From berthold.hoellmann at gl-group.com Mon Sep 1 05:26:33 2008 From: berthold.hoellmann at gl-group.com (=?ISO-8859-15?Q?Berthold_=22H=F6llmann=22?=) Date: Mon, 01 Sep 2008 11:26:33 +0200 Subject: [Numpy-discussion] Problem building NumPy with information from ~/.numpy-site.cfg Message-ID: I am trying to build numpy 1.1.1 with ATLAS support. the f77blas library I am trying to use is named f77blas_gfortran to distinguish it From builds for other compilers. First from the documentation it is unclear whether the default section is named DEFAULT or ALL (I tried both). So I set [DEFAULT] library_dirs = /usr/local/gltools/kinux/lib:/usr/local/gltools/linux_x86_64/lib include_dirs = /usr/local/gltools/include:/usr/local/gltools/linux/include:/usr/local/gltools/linux_x86_64/include [blas_opt] libraries = f77blas_gfortran, cblas, atlas [lapack_opt] libraries = lapack_gfortran, f77blas_gfortran, cblas, atlas in my ~/.numpy-site.cfg but the build process does not find LAPACK support and builds lapack_lite.so with the built in slow lapack. /usr/local/gltools/linux_x86_64/lib/libf77blas_**.so /usr/local/gltools/linux_x86_64/lib/libf77blas_g77.so /usr/local/gltools/linux_x86_64/lib/libf77blas_gfortran.so /usr/local/gltools/linux_x86_64/lib/libf77blas_ifc100.so /usr/local/gltools/linux_x86_64/lib/libf77blas_ifc101.so /usr/local/gltools/linux_x86_64/lib/libf77blas_ifc90.so /usr/local/gltools/linux_x86_64/lib/libf77blas_ifc91.so /usr/local/gltools/linux_x86_64/lib/libf77blas_pg52.so /usr/local/gltools/linux_x86_64/lib/libf77blas_pg60.so /usr/local/gltools/linux_x86_64/lib/libf77blas_pg61.so /usr/local/gltools/linux_x86_64/lib/libf77blas_pg62.so /usr/local/gltools/linux_x86_64/lib/libf77blas_pg70.so /usr/local/gltools/linux_x86_64/lib/libf77blas_pg71.so How do I set the configuration options in ~/.numpy-site.cfg so that the build process finds my libraries? Kind regards Berthold H?llmann -- Germanischer Lloyd AG CAE Development Vorsetzen 35 20459 Hamburg Phone: +49(0)40 36149-7374 Fax: +49(0)40 36149-7320 e-mail: berthold.hoellmann at gl-group.com Internet: http://www.gl-group.com -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 188 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: disclaimer.txt Type: application/octet-stream Size: 2196 bytes Desc: not available URL: From cournapeau at cslab.kecl.ntt.co.jp Mon Sep 1 05:33:33 2008 From: cournapeau at cslab.kecl.ntt.co.jp (David Cournapeau) Date: Mon, 01 Sep 2008 18:33:33 +0900 Subject: [Numpy-discussion] Problem building NumPy with information from ~/.numpy-site.cfg In-Reply-To: References: Message-ID: <1220261613.15478.37.camel@bbc8> On Mon, 2008-09-01 at 11:26 +0200, Berthold "H?llmann" wrote: > I am trying to build numpy 1.1.1 with ATLAS support. the f77blas > library I am trying to use is named f77blas_gfortran to distinguish it > From builds for other compilers. First from the documentation it is > unclear whether the default section is named DEFAULT or ALL (I tried > both). So I set > Hi Berthold, To help you, we need more information: the build log would be useful. Please re-run the python setup.py build from scratch (deleting the build directory): python setup.py build &> build.log Thank you, David From nadavh at visionsense.com Mon Sep 1 10:34:27 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Mon, 01 Sep 2008 17:34:27 +0300 Subject: [Numpy-discussion] Problem building NumPy with information from ~/.numpy-site.cfg In-Reply-To: References: Message-ID: <1220279667.10517.2.camel@nadav.envision.co.il> In the [DEFAULTS] ?library_dirs you wrote kinux instead of linux --- is it on purpose? Nadav On Mon, 2008-09-01 at 11:26 +0200, Berthold "H?llmann" wrote: > I am trying to build numpy 1.1.1 with ATLAS support. the f77blas > library I am trying to use is named f77blas_gfortran to distinguish it > From builds for other compilers. First from the documentation it is > unclear whether the default section is named DEFAULT or ALL (I tried > both). So I set > > [DEFAULT] > library_dirs = /usr/local/gltools/kinux/lib:/usr/local/gltools/linux_x86_64/lib > include_dirs = /usr/local/gltools/include:/usr/local/gltools/linux/include:/usr/local/gltools/linux_x86_64/include > > [blas_opt] > libraries = f77blas_gfortran, cblas, atlas > > [lapack_opt] > libraries = lapack_gfortran, f77blas_gfortran, cblas, atlas > > in my ~/.numpy-site.cfg but the build process does not find LAPACK > support and builds lapack_lite.so with the built in slow lapack. > > /usr/local/gltools/linux_x86_64/lib/libf77blas_**.so > /usr/local/gltools/linux_x86_64/lib/libf77blas_g77.so > /usr/local/gltools/linux_x86_64/lib/libf77blas_gfortran.so > /usr/local/gltools/linux_x86_64/lib/libf77blas_ifc100.so > /usr/local/gltools/linux_x86_64/lib/libf77blas_ifc101.so > /usr/local/gltools/linux_x86_64/lib/libf77blas_ifc90.so > /usr/local/gltools/linux_x86_64/lib/libf77blas_ifc91.so > /usr/local/gltools/linux_x86_64/lib/libf77blas_pg52.so > /usr/local/gltools/linux_x86_64/lib/libf77blas_pg60.so > /usr/local/gltools/linux_x86_64/lib/libf77blas_pg61.so > /usr/local/gltools/linux_x86_64/lib/libf77blas_pg62.so > /usr/local/gltools/linux_x86_64/lib/libf77blas_pg70.so > /usr/local/gltools/linux_x86_64/lib/libf77blas_pg71.so > > How do I set the configuration options in ~/.numpy-site.cfg so that > the build process finds my libraries? > > Kind regards > > Berthold H?llmann > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From berthold.hoellmann at gl-group.com Mon Sep 1 08:35:49 2008 From: berthold.hoellmann at gl-group.com (=?ISO-8859-15?Q?Berthold_=22H=F6llmann=22?=) Date: Mon, 01 Sep 2008 14:35:49 +0200 Subject: [Numpy-discussion] Problem building NumPy with information from ~/.numpy-site.cfg In-Reply-To: <1220279667.10517.2.camel@nadav.envision.co.il> (Nadav Horesh's message of "Mon\, 01 Sep 2008 17\:34\:27 +0300") References: <1220279667.10517.2.camel@nadav.envision.co.il> Message-ID: Nadav Horesh writes: > In the [DEFAULTS] ?library_dirs you wrote kinux instead of linux --- is > it on purpose? It is not on purpose, but for my case it should not matter, because the libraries in `/usr/local/gltools/linux/lib` are 32Bit where I am building a 64Bit version of numpy where the libraries are in `/usr/local/gltools/linux_x86_64/lib`. I am trying to write a '~/.numpy-site.cfg` valid for out 32Bit Linux Systems as well as for our 64Bit Linux System. I get the same NFS mounted home directory on all or our machines. Kind regards Berthold H?llmann -- Germanischer Lloyd AG CAE Development Vorsetzen 35 20459 Hamburg Phone: +49(0)40 36149-7374 Fax: +49(0)40 36149-7320 e-mail: berthold.hoellmann at gl-group.com Internet: http://www.gl-group.com -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 188 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: disclaimer.txt Type: application/octet-stream Size: 2196 bytes Desc: not available URL: From berthe.loic at gmail.com Mon Sep 1 14:49:44 2008 From: berthe.loic at gmail.com (LB) Date: Mon, 1 Sep 2008 11:49:44 -0700 (PDT) Subject: [Numpy-discussion] how to freeze a numpy based script ? Message-ID: Hi, I would like to freeze a numpy based script in order to have an application which could run without having to install numpy and cie. Indeed, I'm not root on the targeted computer and so, I can't easily make a complete install of numpy and scipy. So I decided to test the freeze.py tool shipped with python2.5. To complicate matters, I must say that I only have a local installation of python2.5 and numpy. I used the following command line : > python2.5 ../Python-2.5.1/Tools/freeze/freeze.py ~/Python/numpy/test_freeze.py At first sight, it seems to be fine, as I saw numpy in the liste of frozen dependancies : [...] freezing numpy ... freezing numpy.__config__ ... freezing numpy._import_tools ... freezing numpy.add_newdocs ... freezing numpy.core ... freezing numpy.core._internal ... freezing numpy.core.arrayprint ... freezing numpy.core.defchararray ... freezing numpy.core.defmatrix ... freezing numpy.core.fromnumeric ... freezing numpy.core.info ... freezing numpy.core.memmap ... freezing numpy.core.numeric ... freezing numpy.core.numerictypes ... freezing numpy.core.records ... freezing numpy.ctypeslib ... [...] freezing numpy.version ... But at the end I saw this message : Warning: unknown modules remain: _bisect _csv _ctypes _curses _hashlib _heapq [...] numpy.core._dotblas numpy.core._sort numpy.core.multiarray numpy.core.scalarmath numpy.core.umath numpy.fft.fftpack_lite numpy.lib._compiled_base numpy.linalg.lapack_lite numpy.random.mtrand operator parser pyexpat readline [...] Now run "make" to build the target: test_weibull I runned make without any problem but the final application didn't work : % ./test_freeze Traceback (most recent call last): File "/home/loic/Python/numpy/test_freeze.py", line 8, in import numpy as np [...] File "/home/loic/tmp/bluelagoon/lib/python2.5/site-packages/numpy/ core/__init__.py", line 5, in import multiarray ImportError: No module named multiarray Is there any known issue when freezing a numpy based script ? I should add that I configured my PYTHONPATH to match my local installation > echo $PYTHONPATH /home/loic/lib/python:/home/loic/tmp/bluelagoon/lib/python2.5:/home/ loic/tmp/bluelagoon/lib/python2.5/site-packages/: and this local installation work fine : > python2.5 -c 'import numpy; print numpy.__version__; import numpy.core.multiarray; print "no pb"' 1.2.0b2 no pb Have you got any recipe to freeze numpy based script ? Regards, -- LB From pav at iki.fi Mon Sep 1 16:05:47 2008 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 1 Sep 2008 20:05:47 +0000 (UTC) Subject: [Numpy-discussion] Windows_XP_x86_64_MSVC buildbot needs clean install directory Message-ID: Hi, The buildbot slave Windows_XP_x86_64_MSVC seems to have some bogus files left over in its installation directory (in numpy\doc\) which cause the tests to fail. Removing the installation directory probably helps. -- Pauli Virtanen From millman at berkeley.edu Tue Sep 2 02:22:33 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 1 Sep 2008 23:22:33 -0700 Subject: [Numpy-discussion] numpy failures for rc1 Message-ID: I was hoping to tag the 1.2.0rc1 soon, but I noticed that a couple of failures on the buildbots: http://buildbot.scipy.org/builders/Windows_XP_x86_64_MSVC/builds/701/steps/shell_2/logs/stdio http://buildbot.scipy.org/builders/FreeBSD_64/builds/621/steps/shell_2/logs/stdio Are there any other issues that need to be resolved before tagging the rc? Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From david at ar.media.kyoto-u.ac.jp Tue Sep 2 02:14:58 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 02 Sep 2008 15:14:58 +0900 Subject: [Numpy-discussion] numpy failures for rc1 In-Reply-To: References: Message-ID: <48BCD9E2.5090905@ar.media.kyoto-u.ac.jp> Jarrod Millman wrote: > I was hoping to tag the 1.2.0rc1 soon, but I noticed that a couple of > failures on the buildbots: > http://buildbot.scipy.org/builders/Windows_XP_x86_64_MSVC/builds/701/steps/shell_2/logs/stdio > http://buildbot.scipy.org/builders/FreeBSD_64/builds/621/steps/shell_2/logs/stdio > > The FreeBSD one is a bogus test (again, my fault), which should be trivial to fix. For the Win64 problem, I don't understand how it would appear only on windows; maybe a build bot configuration problem ? cheers, David From theller at ctypes.org Tue Sep 2 02:38:27 2008 From: theller at ctypes.org (Thomas Heller) Date: Tue, 02 Sep 2008 08:38:27 +0200 Subject: [Numpy-discussion] Windows_XP_x86_64_MSVC buildbot needs clean install directory In-Reply-To: References: Message-ID: Pauli Virtanen schrieb: > Hi, > > The buildbot slave Windows_XP_x86_64_MSVC seems to have some bogus files > left over in its installation directory (in numpy\doc\) which cause the > tests to fail. > > Removing the installation directory probably helps. > It did. -- Thanks, Thomas From berthold.hoellmann at gl-group.com Tue Sep 2 02:56:30 2008 From: berthold.hoellmann at gl-group.com (=?ISO-8859-15?Q?Berthold_=22H=F6llmann=22?=) Date: Tue, 02 Sep 2008 08:56:30 +0200 Subject: [Numpy-discussion] Problem building NumPy with information from ~/.numpy-site.cfg In-Reply-To: <1220261613.15478.37.camel@bbc8> (David Cournapeau's message of "Mon\, 01 Sep 2008 18\:33\:33 +0900") References: <1220261613.15478.37.camel@bbc8> Message-ID: David Cournapeau writes: > On Mon, 2008-09-01 at 11:26 +0200, Berthold "H?llmann" wrote: >> I am trying to build numpy 1.1.1 with ATLAS support. the f77blas >> library I am trying to use is named f77blas_gfortran to distinguish it >> From builds for other compilers. First from the documentation it is >> unclear whether the default section is named DEFAULT or ALL (I tried >> both). So I set >> > > Hi Berthold, > > To help you, we need more information: the build log would be useful. > Please re-run the python setup.py build from scratch (deleting the build > directory): > > python setup.py build &> build.log I tried to send the logfile to the list, but the mail still awaits moderation. Kind regards Berthold H?llmann -- Germanischer Lloyd AG CAE Development Vorsetzen 35 20459 Hamburg Phone: +49(0)40 36149-7374 Fax: +49(0)40 36149-7320 e-mail: berthold.hoellmann at gl-group.com Internet: http://www.gl-group.com -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 188 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: disclaimer.txt Type: application/octet-stream Size: 2196 bytes Desc: not available URL: From david at ar.media.kyoto-u.ac.jp Tue Sep 2 02:50:29 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 02 Sep 2008 15:50:29 +0900 Subject: [Numpy-discussion] numpy failures for rc1 In-Reply-To: <48BCD9E2.5090905@ar.media.kyoto-u.ac.jp> References: <48BCD9E2.5090905@ar.media.kyoto-u.ac.jp> Message-ID: <48BCE235.2030409@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > > The FreeBSD one is a bogus test (again, my fault), which should be > trivial to fix. Fixed in r5741 cheers, David From matthieu.brucher at gmail.com Tue Sep 2 03:09:51 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 2 Sep 2008 09:09:51 +0200 Subject: [Numpy-discussion] numpy failures for rc1 In-Reply-To: <48BCD9E2.5090905@ar.media.kyoto-u.ac.jp> References: <48BCD9E2.5090905@ar.media.kyoto-u.ac.jp> Message-ID: I think there is a thread on that matter on the list, and it seems to be fixed now ;) Cheers, Matthieu 2008/9/2, David Cournapeau : > Jarrod Millman wrote: > > I was hoping to tag the 1.2.0rc1 soon, but I noticed that a couple of > > failures on the buildbots: > > http://buildbot.scipy.org/builders/Windows_XP_x86_64_MSVC/builds/701/steps/shell_2/logs/stdio > > http://buildbot.scipy.org/builders/FreeBSD_64/builds/621/steps/shell_2/logs/stdio > > > > > > The FreeBSD one is a bogus test (again, my fault), which should be > trivial to fix. For the Win64 problem, I don't understand how it would > appear only on windows; maybe a build bot configuration problem ? > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher From david at ar.media.kyoto-u.ac.jp Tue Sep 2 03:15:01 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 02 Sep 2008 16:15:01 +0900 Subject: [Numpy-discussion] numpy failures for rc1 In-Reply-To: References: Message-ID: <48BCE7F5.2040908@ar.media.kyoto-u.ac.jp> Jarrod Millman wrote: > I was hoping to tag the 1.2.0rc1 soon, but I noticed that a couple of > failures on the buildbots: > http://buildbot.scipy.org/builders/Windows_XP_x86_64_MSVC/builds/701/steps/shell_2/logs/stdio > http://buildbot.scipy.org/builders/FreeBSD_64/builds/621/steps/shell_2/logs/stdio > > Both are green now (as well as all other ones). cheers, David From charlesr.harris at gmail.com Tue Sep 2 13:22:36 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 2 Sep 2008 11:22:36 -0600 Subject: [Numpy-discussion] Problem building NumPy with information from ~/.numpy-site.cfg In-Reply-To: References: <1220261613.15478.37.camel@bbc8> Message-ID: On Tue, Sep 2, 2008 at 12:56 AM, Berthold "H?llmann" < berthold.hoellmann at gl-group.com> wrote: > David Cournapeau writes: > > > On Mon, 2008-09-01 at 11:26 +0200, Berthold "H?llmann" wrote: > >> I am trying to build numpy 1.1.1 with ATLAS support. the f77blas > >> library I am trying to use is named f77blas_gfortran to distinguish it > >> From builds for other compilers. First from the documentation it is > >> unclear whether the default section is named DEFAULT or ALL (I tried > >> both). So I set > >> > > > > Hi Berthold, > > > > To help you, we need more information: the build log would be > useful. > > Please re-run the python setup.py build from scratch (deleting the build > > directory): > > > > python setup.py build &> build.log > > I tried to send the logfile to the list, but the mail still awaits > moderation. > Yeah, the list has a rather small size limit. Zip it up and send it to David directly. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Tue Sep 2 15:54:15 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 2 Sep 2008 12:54:15 -0700 Subject: [Numpy-discussion] NumPy 1.2.0rc1 to be tagged in 5 hours Message-ID: Hello, I am planning to tag 1.2.0rc1 in 5 hours. The tests are passing on the buildbots and I don't know of anything else that needs to be fixed before making the final 1.2.0 release, which will come out after the release candidate has been more widely tested. If you know of any release blockers, please let me know ASAP. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From doutriaux1 at llnl.gov Tue Sep 2 17:40:54 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Tue, 02 Sep 2008 14:40:54 -0700 Subject: [Numpy-discussion] numpy 1.1.1 fails because of missing md5 In-Reply-To: References: <94156B26-BC6B-4EFA-90DC-7E60B8D57BF2@llnl.gov> <48BD9387.70408@llnl.gov> <48BDAB9D.4050002@llnl.gov> Message-ID: <48BDB2E6.6020503@llnl.gov> Joseph, Ok all failed because numpy couldn't build... It's looking for md5 on my machine i type: whereis md5 md5: /usr/include/md5.h /usr/share/man/man1/md5.1ssl.gz /usr/share/man/man3/md5.3ssl.gz I'm ccing the numpy discussion list on this. The numpy we're trying to build is 1.1.1 I'm attaching the numpy log. Is it fixed in 1.2 ? Joseph in the mean time you might want to try to install md5 on your system. C. Joseph Hargitai wrote: > here are the logs - > > OS: > 2.6.9-55.ELsmp #1 SMP Fri Apr 20 16:36:54 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux > > Rocks 4.3 with Redhat installation cluster. > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: numpy.LOG Type: text/x-log Size: 20380 bytes Desc: not available URL: From robert.kern at gmail.com Tue Sep 2 17:46:30 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 2 Sep 2008 16:46:30 -0500 Subject: [Numpy-discussion] numpy 1.1.1 fails because of missing md5 In-Reply-To: <48BDB2E6.6020503@llnl.gov> References: <94156B26-BC6B-4EFA-90DC-7E60B8D57BF2@llnl.gov> <48BD9387.70408@llnl.gov> <48BDAB9D.4050002@llnl.gov> <48BDB2E6.6020503@llnl.gov> Message-ID: <3d375d730809021446m258d99acqe83151d94a3cd932@mail.gmail.com> On Tue, Sep 2, 2008 at 16:40, Charles Doutriaux wrote: > Joseph, > > Ok all failed because numpy couldn't build... It's looking for md5 That's part of the standard library. Please check your Python installation. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From emanuele at relativita.com Tue Sep 2 18:08:42 2008 From: emanuele at relativita.com (Emanuele Olivetti) Date: Wed, 03 Sep 2008 00:08:42 +0200 Subject: [Numpy-discussion] distance matrix and (weighted) p-norm Message-ID: <48BDB96A.4060506@relativita.com> Hi, I'm trying to compute the distance matrix (weighted p-norm [*]) between two sets of vectors (data1 and data2). Example: import numpy as N p = 3.0 data1 = N.random.randn(100,20) data2 = N.random.randn(80,20) weight = N.random.rand(20) distance_matrix = N.zeros((data1.shape[0],data2.shape[0])) for d in range(data1.shape[1]): distance_matrix += (N.abs(N.subtract.outer(data1[:,d],data2[:,d]))*weight[d])**p pass distance_matrix = distance_matrix**(1.0/p) Is there a way to speed up the for loop? When the dimension of the vectors becomes big (e.g. >1000) the for loop becomes really annoying. Thanks, Emanuele [*] : ||x - x'||_w = (\sum_{i=1...N} (w_i*|x_i - x'_i|)**p)**(1/p) From david at ar.media.kyoto-u.ac.jp Tue Sep 2 22:06:47 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 03 Sep 2008 11:06:47 +0900 Subject: [Numpy-discussion] distance matrix and (weighted) p-norm In-Reply-To: <48BDB96A.4060506@relativita.com> References: <48BDB96A.4060506@relativita.com> Message-ID: <48BDF137.7060702@ar.media.kyoto-u.ac.jp> Emanuele Olivetti wrote: > Hi, > > I'm trying to compute the distance matrix (weighted p-norm [*]) > between two sets of vectors (data1 and data2). Example: > You may want to look at scipy.cluster.distance, which has a bunch of distance matrix implementation. I believe most of them have optional compiled version, for fast execution. cheers, David From millman at berkeley.edu Wed Sep 3 02:16:09 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 2 Sep 2008 23:16:09 -0700 Subject: [Numpy-discussion] 1.2.0rc1 tagged! Message-ID: Hello, The 1.2.0rc1 is now available: http://svn.scipy.org/svn/numpy/tags/1.2.0rc1 The source tarball is here: https://cirl.berkeley.edu/numpy/numpy-1.2.0rc1.tar.gz Here is the universal Mac binary: https://cirl.berkeley.edu/numpy/numpy-1.2.0rc1-py2.5-macosx10.5.dmg Here are the Window's binaries: http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.2.0rc1-win32-superpack-python2.4.exe http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.2.0rc1-win32-superpack-python2.5.exe Here are the release notes: http://scipy.org/scipy/numpy/milestone/1.2.0 Please test this release ASAP and let us know if there are any problems. If there are no show stoppers, this will likely become the 1.2.0 release. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From david at ar.media.kyoto-u.ac.jp Wed Sep 3 02:24:50 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 03 Sep 2008 15:24:50 +0900 Subject: [Numpy-discussion] 1.2.0rc1 tagged! In-Reply-To: References: Message-ID: <48BE2DB2.5060304@ar.media.kyoto-u.ac.jp> Jarrod Millman wrote: > Hello, > > The 1.2.0rc1 is now available: > http://svn.scipy.org/svn/numpy/tags/1.2.0rc1 > > The source tarball is here: > https://cirl.berkeley.edu/numpy/numpy-1.2.0rc1.tar.gz > > Here is the universal Mac binary: > https://cirl.berkeley.edu/numpy/numpy-1.2.0rc1-py2.5-macosx10.5.dmg > > Here are the Window's binaries: > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.2.0rc1-win32-superpack-python2.4.exe > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.2.0rc1-win32-superpack-python2.5.exe > > Here are the release notes: > http://scipy.org/scipy/numpy/milestone/1.2.0 > > Please test this release ASAP and let us know if there are any > problems. If there are no show stoppers, this will likely become the > 1.2.0 release. I just wanted to point that on windows, there are two known test failures concerning long double printing. They should only concern the printing, and do not affect computation; if you don't use long double on windows, this does not affect you at all. Unfortunately, I did not find an easy way to fix it (I could not make the obvious fix to work, and the underlying cause is that our configuration system became a bit of a spaghetti code in this case, which I am hoping to fix for 1.3, but is too big of a change for 1.2). cheers, David From hanni.ali at gmail.com Wed Sep 3 05:04:06 2008 From: hanni.ali at gmail.com (Hanni Ali) Date: Wed, 3 Sep 2008 10:04:06 +0100 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray Message-ID: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> Hi, I have encountered a worrying problem, during migration of software from numarray to numpy, perhaps someone could help me determine how this could be addressed. I have a large array or values 10000 long 12 items per line. The matrix contains floats, dtype=float32 in numpy and type=Float32 in numarray. When I perform a mean of one of the columns we observe a discrepancies in the output values. numarray: >>> port_result.agg_matrix._array[::,2].mean() 193955925.49500328 numpy: >>> port_result.agg_matrix._array[::,2].mean() 193954536.48896 If we examine a specific line in the matrix the arrays appear identical: numarray: >>> port_result.agg_matrix[0] array([ 2.11549568e+08, 4.03735232e+08, 8.47466400e+07, 3.99625600e+07, 7.99853550e+06, 6.68247100e+06, 0.00000000e+00, 1.00018790e+07, 3.43065200e+07, 1.75261240e+07, 4.89246450e+06, 2.06562788e+06], type=Float32) numpy: >>> port_result.agg_matrix[0] array([ 2.11549568e+08, 4.03735232e+08, 8.47466400e+07, 3.99625600e+07, 7.99853550e+06, 6.68247100e+06, 0.00000000e+00, 1.00018790e+07, 3.43065200e+07, 1.75261240e+07, 4.89246450e+06, 2.06562788e+06], dtype=float32) However when examining a specific item numpy appears to report a value to 8 significant figures regardless of the true value, whereas numarray reported the full value, however if I cast the output as a float the full value is present, just not being output. Could this explain the difference in the mean values? How can I get numpy to always provide the exact value in the array, so behave in the same manner as numarray? numarray: >>> port_result.agg_matrix[0][4] 7998535.5 >>> port_result.agg_matrix[0][11] 2065627.875 numpy: >>> port_result.agg_matrix[0][4] 7998535.5 >>> port_result.agg_matrix[0][11] 2065627.9 >>> float(port_result.agg_matrix[0][4]) 7998535.5 >>> float(port_result.agg_matrix[0][11]) 2065627.875 I appreciate any help anyone can give, thank you. Hanni -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Wed Sep 3 05:12:28 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 3 Sep 2008 11:12:28 +0200 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> Message-ID: Hi, I can't help you with the first issues, but the display has nothing to do with the quality of the computation. Numpy only prints a part of a float value, but fir the computations, it obviously uses the correct value. All this can be parametrized by using set_printoptions(). Matthieu 2008/9/3, Hanni Ali : > Hi, > > I have encountered a worrying problem, during migration of software from > numarray to numpy, perhaps someone could help me determine how this could be > addressed. > > I have a large array or values 10000 long 12 items per line. The matrix > contains floats, dtype=float32 in numpy and type=Float32 in numarray. > > When I perform a mean of one of the columns we observe a discrepancies in > the output values. > > numarray: > >>> port_result.agg_matrix._array[::,2].mean() > 193955925.49500328 > > numpy: > > >>> port_result.agg_matrix._array[::,2].mean() > 193954536.48896 > > If we examine a specific line in the matrix the arrays appear identical: > > numarray: > >>> port_result.agg_matrix[0] > array([ 2.11549568e+08, 4.03735232e+08, 8.47466400e+07, > 3.99625600e+07, 7.99853550e+06, 6.68247100e+06, > 0.00000000e+00, 1.00018790e+07, 3.43065200e+07, > 1.75261240e+07, 4.89246450e+06, 2.06562788e+06], type=Float32) > > numpy: > >>> port_result.agg_matrix[0] > array([ 2.11549568e+08, 4.03735232e+08, 8.47466400e+07, > 3.99625600e+07, 7.99853550e+06, 6.68247100e+06, > 0.00000000e+00, 1.00018790e+07, 3.43065200e+07, > 1.75261240e+07, 4.89246450e+06, 2.06562788e+06], dtype=float32) > > However when examining a specific item numpy appears to report a value to 8 > significant figures regardless of the true value, whereas numarray reported > the full value, however if I cast the output as a float the full value is > present, just not being output. Could this explain the difference in the > mean values? How can I get numpy to always provide the exact value in the > array, so behave in the same manner as numarray? > > numarray: > >>> port_result.agg_matrix[0][4] > 7998535.5 > >>> port_result.agg_matrix[0][11] > 2065627.875 > > numpy: > >>> port_result.agg_matrix[0][4] > 7998535.5 > >>> port_result.agg_matrix[0][11] > 2065627.9 > >>> float(port_result.agg_matrix[0][4]) > 7998535.5 > >>> float(port_result.agg_matrix[0][11]) > 2065627.875 > > I appreciate any help anyone can give, thank you. > > Hanni > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- French PhD student Information System Engineer Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From hanni.ali at gmail.com Wed Sep 3 05:31:05 2008 From: hanni.ali at gmail.com (Hanni Ali) Date: Wed, 3 Sep 2008 10:31:05 +0100 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> Message-ID: <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> Hi Matthieu, I thought as much, regarding the computations, but was just presenting the information. Thanks for the set_printoptions but it doesn't seem to apply when accessing a specific item: >>> numpy.set_printoptions(precision=12) >>> port_result.agg_matrix[0] array([ 2.115495680000e+08, 4.037352320000e+08, 8.474664000000e+07, 3.996256000000e+07, 7.998535500000e+06, 6.682471000000e+06, 0.000000000000e+00, 1.000187900000e+07, 3.430652000000e+07, 1.752612400000e+07, 4.892464500000e+06, 2.065627875000e+06], dtype=float32) >>> port_result.agg_matrix[0][11] 2065627.9 No change in the vale output from a specific item in the matrix. Am I missing something? Hanni 2008/9/3 Matthieu Brucher > Hi, > > I can't help you with the first issues, but the display has nothing to > do with the quality of the computation. Numpy only prints a part of a > float value, but fir the computations, it obviously uses the correct > value. All this can be parametrized by using set_printoptions(). > > Matthieu > > 2008/9/3, Hanni Ali : > > Hi, > > > > I have encountered a worrying problem, during migration of software from > > numarray to numpy, perhaps someone could help me determine how this could > be > > addressed. > > > > I have a large array or values 10000 long 12 items per line. The matrix > > contains floats, dtype=float32 in numpy and type=Float32 in numarray. > > > > When I perform a mean of one of the columns we observe a discrepancies in > > the output values. > > > > numarray: > > >>> port_result.agg_matrix._array[::,2].mean() > > 193955925.49500328 > > > > numpy: > > > > >>> port_result.agg_matrix._array[::,2].mean() > > 193954536.48896 > > > > If we examine a specific line in the matrix the arrays appear identical: > > > > numarray: > > >>> port_result.agg_matrix[0] > > array([ 2.11549568e+08, 4.03735232e+08, 8.47466400e+07, > > 3.99625600e+07, 7.99853550e+06, 6.68247100e+06, > > 0.00000000e+00, 1.00018790e+07, 3.43065200e+07, > > 1.75261240e+07, 4.89246450e+06, 2.06562788e+06], > type=Float32) > > > > numpy: > > >>> port_result.agg_matrix[0] > > array([ 2.11549568e+08, 4.03735232e+08, 8.47466400e+07, > > 3.99625600e+07, 7.99853550e+06, 6.68247100e+06, > > 0.00000000e+00, 1.00018790e+07, 3.43065200e+07, > > 1.75261240e+07, 4.89246450e+06, 2.06562788e+06], > dtype=float32) > > > > However when examining a specific item numpy appears to report a value to > 8 > > significant figures regardless of the true value, whereas numarray > reported > > the full value, however if I cast the output as a float the full value is > > present, just not being output. Could this explain the difference in the > > mean values? How can I get numpy to always provide the exact value in the > > array, so behave in the same manner as numarray? > > > > numarray: > > >>> port_result.agg_matrix[0][4] > > 7998535.5 > > >>> port_result.agg_matrix[0][11] > > 2065627.875 > > > > numpy: > > >>> port_result.agg_matrix[0][4] > > 7998535.5 > > >>> port_result.agg_matrix[0][11] > > 2065627.9 > > >>> float(port_result.agg_matrix[0][4]) > > 7998535.5 > > >>> float(port_result.agg_matrix[0][11]) > > 2065627.875 > > > > I appreciate any help anyone can give, thank you. > > > > Hanni > > > > > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > -- > French PhD student > Information System Engineer > Website: http://matthieu-brucher.developpez.com/ > Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn: http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Wed Sep 3 05:26:05 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 03 Sep 2008 18:26:05 +0900 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> Message-ID: <48BE582D.2010606@ar.media.kyoto-u.ac.jp> Hanni Ali wrote: > Hi Matthieu, > > I thought as much, regarding the computations, but was just presenting > the information. Is your matrix available somewhere so that we can reproduce the problem ? Off-hand, I can't see any explanation, but I am not familiar with numarray, so maybe I am missing something obvious, cheers, David From sebastian at sipsolutions.net Wed Sep 3 05:45:24 2008 From: sebastian at sipsolutions.net (Sebastian Stephan Berg) Date: Wed, 03 Sep 2008 11:45:24 +0200 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> Message-ID: <1220435124.20211.10.camel@sebook> Hi, just guessing here. But numarray seems to calculate the result in a bigger dataype, while numpy uses float32 which is the input arrays size (at least I thought so, trying it confused me right now ...). In any case, maybe the difference will be gone if you use .mean(dtype='float64') (or whatever dtype numarray actually uses, which seems to be "numarray.MaximumType(a.type())" where a is the array to take the mean). Sebastian From hanni.ali at gmail.com Wed Sep 3 06:22:56 2008 From: hanni.ali at gmail.com (Hanni Ali) Date: Wed, 3 Sep 2008 11:22:56 +0100 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <48BE582D.2010606@ar.media.kyoto-u.ac.jp> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <48BE582D.2010606@ar.media.kyoto-u.ac.jp> Message-ID: <789d27b10809030322v39f298b3sedb5e288eadb4261@mail.gmail.com> I'm afraid the matrix is not available anywhere and I would not be able to make it available. However I can demonstrate by simply generating a random number and filling a 10x10 matrix with it. I generated a random number in numpy and used that to do the same exercise in numarray. In numpy: >>> matrix = numpy.zeros((10,10),'f') >>> matrix array([[ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]], dtype=float32) >>> number = numpy.random.random_sample() >>> number 0.41582016134572475 >>> matrix.setfield(number,'f',0) >>> matrix array([[ 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567], [ 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567], [ 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567], [ 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567], [ 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567], [ 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567], [ 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567], [ 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567], [ 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567], [ 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567, 0.415820151567]], dtype=float32) >>> matrix.mean() 0.41582069396972654 In numarray: >>> matrix = numarray.zeros((10,10),'f') >>> matrix array([[ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]], type=Float32) >>> number = 0.41582015156745911 >>> for i, item in enumerate(matrix): ... matrix[i] = number ... >>> matrix array([[ 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015], [ 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015], [ 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015], [ 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015], [ 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015], [ 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015], [ 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015], [ 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015], [ 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015], [ 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015, 0.41582015]], type=Float32) >>> matrix.mean() 0.41582015156745911 2008/9/3 David Cournapeau > Hanni Ali wrote: > > Hi Matthieu, > > > > I thought as much, regarding the computations, but was just presenting > > the information. > > Is your matrix available somewhere so that we can reproduce the problem > ? Off-hand, I can't see any explanation, but I am not familiar with > numarray, so maybe I am missing something obvious, > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hanni.ali at gmail.com Wed Sep 3 06:26:43 2008 From: hanni.ali at gmail.com (Hanni Ali) Date: Wed, 3 Sep 2008 11:26:43 +0100 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <1220435124.20211.10.camel@sebook> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> Message-ID: <789d27b10809030326w1000132frb0d530f1dfcd4729@mail.gmail.com> Sebastian you legend, that seems to be it. Thank you very much. >>> matrix.mean(dtype='float64') 0.41582015156745911 What seems odd is that numpy doesn't do this on it's own... 2008/9/3 Sebastian Stephan Berg > Hi, > > just guessing here. But numarray seems to calculate the result in a > bigger dataype, while numpy uses float32 which is the input arrays size > (at least I thought so, trying it confused me right now ...). In any > case, maybe the difference will be gone if you > use .mean(dtype='float64') (or whatever dtype numarray actually uses, > which seems to be "numarray.MaximumType(a.type())" where a is the array > to take the mean). > > Sebastian > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Wed Sep 3 06:42:16 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 3 Sep 2008 12:42:16 +0200 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <789d27b10809030326w1000132frb0d530f1dfcd4729@mail.gmail.com> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> <789d27b10809030326w1000132frb0d530f1dfcd4729@mail.gmail.com> Message-ID: It should never do some black magic without telling you. People are concerned by memory consumption, so if you use more memory than what you think, you can encounter bugs. Least surprise is always better ;) Matthieu 2008/9/3, Hanni Ali : > Sebastian you legend, that seems to be it. > > Thank you very much. > > >>> matrix.mean(dtype='float64') > 0.41582015156745911 > > What seems odd is that numpy doesn't do this on it's own... > > > > 2008/9/3 Sebastian Stephan Berg > > > Hi, > > > > just guessing here. But numarray seems to calculate the result in a > > bigger dataype, while numpy uses float32 which is the input arrays size > > (at least I thought so, trying it confused me right now ...). In any > > case, maybe the difference will be gone if you > > use .mean(dtype='float64') (or whatever dtype numarray actually uses, > > which seems to be "numarray.MaximumType(a.type())" where > a is the array > > to take the mean). > > > > Sebastian > > > > > > > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- French PhD student Information System Engineer Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From hanni.ali at gmail.com Wed Sep 3 06:43:34 2008 From: hanni.ali at gmail.com (Hanni Ali) Date: Wed, 3 Sep 2008 11:43:34 +0100 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <789d27b10809030326w1000132frb0d530f1dfcd4729@mail.gmail.com> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> <789d27b10809030326w1000132frb0d530f1dfcd4729@mail.gmail.com> Message-ID: <789d27b10809030343n5cc5f509iafcbd771ac936e50@mail.gmail.com> Also can you think of a way either dtype is always float64? I have a lot of functions and to add dtype='float64' would require *loads* of testing, whereas if I can set it centrally on the matrix or in the environment that would be so much easier. Hanni 2008/9/3 Hanni Ali > Sebastian you legend, that seems to be it. > > Thank you very much. > > >>> matrix.mean(dtype='float64') > 0.41582015156745911 > > What seems odd is that numpy doesn't do this on it's own... > > > > 2008/9/3 Sebastian Stephan Berg > > Hi, >> >> just guessing here. But numarray seems to calculate the result in a >> bigger dataype, while numpy uses float32 which is the input arrays size >> (at least I thought so, trying it confused me right now ...). In any >> case, maybe the difference will be gone if you >> use .mean(dtype='float64') (or whatever dtype numarray actually uses, >> which seems to be "numarray.MaximumType(a.type())" where a is the array >> to take the mean). >> >> Sebastian >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hanni.ali at gmail.com Wed Sep 3 06:44:42 2008 From: hanni.ali at gmail.com (Hanni Ali) Date: Wed, 3 Sep 2008 11:44:42 +0100 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> <789d27b10809030326w1000132frb0d530f1dfcd4729@mail.gmail.com> Message-ID: <789d27b10809030344k7956b1e8lcdc6b42e2866729f@mail.gmail.com> Understood, but I would generally be more concerned with accuracy than memory? 2008/9/3 Matthieu Brucher > It should never do some black magic without telling you. > People are concerned by memory consumption, so if you use more memory > than what you think, you can encounter bugs. Least surprise is always > better ;) > > Matthieu > > 2008/9/3, Hanni Ali : > > Sebastian you legend, that seems to be it. > > > > Thank you very much. > > > > >>> matrix.mean(dtype='float64') > > 0.41582015156745911 > > > > What seems odd is that numpy doesn't do this on it's own... > > > > > > > > 2008/9/3 Sebastian Stephan Berg > > > > > Hi, > > > > > > just guessing here. But numarray seems to calculate the result in a > > > bigger dataype, while numpy uses float32 which is the input arrays size > > > (at least I thought so, trying it confused me right now ...). In any > > > case, maybe the difference will be gone if you > > > use .mean(dtype='float64') (or whatever dtype numarray actually uses, > > > which seems to be "numarray.MaximumType(a.type())" where > > a is the array > > > to take the mean). > > > > > > Sebastian > > > > > > > > > > > > > > > _______________________________________________ > > > Numpy-discussion mailing list > > > Numpy-discussion at scipy.org > > > > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > -- > French PhD student > Information System Engineer > Website: http://matthieu-brucher.developpez.com/ > Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn: http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Wed Sep 3 06:29:16 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 03 Sep 2008 19:29:16 +0900 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <789d27b10809030322v39f298b3sedb5e288eadb4261@mail.gmail.com> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <48BE582D.2010606@ar.media.kyoto-u.ac.jp> <789d27b10809030322v39f298b3sedb5e288eadb4261@mail.gmail.com> Message-ID: <48BE66FC.5040202@ar.media.kyoto-u.ac.jp> Hanni Ali wrote: > I'm afraid the matrix is not available anywhere and I would not be > able to make it available. > Forget it, Sebastian is right. I was confused by the range of the error, but the error between two floating point numbers is indeed 1e-7 for float on most runtimes (FLT_EPS in C, which is the minimal value such as 1. + FLT_EPS != 1.) I am not sure whether we should define the accumulator to a double in the float case by default; generally, you use float for saving memory and computing speed, and you lose quite a bit speed-wise by using a double accumulator. FWIW, matlab behaves as well (not that it is a justification by itself, but at least it should not surprise matlab users). cheers, David From matthieu.brucher at gmail.com Wed Sep 3 06:47:49 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 3 Sep 2008 12:47:49 +0200 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <789d27b10809030343n5cc5f509iafcbd771ac936e50@mail.gmail.com> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> <789d27b10809030326w1000132frb0d530f1dfcd4729@mail.gmail.com> <789d27b10809030343n5cc5f509iafcbd771ac936e50@mail.gmail.com> Message-ID: By default, numpy uses float64, but you told it to use float32 ;) Matthieu 2008/9/3, Hanni Ali : > Also can you think of a way either dtype is always float64? I have a lot of > functions and to add dtype='float64' would require *loads* of testing, > whereas if I can set it centrally on the matrix or in the environment that > would be so much easier. > > Hanni > > > 2008/9/3 Hanni Ali > > > > Sebastian you legend, that seems to be it. > > > > > > Thank you very much. > > > > >>> matrix.mean(dtype='float64') > > 0.41582015156745911 > > > > What seems odd is that numpy doesn't do this on it's own... > > > > > > > > > > 2008/9/3 Sebastian Stephan Berg > > > > > > > > > > > Hi, > > > > > > just guessing here. But numarray seems to calculate the result in a > > > bigger dataype, while numpy uses float32 which is the input arrays size > > > (at least I thought so, trying it confused me right now ...). In any > > > case, maybe the difference will be gone if you > > > use .mean(dtype='float64') (or whatever dtype numarray actually uses, > > > which seems to be "numarray.MaximumType(a.type())" > where a is the array > > > to take the mean). > > > > > > Sebastian > > > > > > > > > > > > > > > _______________________________________________ > > > Numpy-discussion mailing list > > > Numpy-discussion at scipy.org > > > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- French PhD student Information System Engineer Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From david at ar.media.kyoto-u.ac.jp Wed Sep 3 06:31:09 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 03 Sep 2008 19:31:09 +0900 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <789d27b10809030344k7956b1e8lcdc6b42e2866729f@mail.gmail.com> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> <789d27b10809030326w1000132frb0d530f1dfcd4729@mail.gmail.com> <789d27b10809030344k7956b1e8lcdc6b42e2866729f@mail.gmail.com> Message-ID: <48BE676D.2030201@ar.media.kyoto-u.ac.jp> Hanni Ali wrote: > Understood, but I would generally be more concerned with accuracy than > memory? It is a tradeof: you can choose accuracy if you want, but by using float32, you are already kind of hinting that you care about memory and speed (otherwise, why not using double, which is the default dtype for almost everything in numpy ?) cheers, David From hanni.ali at gmail.com Wed Sep 3 06:49:26 2008 From: hanni.ali at gmail.com (Hanni Ali) Date: Wed, 3 Sep 2008 11:49:26 +0100 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> <789d27b10809030326w1000132frb0d530f1dfcd4729@mail.gmail.com> <789d27b10809030343n5cc5f509iafcbd771ac936e50@mail.gmail.com> Message-ID: <789d27b10809030349y287592e3o9320f617e69e2069@mail.gmail.com> Oh ok, I shall have to find where I did that then. Thanks 2008/9/3 Matthieu Brucher > By default, numpy uses float64, but you told it to use float32 ;) > > Matthieu > > 2008/9/3, Hanni Ali : > > Also can you think of a way either dtype is always float64? I have a lot > of > > functions and to add dtype='float64' would require *loads* of testing, > > whereas if I can set it centrally on the matrix or in the environment > that > > would be so much easier. > > > > Hanni > > > > > > 2008/9/3 Hanni Ali > > > > > > Sebastian you legend, that seems to be it. > > > > > > > > > Thank you very much. > > > > > > >>> matrix.mean(dtype='float64') > > > 0.41582015156745911 > > > > > > What seems odd is that numpy doesn't do this on it's own... > > > > > > > > > > > > > > > 2008/9/3 Sebastian Stephan Berg > > > > > > > > > > > > > > > > Hi, > > > > > > > > just guessing here. But numarray seems to calculate the result in a > > > > bigger dataype, while numpy uses float32 which is the input arrays > size > > > > (at least I thought so, trying it confused me right now ...). In any > > > > case, maybe the difference will be gone if you > > > > use .mean(dtype='float64') (or whatever dtype numarray actually uses, > > > > which seems to be "numarray.MaximumType(a.type())" > > where a is the array > > > > to take the mean). > > > > > > > > Sebastian > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > Numpy-discussion mailing list > > > > Numpy-discussion at scipy.org > > > > > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > > > > > > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > -- > French PhD student > Information System Engineer > Website: http://matthieu-brucher.developpez.com/ > Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn: http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hanni.ali at gmail.com Wed Sep 3 06:51:26 2008 From: hanni.ali at gmail.com (Hanni Ali) Date: Wed, 3 Sep 2008 11:51:26 +0100 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <48BE676D.2030201@ar.media.kyoto-u.ac.jp> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> <789d27b10809030326w1000132frb0d530f1dfcd4729@mail.gmail.com> <789d27b10809030344k7956b1e8lcdc6b42e2866729f@mail.gmail.com> <48BE676D.2030201@ar.media.kyoto-u.ac.jp> Message-ID: <789d27b10809030351r8070551tc689abc60bdbc03@mail.gmail.com> We used to care about memory when we were running on 32-bit platforms, but with the move to 64-bit, enabled by the current work, the issue is removed and I will probably be changing everything for more accuracy. Thanks Hanni 2008/9/3 David Cournapeau > Hanni Ali wrote: > > Understood, but I would generally be more concerned with accuracy than > > memory? > > It is a tradeof: you can choose accuracy if you want, but by using > float32, you are already kind of hinting that you care about memory and > speed (otherwise, why not using double, which is the default dtype for > almost everything in numpy ?) > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From emanuele at relativita.com Wed Sep 3 09:13:41 2008 From: emanuele at relativita.com (Emanuele Olivetti) Date: Wed, 03 Sep 2008 15:13:41 +0200 Subject: [Numpy-discussion] distance matrix and (weighted) p-norm In-Reply-To: <48BDF137.7060702@ar.media.kyoto-u.ac.jp> References: <48BDB96A.4060506@relativita.com> <48BDF137.7060702@ar.media.kyoto-u.ac.jp> Message-ID: <48BE8D85.6020604@relativita.com> David Cournapeau wrote: > Emanuele Olivetti wrote: >> Hi, >> >> I'm trying to compute the distance matrix (weighted p-norm [*]) >> between two sets of vectors (data1 and data2). Example: >> > > You may want to look at scipy.cluster.distance, which has a bunch of > distance matrix implementation. I believe most of them have optional > compiled version, for fast execution. Thanks for the pointer but the distance subpackage in cluster is about the distance matrix of vectors in one set of vectors. So the resulting matrix is symmetric. I need to compute distances between two different sets of vectors (i.e. a non-symmetric distance matrix). It is not clear to me how to use it in my case. Then cluster.distance offers: 1) slow python double for loop for computing each entry of the matrix 2) or fast C implementation (numpy/cluster/distance/src/distance.c). I guess I need to extend distance.c, then work on the wrapper and then on distance.py. But after that it would be meaningless to have those distances under 'cluster', since clustering doesn't need distances between two sets of vectors. In my original post I was looking for a fast python/numpy implementation for my code. In special cases (like p==2, i.e. standard weighted euclidean distance) there is a superfast implementation (e.g., see "Fastest distance matrix calc" 2007 thread). But I'm not able to find something similar for the general case. Any other suggestions on how to speed up my example? Thanks, Emanuele From david at ar.media.kyoto-u.ac.jp Wed Sep 3 09:16:42 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 03 Sep 2008 22:16:42 +0900 Subject: [Numpy-discussion] distance matrix and (weighted) p-norm In-Reply-To: <48BE8D85.6020604@relativita.com> References: <48BDB96A.4060506@relativita.com> <48BDF137.7060702@ar.media.kyoto-u.ac.jp> <48BE8D85.6020604@relativita.com> Message-ID: <48BE8E3A.8070100@ar.media.kyoto-u.ac.jp> Emanuele Olivetti wrote: > > Thanks for the pointer but the distance subpackage in cluster is about > the distance matrix of vectors in one set of vectors. So the resulting > matrix is symmetric. I need to compute distances between two > different sets of vectors (i.e. a non-symmetric distance matrix). > It is not clear to me how to use it in my case. > You may need to extend the code, indeed (although I am more or less responsible for scipy.cluster these days, I have not looked carefully at all the code in distance yet). > Then cluster.distance offers: > 1) slow python double for loop for computing each entry of the matrix > 2) or fast C implementation (numpy/cluster/distance/src/distance.c). > > I guess I need to extend distance.c, then work on the wrapper and then > on distance.py. But after that it would be meaningless to have those > distances under 'cluster', since clustering doesn't need distances between > two sets of vectors. > FWIW, distance is deemed to move to a separate package, because distance computation is useful in other contexts than clustering. cheers, David From emanuele at relativita.com Wed Sep 3 10:22:36 2008 From: emanuele at relativita.com (Emanuele Olivetti) Date: Wed, 03 Sep 2008 16:22:36 +0200 Subject: [Numpy-discussion] distance matrix and (weighted) p-norm In-Reply-To: <48BE8E3A.8070100@ar.media.kyoto-u.ac.jp> References: <48BDB96A.4060506@relativita.com> <48BDF137.7060702@ar.media.kyoto-u.ac.jp> <48BE8D85.6020604@relativita.com> <48BE8E3A.8070100@ar.media.kyoto-u.ac.jp> Message-ID: <48BE9DAC.30103@relativita.com> David Cournapeau wrote: > FWIW, distance is deemed to move to a separate package, because distance > computation is useful in other contexts than clustering. > > Excellent. I was thinking about something similar. I'll have a look to the separate package. Please drop an email to this list when distance will be moved. Thanks, Emanuele From doutriaux1 at llnl.gov Wed Sep 3 11:39:41 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Wed, 03 Sep 2008 08:39:41 -0700 Subject: [Numpy-discussion] numpy 1.1.1 fails because of missing md5 In-Reply-To: <3d375d730809021446m258d99acqe83151d94a3cd932@mail.gmail.com> References: <94156B26-BC6B-4EFA-90DC-7E60B8D57BF2@llnl.gov> <48BD9387.70408@llnl.gov> <48BDAB9D.4050002@llnl.gov> <48BDB2E6.6020503@llnl.gov> <3d375d730809021446m258d99acqe83151d94a3cd932@mail.gmail.com> Message-ID: <48BEAFBD.6050502@llnl.gov> Hi Robert, The first email got intercepted because the attachment was too big (awaits moderator), so i compressed the log and I resend this email. I'm attaching my Python build log, can you spot anything? It "seems" like md5 is built, i get a very similar log on my machine and i have a working import md5. I'm not sure of what's going on. Usually the build is fine. C. Robert Kern wrote: > On Tue, Sep 2, 2008 at 16:40, Charles Doutriaux wrote: > >> Joseph, >> >> Ok all failed because numpy couldn't build... It's looking for md5 >> > > That's part of the standard library. Please check your Python installation. > > -------------- next part -------------- A non-text attachment was scrubbed... Name: Python.LOG.bz2 Type: application/x-bzip Size: 22859 bytes Desc: not available URL: From aisaac at american.edu Wed Sep 3 13:06:25 2008 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 03 Sep 2008 13:06:25 -0400 Subject: [Numpy-discussion] 1.2.0rc1 tagged! In-Reply-To: References: Message-ID: <48BEC411.8090703@american.edu> So the two formatting tests fail, as David warned. But they are known to fail on Windows, and there is no msg to that effect. Might one be added? Alan Isaac Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.test() Running unit tests for numpy NumPy version 1.2.0rc1 NumPy is installed in C:\Python25\lib\site-packages\numpy Python version 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Int el)] nose version 0.10.1 ................................................................................ ................................................................................ ................................................................................ ...FF........................................................................... ................................................................................ .......................................S........................................ ........................................Ignoring "Python was built with Visual S tudio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py." (one s hould fix me in fcompiler/compaq.py) ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ............. ====================================================================== FAIL: Check formatting. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\core\tests\test_print.py", line 28, in test_complex_types assert_equal(str(t(x)), str(complex(x))) File "C:\Python25\Lib\site-packages\numpy\testing\utils.py", line 183, in asse rt_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: '(0+5.9287877500949585e-323j)' DESIRED: '(1+0j)' ====================================================================== FAIL: Check formatting. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\core\tests\test_print.py", line 16, in test_float_types assert_equal(str(t(x)), str(float(x))) File "C:\Python25\Lib\site-packages\numpy\testing\utils.py", line 183, in asse rt_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: '0.0' DESIRED: '1.0' ====================================================================== SKIP: test_umath.TestComplexFunctions.test_branch_cuts_failing ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\lib\site-packages\nose\case.py", line 203, in runTest self.test(*self.arg) File "C:\Python25\Lib\site-packages\numpy\testing\decorators.py", line 93, in skipper raise nose.SkipTest, 'This test is known to fail' SkipTest: This test is known to fail ---------------------------------------------------------------------- Ran 1573 tests in 18.286s FAILED (failures=2) >>> From millman at berkeley.edu Wed Sep 3 14:13:52 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 3 Sep 2008 11:13:52 -0700 Subject: [Numpy-discussion] 1.2.0rc1 tagged! In-Reply-To: <48BEC411.8090703@american.edu> References: <48BEC411.8090703@american.edu> Message-ID: On Wed, Sep 3, 2008 at 10:06 AM, Alan G Isaac wrote: > So the two formatting tests fail, as David warned. > But they are known to fail on Windows, and there > is no msg to that effect. Might one be added? Absolutely, we will make sure to add a message to that effect and possibly point to a bug ticket. Thanks for the suggestion. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From robert.kern at gmail.com Wed Sep 3 15:24:23 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 3 Sep 2008 14:24:23 -0500 Subject: [Numpy-discussion] numpy 1.1.1 fails because of missing md5 In-Reply-To: <48BEAFBD.6050502@llnl.gov> References: <94156B26-BC6B-4EFA-90DC-7E60B8D57BF2@llnl.gov> <48BD9387.70408@llnl.gov> <48BDAB9D.4050002@llnl.gov> <48BDB2E6.6020503@llnl.gov> <3d375d730809021446m258d99acqe83151d94a3cd932@mail.gmail.com> <48BEAFBD.6050502@llnl.gov> Message-ID: <3d375d730809031224t4cd5dff4g13df1ec6708ffac3@mail.gmail.com> On Wed, Sep 3, 2008 at 10:39, Charles Doutriaux wrote: > Hi Robert, > > The first email got intercepted because the attachment was too big (awaits > moderator), so i compressed the log and I resend this email. > > I'm attaching my Python build log, can you spot anything? It "seems" like > md5 is built, i get a very similar log on my machine and i have a working > import md5. md5.py gets installed, but it just (eventually) imports from one of the extension modules _md5 or _hashlib, neither of which is getting built. The errors following the line building '_hashlib' extension are relevant. OpenSSL gets used for its hash function implementations if it is available. The configuration thinks you want it to use OpenSSL, so it tries to build _hashlib, which fails. If the configuration thought you didn't want to use OpenSSL, it would try to build _md5. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Wed Sep 3 15:40:31 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 3 Sep 2008 14:40:31 -0500 Subject: [Numpy-discussion] at my wits end over an error message... In-Reply-To: <05197C5B-52E9-43D1-BDE2-3D869DC42C4B@yale.edu> References: <20080830210246.7474c88e@ajackson.org> <05197C5B-52E9-43D1-BDE2-3D869DC42C4B@yale.edu> Message-ID: <3d375d730809031240y4a0ed897q45e3d47556c55107@mail.gmail.com> On Sat, Aug 30, 2008 at 22:10, Zachary Pincus wrote: > Hi Alan, > >> Traceback (most recent call last): >> File "/usr/local/lib/python2.5/site-packages/enthought.traits-2.0.4- >> py2.5-linux-i686.egg/enthought/traits/trait_notifiers.py", line 325, >> in call_1 >> self.handler( object ) >> File "TrimMapl_1.py", line 98, in _Run_fired >> outdata = np.array(outdata, dtype=dtypes) >> TypeError: expected a readable buffer object > > This would make it appear that the problem is not with numpy per se, > but with the traits library, or how you're using it... I'm not too > familiar with traits, so I can't really provide any advice there. Probably not. While Traits triggers the execution of the failing code, the last line is pure numpy, and the error message comes from numpy.array(). You then do what one normally does to debug a TypeError being thrown by a function: find out the type and value of the inputs being given to the function either by print statements or a debugger. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From doutriaux1 at llnl.gov Wed Sep 3 16:50:52 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Wed, 03 Sep 2008 13:50:52 -0700 Subject: [Numpy-discussion] numpy 1.1.1 fails because of missing md5 In-Reply-To: <3d375d730809031224t4cd5dff4g13df1ec6708ffac3@mail.gmail.com> References: <94156B26-BC6B-4EFA-90DC-7E60B8D57BF2@llnl.gov> <48BD9387.70408@llnl.gov> <48BDAB9D.4050002@llnl.gov> <48BDB2E6.6020503@llnl.gov> <3d375d730809021446m258d99acqe83151d94a3cd932@mail.gmail.com> <48BEAFBD.6050502@llnl.gov> <3d375d730809031224t4cd5dff4g13df1ec6708ffac3@mail.gmail.com> Message-ID: <48BEF8AC.2030608@llnl.gov> Thanks for spotting the origin, I'll pass this along to our user maybe they'll be able to figure out how to build python w/o openssl C. Robert Kern wrote: > On Wed, Sep 3, 2008 at 10:39, Charles Doutriaux wrote: > >> Hi Robert, >> >> The first email got intercepted because the attachment was too big (awaits >> moderator), so i compressed the log and I resend this email. >> >> I'm attaching my Python build log, can you spot anything? It "seems" like >> md5 is built, i get a very similar log on my machine and i have a working >> import md5. >> > > md5.py gets installed, but it just (eventually) imports from one of > the extension modules _md5 or _hashlib, neither of which is getting > built. The errors following the line > > building '_hashlib' extension > > are relevant. OpenSSL gets used for its hash function implementations > if it is available. The configuration thinks you want it to use > OpenSSL, so it tries to build _hashlib, which fails. If the > configuration thought you didn't want to use OpenSSL, it would try to > build _md5. > > From stefan at sun.ac.za Thu Sep 4 03:03:08 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 4 Sep 2008 09:03:08 +0200 Subject: [Numpy-discussion] `out` argument to ufuncs Message-ID: <9457e7c80809040003o74c37b6ercc6e334104be4b77@mail.gmail.com> Hey all, Gael just asked me why `add`, `subtract` etc. didn't have out arguments like `cos` and `sin`. I couldn't give him a good answer. Could we come to an agreement to add a uniform interface to ufuncs for 1.3? Either all or none of them should support the out argument. I see a couple of "deg" (for degrees) flags floating around as well, so we could consider adding those to all ufuncs that accept angles. Thanks for your input, St?fan From robert.kern at gmail.com Thu Sep 4 03:16:19 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 4 Sep 2008 02:16:19 -0500 Subject: [Numpy-discussion] `out` argument to ufuncs In-Reply-To: <9457e7c80809040003o74c37b6ercc6e334104be4b77@mail.gmail.com> References: <9457e7c80809040003o74c37b6ercc6e334104be4b77@mail.gmail.com> Message-ID: <3d375d730809040016v4687b580l84771f23d500a8da@mail.gmail.com> On Thu, Sep 4, 2008 at 02:03, St?fan van der Walt wrote: > Hey all, > > Gael just asked me why `add`, `subtract` etc. didn't have out > arguments like `cos` and `sin`. I couldn't give him a good answer. > > Could we come to an agreement to add a uniform interface to ufuncs for > 1.3? Either all or none of them should support the out argument. I am confused. add() and subtract() *do* take an out argument. In [15]: from numpy import * In [16]: x = arange(10, 20) In [17]: y = arange(10) In [18]: z = zeros(10, int) In [19]: add(x,y,z) Out[19]: array([10, 12, 14, 16, 18, 20, 22, 24, 26, 28]) In [20]: z Out[20]: array([10, 12, 14, 16, 18, 20, 22, 24, 26, 28]) In [21]: subtract(x,y,z) Out[21]: array([10, 10, 10, 10, 10, 10, 10, 10, 10, 10]) In [22]: z Out[22]: array([10, 10, 10, 10, 10, 10, 10, 10, 10, 10]) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cournapeau at cslab.kecl.ntt.co.jp Thu Sep 4 03:47:43 2008 From: cournapeau at cslab.kecl.ntt.co.jp (David Cournapeau) Date: Thu, 04 Sep 2008 16:47:43 +0900 Subject: [Numpy-discussion] numpy 1.3: cleaning the math configuration, toward a warning free build Message-ID: <1220514463.4665.33.camel@bbc8> Hi, For numpy 1.3, there are several things I would like to work on, which I have already hinted previously. The two big things are: - cleaning the math configuration - making numpy buildable warning-free with a good set a warning flags. For the warning-free build, I have started a nep, since some decisions are not obvious. Is this appropriate ? For the first part, I want to clean-up the current umath module, mainly the configuration: we have quite a big mess ATM with several intervened configurations, and everytime I am trying to touch it, I break something (adding trunc, modifying mingw configuration, etc...). I think it would be nice to clean-up, have a per-function check (instead of the HAVE_LONGDOUBLE heuristic); I noticed for example that depending on the configuration, we have different implementations for the same function, etc... Robert mentioned the configuration time cost increase, but I think I have a solution to make it as fast as before for most configurations, the ones which work today, and it will be longer only for platforms which need a better configuration (mingw, etc...). Should I write a nep for that too ? Or a patch is enough ? cheers, David From ondrej at certik.cz Thu Sep 4 05:18:10 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 4 Sep 2008 11:18:10 +0200 Subject: [Numpy-discussion] Updated Numpy reference guide In-Reply-To: References: Message-ID: <85b5c3130809040218m2adb67ffta8f9c53682e4bf68@mail.gmail.com> Hi Pauli! On Sun, Aug 31, 2008 at 8:21 PM, Pauli Virtanen wrote: > > Hi all, > > I finished the first iteration of incorporating material from Travis > Oliphant's "Guide to Numpy" to the Sphinxy reference guide we were > constructing in the Doc marathon. > > Result is here: (the PDF is a bit ugly, though, some content is almost > randomly scattered there) > > http://www.iki.fi/pav/tmp/numpy-refguide/index.xhtml > http://www.iki.fi/pav/tmp/numpy-refguide/NumPy.pdf > > Source is here: (St?fan, if it looks ok to you, could you pull and check > if it builds for you when you have time?) > > https://code.launchpad.net/~pauli-virtanen/scipy/numpy-refguide > > What I did with the "Guide to Numpy" material was: > > - Collapsed each of the reference Chapters 3, 6, 8, 9 (ndarrays, scalars, > dtypes, ufuncs) with the more introductory material in Chapter 2. > > - As this was supposed to be a reference guide, I tried to compress the > text from Chapter 2 as much as possible, by sticking to definitions and > dropping some more tutorial-oriented parts. This may have reduced > readability at some points... > > - I added some small bits or rewrote parts in the above sections in > places where I thought it would improve the result. > > - I did not include material that I thought was better to be put into > appropriate docstrings in Numpy. > > What to do with class docstrings and obscure __xxx__ attributes was not > so clear a decision, so what I did for these varies. > > - The sections about Ufuncs and array indexing are taken almost verbatim > from the "Guide to Numpy". The ndarray, scalar and dtype sections > somewhat follow the structure of the Guide, but the text is more heavily > edited from the original. > > Some things to do: > > - Descriptions about constructing items with __new__ methods should > probably still be clarified; I just replaced references to __new__ with > references to the corresponding classes. > > - What to do with the material from numpy.doc.* should be decided, as the > text there doesn't look like it should go into a reference manual. > > Some questions: > > - Is this good enough to go into Numpy SVN at some point? > > Or should we redo it and base the work closer to the original > "Guide to Numpy"? > > - Does it build for you? > > (I'd recommend using the development 0.5 version of Sphinx, so that you > get the nifty Inter-Sphinx links to the Python documentation.) > > We are unfortunately beating the Sphinx with a big stick to make it > place the documentation of each function or class into a separate file, > and to convert the Numpy docstring format to something the Sphinx can > fathom. :) > > There's also some magic in place to make toctrees:: of function listings > more pleasant to the eye. > > Any comments of what should be improved are welcome. (Even better: clone > the bzr branch, make the changes yourself, and put the result somewhere > available! E.g. as a bzr bundle or a branch on the launchpad.) I think it looks excellent. It'd be cool if all the docs could finally be at one place, instead of scattered all over the wiki. So for me, any form in sphinx is ok. Ondrej From stefan at sun.ac.za Thu Sep 4 06:01:38 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 4 Sep 2008 12:01:38 +0200 Subject: [Numpy-discussion] `out` argument to ufuncs In-Reply-To: <3d375d730809040016v4687b580l84771f23d500a8da@mail.gmail.com> References: <9457e7c80809040003o74c37b6ercc6e334104be4b77@mail.gmail.com> <3d375d730809040016v4687b580l84771f23d500a8da@mail.gmail.com> Message-ID: <9457e7c80809040301k355706f3r2d2ffe026607def5@mail.gmail.com> 2008/9/4 Robert Kern : > I am confused. add() and subtract() *do* take an out argument. So it does. We both tried a keyword-style argument, which I think is a reasonable expectation? Cheers St?fan From robert.kern at gmail.com Thu Sep 4 06:24:53 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 4 Sep 2008 05:24:53 -0500 Subject: [Numpy-discussion] `out` argument to ufuncs In-Reply-To: <9457e7c80809040301k355706f3r2d2ffe026607def5@mail.gmail.com> References: <9457e7c80809040003o74c37b6ercc6e334104be4b77@mail.gmail.com> <3d375d730809040016v4687b580l84771f23d500a8da@mail.gmail.com> <9457e7c80809040301k355706f3r2d2ffe026607def5@mail.gmail.com> Message-ID: <3d375d730809040324x1e45d095hc4741cbfa8c5e207@mail.gmail.com> On Thu, Sep 4, 2008 at 05:01, St?fan van der Walt wrote: > 2008/9/4 Robert Kern : >> I am confused. add() and subtract() *do* take an out argument. > > So it does. We both tried a keyword-style argument, which I think is > a reasonable expectation? It would certainly be *nice*, but many C-implemented functions don't do this ('cause it's a pain in C). It's even harder to do for ufuncs; follow the chain of calls down from PyUFunc_GenericFunction(). Hmm. Now that I look at it, it might be feasible to extract the out= keyword in construct_loop() and append it to the args tuple before passing that down to construct_arrays(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From hanni.ali at gmail.com Thu Sep 4 09:07:28 2008 From: hanni.ali at gmail.com (Hanni Ali) Date: Thu, 4 Sep 2008 14:07:28 +0100 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <1220435124.20211.10.camel@sebook> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> Message-ID: <789d27b10809040607v26b54c59hebbf0760de8dbed0@mail.gmail.com> Hi, Is there a way I can set numpy to use dtype='float64' throughout all my code or force it to use the biggest datatype without adding the dtype='float64' to every call to mean, stdev etc.. 2008/9/3 Sebastian Stephan Berg > Hi, > > just guessing here. But numarray seems to calculate the result in a > bigger dataype, while numpy uses float32 which is the input arrays size > (at least I thought so, trying it confused me right now ...). In any > case, maybe the difference will be gone if you > use .mean(dtype='float64') (or whatever dtype numarray actually uses, > which seems to be "numarray.MaximumType(a.type())" where a is the array > to take the mean). > > Sebastian > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Thu Sep 4 09:01:57 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 04 Sep 2008 22:01:57 +0900 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <789d27b10809040607v26b54c59hebbf0760de8dbed0@mail.gmail.com> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> <789d27b10809040607v26b54c59hebbf0760de8dbed0@mail.gmail.com> Message-ID: <48BFDC45.1090505@ar.media.kyoto-u.ac.jp> Hanni Ali wrote: > Hi, > > Is there a way I can set numpy to use dtype='float64' throughout all > my code or force it to use the biggest datatype without adding the > dtype='float64' to every call to mean, stdev etc.. Since it is the default type for the functions you mention, you can just remove any call to dtype, but removing all the call to dtype='float64' is not much less work than replacing dtype = 'float32' :) More seriously, depending on your program, it may not be doable 100 % automatically. cheers, David From gael.varoquaux at normalesup.org Thu Sep 4 09:32:18 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 4 Sep 2008 15:32:18 +0200 Subject: [Numpy-discussion] `out` argument to ufuncs In-Reply-To: <3d375d730809040324x1e45d095hc4741cbfa8c5e207@mail.gmail.com> References: <9457e7c80809040003o74c37b6ercc6e334104be4b77@mail.gmail.com> <3d375d730809040016v4687b580l84771f23d500a8da@mail.gmail.com> <9457e7c80809040301k355706f3r2d2ffe026607def5@mail.gmail.com> <3d375d730809040324x1e45d095hc4741cbfa8c5e207@mail.gmail.com> Message-ID: <20080904133218.GB23401@phare.normalesup.org> On Thu, Sep 04, 2008 at 05:24:53AM -0500, Robert Kern wrote: > On Thu, Sep 4, 2008 at 05:01, St?fan van der Walt wrote: > > 2008/9/4 Robert Kern : > >> I am confused. add() and subtract() *do* take an out argument. Hum, good! > > So it does. We both tried a keyword-style argument, which I think is > > a reasonable expectation? > It would certainly be *nice*, but many C-implemented functions don't > do this ('cause it's a pain in C). It's even harder to do for ufuncs; > follow the chain of calls down from PyUFunc_GenericFunction(). > Hmm. Now that I look at it, it might be feasible to extract the out= > keyword in construct_loop() and append it to the args tuple before > passing that down to construct_arrays(). Cool. Either that, or fix'n the docs. I had a look at the docstring and it wasn't hinted by the docstring. THe docstring editor is here for fixing these details, but I think I'll simply wait for you to add the keyword argument :). Cheers, Ga?l From oliphant at enthought.com Thu Sep 4 10:07:09 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 04 Sep 2008 09:07:09 -0500 Subject: [Numpy-discussion] numpy 1.3: cleaning the math configuration, toward a warning free build In-Reply-To: <1220514463.4665.33.camel@bbc8> References: <1220514463.4665.33.camel@bbc8> Message-ID: <48BFEB8D.1000102@enthought.com> > Should I write a nep for that too ? Or a patch is enough ? > > I think a patch with a useful explanation of the patch in the ticket is sufficient. -Travis From civija at gmail.com Thu Sep 4 11:08:58 2008 From: civija at gmail.com (Miroslav Sabljic) Date: Thu, 4 Sep 2008 17:08:58 +0200 Subject: [Numpy-discussion] numpy build on windows Message-ID: <59ccde810809040808i19777671s8d30b0a655f68da8@mail.gmail.com> Hy people! I'm quite a newbie regarding numpy so please excuse if I'm asking stupid questions. I need to build Python and couple of his site-packages on Windows using Visual studio 2008. I've built Python 2.5.1 and now it's turn to build numpy and I'm quite stuck because numpy build process is a little confusing for me. So far I've read the instructions on http://www.scipy.org/Installing_SciPy/Windows and created site.cfg file with the following content: [mkl] include_dirs = C:\Program Files (x86)\Intel\MKL\10.0.012\include library_dirs = C:\Program Files (x86)\Intel\MKL\10.0.012\ia32\lib mkl_libs = mkl_ia32, mkl_c_dll, libguide40 lapack_libs = mkl_lapack It keeps telling me that it can't find mkl_ia32, mkl_c_dll and libguide40 at the following location but these libs are there. Could you please instruct me to some detailed build howto for windows and MSVC compiler? I would appreciate any help just to get this done and build it. I've attached a log file, output of 'python setup.py config'. Thanks in advance! -- Best regards, Miroslav -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy_log.txt URL: From david at ar.media.kyoto-u.ac.jp Thu Sep 4 11:10:21 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 05 Sep 2008 00:10:21 +0900 Subject: [Numpy-discussion] numpy build on windows In-Reply-To: <59ccde810809040808i19777671s8d30b0a655f68da8@mail.gmail.com> References: <59ccde810809040808i19777671s8d30b0a655f68da8@mail.gmail.com> Message-ID: <48BFFA5D.6050409@ar.media.kyoto-u.ac.jp> Miroslav Sabljic wrote: > Hy people! > > I'm quite a newbie regarding numpy so please excuse if I'm asking > stupid questions. > No stupid question. Stupid answer may follow, though... > I need to build Python and couple of his site-packages on Windows > using Visual studio 2008. I've built Python 2.5.1 and now it's turn to > build numpy and I'm quite stuck because numpy build process is a > little confusing for me. So far I've read the instructions on > http://www.scipy.org/Installing_SciPy/Windows and created site.cfg > file with the following content: > Ok, you are building on windows, with a custom python, on windows 64 bits, with MKL... That's a lot at the same time. I would first try building numpy without the MKL with VS 2008, because this is already a challenge by itself. Did you you try it ? Did it work ? Did the tests pass ? Are you trying to build a 32 bits numpy (as suggested by the MKL in x86 dir), or a 64 bits numpy ? cheers, David From civija at gmail.com Thu Sep 4 13:30:11 2008 From: civija at gmail.com (=?UTF-8?B?TWlyb3NsYXYgU2FibGppxIc=?=) Date: Thu, 04 Sep 2008 19:30:11 +0200 Subject: [Numpy-discussion] numpy build on windows In-Reply-To: <48BFFA5D.6050409@ar.media.kyoto-u.ac.jp> References: <59ccde810809040808i19777671s8d30b0a655f68da8@mail.gmail.com> <48BFFA5D.6050409@ar.media.kyoto-u.ac.jp> Message-ID: <48C01B23.4050904@gmail.com> David Cournapeau wrote: > Ok, you are building on windows, with a custom python, on windows 64 > bits, with MKL... That's a lot at the same time. I'm building it with custom Python (built with VS 2008) on Win32. It is a lot for me but I just have to do it and learn as much as posible during the process. I'm building it with MKL becuase I followed the instructions from website. > I would first try > building numpy without the MKL with VS 2008, because this is already a > challenge by itself. Did you you try it ? Did it work ? Did the tests > pass ? I haven't tried building it without MKL because I don't know how. :( As I followed the instructions I tried to build it directly with MKL. I can try it tomorrow at work and post a log. Would you be so kind and tell what is the proper way to build it without MKL? > Are you trying to build a 32 bits numpy (as suggested by the MKL > in x86 dir), or a 64 bits numpy ? 32-bit numpy. Everything I'm building for now is 32-bit (Python, PIL, numpy, ...) on 32-bit Windows. If you need any more information please let me know and I will post them. Thanks for your answer! -- Best regards, Miroslav From cournape at gmail.com Thu Sep 4 13:48:31 2008 From: cournape at gmail.com (David Cournapeau) Date: Fri, 5 Sep 2008 02:48:31 +0900 Subject: [Numpy-discussion] numpy build on windows In-Reply-To: <48C01B23.4050904@gmail.com> References: <59ccde810809040808i19777671s8d30b0a655f68da8@mail.gmail.com> <48BFFA5D.6050409@ar.media.kyoto-u.ac.jp> <48C01B23.4050904@gmail.com> Message-ID: <5b8d13220809041048w5b7d64bboe4f608cf9645e211@mail.gmail.com> On Fri, Sep 5, 2008 at 2:30 AM, Miroslav Sablji? wrote: > David Cournapeau wrote: > >> Ok, you are building on windows, with a custom python, on windows 64 >> bits, with MKL... That's a lot at the same time. > > I'm building it with custom Python (built with VS 2008) on Win32. It is > a lot for me but I just have to do it and learn as much as posible > during the process. I'm building it with MKL becuase I followed the > instructions from website. From scipy.org ? If you understood you have to use the MLK, then we have to improve the documentation (the installation pages are a mess ATM). It is certainly no mandatory. Do you have to use VS 2008 ? Building python on windows is not easy because of the dependencies, this alone requires quite a bit of effort. Do you already have your built python ? > > I haven't tried building it without MKL because I don't know how. :( As > I followed the instructions I tried to build it directly with MKL. I can > try it tomorrow at work and post a log. Would you be so kind and tell > what is the proper way to build it without MKL? On windows, if you only build numpy, you only need a C compiler, so in theory, python setup.py build, with an empty site.cfg, should work. But VS 2008 is significantly different from VS 2003, which is the most often used MS compiler to build numpy (on windows, many people use mingw; the official binaries are built with mingw; but there are problems between mingw and VS 2008....). I would try that first (an empty site.cfg build). > > >> Are you trying to build a 32 bits numpy (as suggested by the MKL >> in x86 dir), or a 64 bits numpy ? > > 32-bit numpy. Everything I'm building for now is 32-bit (Python, PIL, > numpy, ...) on 32-bit Windows. If you need any more information please > let me know and I will post them. I am surprised: why you do you have Program File (x86) if you run on 32 bits ? I thought the directory only appeared on windows 64 bits, to hold the 32 bits programs ? cheers, David From rmay31 at gmail.com Thu Sep 4 14:08:17 2008 From: rmay31 at gmail.com (Ryan May) Date: Thu, 04 Sep 2008 13:08:17 -0500 Subject: [Numpy-discussion] BUG in numpy.loadtxt? Message-ID: <48C02411.70602@gmail.com> Stefan (or anyone else who can comment), It appears that the usecols argument to loadtxt no longer accepts numpy arrays: >>>from StringIO import StringIO >>>text = StringIO('1 2 3\n4 5 6\n') >>>data = np.loadtxt(text, usecols=np.arange(1,3)) ValueError Traceback (most recent call last) /usr/lib64/python2.5/site-packages/numpy/lib/io.py in loadtxt(fname, dtype, comments, delimiter, converters, skiprows, usecols, unpack) 323 first_line = fh.readline() 324 first_vals = split_line(first_line) --> 325 N = len(usecols or first_vals) 326 327 dtype_types = flatten_dtype(dtype) ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() >>>data = np.loadtxt(text, usecols=np.arange(1,3).tolist()) >>>data array([[ 2., 3.], [ 5., 6.]]) Was it a conscious design decision that the usecols no longer accept arrays? The new behavior (in 1.1.1) breaks existing code that one of my colleagues has. Can we get a patch in before 1.2 to get this working with arrays again? Thanks, Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From pav at iki.fi Thu Sep 4 14:09:32 2008 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 4 Sep 2008 18:09:32 +0000 (UTC) Subject: [Numpy-discussion] `out` argument to ufuncs References: <9457e7c80809040003o74c37b6ercc6e334104be4b77@mail.gmail.com> <3d375d730809040016v4687b580l84771f23d500a8da@mail.gmail.com> <9457e7c80809040301k355706f3r2d2ffe026607def5@mail.gmail.com> <3d375d730809040324x1e45d095hc4741cbfa8c5e207@mail.gmail.com> <20080904133218.GB23401@phare.normalesup.org> Message-ID: Thu, 04 Sep 2008 15:32:18 +0200, Gael Varoquaux wrote: [clip] > Cool. Either that, or fix'n the docs. I had a look at the docstring and > it wasn't hinted by the docstring. THe docstring editor is here for > fixing these details, but I think I'll simply wait for you to add the > keyword argument :). The ufunc signatures in the docstrings are automatically generated in ufuncobject.c. I changed the code in r5768 so that it's more obvious the various functions take output arguments. When the interface gets adjusted so that 'out' are actually keyword arguments, the signature formatting needs to be changed again so that 'outN' look like keyword arguments. Currently the 'sig' and 'extobj' arguments are not shown in the signature. I'm not sure whether it makes sense to add them, since they appear to be for quite exotic use only... todo.pop(); todo.append(B) -- Pauli Virtanen From civija at gmail.com Thu Sep 4 14:37:37 2008 From: civija at gmail.com (=?UTF-8?B?TWlyb3NsYXYgU2FibGppxIc=?=) Date: Thu, 04 Sep 2008 20:37:37 +0200 Subject: [Numpy-discussion] numpy build on windows In-Reply-To: <5b8d13220809041048w5b7d64bboe4f608cf9645e211@mail.gmail.com> References: <59ccde810809040808i19777671s8d30b0a655f68da8@mail.gmail.com> <48BFFA5D.6050409@ar.media.kyoto-u.ac.jp> <48C01B23.4050904@gmail.com> <5b8d13220809041048w5b7d64bboe4f608cf9645e211@mail.gmail.com> Message-ID: <48C02AF1.5010302@gmail.com> David Cournapeau wrote: > From scipy.org ? If you understood you have to use the MLK, then we > have to improve the documentation (the installation pages are a mess > ATM). It is certainly no mandatory. Yeah, from scipy.org. > Do you have to use VS 2008 ? Building python on windows is not easy > because of the dependencies, this alone requires quite a bit of > effort. Do you already have your built python ? Yes, I have to use VS 2008. I've successfully built Python 2.5.1 with VS 2008 and I'm currently using it to build other neccessary site-packages for it. > On windows, if you only build numpy, you only need a C compiler, so in > theory, python setup.py build, with an empty site.cfg, should work. > But VS 2008 is significantly different from VS 2003, which is the most > often used MS compiler to build numpy (on windows, many people use > mingw; the official binaries are built with mingw; but there are > problems between mingw and VS 2008....). > > I would try that first (an empty site.cfg build). I will try it tomorrow at work. I would use mingw or even cygwin but unfortunately I have to use exactly VS 2008 and only it. > I am surprised: why you do you have Program File (x86) if you run on > 32 bits ? I thought the directory only appeared on windows 64 bits, to > hold the 32 bits programs ? Sorry for this misapprehension! I meant to say that I'm building 32-bit numpy for 32-bit Python. The machine I'm using to build it is 64-bit Windows which has both 32-bit and 64-bit compilers and libraries installed. For now I'm doing only 32-bit builds. Tomorrow I will try to build numpy with empty site.cfg and post results here. Tnx. -- Best regards, Miroslav From matthieu.brucher at gmail.com Thu Sep 4 14:44:53 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 4 Sep 2008 20:44:53 +0200 Subject: [Numpy-discussion] numpy build on windows In-Reply-To: <48C02AF1.5010302@gmail.com> References: <59ccde810809040808i19777671s8d30b0a655f68da8@mail.gmail.com> <48BFFA5D.6050409@ar.media.kyoto-u.ac.jp> <48C01B23.4050904@gmail.com> <5b8d13220809041048w5b7d64bboe4f608cf9645e211@mail.gmail.com> <48C02AF1.5010302@gmail.com> Message-ID: > Sorry for this misapprehension! I meant to say that I'm building 32-bit > numpy for 32-bit Python. The machine I'm using to build it is 64-bit > Windows which has both 32-bit and 64-bit compilers and libraries > installed. For now I'm doing only 32-bit builds. > > Tomorrow I will try to build numpy with empty site.cfg and post results > here. Tnx. It's possible that the spaces in the path are responsible for the non-detection. Matthieu -- French PhD student Information System Engineer Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher From strawman at astraw.com Thu Sep 4 14:51:14 2008 From: strawman at astraw.com (Andrew Straw) Date: Thu, 04 Sep 2008 11:51:14 -0700 Subject: [Numpy-discussion] test results of 1.2.0rc1 Message-ID: <48C02E22.2030704@astraw.com> Hi, with numpy 1.2.0rc1 running 'python -c "import numpy; numpy.test()"' on my Ubuntu Hardy amd64 machine results in 1721 tests being run and 1 skipped. So far, so good. However, if I run numpy.test(10,10,all=True), I get 1846 tests with: the message "FAILED (SKIP=1, errors=8, failures=68)" Furthermore, there are several matplotlib windows that pop up, many of which are non-reassuringly blank: Bartlett window frequency response (twice -- I guess the 2nd is actually for the Blackman window), Hamming window frequency response, Kaiser window, Kaiser window frequency response, sinc function. Additionally, the linspace and logspace tests each generate a plots with green and blue dots at Y values of 0.0 and 0.5, but it would be nice to have an axes title. Should I be concerned that there are so many errors and failures with the numpy test suite? Or am I just running it with unintended settings? If these tests should pass, I will attempt to find time to generate bug reports for them, although I don't think there's anything particularly weird about my setup. -Andrew From pav at iki.fi Thu Sep 4 15:07:24 2008 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 4 Sep 2008 19:07:24 +0000 (UTC) Subject: [Numpy-discussion] test results of 1.2.0rc1 References: <48C02E22.2030704@astraw.com> Message-ID: Thu, 04 Sep 2008 11:51:14 -0700, Andrew Straw wrote: > Hi, with numpy 1.2.0rc1 running 'python -c "import numpy; numpy.test()"' > on my Ubuntu Hardy amd64 machine results in 1721 tests being run and 1 > skipped. So far, so good. > > However, if I run numpy.test(10,10,all=True), I get 1846 tests with: the > message "FAILED (SKIP=1, errors=8, failures=68)" Furthermore, there are > several matplotlib windows that pop up, many of which are > non-reassuringly blank: Bartlett window frequency response (twice -- I > guess the 2nd is actually for the Blackman window), Hamming window > frequency response, Kaiser window, Kaiser window frequency response, > sinc function. Additionally, the linspace and logspace tests each > generate a plots with green and blue dots at Y values of 0.0 and 0.5, > but it would be nice to have an axes title. > > Should I be concerned that there are so many errors and failures with > the numpy test suite? Or am I just running it with unintended settings? > If these tests should pass, I will attempt to find time to generate bug > reports for them, although I don't think there's anything particularly > weird about my setup. I'd say that the settings are "unintended" in the sense that they run all examples in all docstrings. There are quite a few of these, and some indeed plot some graphs. Ideally, all examples should run, but this is not assured at the moment. Some of the examples were added during the summer's documentation marathon, but others date way back. There was some discussion about what to do with the plots, but as far as I know, no conclusions were reached about this, so we sticked writing them like all other examples. Some alternatives offered were 1) Mark them with >>> and live with not being able to doctest docstrings. -0 2) Mark them with >>> and fake matplotlib so that the plots don't appear. +0 3) Mark them with :: and live no syntax highlighting in rendered documentation and not being able to test or render plots easily. -1 4) Steal or adapt the plot:: ReST directive from matplotlib, and use that, as St?fan suggested at some point. Haven't got yet around to implementing this. Tentatively +1; depends a bit of how the implementation goes. -- Pauli Virtanen From oliphant at enthought.com Thu Sep 4 15:13:03 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 04 Sep 2008 14:13:03 -0500 Subject: [Numpy-discussion] BUG in numpy.loadtxt? In-Reply-To: <48C02411.70602@gmail.com> References: <48C02411.70602@gmail.com> Message-ID: <48C0333F.9060709@enthought.com> Ryan May wrote: > Stefan (or anyone else who can comment), > > It appears that the usecols argument to loadtxt no longer accepts numpy > arrays: > Could you enter a ticket so we don't lose track of this. I don't remember anything being intentional. -Travis From pav at iki.fi Thu Sep 4 15:17:00 2008 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 4 Sep 2008 19:17:00 +0000 (UTC) Subject: [Numpy-discussion] test results of 1.2.0rc1 References: <48C02E22.2030704@astraw.com> Message-ID: Thu, 04 Sep 2008 19:07:24 +0000, Pauli Virtanen wrote: [clip] > I'd say that the settings are "unintended" in the sense that they run > all examples in all docstrings. There are quite a few of these, and some > indeed plot some graphs. > > Ideally, all examples should run, but this is not assured at the moment. > Some of the examples were added during the summer's documentation > marathon, but others date way back. Ok, maybe to clarify: yes, I they are bugs, but only bugs in the documentation and not in functionality. Also, some (the plots) are related with unresolved technical issues. -- Pauli Virtanen From robert.kern at gmail.com Thu Sep 4 15:43:15 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 4 Sep 2008 14:43:15 -0500 Subject: [Numpy-discussion] `out` argument to ufuncs In-Reply-To: <20080904133218.GB23401@phare.normalesup.org> References: <9457e7c80809040003o74c37b6ercc6e334104be4b77@mail.gmail.com> <3d375d730809040016v4687b580l84771f23d500a8da@mail.gmail.com> <9457e7c80809040301k355706f3r2d2ffe026607def5@mail.gmail.com> <3d375d730809040324x1e45d095hc4741cbfa8c5e207@mail.gmail.com> <20080904133218.GB23401@phare.normalesup.org> Message-ID: <3d375d730809041243q4a689e90l823e448cc4f5b49f@mail.gmail.com> On Thu, Sep 4, 2008 at 08:32, Gael Varoquaux wrote: > On Thu, Sep 04, 2008 at 05:24:53AM -0500, Robert Kern wrote: >> On Thu, Sep 4, 2008 at 05:01, St?fan van der Walt wrote: >> > 2008/9/4 Robert Kern : >> >> I am confused. add() and subtract() *do* take an out argument. > > Hum, good! > >> > So it does. We both tried a keyword-style argument, which I think is >> > a reasonable expectation? > >> It would certainly be *nice*, but many C-implemented functions don't >> do this ('cause it's a pain in C). It's even harder to do for ufuncs; >> follow the chain of calls down from PyUFunc_GenericFunction(). > >> Hmm. Now that I look at it, it might be feasible to extract the out= >> keyword in construct_loop() and append it to the args tuple before >> passing that down to construct_arrays(). > > Cool. Either that, or fix'n the docs. I had a look at the docstring and > it wasn't hinted by the docstring. THe docstring editor is here for > fixing these details, but I think I'll simply wait for you to add the > keyword argument :). It won't be happening soon. len(todo) == big. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pjssilva at ime.usp.br Thu Sep 4 15:45:31 2008 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Thu, 04 Sep 2008 15:45:31 -0400 Subject: [Numpy-discussion] Getting a numpy array from a ctype pointer Message-ID: <1220557531.22809.3.camel@trinity> Hello, I am writing some code interfacing C and Python using ctypes. In a callback function (in Python) I get in a parameter x which is c_double pointer and a parameter n which is c_int representing the length of the array. How can I transform this information into a numpy array? Paulo From oliphant at enthought.com Thu Sep 4 16:01:17 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 04 Sep 2008 15:01:17 -0500 Subject: [Numpy-discussion] Getting a numpy array from a ctype pointer In-Reply-To: <1220557531.22809.3.camel@trinity> References: <1220557531.22809.3.camel@trinity> Message-ID: <48C03E8D.50105@enthought.com> Paulo J. S. Silva wrote: > Hello, > > I am writing some code interfacing C and Python using ctypes. In a > callback function (in Python) I get in a parameter x which is c_double > pointer and a parameter n which is c_int representing the length of the > array. > > How can I transform this information into a numpy array? > > Something like this may work: from numpy import ctypeslib r = ctypeslib.as_array(x._type_ * n) If that doesn't work, then you can create an array from an arbitrary buffer or any object that exposes the __array_interface__ attribute. So, if you can get the address of c_double then you can use it as the data for the ndarray. Ask if you need more help. -Travis From oc-spam66 at laposte.net Thu Sep 4 16:16:16 2008 From: oc-spam66 at laposte.net (oc-spam66) Date: Thu, 4 Sep 2008 22:16:16 +0200 (CEST) Subject: [Numpy-discussion] How to import data where the decimal separator is a comma ? Message-ID: <14292926.1569891220559376466.JavaMail.www@wwinf8301> Hello, I have data files where the decimal separator is a comma. Can I import this data with numpy.loadtxt ? Notes : - I tried to set the locale LC_NUMERIC="fr_FR.UTF-8" but it didn't change anything. - Python 2.5.2, Numpy 1.1.0 Have a nice day, O.C. Cr?ez votre adresse ?lectronique prenom.nom at laposte.net 1 Go d'espace de stockage, anti-spam et anti-virus int?gr?s. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmay31 at gmail.com Thu Sep 4 16:47:21 2008 From: rmay31 at gmail.com (Ryan May) Date: Thu, 04 Sep 2008 15:47:21 -0500 Subject: [Numpy-discussion] BUG in numpy.loadtxt? In-Reply-To: <48C0333F.9060709@enthought.com> References: <48C02411.70602@gmail.com> <48C0333F.9060709@enthought.com> Message-ID: <48C04959.3080309@gmail.com> Travis E. Oliphant wrote: > Ryan May wrote: >> Stefan (or anyone else who can comment), >> >> It appears that the usecols argument to loadtxt no longer accepts numpy >> arrays: >> > > Could you enter a ticket so we don't lose track of this. I don't > remember anything being intentional. > Done: #905 http://scipy.org/scipy/numpy/ticket/905 I've attached a patch that does the obvious and coerces usecols to a list when it's not None, so it will work for any iterable. I don't think it was a conscious decision, just a consequence of the rewrite using different methods. There are two problems: 1) It's an API break, technically speaking 2) It currently doesn't even accept tuples, which are used in the docstring. Can we hurry and get this into 1.2? Thanks, Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From pjssilva at ime.usp.br Thu Sep 4 17:31:10 2008 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Thu, 04 Sep 2008 17:31:10 -0400 Subject: [Numpy-discussion] Getting a numpy array from a ctype pointer In-Reply-To: <48C03E8D.50105@enthought.com> References: <1220557531.22809.3.camel@trinity> <48C03E8D.50105@enthought.com> Message-ID: <1220563870.22809.8.camel@trinity> Em Qui, 2008-09-04 ?s 15:01 -0500, Travis E. Oliphant escreveu: > Paulo J. S. Silva wrote: > > > Something like this may work: > > from numpy import ctypeslib > > r = ctypeslib.as_array(x._type_ * n) > Unfortunately, it didn't work. > If that doesn't work, then you can create an array from an arbitrary > buffer or any object that exposes the __array_interface__ attribute. > So, if you can get the address of c_double then you can use it as the > data for the ndarray. > > Ask if you need more help. I think I need help. I discovered how to create a Python string from the pointer using ctypes' string_at function: r = C.string_at(x, n*C.sizeof(x._type_)) Now I can use numpy fromstring to get a array version of it. However this is not enough for my application as fromstring actually copies the data, hence change to the array are not reflected in the original array that goes back to the Fortran code. Any hints? Paulo From pjssilva at ime.usp.br Thu Sep 4 18:56:17 2008 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Thu, 04 Sep 2008 18:56:17 -0400 Subject: [Numpy-discussion] Getting a numpy array from a ctype pointer In-Reply-To: <1220563870.22809.8.camel@trinity> References: <1220557531.22809.3.camel@trinity> <48C03E8D.50105@enthought.com> <1220563870.22809.8.camel@trinity> Message-ID: <1220568977.22809.12.camel@trinity> After some trial and erros, I found a solution, described below. Is this the best one? Looks a little convoluted to me (C represents ctypes module and np numpy): Array = n*C.c_double x = Array.from_address(C.addressof(x.contents)) x = np.ctypeslib.as_array(x) Thanks, Paulo Em Qui, 2008-09-04 ?s 17:31 -0400, Paulo J. S. Silva escreveu: > Em Qui, 2008-09-04 ?s 15:01 -0500, Travis E. Oliphant escreveu: > > Paulo J. S. Silva wrote: > > > > > Something like this may work: > > > > from numpy import ctypeslib > > > > r = ctypeslib.as_array(x._type_ * n) > > > > Unfortunately, it didn't work. > > > If that doesn't work, then you can create an array from an arbitrary > > buffer or any object that exposes the __array_interface__ attribute. > > So, if you can get the address of c_double then you can use it as the > > data for the ndarray. > > > > Ask if you need more help. > > I think I need help. I discovered how to create a Python string from the > pointer using ctypes' string_at function: > > r = C.string_at(x, n*C.sizeof(x._type_)) > > Now I can use numpy fromstring to get a array version of it. However > this is not enough for my application as fromstring actually copies the > data, hence change to the array are not reflected in the original array > that goes back to the Fortran code. > > Any hints? > > Paulo > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From oliphant at enthought.com Thu Sep 4 19:21:47 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 04 Sep 2008 18:21:47 -0500 Subject: [Numpy-discussion] Getting a numpy array from a ctype pointer In-Reply-To: <1220568977.22809.12.camel@trinity> References: <1220557531.22809.3.camel@trinity> <48C03E8D.50105@enthought.com> <1220563870.22809.8.camel@trinity> <1220568977.22809.12.camel@trinity> Message-ID: <48C06D8B.6070602@enthought.com> Paulo J. S. Silva wrote: > After some trial and erros, I found a solution, described below. Is this > the best one? Looks a little convoluted to me (C represents ctypes > module and np numpy): > > Array = n*C.c_double > x = Array.from_address(C.addressof(x.contents)) > x = np.ctypeslib.as_array(x) > That's a pretty simple approach. There is a faster approach which uses the undocumented function int_asbuffer function from numpy.core.multiarray (i.e. from numpy.core.multiarray import int_asbuffer) Then: x = np.frombuffer(int_asbuffer(C.addressof(x.contents), n*8)) If you don't like the hard-coded '8', then you can get that number from np.dtype(float).itemsize There may also be a way to get it from ctypes. -Travis From david at ar.media.kyoto-u.ac.jp Fri Sep 5 01:06:07 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 05 Sep 2008 14:06:07 +0900 Subject: [Numpy-discussion] isnan and co: cleaning up Message-ID: <48C0BE3F.1030306@ar.media.kyoto-u.ac.jp> Hi, While working on my branch to clean the math configuration, I noticed that the code for isnan and co became quite convoluted. autoconf info file has a mention of it, and suggests the following for portability (section 5.5.1 of autoconf): The C99 standard says that `isinf' and `isnan' are macros. On some systems just macros are available (e.g., HP-UX and Solaris 10), on some systems both macros and functions (e.g., glibc 2.3.2), and on some systems only functions (e.g., IRIX 6 and Solaris 9). In some cases these functions are declared in nonstandard headers like `' and defined in non-default libraries like `-lm' or `-lsunmath'. The C99 `isinf' and `isnan' macros work correctly with `long double' arguments, but pre-C99 systems that use functions typically assume `double' arguments. On such a system, `isinf' incorrectly returns true for a finite `long double' argument that is outside the range of `double'. To work around this porting mess, you can use code like the following. #include #ifndef isnan # define isnan(x) \ (sizeof (x) == sizeof (long double) ? isnan_ld (x) \ : sizeof (x) == sizeof (double) ? isnan_d (x) \ : isnan_f (x)) static inline int isnan_f (float x) { return x != x; } static inline int isnan_d (double x) { return x != x; } static inline int isnan_ld (long double x) { return x != x; } #endif #ifndef isinf # define isinf(x) \ (sizeof (x) == sizeof (long double) ? isinf_ld (x) \ : sizeof (x) == sizeof (double) ? isinf_d (x) \ : isinf_f (x)) static inline int isinf_f (float x) { return isnan (x - x); } static inline int isinf_d (double x) { return isnan (x - x); } static inline int isinf_ld (long double x) { return isnan (x - x); } #endif Use `AC_C_INLINE' (*note C Compiler::) so that this code works on compilers that lack the `inline' keyword. Some optimizing compilers mishandle these definitions, but systems with that bug typically have missing or broken `isnan' functions anyway, so it's probably not worth worrying about. This is simpler than the current code (I actually understand what the above case does; the current code in numpy has many cases which I do not understand), does not need any C code (_isnan and co), and probably more portable since autoconf has a lot of experience there. Is the code in _isnan really better than just relying on the nan property x != x ? (Do we support platforms without a IEE754 FPU ?) Does anyone has any thought on replacing the current code by the above (modulo the inline part, which we can drop for now) ? cheers, David From david at ar.media.kyoto-u.ac.jp Fri Sep 5 02:27:56 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 05 Sep 2008 15:27:56 +0900 Subject: [Numpy-discussion] Are the CPP symbols HAVE_LONGDOUBLE_FUNCS and HAVE_FLOAT_FUNCS public ? Message-ID: <48C0D16C.9000000@ar.media.kyoto-u.ac.jp> Hi, Those two symbols are mentioned in the numpy C-API book, but with my work on cleaning the math configuration, those are not needed by numpy anymore (I grep into numpy and scipy sources, and they are only used in umathmodule.c.src, where they are obsoleted by my clean-up). Can I remove them, or should I keep them for backward compatibility ? cheers, David From pav at iki.fi Fri Sep 5 03:50:49 2008 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 5 Sep 2008 07:50:49 +0000 (UTC) Subject: [Numpy-discussion] Are the CPP symbols HAVE_LONGDOUBLE_FUNCS and HAVE_FLOAT_FUNCS public ? References: <48C0D16C.9000000@ar.media.kyoto-u.ac.jp> Message-ID: Fri, 05 Sep 2008 15:27:56 +0900, David Cournapeau wrote: > Those two symbols are mentioned in the numpy C-API book, but with my > work on cleaning the math configuration, those are not needed by numpy > anymore (I grep into numpy and scipy sources, and they are only used in > umathmodule.c.src, where they are obsoleted by my clean-up). Can I > remove them, or should I keep them for backward compatibility ? If you remove them, please put a reminder somewhere (eg ticket) to update the documentation accordingly. Pauli From civija at gmail.com Fri Sep 5 04:51:22 2008 From: civija at gmail.com (Miroslav Sabljic) Date: Fri, 5 Sep 2008 10:51:22 +0200 Subject: [Numpy-discussion] numpy build on windows In-Reply-To: <5b8d13220809041048w5b7d64bboe4f608cf9645e211@mail.gmail.com> References: <59ccde810809040808i19777671s8d30b0a655f68da8@mail.gmail.com> <48BFFA5D.6050409@ar.media.kyoto-u.ac.jp> <48C01B23.4050904@gmail.com> <5b8d13220809041048w5b7d64bboe4f608cf9645e211@mail.gmail.com> Message-ID: <59ccde810809050151g457a8f7dv4a530dd780018956@mail.gmail.com> On 9/4/08, David Cournapeau wrote: > On windows, if you only build numpy, you only need a C compiler, so in > theory, python setup.py build, with an empty site.cfg, should work. > But VS 2008 is significantly different from VS 2003, which is the most > often used MS compiler to build numpy (on windows, many people use > mingw; the official binaries are built with mingw; but there are > problems between mingw and VS 2008....). > > I would try that first (an empty site.cfg build). I have tried building it with empty site.cfg file as you told me. The log file is in attachment. -- Best regards, Miroslav -------------- next part -------------- A non-text attachment was scrubbed... Name: numpy_build.log Type: application/octet-stream Size: 7574 bytes Desc: not available URL: From david at ar.media.kyoto-u.ac.jp Fri Sep 5 05:02:51 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 05 Sep 2008 18:02:51 +0900 Subject: [Numpy-discussion] numpy build on windows In-Reply-To: <59ccde810809050151g457a8f7dv4a530dd780018956@mail.gmail.com> References: <59ccde810809040808i19777671s8d30b0a655f68da8@mail.gmail.com> <48BFFA5D.6050409@ar.media.kyoto-u.ac.jp> <48C01B23.4050904@gmail.com> <5b8d13220809041048w5b7d64bboe4f608cf9645e211@mail.gmail.com> <59ccde810809050151g457a8f7dv4a530dd780018956@mail.gmail.com> Message-ID: <48C0F5BB.8080708@ar.media.kyoto-u.ac.jp> Miroslav Sabljic wrote: > > I have tried building it with empty site.cfg file as you told me. The > log file is in attachment. > > First, you are trying to build with numpy 1.0.2, which is really old. Update to a more recent version. The error you are seeing may indicate that it is not possible to build python 2.5 extensions with VS 2008 with distutils (the message says distutils did not detect VS 2008 installation, and that may well be caused by distutils not knowing how to deal with VS 2008). How did you build python with VS 2008 ? cheers, David P.S: Not to discourage you, but it will certainly be painful to build numpy with VS 2008 and python 2.5, because 2.5 distutils does not really handle VS 2008, and we use distutils for the build, and also because AFAIK, numpy has never been built with VS 2008 yet (2.6 will support VS 2008, so that's something which will be fixed sooner rather than later, assuming numpy C code cannot be compiler with VS 2008 now). From matthieu.brucher at gmail.com Fri Sep 5 05:42:13 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Fri, 5 Sep 2008 11:42:13 +0200 Subject: [Numpy-discussion] numpy build on windows In-Reply-To: <48C0F5BB.8080708@ar.media.kyoto-u.ac.jp> References: <59ccde810809040808i19777671s8d30b0a655f68da8@mail.gmail.com> <48BFFA5D.6050409@ar.media.kyoto-u.ac.jp> <48C01B23.4050904@gmail.com> <5b8d13220809041048w5b7d64bboe4f608cf9645e211@mail.gmail.com> <59ccde810809050151g457a8f7dv4a530dd780018956@mail.gmail.com> <48C0F5BB.8080708@ar.media.kyoto-u.ac.jp> Message-ID: > The error you are seeing may indicate that it is not possible to build > python 2.5 extensions with VS 2008 with distutils (the message says > distutils did not detect VS 2008 installation, and that may well be > caused by distutils not knowing how to deal with VS 2008). You may be right, I remember that VS2003 is hardcoded in some places in distutils. I think I ran into this issue some months ago as well. Matthieu -- French PhD student Information System Engineer Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher From david at ar.media.kyoto-u.ac.jp Fri Sep 5 05:27:43 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 05 Sep 2008 18:27:43 +0900 Subject: [Numpy-discussion] numpy build on windows In-Reply-To: References: <59ccde810809040808i19777671s8d30b0a655f68da8@mail.gmail.com> <48BFFA5D.6050409@ar.media.kyoto-u.ac.jp> <48C01B23.4050904@gmail.com> <5b8d13220809041048w5b7d64bboe4f608cf9645e211@mail.gmail.com> <59ccde810809050151g457a8f7dv4a530dd780018956@mail.gmail.com> <48C0F5BB.8080708@ar.media.kyoto-u.ac.jp> Message-ID: <48C0FB8F.5080907@ar.media.kyoto-u.ac.jp> Matthieu Brucher wrote: > > You may be right, I remember that VS2003 is hardcoded in some places > in distutils. I think I ran into this issue some months ago as well. > Mmmh, wonder if I could get around that in numscons :) cheers, David From faltet at pytables.org Fri Sep 5 05:49:48 2008 From: faltet at pytables.org (Francesc Alted) Date: Fri, 5 Sep 2008 11:49:48 +0200 Subject: [Numpy-discussion] numpy build on windows In-Reply-To: <48C0FB8F.5080907@ar.media.kyoto-u.ac.jp> References: <59ccde810809040808i19777671s8d30b0a655f68da8@mail.gmail.com> <48C0FB8F.5080907@ar.media.kyoto-u.ac.jp> Message-ID: <200809051149.49457.faltet@pytables.org> A Friday 05 September 2008, David Cournapeau escrigu?: > Matthieu Brucher wrote: > > You may be right, I remember that VS2003 is hardcoded in some > > places in distutils. I think I ran into this issue some months ago > > as well. > > Mmmh, wonder if I could get around that in numscons :) IIRC, Python 2.4 and 2.5 only supports the VS2003 of the MS series of compilers. With Python 2.6 and 3.0 I think that VS2008 will be the one supported. Cheers, -- Francesc Alted From david at ar.media.kyoto-u.ac.jp Fri Sep 5 05:35:35 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 05 Sep 2008 18:35:35 +0900 Subject: [Numpy-discussion] numpy build on windows In-Reply-To: <200809051149.49457.faltet@pytables.org> References: <59ccde810809040808i19777671s8d30b0a655f68da8@mail.gmail.com> <48C0FB8F.5080907@ar.media.kyoto-u.ac.jp> <200809051149.49457.faltet@pytables.org> Message-ID: <48C0FD67.5030403@ar.media.kyoto-u.ac.jp> Francesc Alted wrote: > > IIRC, Python 2.4 and 2.5 only supports the VS2003 of the MS series of > compilers. With Python 2.6 and 3.0 I think that VS2008 will be the one > supported. > Yes, but that's just stupid when you think about it. I mean, the python code to deal with VS 2008 is there, so if you built your python with VS 2008, everything to build extensions with VS 2008 is there (mismatched VS is another story). The code in numscons to detect and setting up VS 2008 (and 2005) is an improved version of the code in python 2.6 distutils, so if there is a way to prevent distutils from barfing during the configuration, it should be doable, cheers, David From civija at gmail.com Fri Sep 5 07:46:33 2008 From: civija at gmail.com (Miroslav Sabljic) Date: Fri, 5 Sep 2008 13:46:33 +0200 Subject: [Numpy-discussion] numpy build on windows In-Reply-To: <48C0F5BB.8080708@ar.media.kyoto-u.ac.jp> References: <59ccde810809040808i19777671s8d30b0a655f68da8@mail.gmail.com> <48BFFA5D.6050409@ar.media.kyoto-u.ac.jp> <48C01B23.4050904@gmail.com> <5b8d13220809041048w5b7d64bboe4f608cf9645e211@mail.gmail.com> <59ccde810809050151g457a8f7dv4a530dd780018956@mail.gmail.com> <48C0F5BB.8080708@ar.media.kyoto-u.ac.jp> Message-ID: <59ccde810809050446k3d6c2c61pcf7cdd998eede0b6@mail.gmail.com> On 9/5/08, David Cournapeau wrote: > First, you are trying to build with numpy 1.0.2, which is really old. > Update to a more recent version. Yes, 1.0.2 is the version we are using on Linux so I went with that one. I will try newer version. > The error you are seeing may indicate that it is not possible to build > python 2.5 extensions with VS 2008 with distutils (the message says > distutils did not detect VS 2008 installation, and that may well be > caused by distutils not knowing how to deal with VS 2008). The same error was when I was building PIL library, so I had to apply small patch to msvccompiler.py file in Python's distutils and export DISTUTILS_USE_SDK and MSSDK env variables to override builtin settings which were always looking for VS 2003 compiler, then the build process went without a problem. I followed the instructions described here http://jeethurao.com/blog/?p=35. > How did you build python with VS 2008 ? With VS 2008. I opened Python-2.5.1\PCbuild8\pcbuild.sln file with Visual Studio, included project files for tkinter, ssl, bddsb, etc... and everything was built without problems. I have the full Python 2.5.1 distribution build, identical as the Python-2.5.1.msi installation from python.org. > P.S: Not to discourage you, but it will certainly be painful to build > numpy with VS 2008 and python 2.5, because 2.5 distutils does not really > handle VS 2008, and we use distutils for the build, and also because > AFAIK, numpy has never been built with VS 2008 yet (2.6 will support VS > 2008, so that's something which will be fixed sooner rather than later, > assuming numpy C code cannot be compiler with VS 2008 now). Thanks for the info. It would certanly be bad news for me if numpy really can't be build with VS 2008. I will try with numpy 1.1.1 and see what happens. -- Best regards, Miroslav From pjssilva at ime.usp.br Thu Sep 4 21:09:25 2008 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Thu, 04 Sep 2008 21:09:25 -0400 Subject: [Numpy-discussion] Getting a numpy array from a ctype pointer In-Reply-To: <48C06D8B.6070602@enthought.com> References: <1220557531.22809.3.camel@trinity> <48C03E8D.50105@enthought.com> <1220563870.22809.8.camel@trinity> <1220568977.22809.12.camel@trinity> <48C06D8B.6070602@enthought.com> Message-ID: <1220576965.6521.3.camel@trinity> > x = np.frombuffer(int_asbuffer(C.addressof(x.contents), n*8)) I'll go with your faster solution. Very good. Thank you very much. Paulo From david at ar.media.kyoto-u.ac.jp Fri Sep 5 08:11:35 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 05 Sep 2008 21:11:35 +0900 Subject: [Numpy-discussion] numpy build on windows In-Reply-To: <59ccde810809050446k3d6c2c61pcf7cdd998eede0b6@mail.gmail.com> References: <59ccde810809040808i19777671s8d30b0a655f68da8@mail.gmail.com> <48BFFA5D.6050409@ar.media.kyoto-u.ac.jp> <48C01B23.4050904@gmail.com> <5b8d13220809041048w5b7d64bboe4f608cf9645e211@mail.gmail.com> <59ccde810809050151g457a8f7dv4a530dd780018956@mail.gmail.com> <48C0F5BB.8080708@ar.media.kyoto-u.ac.jp> <59ccde810809050446k3d6c2c61pcf7cdd998eede0b6@mail.gmail.com> Message-ID: <48C121F7.4070209@ar.media.kyoto-u.ac.jp> Miroslav Sabljic wrote: > > > Thanks for the info. It would certanly be bad news for me if numpy > really can't be build with VS 2008. I will try with numpy 1.1.1 and > see what happens. > The code itself will have to be buildable with VS 2008 (if only because of python 2.6, for which VS 2008 will be the default compiler), so it will happen eventually. The problem is that distutils is really inflexible, and it is hard to know how to have it do what you want; I unfortunately don't have the time to build a VS 2008 python ATM. If you can't get further, don't hesitate to ask again in a few days, cheers, David From civija at gmail.com Fri Sep 5 09:00:42 2008 From: civija at gmail.com (Miroslav Sabljic) Date: Fri, 5 Sep 2008 15:00:42 +0200 Subject: [Numpy-discussion] numpy build on windows In-Reply-To: <48C121F7.4070209@ar.media.kyoto-u.ac.jp> References: <59ccde810809040808i19777671s8d30b0a655f68da8@mail.gmail.com> <48BFFA5D.6050409@ar.media.kyoto-u.ac.jp> <48C01B23.4050904@gmail.com> <5b8d13220809041048w5b7d64bboe4f608cf9645e211@mail.gmail.com> <59ccde810809050151g457a8f7dv4a530dd780018956@mail.gmail.com> <48C0F5BB.8080708@ar.media.kyoto-u.ac.jp> <59ccde810809050446k3d6c2c61pcf7cdd998eede0b6@mail.gmail.com> <48C121F7.4070209@ar.media.kyoto-u.ac.jp> Message-ID: <59ccde810809050600i29eca7b5i85e22373a85c7b65@mail.gmail.com> On 9/5/08, David Cournapeau wrote: > The code itself will have to be buildable with VS 2008 (if only because > of python 2.6, for which VS 2008 will be the default compiler), so it > will happen eventually. > > The problem is that distutils is really inflexible, and it is hard to > know how to have it do what you want; I have tried with numpy 1.1.1 and made some progress that actually something did build this time. :) I did the same trick when I was building PIL. set DISTUTILS_USE_SDK=1 set MSSDK=1 python setup.py build (without site.cfg) and there is no more this error message: error: Python was built with Visual Studio version 9, and extensions need to be built with the same version of the compiler, but it isn't installed. The build log is in attachment. Now I just have to see about this error message I'm getting, "don't know how to compile Fortran code on platform 'nt'". > I unfortunately don't have the time to build a VS 2008 python ATM. If you can't get > further, don't hesitate to ask again in a few days, If you are willing to try I can send you my Python25_VC9 installation so you can try with it? BTW, thanks for your help so far! Second try: The attach was over 40 KB, this is ziped one. -- Best regards, Miroslav -------------- next part -------------- A non-text attachment was scrubbed... Name: numpy-1.1.1_build_log.txt.zip Type: application/zip Size: 4461 bytes Desc: not available URL: From david at ar.media.kyoto-u.ac.jp Fri Sep 5 08:47:17 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 05 Sep 2008 21:47:17 +0900 Subject: [Numpy-discussion] numpy build on windows In-Reply-To: <59ccde810809050600i29eca7b5i85e22373a85c7b65@mail.gmail.com> References: <59ccde810809040808i19777671s8d30b0a655f68da8@mail.gmail.com> <48BFFA5D.6050409@ar.media.kyoto-u.ac.jp> <48C01B23.4050904@gmail.com> <5b8d13220809041048w5b7d64bboe4f608cf9645e211@mail.gmail.com> <59ccde810809050151g457a8f7dv4a530dd780018956@mail.gmail.com> <48C0F5BB.8080708@ar.media.kyoto-u.ac.jp> <59ccde810809050446k3d6c2c61pcf7cdd998eede0b6@mail.gmail.com> <48C121F7.4070209@ar.media.kyoto-u.ac.jp> <59ccde810809050600i29eca7b5i85e22373a85c7b65@mail.gmail.com> Message-ID: <48C12A55.40900@ar.media.kyoto-u.ac.jp> Miroslav Sabljic wrote: > > The build log is in attachment. Now I just have to see about this > error message I'm getting, "don't know how to compile Fortran code on > platform 'nt'". > This is not an error per se: fortran compiler is not mandatory. You will not get all the features of numpy, but you will get a working numpy without a fortran compiler. Looking at your build.log, it looks like it built successfully, no ? cheers, David From civija at gmail.com Fri Sep 5 09:28:28 2008 From: civija at gmail.com (Miroslav Sabljic) Date: Fri, 5 Sep 2008 15:28:28 +0200 Subject: [Numpy-discussion] numpy build on windows In-Reply-To: <48C12A55.40900@ar.media.kyoto-u.ac.jp> References: <59ccde810809040808i19777671s8d30b0a655f68da8@mail.gmail.com> <48BFFA5D.6050409@ar.media.kyoto-u.ac.jp> <48C01B23.4050904@gmail.com> <5b8d13220809041048w5b7d64bboe4f608cf9645e211@mail.gmail.com> <59ccde810809050151g457a8f7dv4a530dd780018956@mail.gmail.com> <48C0F5BB.8080708@ar.media.kyoto-u.ac.jp> <59ccde810809050446k3d6c2c61pcf7cdd998eede0b6@mail.gmail.com> <48C121F7.4070209@ar.media.kyoto-u.ac.jp> <59ccde810809050600i29eca7b5i85e22373a85c7b65@mail.gmail.com> <48C12A55.40900@ar.media.kyoto-u.ac.jp> Message-ID: <59ccde810809050628k5dc3fef8q3375ba111e01c17c@mail.gmail.com> On 9/5/08, David Cournapeau wrote: > This is not an error per se: fortran compiler is not mandatory. You will > not get all the features of numpy, but you will get a working numpy > without a fortran compiler. Looking at your build.log, it looks like it > built successfully, no ? Yes it looks so, but as this is first time I'm building numpy I wasn't sure. I will now experiment further and try to build it with MKL. Tnx for your help, I really appreciate it! -- Best regards, Miroslav From david at ar.media.kyoto-u.ac.jp Fri Sep 5 09:16:41 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 05 Sep 2008 22:16:41 +0900 Subject: [Numpy-discussion] numpy build on windows In-Reply-To: <59ccde810809050628k5dc3fef8q3375ba111e01c17c@mail.gmail.com> References: <59ccde810809040808i19777671s8d30b0a655f68da8@mail.gmail.com> <48BFFA5D.6050409@ar.media.kyoto-u.ac.jp> <48C01B23.4050904@gmail.com> <5b8d13220809041048w5b7d64bboe4f608cf9645e211@mail.gmail.com> <59ccde810809050151g457a8f7dv4a530dd780018956@mail.gmail.com> <48C0F5BB.8080708@ar.media.kyoto-u.ac.jp> <59ccde810809050446k3d6c2c61pcf7cdd998eede0b6@mail.gmail.com> <48C121F7.4070209@ar.media.kyoto-u.ac.jp> <59ccde810809050600i29eca7b5i85e22373a85c7b65@mail.gmail.com> <48C12A55.40900@ar.media.kyoto-u.ac.jp> <59ccde810809050628k5dc3fef8q3375ba111e01c17c@mail.gmail.com> Message-ID: <48C13139.6040705@ar.media.kyoto-u.ac.jp> Miroslav Sabljic wrote: > > Yes it looks so, but as this is first time I'm building numpy I wasn't > sure. I will now experiment further and try to build it with MKL. Tnx > for your help, I really appreciate it! > You should first test the binary (import numpy; numpy.test()), to see whether the built was really sucessful. David From simon.palmer at gmail.com Fri Sep 5 09:46:20 2008 From: simon.palmer at gmail.com (SimonPalmer) Date: Fri, 5 Sep 2008 06:46:20 -0700 (PDT) Subject: [Numpy-discussion] sum of positive values in an array Message-ID: Hi, probably a basic question, but I'm looking for a neat way to sum all the positive values in an array of floats. I'm currently doing it the hard way, but am hoping there is some cunning and elegant syntax I can use instead From simon.palmer at gmail.com Fri Sep 5 10:02:38 2008 From: simon.palmer at gmail.com (SimonPalmer) Date: Fri, 5 Sep 2008 07:02:38 -0700 (PDT) Subject: [Numpy-discussion] argsort in descending order Message-ID: another newb question I suspect, but is there a way to instruct argsort to sort in descending order or should I just sort and reverse? From david.huard at gmail.com Fri Sep 5 10:04:49 2008 From: david.huard at gmail.com (David Huard) Date: Fri, 5 Sep 2008 10:04:49 -0400 Subject: [Numpy-discussion] BUG in numpy.loadtxt? In-Reply-To: <48C04959.3080309@gmail.com> References: <48C02411.70602@gmail.com> <48C0333F.9060709@enthought.com> <48C04959.3080309@gmail.com> Message-ID: <91cf711d0809050704r16c1a40cw3176d3aee1f33606@mail.gmail.com> Hi Ryan, I applied your patch in r5788 on the trunk. I noticed there was another bug occurring when both converters and usecols are provided. I've added regression tests for both bugs. Could you confirm that everything is fine on your side ? Thanks, On Thu, Sep 4, 2008 at 4:47 PM, Ryan May wrote: > Travis E. Oliphant wrote: > > Ryan May wrote: > >> Stefan (or anyone else who can comment), > >> > >> It appears that the usecols argument to loadtxt no longer accepts numpy > >> arrays: > >> > > > > Could you enter a ticket so we don't lose track of this. I don't > > remember anything being intentional. > > > > Done: #905 > http://scipy.org/scipy/numpy/ticket/905 > > I've attached a patch that does the obvious and coerces usecols to a > list when it's not None, so it will work for any iterable. > > I don't think it was a conscious decision, just a consequence of the > rewrite using different methods. There are two problems: > > 1) It's an API break, technically speaking > 2) It currently doesn't even accept tuples, which are used in the > docstring. > > Can we hurry and get this into 1.2? > > Thanks, > Ryan > > -- > Ryan May > Graduate Research Assistant > School of Meteorology > University of Oklahoma > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.pincus at yale.edu Fri Sep 5 10:33:17 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Fri, 5 Sep 2008 10:33:17 -0400 Subject: [Numpy-discussion] sum of positive values in an array In-Reply-To: References: Message-ID: <27B24AB4-427B-4C80-B2FD-12AB885E2B31@yale.edu> > > Hi, probably a basic question, but I'm looking for a neat way to sum > all the positive values in an array of floats. I'm currently doing it > the hard way, but am hoping there is some cunning and elegant syntax I > can use instead Fancy indexing's my favorite cunning and elegant syntax: a = numpy.random.randn(100) a.sum() a > 0 # gives a boolean array that's True where the elements of a are > 0. a[a > 0] # retrieve from a only the elements with True in the boolean array. a[a > 0].sum() # your desired result. Note that you can also fancy-index with a list/array of indices, as well as a boolean "mask" array: a[[1, 40, 55]] # pulls just the values at index 1, 40, and 55 out of a into a shape (3,) array. Zach From ludwigbrinckmann at gmail.com Fri Sep 5 10:45:57 2008 From: ludwigbrinckmann at gmail.com (Ludwig) Date: Fri, 5 Sep 2008 15:45:57 +0100 Subject: [Numpy-discussion] sum of positive values in an array In-Reply-To: <27B24AB4-427B-4C80-B2FD-12AB885E2B31@yale.edu> References: <27B24AB4-427B-4C80-B2FD-12AB885E2B31@yale.edu> Message-ID: <3f7a6e1c0809050745r67db1d12hddc5de6459439957@mail.gmail.com> What are the relative merits of sum(a[where(a>0]) to a[a > 0].sum() ? Second one is more OO, takes a few keystrokes less to type. Is there any real difference if it came to very large arrays? Or is it 'just' a style question? Ludwig 2008/9/5 Zachary Pincus > > > > Hi, probably a basic question, but I'm looking for a neat way to sum > > all the positive values in an array of floats. I'm currently doing it > > the hard way, but am hoping there is some cunning and elegant syntax I > > can use instead > > Fancy indexing's my favorite cunning and elegant syntax: > > a = numpy.random.randn(100) > a.sum() > a > 0 # gives a boolean array that's True where the elements of a are > > 0. > a[a > 0] # retrieve from a only the elements with True in the boolean > array. > a[a > 0].sum() # your desired result. > > Note that you can also fancy-index with a list/array of indices, as > well as a boolean "mask" array: > a[[1, 40, 55]] # pulls just the values at index 1, 40, and 55 out of a > into a shape (3,) array. > > Zach > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Fri Sep 5 10:32:41 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 05 Sep 2008 23:32:41 +0900 Subject: [Numpy-discussion] sum of positive values in an array In-Reply-To: <3f7a6e1c0809050745r67db1d12hddc5de6459439957@mail.gmail.com> References: <27B24AB4-427B-4C80-B2FD-12AB885E2B31@yale.edu> <3f7a6e1c0809050745r67db1d12hddc5de6459439957@mail.gmail.com> Message-ID: <48C14309.308@ar.media.kyoto-u.ac.jp> Ludwig wrote: > What are the relative merits of > > sum(a[where(a>0]) > > to > > a[a > 0].sum() > > ? > Second one is more OO, takes a few keystrokes less to type. Is there > any real difference if it came to very large arrays? Or is it 'just' a > style question? In this case, it is style: sum(a) really calls a.sum() internally; in the second case, you are sure to call numpy sum, and not python sum, which is a common mistake of people new to numpy. cheers, David From kwgoodman at gmail.com Fri Sep 5 10:52:39 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 5 Sep 2008 07:52:39 -0700 Subject: [Numpy-discussion] sum of positive values in an array In-Reply-To: <48C14309.308@ar.media.kyoto-u.ac.jp> References: <27B24AB4-427B-4C80-B2FD-12AB885E2B31@yale.edu> <3f7a6e1c0809050745r67db1d12hddc5de6459439957@mail.gmail.com> <48C14309.308@ar.media.kyoto-u.ac.jp> Message-ID: On Fri, Sep 5, 2008 at 7:32 AM, David Cournapeau wrote: > Ludwig wrote: >> What are the relative merits of >> >> sum(a[where(a>0]) >> >> to >> >> a[a > 0].sum() >> >> ? >> Second one is more OO, takes a few keystrokes less to type. Is there >> any real difference if it came to very large arrays? Or is it 'just' a >> style question? > > In this case, it is style: sum(a) really calls a.sum() internally; in > the second case, you are sure to call numpy sum, and not python sum, > which is a common mistake of people new to numpy. Here's another difference: >> a = np.random.randn(100000) >> timeit np.sum(a[np.where(a>0)]) 100 loops, best of 3: 3.44 ms per loop >> timeit a[a > 0].sum() 100 loops, best of 3: 2.21 ms per loop From cournape at gmail.com Fri Sep 5 12:08:54 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 6 Sep 2008 01:08:54 +0900 Subject: [Numpy-discussion] sum of positive values in an array In-Reply-To: References: <27B24AB4-427B-4C80-B2FD-12AB885E2B31@yale.edu> <3f7a6e1c0809050745r67db1d12hddc5de6459439957@mail.gmail.com> <48C14309.308@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220809050908v6885123ay62dba32f6e32b174@mail.gmail.com> On Fri, Sep 5, 2008 at 11:52 PM, Keith Goodman wrote: > Here's another difference: > >>> a = np.random.randn(100000) >>> timeit np.sum(a[np.where(a>0)]) > 100 loops, best of 3: 3.44 ms per loop >>> timeit a[a > 0].sum() > 100 loops, best of 3: 2.21 ms per loop But you're not comparing the same thing: why calling where in one case and not in the other ? The difference is in the where call, not in the a.sum() vs sum(a) cheers, David From kwgoodman at gmail.com Fri Sep 5 12:22:15 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 5 Sep 2008 09:22:15 -0700 Subject: [Numpy-discussion] sum of positive values in an array In-Reply-To: <5b8d13220809050908v6885123ay62dba32f6e32b174@mail.gmail.com> References: <27B24AB4-427B-4C80-B2FD-12AB885E2B31@yale.edu> <3f7a6e1c0809050745r67db1d12hddc5de6459439957@mail.gmail.com> <48C14309.308@ar.media.kyoto-u.ac.jp> <5b8d13220809050908v6885123ay62dba32f6e32b174@mail.gmail.com> Message-ID: On Fri, Sep 5, 2008 at 9:08 AM, David Cournapeau wrote: > On Fri, Sep 5, 2008 at 11:52 PM, Keith Goodman wrote: > >> Here's another difference: >> >>>> a = np.random.randn(100000) >>>> timeit np.sum(a[np.where(a>0)]) >> 100 loops, best of 3: 3.44 ms per loop >>>> timeit a[a > 0].sum() >> 100 loops, best of 3: 2.21 ms per loop > > But you're not comparing the same thing: why calling where in one case > and not in the other ? The difference is in the where call, not in the > a.sum() vs sum(a) I was comparing the two techniques from this question: Ludwig wrote: > What are the relative merits of > > sum(a[where(a>0]) > > to > > a[a > 0].sum() From rmay31 at gmail.com Fri Sep 5 12:36:45 2008 From: rmay31 at gmail.com (Ryan May) Date: Fri, 05 Sep 2008 11:36:45 -0500 Subject: [Numpy-discussion] BUG in numpy.loadtxt? In-Reply-To: <91cf711d0809050704r16c1a40cw3176d3aee1f33606@mail.gmail.com> References: <48C02411.70602@gmail.com> <48C0333F.9060709@enthought.com> <48C04959.3080309@gmail.com> <91cf711d0809050704r16c1a40cw3176d3aee1f33606@mail.gmail.com> Message-ID: <48C1601D.6010307@gmail.com> David Huard wrote: > Hi Ryan, > > I applied your patch in r5788 on the trunk. > I noticed there was another bug occurring when both converters and > usecols are provided. > I've added regression tests for both bugs. Could you confirm that > everything is fine on your side ? > I can confirm that it works fine for me. Can you or someone else backport this to the 1.2 branch so that this bug is fixed in the next release? Thanks, Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From charlesr.harris at gmail.com Fri Sep 5 12:43:15 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 5 Sep 2008 10:43:15 -0600 Subject: [Numpy-discussion] argsort in descending order In-Reply-To: References: Message-ID: On Fri, Sep 5, 2008 at 8:02 AM, SimonPalmer wrote: > another newb question I suspect, but is there a way to instruct > argsort to sort in descending order or should I just sort and reverse? You'll just have to reverse the indices. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.huard at gmail.com Fri Sep 5 12:56:37 2008 From: david.huard at gmail.com (David Huard) Date: Fri, 5 Sep 2008 12:56:37 -0400 Subject: [Numpy-discussion] BUG in numpy.loadtxt? In-Reply-To: <48C1601D.6010307@gmail.com> References: <48C02411.70602@gmail.com> <48C0333F.9060709@enthought.com> <48C04959.3080309@gmail.com> <91cf711d0809050704r16c1a40cw3176d3aee1f33606@mail.gmail.com> <48C1601D.6010307@gmail.com> Message-ID: <91cf711d0809050956w7986f0ax649abb3944491541@mail.gmail.com> Done in r5790. On Fri, Sep 5, 2008 at 12:36 PM, Ryan May wrote: > David Huard wrote: > > Hi Ryan, > > > > I applied your patch in r5788 on the trunk. > > I noticed there was another bug occurring when both converters and > > usecols are provided. > > I've added regression tests for both bugs. Could you confirm that > > everything is fine on your side ? > > > > I can confirm that it works fine for me. Can you or someone else > backport this to the 1.2 branch so that this bug is fixed in the next > release? > > Thanks, > > Ryan > > -- > Ryan May > Graduate Research Assistant > School of Meteorology > University of Oklahoma > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmay31 at gmail.com Fri Sep 5 13:35:32 2008 From: rmay31 at gmail.com (Ryan May) Date: Fri, 05 Sep 2008 12:35:32 -0500 Subject: [Numpy-discussion] BUG in numpy.loadtxt? In-Reply-To: <91cf711d0809050956w7986f0ax649abb3944491541@mail.gmail.com> References: <48C02411.70602@gmail.com> <48C0333F.9060709@enthought.com> <48C04959.3080309@gmail.com> <91cf711d0809050704r16c1a40cw3176d3aee1f33606@mail.gmail.com> <48C1601D.6010307@gmail.com> <91cf711d0809050956w7986f0ax649abb3944491541@mail.gmail.com> Message-ID: <48C16DE4.9090405@gmail.com> Thanks a bunch for getting these done. David Huard wrote: > Done in r5790. > > On Fri, Sep 5, 2008 at 12:36 PM, Ryan May > wrote: > > David Huard wrote: > > Hi Ryan, > > > > I applied your patch in r5788 on the trunk. > > I noticed there was another bug occurring when both converters and > > usecols are provided. > > I've added regression tests for both bugs. Could you confirm that > > everything is fine on your side ? > > > > I can confirm that it works fine for me. Can you or someone else > backport this to the 1.2 branch so that this bug is fixed in the next > release? > > Thanks, > > Ryan > > -- > Ryan May > Graduate Research Assistant > School of Meteorology > University of Oklahoma > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From simon.palmer at gmail.com Fri Sep 5 15:12:15 2008 From: simon.palmer at gmail.com (SimonPalmer) Date: Fri, 5 Sep 2008 12:12:15 -0700 (PDT) Subject: [Numpy-discussion] sum of positive values in an array In-Reply-To: <27B24AB4-427B-4C80-B2FD-12AB885E2B31@yale.edu> References: <27B24AB4-427B-4C80-B2FD-12AB885E2B31@yale.edu> Message-ID: thank you very much, deliciously satisfying On Sep 5, 3:33?pm, Zachary Pincus wrote: > > Hi, probably a basic question, but I'm looking for a neat way to sum > > all the positive values in an array of floats. ?I'm currently doing it > > the hard way, but am hoping there is some cunning and elegant syntax I > > can use instead > > Fancy indexing's my favorite cunning and elegant syntax: > > a = numpy.random.randn(100) > a.sum() > a > 0 # gives a boolean array that's True where the elements of a are ? > ?> 0. > a[a > 0] # retrieve from a only the elements with True in the boolean ? > array. > a[a > 0].sum() # your desired result. > > Note that you can also fancy-index with a list/array of indices, as ? > well as a boolean "mask" array: > a[[1, 40, 55]] # pulls just the values at index 1, 40, and 55 out of a ? > into a shape (3,) array. > > Zach > _______________________________________________ > Numpy-discussion mailing list > Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Fri Sep 5 19:38:39 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 5 Sep 2008 17:38:39 -0600 Subject: [Numpy-discussion] isnan and co: cleaning up In-Reply-To: <48C0BE3F.1030306@ar.media.kyoto-u.ac.jp> References: <48C0BE3F.1030306@ar.media.kyoto-u.ac.jp> Message-ID: On Thu, Sep 4, 2008 at 11:06 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Hi, > > While working on my branch to clean the math configuration, I > noticed that the code for isnan and co became quite convoluted. autoconf > info file has a mention of it, and suggests the following for > portability (section 5.5.1 of autoconf): > > The C99 standard says that `isinf' and `isnan' are macros. On > some systems just macros are available (e.g., HP-UX and Solaris > 10), on some systems both macros and functions (e.g., glibc > 2.3.2), and on some systems only functions (e.g., IRIX 6 and > Solaris 9). In some cases these functions are declared in > nonstandard headers like `' and defined in non-default > libraries like `-lm' or `-lsunmath'. > > The C99 `isinf' and `isnan' macros work correctly with `long > double' arguments, but pre-C99 systems that use functions > typically assume `double' arguments. On such a system, `isinf' > incorrectly returns true for a finite `long double' argument that > is outside the range of `double'. > > To work around this porting mess, you can use code like the > following. > > #include > > #ifndef isnan > # define isnan(x) \ > (sizeof (x) == sizeof (long double) ? isnan_ld (x) \ > : sizeof (x) == sizeof (double) ? isnan_d (x) \ > : isnan_f (x)) > static inline int isnan_f (float x) { return x != x; } > static inline int isnan_d (double x) { return x != x; } > static inline int isnan_ld (long double x) { return x != x; } > #endif > > #ifndef isinf > # define isinf(x) \ > (sizeof (x) == sizeof (long double) ? isinf_ld (x) \ > : sizeof (x) == sizeof (double) ? isinf_d (x) \ > : isinf_f (x)) > static inline int isinf_f (float x) { return isnan (x - > x); } > static inline int isinf_d (double x) { return isnan (x - > x); } > static inline int isinf_ld (long double x) { return isnan (x - > x); } > #endif > > Use `AC_C_INLINE' (*note C Compiler::) so that this code works on > compilers that lack the `inline' keyword. Some optimizing > compilers mishandle these definitions, but systems with that bug > typically have missing or broken `isnan' functions anyway, so it's > probably not worth worrying about. > > This is simpler than the current code (I actually understand what the > above case does; the current code in numpy has many cases which I do not > understand), does not need any C code (_isnan and co), and probably more > portable since autoconf has a lot of experience there. Is the code in > _isnan really better than just relying on the nan property x != x ? (Do > we support platforms without a IEE754 FPU ?) > > Does anyone has any thought on replacing the current code by the above > (modulo the inline part, which we can drop for now) ? > +1, but (putting on pedant hat) I think the functions should be formatted in the way revealed to us by K&R. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Fri Sep 5 23:15:00 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 6 Sep 2008 12:15:00 +0900 Subject: [Numpy-discussion] isnan and co: cleaning up In-Reply-To: References: <48C0BE3F.1030306@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220809052015o9cda310v86637e28595bc10f@mail.gmail.com> On Sat, Sep 6, 2008 at 8:38 AM, Charles R Harris wrote: > > +1, but (putting on pedant hat) I think the functions should be formatted in > the way revealed to us by K&R. By K&R, you mean the K&R style, right , not the K&R function declaration style ? :) cheers, David From cournape at gmail.com Fri Sep 5 23:15:59 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 6 Sep 2008 12:15:59 +0900 Subject: [Numpy-discussion] Are the CPP symbols HAVE_LONGDOUBLE_FUNCS and HAVE_FLOAT_FUNCS public ? In-Reply-To: References: <48C0D16C.9000000@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220809052015t4e05dac9yd2dea7c02b8c9a73@mail.gmail.com> On Fri, Sep 5, 2008 at 4:50 PM, Pauli Virtanen wrote: > > If you remove them, please put a reminder somewhere (eg ticket) to update > the documentation accordingly. Is there a reason for not updating the documentation in the patch (well, branch) itself ? cheers, David From charlesr.harris at gmail.com Fri Sep 5 23:43:30 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 5 Sep 2008 21:43:30 -0600 Subject: [Numpy-discussion] isnan and co: cleaning up In-Reply-To: <5b8d13220809052015o9cda310v86637e28595bc10f@mail.gmail.com> References: <48C0BE3F.1030306@ar.media.kyoto-u.ac.jp> <5b8d13220809052015o9cda310v86637e28595bc10f@mail.gmail.com> Message-ID: On Fri, Sep 5, 2008 at 9:15 PM, David Cournapeau wrote: > On Sat, Sep 6, 2008 at 8:38 AM, Charles R Harris > wrote: > > > > +1, but (putting on pedant hat) I think the functions should be formatted > in > > the way revealed to us by K&R. > > By K&R, you mean the K&R style, right , not the K&R function > declaration style ? :) > This one int main(int argc, char *argv[]) { ... while (x == y) { something(); somethingelse(); if (some_error) do_correct(); else continue_as_usual(); } finalthing(); ... } I'm a bit of a heretic myself, I always use braces in if statements, even for one liners. It's just a bit easier on my eyes. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dlenski at gmail.com Sat Sep 6 02:21:57 2008 From: dlenski at gmail.com (Dan Lenski) Date: Sat, 6 Sep 2008 06:21:57 +0000 (UTC) Subject: [Numpy-discussion] argsort in descending order References: Message-ID: On Fri, 05 Sep 2008 07:02:38 -0700, SimonPalmer wrote: > another newb question I suspect, but is there a way to instruct argsort > to sort in descending order or should I just sort and reverse? Just sort and subtract to get the reverse order. I can think of two reasonable ways to do it with no copying: from numpy import * a = rand(100) # this does NOT make a copy of a, simply indexes it backward, and should # be very fast reverseorder = argsort(a[::-1]) # this way doesn't make a copy either, and probably is a little slower # since it has to do many subtractions reverseorder = len(a)-1-argsort(a) HTH, Dan From dmitrey.kroshko at scipy.org Sat Sep 6 03:21:51 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Sat, 06 Sep 2008 10:21:51 +0300 Subject: [Numpy-discussion] does numpy have funcs like isanynan() or isallfinite()? Message-ID: <48C22F8F.3030509@scipy.org> hi all, does numpy have funcs like isanynan(array) or isallfinite(array)? I very often use any(isnan(my_array)) or all(isfinite(my_array)), I guess having a single case triggered on would be enough here to omit further checks. Regards, D. From david at ar.media.kyoto-u.ac.jp Sat Sep 6 04:40:33 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 06 Sep 2008 17:40:33 +0900 Subject: [Numpy-discussion] isnan and co: cleaning up In-Reply-To: References: <48C0BE3F.1030306@ar.media.kyoto-u.ac.jp> <5b8d13220809052015o9cda310v86637e28595bc10f@mail.gmail.com> Message-ID: <48C24201.90103@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > > This one > > int main(int argc, char *argv[]) > > { > ... > while (x == y) { > something(); > > somethingelse(); > if (some_error) > do_correct(); > > else > continue_as_usual(); > } > finalthing(); > > ... > } > > I'm a bit of a heretic myself, I always use braces in if statements, even for one liners. me too ( I believe it avoids mistakes when the one statement becomes 2 - although recent gcc may have a warning for that). I think I will use the same technique as R to detect those functions (because it may be a macro or a function; C99 says function, but old platform and the MS one do not have macro). cheers, David From robert.kern at gmail.com Sat Sep 6 06:18:29 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 6 Sep 2008 05:18:29 -0500 Subject: [Numpy-discussion] does numpy have funcs like isanynan() or isallfinite()? In-Reply-To: <48C22F8F.3030509@scipy.org> References: <48C22F8F.3030509@scipy.org> Message-ID: <3d375d730809060318yf2ed65bxf71220fc024bda74@mail.gmail.com> On Sat, Sep 6, 2008 at 02:21, dmitrey wrote: > hi all, > > does numpy have funcs like isanynan(array) or isallfinite(array)? No. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pav at iki.fi Sat Sep 6 06:45:24 2008 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 6 Sep 2008 10:45:24 +0000 (UTC) Subject: [Numpy-discussion] Are the CPP symbols HAVE_LONGDOUBLE_FUNCS and HAVE_FLOAT_FUNCS public ? References: <48C0D16C.9000000@ar.media.kyoto-u.ac.jp> <5b8d13220809052015t4e05dac9yd2dea7c02b8c9a73@mail.gmail.com> Message-ID: Sat, 06 Sep 2008 12:15:59 +0900, David Cournapeau wrote: > On Fri, Sep 5, 2008 at 4:50 PM, Pauli Virtanen wrote: >> >> If you remove them, please put a reminder somewhere (eg ticket) to >> update the documentation accordingly. > > Is there a reason for not updating the documentation in the patch (well, > branch) itself ? Not really. Also that would be OK, just as long as someone remembers to do it. -- Pauli Virtanen From cournape at gmail.com Sat Sep 6 23:52:59 2008 From: cournape at gmail.com (David Cournapeau) Date: Sun, 7 Sep 2008 12:52:59 +0900 Subject: [Numpy-discussion] isnan and co: cleaning up In-Reply-To: <48C24201.90103@ar.media.kyoto-u.ac.jp> References: <48C0BE3F.1030306@ar.media.kyoto-u.ac.jp> <5b8d13220809052015o9cda310v86637e28595bc10f@mail.gmail.com> <48C24201.90103@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220809062052v31d319a7lc6fc8d0fbcbd37d@mail.gmail.com> On Sat, Sep 6, 2008 at 5:40 PM, David Cournapeau wrote: > Charles R Harris wrote: >> >> >> This one >> Patch available here: http://scipy.org/scipy/numpy/ticket/907 Tested with gcc on mac os X, it also builds with VS 2003 (but I could not test there because of another problem with numpy and VS 2003, unrelated to this). David From mforbes at physics.ubc.ca Sun Sep 7 10:36:20 2008 From: mforbes at physics.ubc.ca (Michael McNeil Forbes) Date: Sun, 7 Sep 2008 08:36:20 -0600 Subject: [Numpy-discussion] sum of positive values in an array In-Reply-To: References: <27B24AB4-427B-4C80-B2FD-12AB885E2B31@yale.edu> <3f7a6e1c0809050745r67db1d12hddc5de6459439957@mail.gmail.com> <48C14309.308@ar.media.kyoto-u.ac.jp> Message-ID: <519079E1-AFB6-4D77-BB1C-7E609AE4A013@physics.ubc.ca> On Sep 5, 2008, at 8:52 AM, Keith Goodman wrote: > Here's another difference: > >>> a = np.random.randn(100000) >>> timeit np.sum(a[np.where(a>0)]) > 100 loops, best of 3: 3.44 ms per loop >>> timeit a[a > 0].sum() > 100 loops, best of 3: 2.21 ms per loop Here is an even faster method (but much more ugly!): 0.25ms: (a.sum() + abs(a).sum())/2 0.34ms: a[a > 0].sum() 0.55ms: np.sum(a[np.where(a>0)]) Michael. From mforbes at physics.ubc.ca Sun Sep 7 10:35:25 2008 From: mforbes at physics.ubc.ca (Michael McNeil Forbes) Date: Sun, 7 Sep 2008 08:35:25 -0600 Subject: [Numpy-discussion] sum of positive values in an array In-Reply-To: References: <27B24AB4-427B-4C80-B2FD-12AB885E2B31@yale.edu> <3f7a6e1c0809050745r67db1d12hddc5de6459439957@mail.gmail.com> <48C14309.308@ar.media.kyoto-u.ac.jp> Message-ID: On Sep 5, 2008, at 8:52 AM, Keith Goodman wrote: > Here's another difference: > >>> a = np.random.randn(100000) >>> timeit np.sum(a[np.where(a>0)]) > 100 loops, best of 3: 3.44 ms per loop >>> timeit a[a > 0].sum() > 100 loops, best of 3: 2.21 ms per loop Here is an even faster method (but much more ugly!): 0.25ms: (a.sum() + abs(a).sum())/2 0.34ms: a[a > 0].sum() 0.55ms: np.sum(a[np.where(a>0)]) Michael. From eads at soe.ucsc.edu Sun Sep 7 15:56:50 2008 From: eads at soe.ucsc.edu (Damian Eads) Date: Sun, 07 Sep 2008 13:56:50 -0600 Subject: [Numpy-discussion] distance matrix and (weighted) p-norm In-Reply-To: <48BE9DAC.30103@relativita.com> References: <48BDB96A.4060506@relativita.com> <48BDF137.7060702@ar.media.kyoto-u.ac.jp> <48BE8D85.6020604@relativita.com> <48BE8E3A.8070100@ar.media.kyoto-u.ac.jp> <48BE9DAC.30103@relativita.com> Message-ID: <48C43202.6040805@soe.ucsc.edu> Hi there, The pdist function computes pairwise distances between vectors in a single collection, storing the distances in a condensed distance matrix. This is not exactly what you want--you want to compute distance between two collections of vectors. Suppose XA is a m_A by n array and XB is a m_B by n array, M=scipy.cluster.distance.cdist(XA, XB, metric='mahalanobis') computes a m_A by m_B distance matrix M. The ij'th entry is the distance between XA[i,:] and XB[j,:]. The core computation is implemented in C for efficiency. I've committed the new function along with documentation and about two dozen tests. Cheers, Damian Emanuele Olivetti wrote: > David Cournapeau wrote: >> FWIW, distance is deemed to move to a separate package, because distance >> computation is useful in other contexts than clustering. >> >> > > Excellent. I was thinking about something similar. I'll have a look > to the separate package. Please drop an email to this list when > distance will be moved. > > Thanks, > > Emanuele ----------------------------------------------------- Damian Eads Ph.D. Student Jack Baskin School of Engineering, UCSC E2-479 1156 High Street Santa Cruz, CA 95064 http://www.soe.ucsc.edu/~eads From emanuele at relativita.com Sun Sep 7 19:07:49 2008 From: emanuele at relativita.com (Emanuele Olivetti) Date: Mon, 08 Sep 2008 01:07:49 +0200 Subject: [Numpy-discussion] distance matrix and (weighted) p-norm In-Reply-To: <48C43202.6040805@soe.ucsc.edu> References: <48BDB96A.4060506@relativita.com> <48BDF137.7060702@ar.media.kyoto-u.ac.jp> <48BE8D85.6020604@relativita.com> <48BE8E3A.8070100@ar.media.kyoto-u.ac.jp> <48BE9DAC.30103@relativita.com> <48C43202.6040805@soe.ucsc.edu> Message-ID: <48C45EC5.1020305@relativita.com> Excellent. David said that distance computation will be moved in a separate package soon. I guess that your implementation will be the suitable one for this package. Am I wrong? Thanks again, Emanuele Damian Eads wrote: > Hi there, > > The pdist function computes pairwise distances between vectors in a > single collection, storing the distances in a condensed distance matrix. > This is not exactly what you want--you want to compute distance > between two collections of vectors. > > Suppose XA is a m_A by n array and XB is a m_B by n array, > > M=scipy.cluster.distance.cdist(XA, XB, metric='mahalanobis') > > computes a m_A by m_B distance matrix M. The ij'th entry is the distance > between XA[i,:] and XB[j,:]. The core computation is implemented in C > for efficiency. I've committed the new function along with documentation > and about two dozen tests. > > Cheers, > > Damian > > Emanuele Olivetti wrote: > >> David Cournapeau wrote: >> >>> FWIW, distance is deemed to move to a separate package, because distance >>> computation is useful in other contexts than clustering. >>> >>> >>> >> Excellent. I was thinking about something similar. I'll have a look >> to the separate package. Please drop an email to this list when >> distance will be moved. >> >> Thanks, >> >> Emanuele >> > > ----------------------------------------------------- > Damian Eads Ph.D. Student > Jack Baskin School of Engineering, UCSC E2-479 > 1156 High Street > Santa Cruz, CA 95064 http://www.soe.ucsc.edu/~eads > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From millman at berkeley.edu Sun Sep 7 19:33:11 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Sun, 7 Sep 2008 16:33:11 -0700 Subject: [Numpy-discussion] distance matrix and (weighted) p-norm In-Reply-To: <48C45EC5.1020305@relativita.com> References: <48BDB96A.4060506@relativita.com> <48BDF137.7060702@ar.media.kyoto-u.ac.jp> <48BE8D85.6020604@relativita.com> <48BE8E3A.8070100@ar.media.kyoto-u.ac.jp> <48BE9DAC.30103@relativita.com> <48C43202.6040805@soe.ucsc.edu> <48C45EC5.1020305@relativita.com> Message-ID: On Sun, Sep 7, 2008 at 4:07 PM, Emanuele Olivetti wrote: > David said that distance computation will be moved in a > separate package soon. I guess that your implementation > will be the suitable one for this package. Am I wrong? Yes, that is correct. David was talking about pulling Damian's distance code out of scipy.cluster and into its own scipy subpackage. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From eads at soe.ucsc.edu Mon Sep 8 11:15:24 2008 From: eads at soe.ucsc.edu (Damian Eads) Date: Mon, 08 Sep 2008 09:15:24 -0600 Subject: [Numpy-discussion] distance matrix and (weighted) p-norm In-Reply-To: <48BDB96A.4060506@relativita.com> References: <48BDB96A.4060506@relativita.com> Message-ID: <48C5418C.3080801@soe.ucsc.edu> Emanuele Olivetti wrote: > Hi, > > I'm trying to compute the distance matrix (weighted p-norm [*]) > between two sets of vectors (data1 and data2). Example: > > import numpy as N > p = 3.0 > data1 = N.random.randn(100,20) > data2 = N.random.randn(80,20) > weight = N.random.rand(20) > distance_matrix = N.zeros((data1.shape[0],data2.shape[0])) > for d in range(data1.shape[1]): > distance_matrix += > (N.abs(N.subtract.outer(data1[:,d],data2[:,d]))*weight[d])**p > pass > distance_matrix = distance_matrix**(1.0/p) > > > Is there a way to speed up the for loop? When the dimension > of the vectors becomes big (e.g. >1000) the for loop > becomes really annoying. > > Thanks, > > Emanuele > > [*] : ||x - x'||_w = (\sum_{i=1...N} (w_i*|x_i - x'_i|)**p)**(1/p) This feature could be implemented easily. However, I must admit I'm not very familiar with weighted p-norms. What is the reason for raising w to the p instead of w_i*(|x_i-x'_i|)**p? Damian From emanuele at relativita.com Mon Sep 8 12:04:32 2008 From: emanuele at relativita.com (Emanuele Olivetti) Date: Mon, 08 Sep 2008 18:04:32 +0200 Subject: [Numpy-discussion] distance matrix and (weighted) p-norm In-Reply-To: <48C5418C.3080801@soe.ucsc.edu> References: <48BDB96A.4060506@relativita.com> <48C5418C.3080801@soe.ucsc.edu> Message-ID: <48C54D10.5080002@relativita.com> Damian Eads wrote: > Emanuele Olivetti wrote: >> ... >> [*] : ||x - x'||_w = (\sum_{i=1...N} (w_i*|x_i - x'_i|)**p)**(1/p) > > This feature could be implemented easily. However, I must admit I'm not > very familiar with weighted p-norms. What is the reason for raising w > to the p instead of w_i*(|x_i-x'_i|)**p? > I believe that it is just a choice, that should be clearly expressed since the two formulations lead to different results. I think the expression I wrote is more convenient, since it gives what it is expected even in limit cases. 2 examples: 1) if |x-x'|=N.ones(n) , then ||x-x'||_w,p = ||w||_p in your case: ||x-x'||_w,p = (\sum(w_i))**(1/p) breaking this symmetry 2) when p goes to inifinity in my case: ||x-x'||_w,inf = max(w_i*|x_i-x'_i|_{i=1,...,n}) in your case: ||x-x'||_w,inf = max(|x_i-x'_i|_{i=1,...,n}) = ||x-x'||_1,inf But I welcome any comment on this topic! Emanuele From dblubaugh at belcan.com Mon Sep 8 13:35:20 2008 From: dblubaugh at belcan.com (Blubaugh, David A.) Date: Mon, 8 Sep 2008 13:35:20 -0400 Subject: [Numpy-discussion] F2PY ?? Has anyone worked with the F2PY generator? Message-ID: <27CC3060AF71DA40A5DC85F7D5B70F3804F0EC8E@AWMAIL04.belcan.com> To All, Has anyone worked with the F2PY generator? This is something that is supposedly built within numpy and scipy. I was wondering if anyone has had any issues with this environment?? It is important for my current employment!! Thanks, David Blubaugh This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated. From pav at iki.fi Mon Sep 8 14:04:13 2008 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 8 Sep 2008 18:04:13 +0000 (UTC) Subject: [Numpy-discussion] F2PY ?? Has anyone worked with the F2PY generator? References: <27CC3060AF71DA40A5DC85F7D5B70F3804F0EC8E@AWMAIL04.belcan.com> Message-ID: Hi, Mon, 08 Sep 2008 13:35:20 -0400, Blubaugh, David A. wrote: > Has anyone worked with the F2PY generator? This is something that is > supposedly built within numpy and scipy. I was wondering if anyone has > had any issues with this environment? I haven't had any problems with it, and I'm using it daily. If you have some problems, please give more information, so that we have better change to understand what's wrong. For example - What are you trying to do? - Which platform are you running f2py on (Windows, Cygwin on Windows, Solaris, etc.) - Which version you have installed. Eg. what does "f2py -v" output? And "python -c 'import numpy; print numpy.__version__'" - The exact commands you typed that didn't work as expected. - Full output produced by the commands. - A minimal example, with source code, that should work but fails. -- Pauli Virtanen From Chris.Barker at noaa.gov Mon Sep 8 14:47:45 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 08 Sep 2008 11:47:45 -0700 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> <789d27b10809030326w1000132frb0d530f1dfcd4729@mail.gmail.com> Message-ID: <48C57351.4070006@noaa.gov> Matthieu Brucher wrote: > People are concerned by memory consumption, so if you use more memory > than what you think, you can encounter bugs. Least surprise is always > better ;) sure, but I'm a bit confused -- there is a tiny amount of memory involved in the accumulator -- isn't it a single number? (or, for doing it along an axis, at least it's one order of magnitude fewer numbers) And with a double precision floating point unit, is the math any slower? would you have to push any more memory around? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From markperrymiller at gmail.com Mon Sep 8 16:14:31 2008 From: markperrymiller at gmail.com (Mark Miller) Date: Mon, 8 Sep 2008 13:14:31 -0700 Subject: [Numpy-discussion] just curious...why does numpy.where() return tuples? Message-ID: Just for my own benefit, I am curious about this. I am running into problems because I need to archive the result (tuple) returned by a numpy.where statement. Pickle does not seem to like to deal with numpy scalars, and numpy's archiving functions (memmap) can't work on the tuple that gets returned by the where functions (I think). Is there a way around this? All would be good if the where statements actually returned numpy arrays instead. Thanks, -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 8 17:01:30 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 8 Sep 2008 16:01:30 -0500 Subject: [Numpy-discussion] just curious...why does numpy.where() return tuples? In-Reply-To: References: Message-ID: <3d375d730809081401y3f694125la8db628c5d4b4325@mail.gmail.com> On Mon, Sep 8, 2008 at 15:14, Mark Miller wrote: > Just for my own benefit, I am curious about this. > > I am running into problems because I need to archive the result (tuple) > returned by a numpy.where statement. Pickle does not seem to like to deal > with numpy scalars, and numpy's archiving functions (memmap) can't work on > the tuple that gets returned by the where functions (I think). > > Is there a way around this? All would be good if the where statements > actually returned numpy arrays instead. I assume that you are talking about where()'s single-argument form which is equivalent to nonzero() (which I recommend using instead). The reason that it returns a tuple is that the result is intended to be usable as a "fancy" index. E.g. In [4]: x = random.randint(0, 2, [3, 10]) In [5]: x Out[5]: array([[1, 0, 0, 0, 0, 0, 1, 1, 0, 0], [1, 1, 0, 0, 1, 1, 1, 0, 0, 1], [1, 0, 0, 1, 0, 1, 0, 1, 0, 1]]) In [6]: nonzero(x) Out[6]: (array([0, 0, 0, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2]), array([0, 6, 7, 0, 1, 4, 5, 6, 9, 0, 3, 5, 7, 9])) In [7]: x[nonzero(x)] Out[7]: array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) In Python, x[i,j] is implicitly x[(i,j)]. In order to support both x[i,j] and x[some_array] (i.e., some_array is an array indexing on the first axis only), we differentiate the inputs by type. In [8]: x[array(nonzero(x))] --------------------------------------------------------------------------- IndexError Traceback (most recent call last) /Users/rkern/svn/shell/ShellIO/ in () IndexError: index (6) out of range (0<=index<2) in dimension 0 If you want to use memmap or numpy.lib.format to save your data, then I recommend explicitly casting to an array, and possibly converting back to a tuple when you read it again. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From sebastian at sipsolutions.net Mon Sep 8 17:55:55 2008 From: sebastian at sipsolutions.net (Sebastian Stephan Berg) Date: Mon, 08 Sep 2008 23:55:55 +0200 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <48C57351.4070006@noaa.gov> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> <789d27b10809030326w1000132frb0d530f1dfcd4729@mail.gmail.com> <48C57351.4070006@noaa.gov> Message-ID: <1220910955.8226.45.camel@sebook> Yeah, I memory wise it doesn't matter to sum to a double, but just trying around it seems that the mixing of float and double is very slow (at least on my comp) while if the starting array is already double there is almost no difference for summing. Generally double precision calculations should be slower though. Don't extensions like SSE2 operate either on 2 doubles or 4 floats at once and thus should be about twice as fast for floats? For add/multiply this behaviour is for me visible anyways. Some timeings for sum and also add with different datatypes, f64 being a float64 and f32 a float32 array (times not very exact): In [20]: %timeit f32.sum(dtype='float32') 10 loops, best of 3: 70.5 ms per loop In [21]: %timeit f32.sum(dtype='float64') 10 loops, best of 3: 334 ms per loop In [22]: %timeit f64.sum(dtype='float32') 10 loops, best of 3: 329 ms per loop In [23]: %timeit f64.sum(dtype='float64') 10 loops, best of 3: 75.3 ms per loop And add (similar for multiplying): In [50]: %timeit add(f64, 1.32, f64) 10 loops, best of 3: 158 ms per loop In [51]: %timeit add(f32, 1.32, f32) 10 loops, best of 3: 78.1 ms per loop It seems here numpy picked float32 for 1.32 in the second case. If I give array([1.32], dtype='float64') performance drops to 200+ ms! Maybe this speed difference when summing to a different type is a reason for not changing the default type though. Sebastian From Chris.Barker at noaa.gov Mon Sep 8 18:33:18 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 08 Sep 2008 15:33:18 -0700 Subject: [Numpy-discussion] 1.2.0rc1 tagged! In-Reply-To: <48BE2DB2.5060304@ar.media.kyoto-u.ac.jp> References: <48BE2DB2.5060304@ar.media.kyoto-u.ac.jp> Message-ID: <48C5A82E.8000002@noaa.gov> > Jarrod Millman wrote: >> Here is the universal Mac binary: >> https://cirl.berkeley.edu/numpy/numpy-1.2.0rc1-py2.5-macosx10.5.dmg >> Please test this release ASAP OS-X 10.4.11 dual G5 PPC looks OK to me, except: ERROR: Failure: ImportError (No module named scipy) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/loader.py", line 364, in loadTestsFromName addr.filename, addr.module) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/importer.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/doc/__init__.py", line 10, in __import__(__name__ + '.' + f) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/doc/example.py", line 27, in import scipy as sp ImportError: No module named scipy ---------------------------------------------------------------------- Ran 1722 tests in 15.940s FAILED (SKIP=1, errors=1) I shouldn't need scipy to test numpy. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Mon Sep 8 18:40:11 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 8 Sep 2008 17:40:11 -0500 Subject: [Numpy-discussion] 1.2.0rc1 tagged! In-Reply-To: <48C5A82E.8000002@noaa.gov> References: <48BE2DB2.5060304@ar.media.kyoto-u.ac.jp> <48C5A82E.8000002@noaa.gov> Message-ID: <3d375d730809081540l53a065bckb505a81852cdfc49@mail.gmail.com> On Mon, Sep 8, 2008 at 17:33, Christopher Barker wrote: >> Jarrod Millman wrote: >>> Here is the universal Mac binary: >>> https://cirl.berkeley.edu/numpy/numpy-1.2.0rc1-py2.5-macosx10.5.dmg > >>> Please test this release ASAP > > OS-X 10.4.11 dual G5 PPC > > looks OK to me, except: > > ERROR: Failure: ImportError (No module named scipy) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/loader.py", > line 364, in loadTestsFromName > addr.filename, addr.module) > File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/importer.py", > line 39, in importFromPath > return self.importFromDir(dir_path, fqname) > File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/importer.py", > line 84, in importFromDir > mod = load_module(part_fqname, fh, filename, desc) > File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/doc/__init__.py", > line 10, in > __import__(__name__ + '.' + f) > File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/doc/example.py", > line 27, in > import scipy as sp > ImportError: No module named scipy > > ---------------------------------------------------------------------- > Ran 1722 tests in 15.940s > > FAILED (SKIP=1, errors=1) > > > > I shouldn't need scipy to test numpy. Hmm. That file shouldn't be there. It's not in the 1.2.0rc1 tag in SVN. svn ls https://projects.scipy.org/svn/numpy/tags/1.2.0rc1/numpy/doc $ svn ls https://projects.scipy.org/svn/numpy/tags/1.2.0rc1/numpy/doc __init__.py basics.py broadcasting.py creation.py glossary.py howtofind.py indexing.py internals.py io.py jargon.py methods_vs_functions.py misc.py performance.py structured_arrays.py subclassing.py ufuncs.py Are you sure you don't have old files from a previous installation? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Chris.Barker at noaa.gov Mon Sep 8 18:54:56 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 08 Sep 2008 15:54:56 -0700 Subject: [Numpy-discussion] 1.2.0rc1 tagged! In-Reply-To: <3d375d730809081540l53a065bckb505a81852cdfc49@mail.gmail.com> References: <48BE2DB2.5060304@ar.media.kyoto-u.ac.jp> <48C5A82E.8000002@noaa.gov> <3d375d730809081540l53a065bckb505a81852cdfc49@mail.gmail.com> Message-ID: <48C5AD40.2020607@noaa.gov> Robert Kern wrote: > Hmm. That file shouldn't be there. It's not in the 1.2.0rc1 tag in SVN. OK. I cleaned out everything numpy from site-packages, re-installed, and presto: Ran 1721 tests in 12.736s OK (SKIP=1) >>> It must have been left over from a previous install -- I should have thought of that. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From millman at berkeley.edu Mon Sep 8 19:25:00 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 8 Sep 2008 16:25:00 -0700 Subject: [Numpy-discussion] mixed-EOL problems w/ Message-ID: I have been trying to get Launchpad to do continuous imports of the SciPy subversion repository into Bazaar: https://code.launchpad.net/~vcs-imports/scipy/trunk This works fine for NumPy: https://code.launchpad.net/~vcs-imports/numpy/trunk However, this doesn't work for SciPy due to a mixed-EOL issue: https://bugs.launchpad.net/launchpad-cscvs/+bug/256050 Colin Watson has recently volunteered to look into fixing this issue and was asking if anyone had a preference about how this should be solved. Please let me know if you see any reason for one solution over another. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From dblubaugh at belcan.com Mon Sep 8 19:26:43 2008 From: dblubaugh at belcan.com (Blubaugh, David A.) Date: Mon, 8 Sep 2008 19:26:43 -0400 Subject: [Numpy-discussion] F2PY ?? Has anyone worked with the F2PY generator? Message-ID: <27CC3060AF71DA40A5DC85F7D5B70F3804F0F01C@AWMAIL04.belcan.com> Pauli, Yes, I am utilizing the windows environment. I cannot install f2py. I obtain the following error when I try to execute the setup.py file within the f2py folder located within the numpy master folder: Warning: Assuming default configuration (lib\parser/{setup_parser,setup}.py was not found) Appending f2py.lib.parser configuration to f2py.lib Ignoring attempt to set 'name' (from 'f2py.lib' to 'f2py.lib.parser') Warning: Assuming default configuration (lib\extgen/{setup_extgen,setup}.py was not found) Appending f2py.lib.extgen configuration to f2py.lib Ignoring attempt to set 'name' (from 'f2py.lib' to 'f2py.lib.extgen') Appending f2py.lib configuration to f2py Ignoring attempt to set 'name' (from 'f2py' to 'f2py.lib') F2PY Version 2_4423 Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\f2py\setup.py", line 130, in **config) TypeError: setup() got multiple values for keyword argument 'version' >>> I do not know as to how to fix the multiple values for version?? PLEASE HELP!!! David Blubaugh -----Original Message----- From: Blubaugh, David A. Sent: Monday, September 08, 2008 6:04 PM To: 'python-list at python.org' Subject: F2PY ?? Has anyone worked with the F2PY generator? To All, Has anyone worked with the F2PY generator? This is something that is supposedly built within numpy and scipy for the Python environment. I was wondering if anyone has encountered any issues with this environment?? This is important to find the answers to these questions. Thanks, David Blubaugh This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated. From robert.kern at gmail.com Mon Sep 8 19:29:14 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 8 Sep 2008 18:29:14 -0500 Subject: [Numpy-discussion] F2PY ?? Has anyone worked with the F2PY generator? In-Reply-To: <27CC3060AF71DA40A5DC85F7D5B70F3804F0F01C@AWMAIL04.belcan.com> References: <27CC3060AF71DA40A5DC85F7D5B70F3804F0F01C@AWMAIL04.belcan.com> Message-ID: <3d375d730809081629ydc5f0cer4d8ba64e17df1ca4@mail.gmail.com> On Mon, Sep 8, 2008 at 18:26, Blubaugh, David A. wrote: > Pauli, > > > Yes, I am utilizing the windows environment. I cannot install f2py. > > I obtain the following error when I try to execute the setup.py file > within the f2py folder located within the numpy master folder: Don't do that. Once numpy is installed, f2py is installed. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Chris.Barker at noaa.gov Mon Sep 8 19:32:37 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 08 Sep 2008 16:32:37 -0700 Subject: [Numpy-discussion] mixed-EOL problems w/ In-Reply-To: References: Message-ID: <48C5B615.3060005@noaa.gov> Jarrod Millman wrote: > However, this doesn't work for SciPy due to a mixed-EOL issue: > https://bugs.launchpad.net/launchpad-cscvs/+bug/256050 > > Colin Watson has recently volunteered to look into fixing this issue > and was asking if anyone had a preference about how this should be > solved. I vote for: "blatting all the newlines in the file into the native format" then you're really fixing the problem, not putting it off. Funny, I'm right in the middle of dealing with eol issues on a unrelated SVN project -- I'd really be much happier if SVN just did what CVS did -- this is far more painful. It's just not that hard to go back and fix it if you accidentally put a binary file in as text. Oh well. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From markperrymiller at gmail.com Mon Sep 8 19:36:19 2008 From: markperrymiller at gmail.com (Mark Miller) Date: Mon, 8 Sep 2008 16:36:19 -0700 Subject: [Numpy-discussion] F2PY ?? Has anyone worked with the F2PY generator? In-Reply-To: <27CC3060AF71DA40A5DC85F7D5B70F3804F0F01C@AWMAIL04.belcan.com> References: <27CC3060AF71DA40A5DC85F7D5B70F3804F0F01C@AWMAIL04.belcan.com> Message-ID: If numpy is installed, then f2py will be too. On the windows environment, there is a file called f2py.py that you can call from the command line. It should be in the 'scripts' directory of your Python installation. Try something like this: python c:\python25\scripts\f2py.py (of course change to reflect your own python installation directory) -Mark On Mon, Sep 8, 2008 at 4:26 PM, Blubaugh, David A. wrote: > Pauli, > > > Yes, I am utilizing the windows environment. I cannot install f2py. > > I obtain the following error when I try to execute the setup.py file > within the f2py folder located within the numpy master folder: > > > Warning: Assuming default configuration > (lib\parser/{setup_parser,setup}.py was not found) Appending > f2py.lib.parser configuration to f2py.lib Ignoring attempt to set 'name' > (from 'f2py.lib' to 'f2py.lib.parser') > Warning: Assuming default configuration > (lib\extgen/{setup_extgen,setup}.py was not found) Appending > f2py.lib.extgen configuration to f2py.lib Ignoring attempt to set 'name' > (from 'f2py.lib' to 'f2py.lib.extgen') Appending f2py.lib configuration > to f2py Ignoring attempt to set 'name' (from 'f2py' to 'f2py.lib') F2PY > Version 2_4423 > > Traceback (most recent call last): > File "C:\Python25\Lib\site-packages\numpy\f2py\setup.py", line 130, in > > **config) > TypeError: setup() got multiple values for keyword argument 'version' > >>> > > > I do not know as to how to fix the multiple values for version?? > PLEASE HELP!!! > > > David Blubaugh > > > > > > > > -----Original Message----- > From: Blubaugh, David A. > Sent: Monday, September 08, 2008 6:04 PM > To: 'python-list at python.org' > Subject: F2PY ?? Has anyone worked with the F2PY generator? > > To All, > > > Has anyone worked with the F2PY generator? This is something that is > supposedly built within numpy and scipy for the Python environment. I > was wondering if anyone has encountered any issues with this > environment?? This is important to find the answers to these questions. > > > > Thanks, > > > David Blubaugh > > > > This e-mail transmission contains information that is confidential and may > be > privileged. It is intended only for the addressee(s) named above. If you > receive > this e-mail in error, please do not read, copy or disseminate it in any > manner. > If you are not the intended recipient, any disclosure, copying, > distribution or > use of the contents of this information is prohibited. Please reply to the > message immediately by informing the sender that the message was > misdirected. > After replying, please erase it from your computer system. Your assistance > in > correcting this error is appreciated. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Mon Sep 8 19:45:57 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 8 Sep 2008 16:45:57 -0700 Subject: [Numpy-discussion] upcoming 1.2.0rc2 (1.2.0 final soon to follow)! Message-ID: Hello, There have been a few updates to the 1.2.x branch that resolve some issues with the last release candidate: - a loadtxt regression: http://projects.scipy.org/scipy/numpy/changeset/5790 - a knownfailureif decorator was added: http://projects.scipy.org/scipy/numpy/changeset/5792 - and used for a known Window's issue: http://projects.scipy.org/scipy/numpy/changeset/5795 - a numscons build directory issue: http://projects.scipy.org/scipy/numpy/changeset/5794 There are no other issues that I am aware of that would block a final release. If you know of a major regression or bug that would constitute a release blocker, please let me know ASAP. Otherwise I will tag a 1.2.0rc2 later tonight and announce it tomorrow. Assuming there are no new issues with the new release candidate, I plan to tag the final 1.2.0 release on Sunday. Also, please take a look at the release notes and let me know if anything needs to be changed or added: http://projects.scipy.org/scipy/numpy/milestone/1.2.0 Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From jinbow at MIT.EDU Mon Sep 8 19:46:38 2008 From: jinbow at MIT.EDU (jinbo wang) Date: Mon, 08 Sep 2008 19:46:38 -0400 Subject: [Numpy-discussion] write binary data Message-ID: <48C5B95E.9080605@mit.edu> Hi guys, I am new to python, and still wondering around to solve a simple problem. I have an array A, and want to save it to a binary data file with 'float64' and 'big-endian' specified. I searched online, but still don't know how to do it. I appreciate your help! regards, Jinbo Wang From robert.kern at gmail.com Mon Sep 8 19:52:02 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 8 Sep 2008 18:52:02 -0500 Subject: [Numpy-discussion] upcoming 1.2.0rc2 (1.2.0 final soon to follow)! In-Reply-To: References: Message-ID: <3d375d730809081652t3f46e978q82dae1237456223b@mail.gmail.com> On Mon, Sep 8, 2008 at 18:45, Jarrod Millman wrote: > Hello, > > There have been a few updates to the 1.2.x branch that resolve some > issues with the last release candidate: > - a loadtxt regression: http://projects.scipy.org/scipy/numpy/changeset/5790 > - a knownfailureif decorator was added: > http://projects.scipy.org/scipy/numpy/changeset/5792 > - and used for a known Window's issue: > http://projects.scipy.org/scipy/numpy/changeset/5795 > - a numscons build directory issue: > http://projects.scipy.org/scipy/numpy/changeset/5794 > > There are no other issues that I am aware of that would block a final > release. If you know of a major regression or bug that would > constitute a release blocker, please let me know ASAP. Otherwise I > will tag a 1.2.0rc2 later tonight and announce it tomorrow. Assuming > there are no new issues with the new release candidate, I plan to tag > the final 1.2.0 release on Sunday. > > Also, please take a look at the release notes and let me know if > anything needs to be changed or added: > http://projects.scipy.org/scipy/numpy/milestone/1.2.0 Some of the non-uniform RNG distributions got fixed/improved. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Mon Sep 8 19:56:43 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 8 Sep 2008 18:56:43 -0500 Subject: [Numpy-discussion] write binary data In-Reply-To: <48C5B95E.9080605@mit.edu> References: <48C5B95E.9080605@mit.edu> Message-ID: <3d375d730809081656x10fb7fd3p8a69efab982269b4@mail.gmail.com> On Mon, Sep 8, 2008 at 18:46, jinbo wang wrote: > Hi guys, > I am new to python, and still wondering around to solve a simple problem. > I have an array A, and want to save it to a binary data file with > 'float64' and 'big-endian' specified. I searched online, but still don't > know how to do it. I appreciate your help! x.astype('>float64').tofile('foo.dat') -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From millman at berkeley.edu Mon Sep 8 20:12:58 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 8 Sep 2008 17:12:58 -0700 Subject: [Numpy-discussion] planning for numpy 1.3.0 release Message-ID: Now that 1.2.0 is almost finalized, I wanted to ask everyone to start thinking about what they plan to work on for the next minor release: http://scipy.org/scipy/numpy/milestone/1.3.0 We have been gradually moving toward a more time-based release schedule over the last several months. In this vein, I would like to move feature and code discussion and planning to the beginning of the release cycle, rather than waiting until the end of the release cycle. So if there is something that you would like to work on during this release cycle, please bring it up now. Also it is important for us to keep the trunk as stable as possible; if you wish to work on a major new feature or an significant refactoring of the codebase, please work in a branch. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From FDU.xiaojf at gmail.com Mon Sep 8 23:03:46 2008 From: FDU.xiaojf at gmail.com (xiaojf) Date: Mon, 8 Sep 2008 20:03:46 -0700 (PDT) Subject: [Numpy-discussion] numpy or scipy ? Message-ID: <75951306-5109-4063-ac04-9623a2349246@i76g2000hsf.googlegroups.com> Hi all, I'm trying to replace matlab with python in my daily work and I have several problems now. 1) Which one should I use, numpy or scipy ? I found that numpy and scipy have many overlap functions and modules, such as numpy.linalg and scipy.linaly. Maybe I should use numpy in some occasions and use scipy in the other occasions? 2) Are there any functions in numpy/scipy that can do Partial Least Square(PLS) regression? How to calculate adjusted R squared in numpy/ scipy? I have googled to find the answer, but have on success. Thanks in advance. From charlesr.harris at gmail.com Tue Sep 9 00:05:50 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 8 Sep 2008 22:05:50 -0600 Subject: [Numpy-discussion] numpy or scipy ? In-Reply-To: <75951306-5109-4063-ac04-9623a2349246@i76g2000hsf.googlegroups.com> References: <75951306-5109-4063-ac04-9623a2349246@i76g2000hsf.googlegroups.com> Message-ID: On Mon, Sep 8, 2008 at 9:03 PM, xiaojf wrote: > Hi all, > > I'm trying to replace matlab with python in my daily work and I have > several problems now. > > 1) Which one should I use, numpy or scipy ? I found that numpy and > scipy have many overlap functions and modules, such as numpy.linalg > and scipy.linaly. > > Maybe I should use numpy in some occasions and use scipy in the other > occasions? > Numpy is the base, Scipy is a supplement. > > 2) Are there any functions in numpy/scipy that can do Partial Least > Square(PLS) regression? How to calculate adjusted R squared in numpy/ > scipy? I have googled to find the answer, but have on success. > I don't know. There was some discussion on the list but I don't think there are any packages. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Tue Sep 9 00:14:43 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 08 Sep 2008 21:14:43 -0700 Subject: [Numpy-discussion] planning for numpy 1.3.0 release In-Reply-To: References: Message-ID: <48C5F833.8060609@noaa.gov> Jarrod Millman wrote: > So if there is something that you would like to work on during this > release cycle, please bring it up now. Anyone want to help with improvements to fromfile() for text files? See some notes from me about it on this list in the last month or so. -CHB -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception From cournapeau at cslab.kecl.ntt.co.jp Tue Sep 9 01:39:30 2008 From: cournapeau at cslab.kecl.ntt.co.jp (David Cournapeau) Date: Tue, 09 Sep 2008 14:39:30 +0900 Subject: [Numpy-discussion] numpy or scipy ? In-Reply-To: References: <75951306-5109-4063-ac04-9623a2349246@i76g2000hsf.googlegroups.com> Message-ID: <1220938770.14526.6.camel@bbc8> On Mon, 2008-09-08 at 22:05 -0600, Charles R Harris wrote: > > > On Mon, Sep 8, 2008 at 9:03 PM, xiaojf wrote: > Hi all, > > I'm trying to replace matlab with python in my daily work and > I have > several problems now. > > 1) Which one should I use, numpy or scipy ? I found that numpy > and > scipy have many overlap functions and modules, such as > numpy.linalg > and scipy.linaly. > > Maybe I should use numpy in some occasions and use scipy in > the other > occasions? > > Numpy is the base, Scipy is a supplement. To complete what Charles said, to compare with matlab: - numpy is the core of the scientific stack for python. - scipy is built on the top of numpy: it leverages numpy and can be seen as a set of toolboxes for numpy. - matplotlib gives you something similar to matlab plotting capabilities. Numpy is more restricted than what bare matlab (wo any toolbox) can do: basic linear algebra (inverse, eigen values, cholesky and a few more), basic fft and random generators. With scipy, you should get almost everything available in bare matlab, plus some features found in toolboxes (mostly signal and statistics toolboxes), but not all. cheers, David From cournapeau at cslab.kecl.ntt.co.jp Tue Sep 9 01:54:44 2008 From: cournapeau at cslab.kecl.ntt.co.jp (David Cournapeau) Date: Tue, 09 Sep 2008 14:54:44 +0900 Subject: [Numpy-discussion] planning for numpy 1.3.0 release In-Reply-To: References: Message-ID: <1220939684.14526.12.camel@bbc8> On Mon, 2008-09-08 at 17:12 -0700, Jarrod Millman wrote: > Now that 1.2.0 is almost finalized, I wanted to ask everyone to start > thinking about what they plan to work on for the next minor release: > http://scipy.org/scipy/numpy/milestone/1.3.0 > > We have been gradually moving toward a more time-based release > schedule over the last several months. In this vein, I would like to > move feature and code discussion and planning to the beginning of the > release cycle, rather than waiting until the end of the release cycle. > So if there is something that you would like to work on during this > release cycle, please bring it up now. > > Also it is important for us to keep the trunk as stable as possible; > if you wish to work on a major new feature or an significant > refactoring of the codebase, please work in a branch. Hi Jarrod, hi everyone, Here are some things I would like to work/seen fixed on for 1.3: - fix VS 2003 compilation for mtrand.c (numpy is not buildable with VS 2003.2005.2008 ATM because of a compiler limitation) - clean umathmodule (work almost done in the clean_math branch, still need some testing on windows and solaris before merging). Something else I would like to see is integration of numscons into numpy trunk ? I am not talking about making numscons the default build, just about the availability of numscons with numpy sources (the main rationale for this is my own convenience: tracking different numpy version is just too much work for me, having it integrated in numpy svn would be helpful). cheers, David From nadavh at visionsense.com Tue Sep 9 02:04:13 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Tue, 9 Sep 2008 09:04:13 +0300 Subject: [Numpy-discussion] numpy or scipy ? References: <75951306-5109-4063-ac04-9623a2349246@i76g2000hsf.googlegroups.com> Message-ID: <710F2847B0018641891D9A216027636029C26A@ex3.envision.co.il> You can find PLS in pychem and MDP (Modular data processing) Nadav. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Charles R Harris ????: ? 09-??????-08 07:05 ??: Discussion of Numerical Python ????: Re: [Numpy-discussion] numpy or scipy ? On Mon, Sep 8, 2008 at 9:03 PM, xiaojf wrote: > Hi all, > > I'm trying to replace matlab with python in my daily work and I have > several problems now. > > 1) Which one should I use, numpy or scipy ? I found that numpy and > scipy have many overlap functions and modules, such as numpy.linalg > and scipy.linaly. > > Maybe I should use numpy in some occasions and use scipy in the other > occasions? > Numpy is the base, Scipy is a supplement. > > 2) Are there any functions in numpy/scipy that can do Partial Least > Square(PLS) regression? How to calculate adjusted R squared in numpy/ > scipy? I have googled to find the answer, but have on success. > I don't know. There was some discussion on the list but I don't think there are any packages. Chuck -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3336 bytes Desc: not available URL: From cournape at gmail.com Tue Sep 9 02:25:19 2008 From: cournape at gmail.com (David Cournapeau) Date: Tue, 9 Sep 2008 15:25:19 +0900 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <1220910955.8226.45.camel@sebook> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> <789d27b10809030326w1000132frb0d530f1dfcd4729@mail.gmail.com> <48C57351.4070006@noaa.gov> <1220910955.8226.45.camel@sebook> Message-ID: <5b8d13220809082325x74803d6t751e3518f03501fc@mail.gmail.com> On Tue, Sep 9, 2008 at 6:55 AM, Sebastian Stephan Berg wrote: > Yeah, I memory wise it doesn't matter to sum to a double, but just > trying around it seems that the mixing of float and double is very slow Yes, the memory argument explains why you would float32 data vs float64 data, not the accumulator (that certainly what Matthieu meant). Having an accumulator of float64 means you don't care so much about speed, but more about memory, and are willing to trade speed for less memory. > (at least on my comp) while if the starting array is already double > there is almost no difference for summing. Generally double precision > calculations should be slower though. Don't extensions like SSE2 operate > either on 2 doubles or 4 floats at once and thus should be about twice > as fast for floats? For add/multiply this behaviour is for me visible > anyways. We don't use SSE and co in numpy, and I doubt the compilers (even Intel one) are able to generate effective SSE for numpy ATM. Actually, double and float are about the same speed for x86 (using the x87 FPU and not the SSE units), because internally, the register is 80 bits wide when doing computation. The real difference is the memory pressure induced by double (8 bytes per items) compared to float when doing computation with double, and for certain operations, for a reason I don't understand (log, sin and co are as fast for float and double using the FPU, but sqrt and divide are twice faster for float, for example). I don't have any clear explanation on why mixing float with a double accumulator would be slow; maybe the default code to convert float to double - as generated by the compiler - is bad. Or maybe numpy does something funny. cheers, David From peridot.faceted at gmail.com Tue Sep 9 02:45:11 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Tue, 9 Sep 2008 02:45:11 -0400 Subject: [Numpy-discussion] numpy or scipy ? In-Reply-To: <75951306-5109-4063-ac04-9623a2349246@i76g2000hsf.googlegroups.com> References: <75951306-5109-4063-ac04-9623a2349246@i76g2000hsf.googlegroups.com> Message-ID: 2008/9/8 xiaojf : > Hi all, > > I'm trying to replace matlab with python in my daily work and I have > several problems now. > > 1) Which one should I use, numpy or scipy ? I found that numpy and > scipy have many overlap functions and modules, such as numpy.linalg > and scipy.linaly. > > Maybe I should use numpy in some occasions and use scipy in the other > occasions? In addition to what others have said - numpy is basic array operations, scipy is any sophisticated calculations - I should point out that there are many functions from numpy that are, for historical reasons, simply reexported as names in scipy. For example, In [7]: scipy.sin == numpy.sin Out[7]: True These functions will obviously behave identically. However, some functions provide extended versions in scipy: In [8]: numpy.sqrt(-1.) Out[8]: nan In [9]: scipy.sqrt(-1.) Out[9]: 1j The rule here is that scipy has the more "scientifically-correct" versions, while numpy has the more efficient and less surprising versions (notice that scipy.sqrt must return a complex array if any element of its input is real but negative, so its return type depends on the input value...) For the most part, though, scipy has tools and algorithms you might need for numerical calculations - a good rule of thumb is that it has tools to solve most of the problems described in the "Numerical Recipes" books - while numpy is a simple array library. Anne From hanni.ali at gmail.com Tue Sep 9 02:53:30 2008 From: hanni.ali at gmail.com (Hanni Ali) Date: Tue, 9 Sep 2008 07:53:30 +0100 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <48BFDC45.1090505@ar.media.kyoto-u.ac.jp> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> <789d27b10809040607v26b54c59hebbf0760de8dbed0@mail.gmail.com> <48BFDC45.1090505@ar.media.kyoto-u.ac.jp> Message-ID: <789d27b10809082353n60e8eaa2jdbd7667bad198f21@mail.gmail.com> Hi David, Forgot to answer last week, I was under a fair bit of pressure time wise, but thanks for your input. I sorted it all in the end and just in time, but the main issue here was the change from numarray to numpy. Previously where a typecode of 'f' was used in numarray, the calculation was performed in double precision whereas in numpy it was calculated in single precision. Hence when migrating the code, the differences popped up, which were fairly big when considering the size and number of mean calcs we perform. I now have a distinct dislike of float values (it'll probably wear off over time), how can the sum of 100,000 numbers be anything other than the sum of those numbers. I know the reasoning, as highlighted by the couple of other e-mails we have had, but I feel the default should probably lean towards accuracy than speed. 2.0+2.0=4.0 and 2.0+2.0.....=200,000.0 not 2array.sum() != 200,000... Just an opinion though Hanni 2008/9/4 David Cournapeau > Hanni Ali wrote: > > Hi, > > > > Is there a way I can set numpy to use dtype='float64' throughout all > > my code or force it to use the biggest datatype without adding the > > dtype='float64' to every call to mean, stdev etc.. > > Since it is the default type for the functions you mention, you can just > remove any call to dtype, but removing all the call to dtype='float64' > is not much less work than replacing dtype = 'float32' :) More > seriously, depending on your program, it may not be doable 100 % > automatically. > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Tue Sep 9 03:16:43 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 9 Sep 2008 09:16:43 +0200 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <789d27b10809082353n60e8eaa2jdbd7667bad198f21@mail.gmail.com> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> <789d27b10809040607v26b54c59hebbf0760de8dbed0@mail.gmail.com> <48BFDC45.1090505@ar.media.kyoto-u.ac.jp> <789d27b10809082353n60e8eaa2jdbd7667bad198f21@mail.gmail.com> Message-ID: > I now have a distinct dislike of float values (it'll probably wear off over > time), how can the sum of 100,000 numbers be anything other than the sum of > those numbers. I know the reasoning, as highlighted by the couple of other > e-mails we have had, but I feel the default should probably lean towards > accuracy than speed. 2.0+2.0=4.0 and 2.0+2.0.....=200,000.0 not 2array.sum() > != 200,000... In that case, we should not use doubles, but long double or even better, the real numbers themselves. Which would mean that computations would be very very very slow. Numpy leans somehow towards accuracy. If you want more accuracy (because even with double, you can hit the limit very fast), use another type. You said : how can the sum of 100,000 numbers be anything other than the sum of > those numbers This will always be a problem. With doubles, try to sum 1/n (1...100000), you'll be surprized. And then do sum 1/n (100000...1) with float values, and here the result should be better than when using doubles. Numerical issues in scientific computing are tricky. There is no single answer, it depends on your problem. Matthieu -- French PhD student Information System Engineer Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From cournapeau at cslab.kecl.ntt.co.jp Tue Sep 9 03:25:17 2008 From: cournapeau at cslab.kecl.ntt.co.jp (David Cournapeau) Date: Tue, 09 Sep 2008 16:25:17 +0900 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <789d27b10809082353n60e8eaa2jdbd7667bad198f21@mail.gmail.com> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> <789d27b10809040607v26b54c59hebbf0760de8dbed0@mail.gmail.com> <48BFDC45.1090505@ar.media.kyoto-u.ac.jp> <789d27b10809082353n60e8eaa2jdbd7667bad198f21@mail.gmail.com> Message-ID: <1220945117.14526.31.camel@bbc8> On Tue, 2008-09-09 at 07:53 +0100, Hanni Ali wrote: > Hi David, > > Forgot to answer last week, I was under a fair bit of pressure time > wise, but thanks for your input. I sorted it all in the end and just > in time, but the main issue here was the change from numarray to > numpy. Previously where a typecode of 'f' was used in numarray, the > calculation was performed in double precision whereas in numpy it was > calculated in single precision. Hence when migrating the code, the > differences popped up, which were fairly big when considering the size > and number of mean calcs we perform. Hi Hanni, glad it worked ok for you. > > I now have a distinct dislike of float values (it'll probably wear off > over time), how can the sum of 100,000 numbers be anything other than > the sum of those numbers. I know the reasoning, as highlighted by the > couple of other e-mails we have had, but I feel the default should > probably lean towards accuracy than speed. 2.0+2.0=4.0 and 2.0 > +2.0.....=200,000.0 not 2array.sum() != 200,000... I think it is a fallacy to say you prefer accuracy over speed: the fallacy is in thinking it is binary choice. You care about speed, because otherwise, you would not use a computer at all, you would do everything by hand [1]. Floating point is by itself an approximation: it can not even represent rational number accurately, let alone algebraic numbers or transcendent ones ! There are packages to do exact computation (look at sage for example for something based on python), but numpy/scipy are first numerical computation, meaning approximation along the way. It is true that it can give some unexpected results, and you should be aware of floating point limitations [2]. That being said, for a lot of computations, when you have unexpected difference between float and double, you have a problem in your implementation. For example, IIRC, you computed average of a big number numbers, at once: you can get better results if you first normalize your numbers. Another example which bites me all the time in statistic is when computing exponential of small numbers: log(exp(-1000)) will be -Inf done naively, but you and me know the solution is of course -1000; again, you should think more about your computation. IOW, floating point are a useful approximation/abstraction (I don't know if you are familiar with fixed point computation, as done in some DSP, but it is not pretty), but it breaks in some cases. cheers, David [1] I know some people do this for some kind of computation; in a different context from numerical computation, I found the following interview from Alain Connes (one of the most famous French Mathematician currently alive), to be extemely enlightening: http://www.ipm.ac.ir/IPM/news/connes-interview.pdf (see page 2-3 for the discussion about computer and computation) [2] "What every computer scientist should know about floating-point arithmetic", in ACM Computer Survey, 1991, By David Goldberg. From stefan at sun.ac.za Tue Sep 9 03:49:32 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 9 Sep 2008 09:49:32 +0200 Subject: [Numpy-discussion] planning for numpy 1.3.0 release In-Reply-To: <48C5F833.8060609@noaa.gov> References: <48C5F833.8060609@noaa.gov> Message-ID: <9457e7c80809090049h29b5f019g41457c247901a16@mail.gmail.com> 2008/9/9 Christopher Barker : > Jarrod Millman wrote: >> So if there is something that you would like to work on during this >> release cycle, please bring it up now. > > Anyone want to help with improvements to fromfile() for text files? This is low hanging fruit for anyone with some experience in C. We can definitely get it done for 1.3. Chris, would you file a ticket and add the detail from your mailing list posts, if that hasn't already been done? Thanks St?fan From hanni.ali at gmail.com Tue Sep 9 03:57:04 2008 From: hanni.ali at gmail.com (Hanni Ali) Date: Tue, 9 Sep 2008 08:57:04 +0100 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> <789d27b10809040607v26b54c59hebbf0760de8dbed0@mail.gmail.com> <48BFDC45.1090505@ar.media.kyoto-u.ac.jp> <789d27b10809082353n60e8eaa2jdbd7667bad198f21@mail.gmail.com> Message-ID: <789d27b10809090057i4dd717d6i89b4b81486a8edf5@mail.gmail.com> Hi Matthieu, Interesting example thanks. I can't however seem to get anything other than zero for the 100,000 to 1 sum. Cheers, Hanni 2008/9/9 Matthieu Brucher > > I now have a distinct dislike of float values (it'll probably wear off > over > > time), how can the sum of 100,000 numbers be anything other than the sum > of > > those numbers. I know the reasoning, as highlighted by the couple of > other > > e-mails we have had, but I feel the default should probably lean towards > > accuracy than speed. 2.0+2.0=4.0 and 2.0+2.0.....=200,000.0 not > 2array.sum() > > != 200,000... > > In that case, we should not use doubles, but long double or even > better, the real numbers themselves. Which would mean that > computations would be very very very slow. > Numpy leans somehow towards accuracy. If you want more accuracy > (because even with double, you can hit the limit very fast), use > another type. > > You said : > how can the sum of 100,000 numbers be anything other than the sum of > > those numbers > > This will always be a problem. With doubles, try to sum 1/n > (1...100000), you'll be surprized. And then do sum 1/n (100000...1) > with float values, and here the result should be better than when > using doubles. Numerical issues in scientific computing are tricky. > There is no single answer, it depends on your problem. > > Matthieu > -- > French PhD student > Information System Engineer > Website: http://matthieu-brucher.developpez.com/ > Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn: http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hanni.ali at gmail.com Tue Sep 9 03:59:21 2008 From: hanni.ali at gmail.com (Hanni Ali) Date: Tue, 9 Sep 2008 08:59:21 +0100 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <1220945117.14526.31.camel@bbc8> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> <789d27b10809040607v26b54c59hebbf0760de8dbed0@mail.gmail.com> <48BFDC45.1090505@ar.media.kyoto-u.ac.jp> <789d27b10809082353n60e8eaa2jdbd7667bad198f21@mail.gmail.com> <1220945117.14526.31.camel@bbc8> Message-ID: <789d27b10809090059r789e888buf56205ce3fddd044@mail.gmail.com> > > > I think it is a fallacy to say you prefer accuracy over speed: the > fallacy is in thinking it is binary choice. You care about speed, > because otherwise, you would not use a computer at all, you would do > everything by hand [1]. Floating point is by itself an approximation: it > can not even represent rational number accurately, let alone algebraic > numbers or transcendent ones ! There are packages to do exact > computation (look at sage for example for something based on python), > but numpy/scipy are first numerical computation, meaning approximation > along the way. > True perhaps I should have said consistency, rather than accuracy. Cheers -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Tue Sep 9 05:06:14 2008 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 9 Sep 2008 09:06:14 +0000 (UTC) Subject: [Numpy-discussion] planning for numpy 1.3.0 release References: <48C5F833.8060609@noaa.gov> <9457e7c80809090049h29b5f019g41457c247901a16@mail.gmail.com> Message-ID: Tue, 09 Sep 2008 09:49:32 +0200, St?fan van der Walt wrote: > 2008/9/9 Christopher Barker : >> Jarrod Millman wrote: >>> So if there is something that you would like to work on during this >>> release cycle, please bring it up now. >> >> Anyone want to help with improvements to fromfile() for text files? > > This is low hanging fruit for anyone with some experience in C. We can > definitely get it done for 1.3. Chris, would you file a ticket and add > the detail from your mailing list posts, if that hasn't already been > done? I'd like to address the C locale dependency of fromfile and fromstring. (setlocale() affects fromfile, but does not affect fromstring; locale can make eg. "," become the decimal separator instead of "."). I'd assume we want to follow Fortran and Python here, and *not* format or parse floating point numbers according the C locale. Use of fscanf might need to be reconsidered, or the calls need to be wrapped in setlocale ("C"); setlocale(old_locale); pairs... Locale-related tickets: #884, #902, #695 Running numpy tests under different locales and calling setlocale(LC_ALL, "") may reveal some additional points to address. -- Pauli Virtanen From charlesr.harris at gmail.com Tue Sep 9 06:45:15 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 9 Sep 2008 04:45:15 -0600 Subject: [Numpy-discussion] planning for numpy 1.3.0 release In-Reply-To: <1220939684.14526.12.camel@bbc8> References: <1220939684.14526.12.camel@bbc8> Message-ID: On Mon, Sep 8, 2008 at 11:54 PM, David Cournapeau < cournapeau at cslab.kecl.ntt.co.jp> wrote: > On Mon, 2008-09-08 at 17:12 -0700, Jarrod Millman wrote: > > Now that 1.2.0 is almost finalized, I wanted to ask everyone to start > > thinking about what they plan to work on for the next minor release: > > http://scipy.org/scipy/numpy/milestone/1.3.0 > > > > We have been gradually moving toward a more time-based release > > schedule over the last several months. In this vein, I would like to > > move feature and code discussion and planning to the beginning of the > > release cycle, rather than waiting until the end of the release cycle. > > So if there is something that you would like to work on during this > > release cycle, please bring it up now. > > > > Also it is important for us to keep the trunk as stable as possible; > > if you wish to work on a major new feature or an significant > > refactoring of the codebase, please work in a branch. > > Hi Jarrod, hi everyone, > > Here are some things I would like to work/seen fixed on for 1.3: > - fix VS 2003 compilation for mtrand.c (numpy is not > buildable with VS > 2003.2005.2008 ATM because of a compiler limitation) > - clean umathmodule (work almost done in the clean_math > branch, still > need some testing on windows and solaris before merging). > I also have a big rearrangement/cleanup of the inner loops in the umathmodule that has been sitting around for half a year. I think it is almost orthogonal to your work because it deals with a different part of the code. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Tue Sep 9 08:16:59 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 9 Sep 2008 14:16:59 +0200 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <789d27b10809090057i4dd717d6i89b4b81486a8edf5@mail.gmail.com> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> <789d27b10809040607v26b54c59hebbf0760de8dbed0@mail.gmail.com> <48BFDC45.1090505@ar.media.kyoto-u.ac.jp> <789d27b10809082353n60e8eaa2jdbd7667bad198f21@mail.gmail.com> <789d27b10809090057i4dd717d6i89b4b81486a8edf5@mail.gmail.com> Message-ID: Use 1./n instead of 1/n. If n is an integer, 1/n equals 0. Matthieu 2008/9/9 Hanni Ali : > Hi Matthieu, > > Interesting example thanks. I can't however seem to get anything other than > zero for the 100,000 to 1 sum. > > Cheers, > > Hanni > > 2008/9/9 Matthieu Brucher >> >> > I now have a distinct dislike of float values (it'll probably wear off >> > over >> > time), how can the sum of 100,000 numbers be anything other than the sum >> > of >> > those numbers. I know the reasoning, as highlighted by the couple of >> > other >> > e-mails we have had, but I feel the default should probably lean towards >> > accuracy than speed. 2.0+2.0=4.0 and 2.0+2.0.....=200,000.0 not >> > 2array.sum() >> > != 200,000... >> >> In that case, we should not use doubles, but long double or even >> better, the real numbers themselves. Which would mean that >> computations would be very very very slow. >> Numpy leans somehow towards accuracy. If you want more accuracy >> (because even with double, you can hit the limit very fast), use >> another type. >> >> You said : >> how can the sum of 100,000 numbers be anything other than the sum of >> > those numbers >> >> This will always be a problem. With doubles, try to sum 1/n >> (1...100000), you'll be surprized. And then do sum 1/n (100000...1) >> with float values, and here the result should be better than when >> using doubles. Numerical issues in scientific computing are tricky. >> There is no single answer, it depends on your problem. >> >> Matthieu >> -- >> French PhD student >> Information System Engineer >> Website: http://matthieu-brucher.developpez.com/ >> Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 >> LinkedIn: http://www.linkedin.com/in/matthieubrucher >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- French PhD student Information System Engineer Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From hanni.ali at gmail.com Tue Sep 9 08:32:54 2008 From: hanni.ali at gmail.com (Hanni Ali) Date: Tue, 9 Sep 2008 13:32:54 +0100 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> <789d27b10809040607v26b54c59hebbf0760de8dbed0@mail.gmail.com> <48BFDC45.1090505@ar.media.kyoto-u.ac.jp> <789d27b10809082353n60e8eaa2jdbd7667bad198f21@mail.gmail.com> <789d27b10809090057i4dd717d6i89b4b81486a8edf5@mail.gmail.com> Message-ID: <789d27b10809090532y67b0c119xde984d815b0b03a1@mail.gmail.com> I already was... 2008/9/9 Matthieu Brucher > Use 1./n instead of 1/n. If n is an integer, 1/n equals 0. > > Matthieu > > 2008/9/9 Hanni Ali : > > Hi Matthieu, > > > > Interesting example thanks. I can't however seem to get anything other > than > > zero for the 100,000 to 1 sum. > > > > Cheers, > > > > Hanni > > > > 2008/9/9 Matthieu Brucher > >> > >> > I now have a distinct dislike of float values (it'll probably wear off > >> > over > >> > time), how can the sum of 100,000 numbers be anything other than the > sum > >> > of > >> > those numbers. I know the reasoning, as highlighted by the couple of > >> > other > >> > e-mails we have had, but I feel the default should probably lean > towards > >> > accuracy than speed. 2.0+2.0=4.0 and 2.0+2.0.....=200,000.0 not > >> > 2array.sum() > >> > != 200,000... > >> > >> In that case, we should not use doubles, but long double or even > >> better, the real numbers themselves. Which would mean that > >> computations would be very very very slow. > >> Numpy leans somehow towards accuracy. If you want more accuracy > >> (because even with double, you can hit the limit very fast), use > >> another type. > >> > >> You said : > >> how can the sum of 100,000 numbers be anything other than the sum of > >> > those numbers > >> > >> This will always be a problem. With doubles, try to sum 1/n > >> (1...100000), you'll be surprized. And then do sum 1/n (100000...1) > >> with float values, and here the result should be better than when > >> using doubles. Numerical issues in scientific computing are tricky. > >> There is no single answer, it depends on your problem. > >> > >> Matthieu > >> -- > >> French PhD student > >> Information System Engineer > >> Website: http://matthieu-brucher.developpez.com/ > >> Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > >> LinkedIn: http://www.linkedin.com/in/matthieubrucher > >> _______________________________________________ > >> Numpy-discussion mailing list > >> Numpy-discussion at scipy.org > >> http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > -- > French PhD student > Information System Engineer > Website: http://matthieu-brucher.developpez.com/ > Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn: http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From emanuele at relativita.com Tue Sep 9 08:42:31 2008 From: emanuele at relativita.com (Emanuele Olivetti) Date: Tue, 09 Sep 2008 14:42:31 +0200 Subject: [Numpy-discussion] [Fwd: Fwd: [ML-news] Call for Submissions: Workshop on Machine Learning Open Source Software (MLOSS), NIPS*08] Message-ID: <48C66F37.8050409@relativita.com> Maybe of interest. E. -------- Original Message -------- ---------- Forwarded message ---------- From: mikiobraun Date: 2008/9/8 Subject: [ML-news] Call for Submissions: Workshop on Machine Learning Open Source Software (MLOSS), NIPS*08 To: Machine Learning News ********************************************************************** Call for Submissions Workshop on Machine Learning Open Source Software 2008 http://mloss.org/workshop/nips08 held at NIPS*08, Whistler, Canada, December 12th, 2008 ********************************************************************** The NIPS workshop on Workshop on Machine Learning Open Source Software (MLOSS) will held in Whistler (B.C.) on the 12th of December, 2008. Important Dates =============== * Submission Date: October 1st, 2008 * Notification of Acceptance: October 14th, 2008 * Workshop date: December 12 or 13th, 2008 Call for Contributions ====================== The organizing committee is currently seeking abstracts for talks at MLOSS 2008. MLOSS is a great opportunity for you to tell the community about your use, development, or philosophy of open source software in machine learning. This includes (but is not limited to) numeric packages (as e.g. R,octave,numpy), machine learning toolboxes and implementations of ML-algorithms. The committee will select several submitted abstracts for 20-minute talks. The submission process is very simple: * Tag your mloss.org project with the tag nips2008 * Ensure that you have a good description (limited to 500 words) * Any bells and whistles can be put on your own project page, and of course provide this link on mloss.org On 1 October 2008, we will collect all projects tagged with nips2008 for review. Note: Projects must adhere to a recognized Open Source License (cf. http://www.opensource.org/licenses/ ) and the source code must have been released at the time of submission. Submissions will be reviewed based on the status of the project at the time of the submission deadline. Description =========== We believe that the wide-spread adoption of open source software policies will have a tremendous impact on the field of machine learning. The goal of this workshop is to further support the current developments in this area and give new impulses to it. Following the success of the inaugural NIPS-MLOSS workshop held at NIPS 2006, the Journal of Machine Learning Research (JMLR) has started a new track for machine learning open source software initiated by the workshop's organizers. Many prominent machine learning researchers have co-authored a position paper advocating the need for open source software in machine learning. Furthermore, the workshop's organizers have set up a community website mloss.org where people can register their software projects, rate existing projects and initiate discussions about projects and related topics. This website currently lists 123 such projects including many prominent projects in the area of machine learning. The main goal of this workshop is to bring the main practitioners in the area of machine learning open source software together in order to initiate processes which will help to further improve the development of this area. In particular, we have to move beyond a mere collection of more or less unrelated software projects and provide a common foundation to stimulate cooperation and interoperability between different projects. An important step in this direction will be a common data exchange format such that different methods can exchange their results more easily. This year's workshop sessions will consist of three parts. * We have two invited speakers: John Eaton, the lead developer of Octave and John Hunter, the lead developer of matplotlib. * Researchers are invited to submit their open source project to present it at the workshop. * In discussion sessions, important questions regarding the future development of this area will be discussed. In particular, we will discuss what makes a good machine learning software project and how to improve interoperability between programs. In addition, the question of how to deal with data sets and reproducibility will also be addressed. Taking advantage of the large number of key research groups which attend NIPS, decisions and agreements taken at the workshop will have the potential to significantly impact the future of machine learning software. Invited Speakers ================ * John D. Hunter - Main author of matplotlib. * John W. Eaton - Main author of Octave. Tentative Program ================= The 1 day workshop will be a mixture of talks (including a mandatory demo of the software) and panel/open/hands-on discussions. Morning session: 7:30am - ?10:30am * Introduction and overview * Octave (John W. Eaton) * Contributed Talks * Discussion: What is a good mloss project? o Review criteria for JMLR mloss o Interoperable software o Test suites Afternoon session: 3:30pm - ?6:30pm * Matplotlib (John D. Hunter) * Contributed Talks * Discussion: Reproducible research o Data exchange standards o Shall datasets be open too? How to provide access to data sets. o Reproducible research, the next level after UCI datasets. Program Committee ================= * Jason Weston (NEC Princeton, USA) * Gunnar R?tsch (FML Tuebingen, Germany) * Lieven Vandenberghe (University of California LA, USA) * Joachim Dahl (Aalborg University, Denmark) * Torsten Hothorn (Ludwig Maximilians University, Munich, Germany) * Asa Ben-Hur (Colorado State University, USA) * William Stafford Noble (Department of Genome Sciences Seattle, USA) * Klaus-Robert Mueller (Fraunhofer Institute First, Germany) * Geoff Holmes (University of Waikato, New Zealand) * Alain Rakotomamonjy (University of Rouen, France) Organizers ========== * Soeren Sonnenburg Fraunhofer FIRST Kekul?str. 7, 12489 Berlin, Germany * Mikio Braun Technische Universit?t Berlin, Franklinstr. 28/29, FR 6-9, 10587 Berlin, Germany * Cheng Soon Ong ETH Z?rich, Universit?tstr. 6, 8092 Z?rich, Switzerland Funding ======= The workshop is supported by PASCAL (Pattern Analysis, Statistical Modelling and Computational Learning) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Machine Learning News" group. To post to this group, send email to ML-news at googlegroups.com To unsubscribe from this group, send email to ML-news+unsubscribe at googlegroups.com For more options, visit this group at http://groups.google.com/group/ML-news?hl=en -~----------~----~----~----~------~----~------~--~--- From cournape at gmail.com Tue Sep 9 10:01:42 2008 From: cournape at gmail.com (David Cournapeau) Date: Tue, 9 Sep 2008 23:01:42 +0900 Subject: [Numpy-discussion] planning for numpy 1.3.0 release In-Reply-To: References: <1220939684.14526.12.camel@bbc8> Message-ID: <5b8d13220809090701h426258d9xed04ca571f2f44e6@mail.gmail.com> On Tue, Sep 9, 2008 at 7:45 PM, Charles R Harris wrote: > > I also have a big rearrangement/cleanup of the inner loops in the > umathmodule that has been sitting around for half a year. I think it is > almost orthogonal to your work because it deals with a different part of the > code. Maybe you could commit your changes to my branch, then ? For now, most of my changes are under git, but I can commit them to svn. cheers, David From cournape at gmail.com Tue Sep 9 10:03:31 2008 From: cournape at gmail.com (David Cournapeau) Date: Tue, 9 Sep 2008 23:03:31 +0900 Subject: [Numpy-discussion] planning for numpy 1.3.0 release In-Reply-To: References: Message-ID: <5b8d13220809090703m7d1858bfy1a4b4c1a5d588a33@mail.gmail.com> On Tue, Sep 9, 2008 at 9:12 AM, Jarrod Millman wrote: > Now that 1.2.0 is almost finalized, I wanted to ask everyone to start > thinking about what they plan to work on for the next minor release: > http://scipy.org/scipy/numpy/milestone/1.3.0 > > We have been gradually moving toward a more time-based release > schedule over the last several months. In this vein, I would like to > move feature and code discussion and planning to the beginning of the > release cycle, rather than waiting until the end of the release cycle. > Something else which could be nice - but arguably difficult depending on what is needed - is making sure numpy does work with python 2.6. I am saying this because I just saw Barry Warsaw mentioning splitting python 2.6/3.0 releases date such as python 2.6 will be integrated in next version of Mac OS X. cheers, David From focke at slac.stanford.edu Tue Sep 9 10:11:31 2008 From: focke at slac.stanford.edu (Warren Focke) Date: Tue, 9 Sep 2008 07:11:31 -0700 (PDT) Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <5b8d13220809082325x74803d6t751e3518f03501fc@mail.gmail.com> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> <789d27b10809030326w1000132frb0d530f1dfcd4729@mail.gmail.com> <48C57351.4070006@noaa.gov> <1220910955.8226.45.camel@sebook> <5b8d13220809082325x74803d6t751e3518f03501fc@mail.gmail.com> Message-ID: On Tue, 9 Sep 2008, David Cournapeau wrote: > We don't use SSE and co in numpy, and I doubt the compilers (even > Intel one) are able to generate effective SSE for numpy ATM. Actually, > double and float are about the same speed for x86 (using the x87 FPU > and not the SSE units), because internally, the register is 80 bits > wide when doing computation. The real difference is the memory > pressure induced by double (8 bytes per items) compared to float when > doing computation with double, and for certain operations, for a > reason I don't understand (log, sin and co are as fast for float and > double using the FPU, but sqrt and divide are twice faster for float, > for example). Some of the transcendental functions are implemented with polynomials, so should take constant time. Others may use iterative algorithms, which would require fewer iterations to reach single accuracy than double. w From Chris.Barker at noaa.gov Tue Sep 9 13:19:16 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 09 Sep 2008 10:19:16 -0700 Subject: [Numpy-discussion] Floating Point Difference between numpy and numarray In-Reply-To: <789d27b10809090532y67b0c119xde984d815b0b03a1@mail.gmail.com> References: <789d27b10809030204h3997480r8bf1dfaac4c1a0af@mail.gmail.com> <789d27b10809030231gdda0684nc893c2236fcbefab@mail.gmail.com> <1220435124.20211.10.camel@sebook> <789d27b10809040607v26b54c59hebbf0760de8dbed0@mail.gmail.com> <48BFDC45.1090505@ar.media.kyoto-u.ac.jp> <789d27b10809082353n60e8eaa2jdbd7667bad198f21@mail.gmail.com> <789d27b10809090057i4dd717d6i89b4b81486a8edf5@mail.gmail.com> <789d27b10809090532y67b0c119xde984d815b0b03a1@mail.gmail.com> Message-ID: <48C6B014.2070402@noaa.gov> Hanni Ali wrote: > > Interesting example thanks. I can't however seem to get anything > other than > > zero for the 100,000 to 1 sum. what is the "100,000 to 1 sum." this is my interpretation: a = np.linspace(100000, 1, 100000) >>> a array([ 1.00000000e+05, 9.99990000e+04, 9.99980000e+04, ..., 3.00000000e+00, 2.00000000e+00, 1.00000000e+00]) >>> a.sum() 5000050000.0 which is exact, but it won't be if you use single precision: >>> a = a.astype(np.float32) >>> a.sum() 5.0000865e+09 You'd have to use tricks like compensated summation and the like to improve the answers, if you want. Actually, it would be kind of neat to have that in numpy or scipy... What are you trying to do? Show us the code, and I'm sure someone can explain why you are getting what you are getting. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From doutriaux1 at llnl.gov Tue Sep 9 14:03:04 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Tue, 09 Sep 2008 11:03:04 -0700 Subject: [Numpy-discussion] small bug in numpy.ma.minimum.outer Message-ID: <48C6BA58.4000506@llnl.gov> a=numpy.ones((2,2)) >>> numpy.minimum.outer(a,a) array([[[[ 1., 1.], [ 1., 1.]], [[ 1., 1.], [ 1., 1.]]], [[[ 1., 1.], [ 1., 1.]], [[ 1., 1.], [ 1., 1.]]]]) >>> numpy.ma.minimum.outer(a,a) Traceback (most recent call last): File "", line 1, in File "/lgm/cdat/latest/lib/python2.5/site-packages/numpy/ma/core.py", line 3317, in outer result._mask = m AttributeError: 'numpy.ndarray' object has no attribute '_mask' From doutriaux1 at llnl.gov Tue Sep 9 14:14:48 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Tue, 09 Sep 2008 11:14:48 -0700 Subject: [Numpy-discussion] compressed and sqrt in numpy.ma Message-ID: <48C6BD18.7040902@llnl.gov> The following is causing our code to crash, shouldn't .data be just ones ? >>> a = numpy.ma.ones((2,2)) >>> b = numpy.ma.sqrt(a) >>> b.compressed().data From pgmdevlist at gmail.com Tue Sep 9 14:18:17 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 9 Sep 2008 14:18:17 -0400 Subject: [Numpy-discussion] compressed and sqrt in numpy.ma In-Reply-To: <48C6BD18.7040902@llnl.gov> References: <48C6BD18.7040902@llnl.gov> Message-ID: <200809091418.19339.pgmdevlist@gmail.com> On Tuesday 09 September 2008 14:14:48 Charles Doutriaux wrote: > The following is causing our code to crash, shouldn't .data be just ones ? > > >>> a = numpy.ma.ones((2,2)) > >>> b = numpy.ma.sqrt(a) > >>> b.compressed().data Nope. When you use compressed(), you get a ndarray (or a subclass, depending on the _baseclass attribute), but no longer a MaskedArray. Therefore, the '.data' attribute is the corresponding ndarray.data. In your example, just use b.compressed() From pgmdevlist at gmail.com Tue Sep 9 14:20:08 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 9 Sep 2008 14:20:08 -0400 Subject: [Numpy-discussion] small bug in numpy.ma.minimum.outer In-Reply-To: <48C6BA58.4000506@llnl.gov> References: <48C6BA58.4000506@llnl.gov> Message-ID: <200809091420.09031.pgmdevlist@gmail.com> On Tuesday 09 September 2008 14:03:04 Charles Doutriaux wrote: > a=numpy.ones((2,2)) > > >>> numpy.minimum.outer(a,a) > >>> numpy.ma.minimum.outer(a,a) Fixed in SVN r5800. From doutriaux1 at llnl.gov Tue Sep 9 14:33:22 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Tue, 09 Sep 2008 11:33:22 -0700 Subject: [Numpy-discussion] small bug in numpy.ma.minimum.outer In-Reply-To: <200809091420.09031.pgmdevlist@gmail.com> References: <48C6BA58.4000506@llnl.gov> <200809091420.09031.pgmdevlist@gmail.com> Message-ID: <48C6C172.2010504@llnl.gov> Thx Pierre Pierre GM wrote: > On Tuesday 09 September 2008 14:03:04 Charles Doutriaux wrote: > >> a=numpy.ones((2,2)) >> >> >>> numpy.minimum.outer(a,a) >> >>> numpy.ma.minimum.outer(a,a) >> > > Fixed in SVN r5800. > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http:// projects.scipy.org/mailman/listinfo/numpy-discussion > > > From millman at berkeley.edu Tue Sep 9 16:40:26 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 9 Sep 2008 13:40:26 -0700 Subject: [Numpy-discussion] upcoming 1.2.0rc2 (1.2.0 final soon to follow)! In-Reply-To: References: Message-ID: I will be branching for 1.2.0rc2 shortly. Please don't commit anything to the branch without running it by the list: http://projects.scipy.org/scipy/numpy/browser/branches/1.2.x Once the binaries have been made, I will send out an announcement. Thanks, Jarrod On Mon, Sep 8, 2008 at 4:45 PM, Jarrod Millman wrote: > Hello, > > There have been a few updates to the 1.2.x branch that resolve some > issues with the last release candidate: > - a loadtxt regression: http://projects.scipy.org/scipy/numpy/changeset/5790 > - a knownfailureif decorator was added: > http://projects.scipy.org/scipy/numpy/changeset/5792 > - and used for a known Window's issue: > http://projects.scipy.org/scipy/numpy/changeset/5795 > - a numscons build directory issue: > http://projects.scipy.org/scipy/numpy/changeset/5794 > > There are no other issues that I am aware of that would block a final > release. If you know of a major regression or bug that would > constitute a release blocker, please let me know ASAP. Otherwise I > will tag a 1.2.0rc2 later tonight and announce it tomorrow. Assuming > there are no new issues with the new release candidate, I plan to tag > the final 1.2.0 release on Sunday. > > Also, please take a look at the release notes and let me know if > anything needs to be changed or added: > http://projects.scipy.org/scipy/numpy/milestone/1.2.0 > > Thanks, > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From dblubaugh at belcan.com Tue Sep 9 18:53:02 2008 From: dblubaugh at belcan.com (Blubaugh, David A.) Date: Tue, 9 Sep 2008 18:53:02 -0400 Subject: [Numpy-discussion] Still having issues Message-ID: <27CC3060AF71DA40A5DC85F7D5B70F3804F8A915@AWMAIL04.belcan.com> Mark, I have taken the advice that you had given to me the other day. However, I am still having issues with actually executing the f2py.py script. > If numpy is installed, then f2py will be too. > On the windows environment, > there is a file called f2py.py that you can >>> > call from the command line. It > should be in the 'scripts' directory of your > > Python installation. > > Try something like this: > > python c:\python25\scripts\f2py.py > (of course change to reflect your own python > > installation directory) It appears that once I run the f2py.py script from the IDLE environment, I will then obtain the following error: You will have to read at the end of the listed output. It looks as though there was a premature system exit once f2py.py was executed. >>> Usage: 1) To construct extension module sources: f2py [] [[[only:]||[skip:]] \ ] \ [: ...] 2) To compile fortran files and build extension modules: f2py -c [, , ] 3) To generate signature files: f2py -h ...< same options as in (1) > Description: This program generates a Python C/API file (module.c) that contains wrappers for given fortran functions so that they can be called from Python. With the -c option the corresponding extension modules are built. Options: --g3-numpy Use numpy.f2py.lib tool, the 3rd generation of F2PY, with NumPy support. --2d-numpy Use numpy.f2py tool with NumPy support. [DEFAULT] --2d-numeric Use f2py2e tool with Numeric support. --2d-numarray Use f2py2e tool with Numarray support. -h Write signatures of the fortran routines to file and exit. You can then edit and use it instead of . If ==stdout then the signatures are printed to stdout. Names of fortran routines for which Python C/API functions will be generated. Default is all that are found in . Paths to fortran/signature files that will be scanned for in order to determine their signatures. skip: Ignore fortran functions that follow until `:'. only: Use only fortran functions that follow until `:'. : Get back to mode. -m Name of the module; f2py generates a Python/C API file module.c or extension module . Default is 'untitled'. --[no-]lower Do [not] lower the cases in . By default, --lower is assumed with -h key, and --no-lower without -h key. --build-dir All f2py generated files are created in . Default is tempfile.mktemp(). --overwrite-signature Overwrite existing signature file. --[no-]latex-doc Create (or not) module.tex. Default is --no-latex-doc. --short-latex Create 'incomplete' LaTeX document (without commands \documentclass, \tableofcontents, and \begin{document}, \end{document}). --[no-]rest-doc Create (or not) module.rst. Default is --no-rest-doc. --debug-capi Create C/API code that reports the state of the wrappers during runtime. Useful for debugging. --[no-]wrap-functions Create Fortran subroutine wrappers to Fortran 77 functions. --wrap-functions is default because it ensures maximum portability/compiler independence. --include_paths ::... Search include files from the given directories. --help-link [..] List system resources found by system_info.py. See also --link- switch below. [..] is optional list of resources names. E.g. try 'f2py --help-link lapack_opt'. --quiet Run quietly. --verbose Run with extra verbosity. -v Print f2py version ID and exit. numpy.distutils options (only effective with -c): --fcompiler= Specify Fortran compiler type by vendor --compiler= Specify C compiler type (as defined by distutils) --help-fcompiler List available Fortran compilers and exit --f77exec= Specify the path to F77 compiler --f90exec= Specify the path to F90 compiler --f77flags= Specify F77 compiler flags --f90flags= Specify F90 compiler flags --opt= Specify optimization flags --arch= Specify architecture specific optimization flags --noopt Compile without optimization --noarch Compile without arch-dependent optimization --debug Compile with debugging information Extra options (only effective with -c): --link- Link extension module with as defined by numpy.distutils/system_info.py. E.g. to link with optimized LAPACK libraries (vecLib on MacOSX, ATLAS elsewhere), use --link-lapack_opt. See also --help-link switch. -L/path/to/lib/ -l -D -U -I/path/to/include/ .o .so .a Using the following macros may be required with non-gcc Fortran compilers: -DPREPEND_FORTRAN -DNO_APPEND_FORTRAN -DUPPERCASE_FORTRAN -DUNDERSCORE_G77 When using -DF2PY_REPORT_ATEXIT, a performance report of F2PY interface is printed out at exit (platforms: Linux). When using -DF2PY_REPORT_ON_ARRAY_COPY=, a message is sent to stderr whenever F2PY interface makes a copy of an array. Integer sets the threshold for array sizes when a message should be shown. Version: 2_4423 numpy Version: 1.0.4 Requires: Python 2.3 or higher. License: NumPy license (see LICENSE.txt in the NumPy source code) Copyright 1999 - 2005 Pearu Peterson all rights reserved. http://cens.ioc.ee/projects/f2py2e/ Traceback (most recent call last): File "C:\Python25\Scripts\f2py.py", line 26, in main() File "C:\Python25\lib\site-packages\numpy\f2py\f2py2e.py", line 560, in main run_main(sys.argv[1:]) File "C:\Python25\lib\site-packages\numpy\f2py\f2py2e.py", line 339, in run_main files,options=scaninputline(comline_list) File "C:\Python25\lib\site-packages\numpy\f2py\f2py2e.py", line 241, in scaninputline sys.exit() SystemExit Mark, I appears that the system has prematurely exited from execution. Any solutions??? David This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated. From robert.kern at gmail.com Tue Sep 9 18:57:52 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 9 Sep 2008 17:57:52 -0500 Subject: [Numpy-discussion] Still having issues In-Reply-To: <27CC3060AF71DA40A5DC85F7D5B70F3804F8A915@AWMAIL04.belcan.com> References: <27CC3060AF71DA40A5DC85F7D5B70F3804F8A915@AWMAIL04.belcan.com> Message-ID: <3d375d730809091557q54424dcfje131edb213af3a07@mail.gmail.com> On Tue, Sep 9, 2008 at 17:53, Blubaugh, David A. wrote: > It appears that once I run the f2py.py script from the IDLE environment, That's your problem. It is a script to run from the command line, not a module for you to import. Please reply to messages with the reply button in your mail reader instead of creating a new thread every time. Thanks. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From simon.palmer at gmail.com Tue Sep 9 20:59:45 2008 From: simon.palmer at gmail.com (SimonPalmer) Date: Tue, 9 Sep 2008 17:59:45 -0700 (PDT) Subject: [Numpy-discussion] norm_gen Message-ID: <503f00ce-3b68-46df-9e09-34de50aca101@w24g2000prd.googlegroups.com> Hi, can anyone point me in the direction of some example code to use the norm_gen function in scipy.stats? I know this is the wrong board but I seem to be at the wrong end of the moderation queue on the scipy board and have been for a couple of weeks, so my posts over there don't show up. I'm hoping that someone here might be able to point me in the right direction Thanks in advance Simon From oliphant at enthought.com Wed Sep 10 00:38:24 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 09 Sep 2008 23:38:24 -0500 Subject: [Numpy-discussion] NumPy arrays that use memory allocated from other libraries or tools Message-ID: <48C74F40.3090103@enthought.com> I wanted to point anybody interested to a blog post that describes a useful pattern for having a NumPy array that points to the memory created by a different memory manager than the standard one used by NumPy. The pattern shows how to create a NumPy array that points to previously allocated memory and then shows how to construct an object that allows the correct deallocator to be called when the NumPy array is freed. This may be useful if you are wrapping code that has it's own memory management scheme. Comments and feedback is welcome. The post is http://blog.enthought.com/?p=62 Best regards, -Travis Oliphant From Chris.Barker at noaa.gov Wed Sep 10 01:44:38 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 09 Sep 2008 22:44:38 -0700 Subject: [Numpy-discussion] fromfile() improvements (was: planning for numpy 1.3.0 release) In-Reply-To: <9457e7c80809090049h29b5f019g41457c247901a16@mail.gmail.com> References: <48C5F833.8060609@noaa.gov> <9457e7c80809090049h29b5f019g41457c247901a16@mail.gmail.com> Message-ID: <48C75EC6.3070407@noaa.gov> St?fan van der Walt wrote: > 2008/9/9 Christopher Barker : >> Anyone want to help with improvements to fromfile() for text files? > > This is low hanging fruit for anyone with some experience in C. We > can definitely get it done for 1.3. Chris, would you file a ticket > and add the detail from your mailing list posts, if that hasn't > already been done? Done: http://scipy.org/scipy/numpy/ticket/909 ( By the way, is there a way to fix the typo in the ticket title? --oops!) There are a few fromfile() related tickets that I referenced as well. It's not totally straightforward what should be done, so I've included the text of the ticket here to start a discussion: Proposed Enhancements and bug fixes for fromfile() and fromstring() text handling: Motivation: The goal of the fromfile() text file handling capability is to enable users to write code that can read a lot of numbers from a text file into an array. Python provides a lot of nifty text processing capabilities, and there are a number of higher level facilities for reading blocks of data (including numpy.loadtxt). These are very capable, but there really is a significant performance hit, at least when loading 10s of thousands of numbers into a file. We don't want to write all of loadtxt() and friends in C. Rather, the goal is to allow the simple cases to be done very efficiently, and hopefully fancier text reading packages can build on it to add more features. Unfortunately, the current (numpy version 1.2) version has a few bugs and limitations that keep of from being nearly as useful as it could be. Possible features: * Create fromtextfile() and fromtextstring functions, distinct from fromfile() and fromstring(). It really is a different functionality. fromfile() could still call fromtextfile() for backward compatibility. * Allow more than one separator? for example, a comma or whitespace? In the general case, the user could perhaps specify any number of separators, though I doubt that would be useful in practice. At the very least, however, fromtextfile() should support reading files that look like: 43.5, 345.6, 123.456, 234.33 34.5, 22.57, 2345, 2345, 252 ... That is, comma separated, but being able to read multiple lines in one shot. The easiest way to support that would probably be to always allow whitespace as a separator, and add the one passed in. I can't think of a reason not to do this, but maybe I'm not very imaginative. * Allow the user to specify a shape for the output array. There may be little point, as all this does is save a calls to reshape(), but it may be another way to support the above. i.e. you could read that data with: a = np.fromtextfile(infile, dtype=np.float, sep=',', shape=(-1, 4)) Then it would know to skip the newlines every 4 elements. * Allow the user to specify a comment string. The reader would then skip everything in the file between the comment string and a newline. Maybe Universal newline -- any of \r, \n or \r\n. Or simply expect that the user has opened the file with mode 'U' if they want that. This could also be extended to support C-style comments with an opening and closing character sequence, but that's a lot less common. * Allow the user to specify a Locale. It may be best to be able to specify a locale, rather than relying on the system on (whether '.' or ',' is the decimal separator, for instance. (ticket #884) * parsing of "Inf" and the like that doesn't depend on system (ticket #510). This would be nice, but maybe too difficult -- would we need to write our own scanf? Bugs to be fixed: ? * fromfile() and fromstring handling malformed data poorly: ticket #883 * Any others? NOTE: my C is pretty lame, or I'd do some of this. I could help out with writing tests, etc. though. Thanks all, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From stefan at sun.ac.za Wed Sep 10 03:55:04 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 10 Sep 2008 09:55:04 +0200 Subject: [Numpy-discussion] NumPy arrays that use memory allocated from other libraries or tools In-Reply-To: <48C74F40.3090103@enthought.com> References: <48C74F40.3090103@enthought.com> Message-ID: <9457e7c80809100055m1d0f754fv95f2a566ffcc2645@mail.gmail.com> 2008/9/10 Travis E. Oliphant : > The post is > > http://blog.enthought.com/?p=62 Very cool post, thank you! I agree that it would be great to have such a mechanism in NumPy. Cheers St?fan From stefan at sun.ac.za Wed Sep 10 04:44:52 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 10 Sep 2008 10:44:52 +0200 Subject: [Numpy-discussion] fromfile() improvements (was: planning for numpy 1.3.0 release) In-Reply-To: <48C75EC6.3070407@noaa.gov> References: <48C5F833.8060609@noaa.gov> <9457e7c80809090049h29b5f019g41457c247901a16@mail.gmail.com> <48C75EC6.3070407@noaa.gov> Message-ID: <9457e7c80809100144v57f9252dt2e059d24122f8dfa@mail.gmail.com> 2008/9/10 Christopher Barker : > http://scipy.org/scipy/numpy/ticket/909 > > ( By the way, is there a way to fix the typo in the ticket title? --oops!) Fixed. Thanks a lot for writing this up! Cheers St?fan From pgmdevlist at gmail.com Wed Sep 10 11:17:10 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 10 Sep 2008 11:17:10 -0400 Subject: [Numpy-discussion] Issue w/ ndarray.view Message-ID: <200809101117.10546.pgmdevlist@gmail.com> All, I was fixing MaskedArray.view for masked arrays with flexible type when I ran into a small pb with view. The new definition accepts 2 keywords dtype and type. I thought I could easily redefined MaskedArray.view as def view(self, dtype=None, type=None): output = ndarray.view(self, dtype=dtype, type=type) [some extra lines to view the mask if needed] return output Unfortunately, that doesn't work: if type is None, I get a ValueError that type must be a subtype of ndarray. Using output = ndarray.view(self, dtype=dtype, type=type or ndarray) doesn't work either if dtype was already a type, as it complains that type is given twice. I end up having to test whether dtype is None, type is None, build a tuple of arguments (args in one of (dtype,) or (type,) or (dtype,type) or ()) and parse that to ndarray.view(self, *args). That's a tad ugly and irritating. Shouldn't the checks on the arguments dealt with on the C side, instead of having to redo them in Python all the time ? As a side-note, recarray.view still accepts only one argument. Looks like there's a pb of consistency. Looking forward to reading your comments P. From civija at gmail.com Wed Sep 10 11:32:57 2008 From: civija at gmail.com (Miroslav Sabljic) Date: Wed, 10 Sep 2008 17:32:57 +0200 Subject: [Numpy-discussion] numpy build on windows In-Reply-To: <48C13139.6040705@ar.media.kyoto-u.ac.jp> References: <59ccde810809040808i19777671s8d30b0a655f68da8@mail.gmail.com> <5b8d13220809041048w5b7d64bboe4f608cf9645e211@mail.gmail.com> <59ccde810809050151g457a8f7dv4a530dd780018956@mail.gmail.com> <48C0F5BB.8080708@ar.media.kyoto-u.ac.jp> <59ccde810809050446k3d6c2c61pcf7cdd998eede0b6@mail.gmail.com> <48C121F7.4070209@ar.media.kyoto-u.ac.jp> <59ccde810809050600i29eca7b5i85e22373a85c7b65@mail.gmail.com> <48C12A55.40900@ar.media.kyoto-u.ac.jp> <59ccde810809050628k5dc3fef8q3375ba111e01c17c@mail.gmail.com> <48C13139.6040705@ar.media.kyoto-u.ac.jp> Message-ID: <59ccde810809100832m775ed51aq6b3c1e41bda5d557@mail.gmail.com> On 9/5/08, David Cournapeau wrote: > You should first test the binary (import numpy; numpy.test()), to see > whether the built was really sucessful. Hy again! :) I've managed to build numpy and install it. 'import numpy' works OK, numpy.show_config() looks OK, but when I try tu run numpy.test() python crashes. I've attached log of the python session and screenshot. It always crashes at the same place, so my question is what am I doing wrong and how can I debug the problem that is cousing python.exe to crash? TIA! This is my site.cfg: [mkl] include_dirs = M:\ASTLIB\ia32-unknown-winnt_vc9\intel\mkl\v10.0.1\include library_dirs = M:\ASTLIB\ia32-unknown-winnt_vc9\intel\mkl\v10.0.1\ia32\lib mkl_libs = mkl_c, libguide40 lapack_libs = mkl_lapack -- Best regards, Miroslav -------------- next part -------------- A non-text attachment was scrubbed... Name: python_crash.png Type: image/png Size: 24596 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: numpy_crash.log Type: application/octet-stream Size: 4193 bytes Desc: not available URL: From pgmdevlist at gmail.com Wed Sep 10 12:47:26 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 10 Sep 2008 12:47:26 -0400 Subject: [Numpy-discussion] Pb while compiling numpy svn Message-ID: <200809101247.27377.pgmdevlist@gmail.com> All, I can't compile the latest SVN on my machine. You'll find the corresponding error message at the end. Any help/comment welcome. Thx a lot in advance, P. x86_64-pc-linux-gnu-gcc -pthread _configtest.o -L/usr/lib64 -lf77blas -lcblas -latlas -o _configtest ATLAS version 3.8.2 built by root on Thu Jun 19 14:40:27 EDT 2008: UNAME : Linux back2tux 2.6.22-suspend2-r1 #8 Fri Feb 15 17:00:17 EST 2008 x86_64 Mobile AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux INSTFLG : -1 0 -a 1 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_HAMMER -DATL_CPUMHZ=2205 -DATL_SSE2 -DATL_SSE1 -DATL_3DNow -DATL_USE64BITS -DATL_GAS_x8664 F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 262144 F77 : gfortran, version GNU Fortran 95 (GCC) 4.1.2 (Gentoo 4.1.2 p1.0.1) F77FLAGS : -O2 -m64 SMC : x86_64-pc-linux-gnu-gcc, version x86_64-pc-linux-gnu-gcc (GCC) 4.1.2 (Gentoo 4.1.2 p1.0.1) SMCFLAGS : -march=k8 -O2 -pipe -m64 SKC : x86_64-pc-linux-gnu-gcc, version x86_64-pc-linux-gnu-gcc (GCC) 4.1.2 (Gentoo 4.1.2 p1.0.1) SKCFLAGS : -march=k8 -O2 -pipe -m64 ... creating build/temp.linux-x86_64-2.4/build/src.linux-x86_64-2.4/numpy/core/src compile options: '-Ibuild/src.linux-x86_64-2.4/numpy/core/src -Inumpy/core/include -Ibuild/src.linux-x86_64-2.4/numpy/core/include/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' x86_64-pc-linux-gnu-gcc: build/src.linux-x86_64-2.4/numpy/core/src/umathmodule.c numpy/core/src/umathmodule.c.src:329: error: expected identifier or ?(? before ?< References: Message-ID: Hi Jarrod et al: On Sep 3, 2008, at 2:16 AM, Jarrod Millman wrote: > Here is the universal Mac binary: > https://cirl.berkeley.edu/numpy/numpy-1.2.0rc1-py2.5-macosx10.5.dmg I've been running the enthought distribution: Python 2.5.2 |EPD 4.0.30001| (r252:60911, Aug 17 2008, 17:29:54) The new numpy-1.2.0rc1-py2.5-macosx10.5.dmg apparently won't install under anything other than a python.org python framework, so it ended up in the wrong (for me) place. Is this expected? Thanks to everyone for all the hard work. I really appreciate the existence of numpy and scipy. Bob Pyle From oliphant at enthought.com Wed Sep 10 13:09:24 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Wed, 10 Sep 2008 12:09:24 -0500 Subject: [Numpy-discussion] Issue w/ ndarray.view In-Reply-To: <200809101117.10546.pgmdevlist@gmail.com> References: <200809101117.10546.pgmdevlist@gmail.com> Message-ID: <48C7FF44.9070505@enthought.com> Pierre GM wrote: > All, > I was fixing MaskedArray.view for masked arrays with flexible type when I ran > into a small pb with view. > The new definition accepts 2 keywords dtype and type. I thought I could easily > redefined MaskedArray.view as > > def view(self, dtype=None, type=None): > output = ndarray.view(self, dtype=dtype, type=type) > [some extra lines to view the mask if needed] > return output > > Unfortunately, that doesn't work: if type is None, I get a ValueError that > type must be a subtype of ndarray. Using > output = ndarray.view(self, dtype=dtype, type=type or ndarray) > doesn't work either if dtype was already a type, as it complains that type is > given twice. > > I end up having to test whether dtype is None, type is None, build a tuple of > arguments (args in one of (dtype,) or (type,) or (dtype,type) or ()) and > parse that to ndarray.view(self, *args). > That's a tad ugly and irritating. Shouldn't the checks on the arguments dealt > with on the C side, instead of having to redo them in Python all the time ? > That would be nicer. It's just a bit harder to do on the C-side where PyArg_ParseTuple is the heavy lifter and creates habits of argument acceptance that are different than on the Python side. > As a side-note, recarray.view still accepts only one argument. Looks like > there's a pb of consistency. > Yes, it should be changed also. -Travis From robert.kern at gmail.com Wed Sep 10 15:57:15 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 10 Sep 2008 14:57:15 -0500 Subject: [Numpy-discussion] 1.2.0rc1 tagged! In-Reply-To: References: Message-ID: <3d375d730809101257y61dc25c8v927c6a2cbaa66fd4@mail.gmail.com> On Wed, Sep 10, 2008 at 11:07, Robert Pyle wrote: > Hi Jarrod et al: > > On Sep 3, 2008, at 2:16 AM, Jarrod Millman wrote: > >> Here is the universal Mac binary: >> https://cirl.berkeley.edu/numpy/numpy-1.2.0rc1-py2.5-macosx10.5.dmg > > I've been running the enthought distribution: Python 2.5.2 |EPD > 4.0.30001| (r252:60911, Aug 17 2008, 17:29:54) > > The new numpy-1.2.0rc1-py2.5-macosx10.5.dmg apparently won't install > under anything other than a python.org python framework, so it ended > up in the wrong (for me) place. Is this expected? Yes. It is difficult (or impossible without writing new code) to build these kinds of installers to work with different locations of Python frameworks. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Chris.Barker at noaa.gov Wed Sep 10 16:28:49 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 10 Sep 2008 13:28:49 -0700 Subject: [Numpy-discussion] 1.2.0rc1 tagged! In-Reply-To: <3d375d730809101257y61dc25c8v927c6a2cbaa66fd4@mail.gmail.com> References: <3d375d730809101257y61dc25c8v927c6a2cbaa66fd4@mail.gmail.com> Message-ID: <48C82E01.6070100@noaa.gov> Robert Kern wrote: > Yes. It is difficult (or impossible without writing new code) to build > these kinds of installers to work with different locations of Python > frameworks. yup. however, perhaps the approach taken by wxPython would be worth considering. Robin is putting the whole package in: /usr/local/lib/wxPython-unicode-2.8.8.1/lib/python2.5/site-packages Then putting a *.pth file in both /Library and /System pythons so that it will work with either install of 2.5. It seems it would be easy to add Enthought's too, though I have no idea where Enthought puts it. Actually, perhaps we should try to establish a standard, putting packages that work with multiple pythons in something like: /usr/local/lib/UniversalPython2.5/site-packages and add that to a *.pth file in various versions. Note that Robin put: import site; site.addsitedir('/usr/local/lib/wxPython-unicode-2.8.8.1/lib/python2.5') in the pth file, rather than just the path -- I have no idea why. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Wed Sep 10 16:38:36 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 10 Sep 2008 15:38:36 -0500 Subject: [Numpy-discussion] 1.2.0rc1 tagged! In-Reply-To: <48C82E01.6070100@noaa.gov> References: <3d375d730809101257y61dc25c8v927c6a2cbaa66fd4@mail.gmail.com> <48C82E01.6070100@noaa.gov> Message-ID: <3d375d730809101338r155eef9fh4a56bcc43e7e0891@mail.gmail.com> On Wed, Sep 10, 2008 at 15:28, Christopher Barker wrote: > Robert Kern wrote: >> Yes. It is difficult (or impossible without writing new code) to build >> these kinds of installers to work with different locations of Python >> frameworks. > > yup. however, perhaps the approach taken by wxPython would be worth > considering. Robin is putting the whole package in: > > /usr/local/lib/wxPython-unicode-2.8.8.1/lib/python2.5/site-packages > > Then putting a *.pth file in both /Library and /System pythons so that > it will work with either install of 2.5. It seems it would be easy to > add Enthought's too, though I have no idea where Enthought puts it. We're in /Library/Frameworks/Python.framework/..., too, just with a different version number to keep the installations separate. > Actually, perhaps we should try to establish a standard, putting > packages that work with multiple pythons in something like: > > /usr/local/lib/UniversalPython2.5/site-packages > > and add that to a *.pth file in various versions. The problem is that each Python has to support this. If each package has to drop in a .pth file to hack in support for this, it'll just be a shambles. It's possible that /Library/Python/2.x/site-packages was designed for this, but I more strongly suspect that it was just a place outside of /System for third-party packages for the /System Python to go, and it was not intended to be used by any other Python. > Note that Robin put: > > import site; > site.addsitedir('/usr/local/lib/wxPython-unicode-2.8.8.1/lib/python2.5') > > in the pth file, rather than just the path -- I have no idea why. Because it, too, has a .pth file. Python only reads .pth files in paths that it knows about. site.addsitedir() registers such paths. This is not a path (and my sincerest apologies for the pun) that I would like to go down. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pav at iki.fi Wed Sep 10 17:04:06 2008 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 10 Sep 2008 21:04:06 +0000 (UTC) Subject: [Numpy-discussion] Still having issues References: <27CC3060AF71DA40A5DC85F7D5B70F3804F8A915@AWMAIL04.belcan.com> <3d375d730809091557q54424dcfje131edb213af3a07@mail.gmail.com> Message-ID: Tue, 09 Sep 2008 17:57:52 -0500, Robert Kern wrote: > On Tue, Sep 9, 2008 at 17:53, Blubaugh, David A. > wrote: >> It appears that once I run the f2py.py script from the IDLE >> environment, > > That's your problem. It is a script to run from the command line, not a > module for you to import. > > Please reply to messages with the reply button in your mail reader > instead of creating a new thread every time. Thanks. Please also see the documentation for f2py, http://www.scipy.org/F2py http://www.scipy.org/F2PY_Windows http://cens.ioc.ee/projects/f2py2e/usersguide/index.html -- Pauli Virtanen From Chris.Barker at noaa.gov Wed Sep 10 17:42:37 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 10 Sep 2008 14:42:37 -0700 Subject: [Numpy-discussion] 1.2.0rc1 tagged! In-Reply-To: <3d375d730809101338r155eef9fh4a56bcc43e7e0891@mail.gmail.com> References: <3d375d730809101257y61dc25c8v927c6a2cbaa66fd4@mail.gmail.com> <48C82E01.6070100@noaa.gov> <3d375d730809101338r155eef9fh4a56bcc43e7e0891@mail.gmail.com> Message-ID: <48C83F4D.3020609@noaa.gov> Robert Kern wrote: >> Actually, perhaps we should try to establish a standard, putting >> packages that work with multiple pythons in something like: >> >> /usr/local/lib/UniversalPython2.5/site-packages >> >> and add that to a *.pth file in various versions. > > The problem is that each Python has to support this. If each package > has to drop in a .pth file to hack in support for this, it'll just be > a shambles. maybe, but what we have is already a shambles, and a constant annoyance, IF we could get it established as something of standard , things would be better. > It's possible that /Library/Python/2.x/site-packages was designed for > this, but I more strongly suspect that it was just a place outside of > /System for third-party packages for the /System Python to go, and it > was not intended to be used by any other Python. I agree -- Apple doesn't ever seem to concern itself with other pythons. However, is there any reason not to use it that way? It's a logical place to put it. >> import site; >> site.addsitedir('/usr/local/lib/wxPython-unicode-2.8.8.1/lib/python2.5') >> >> in the pth file, rather than just the path -- I have no idea why. > > Because it, too, has a .pth file. Python only reads .pth files in > paths that it knows about. site.addsitedir() registers such paths. ahh, thanks. Now I get it. > This is not a path (and my sincerest apologies for the pun) that I > would like to go down. I'm not sure either, but I now that this is an issue for every compiled package out there, so it would be nice to make it a bit easier for the users and package builders one way or another. Binary eggs always seem to be a struggle on the mac too.... oh well, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From dblubaugh at belcan.com Wed Sep 10 20:33:31 2008 From: dblubaugh at belcan.com (Blubaugh, David A.) Date: Wed, 10 Sep 2008 20:33:31 -0400 Subject: [Numpy-discussion] Still having issues with f2py In-Reply-To: References: <27CC3060AF71DA40A5DC85F7D5B70F3804F8A915@AWMAIL04.belcan.com> Message-ID: <27CC3060AF71DA40A5DC85F7D5B70F3804F8B21B@AWMAIL04.belcan.com> Mark, THANK YOU GREATLY FOR YOUR HELP!! I have executed all that suggestions that you have made: I was able to have f2py be able to detect two distinct fortran compilers on the computer that I am utilizing. However, I have found a new error that I found to be somewhat unclear. The error is as follows: I will try to input the following commands into an MS-DOS prompt: C:\Python25\Scripts>c:\python25\python f2py.py -c fib1_proj.f90 -m fib1_proj You should know that I have the FORTRAN file located within the Scripts directory along with the f2py.py file, which is also located in the Scripts directory. The scripts directory is as follows: C:\Python25\Scripts Therefore, once this line was executed, it was then able to execute python as well as f2py.py and be able to obtain fib1_proj.f90. However, this is the error that I obtained: running build running scons No module named msvccompiler in numpy.distutils; trying from distutils error: Python was built with Visual Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py. I was wondering if the (.f90) extension was causing the problems?? I was also wondering as to what is involved with "compiling with MingW32, by passing "-c mingw32" to setup.py."?? Thank You All Greatly, David Blubaugh ________________________________ From: Mark Miller [mailto:markperrymiller at gmail.com] Sent: Tuesday, September 09, 2008 7:06 PM To: Blubaugh, David A. Subject: Re: Still having issues Hi David: I'm not 100% sure of what's going on. I've used F2py many times myself in the windows environment. Once thing that struck me from your message is that you mentioned running the command from within IDLE. Try opening up a command prompt window (in the Accessories section within the START menu of Windows) and issuing the commands from there. There is no need to import F2py modules into the python shell itself. If issuing things from the command line does not work, then I would suggest the following: 1) remove Numpy 2) remove Python 3) re-install Python from the 'official' binary source (the simple windows setup program) 4) re-install Numpy using the simple windows setup program Once that is done, try the following two things(I assume that your Python will be in c:\python25. If not, change the commands as appropriate!): First, make sure that you can run F2py. Just run the following to see if you get any errors. python c:\python25\scripts\f2py.py If that works, issue the following command to see if f2py is picking up your fortran compiler. Python C:\python25\scripts\f2py.py -c --help-fcompiler If the second one fails, you may need to make sure that you add your fortran compiler's path to the Windows environment path variable. Does that make sense? -Mark On Tue, Sep 9, 2008 at 3:53 PM, Blubaugh, David A. wrote: Mark, I have taken the advice that you had given to me the other day. However, I am still having issues with actually executing the f2py.py script. > If numpy is installed, then f2py will be too. > On the windows environment, > there is a file called f2py.py that you can >>> > call from the command line. It > should be in the 'scripts' directory of your > > Python installation. > > Try something like this: > > python c:\python25\scripts\f2py.py > (of course change to reflect your own python > > installation directory) It appears that once I run the f2py.py script from the IDLE environment, I will then obtain the following error: You will have to read at the end of the listed output. It looks as though there was a premature system exit once f2py.py was executed. >>> Usage: 1) To construct extension module sources: f2py [] [[[only:]||[skip:]] \ ] \ [: ...] 2) To compile fortran files and build extension modules: f2py -c [, , ] 3) To generate signature files: f2py -h ...< same options as in (1) > Description: This program generates a Python C/API file (module.c) that contains wrappers for given fortran functions so that they can be called from Python. With the -c option the corresponding extension modules are built. Options: --g3-numpy Use numpy.f2py.lib tool, the 3rd generation of F2PY, with NumPy support. --2d-numpy Use numpy.f2py tool with NumPy support. [DEFAULT] --2d-numeric Use f2py2e tool with Numeric support. --2d-numarray Use f2py2e tool with Numarray support. -h Write signatures of the fortran routines to file and exit. You can then edit and use it instead of . If ==stdout then the signatures are printed to stdout. Names of fortran routines for which Python C/API functions will be generated. Default is all that are found in . Paths to fortran/signature files that will be scanned for in order to determine their signatures. skip: Ignore fortran functions that follow until `:'. only: Use only fortran functions that follow until `:'. : Get back to mode. -m Name of the module; f2py generates a Python/C API file module.c or extension module . Default is 'untitled'. --[no-]lower Do [not] lower the cases in . By default, --lower is assumed with -h key, and --no-lower without -h key. --build-dir All f2py generated files are created in . Default is tempfile.mktemp(). --overwrite-signature Overwrite existing signature file. --[no-]latex-doc Create (or not) module.tex. Default is --no-latex-doc. --short-latex Create 'incomplete' LaTeX document (without commands \documentclass, \tableofcontents, and \begin{document}, \end{document}). --[no-]rest-doc Create (or not) module.rst. Default is --no-rest-doc. --debug-capi Create C/API code that reports the state of the wrappers during runtime. Useful for debugging. --[no-]wrap-functions Create Fortran subroutine wrappers to Fortran 77 functions. --wrap-functions is default because it ensures maximum portability/compiler independence. --include_paths ::... Search include files from the given directories. --help-link [..] List system resources found by system_info.py. See also --link- switch below. [..] is optional list of resources names. E.g. try 'f2py --help-link lapack_opt'. --quiet Run quietly. --verbose Run with extra verbosity. -v Print f2py version ID and exit. numpy.distutils options (only effective with -c): --fcompiler= Specify Fortran compiler type by vendor --compiler= Specify C compiler type (as defined by distutils) --help-fcompiler List available Fortran compilers and exit --f77exec= Specify the path to F77 compiler --f90exec= Specify the path to F90 compiler --f77flags= Specify F77 compiler flags --f90flags= Specify F90 compiler flags --opt= Specify optimization flags --arch= Specify architecture specific optimization flags --noopt Compile without optimization --noarch Compile without arch-dependent optimization --debug Compile with debugging information Extra options (only effective with -c): --link- Link extension module with as defined by numpy.distutils/system_info.py. E.g. to link with optimized LAPACK libraries (vecLib on MacOSX, ATLAS elsewhere), use --link-lapack_opt. See also --help-link switch. -L/path/to/lib/ -l -D -U -I/path/to/include/ .o .so .a Using the following macros may be required with non-gcc Fortran compilers: -DPREPEND_FORTRAN -DNO_APPEND_FORTRAN -DUPPERCASE_FORTRAN -DUNDERSCORE_G77 When using -DF2PY_REPORT_ATEXIT, a performance report of F2PY interface is printed out at exit (platforms: Linux). When using -DF2PY_REPORT_ON_ARRAY_COPY=, a message is sent to stderr whenever F2PY interface makes a copy of an array. Integer sets the threshold for array sizes when a message should be shown. Version: 2_4423 numpy Version: 1.0.4 Requires: Python 2.3 or higher. License: NumPy license (see LICENSE.txt in the NumPy source code) Copyright 1999 - 2005 Pearu Peterson all rights reserved. http://cens.ioc.ee/projects/f2py2e/ Traceback (most recent call last): File "C:\Python25\Scripts\f2py.py", line 26, in main() File "C:\Python25\lib\site-packages\numpy\f2py\f2py2e.py", line 560, in main run_main(sys.argv[1:]) File "C:\Python25\lib\site-packages\numpy\f2py\f2py2e.py", line 339, in run_main files,options=scaninputline(comline_list) File "C:\Python25\lib\site-packages\numpy\f2py\f2py2e.py", line 241, in scaninputline sys.exit() SystemExit Mark, I appears that the system has prematurely exited from execution. Any solutions??? David This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lbrooks at mit.edu Wed Sep 10 20:06:41 2008 From: lbrooks at mit.edu (Lane Brooks) Date: Wed, 10 Sep 2008 18:06:41 -0600 Subject: [Numpy-discussion] Dealing with types in extension modules Message-ID: <48C86111.1040608@mit.edu> When writing an numpy extension module, what is the preferred way to deal with the all the possible types an ndarray can have? I have some data processing functions I need to implement and they need to be generic and work for all the possible numerical dtypes. I do not want to have to re-implement the same C-code for all the possible types, so the way I approached it was to use a C++ template function to implement the processing. Then I have a dispatching function that checks the type of the input ndarray and calls the correct template. Is there a better way? For example, suppose I have a processing template function: template int Resize(T *datai) { ... } Then in my dispatch function, I do: switch(PyArray_TYPE(bufi)) { case NPY_UBYTE: Resize((npy_ubyte *) PyArray_DATA(bufi)); break; case NPY_BYTE: Resize((npy_byte *) PyArray_DATA(bufi)); break; case NPY_USHORT: Resize((npy_ushort *) PyArray_DATA(bufi)); break; case NPY_SHORT: Resize((npy_short *) PyArray_DATA(bufi)); break; case NPY_UINT: Resize((npy_uint *) PyArray_DATA(bufi)); break; .... } From tgrav at mac.com Wed Sep 10 20:58:28 2008 From: tgrav at mac.com (Tommy Grav) Date: Wed, 10 Sep 2008 20:58:28 -0400 Subject: [Numpy-discussion] 1.2.0rc1 tagged! In-Reply-To: <48C83F4D.3020609@noaa.gov> References: <3d375d730809101257y61dc25c8v927c6a2cbaa66fd4@mail.gmail.com> <48C82E01.6070100@noaa.gov> <3d375d730809101338r155eef9fh4a56bcc43e7e0891@mail.gmail.com> <48C83F4D.3020609@noaa.gov> Message-ID: <646EE7F0-58D0-4268-91EA-52A10EC6176B@mac.com> I thought is was pretty standard that non-system versions of python should go into /Library/Frameworks/Python.framework/ on the OS X? Is this not the case? Cheers Tommy From robert.kern at gmail.com Wed Sep 10 22:12:18 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 10 Sep 2008 21:12:18 -0500 Subject: [Numpy-discussion] 1.2.0rc1 tagged! In-Reply-To: <646EE7F0-58D0-4268-91EA-52A10EC6176B@mac.com> References: <3d375d730809101257y61dc25c8v927c6a2cbaa66fd4@mail.gmail.com> <48C82E01.6070100@noaa.gov> <3d375d730809101338r155eef9fh4a56bcc43e7e0891@mail.gmail.com> <48C83F4D.3020609@noaa.gov> <646EE7F0-58D0-4268-91EA-52A10EC6176B@mac.com> Message-ID: <3d375d730809101912k4ffbbee2g1bfff79c9070514a@mail.gmail.com> On Wed, Sep 10, 2008 at 19:58, Tommy Grav wrote: > I thought is was pretty standard that non-system versions of python > should go > into /Library/Frameworks/Python.framework/ on the OS X? Is this not > the case? Yes, but frameworks are versioned, and the files installed by .mpkg packages have the version fully specified. EPD uses a different version number than the www.python.org release in order to keep the two installations separate (so you can install EPD, then delete it with few, if any repercussions). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at enthought.com Wed Sep 10 22:21:20 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Wed, 10 Sep 2008 21:21:20 -0500 Subject: [Numpy-discussion] Dealing with types in extension modules In-Reply-To: <48C86111.1040608@mit.edu> References: <48C86111.1040608@mit.edu> Message-ID: <48C880A0.1030804@enthought.com> Lane Brooks wrote: > When writing an numpy extension module, what is the preferred way to > deal with the all the possible types an ndarray can have? > > I have some data processing functions I need to implement and they need > to be generic and work for all the possible numerical dtypes. I do not > want to have to re-implement the same C-code for all the possible types, > so the way I approached it was to use a C++ template function to > implement the processing. Then I have a dispatching function that > checks the type of the input ndarray and calls the correct template. Is > there a better way? > You could store the functions in an array of function pointers and look-up the correct one using the typenum: resize_funcs[PyArray_Type(buf1)](PyArray_DATA(bufi)) with resize_funcs filled appropriately. -Travis From tgrav at mac.com Wed Sep 10 22:44:01 2008 From: tgrav at mac.com (Tommy Grav) Date: Wed, 10 Sep 2008 22:44:01 -0400 Subject: [Numpy-discussion] 1.2.0rc1 tagged! In-Reply-To: <3d375d730809101912k4ffbbee2g1bfff79c9070514a@mail.gmail.com> References: <3d375d730809101257y61dc25c8v927c6a2cbaa66fd4@mail.gmail.com> <48C82E01.6070100@noaa.gov> <3d375d730809101338r155eef9fh4a56bcc43e7e0891@mail.gmail.com> <48C83F4D.3020609@noaa.gov> <646EE7F0-58D0-4268-91EA-52A10EC6176B@mac.com> <3d375d730809101912k4ffbbee2g1bfff79c9070514a@mail.gmail.com> Message-ID: <074AB5DC-63F5-47EC-806C-E6F6CE668AFB@mac.com> On Sep 10, 2008, at 10:12 PM, Robert Kern wrote: > On Wed, Sep 10, 2008 at 19:58, Tommy Grav wrote: >> I thought is was pretty standard that non-system versions of python >> should go >> into /Library/Frameworks/Python.framework/ on the OS X? Is this not >> the case? > > Yes, but frameworks are versioned, and the files installed by .mpkg > packages have the version fully specified. EPD uses a different > version number than the www.python.org release in order to keep the > two installations separate (so you can install EPD, then delete it > with few, if any repercussions). I see. I have been using ActivePython and the numpy packages has always worked fine for me :) Cheers Tommy From lbrooks at mit.edu Wed Sep 10 23:38:33 2008 From: lbrooks at mit.edu (Lane Brooks) Date: Wed, 10 Sep 2008 21:38:33 -0600 Subject: [Numpy-discussion] Dealing with types in extension modules In-Reply-To: <48C880A0.1030804@enthought.com> References: <48C86111.1040608@mit.edu> <48C880A0.1030804@enthought.com> Message-ID: <48C892B9.4060004@mit.edu> Travis E. Oliphant wrote: > Lane Brooks wrote: > >> When writing an numpy extension module, what is the preferred way to >> deal with the all the possible types an ndarray can have? >> >> I have some data processing functions I need to implement and they need >> to be generic and work for all the possible numerical dtypes. I do not >> want to have to re-implement the same C-code for all the possible types, >> so the way I approached it was to use a C++ template function to >> implement the processing. Then I have a dispatching function that >> checks the type of the input ndarray and calls the correct template. Is >> there a better way? >> >> > You could store the functions in an array of function pointers and > look-up the correct one using the typenum: > > resize_funcs[PyArray_Type(buf1)](PyArray_DATA(bufi)) > > with resize_funcs filled appropriately. > > -Travis Would this require implementing a unique function for each of the possible types, though? That is mostly what I want to avoid. I do not want to have to implement 10 to 15 different functions that all do the same exact thing but to different types of data. I guess with your proposal I can still use templates to have a single function definition. Lane -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Wed Sep 10 23:50:53 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Wed, 10 Sep 2008 22:50:53 -0500 Subject: [Numpy-discussion] Dealing with types in extension modules In-Reply-To: <48C892B9.4060004@mit.edu> References: <48C86111.1040608@mit.edu> <48C880A0.1030804@enthought.com> <48C892B9.4060004@mit.edu> Message-ID: <48C8959D.8040206@enthought.com> Lane Brooks wrote: > Travis E. Oliphant wrote: >> Lane Brooks wrote: >> >>> When writing an numpy extension module, what is the preferred way to >>> deal with the all the possible types an ndarray can have? >>> >>> I have some data processing functions I need to implement and they need >>> to be generic and work for all the possible numerical dtypes. I do not >>> want to have to re-implement the same C-code for all the possible types, >>> so the way I approached it was to use a C++ template function to >>> implement the processing. Then I have a dispatching function that >>> checks the type of the input ndarray and calls the correct template. Is >>> there a better way? >>> >>> >> You could store the functions in an array of function pointers and >> look-up the correct one using the typenum: >> >> resize_funcs[PyArray_Type(buf1)](PyArray_DATA(bufi)) >> >> with resize_funcs filled appropriately. >> >> -Travis > Would this require implementing a unique function for each of the > possible types, though? That is mostly what I want to avoid. I do > not want to have to implement 10 to 15 different functions that all do > the same exact thing but to different types of data. I guess with > your proposal I can still use templates to have a single function > definition. You could have a default function which does type coercion or uses type-indifferent code. It really depends on what you are doing. But generally if you want to support multiple types you have to repeat the code for that type, I don't know of anyway around that. Also, you don't have to fill in all the functions (some could be NULL and you could use coercion to another type or some other default implementation as needed). -Travis From lbrooks at mit.edu Wed Sep 10 23:59:59 2008 From: lbrooks at mit.edu (Lane Brooks) Date: Wed, 10 Sep 2008 21:59:59 -0600 Subject: [Numpy-discussion] Dealing with types in extension modules In-Reply-To: <48C8959D.8040206@enthought.com> References: <48C86111.1040608@mit.edu> <48C880A0.1030804@enthought.com> <48C892B9.4060004@mit.edu> <48C8959D.8040206@enthought.com> Message-ID: <48C897BF.1020806@mit.edu> Travis E. Oliphant wrote: > Lane Brooks wrote: > >> Travis E. Oliphant wrote: >> >>> Lane Brooks wrote: >>> >>> >>>> When writing an numpy extension module, what is the preferred way to >>>> deal with the all the possible types an ndarray can have? >>>> >>>> I have some data processing functions I need to implement and they need >>>> to be generic and work for all the possible numerical dtypes. I do not >>>> want to have to re-implement the same C-code for all the possible types, >>>> so the way I approached it was to use a C++ template function to >>>> implement the processing. Then I have a dispatching function that >>>> checks the type of the input ndarray and calls the correct template. Is >>>> there a better way? >>>> >>>> >>>> >>> You could store the functions in an array of function pointers and >>> look-up the correct one using the typenum: >>> >>> resize_funcs[PyArray_Type(buf1)](PyArray_DATA(bufi)) >>> >>> with resize_funcs filled appropriately. >>> >>> -Travis >>> >> Would this require implementing a unique function for each of the >> possible types, though? That is mostly what I want to avoid. I do >> not want to have to implement 10 to 15 different functions that all do >> the same exact thing but to different types of data. I guess with >> your proposal I can still use templates to have a single function >> definition. >> > You could have a default function which does type coercion or uses > type-indifferent code. It really depends on what you are doing. > > But generally if you want to support multiple types you have to repeat > the code for that type, I don't know of anyway around that. > > Also, you don't have to fill in all the functions (some could be NULL > and you could use coercion to another type or some other default > implementation as needed). > > -Travis Is such an approach preferred to templates? I think duplicating the code for different types sound tedious and error-prone. What is type-indifferent code? Is there a way, for example, to add or multiple two variables in a type-indifferent way? Thanks, Lane -------------- next part -------------- An HTML attachment was scrubbed... URL: From rocksportrocker at googlemail.com Thu Sep 11 07:25:33 2008 From: rocksportrocker at googlemail.com (Uwe Schmitt) Date: Thu, 11 Sep 2008 04:25:33 -0700 (PDT) Subject: [Numpy-discussion] Still having issues with f2py In-Reply-To: <27CC3060AF71DA40A5DC85F7D5B70F3804F8B21B@AWMAIL04.belcan.com> References: <27CC3060AF71DA40A5DC85F7D5B70F3804F8A915@AWMAIL04.belcan.com> <27CC3060AF71DA40A5DC85F7D5B70F3804F8B21B@AWMAIL04.belcan.com> Message-ID: On 11 Sep., 02:33, "Blubaugh, David A." wrote: > Mark, > ... ?I > was also wondering as to what is involved with "compiling with MingW32, > by passing "-c mingw32" to setup.py."?? If you do not have the right MS compiler on your machine, you can use the mingw port of gcc instead. If you use enthougths edition you already have it, else you have to install the compiler. Passing the compiler option to f2py is easiest if you write your own setup.py Mine look like this: from numpy.distutils.core import Extension import sys ext1 = Extension(name = '_module', sources = ['module.pyf', "algorithm.f",]) if __name__ == "__main__": from numpy.distutils.core import setup setup(name = 'testdmoude', ext_modules = [ext1,], ) You can call this as follows python setup.py build -cmingw32 Greetings, Uwe From rocksportrocker at googlemail.com Thu Sep 11 08:32:35 2008 From: rocksportrocker at googlemail.com (Uwe Schmitt) Date: Thu, 11 Sep 2008 05:32:35 -0700 (PDT) Subject: [Numpy-discussion] cross product of two vectors as sets Message-ID: Hi, I want to calculate the crossproduct (from set theory, not vectorspace crossproduct) of two vectors x,y. My method is as follows: >>> x = array([1,2,3]) >>> y = array([4,5]) >>> xx, yy = meshgrid(x,y) >>> array(zip(xx.flatten(), yy.flatten())) array([[1, 4], [2, 4], [3, 4], [1, 5], [2, 5], [3, 5]]) I guess there is an easier way to achieve this. Any hints ? Greetings, Uwe From aisaac at american.edu Thu Sep 11 09:39:25 2008 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 11 Sep 2008 09:39:25 -0400 Subject: [Numpy-discussion] cross product of two vectors as sets In-Reply-To: References: Message-ID: <48C91F8D.6090703@american.edu> Set product could be produced as:: dp = list((xi,yi) for xi in x for yi in y) dpasarray = np.array(dp) but unless you really need the array, you can use a generator. Alan Isaac From doutriaux1 at llnl.gov Thu Sep 11 15:12:10 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Thu, 11 Sep 2008 12:12:10 -0700 Subject: [Numpy-discussion] f2py related question Message-ID: <48C96D8A.5050800@llnl.gov> Hello, I have a quick question that I'm hoping f2py developpers will be able to quickly answer I have some C code, input type can be either int or long I'm trying to write some fortran interface to it my understanding is that integer in fortran corresponds to int integer(kind=2) matches short but what would match long ? Thanks, c. From dblubaugh at belcan.com Thu Sep 11 15:17:32 2008 From: dblubaugh at belcan.com (Blubaugh, David A.) Date: Thu, 11 Sep 2008 15:17:32 -0400 Subject: [Numpy-discussion] Still having issues with f2py Message-ID: <27CC3060AF71DA40A5DC85F7D5B70F380500CDFC@AWMAIL04.belcan.com> Uwe, I have tried your example and it look as though it would have worked. However, I did obtain the following error!!! python25> python setup.py build -cmingw32 running build running scons customize Mingw32CCompiler Found executable C:\gfortran\bin\gcc.exe Ignoring "Python was built with Visual Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py." (one s hould fix me in fcompiler/compaq.py) customize GnuFCompiler Could not locate executable g77 Found executable C:\Program Files\Microsoft Visual Studio\DF98\BIN\f77.exe gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found Found executable C:\Program Files\Microsoft Visual Studio\DF98\BIN\f77.exe customize IntelVisualFCompiler Could not locate executable ifl customize AbsoftFCompiler Found executable C:\Program Files\Microsoft Visual Studio\DF98\BIN\f90.exe Found executable C:\Program Files\Microsoft Visual Studio\DF98\BIN\f90.exe customize CompaqVisualFCompiler Found executable C:\apps\cygwin\bin\DF.exe Found executable C:\apps\cygwin\bin\DF.exe customize IntelItaniumVisualFCompiler Could not locate executable efl customize Gnu95FCompiler Found executable C:\gfortran\bin\gfortran.exe Found executable C:\gfortran\bin\gfortran.exe customize Gnu95FCompiler customize Mingw32CCompiler customize Mingw32CCompiler using scons Could not locate executable g++ running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler opti ons running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler opt ions running build_src building extension "_module" sources target build\src.win32-2.5\_modulemodule.c does not exist: Assuming _modulemodule.c was generated with "build_src --inplace" command. error: '_modulemodule.c' missing python25> It looks as though I am missing _modulemodule.c. I was wondering as to how I may obtain this file? So I may finally get f2py working. Also, you should all know that I have executed all of these commands within Cygwin!! Thanks, David Blubaugh This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated. From robert.kern at gmail.com Thu Sep 11 15:54:37 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 11 Sep 2008 14:54:37 -0500 Subject: [Numpy-discussion] f2py related question In-Reply-To: <48C96D8A.5050800@llnl.gov> References: <48C96D8A.5050800@llnl.gov> Message-ID: <3d375d730809111254rb84c931hf5b235632425da5e@mail.gmail.com> On Thu, Sep 11, 2008 at 14:12, Charles Doutriaux wrote: > Hello, > > I have a quick question that I'm hoping f2py developpers will be able to > quickly answer > > I have some C code, input type can be either int or long > I'm trying to write some fortran interface to it > > my understanding is that > integer in fortran corresponds to int > integer(kind=2) matches short > but what would match long ? Unfortunately, it depends on your system. A C long is defined to be at least as large as an int. Incidentally, an int is defined to be at least as long as a short and a short is defined to be at least as long as a char. So you *could* have char == short == int == long == 1 byte, but no such system exists (I hope!). On pretty much all 32-bit systems I am aware of, int == long == 4 bytes. On 64-bit systems, things get a little more fragmented. Most UNIX-type 64-bit systems, long == 8 bytes. On Win64, however, long == 4 bytes. This section of the Wikipedia article on 64-bit computing gives more detail: http://en.wikipedia.org/wiki/64-bit#64-bit_data_models Furthermore, it's not just the OS that determines the data model, but also the options that are given to your compiler when building your program *and also all of the libraries and system libraries* that it uses. Fortunately, there's an easy way for you to tell: just ask numpy. Python ints are C longs, and the default numpy int dtype matches Python ints. On my 32-bit system: >>> import numpy >>> numpy.dtype(int).itemsize 4 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pgmdevlist at gmail.com Thu Sep 11 15:54:21 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 11 Sep 2008 15:54:21 -0400 Subject: [Numpy-discussion] Pb while compiling numpy svn In-Reply-To: <200809101247.27377.pgmdevlist@gmail.com> References: <200809101247.27377.pgmdevlist@gmail.com> Message-ID: <200809111554.23122.pgmdevlist@gmail.com> On Wednesday 10 September 2008 12:47:26 you wrote: > All, > I can't compile the latest SVN on my machine. You'll find the corresponding > error message at the end. Sorry for the noise, my local checkout was borked... From doutriaux1 at llnl.gov Thu Sep 11 16:17:01 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Thu, 11 Sep 2008 13:17:01 -0700 Subject: [Numpy-discussion] f2py related question In-Reply-To: <3d375d730809111254rb84c931hf5b235632425da5e@mail.gmail.com> References: <48C96D8A.5050800@llnl.gov> <3d375d730809111254rb84c931hf5b235632425da5e@mail.gmail.com> Message-ID: <48C97CBD.6070100@llnl.gov> Thanks robert, That's exactly the information I needed. Thanks for the link too. C. Robert Kern wrote: > On Thu, Sep 11, 2008 at 14:12, Charles Doutriaux wrote: > >> Hello, >> >> I have a quick question that I'm hoping f2py developpers will be able to >> quickly answer >> >> I have some C code, input type can be either int or long >> I'm trying to write some fortran interface to it >> >> my understanding is that >> integer in fortran corresponds to int >> integer(kind=2) matches short >> but what would match long ? >> > > Unfortunately, it depends on your system. A C long is defined to be at > least as large as an int. Incidentally, an int is defined to be at > least as long as a short and a short is defined to be at least as long > as a char. So you *could* have char == short == int == long == 1 byte, > but no such system exists (I hope!). > > On pretty much all 32-bit systems I am aware of, int == long == 4 > bytes. On 64-bit systems, things get a little more fragmented. Most > UNIX-type 64-bit systems, long == 8 bytes. On Win64, however, long == > 4 bytes. This section of the Wikipedia article on 64-bit computing > gives more detail: > > http:// en.wikipedia.org/wiki/64-bit#64-bit_data_models > > Furthermore, it's not just the OS that determines the data model, but > also the options that are given to your compiler when building your > program *and also all of the libraries and system libraries* that it > uses. > > Fortunately, there's an easy way for you to tell: just ask numpy. > Python ints are C longs, and the default numpy int dtype matches > Python ints. On my 32-bit system: > > >>>> import numpy >>>> numpy.dtype(int).itemsize >>>> > 4 > > From dalcinl at gmail.com Thu Sep 11 16:41:02 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 11 Sep 2008 17:41:02 -0300 Subject: [Numpy-discussion] NumPy arrays that use memory allocated from other libraries or tools In-Reply-To: <9457e7c80809100055m1d0f754fv95f2a566ffcc2645@mail.gmail.com> References: <48C74F40.3090103@enthought.com> <9457e7c80809100055m1d0f754fv95f2a566ffcc2645@mail.gmail.com> Message-ID: On Wed, Sep 10, 2008 at 4:55 AM, St?fan van der Walt wrote: > 2008/9/10 Travis E. Oliphant : >> The post is >> >> http://blog.enthought.com/?p=62 > > Very cool post, thank you! I agree that it would be great to have > such a mechanism in NumPy. > And then builtin PyCObject's (using PyCObject_FromVoidPtrAndDesc) is your friend here, no need to implement a brand-new type :-). -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From Catherine.M.Moroney at jpl.nasa.gov Thu Sep 11 16:54:49 2008 From: Catherine.M.Moroney at jpl.nasa.gov (Catherine Moroney) Date: Thu, 11 Sep 2008 13:54:49 -0700 Subject: [Numpy-discussion] creating or truncating SD files in pyhdf Message-ID: <4B471B84-8E50-4304-B9B2-0902755DE5A0@jpl.nasa.gov> What is the correct way to create an SD file (if it doesn't already exist), or truncate and then rewrite an existing one? Does it matter which order the file-access options are set? For instance, here's my code: self.flhdf_stereo = SD.SD(flname_stereo, \ SD.SDC.TRUNC|SD.SDC.WRITE|SD.SDC.CREATE) and this fails if the file already exists, complaining that it can't open the path, even though the path is valid. This is the case if the file already exists. If the file doesn't exist, then it works just fine. Additionally, if I remove the "TRUNC" option then the file is opened without any problems. However, I need to truncate the file if it already exists. How do I do this? Catherine From robert.kern at gmail.com Thu Sep 11 17:00:44 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 11 Sep 2008 16:00:44 -0500 Subject: [Numpy-discussion] creating or truncating SD files in pyhdf In-Reply-To: <4B471B84-8E50-4304-B9B2-0902755DE5A0@jpl.nasa.gov> References: <4B471B84-8E50-4304-B9B2-0902755DE5A0@jpl.nasa.gov> Message-ID: <3d375d730809111400y7543d452n817c2372eda5928a@mail.gmail.com> On Thu, Sep 11, 2008 at 15:54, Catherine Moroney wrote: > What is the correct way to create an SD file (if it doesn't already > exist), > or truncate and then rewrite an existing one? I would suggest testing with os.path.exists() first, then deciding whether to use TRUNC or not based on that. > Does it matter which > order > the file-access options are set? No, the | operator is commutative. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From thrabe at gmail.com Thu Sep 11 17:04:57 2008 From: thrabe at gmail.com (Thomas Hrabe) Date: Thu, 11 Sep 2008 14:04:57 -0700 (PDT) Subject: [Numpy-discussion] Numpy array flags - BUG? Message-ID: <19444726.post@talk.nabble.com> Hello everyone, I must report odd behaviour of the numpy arrays regarding the flags set for each array object in C++. Please have a look at the following code: static PyObject* test(PyObject* self,PyObject* args){ int s[2]; s[0] = 1; s[1] = 1; char* value = (char*)PyMem_Malloc(2*sizeof(int)); PyObject* obj= PyArray_FromDimsAndData(1,s,NPY_INTLTR,(char*)value); PyArrayObject* array = (PyArrayObject*) obj; printf("%d\n",array->flags); array->flags=0; printf("%d\n",array->flags); return obj; } By creating my object with the code above, the flags of the object are not set right. The explicit setting of array->flags=0; would yield a print out of 0 for the next line, but when I look at the value from the interpreter, I would get a >>> a.flags.num 1 ??? It seems to me the flags are overridden and that C alignment is a hard coded default value. I need to set the Flags to 1286, which means (Fortran alignment) C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False Thats exactly what I want to achieve in my real code(similar as above, but more complex). But for that, however, I would get a >>> a.flags.num 1285 C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False even though I set the flags value in my C++ code to 1286 explicitly and a printf directive prints 1286 . Does anybody have a clue where the magic happens? Can I override the flags settings of an array in python and set them to some value? I tried it with the PyArray_UpdateFlags function, but it did not work, same results. Thank you in advance for your help, Thomas -- View this message in context: http://www.nabble.com/Numpy-array-flags---BUG--tp19444726p19444726.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From dblubaugh at belcan.com Thu Sep 11 17:42:17 2008 From: dblubaugh at belcan.com (Blubaugh, David A.) Date: Thu, 11 Sep 2008 17:42:17 -0400 Subject: [Numpy-discussion] F2py errors still Message-ID: <27CC3060AF71DA40A5DC85F7D5B70F380500CF83@AWMAIL04.belcan.com> To All, I am still not sure as to why this is not working. I believe it could be my windows environment variables. However, I have set these variables based on the following webpage: http://www.scipy.org/F2PY_Windows The error that I obtain is the following: C:\python25\python C:\python25\scripts\f2py.py -c --fcomplier=gnu95 --compiler=mingw32 -m foo foo.f90 running build running scons No module named msvccompiler in numpy.distutils; trying from distutils error: Python was built with Visual Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py. Why that I need to have Visual Studio 2003 is beyond me. Does anyone have any suggestions. David Blubaugh This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated. From robert.kern at gmail.com Thu Sep 11 17:46:22 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 11 Sep 2008 16:46:22 -0500 Subject: [Numpy-discussion] F2py errors still In-Reply-To: <27CC3060AF71DA40A5DC85F7D5B70F380500CF83@AWMAIL04.belcan.com> References: <27CC3060AF71DA40A5DC85F7D5B70F380500CF83@AWMAIL04.belcan.com> Message-ID: <3d375d730809111446g438fb6e2j61ffceeaef94f6b2@mail.gmail.com> On Thu, Sep 11, 2008 at 16:42, Blubaugh, David A. wrote: > To All, > > I am still not sure as to why this is not working. I believe it could > be my windows environment variables. However, I have set these > variables based on the following webpage: > > > http://www.scipy.org/F2PY_Windows > > > The error that I obtain is the following: > > > C:\python25\python C:\python25\scripts\f2py.py -c --fcomplier=gnu95 > --compiler=mingw32 -m foo foo.f90 > > running build > running scons > No module named msvccompiler in numpy.distutils; trying from distutils > error: Python was built with Visual Studio 2003; > extensions must be built with a compiler than can generate compatible > binaries. > Visual Studio 2003 was not found on this system. If you have Cygwin > installed, > you can try compiling with MingW32, by passing "-c mingw32" to setup.py. > > Why that I need to have Visual Studio 2003 is beyond me. Does anyone > have any suggestions. I believe this got fixed recently. Please try the latest release candidate of numpy 1.2. And please try to stay in one mailing list thread. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Catherine.M.Moroney at jpl.nasa.gov Thu Sep 11 18:01:02 2008 From: Catherine.M.Moroney at jpl.nasa.gov (Catherine Moroney) Date: Thu, 11 Sep 2008 15:01:02 -0700 Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 24, Issue 32 In-Reply-To: References: Message-ID: <88AE1EEC-101A-4706-AD00-F1201A2BE786@jpl.nasa.gov> I'm able to remove the file using python's "os" module, but I still get an error when trying to create the file using pyhdf, this time specifiying "SD.SDC.CREATE|SD.SDC.WRITE" on the file-create line. It's still complaining that it can't open the path, even though the specified filename is valid. Catherine On Sep 11, 2008, at 2:42 PM, numpy-discussion-request at scipy.org wrote: >> What is the correct way to create an SD file (if it doesn't already >> exist), >> or truncate and then rewrite an existing one? > > I would suggest testing with os.path.exists() first, then deciding > whether to use TRUNC or not based on that. > >> Does it matter which >> order >> the file-access options are set? > > No, the | operator is commutative. From dblubaugh at belcan.com Thu Sep 11 18:01:48 2008 From: dblubaugh at belcan.com (Blubaugh, David A.) Date: Thu, 11 Sep 2008 18:01:48 -0400 Subject: [Numpy-discussion] Has any one out there ever worked with the Rpyc, which is a remote process call for python Message-ID: <27CC3060AF71DA40A5DC85F7D5B70F380500CF94@AWMAIL04.belcan.com> To All, Has any one out there ever worked with the Rpyc, which is a remote process call for python? David Blubaugh This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated. From robert.kern at gmail.com Thu Sep 11 18:11:22 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 11 Sep 2008 17:11:22 -0500 Subject: [Numpy-discussion] Has any one out there ever worked with the Rpyc, which is a remote process call for python In-Reply-To: <27CC3060AF71DA40A5DC85F7D5B70F380500CF94@AWMAIL04.belcan.com> References: <27CC3060AF71DA40A5DC85F7D5B70F380500CF94@AWMAIL04.belcan.com> Message-ID: <3d375d730809111511v7ae020cs2725588757f2df5@mail.gmail.com> On Thu, Sep 11, 2008 at 17:01, Blubaugh, David A. wrote: > To All, > > > Has any one out there ever worked with the Rpyc, which is a remote > process call for python? Please do not cross-post. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Thu Sep 11 18:13:41 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 11 Sep 2008 17:13:41 -0500 Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 24, Issue 32 In-Reply-To: <88AE1EEC-101A-4706-AD00-F1201A2BE786@jpl.nasa.gov> References: <88AE1EEC-101A-4706-AD00-F1201A2BE786@jpl.nasa.gov> Message-ID: <3d375d730809111513n13c98bd2tffd0be392db0c4c5@mail.gmail.com> On Thu, Sep 11, 2008 at 17:01, Catherine Moroney wrote: > I'm able to remove the file using python's "os" module, but > I still get an error when trying to create the file using > pyhdf, this time specifiying "SD.SDC.CREATE|SD.SDC.WRITE" > on the file-create line. > > It's still complaining that it can't open the path, even though > the specified filename is valid. Can you show me the code and copy-and-paste the full traceback? Thanks. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From thrabe at gmail.com Thu Sep 11 19:08:08 2008 From: thrabe at gmail.com (Thomas Hrabe) Date: Thu, 11 Sep 2008 16:08:08 -0700 (PDT) Subject: [Numpy-discussion] Numpy array flags - BUG? In-Reply-To: <19444726.post@talk.nabble.com> References: <19444726.post@talk.nabble.com> Message-ID: <19446563.post@talk.nabble.com> I tried another approach, creating an array in python with >>>import numpy; >>>a = numpy.zeros((2,2),order="C"); >>>a.flags.num 1285 and setting the flags within C++ to 1286. The value remains the same (1285) after setting it to 1286 in embedded C. PyArg_ParseTuple(args, "O!",&PyArray_Type, &array) printf("%d\n",array->flags); //prints 1285 array->flags = 1286; printf("%d\n",array->flags); //prints 1286 return Py_None; >>>a.flags.num 1285 -- View this message in context: http://www.nabble.com/Numpy-array-flags---BUG--tp19444726p19446563.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From thrabe at gmail.com Thu Sep 11 19:21:41 2008 From: thrabe at gmail.com (Thomas Hrabe) Date: Thu, 11 Sep 2008 16:21:41 -0700 (PDT) Subject: [Numpy-discussion] C to Fortran Conversion Message-ID: <19446690.post@talk.nabble.com> Hi all, this post might not really belong into this forum, but as the functions I am looking for are member of numpy, I thought I might drop a message at this point. I am looking for some functions/ a library converting C aligned arrays to Fortran, due to the flags problem I mentioned recently in this forum. Do you know the C function that is used for the conversion by numpy.asfortranarray, for instance? Is it callable from C code,too? Is it based on some external library? What I have in mind is a function header like: void CToFortran(void* cArray /*source*/,void* fArray /*target*/,int dimensions,int* sizes) where the source array cArray would be copied in fortran order to fArray according to dimensions and sizes. Anybody having an idea? Thomas -- View this message in context: http://www.nabble.com/C-to-Fortran-Conversion-tp19446690p19446690.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From oliphant at enthought.com Thu Sep 11 20:35:03 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 11 Sep 2008 19:35:03 -0500 Subject: [Numpy-discussion] Numpy array flags - BUG? In-Reply-To: <19444726.post@talk.nabble.com> References: <19444726.post@talk.nabble.com> Message-ID: <48C9B937.4010601@enthought.com> Thomas Hrabe wrote: > Hello everyone, > > I must report odd behaviour of the numpy arrays regarding the flags set for > each array object in C++. > Please have a look at the following code: > > static PyObject* test(PyObject* self,PyObject* args){ > > int s[2]; > s[0] = 1; > s[1] = 1; > > char* value = (char*)PyMem_Malloc(2*sizeof(int)); > > PyObject* obj= PyArray_FromDimsAndData(1,s,NPY_INTLTR,(char*)value); > PyArrayObject* array = (PyArrayObject*) obj; > printf("%d\n",array->flags); > array->flags=0; > printf("%d\n",array->flags); > return obj; > } > You are using an old API and a new feature. If you want to be able to create a Fortran array from data you need to use PyArray_NewFromDescr which allows setting flags. -Travis From alan at ajackson.org Thu Sep 11 21:03:53 2008 From: alan at ajackson.org (Alan Jackson) Date: Thu, 11 Sep 2008 20:03:53 -0500 Subject: [Numpy-discussion] array gymnastics Message-ID: <20080911200353.12aa4853@ajackson.org> There has got to be a simple way to do this, but I'm just not seeing it. >>> a = array([[1,2,3,4,5,6], [7,8,9,10,11,12]]) >>> b = array([21,22,23,24,25,26]) What I want to end up with is : c = array([[1,7,21], [2,8,22], ...... [6,12,26]]) -- ----------------------------------------------------------------------- | Alan K. Jackson | To see a World in a Grain of Sand | | alan at ajackson.org | And a Heaven in a Wild Flower, | | www.ajackson.org | Hold Infinity in the palm of your hand | | Houston, Texas | And Eternity in an hour. - Blake | ----------------------------------------------------------------------- From bpederse at gmail.com Thu Sep 11 21:19:22 2008 From: bpederse at gmail.com (Brent Pedersen) Date: Thu, 11 Sep 2008 18:19:22 -0700 Subject: [Numpy-discussion] array gymnastics In-Reply-To: <20080911200353.12aa4853@ajackson.org> References: <20080911200353.12aa4853@ajackson.org> Message-ID: On Thu, Sep 11, 2008 at 6:03 PM, Alan Jackson wrote: > There has got to be a simple way to do this, but I'm just not seeing it. > >>>> a = array([[1,2,3,4,5,6], > [7,8,9,10,11,12]]) >>>> b = array([21,22,23,24,25,26]) > > What I want to end up with is : > > c = array([[1,7,21], > [2,8,22], > ...... > [6,12,26]]) > > > -- > ----------------------------------------------------------------------- > | Alan K. Jackson | To see a World in a Grain of Sand | > | alan at ajackson.org | And a Heaven in a Wild Flower, | > | www.ajackson.org | Hold Infinity in the palm of your hand | > | Houston, Texas | And Eternity in an hour. - Blake | > ----------------------------------------------------------------------- > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > hi, i think >>> numpy.column_stack((a[0], a[1], b)) does what you want. From nadavh at visionsense.com Thu Sep 11 21:57:46 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Fri, 12 Sep 2008 04:57:46 +0300 Subject: [Numpy-discussion] array gymnastics References: <20080911200353.12aa4853@ajackson.org> Message-ID: <710F2847B0018641891D9A216027636029C270@ex3.envision.co.il> or hstack((a.T, b[:,None])) Nadav. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Brent Pedersen ????: ? 12-??????-08 04:19 ??: Discussion of Numerical Python ????: Re: [Numpy-discussion] array gymnastics On Thu, Sep 11, 2008 at 6:03 PM, Alan Jackson wrote: > There has got to be a simple way to do this, but I'm just not seeing it. > >>>> a = array([[1,2,3,4,5,6], > [7,8,9,10,11,12]]) >>>> b = array([21,22,23,24,25,26]) > > What I want to end up with is : > > c = array([[1,7,21], > [2,8,22], > ...... > [6,12,26]]) > > > -- > ----------------------------------------------------------------------- > | Alan K. Jackson | To see a World in a Grain of Sand | > | alan at ajackson.org | And a Heaven in a Wild Flower, | > | www.ajackson.org | Hold Infinity in the palm of your hand | > | Houston, Texas | And Eternity in an hour. - Blake | > ----------------------------------------------------------------------- > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > hi, i think >>> numpy.column_stack((a[0], a[1], b)) does what you want. _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3308 bytes Desc: not available URL: From alan at ajackson.org Thu Sep 11 22:46:35 2008 From: alan at ajackson.org (Alan Jackson) Date: Thu, 11 Sep 2008 21:46:35 -0500 Subject: [Numpy-discussion] array gymnastics In-Reply-To: <710F2847B0018641891D9A216027636029C270@ex3.envision.co.il> References: <20080911200353.12aa4853@ajackson.org> <710F2847B0018641891D9A216027636029C270@ex3.envision.co.il> Message-ID: <20080911214635.5ebcac92@ajackson.org> Thanks! I think I figured that out about a year ago, and I couldn't remember what I had done. On Fri, 12 Sep 2008 04:57:46 +0300 "Nadav Horesh" wrote: > or > hstack((a.T, b[:,None])) > > Nadav. > > > -----????? ??????----- > ???: numpy-discussion-bounces at scipy.org ??? Brent Pedersen > ????: ? 12-??????-08 04:19 > ??: Discussion of Numerical Python > ????: Re: [Numpy-discussion] array gymnastics > > On Thu, Sep 11, 2008 at 6:03 PM, Alan Jackson wrote: > > There has got to be a simple way to do this, but I'm just not seeing it. > > > >>>> a = array([[1,2,3,4,5,6], > > [7,8,9,10,11,12]]) > >>>> b = array([21,22,23,24,25,26]) > > > > What I want to end up with is : > > > > c = array([[1,7,21], > > [2,8,22], > > ...... > > [6,12,26]]) > > > > > > -- > > ----------------------------------------------------------------------- > > | Alan K. Jackson | To see a World in a Grain of Sand | > > | alan at ajackson.org | And a Heaven in a Wild Flower, | > > | www.ajackson.org | Hold Infinity in the palm of your hand | > > | Houston, Texas | And Eternity in an hour. - Blake | > > ----------------------------------------------------------------------- > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > hi, i think > > >>> numpy.column_stack((a[0], a[1], b)) > > does what you want. > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- ----------------------------------------------------------------------- | Alan K. Jackson | To see a World in a Grain of Sand | | alan at ajackson.org | And a Heaven in a Wild Flower, | | www.ajackson.org | Hold Infinity in the palm of your hand | | Houston, Texas | And Eternity in an hour. - Blake | ----------------------------------------------------------------------- From pricewhelan at mac.com Fri Sep 12 02:20:15 2008 From: pricewhelan at mac.com (Adrian Price-Whelan) Date: Fri, 12 Sep 2008 02:20:15 -0400 Subject: [Numpy-discussion] GCC Error (?) Message-ID: <1AA95660-4EBC-4690-8056-7109040A2C35@mac.com> Hey - Trying to reinstall numpy and scipy on my Macbook Pro running Mac OS X 10.5.4 but now both of them give me nasty errors akin to: Could not locate executable f95 customize AbsoftFCompiler Could not locate executable f90 Could not locate executable f77 customize IBMFCompiler Could not locate executable xlf90 Could not locate executable xlf customize IntelFCompiler Could not locate executable ifort Could not locate executable ifc customize GnuFCompiler Could not locate executable g77 I've tried reinstalling the developer tools, and on a side note, I tried installing scipy through FINK and fink couldn't find the package...? fink install scipy-py25 Password: Information about 2516 packages read in 2 seconds. Failed: no package found for specification 'scipy-py25'! weird...anyways, i need to get this compiler issue settled and would appreciate any and all help! Thanks, Adrian From robert.kern at gmail.com Fri Sep 12 03:24:11 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 12 Sep 2008 02:24:11 -0500 Subject: [Numpy-discussion] GCC Error (?) In-Reply-To: <1AA95660-4EBC-4690-8056-7109040A2C35@mac.com> References: <1AA95660-4EBC-4690-8056-7109040A2C35@mac.com> Message-ID: <3d375d730809120024s28a90e77sd0f3f9f64d22cb4@mail.gmail.com> On Fri, Sep 12, 2008 at 01:20, Adrian Price-Whelan wrote: > Hey - > > Trying to reinstall numpy and scipy on my Macbook Pro running Mac OS X > 10.5.4 but now both of them give me nasty errors akin to: > Could not locate executable f95 > customize AbsoftFCompiler > Could not locate executable f90 > Could not locate executable f77 > customize IBMFCompiler > Could not locate executable xlf90 > Could not locate executable xlf > customize IntelFCompiler > Could not locate executable ifort > Could not locate executable ifc > customize GnuFCompiler > Could not locate executable g77 Do you have a Fortran compiler installed? It doesn't come from Apple. I recommend installing this one: http://r.research.att.com/gfortran-4.2.3.dmg -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Fri Sep 12 03:25:49 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 12 Sep 2008 02:25:49 -0500 Subject: [Numpy-discussion] GCC Error (?) In-Reply-To: <1AA95660-4EBC-4690-8056-7109040A2C35@mac.com> References: <1AA95660-4EBC-4690-8056-7109040A2C35@mac.com> Message-ID: <3d375d730809120025i21d6565am15c61508469ee3db@mail.gmail.com> On Fri, Sep 12, 2008 at 01:20, Adrian Price-Whelan wrote: > Hey - > > Trying to reinstall numpy and scipy on my Macbook Pro running Mac OS X > 10.5.4 but now both of them give me nasty errors akin to: > Could not locate executable f95 > customize AbsoftFCompiler > Could not locate executable f90 > Could not locate executable f77 > customize IBMFCompiler > Could not locate executable xlf90 > Could not locate executable xlf > customize IntelFCompiler > Could not locate executable ifort > Could not locate executable ifc > customize GnuFCompiler > Could not locate executable g77 Ah, you're having a problem with numpy, too. Note that in the case of numpy, you'll see these warnings, but they shouldn't trouble you. What problem are you actually having with building numpy? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From nwagner at iam.uni-stuttgart.de Fri Sep 12 14:07:57 2008 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Fri, 12 Sep 2008 20:07:57 +0200 Subject: [Numpy-discussion] f2py usage Message-ID: Hi all, How can I use f2py ? I followed the instructions given at http://www.scipy.org/F2py To check if F2PY is installed correctly, I run /usr/bin/f2py Traceback (most recent call last): File "/usr/bin/f2py", line 20, in ? from numpy.f2py import main ImportError: No module named numpy.f2py Any pointer ? nwagner at linux:~> /usr/bin/python Python 2.4 (#1, Oct 13 2006, 17:13:31) [GCC 3.3.5 20050117 (prerelease) (SUSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from numpy.f2py import main >>> Nils From pav at iki.fi Fri Sep 12 14:21:46 2008 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 12 Sep 2008 18:21:46 +0000 (UTC) Subject: [Numpy-discussion] f2py usage References: Message-ID: Fri, 12 Sep 2008 20:07:57 +0200, Nils Wagner wrote: > How can I use f2py ? > > I followed the instructions given at > > http://www.scipy.org/F2py > > To check if F2PY is installed correctly, I run > > /usr/bin/f2py > Traceback (most recent call last): > File "/usr/bin/f2py", line 20, in ? > from numpy.f2py import main > ImportError: No module named numpy.f2py > > Any pointer ? > > nwagner at linux:~> /usr/bin/python > Python 2.4 (#1, Oct 13 2006, 17:13:31) [GCC 3.3.5 20050117 (prerelease) > (SUSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" > for more information. >>>> from numpy.f2py import main >>>> >>>> > Nils Do you have different versions of Python installed? Check that the first line of /usr/bin/f2py is something like #!/usr/bin/python Or type f2py2.4 instead of f2py. -- Pauli Virtanen From thrabe at gmail.com Fri Sep 12 14:47:03 2008 From: thrabe at gmail.com (Thomas Hrabe) Date: Fri, 12 Sep 2008 11:47:03 -0700 (PDT) Subject: [Numpy-discussion] Numpy array flags - BUG? In-Reply-To: <48C9B937.4010601@enthought.com> References: <19444726.post@talk.nabble.com> <48C9B937.4010601@enthought.com> Message-ID: <19461959.post@talk.nabble.com> Thank you, first tests work. Thomas Travis E. Oliphant wrote: > > Thomas Hrabe wrote: >> Hello everyone, >> >> I must report odd behaviour of the numpy arrays regarding the flags set >> for >> each array object in C++. >> Please have a look at the following code: >> >> static PyObject* test(PyObject* self,PyObject* args){ >> >> int s[2]; >> s[0] = 1; >> s[1] = 1; >> >> char* value = (char*)PyMem_Malloc(2*sizeof(int)); >> >> PyObject* obj= >> PyArray_FromDimsAndData(1,s,NPY_INTLTR,(char*)value); >> PyArrayObject* array = (PyArrayObject*) obj; >> printf("%d\n",array->flags); >> array->flags=0; >> printf("%d\n",array->flags); >> return obj; >> } >> > You are using an old API and a new feature. If you want to be able to > create a Fortran array from data you need to use > > PyArray_NewFromDescr > > which allows setting flags. > > -Travis > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- View this message in context: http://www.nabble.com/Numpy-array-flags---BUG--tp19444726p19461959.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From dblubaugh at belcan.com Fri Sep 12 15:34:06 2008 From: dblubaugh at belcan.com (Blubaugh, David A.) Date: Fri, 12 Sep 2008 15:34:06 -0400 Subject: [Numpy-discussion] F2py errors still In-Reply-To: References: Message-ID: <27CC3060AF71DA40A5DC85F7D5B70F380500D643@AWMAIL04.belcan.com> Robert, In response to your statement I cannot find Numpy 1.2 listed. Where is it??? David Blubaugh > I believe this got fixed recently. Please try the latest release candidate of numpy 1.2. > And please try to stay in one mailing list thread. This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated. From millman at berkeley.edu Fri Sep 12 15:42:25 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Fri, 12 Sep 2008 12:42:25 -0700 Subject: [Numpy-discussion] F2py errors still In-Reply-To: <27CC3060AF71DA40A5DC85F7D5B70F380500D643@AWMAIL04.belcan.com> References: <27CC3060AF71DA40A5DC85F7D5B70F380500D643@AWMAIL04.belcan.com> Message-ID: On Fri, Sep 12, 2008 at 12:34 PM, Blubaugh, David A. wrote: > In response to your statement I cannot find Numpy 1.2 listed. Where is > it??? I haven't announced rc2 yet, since I am still waiting to get the binaries built (should be out later today). But since you are building yourself, here are the links: The 1.2.0rc2 svn tag: http://svn.scipy.org/svn/numpy/tags/1.2.0rc2 The source tarball: https://cirl.berkeley.edu/numpy/numpy-1.2.0rc2.tar.gz -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From stefan at sun.ac.za Sat Sep 13 02:34:33 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sat, 13 Sep 2008 08:34:33 +0200 Subject: [Numpy-discussion] patch for new mgrid / ogrid functionality In-Reply-To: <1218812064.7194.17.camel@localhost> References: <1218812064.7194.17.camel@localhost> Message-ID: <9457e7c80809122334q2c305be4r977f6803f029b193@mail.gmail.com> Hey all, David Kaplan implemented a generalised ogrid/mgrid, and would like to have some feedback. See http://projects.scipy.org/pipermail/numpy-discussion/2008-August/036691.html for his original email and attachment. If no one objects to such functionality, I shall review the patch and apply it. Cheers St?fan 2008/8/15 David M. Kaplan : > Hi, > > A while back, I sent some changes to index_tricks.py that would allow > mgrid and ogrid to mesh things other than slices. For example: > >>>> mgrid[['a','b'],[float,int],:3] > [array([[['a', 'a', 'a'], > ['a', 'a', 'a']], > > [['b', 'b', 'b'], > ['b', 'b', 'b']]], > dtype='|S1'), array([[[, , ], > [, , ]], > > [[, , ], > [, , ]]], dtype=object), array([[[0, 1, 2], > [0, 1, 2]], > > [[0, 1, 2], > [0, 1, 2]]])] > > At the time, there wasn't much follow-up, but I am hoping that there is > still interest in this functionality, as I have gone ahead and finished > the patch including documentation changes and updates to > test_index_tricks.py. Attached is a patch set to the latest subversion > of the numpy trunk. I don't think I am allowed to commit the changes > myself - correct me if I am wrong. > > This functionality seems like a nice addition to me as it allows one to > mesh things that are not uniformly spaced and potentially not even > numbers. The changes don't affect functionality that existed previously > except for one minor change - instead of returning a numpy array of > arrays, mgrid/ogrid now return a list of arrays. However, this is > unlikely to be a problem as the majority of users generally unpack the > results of mgrid/ogrid so that each matrix can be used individually. > > Comments welcome. > > Cheers, > David > > -- > ********************************** > David M. Kaplan > Charge de Recherche 1 > Institut de Recherche pour le Developpement > Centre de Recherche Halieutique Mediterraneenne et Tropicale > av. Jean Monnet > B.P. 171 > 34203 Sete cedex > France > > Phone: +33 (0)4 99 57 32 27 > Fax: +33 (0)4 99 57 32 95 > http://www.ur097.ird.fr/team/dkaplan/index.html > ********************************** From oliphant at enthought.com Sat Sep 13 10:29:56 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sat, 13 Sep 2008 09:29:56 -0500 Subject: [Numpy-discussion] patch for new mgrid / ogrid functionality In-Reply-To: <9457e7c80809122334q2c305be4r977f6803f029b193@mail.gmail.com> References: <1218812064.7194.17.camel@localhost> <9457e7c80809122334q2c305be4r977f6803f029b193@mail.gmail.com> Message-ID: <48CBCE64.1020207@enthought.com> St?fan van der Walt wrote: > Hey all, > > David Kaplan implemented a generalised ogrid/mgrid, and would like to > have some feedback. See > > http://projects.scipy.org/pipermail/numpy-discussion/2008-August/036691.html > > for his original email and attachment. > > If no one objects to such functionality, I shall review the patch and apply it. > Generally, it is O.K. except that he changes the output from a numpy array to a list of numpy arrays. This is a small change, but it is an API change. I'm not sure why he found it necessary to do that, but it would be easier to apply the patch if it didn't change the API. -Travis From wnbell at gmail.com Sat Sep 13 12:32:20 2008 From: wnbell at gmail.com (Nathan Bell) Date: Sat, 13 Sep 2008 12:32:20 -0400 Subject: [Numpy-discussion] Dealing with types in extension modules In-Reply-To: <48C897BF.1020806@mit.edu> References: <48C86111.1040608@mit.edu> <48C880A0.1030804@enthought.com> <48C892B9.4060004@mit.edu> <48C8959D.8040206@enthought.com> <48C897BF.1020806@mit.edu> Message-ID: On Wed, Sep 10, 2008 at 11:59 PM, Lane Brooks wrote: > > When writing an numpy extension module, what is the preferred way to > deal with the all the possible types an ndarray can have? > > I have some data processing functions I need to implement and they need > to be generic and work for all the possible numerical dtypes. I do not > want to have to re-implement the same C-code for all the possible types, > so the way I approached it was to use a C++ template function to > implement the processing. Then I have a dispatching function that > checks the type of the input ndarray and calls the correct template. Is > there a better way? > In scipy.sparse there is a 'sparsetools' extension module that uses C++ templates to support 14 (or so) numpy types. In sparsetools, SWIG typemaps are used to dispatch the correct template based on the numpy data type. For instance, a typemap decides to call foo if the array has type PyArray_FLOAT. SWIG can be a little overwhelming, but it automates dispatching and can even do automatic upcasting for you. For instance, if your C++ function is foo(double, double) and you pass it (float,int) then SWIG can upcast both arguments to double before passing them into foo(). Also, you can force arrays to be C-contiguous (as opposed to strided or Fortran order) and have the correct endianness. Basically, you can use SWIG to tame the input numpy arrays. The sparsetools module is here: http://projects.scipy.org/scipy/scipy/browser/trunk/scipy/sparse/sparsetools The file numpy.i does most of the heavy lifiting. For each sparse matrix format, for instance 'csr', there is a separate SWIG file ('csr.i') that instantiates each function in the corresponding header file ('csr.h'). The output of 'swig -c++ -python csr.i' is the actual extension module ('csr_wrap.cxx'). If you look at one of these (massive!) files you can see how SWIG dispatches the appropriate function. Also, in order to support complex types (e.g. complex128), there is a wrapper in complex_ops.h that overloads the standard arithmetic operators for complex numbers. Then complex types are handled exactly as above. http://projects.scipy.org/scipy/scipy/browser/trunk/scipy/sparse/sparsetools/complex_ops.h Depending on your problem, SWIG may be overkill. OTOH you may be able to get everything to need from the sparsetools source code. Feel free to pillage it as you require :) Should you go the SWIG path, I can help explain some of the more cryptic parts. -- Nathan Bell wnbell at gmail.com http://graphics.cs.uiuc.edu/~wnbell/ From stefan at sun.ac.za Sat Sep 13 14:49:44 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sat, 13 Sep 2008 20:49:44 +0200 Subject: [Numpy-discussion] patch for new mgrid / ogrid functionality In-Reply-To: <48CBCE64.1020207@enthought.com> References: <1218812064.7194.17.camel@localhost> <9457e7c80809122334q2c305be4r977f6803f029b193@mail.gmail.com> <48CBCE64.1020207@enthought.com> Message-ID: <9457e7c80809131149u5ed7c7efw80b70828539107b7@mail.gmail.com> 2008/9/13 Travis E. Oliphant : > Generally, it is O.K. except that he changes the output from a numpy > array to a list of numpy arrays. This is a small change, but it is an > API change. I'm not sure why he found it necessary to do that, but it > would be easier to apply the patch if it didn't change the API. The example he gave illustrated having a mixed bag of types in one grid. You can do that with two arrays, but not one, unless you use object-arrays for everything. Maybe that makes sense. Cheers St?fan From oliphant at enthought.com Sat Sep 13 15:03:48 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sat, 13 Sep 2008 14:03:48 -0500 Subject: [Numpy-discussion] patch for new mgrid / ogrid functionality In-Reply-To: <9457e7c80809131149u5ed7c7efw80b70828539107b7@mail.gmail.com> References: <1218812064.7194.17.camel@localhost> <9457e7c80809122334q2c305be4r977f6803f029b193@mail.gmail.com> <48CBCE64.1020207@enthought.com> <9457e7c80809131149u5ed7c7efw80b70828539107b7@mail.gmail.com> Message-ID: <48CC0E94.60702@enthought.com> St?fan van der Walt wrote: > 2008/9/13 Travis E. Oliphant : > >> Generally, it is O.K. except that he changes the output from a numpy >> array to a list of numpy arrays. This is a small change, but it is an >> API change. I'm not sure why he found it necessary to do that, but it >> would be easier to apply the patch if it didn't change the API. >> > > The example he gave illustrated having a mixed bag of types in one > grid. You can do that with two arrays, but not one, unless you use > object-arrays for everything. Maybe that makes sense. > > I don't see why you have to use object-arrays for everything, but obviously you would for the mixed types. -Travis From millman at berkeley.edu Sat Sep 13 19:00:35 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Sat, 13 Sep 2008 16:00:35 -0700 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- Message-ID: Hello, The 1.2.0rc2 is now available: http://svn.scipy.org/svn/numpy/tags/1.2.0rc2 The source tarball is here: https://cirl.berkeley.edu/numpy/numpy-1.2.0rc2.tar.gz Here is the universal Mac binary: https://cirl.berkeley.edu/numpy/numpy-1.2.0rc2-py2.5-macosx10.5.dmg Here are the Window's binaries: http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.2.0rc2-win32-superpack-python2.4.exe http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.2.0rc2-win32-superpack-python2.5.exe Here are the release notes: http://scipy.org/scipy/numpy/milestone/1.2.0 Please test this release ASAP and let us know if there are any problems. If there are no show stoppers, this will become the 1.2.0 release. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From David.Kaplan at ird.fr Sun Sep 14 03:49:11 2008 From: David.Kaplan at ird.fr (David M. Kaplan) Date: Sun, 14 Sep 2008 09:49:11 +0200 Subject: [Numpy-discussion] patch for new mgrid / ogrid functionality In-Reply-To: References: Message-ID: <1221378551.7007.13.camel@localhost> Hi, On Sat, 2008-09-13 at 12:00 -0500, numpy-discussion-request at scipy.org wrote: > St?fan van der Walt wrote: > > Hey all, > > > > David Kaplan implemented a generalised ogrid/mgrid, and would like > to > > have some feedback. See > > > > > http://projects.scipy.org/pipermail/numpy-discussion/2008-August/036691.html > > > > for his original email and attachment. > > > > If no one objects to such functionality, I shall review the patch > and apply it. > > > > Generally, it is O.K. except that he changes the output from a numpy > array to a list of numpy arrays. This is a small change, but it is > an > API change. I'm not sure why he found it necessary to do that, but > it > would be easier to apply the patch if it didn't change the API. > My reason for returning a list is that with the new functionality it was more efficient to use a list (note that ogrid always returns a list). It is easy to force the list to an array before returning (if not sparse), but if I remember correctly that had a performance hit. Also, the vaste majority of users probably unpack the array immediately making conversion to an array unnecessary. I leave it up to you which is best. I can add the line to convert to array if desired. Cheers, David > -Travis > -- ********************************** David M. Kaplan Charge de Recherche 1 Institut de Recherche pour le Developpement Centre de Recherche Halieutique Mediterraneenne et Tropicale av. Jean Monnet B.P. 171 34203 Sete cedex France Phone: +33 (0)4 99 57 32 27 Fax: +33 (0)4 99 57 32 95 http://www.ur097.ird.fr/team/dkaplan/index.html ********************************** From talumbau at llnl.gov Sun Sep 14 04:09:43 2008 From: talumbau at llnl.gov (T.J. Alumbaugh) Date: Sun, 14 Sep 2008 01:09:43 -0700 Subject: [Numpy-discussion] making arithmetic operations compatible with ndarray Message-ID: <07050703-96E2-4B82-AD23-1128CB447EFB@llnl.gov> Hi, I'm wondering if someone can give me a tip on how to modify my Python type to work with certain arithmetic operations with ndarray. The actual operation I would like is as follows: ndarrayRef += myObj where myObj is a reference to an object of a user defined Python type (with array-type data as one of its members). I need a solution that does not involve my type inheriting from the ndarray class. A fast solution would be great, but is not required, so a pure Python solution would be fine. My user defined type does use the Numpy C API to expose its data as an ndarray (with PyArray_SimpleNewFromData), but that involves a function call to return that reference. I'm hoping there is a solution that is along the lines of 'define these three member functions on your type and the object will be numpy-ish enough to do the job.' Any ideas? Thanks, -T.J. Alumbaugh talumbau at llnl.gov From robince at gmail.com Sun Sep 14 13:01:28 2008 From: robince at gmail.com (Robin) Date: Sun, 14 Sep 2008 18:01:28 +0100 Subject: [Numpy-discussion] funny matrix product calculation Message-ID: Hi, I am trying to compute the following (with summation convention): A_{ij} B_{jm} x_{j} where A and B are non-square arrays (in fact B=A^T) and x is a vector. I'm actually not sure if it's valid to have 3 summation indices as above - but I'm pretty sure that's what I want... So it's kind of matric multiplication, but I want another vector dot'ed into the sum. How can I implement this efficiently in numpy/scipy? I'm having trouble breaking it down to vectorised operations. I suppose I could try a loop in Cython but I was hoping there might be a tricky numpy solution. Thanks Robin From nadavh at visionsense.com Sun Sep 14 13:50:53 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Sun, 14 Sep 2008 20:50:53 +0300 Subject: [Numpy-discussion] funny matrix product calculation References: Message-ID: <710F2847B0018641891D9A216027636029C276@ex3.envision.co.il> Not a supper efficient solution, but mat sill may work: dot(A, B*x[:,None]) or dot(A*x, B) Nadav -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Robin ????: ? 14-??????-08 20:01 ??: Discussion of Numerical Python ????: [Numpy-discussion] funny matrix product calculation Hi, I am trying to compute the following (with summation convention): A_{ij} B_{jm} x_{j} where A and B are non-square arrays (in fact B=A^T) and x is a vector. I'm actually not sure if it's valid to have 3 summation indices as above - but I'm pretty sure that's what I want... So it's kind of matric multiplication, but I want another vector dot'ed into the sum. How can I implement this efficiently in numpy/scipy? I'm having trouble breaking it down to vectorised operations. I suppose I could try a loop in Cython but I was hoping there might be a tricky numpy solution. Thanks Robin _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3296 bytes Desc: not available URL: From silva at lma.cnrs-mrs.fr Sun Sep 14 17:17:56 2008 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Sun, 14 Sep 2008 23:17:56 +0200 Subject: [Numpy-discussion] funny matrix product calculation In-Reply-To: References: Message-ID: <1221427076.6915.5.camel@localhost.localdomain> Le dimanche 14 septembre 2008 ? 18:01 +0100, Robin a ?crit : > Hi, > > I am trying to compute the following (with summation convention): > > A_{ij} B_{jm} x_{j} > > where A and B are non-square arrays (in fact B=A^T) and x is a vector. > I'm actually not sure if it's valid to have 3 summation indices as > above - but I'm pretty sure that's what I want... So it's kind of > matric multiplication, but I want another vector dot'ed into the sum. What about considering a square diagonal matrix having vector x on its main diagonal ? Calling X such a matrix: X = numpy.matrix(numpy.diag(x, k=0)) it seems that computing matrix multiplication A*X*B produces the requirements, doesn't it? -- Fabrice Silva From dalke at dalkescientific.com Sun Sep 14 19:21:05 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Mon, 15 Sep 2008 01:21:05 +0200 Subject: [Numpy-discussion] isnan and co: cleaning up In-Reply-To: References: <48C0BE3F.1030306@ar.media.kyoto-u.ac.jp> Message-ID: <0C000502-8983-4289-8CF4-83CAE5A39E13@dalkescientific.com> The following seems relevant in this thread. On a Mac, OS X 10.4, I just rebuilt from SVN and got compile options: '-Inumpy/core/src -Inumpy/core/include -I/Library/ Frameworks/Python.framework/Versions/2.5/include/python2.5 -c' gcc: _configtest.c _configtest.c: In function 'main': _configtest.c:4: error: 'isnan' undeclared (first use in this function) _configtest.c:4: error: (Each undeclared identifier is reported only once _configtest.c:4: error: for each function it appears in.) _configtest.c: In function 'main': _configtest.c:4: error: 'isnan' undeclared (first use in this function) _configtest.c:4: error: (Each undeclared identifier is reported only once _configtest.c:4: error: for each function it appears in.) lipo: can't figure out the architecture type of: /var/tmp//ccTy1pzZ.out _configtest.c: In function 'main': _configtest.c:4: error: 'isnan' undeclared (first use in this function) _configtest.c:4: error: (Each undeclared identifier is reported only once _configtest.c:4: error: for each function it appears in.) _configtest.c: In function 'main': _configtest.c:4: error: 'isnan' undeclared (first use in this function) _configtest.c:4: error: (Each undeclared identifier is reported only once _configtest.c:4: error: for each function it appears in.) lipo: can't figure out the architecture type of: /var/tmp//ccTy1pzZ.out failure. removing: _configtest.c _configtest.o followed by a failure for "isinf". It kept on going and finished the compile cleanly, so I don't know if this is an informational warning or an indication of a suppressed error that shouldn't have been suppressed. Should that error message be there? In my understanding, errors are bad. I tried looking for where that test as being done, but it seems to be somewhere in SCons (in numpy/core/SConscript) and I figured it would be easier than ask than to root around. Andrew dalke at dalkescientific.com From david at ar.media.kyoto-u.ac.jp Sun Sep 14 23:45:43 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 15 Sep 2008 12:45:43 +0900 Subject: [Numpy-discussion] isnan and co: cleaning up In-Reply-To: <0C000502-8983-4289-8CF4-83CAE5A39E13@dalkescientific.com> References: <48C0BE3F.1030306@ar.media.kyoto-u.ac.jp> <0C000502-8983-4289-8CF4-83CAE5A39E13@dalkescientific.com> Message-ID: <48CDDA67.3080505@ar.media.kyoto-u.ac.jp> Andrew Dalke wrote: > The following seems relevant in this thread. > > On a Mac, OS X 10.4, I just rebuilt from SVN and got > > compile options: '-Inumpy/core/src -Inumpy/core/include -I/Library/ > Frameworks/Python.framework/Versions/2.5/include/python2.5 -c' > gcc: _configtest.c > _configtest.c: In function 'main': > _configtest.c:4: error: 'isnan' undeclared (first use in this function) > _configtest.c:4: error: (Each undeclared identifier is reported only > once > _configtest.c:4: error: for each function it appears in.) > _configtest.c: In function 'main': > _configtest.c:4: error: 'isnan' undeclared (first use in this function) > _configtest.c:4: error: (Each undeclared identifier is reported only > once > _configtest.c:4: error: for each function it appears in.) > lipo: can't figure out the architecture type of: /var/tmp//ccTy1pzZ.out > _configtest.c: In function 'main': > _configtest.c:4: error: 'isnan' undeclared (first use in this function) > _configtest.c:4: error: (Each undeclared identifier is reported only > once > _configtest.c:4: error: for each function it appears in.) > _configtest.c: In function 'main': > _configtest.c:4: error: 'isnan' undeclared (first use in this function) > _configtest.c:4: error: (Each undeclared identifier is reported only > once > _configtest.c:4: error: for each function it appears in.) > lipo: can't figure out the architecture type of: /var/tmp//ccTy1pzZ.out > failure. > removing: _configtest.c _configtest.o > > followed by a failure for "isinf". > > It kept on going and finished the compile cleanly, so I don't know if > this is an informational warning or an indication of a suppressed > error that shouldn't have been suppressed. Should that error message > be there? In my understanding, errors are bad. It is not an error for the build, but an error at the configuration stage. To get some informations about the platform, we do compile some code snippets, and do our configuration depending on whether they fail or not. We could log those in a different location than the actual build, but I am not sure anyone wants to spend some time on it. > > I tried looking for where that test as being done, but it seems to be > somewhere in SCons (in numpy/core/SConscript) and I figured it would > be easier than ask than to root around. It has nothing to do with scons, since the _configtest.c are generated by the distutils build. With scons, the errors during configuration stage logged in a separate file (which is one advantage of scons, actually, since it does the work for us). cheers, David From David.Kaplan at ird.fr Mon Sep 15 03:56:53 2008 From: David.Kaplan at ird.fr (David M. Kaplan) Date: Mon, 15 Sep 2008 09:56:53 +0200 Subject: [Numpy-discussion] patch for new mgrid / ogrid functionality In-Reply-To: References: Message-ID: <1221465413.7117.9.camel@localhost> Hi, Ahh, yes, I forgot about that problem - if you have mixed types forcing to an array doesn't make much sense and can have undesired results such as forcing numbers to strings unless you explicitly force to an object array. As this would be undesirable in most non-mixed cases, you would need to type check everything.... This ability to mesh non-numerical arrays has lots of uses and I think the change to a list will be transparent for most users who never use the packed array. Also, this makes ogrid and mgrid return consistent type objects. Therefore, I think it is worth the API change. Cheers, David On Sun, 2008-09-14 at 12:00 -0500, numpy-discussion-request at scipy.org wrote: > Message: 2 > Date: Sat, 13 Sep 2008 14:03:48 -0500 > From: "Travis E. Oliphant" > Subject: Re: [Numpy-discussion] patch for new mgrid / ogrid > functionality > To: Discussion of Numerical Python > Message-ID: <48CC0E94.60702 at enthought.com> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > St?fan van der Walt wrote: > > 2008/9/13 Travis E. Oliphant : > > > >> Generally, it is O.K. except that he changes the output from a > numpy > >> array to a list of numpy arrays. This is a small change, but it is > an > >> API change. I'm not sure why he found it necessary to do that, > but it > >> would be easier to apply the patch if it didn't change the API. > >> > > > > The example he gave illustrated having a mixed bag of types in one > > grid. You can do that with two arrays, but not one, unless you use > > object-arrays for everything. Maybe that makes sense. > > > > > I don't see why you have to use object-arrays for everything, but > obviously you would for the mixed types. > > -Travis > -- ********************************** David M. Kaplan Charge de Recherche 1 Institut de Recherche pour le Developpement Centre de Recherche Halieutique Mediterraneenne et Tropicale av. Jean Monnet B.P. 171 34203 Sete cedex France Phone: +33 (0)4 99 57 32 27 Fax: +33 (0)4 99 57 32 95 http://www.ur097.ird.fr/team/dkaplan/index.html ********************************** From david at ar.media.kyoto-u.ac.jp Mon Sep 15 03:50:20 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 15 Sep 2008 16:50:20 +0900 Subject: [Numpy-discussion] math directive, rest files and sphinx Message-ID: <48CE13BC.60908@ar.media.kyoto-u.ac.jp> Hi, I have to confess I am not always up-to-date to the whole documentation effort, and I have a problem in one of my package with the math directive. Can I use it for 'complex' latex, like linear algebra stuff (things like array environment, etc...); I don't intend to use it in docstring themselves, but in rst document generated by sphinx. I could not make it work with the numpydoc and mathml sphinx extensions from the numpyref project, cheers, David From stefan at sun.ac.za Mon Sep 15 05:32:29 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 15 Sep 2008 11:32:29 +0200 Subject: [Numpy-discussion] math directive, rest files and sphinx In-Reply-To: <48CE13BC.60908@ar.media.kyoto-u.ac.jp> References: <48CE13BC.60908@ar.media.kyoto-u.ac.jp> Message-ID: <9457e7c80809150232m7f81304en33f695d8c6cb4f02@mail.gmail.com> 2008/9/15 David Cournapeau : > I have to confess I am not always up-to-date to the whole > documentation effort, and I have a problem in one of my package with the > math directive. Can I use it for 'complex' latex, like linear algebra > stuff (things like array environment, etc...); I don't intend to use it > in docstring themselves, but in rst document generated by sphinx. I > could not make it work with the numpydoc and mathml sphinx extensions > from the numpyref project, It should work, as long as you include the necessary packages in your LaTeX preamble, which is included in the Sphinx project config file. Could you give a bit more info on what exactly breaks? Cheers St?fan From simon.palmer at gmail.com Mon Sep 15 05:55:23 2008 From: simon.palmer at gmail.com (SimonPalmer) Date: Mon, 15 Sep 2008 02:55:23 -0700 (PDT) Subject: [Numpy-discussion] product of arrays of different lengths Message-ID: <75aebd81-b191-4309-898a-7c1988e1daa1@s50g2000hsb.googlegroups.com> I have two 1D arrays of different lengths and I want the calculate sum of the product of the two to the extent of the smaller of them (make sense?). I don't know which of the two will be the longer ahead of time. I expected (hoped) to be able to do this: sp = (A * B).sum() and imagined that the trailing end of the longer of the two arrays might just be ignored, but instead I get an error "objects cannot be broadcast to a single shape". Both arrays are generally very short (<100 elements) so I can tolerate a copy, however this calculation is right in the heart of the innermost loop in my code, so I have to tread a bit carefully. Here's what I do at the moment, which is ugly, but works max_idx = min(len(A), len(B)) total_product = 0 for idx in range(0, max_idx): total_product += A[idx] * B[idx] ...but this is numpy, so I'm looking for a neat way. From faltet at pytables.org Mon Sep 15 06:25:27 2008 From: faltet at pytables.org (Francesc Alted) Date: Mon, 15 Sep 2008 12:25:27 +0200 Subject: [Numpy-discussion] product of arrays of different lengths In-Reply-To: <75aebd81-b191-4309-898a-7c1988e1daa1@s50g2000hsb.googlegroups.com> References: <75aebd81-b191-4309-898a-7c1988e1daa1@s50g2000hsb.googlegroups.com> Message-ID: <200809151225.27842.faltet@pytables.org> A Monday 15 September 2008, SimonPalmer escrigu?: > I have two 1D arrays of different lengths and I want the calculate > sum of the product of the two to the extent of the smaller of them > (make sense?). I don't know which of the two will be the longer > ahead of time. > > I expected (hoped) to be able to do this: > > sp = (A * B).sum() > > and imagined that the trailing end of the longer of the two arrays > might just be ignored, but instead I get an error "objects cannot be > broadcast to a single shape". > > Both arrays are generally very short (<100 elements) so I can > tolerate a copy, however this calculation is right in the heart of > the innermost loop in my code, so I have to tread a bit carefully. > > Here's what I do at the moment, which is ugly, but works > > max_idx = min(len(A), len(B)) > total_product = 0 > for idx in range(0, max_idx): > total_product += A[idx] * B[idx] > > ...but this is numpy, so I'm looking for a neat way. If I understand you correctly, maybe the next is what you want: max_idx = min(len(A), len(B)) (A[:max_idx] * B[:max_idx]).sum() which does not require a copy becuase the [:max_idx] operator returns just a view of the arrays. Hope that helps, -- Francesc Alted From robince at gmail.com Mon Sep 15 06:26:02 2008 From: robince at gmail.com (Robin) Date: Mon, 15 Sep 2008 11:26:02 +0100 Subject: [Numpy-discussion] funny matrix product calculation In-Reply-To: <1221427076.6915.5.camel@localhost.localdomain> References: <1221427076.6915.5.camel@localhost.localdomain> Message-ID: On Sun, Sep 14, 2008 at 10:17 PM, Fabrice Silva wrote: > What about considering a square diagonal matrix having vector x on its > main diagonal ? > Calling X such a matrix: > X = numpy.matrix(numpy.diag(x, k=0)) Thanks folks - I think your suggestions will work. Don't know why I didn't see it myself. Cheers Robin From robince at gmail.com Mon Sep 15 06:27:54 2008 From: robince at gmail.com (Robin) Date: Mon, 15 Sep 2008 11:27:54 +0100 Subject: [Numpy-discussion] profiling line by line Message-ID: Hi, I am using the prun feature of Ipython which is very helpful. I was wondering though if theres anything for Python that would allow line-by-line profiling (ie a time for each line of code) like the MATLAB profiler? Cheers Robin From stefan at sun.ac.za Mon Sep 15 06:28:42 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 15 Sep 2008 12:28:42 +0200 Subject: [Numpy-discussion] patch for new mgrid / ogrid functionality In-Reply-To: <1221465413.7117.9.camel@localhost> References: <1221465413.7117.9.camel@localhost> Message-ID: <9457e7c80809150328o3f62dd5co970ed07a24ff97a8@mail.gmail.com> 2008/9/15 David M. Kaplan : > This ability to mesh non-numerical arrays has lots of uses and I think > the change to a list will be transparent for most users who never use > the packed array. Also, this makes ogrid and mgrid return consistent > type objects. Therefore, I think it is worth the API change. I agree. But that means that we can only include it in 1.3 (which is currently being developed on trunnk, IIRC?). Cheers St?fan From simon.palmer at gmail.com Mon Sep 15 07:01:29 2008 From: simon.palmer at gmail.com (SimonPalmer) Date: Mon, 15 Sep 2008 04:01:29 -0700 (PDT) Subject: [Numpy-discussion] product of arrays of different lengths In-Reply-To: <200809151225.27842.faltet@pytables.org> References: <75aebd81-b191-4309-898a-7c1988e1daa1@s50g2000hsb.googlegroups.com> <200809151225.27842.faltet@pytables.org> Message-ID: nice, much neater, thanks On Sep 15, 11:25?am, Francesc Alted wrote: > A Monday 15 September 2008, SimonPalmer escrigu?: > > > > > I have two 1D arrays of different lengths and I want the calculate > > sum of the product of the two to the extent of the smaller of them > > (make sense?). ?I don't know which of the two will be the longer > > ahead of time. > > > I expected (hoped) to be able to do this: > > > sp = (A * B).sum() > > > and imagined that the trailing end of the longer of the two arrays > > might just be ignored, but instead I get an error "objects cannot be > > broadcast to a single shape". > > > Both arrays are generally very short (<100 elements) so I can > > tolerate a copy, however this calculation is right in the heart of > > the innermost loop in my code, so I have to tread a bit carefully. > > > Here's what I do at the moment, which is ugly, but works > > > ? ? ? ? max_idx = min(len(A), len(B)) > > ? ? ? ? total_product = 0 > > ? ? ? ? for idx in range(0, max_idx): > > ? ? ? ? ? ? total_product += A[idx] * B[idx] > > > ...but this is numpy, so I'm looking for a neat way. > > If I understand you correctly, maybe the next is what you want: > > max_idx = min(len(A), len(B)) > (A[:max_idx] * B[:max_idx]).sum() > > which does not require a copy becuase the [:max_idx] operator returns > just a view of the arrays. > > Hope that helps, > > -- > Francesc Alted > _______________________________________________ > Numpy-discussion mailing list > Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion From aisaac at american.edu Mon Sep 15 09:08:20 2008 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 15 Sep 2008 09:08:20 -0400 Subject: [Numpy-discussion] product of arrays of different lengths In-Reply-To: <200809151225.27842.faltet@pytables.org> References: <75aebd81-b191-4309-898a-7c1988e1daa1@s50g2000hsb.googlegroups.com> <200809151225.27842.faltet@pytables.org> Message-ID: <48CE5E44.6090802@american.edu> On 9/15/2008 6:25 AM Francesc Alted apparently wrote: > max_idx = min(len(A), len(B)) > (A[:max_idx] * B[:max_idx]).sum() > > which does not require a copy becuase the [:max_idx] operator returns > just a view of the arrays. But it still requires creating a new array, so perhaps use of ``dot`` above or even:: sum(ai*bi for ai,bi in izip(A,B)) is worth considering in this case. (That is the built-in ``sum``.) Alan Isaac From faltet at pytables.org Mon Sep 15 09:32:15 2008 From: faltet at pytables.org (Francesc Alted) Date: Mon, 15 Sep 2008 15:32:15 +0200 Subject: [Numpy-discussion] product of arrays of different lengths In-Reply-To: <48CE5E44.6090802@american.edu> References: <75aebd81-b191-4309-898a-7c1988e1daa1@s50g2000hsb.googlegroups.com> <200809151225.27842.faltet@pytables.org> <48CE5E44.6090802@american.edu> Message-ID: <200809151532.15648.faltet@pytables.org> A Monday 15 September 2008, Alan G Isaac escrigu?: > On 9/15/2008 6:25 AM Francesc Alted apparently wrote: > > max_idx = min(len(A), len(B)) > > (A[:max_idx] * B[:max_idx]).sum() > > > > which does not require a copy becuase the [:max_idx] operator > > returns just a view of the arrays. > > But it still requires creating a new array, > so perhaps use of ``dot`` above or even:: > > sum(ai*bi for ai,bi in izip(A,B)) > > is worth considering in this case. > (That is the built-in ``sum``.) Yeah. Good point. -- Francesc Alted From simon.palmer at gmail.com Mon Sep 15 10:14:04 2008 From: simon.palmer at gmail.com (SimonPalmer) Date: Mon, 15 Sep 2008 07:14:04 -0700 (PDT) Subject: [Numpy-discussion] product of arrays of different lengths In-Reply-To: <200809151532.15648.faltet@pytables.org> References: <75aebd81-b191-4309-898a-7c1988e1daa1@s50g2000hsb.googlegroups.com> <200809151225.27842.faltet@pytables.org> <48CE5E44.6090802@american.edu> <200809151532.15648.faltet@pytables.org> Message-ID: what is the overhead associated with importing a new module (whichever includes izip)? I am wondering whether it is actually more efficient for me to put my aesthetics aside and stick with my ugly but efficient loop On Sep 15, 2:32?pm, Francesc Alted wrote: > A Monday 15 September 2008, Alan G Isaac escrigu?: > > > On 9/15/2008 6:25 AM Francesc Alted apparently wrote: > > > max_idx = min(len(A), len(B)) > > > (A[:max_idx] * B[:max_idx]).sum() > > > > which does not require a copy becuase the [:max_idx] operator > > > returns just a view of the arrays. > > > But it still requires creating a new array, > > so perhaps use of ``dot`` above or even:: > > > ? ?sum(ai*bi for ai,bi in izip(A,B)) > > > is worth considering in this case. > > (That is the built-in ``sum``.) > > Yeah. ?Good point. > > -- > Francesc Alted > _______________________________________________ > Numpy-discussion mailing list > Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion From gael.varoquaux at normalesup.org Mon Sep 15 10:33:19 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 15 Sep 2008 16:33:19 +0200 Subject: [Numpy-discussion] patch for new mgrid / ogrid functionality In-Reply-To: <9457e7c80809150328o3f62dd5co970ed07a24ff97a8@mail.gmail.com> References: <1221465413.7117.9.camel@localhost> <9457e7c80809150328o3f62dd5co970ed07a24ff97a8@mail.gmail.com> Message-ID: <20080915143319.GB21135@phare.normalesup.org> On Mon, Sep 15, 2008 at 12:28:42PM +0200, St?fan van der Walt wrote: > 2008/9/15 David M. Kaplan : > > This ability to mesh non-numerical arrays has lots of uses and I think > > the change to a list will be transparent for most users who never use > > the packed array. Also, this makes ogrid and mgrid return consistent > > type objects. Therefore, I think it is worth the API change. > I agree. But that means that we can only include it in 1.3 (which is > currently being developed on trunnk, IIRC?). I'd prefer not seeing any non-backwards-compatible API change before 2.0. I think we would be breaking the contract that we have with our users, that are going to see some code that worked suddenly break. Ga?l From dalke at dalkescientific.com Mon Sep 15 10:38:06 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Mon, 15 Sep 2008 16:38:06 +0200 Subject: [Numpy-discussion] isnan and co: cleaning up In-Reply-To: <48CDDA67.3080505@ar.media.kyoto-u.ac.jp> References: <48C0BE3F.1030306@ar.media.kyoto-u.ac.jp> <0C000502-8983-4289-8CF4-83CAE5A39E13@dalkescientific.com> <48CDDA67.3080505@ar.media.kyoto-u.ac.jp> Message-ID: <8D939C2A-EA79-4C92-A8FE-C7C7F2E357DC@dalkescientific.com> On Sep 15, 2008, at 5:45 AM, David Cournapeau wrote: > It is not an error for the build, but an error at the configuration > stage. To get some informations about the platform, we do compile some > code snippets, and do our configuration depending on whether they fail > or not. We could log those in a different location than the actual > build, but I am not sure anyone wants to spend some time on it. Okay, then I'll let it stand, and if someone else is pensive and does a web search for it then they'll at least find this thread. Andrew dalke at dalkescientific.com From gael.varoquaux at normalesup.org Mon Sep 15 10:38:30 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 15 Sep 2008 16:38:30 +0200 Subject: [Numpy-discussion] product of arrays of different lengths In-Reply-To: References: <75aebd81-b191-4309-898a-7c1988e1daa1@s50g2000hsb.googlegroups.com> <200809151225.27842.faltet@pytables.org> <48CE5E44.6090802@american.edu> <200809151532.15648.faltet@pytables.org> Message-ID: <20080915143830.GC21135@phare.normalesup.org> On Mon, Sep 15, 2008 at 07:14:04AM -0700, SimonPalmer wrote: > what is the overhead associated with importing a new module (whichever > includes izip)? Modules are cached, so the cost of importing a module, when it has already been loaded, is simply the cost of checking if it has been loaded, and copying the references to the variables in the locals namespace. Changes are izip has already been loaded, so the overhead is going to be relly low. Don't put the import statement in the inner loop, so as not to have this overhead while running the loop. Put it at the module level, so as to have it only once. By the way, speed question are not something one should consider from prior knowledge, at least not too much, because it is so easy to get it wrong. Just use the "%timeit" magic of IPython to check for yourself, and when you have different approaches in mind, see which one is the fastest. HTH, Ga?l From stefan at sun.ac.za Mon Sep 15 10:40:51 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 15 Sep 2008 16:40:51 +0200 Subject: [Numpy-discussion] patch for new mgrid / ogrid functionality In-Reply-To: <20080915143319.GB21135@phare.normalesup.org> References: <1221465413.7117.9.camel@localhost> <9457e7c80809150328o3f62dd5co970ed07a24ff97a8@mail.gmail.com> <20080915143319.GB21135@phare.normalesup.org> Message-ID: <9457e7c80809150740w32f75247r38e9b029328e0dcf@mail.gmail.com> 2008/9/15 Gael Varoquaux : >> I agree. But that means that we can only include it in 1.3 (which is >> currently being developed on trunk, IIRC?). > > I'd prefer not seeing any non-backwards-compatible API change before 2.0. > I think we would be breaking the contract that we have with our users, > that are going to see some code that worked suddenly break. We could always return lists *only* for mixed type ogrids. While that preserves backwards compatibility, it is really ugly :/ I understand all the arguments involved, but I see much pain and suffering (Yoda be my witness) in cementing ourselves to an immature Python-level API. Cheers St?fan From faltet at pytables.org Mon Sep 15 11:18:26 2008 From: faltet at pytables.org (Francesc Alted) Date: Mon, 15 Sep 2008 17:18:26 +0200 Subject: [Numpy-discussion] product of arrays of different lengths In-Reply-To: References: <75aebd81-b191-4309-898a-7c1988e1daa1@s50g2000hsb.googlegroups.com> <200809151532.15648.faltet@pytables.org> Message-ID: <200809151718.27163.faltet@pytables.org> A Monday 15 September 2008, SimonPalmer escrigu?: > what is the overhead associated with importing a new module > (whichever includes izip)? > > I am wondering whether it is actually more efficient for me to put my > aesthetics aside and stick with my ugly but efficient loop If the loop is important for you in terms of time consumption, always do some timing mesurements so as to be sure that you are choosing the most efficient approach. In your case I'd say that, as your arrays are small (<100 elements), the memory penalty of using the: (A[:max_idx] * B[:max_idx]).sum() idiom is nothing compared with the speed gains of using NumPy. Here are some timings: #1 Your initial solution In [46]: t1 = timeit.Timer("max_idx = min(len(A), len(B)); total_product = sum( A[idx] * B[idx] for idx in range(0, max_idx))", "import numpy as N; A=N.arange(100); B=N.arange(88)") In [47]: t1.repeat(3,1000) Out[47]: [0.4705650806427002, 0.35150599479675293, 0.34992504119873047] #2 Using NumPy multiplication In [48]: t2 = timeit.Timer("max_idx = min(len(A), len(B)); total_product = (A[:max_idx] * B[:max_idx]).sum()", "import numpy as N; A=N.arange(100); B=N.arange(88)") In [49]: t2.repeat(3,1000) Out[49]: [0.044930934906005859, 0.043951034545898438, 0.042618989944458008] #3 Using izip iterator In [50]: t3 = timeit.Timer("total_product = sum(ai*bi for ai,bi in izip(A,B))", "from itertools import izip; import numpy as N; A=N.arange(100); B=N.arange(88)") In [51]: t3.repeat(3,1000) Out[51]: [0.45449995994567871, 0.37039089202880859, 0.3395388126373291] As you can see solution #2 is around 10x faster than the others. Cheers, -- Francesc Alted From robert.kern at gmail.com Mon Sep 15 11:18:45 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 15 Sep 2008 10:18:45 -0500 Subject: [Numpy-discussion] profiling line by line In-Reply-To: References: Message-ID: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> On Mon, Sep 15, 2008 at 05:27, Robin wrote: > Hi, > > I am using the prun feature of Ipython which is very helpful. > > I was wondering though if theres anything for Python that would allow > line-by-line profiling (ie a time for each line of code) like the > MATLAB profiler? I don't think there is anything out there at the moment. Python does have a hook for line-by-line tracing, and one can make a line-by-line profiler from it. To do it with the least overhead, you need to use the C API for it and a C callback. I do have some Cython code that does this. It needs a little bit more work, though. I'll try to push it out soonish. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From e-alerts at proteinkemi.dk Mon Sep 15 10:50:01 2008 From: e-alerts at proteinkemi.dk (Tommy Carstensen) Date: Mon, 15 Sep 2008 15:50:01 +0100 Subject: [Numpy-discussion] numpy.linalg.eig Message-ID: <003f01c91742$54b5c9d0$fe215d70$@dk> To the numpy mailing list, Previously I used python2.3 and the module LinearAlgebra for matrix diagonalization. I recently upgraded to python2.5 and I am now using numpy.linalg.eig for matrix diagonalization. This is approximately 3 times slower than previously. I am using the same work station as previously and it has all lapack libraries installed to my knowledge. Does anyone have an explanation as to why I am experiencing the reduced speed of calculation? This is my first post to this mailing list, so I hope this mail complies with the mailing list standards. Kind regards, Tommy Tommy Carstensen UCD School of Biomolecular and Biomedical Sciences Conway Institute University College Dublin Dublin 4, Ireland http://enzyme.ucd.ie/group_members/tommy From arnar.flatberg at gmail.com Mon Sep 15 12:13:03 2008 From: arnar.flatberg at gmail.com (Arnar Flatberg) Date: Mon, 15 Sep 2008 18:13:03 +0200 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> Message-ID: <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> On Mon, Sep 15, 2008 at 5:18 PM, Robert Kern wrote: > I do have some Cython code that > does this. It needs a little bit more work, though. I'll try to push > it out soonish. > That would make me an extremely happy user, I've been looking for this for years! I can't imagine I'm the only one who profiles some hundred lines of code and ends up with 90% of total time in the dot-function Arnar -------------- next part -------------- An HTML attachment was scrubbed... URL: From Achim.Gaedke at physik.tu-darmstadt.de Mon Sep 15 12:11:06 2008 From: Achim.Gaedke at physik.tu-darmstadt.de (Achim Gaedke) Date: Mon, 15 Sep 2008 18:11:06 +0200 Subject: [Numpy-discussion] trouble with locale in combination with numpy and pylab Message-ID: <48CE891A.4080800@physik.tu-darmstadt.de> Hi there! Consider this python program (saved as locale_trouble.py): import numpy a=numpy.float32(1.2) print a, float(a) import gtk print a, float(a) everything is fine, if you call this program as follows: LANG="C" python locale_trouble.py 1.2 1.20000004768 1.2 1.20000004768 LANG="de_DE" python locale_trouble.py 1.2 1.20000004768 1,2 1.20000004768 The coma in the last line is unexpected. This error also occurs when using pylab or matplotlib with gtk backend: import pylab print pylab.rcParams["backend"] GTKAgg This behaviour is observed with debian lenny and python 2.4 and python 2.5, numpy-1.1.0 and numpy-1.1.1 . On debian etch with numpy-1.0.1 we were not able to provoke a coma. This behaviour is highly irritating, so we believe that is a bug. Following the python policy to use a dot for numbers, independent of locale definitions. See note (3) on page: http://docs.python.org/lib/typesseq-strings.html Yours, Achim From stefan at sun.ac.za Mon Sep 15 12:29:18 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 15 Sep 2008 18:29:18 +0200 Subject: [Numpy-discussion] trouble with locale in combination with numpy and pylab In-Reply-To: <48CE891A.4080800@physik.tu-darmstadt.de> References: <48CE891A.4080800@physik.tu-darmstadt.de> Message-ID: <9457e7c80809150929q6fda09a4yc01bfc5a7daec2f0@mail.gmail.com> 2008/9/15 Achim Gaedke : > Hi there! > > Consider this python program (saved as locale_trouble.py): This has already been filed as http://projects.scipy.org/scipy/numpy/ticket/902 Please add any additional info to that ticket. Thanks! St?fan From robince at gmail.com Mon Sep 15 12:38:14 2008 From: robince at gmail.com (Robin) Date: Mon, 15 Sep 2008 17:38:14 +0100 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> Message-ID: On Mon, Sep 15, 2008 at 5:13 PM, Arnar Flatberg wrote: > On Mon, Sep 15, 2008 at 5:18 PM, Robert Kern wrote: >> >> I do have some Cython code that >> does this. It needs a little bit more work, though. I'll try to push >> it out soonish. > > That would make me an extremely happy user, I've been looking for this for > years! > I can't imagine I'm the only one who profiles some hundred lines of code and > ends up with 90% of total time in the dot-function Yes, that would be terrific - the profiler is about the only thing I miss from Matlab... Especially after an afternoon of refactoring everything into tiny functions to get anything useful out of the normal profiler and see where the bottleneck in my code was. Robin From robert.kern at gmail.com Mon Sep 15 12:38:51 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 15 Sep 2008 11:38:51 -0500 Subject: [Numpy-discussion] numpy.linalg.eig In-Reply-To: <003f01c91742$54b5c9d0$fe215d70$@dk> References: <003f01c91742$54b5c9d0$fe215d70$@dk> Message-ID: <3d375d730809150938x6bc994aai197dfa28c5e50092@mail.gmail.com> On Mon, Sep 15, 2008 at 09:50, Tommy Carstensen wrote: > To the numpy mailing list, > > Previously I used python2.3 and the module LinearAlgebra for matrix > diagonalization. I recently upgraded to python2.5 and I am now using > numpy.linalg.eig for matrix diagonalization. This is approximately 3 times > slower than previously. I am using the same work station as previously and > it has all lapack libraries installed to my knowledge. Does anyone have an > explanation as to why I am experiencing the reduced speed of calculation? Although you have the same LAPACK libraries installed, Numeric and numpy may have been built differently, and numpy may not be using the same libraries. Can you check? > This is my first post to this mailing list, so I hope this mail complies > with the mailing list standards. Yes, certainly. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From millman at berkeley.edu Mon Sep 15 13:10:04 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 15 Sep 2008 10:10:04 -0700 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: References: Message-ID: Hey, I would like to release 1.2.0 final soon, but I need to know whether anyone is having any problems with the rc2 binaries. Please test them and let me know whether or not you have any problems with them. Thanks, Jarrod On Sat, Sep 13, 2008 at 4:00 PM, Jarrod Millman wrote: > Hello, > > The 1.2.0rc2 is now available: > http://svn.scipy.org/svn/numpy/tags/1.2.0rc2 > > The source tarball is here: > https://cirl.berkeley.edu/numpy/numpy-1.2.0rc2.tar.gz > > Here is the universal Mac binary: > https://cirl.berkeley.edu/numpy/numpy-1.2.0rc2-py2.5-macosx10.5.dmg > > Here are the Window's binaries: > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.2.0rc2-win32-superpack-python2.4.exe > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.2.0rc2-win32-superpack-python2.5.exe > > Here are the release notes: > http://scipy.org/scipy/numpy/milestone/1.2.0 > > Please test this release ASAP and let us know if there are any > problems. If there are no show stoppers, this will become the > 1.2.0 release. > > Thanks, > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From aisaac at american.edu Mon Sep 15 13:33:56 2008 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 15 Sep 2008 13:33:56 -0400 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: References: Message-ID: <48CE9C84.7060406@american.edu> Python 2.5 Superpack looks fine here. Alan ################################################################### Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.__version__ '1.2.0rc2' >>> numpy.test() Running unit tests for numpy NumPy version 1.2.0rc2 NumPy is installed in C:\Python25\lib\site-packages\numpy Python version 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Int el)] nose version 0.10.1 ................................................................................ ................................................................................ ................................................................................ .....K..K....................................................................... ................................................................................ ...........................................K.................................... ............................................Ignoring "Python was built with Visu al Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py." (one s hould fix me in fcompiler/compaq.py) ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ .............. ====================================================================== KNOWNFAIL: test_complex_longdouble (test_print.TestPrint) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\testing\decorators.py", line 119, in skipper raise KnownFailureTest, msg KnownFailureTest: complex long double print is known to fail on windows ====================================================================== KNOWNFAIL: test_longdouble (test_print.TestPrint) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\testing\decorators.py", line 119, in skipper raise KnownFailureTest, msg KnownFailureTest: long double print is known to fail on windows ====================================================================== KNOWNFAIL: test_umath.TestComplexFunctions.test_branch_cuts_failing ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\lib\site-packages\nose\case.py", line 203, in runTest self.test(*self.arg) File "C:\Python25\Lib\site-packages\numpy\testing\decorators.py", line 119, in skipper raise KnownFailureTest, msg KnownFailureTest: These branch cuts are known to fail ---------------------------------------------------------------------- Ran 1578 tests in 17.856s OK >>> From aisaac at american.edu Mon Sep 15 13:48:54 2008 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 15 Sep 2008 13:48:54 -0400 Subject: [Numpy-discussion] product of arrays of different lengths In-Reply-To: References: <75aebd81-b191-4309-898a-7c1988e1daa1@s50g2000hsb.googlegroups.com> <200809151225.27842.faltet@pytables.org> <48CE5E44.6090802@american.edu> <200809151532.15648.faltet@pytables.org> Message-ID: <48CEA006.3010607@american.edu> SimonPalmer wrote: > what is the overhead associated with importing a new module (whichever > includes izip)? > I am wondering whether it is actually more efficient for me to put my > aesthetics aside and stick with my ugly but efficient loop If you do not want to use izip, you can still avoid array creation by using ``dot``:: max_idx = min(len(A), len(B)) np.dot(A[:max_idx], B[:max_idx]) Cheers, Alan Isaac From markperrymiller at gmail.com Mon Sep 15 13:49:27 2008 From: markperrymiller at gmail.com (Mark Miller) Date: Mon, 15 Sep 2008 10:49:27 -0700 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: <48CE9C84.7060406@american.edu> References: <48CE9C84.7060406@american.edu> Message-ID: Warning, errors, and failures here on XP Pro (numpy installed with the python 2.5 superpack). Just passing it along, and apologies if these have already been caught. >>> import numpy >>> numpy.__version__ '1.2.0rc2' >>> numpy.test() Running unit tests for numpy NumPy version 1.2.0rc2 NumPy is installed in C:\Python25\lib\site-packages\numpy Python version 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] nose version 0.10.3 ....................................... Warning (from warnings module): File "C:\Python25\lib\unittest.py", line 507 return self.suiteClass(map(testCaseClass, testCaseNames)) DeprecationWarning: NumpyTestCase will be removed in the next release; please update your code to use nose or unittest Warning (from warnings module): File "C:\Python25\lib\site-packages\numpy\core\tests\test_ma.py", line 18 NumpyTestCase.__init__(self, *args, **kwds) DeprecationWarning: NumpyTestCase will be removed in the next release; please update your code to use nose or unittest ............................... Warning (from warnings module): File "C:\Python25\lib\site-packages\numpy\core\ma.py", line 609 warnings.warn("Cannot automatically convert masked array to "\ UserWarning: Cannot automatically convert masked array to numeric because data is masked in one or more locations. F...EEEEE..............................................................................................................................................................................................................K..K..................................................................................................................................................................................................K................................................................................Ignoring "Python was built with Visual Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py." (one should fix me in fcompiler/compaq.py) .............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................. ====================================================================== ERROR: test_ma.testta ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\python25\Lib\site-packages\nose\case.py", line 182, in runTest self.test(*self.arg) TypeError: testta() takes exactly 2 arguments (0 given) ====================================================================== ERROR: test_ma.testtb ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\python25\Lib\site-packages\nose\case.py", line 182, in runTest self.test(*self.arg) TypeError: testtb() takes exactly 2 arguments (0 given) ====================================================================== ERROR: test_ma.testtc ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\python25\Lib\site-packages\nose\case.py", line 182, in runTest self.test(*self.arg) TypeError: testtc() takes exactly 2 arguments (0 given) ====================================================================== ERROR: test_ma.testf ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\python25\Lib\site-packages\nose\case.py", line 182, in runTest self.test(*self.arg) TypeError: testf() takes exactly 1 argument (0 given) ====================================================================== ERROR: test_ma.testinplace ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\python25\Lib\site-packages\nose\case.py", line 182, in runTest self.test(*self.arg) TypeError: testinplace() takes exactly 1 argument (0 given) ====================================================================== FAIL: check_testUfuncRegression (test_ma.TestUfuncs) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\lib\site-packages\numpy\core\tests\test_ma.py", line 692, in check_testUfuncRegression self.failUnless(eqmask(ur.mask, mr.mask)) AssertionError ---------------------------------------------------------------------- Ran 1618 tests in 7.485s FAILED (KNOWNFAIL=3, errors=5, failures=1) >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Mon Sep 15 14:03:40 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 15 Sep 2008 11:03:40 -0700 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: References: <48CE9C84.7060406@american.edu> Message-ID: On Mon, Sep 15, 2008 at 10:49 AM, Mark Miller wrote: > Warning, errors, and failures here on XP Pro (numpy installed with the > python 2.5 superpack). > > Just passing it along, and apologies if these have already been caught. Thanks for testing this. It looks like you have some old files sitting around (e.g., numpy\core\tests\test_ma.py). Could you delete C:\Python25\lib\site-packages\numpy and reinstall 1.2.0rc2? Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From markperrymiller at gmail.com Mon Sep 15 14:10:02 2008 From: markperrymiller at gmail.com (Mark Miller) Date: Mon, 15 Sep 2008 11:10:02 -0700 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: References: <48CE9C84.7060406@american.edu> Message-ID: OK..thanks. That did the trick. All clear now, save for 3 known failures. Again, thanks for letting me know about this. -Mark On Mon, Sep 15, 2008 at 11:03 AM, Jarrod Millman wrote: > On Mon, Sep 15, 2008 at 10:49 AM, Mark Miller > wrote: > > Warning, errors, and failures here on XP Pro (numpy installed with the > > python 2.5 superpack). > > > > Just passing it along, and apologies if these have already been caught. > > Thanks for testing this. It looks like you have some old files > sitting around (e.g., numpy\core\tests\test_ma.py). Could you delete > C:\Python25\lib\site-packages\numpy and reinstall 1.2.0rc2? > > Thanks, > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Mon Sep 15 14:23:38 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 15 Sep 2008 11:23:38 -0700 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: References: <48CE9C84.7060406@american.edu> Message-ID: On Mon, Sep 15, 2008 at 11:10 AM, Mark Miller wrote: > OK..thanks. That did the trick. All clear now, save for 3 known failures. Excellent. Thanks for testing this. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From josef.pktd at gmail.com Mon Sep 15 14:50:21 2008 From: josef.pktd at gmail.com (joep) Date: Mon, 15 Sep 2008 11:50:21 -0700 (PDT) Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: References: <48CE9C84.7060406@american.edu> Message-ID: all ok on python 2.4 WindowsXP sse2 scipy test results are the same as with numpy 1.1.0 C:\Josef\work-oth\sort\pypi>python Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.version.version '1.2.0rc2' >>> numpy.test() Running unit tests for numpy NumPy version 1.2.0rc2 NumPy is installed in c:\programs\python24\lib\site-packages\numpy Python version 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)] nose version 0.10.3 ................................................................................ ................................................................................ ................................................................................ ..K..K.......................................................................... ................................................................................ ........................................K....................................... .........................................Ignoring "The .NET Framework SDK needs to be installed before building extensions for Python." (one should fix me in fc ompiler/compaq.py) ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ .............. ---------------------------------------------------------------------- Ran 1575 tests in 10.125s OK (KNOWNFAIL=3) (doesn't show details of known fails) >>> import scipy >>> scipy.version.version '0.6.0' >>> scipy.test() Ran 1862 tests in 8.984s FAILED (failures=4, errors=2) From simon.palmer at gmail.com Mon Sep 15 14:55:54 2008 From: simon.palmer at gmail.com (Simon Palmer) Date: Mon, 15 Sep 2008 19:55:54 +0100 Subject: [Numpy-discussion] product of arrays of different lengths In-Reply-To: <48CEA006.3010607@american.edu> References: <75aebd81-b191-4309-898a-7c1988e1daa1@s50g2000hsb.googlegroups.com> <200809151225.27842.faltet@pytables.org> <48CE5E44.6090802@american.edu> <200809151532.15648.faltet@pytables.org> <48CEA006.3010607@american.edu> Message-ID: Thanks very much everyone, this has been very helpful. I have been doing some timing of my own and the order of magnitude difference that numpy provides wins. I'm learning python as I go so some of my questions come from a lack of language skillls, but it is good to get to know the numpy dialect too. On Mon, Sep 15, 2008 at 6:48 PM, Alan G Isaac wrote: > SimonPalmer wrote: > > what is the overhead associated with importing a new module (whichever > > includes izip)? > > I am wondering whether it is actually more efficient for me to put my > > aesthetics aside and stick with my ugly but efficient loop > > If you do not want to use izip, you can still avoid array > creation by using ``dot``:: > > max_idx = min(len(A), len(B)) > np.dot(A[:max_idx], B[:max_idx]) > > Cheers, > Alan Isaac > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mforbes at physics.ubc.ca Mon Sep 15 16:45:14 2008 From: mforbes at physics.ubc.ca (Michael McNeil Forbes) Date: Mon, 15 Sep 2008 16:45:14 -0400 Subject: [Numpy-discussion] profiling line by line In-Reply-To: References: Message-ID: The hotshot profiler used to do this, but I don't think it is really supported anymore... I have not used it in a while, but agree that a line-by-line profiler can be very nice. Michael. On Sep 15, 2008, at 6:27 AM, Robin wrote: > Hi, > > I am using the prun feature of Ipython which is very helpful. > > I was wondering though if theres anything for Python that would allow > line-by-line profiling (ie a time for each line of code) like the > MATLAB profiler? > > Cheers > > Robin > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From neilcrighton at gmail.com Mon Sep 15 18:11:04 2008 From: neilcrighton at gmail.com (Neil) Date: Mon, 15 Sep 2008 15:11:04 -0700 (PDT) Subject: [Numpy-discussion] loadtxt improvement Message-ID: <5d88a931-b5cd-48dd-94ca-378fe6791f04@p25g2000hsf.googlegroups.com> I'd like to add a way for loadtxt to infer a dtype from the data it reads in. Record arrays/structured arrays are the best thing ever, and ideally I'd like to read in a csv-style file into a structured array in one easy step. loadtxt almost does this - if I know the number and type of fields before reading the file, I can specify the dtype keyword, and loadtxt will give me a structured array. But having to know the dtype before you read a file (including the the required string lengths!) is a real pain. It would be great if you could tell loadtxt to read a file into a structured array and guess the dtype for each field. I've made some changes to lib/io.py that does this by adding a 'names' keyword to loadtxt. If a list of field names is given, then loadtxt reads the file data into a structured array, trying int, float and str types for each column and keeping whichever is suitable for all the data in that column. Does this sound like a good approach? From wnbell at gmail.com Mon Sep 15 22:40:10 2008 From: wnbell at gmail.com (Nathan Bell) Date: Mon, 15 Sep 2008 22:40:10 -0400 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: References: Message-ID: On Sat, Sep 13, 2008 at 7:00 PM, Jarrod Millman wrote: > > Please test this release ASAP and let us know if there are any > problems. If there are no show stoppers, this will become the > 1.2.0 release. > Looks OK on Ubuntu 8.04 x86_64. Thanks for your continued work! $ python -c 'from numpy import test; test()' Running unit tests for numpy NumPy version 1.2.0rc2 NumPy is installed in /usr/lib/python2.5/site-packages/numpy Python version 2.5.2 (r252:60911, Jul 31 2008, 17:31:22) [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] nose version 0.10.3 ...........................................................................................................................................................................................................................................................................................................................................................................................................................................................K.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................. ---------------------------------------------------------------------- Ran 1726 tests in 8.813s OK (KNOWNFAIL=1) -- Nathan Bell wnbell at gmail.com http://graphics.cs.uiuc.edu/~wnbell/ From david at ar.media.kyoto-u.ac.jp Mon Sep 15 22:43:20 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 16 Sep 2008 11:43:20 +0900 Subject: [Numpy-discussion] profiling line by line In-Reply-To: References: Message-ID: <48CF1D48.8050403@ar.media.kyoto-u.ac.jp> Michael McNeil Forbes wrote: > The hotshot profiler used to do this, but I don't think it is really > supported anymore... I have not used it in a while, but agree that a > line-by-line profiler can be very nice. > You can do it with kcachegrind, which is arguably more powerful, but it is a bit a pain to set up. http://jcalderone.livejournal.com/21124.html cheers, David From robert.kern at gmail.com Mon Sep 15 23:09:50 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 15 Sep 2008 22:09:50 -0500 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <48CF1D48.8050403@ar.media.kyoto-u.ac.jp> References: <48CF1D48.8050403@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730809152009ga634dara6ee25535651b51d@mail.gmail.com> On Mon, Sep 15, 2008 at 21:43, David Cournapeau wrote: > Michael McNeil Forbes wrote: >> The hotshot profiler used to do this, but I don't think it is really >> supported anymore... I have not used it in a while, but agree that a >> line-by-line profiler can be very nice. > > You can do it with kcachegrind, which is arguably more powerful, but it > is a bit a pain to set up. > > http://jcalderone.livejournal.com/21124.html No, that's with cProfile, which doesn't support line-by-line profiling. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Mon Sep 15 23:12:19 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 16 Sep 2008 12:12:19 +0900 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <3d375d730809152009ga634dara6ee25535651b51d@mail.gmail.com> References: <48CF1D48.8050403@ar.media.kyoto-u.ac.jp> <3d375d730809152009ga634dara6ee25535651b51d@mail.gmail.com> Message-ID: <48CF2413.40807@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > On Mon, Sep 15, 2008 at 21:43, David Cournapeau > wrote: > >> You can do it with kcachegrind, which is arguably more powerful, but it >> is a bit a pain to set up. >> >> http://jcalderone.livejournal.com/21124.html >> > > No, that's with cProfile, which doesn't support line-by-line profiling. > You're right, I did it with hotshot, I did not paste the right link, sorry. The right one should be: http://ddaa.net/blog/python/lsprof-calltree Both profiles (cProfile and hotshot) formats are usable with kcachegrind, but you have to convert first the profiles, so it is not as integrated as if everything was in python. It does give you information similar as matlab, which is one thing I missed from matlab too when I started using python. cheers, David From david at ar.media.kyoto-u.ac.jp Tue Sep 16 00:32:58 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 16 Sep 2008 13:32:58 +0900 Subject: [Numpy-discussion] math directive, rest files and sphinx In-Reply-To: <9457e7c80809150232m7f81304en33f695d8c6cb4f02@mail.gmail.com> References: <48CE13BC.60908@ar.media.kyoto-u.ac.jp> <9457e7c80809150232m7f81304en33f695d8c6cb4f02@mail.gmail.com> Message-ID: <48CF36FA.9060508@ar.media.kyoto-u.ac.jp> St?fan van der Walt wrote: > > It should work, as long as you include the necessary packages in your > LaTeX preamble, which is included in the Sphinx project config file. > Could you give a bit more info on what exactly breaks? I managed to get further, but I don't get what I want. A minimal latex code which has the problem is the following .. math:: \begin{matrix}x \\ y \end{array} This shows correctly with latex, but not with html (which shows xy instead of a column with x in one row and y in the other). I guess this is a mathml problem ? Also, I noticed a trivial but confusing error in error messages in mathml.py. Shall I report those problems with fixes in the numpyref package at launchpad ? IOW, what is the 'official' gate to this ? Finally, are are gearing toward math support from sphinx directly (available in future 0.5), or will we keep our own ? cheers, David From spam4bsimons at yahoo.ca Tue Sep 16 00:52:19 2008 From: spam4bsimons at yahoo.ca (brendan simons) Date: Mon, 15 Sep 2008 21:52:19 -0700 (PDT) Subject: [Numpy-discussion] Help speeding up element-wise operations for video processing Message-ID: <458186.45235.qm@web51601.mail.re2.yahoo.com> Hi everyone I have a quick problem. I'm trying to write a routine to "demosaic" video data extremely quickly. I thought my algorithm would make good use of my processor's vector math units, but it isn't fast enough (it currently takes about 300ms for each 8 megapixel frame) Maybe you can help me speed it up? Lets just look at one piece of the algorithm - interpolating the green pixels: My camera hands me a frame as an int* c array of 8 bit integers. Forget the problem of turning that into a numpy array for now and assume I have the image as a numpy uint8 array "ain" . Also assume that Greenmask is a precomputed array of 1s and 0s of the same size as ain. Now, I do the following: #interpolate the green pixels from the bayer filter image ain g = greenMask * ain gi = g[:-2, 1:-1].astype('uint16') gi += g[2:, 1:-1] gi += g[1:-1, :-2] gi += g[1:-1, 2:] gi /= 4 gi += g[1:-1, 1:-1] return gi I do something similar for red and blue, then stack the interpolated red, green and blue integers into an array of 24 bit integers and blit to the screen. I was hoping that none of the lines would have to iterate over pixels, and would instead do the adds and multiplies as single operations. Perhaps numpy has to iterate when copying a subset of an array? Is there a faster array "crop" ? Any hints as to how I might code this part up using ctypes? Thanks in advance, Brendan __________________________________________________________________ Ask a question on any topic and get answers from real people. Go to Yahoo! Answers and share what you know at http://ca.answers.yahoo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From f.drossaert at googlemail.com Tue Sep 16 02:58:20 2008 From: f.drossaert at googlemail.com (Francis Drossaert) Date: Tue, 16 Sep 2008 07:58:20 +0100 Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ Message-ID: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> Hi everybody, I am trying to install python2.5/scipy/numpy/sympy/matplotlib locally, because of various reasons. I am not root. At home I am using Ubuntu and I am root and everything works as it should. So far I managed to make python2.5, sympy and ipython to run fine on my computer at work, but I am struggling to make numpy+scipy since I am trying to install it locally. The problem is likely the lapack+atlas build. I have followed the build by hand instructions on http://www.scipy.org/Installing_SciPy/Linux. I am not getting any errors during the build and install. However after installing I tried to import numpy in python and received to following error: ImportError: /users/francisd/local/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so: undefined symbol: zgesdd_ Googling this error, it seems that the error is caused by having a wrong lapack version. I am running Centos 5 (basically Red Hat Enterprise Linux 5) on a x86_64 machine. Does anybody know how I can get the right version, by changing some flags, or whatever? FYI I have added the -m64 flag for the lapack build, but no change. Cheers, Francis From david at ar.media.kyoto-u.ac.jp Tue Sep 16 02:49:29 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 16 Sep 2008 15:49:29 +0900 Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> Message-ID: <48CF56F9.9070206@ar.media.kyoto-u.ac.jp> Francis Drossaert wrote: > Hi everybody, > > I am trying to install python2.5/scipy/numpy/sympy/matplotlib locally, > because of various reasons. I am not root. Do you really need python 2.5 ? By building your own python, you are forcing yourself to build every package you will need for python, including the dependencies. For matplotlib, it will be painful. Python 2.4 is enough (incidentally, I have to use Centos 5 at some place, on the same architecture). For Blas/Lapack, how did you build them ? (Which version, which makefile, which fortran compiler). > However after installing I tried to import numpy in python and > received to following error: ImportError: > /users/francisd/local/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so: > undefined symbol: zgesdd_ What lapack_lite.so does depend on ? (ldd /users/francisd/local/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so) > > Googling this error, it seems that the error is caused by having a > wrong lapack version. I am running Centos 5 (basically Red Hat > Enterprise Linux 5) on a x86_64 machine. Unfortunately, it can be many things. Blas/Lapack/Atlas are difficult to build by yourself, there are tens of way to screw up at any point. If you give us the above information, it should be clearer where exactly an error was made cheers, David From stefan at sun.ac.za Tue Sep 16 04:43:31 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 16 Sep 2008 10:43:31 +0200 Subject: [Numpy-discussion] math directive, rest files and sphinx In-Reply-To: <48CF36FA.9060508@ar.media.kyoto-u.ac.jp> References: <48CE13BC.60908@ar.media.kyoto-u.ac.jp> <9457e7c80809150232m7f81304en33f695d8c6cb4f02@mail.gmail.com> <48CF36FA.9060508@ar.media.kyoto-u.ac.jp> Message-ID: <9457e7c80809160143i73b41d0ci868244f2005f9120@mail.gmail.com> Hi David 2008/9/16 David Cournapeau : > I managed to get further, but I don't get what I want. A minimal latex > code which has the problem is the following > > .. math:: \begin{matrix}x \\ y \end{array} > > This shows correctly with latex, but not with html (which shows xy > instead of a column with x in one row and y in the other). I guess this > is a mathml problem ? I expanded the mathml module in the reference guide source to handle constructs as they came up. I never did try to do matrices, though, so it's quite likely that it hasn't been implemented. > Also, I noticed a trivial but confusing error in error messages in > mathml.py. Shall I report those problems with fixes in the numpyref > package at launchpad ? IOW, what is the 'official' gate to this ? I'm not sure if there is an "official" version anywhere. The URL I got it from is in the top of the source, but I've modified the code since. I guess fixing our version on launchpad is a good start. > Finally, are are gearing toward math support from sphinx directly > (available in future 0.5), or will we keep our own ? While I like the mathml approach, it does not render perfectly on all browsers and, as you've discovered, doesn't support all the constructs. It may be easier to switch to the math module now in Sphinx, rather than to update the mathml module. I'll leave the decision up to Pauli and yourself; unfortunately I don't have time to work on this right now. Cheers St?fan From stefan at sun.ac.za Tue Sep 16 04:50:58 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 16 Sep 2008 10:50:58 +0200 Subject: [Numpy-discussion] Help speeding up element-wise operations for video processing In-Reply-To: <458186.45235.qm@web51601.mail.re2.yahoo.com> References: <458186.45235.qm@web51601.mail.re2.yahoo.com> Message-ID: <9457e7c80809160150i5acb33d7t95193c58a27ab76d@mail.gmail.com> Hi Brendan 2008/9/16 brendan simons : > #interpolate the green pixels from the bayer filter image ain > g = greenMask * ain > gi = g[:-2, 1:-1].astype('uint16') > gi += g[2:, 1:-1] > gi += g[1:-1, :-2] > gi += g[1:-1, 2:] > gi /= 4 > gi += g[1:-1, 1:-1] > return gi I may be completely off base here, but you should be able to do this *very* quickly using your GPU, or even just using OpenGL. Otherwise, coding it up in ctypes is easy as well (I can send you a code snippet, if you need). > I do something similar for red and blue, then stack the interpolated red, > green and blue integers into an array of 24 bit integers and blit to the > screen. > > I was hoping that none of the lines would have to iterate over pixels, and > would instead do the adds and multiplies as single operations. Perhaps numpy > has to iterate when copying a subset of an array? Is there a faster array > "crop" ? Any hints as to how I might code this part up using ctypes? Have you tried formulating this as a convolution, and using scipy.signal's 2-d convolve or fftconvolve? Cheers St?fan From F.Drossaert at googlemail.com Tue Sep 16 05:16:46 2008 From: F.Drossaert at googlemail.com (Francis) Date: Tue, 16 Sep 2008 02:16:46 -0700 (PDT) Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: <48CF56F9.9070206@ar.media.kyoto-u.ac.jp> References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <48CF56F9.9070206@ar.media.kyoto-u.ac.jp> Message-ID: <2a66de50-78a4-47ce-adac-611d808cfe80@a1g2000hsb.googlegroups.com> > Do you really need python 2.5 ? By building your own python, you are > forcing yourself to build every package you will need for python, > including the dependencies. For matplotlib, it will be painful. Python > 2.4 is enough (incidentally, I have to use Centos 5 at some place, on > the same architecture). I don't really need Python2.5. Python2.4 is actually available on the network but I think I will run into problems using the network one and try using local libraries. I assume one would need to change the paths on a file which I can't change. Anyway I installed Python2.5 without any problems. I guess the problem is combining numpy/scipy with the lapack/blas/atlas package. > For Blas/Lapack, how did you build them ? (Which version, which > makefile, which fortran compiler). I forgot to include the link: http://www.scipy.org/Installing_SciPy/. I basically followed the instructions mentioned at `building Atlas by hand`. The build did not result in any errors. I have also tried to build it using OPTS = -O2 -fPIC -m64 and NOOPT = -O0 -fPIC -m64 when building the Lapack. > > However after installing I tried to import numpy in python and > > received to following error: ?ImportError: > > /users/francisd/local/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so: > > undefined symbol: zgesdd_ > > What lapack_lite.so does depend on ? (ldd > /users/francisd/local/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so) Output: libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b4187472000) libc.so.6 => /lib64/libc.so.6 (0x00002b418768c000) /lib64/ld-linux-x86-64.so.2 (0x00000036b8800000) > > Googling this error, it seems that the error is caused by having a > > wrong lapack version. I am running Centos 5 (basically Red Hat > > Enterprise Linux 5) on a x86_64 machine. > > Unfortunately, it can be many things. Blas/Lapack/Atlas are difficult to > build by yourself, there are tens of way to screw up at any point. If > you give us the above information, it should be clearer where exactly an > error was made You make it sound as there is no easy solution. > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion From david at ar.media.kyoto-u.ac.jp Tue Sep 16 05:18:04 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 16 Sep 2008 18:18:04 +0900 Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: <2a66de50-78a4-47ce-adac-611d808cfe80@a1g2000hsb.googlegroups.com> References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <48CF56F9.9070206@ar.media.kyoto-u.ac.jp> <2a66de50-78a4-47ce-adac-611d808cfe80@a1g2000hsb.googlegroups.com> Message-ID: <48CF79CC.9090609@ar.media.kyoto-u.ac.jp> Francis wrote: > > I don't really need Python2.5. Python2.4 is actually available on the > network but I think I will run into problems using the network one and > try using local libraries. No, it is perfectly supported (I do it all the time, even on my own machines where I have root rights). You use --prefix when installing packages, and you need to make sure that PYTHONPATH is correctly set up, e.g.: python setup.py install --prefix=$HOME/local # will install in $HOME/local/lib/python2.4/site-packages for python2.4 And set PYTHONPATH to: PYTHONPATH=$HOME/local/lib/python2.4/site-packages:$PYTHONPATH. You can check which path you use by doing: python -c "import numpy; print numpy.__file__". > I assume one would need to change the paths > on a file which I can't change. Anyway I installed Python2.5 without > any problems. The problem is that when building software, you generally don't know whether you did it right or wrong before much later, by building other softwares on the top of it. For numpy/scipy, it is not too much of a problem, because they only depend on the interpreter, but for matplotlib, you will have to build pygtk/pyqt/etc... Since it does not make a difference to use your own python or the included python, you should use it. I personally use python2.4 on Centos to build numpy/scipy myself. > > Output: > > libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b4187472000) > libc.so.6 => /lib64/libc.so.6 (0x00002b418768c000) > /lib64/ld-linux-x86-64.so.2 (0x00000036b8800000) For some reason, lapack has not been linked. Can you build numpy from scratch and give us the build.log ? For example, in the source tree: rm -rf ./build && python setup.py build &> build.log cheers, David From F.Drossaert at googlemail.com Tue Sep 16 07:28:17 2008 From: F.Drossaert at googlemail.com (Francis) Date: Tue, 16 Sep 2008 04:28:17 -0700 (PDT) Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: <48CF79CC.9090609@ar.media.kyoto-u.ac.jp> References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <48CF56F9.9070206@ar.media.kyoto-u.ac.jp> <2a66de50-78a4-47ce-adac-611d808cfe80@a1g2000hsb.googlegroups.com> <48CF79CC.9090609@ar.media.kyoto-u.ac.jp> Message-ID: <07bbbcff-365f-4051-924b-dae8d353b0ed@8g2000hse.googlegroups.com> On 16 Sep, 10:18, David Cournapeau wrote: > Francis wrote: > > > I don't really need Python2.5. Python2.4 is actually available on the > > network but I think I will run into problems using the network one and > > try using local libraries. > > No, it is perfectly supported (I do it all the time, even on my own > machines where I have root rights). You use --prefix when installing > packages, and you need to make sure that PYTHONPATH is correctly set up, > e.g.: > > python setup.py install --prefix=$HOME/local # will install in > $HOME/local/lib/python2.4/site-packages for python2.4 > > And set PYTHONPATH to: > PYTHONPATH=$HOME/local/lib/python2.4/site-packages:$PYTHONPATH. > > You can check which path you use by doing: python -c "import numpy; > print numpy.__file__". Okay. That should not be much of a problem. > > ?I assume one would need to change the paths > > on a file which I can't change. Anyway I installed Python2.5 without > > any problems. > > The problem is that when building software, you generally don't know > whether you did it right or wrong before much later, by building other > softwares on the top of it. For numpy/scipy, it is not too much of a > problem, because they only depend on the interpreter, but for > matplotlib, you will have to build pygtk/pyqt/etc... Since it does not > make a difference to use your own python or the included python, you > should use it. I personally use python2.4 on Centos to build numpy/scipy > myself. > > > > > Output: > > > ? ? ? ? libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b4187472000) > > ? ? ? ? libc.so.6 => /lib64/libc.so.6 (0x00002b418768c000) > > ? ? ? ? /lib64/ld-linux-x86-64.so.2 (0x00000036b8800000) > > For some reason, lapack has not been linked. Can you build numpy from > scratch and give us the build.log ? For example, in the source tree: > > rm -rf ./build && python setup.py build &> build.log > > cheers, > > David I only included the first and last part which I think are relevant. F2PY Version 2_5585 blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['/users/francisd/local/lib/atlas'] language = c customize GnuFCompiler Found executable /usr/bin/g77 gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config compiling '_configtest.c': /* This file is generated from numpy/distutils/system_info.py */ void ATL_buildinfo(void); int main(void) { ATL_buildinfo(); return 0; } C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe - Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector -- param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC compile options: '-c' gcc: _configtest.c gcc -pthread _configtest.o -L/users/francisd/local/lib/atlas -llapack - lf77blas -lcblas -latlas -o _configtest ATLAS version 3.8.2 built by francisd on Mon Sep 15 15:20:08 BST 2008: UNAME : Linux red-67 2.6.18-92.el5 #1 SMP Tue Jun 10 18:51:06 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux INSTFLG : -1 0 -a 1 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_PIII -DATL_CPUMHZ=2500 - DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_USE64BITS -DATL_GAS_x8664 F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 262144 F77 : gfortran, version GNU Fortran (GCC) 4.1.2 20071124 (Red Hat 4.1.2-42) F77FLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 - fPIC -m64 SMC : gcc, version gcc (GCC) 4.1.2 20071124 (Red Hat 4.1.2-42) SMCFLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 - fPIC -m64 SKC : gcc, version gcc (GCC) 4.1.2 20071124 (Red Hat 4.1.2-42) SKCFLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 - fPIC -m64 success! removing: _configtest.c _configtest.o _configtest FOUND: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['/users/francisd/local/lib/atlas'] language = c define_macros = [('ATLAS_INFO', '"\\"3.8.2\\""')] lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries lapack_atlas not found in /users/francisd/local/lib/atlas numpy.distutils.system_info.atlas_threads_info Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['lapack', 'lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['/users/francisd/local/lib/atlas'] language = f77 customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config compiling '_configtest.c': /* This file is generated from numpy/distutils/system_info.py */ void ATL_buildinfo(void); int main(void) { ATL_buildinfo(); return 0; } C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe - Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector -- param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC compile options: '-c' gcc: _configtest.c gcc -pthread _configtest.o -L/users/francisd/local/lib/atlas -llapack - llapack -lf77blas -lcblas -latlas -o _configtest ATLAS version 3.8.2 built by francisd on Mon Sep 15 15:20:08 BST 2008: UNAME : Linux red-67 2.6.18-92.el5 #1 SMP Tue Jun 10 18:51:06 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux INSTFLG : -1 0 -a 1 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_PIII -DATL_CPUMHZ=2500 - DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_USE64BITS -DATL_GAS_x8664 F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 262144 F77 : gfortran, version GNU Fortran (GCC) 4.1.2 20071124 (Red Hat 4.1.2-42) F77FLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 - fPIC -m64 SMC : gcc, version gcc (GCC) 4.1.2 20071124 (Red Hat 4.1.2-42) SMCFLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 - fPIC -m64 SKC : gcc, version gcc (GCC) 4.1.2 20071124 (Red Hat 4.1.2-42) SKCFLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 - fPIC -m64 success! removing: _configtest.c _configtest.o _configtest FOUND: libraries = ['lapack', 'lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['/users/francisd/local/lib/atlas'] language = f77 define_macros = [('ATLAS_INFO', '"\\"3.8.2\\""')] running build running scons . . . . . creating build/temp.linux-x86_64-2.4/numpy/fft compile options: '-Inumpy/core/include -Ibuild/src.linux-x86_64-2.4/ numpy/core/include/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/ include/python2.4 -I/usr/include/python2.4 -c' gcc: numpy/fft/fftpack_litemodule.c gcc: numpy/fft/fftpack.c gcc -pthread -shared build/temp.linux-x86_64-2.4/numpy/fft/ fftpack_litemodule.o build/temp.linux-x86_64-2.4/numpy/fft/fftpack.o - o build/lib.linux-x86_64-2.4/numpy/fft/fftpack_lite.so building 'numpy.linalg.lapack_lite' extension compiling C sources C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe - Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector -- param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC creating build/temp.linux-x86_64-2.4/numpy/linalg compile options: '-DATLAS_INFO="\"3.8.2\"" -Inumpy/core/include - Ibuild/src.linux-x86_64-2.4/numpy/core/include/numpy -Inumpy/core/src - Inumpy/core/include -I/usr/include/python2.4 -I/usr/include/python2.4 - c' gcc: numpy/linalg/lapack_litemodule.c gcc: numpy/linalg/python_xerbla.c /usr/bin/g77 -g -Wall -g -Wall -shared build/temp.linux-x86_64-2.4/ numpy/linalg/lapack_litemodule.o build/temp.linux-x86_64-2.4/numpy/ linalg/python_xerbla.o -L/users/francisd/local/lib/atlas -llapack - llapack -lf77blas -lcblas -latlas -lg2c -o build/lib.linux-x86_64-2.4/ numpy/linalg/lapack_lite.so building 'numpy.random.mtrand' extension compiling C sources C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe - Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector -- param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC creating build/temp.linux-x86_64-2.4/numpy/random creating build/temp.linux-x86_64-2.4/numpy/random/mtrand compile options: '-Inumpy/core/include -Ibuild/src.linux-x86_64-2.4/ numpy/core/include/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/ include/python2.4 -I/usr/include/python2.4 -c' gcc: numpy/random/mtrand/initarray.c gcc: numpy/random/mtrand/distributions.c gcc: numpy/random/mtrand/randomkit.c gcc: numpy/random/mtrand/mtrand.c gcc -pthread -shared build/temp.linux-x86_64-2.4/numpy/random/mtrand/ mtrand.o build/temp.linux-x86_64-2.4/numpy/random/mtrand/randomkit.o build/temp.linux-x86_64-2.4/numpy/random/mtrand/initarray.o build/ temp.linux-x86_64-2.4/numpy/random/mtrand/distributions.o -o build/ lib.linux-x86_64-2.4/numpy/random/mtrand.so running build_scripts creating build/scripts.linux-x86_64-2.4 Creating build/scripts.linux-x86_64-2.4/f2py adding 'build/scripts.linux-x86_64-2.4/f2py' to scripts changing mode of build/scripts.linux-x86_64-2.4/f2py from 664 to 775 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion From david at ar.media.kyoto-u.ac.jp Tue Sep 16 07:22:37 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 16 Sep 2008 20:22:37 +0900 Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: <07bbbcff-365f-4051-924b-dae8d353b0ed@8g2000hse.googlegroups.com> References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <48CF56F9.9070206@ar.media.kyoto-u.ac.jp> <2a66de50-78a4-47ce-adac-611d808cfe80@a1g2000hsb.googlegroups.com> <48CF79CC.9090609@ar.media.kyoto-u.ac.jp> <07bbbcff-365f-4051-924b-dae8d353b0ed@8g2000hse.googlegroups.com> Message-ID: <48CF96FD.9090805@ar.media.kyoto-u.ac.jp> Francis wrote: > > > ATLAS version 3.8.2 built by francisd on Mon Sep 15 15:20:08 BST 2008: > UNAME : Linux red-67 2.6.18-92.el5 #1 SMP Tue Jun 10 18:51:06 > EDT 2008 x86_64 x86_64 x86_64 GNU/Linux > INSTFLG : -1 0 -a 1 > ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_PIII -DATL_CPUMHZ=2500 - > DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_USE64BITS -DATL_GAS_x8664 > F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle > CACHEEDGE: 262144 > F77 : gfortran, version GNU Fortran (GCC) 4.1.2 20071124 (Red > Hat 4.1.2-42) ^^ You built atlas with gfortran > /usr/bin/g77 -g -Wall -g -Wall -shared build/temp.linux-x86_64-2.4/ > numpy/linalg/lapack_litemodule.o build/temp.linux-x86_64-2.4/numpy/ > linalg/python_xerbla.o -L/users/francisd/local/lib/atlas -llapack - > llapack -lf77blas -lcblas -latlas -lg2c -o build/lib.linux-x86_64-2.4/ > numpy/linalg/lapack_lite.so ^^ Here you use g77. This cannot work. You have to use the same fortran compiler for *everything*, because g77 and gfortran are not ABI compatible. To build numpy and scipy with gfortran, do: python setup.py build --fcompiler=gnu95 But I still find a bit strange that it did not fail during the link. You may have other errors, cheers, David From faltet at pytables.org Tue Sep 16 08:54:05 2008 From: faltet at pytables.org (Francesc Alted) Date: Tue, 16 Sep 2008 14:54:05 +0200 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: References: Message-ID: <200809161454.06195.faltet@pytables.org> A Sunday 14 September 2008, Jarrod Millman escrigu?: > Hello, > > The 1.2.0rc2 is now available: > http://svn.scipy.org/svn/numpy/tags/1.2.0rc2 > > The source tarball is here: > https://cirl.berkeley.edu/numpy/numpy-1.2.0rc2.tar.gz > > Here is the universal Mac binary: > https://cirl.berkeley.edu/numpy/numpy-1.2.0rc2-py2.5-macosx10.5.dmg > > Here are the Window's binaries: > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy- >1.2.0rc2-win32-superpack-python2.4.exe > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy- >1.2.0rc2-win32-superpack-python2.5.exe > > Here are the release notes: > http://scipy.org/scipy/numpy/milestone/1.2.0 > > Please test this release ASAP and let us know if there are any > problems. If there are no show stoppers, this will become the > 1.2.0 release. Seems that there is a small glitch here: ====================================================================== ERROR: Failure: ImportError (No module named scipy) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib64/python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/loader.py", line 364, in loadTestsFromName addr.filename, addr.module) File "/usr/local/lib64/python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/local/lib64/python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/importer.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/usr/local/lib64/python2.5/site-packages/numpy/doc/__init__.py", line 10, in __import__(__name__ + '.' + f) File "/usr/local/lib64/python2.5/site-packages/numpy/doc/example.py", line 27, in import scipy as sp ImportError: No module named scipy ---------------------------------------------------------------------- Ran 1727 tests in 12.617s FAILED (KNOWNFAIL=1, errors=1) Running unit tests for numpy NumPy version 1.2.0.dev5825 NumPy is installed in /usr/local/lib64/python2.5/site-packages/numpy Python version 2.5.2 (r252:60911, Aug 1 2008, 00:37:21) [GCC 4.3.1 20080507 (prerelease) [gcc-4_3-branch revision 135036]] nose version 0.10.3 Using OpenSUSE GNU/Linux 11.0 64-bit and Python 2.5. I've been unable to run the tests with Python 2.6b3 (nose seems that does not work yet against Python 2.6). However, I've run the complete test suite of PyTables Pro (using Python 2.6 now) against NumPy 1.2 and everthing went fine: -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= PyTables version: 2.1b2.devpro HDF5 version: 1.8.1 NumPy version: 1.2.0.dev5825 Zlib version: 1.2.3 LZO version: 2.02 (Oct 17 2005) BZIP2 version: 1.0.5 (10-Dec-2007) Python version: 2.6b3 (r26b3:65922, Sep 8 2008, 18:38:56) [GCC 4.3.1 20080507 (prerelease) [gcc-4_3-branch revision 135036]] Platform: linux2-x86_64 Byte-ordering: little -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Performing the complete test suite! -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Skipping Numeric test suite. Skipping numarray test suite. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= [clip] ........................................... ---------------------------------------------------------------------- Ran 52443 tests in 11592.913s OK Also, it seems that NumPy 1.2 is relatively free of leaks or such a huge test suite would have exploded violently. Good job! -- Francesc Alted From charlesr.harris at gmail.com Tue Sep 16 09:28:26 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 16 Sep 2008 07:28:26 -0600 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: <200809161454.06195.faltet@pytables.org> References: <200809161454.06195.faltet@pytables.org> Message-ID: On Tue, Sep 16, 2008 at 6:54 AM, Francesc Alted wrote: > A Sunday 14 September 2008, Jarrod Millman escrigu?: > > Hello, > > > > The 1.2.0rc2 is now available: > > http://svn.scipy.org/svn/numpy/tags/1.2.0rc2 > > > > The source tarball is here: > > https://cirl.berkeley.edu/numpy/numpy-1.2.0rc2.tar.gz > > > > Here is the universal Mac binary: > > https://cirl.berkeley.edu/numpy/numpy-1.2.0rc2-py2.5-macosx10.5.dmg > > > > Here are the Window's binaries: > > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy- > >1.2.0rc2-win32-superpack-python2.4.exe > > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy- > >1.2.0rc2-win32-superpack-python2.5.exe > > > > Here are the release notes: > > http://scipy.org/scipy/numpy/milestone/1.2.0 > > > > Please test this release ASAP and let us know if there are any > > problems. If there are no show stoppers, this will become the > > 1.2.0 release. > > Seems that there is a small glitch here: > > ====================================================================== > ERROR: Failure: ImportError (No module named scipy) > ---------------------------------------------------------------------- > Traceback (most recent call last): > Can you delete the installed numpy directory and try a clean install? I think you might be seeing stale files. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at pytables.org Tue Sep 16 09:37:21 2008 From: faltet at pytables.org (Francesc Alted) Date: Tue, 16 Sep 2008 15:37:21 +0200 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: References: <200809161454.06195.faltet@pytables.org> Message-ID: <200809161537.21206.faltet@pytables.org> A Tuesday 16 September 2008, Charles R Harris escrigu?: > On Tue, Sep 16, 2008 at 6:54 AM, Francesc Alted wrote: > > A Sunday 14 September 2008, Jarrod Millman escrigu?: > > > Hello, > > > > > > The 1.2.0rc2 is now available: > > > http://svn.scipy.org/svn/numpy/tags/1.2.0rc2 > > > > > > The source tarball is here: > > > https://cirl.berkeley.edu/numpy/numpy-1.2.0rc2.tar.gz > > > > > > Here is the universal Mac binary: > > > https://cirl.berkeley.edu/numpy/numpy-1.2.0rc2-py2.5-macosx10.5.d > > >mg > > > > > > Here are the Window's binaries: > > > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/nu > > >mpy- 1.2.0rc2-win32-superpack-python2.4.exe > > > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/nu > > >mpy- 1.2.0rc2-win32-superpack-python2.5.exe > > > > > > Here are the release notes: > > > http://scipy.org/scipy/numpy/milestone/1.2.0 > > > > > > Please test this release ASAP and let us know if there are any > > > problems. If there are no show stoppers, this will become the > > > 1.2.0 release. > > > > Seems that there is a small glitch here: > > > > =================================================================== > >=== ERROR: Failure: ImportError (No module named scipy) > > ------------------------------------------------------------------- > >--- Traceback (most recent call last): > > Can you delete the installed numpy directory and try a clean install? > I think you might be seeing stale files. You are right: ---------------------------------------------------------------------- Ran 1726 tests in 12.981s OK (KNOWNFAIL=1) Running unit tests for numpy NumPy version 1.2.0.dev5825 NumPy is installed in /usr/local/lib64/python2.5/site-packages/numpy Python version 2.5.2 (r252:60911, Aug 1 2008, 00:37:21) [GCC 4.3.1 20080507 (prerelease) [gcc-4_3-branch revision 135036]] nose version 0.10.3 Excellent! -- Francesc Alted From david.huard at gmail.com Tue Sep 16 10:04:33 2008 From: david.huard at gmail.com (David Huard) Date: Tue, 16 Sep 2008 10:04:33 -0400 Subject: [Numpy-discussion] Help speeding up element-wise operations for video processing In-Reply-To: <9457e7c80809160150i5acb33d7t95193c58a27ab76d@mail.gmail.com> References: <458186.45235.qm@web51601.mail.re2.yahoo.com> <9457e7c80809160150i5acb33d7t95193c58a27ab76d@mail.gmail.com> Message-ID: <91cf711d0809160704n48c0e3c0t389e2649317330eb@mail.gmail.com> Brendan, Not sure if I understand correctly what you want, but ... Numpy vector operations are performed in C, so there will be an iteration over the array elements. For parallel operations over all pixels, you'd need a package that talks to your GPU, such as pyGPU. I've never tried it and if you do, please report your experience, I'd be very interested to hear about it. HTH, David On Tue, Sep 16, 2008 at 4:50 AM, St?fan van der Walt wrote: > Hi Brendan > > 2008/9/16 brendan simons : > > #interpolate the green pixels from the bayer filter image ain > > g = greenMask * ain > > gi = g[:-2, 1:-1].astype('uint16') > > gi += g[2:, 1:-1] > > gi += g[1:-1, :-2] > > gi += g[1:-1, 2:] > > gi /= 4 > > gi += g[1:-1, 1:-1] > > return gi > > I may be completely off base here, but you should be able to do this > *very* quickly using your GPU, or even just using OpenGL. Otherwise, > coding it up in ctypes is easy as well (I can send you a code snippet, > if you need). > > > I do something similar for red and blue, then stack the interpolated red, > > green and blue integers into an array of 24 bit integers and blit to the > > screen. > > > > I was hoping that none of the lines would have to iterate over pixels, > and > > would instead do the adds and multiplies as single operations. Perhaps > numpy > > has to iterate when copying a subset of an array? Is there a faster > array > > "crop" ? Any hints as to how I might code this part up using ctypes? > > Have you tried formulating this as a convolution, and using > scipy.signal's 2-d convolve or fftconvolve? > > Cheers > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Tue Sep 16 10:09:18 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 16 Sep 2008 23:09:18 +0900 Subject: [Numpy-discussion] math directive, rest files and sphinx In-Reply-To: <9457e7c80809160143i73b41d0ci868244f2005f9120@mail.gmail.com> References: <48CE13BC.60908@ar.media.kyoto-u.ac.jp> <9457e7c80809150232m7f81304en33f695d8c6cb4f02@mail.gmail.com> <48CF36FA.9060508@ar.media.kyoto-u.ac.jp> <9457e7c80809160143i73b41d0ci868244f2005f9120@mail.gmail.com> Message-ID: <48CFBE0E.3090003@ar.media.kyoto-u.ac.jp> Hi St?fan St?fan van der Walt wrote: > > I expanded the mathml module in the reference guide source to handle > constructs as they came up. I never did try to do matrices, though, > so it's quite likely that it hasn't been implemented. > Does that mean you have to implement latex parsing by yourself ? > > I'm not sure if there is an "official" version anywhere. The URL I > got it from is in the top of the source, but I've modified the code > since. I guess fixing our version on launchpad is a good start. Ok, thanks. I thought this was done for numpy, but I guess not. > > While I like the mathml approach, it does not render perfectly on all > browsers and, as you've discovered, doesn't support all the > constructs. It may be easier to switch to the math module now in > Sphinx, rather than to update the mathml module. I'll leave the > decision up to Pauli and yourself; I don't think I am entitled to make any decision here: I have no knowledge of the stack involved here. I can only confirm than using sphinx 0.5 (the dev version) does work for latex constructs; I could not make the JS backend work, but again, I know nothing about web frameworks and co, and I guess setting up a webserver for this to work to be trivial for anyone who knows his stuff here. cheers, David From stefan at sun.ac.za Tue Sep 16 10:39:29 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 16 Sep 2008 16:39:29 +0200 Subject: [Numpy-discussion] math directive, rest files and sphinx In-Reply-To: <48CFBE0E.3090003@ar.media.kyoto-u.ac.jp> References: <48CE13BC.60908@ar.media.kyoto-u.ac.jp> <9457e7c80809150232m7f81304en33f695d8c6cb4f02@mail.gmail.com> <48CF36FA.9060508@ar.media.kyoto-u.ac.jp> <9457e7c80809160143i73b41d0ci868244f2005f9120@mail.gmail.com> <48CFBE0E.3090003@ar.media.kyoto-u.ac.jp> Message-ID: <9457e7c80809160739q53d19efbj45f3e9af6a027a44@mail.gmail.com> 2008/9/16 David Cournapeau : >> I expanded the mathml module in the reference guide source to handle >> constructs as they came up. I never did try to do matrices, though, >> so it's quite likely that it hasn't been implemented. > > Does that mean you have to implement latex parsing by yourself ? That's what mathml.py does already, but you'll have to modify it. > I don't think I am entitled to make any decision here: I have no > knowledge of the stack involved here. I can only confirm than using > sphinx 0.5 (the dev version) does work for latex constructs; I could not > make the JS backend work, but again, I know nothing about web frameworks > and co, and I guess setting up a webserver for this to work to be > trivial for anyone who knows his stuff here. Pauli used dvipng to do it on the web framework, and that worked very well. Maybe we should just switch and save ourselves a big of headache. Cheers St?fan From LisaAnnDelgado at gmail.com Tue Sep 16 11:05:21 2008 From: LisaAnnDelgado at gmail.com (Lisa Delgado) Date: Tue, 16 Sep 2008 15:05:21 +0000 (UTC) Subject: [Numpy-discussion] ValueError: setting an array element with a sequence. Message-ID: Hi, I had previously run a program successfully when using numpy 1.1.0. I just downloaded the Enthought Python Distribution, which has numpy 1.0.4 and am now having problems running my program. I get the following error. Traceback (most recent call last): File "/Users/Lisa/Documents/Dissertation work/PythonPrograms/NashEq.py", line 57, in EffortCosts[i,j] = buyeragents[i][j] ValueError: setting an array element with a sequence. Can anyone help me with this? Thank you. Lisa From dalke at dalkescientific.com Tue Sep 16 11:11:04 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Tue, 16 Sep 2008 17:11:04 +0200 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: References: Message-ID: <3EA7103F-23FC-4078-8255-75ADDD978EA8@dalkescientific.com> On Sep 14, 2008, at 1:00 AM, Jarrod Millman wrote: > Please test this release ASAP and let us know if there are any > problems. If there are no show stoppers, this will become the > 1.2.0 release. Running on OS X 10.4 , testing with 'import numpy; numpy.test()' Ran 1726 tests in 7.544s OK (KNOWNFAIL=1) Also, I'm pleased to see that most of my patches for improved startup performance were applied. Here are timings for 1.2.0rc2 [josiah:~] dalke% time python -c 'import os' 0.014u 0.029s 0:00.04 75.0% 0+0k 0+1io 0pf+0w [josiah:~] dalke% time python -c 'import numpy' 0.067u 0.123s 0:00.19 94.7% 0+0k 0+9io 0pf+0w [josiah:~] dalke% Only 0.15s extra startup time. Vs. 0.34 seconds with 1.1.0: (sudo rm .../numpy; cd 1.1.0; setup.py install) [josiah:~] dalke% time python -c 'import os' 0.014u 0.028s 0:00.04 75.0% 0+0k 0+0io 0pf+0w [josiah:~] dalke% time python -c 'import numpy' 0.160u 0.227s 0:00.38 100.0% 0+0k 0+0io 0pf+0w [josiah:~] dalke% About twice as fast. Programs that ran in under a second should now be at least 30% faster. :) > Andrew dalke at dalkescientific.com From F.Drossaert at googlemail.com Tue Sep 16 11:22:15 2008 From: F.Drossaert at googlemail.com (Francis) Date: Tue, 16 Sep 2008 08:22:15 -0700 (PDT) Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: <48CF96FD.9090805@ar.media.kyoto-u.ac.jp> References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <48CF56F9.9070206@ar.media.kyoto-u.ac.jp> <2a66de50-78a4-47ce-adac-611d808cfe80@a1g2000hsb.googlegroups.com> <48CF79CC.9090609@ar.media.kyoto-u.ac.jp> <07bbbcff-365f-4051-924b-dae8d353b0ed@8g2000hse.googlegroups.com> <48CF96FD.9090805@ar.media.kyoto-u.ac.jp> Message-ID: <70cd1d75-3a32-4140-8a72-d2cdd650ef62@m3g2000hsc.googlegroups.com> You are right. I have recompiled everything with gfortran, but now I get the following error when I try to import numpy: undefined symbol: _gfortran_concat_string. I have build a while ago sage-math (where everything is build locally), but I delete it since it was over 1GB and I am only interested in numpy+scipy+sympy+matlibplot. Although I did not test it extensive, importing numpy and scipy was not a problem. Perhaps I could try it again and try to use the libraries they use to install numpy if it is clear how it is done. From david at ar.media.kyoto-u.ac.jp Tue Sep 16 11:13:12 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 17 Sep 2008 00:13:12 +0900 Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: <70cd1d75-3a32-4140-8a72-d2cdd650ef62@m3g2000hsc.googlegroups.com> References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <48CF56F9.9070206@ar.media.kyoto-u.ac.jp> <2a66de50-78a4-47ce-adac-611d808cfe80@a1g2000hsb.googlegroups.com> <48CF79CC.9090609@ar.media.kyoto-u.ac.jp> <07bbbcff-365f-4051-924b-dae8d353b0ed@8g2000hse.googlegroups.com> <48CF96FD.9090805@ar.media.kyoto-u.ac.jp> <70cd1d75-3a32-4140-8a72-d2cdd650ef62@m3g2000hsc.googlegroups.com> Message-ID: <48CFCD08.10109@ar.media.kyoto-u.ac.jp> Francis wrote: > You are right. I have recompiled everything with gfortran, but now I > get the following error when I try to import numpy: undefined symbol: > _gfortran_concat_string. This means something at some point was not built with gfortran, or that you did not use gfortran to build numpy. Which .so file generates this error ? Can you give us the result of ldd on that .so ? Also, are you sure libraries paths are correctly propagated ? In particular, you should have LD_LIBRARY_PATH set up correctly to where you put your bustom-built libraries. > > I have build a while ago sage-math (where everything is build > locally), but I delete it since it was over 1GB and I am only > interested in numpy+scipy+sympy+matlibplot. Although I did not test it > extensive, importing numpy and scipy was not a problem. Yes, I think they package everything by themselves. I am starting to think that we should do the same for blas/lapack, that's also how R does it by default, and I am so tired of seeing the same errors coming again and again. I think that distributions will never package blas/lapack correctly, it is hopeless. You could also try EPD, cheers, David From aisaac at american.edu Tue Sep 16 11:42:43 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 16 Sep 2008 11:42:43 -0400 Subject: [Numpy-discussion] ValueError: setting an array element with sequence. In-Reply-To: References: Message-ID: <48CFD3F3.3070003@american.edu> On 9/16/2008 11:05 AM Lisa Delgado apparently wrote: > I had previously run a program successfully when using numpy 1.1.0. I just > downloaded the Enthought Python Distribution, which has numpy 1.0.4 and am now > having problems running my program. I get the following error. > > Traceback (most recent call last): > File "/Users/Lisa/Documents/Dissertation work/PythonPrograms/NashEq.py", line > 57, in > EffortCosts[i,j] = buyeragents[i][j] > ValueError: setting an array element with a sequence. > > Can anyone help me with this? It looks from the error that buyeragents[i][j] is a sequence, while you expect it to be a number. Most likely, this is *not* due to your change of NumPy version, but rather to some change you have made in the descriptionof buyeragents. Alan Isaac PS Is this your own program for finding Nash equilibrium? If not, which one are you using? From mailinglist.honeypot at gmail.com Tue Sep 16 11:57:22 2008 From: mailinglist.honeypot at gmail.com (Steve Lianoglou) Date: Tue, 16 Sep 2008 11:57:22 -0400 Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: <48CFCD08.10109@ar.media.kyoto-u.ac.jp> References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <48CF56F9.9070206@ar.media.kyoto-u.ac.jp> <2a66de50-78a4-47ce-adac-611d808cfe80@a1g2000hsb.googlegroups.com> <48CF79CC.9090609@ar.media.kyoto-u.ac.jp> <07bbbcff-365f-4051-924b-dae8d353b0ed@8g2000hse.googlegroups.com> <48CF96FD.9090805@ar.media.kyoto-u.ac.jp> <70cd1d75-3a32-4140-8a72-d2cdd650ef62@m3g2000hsc.googlegroups.com> <48CFCD08.10109@ar.media.kyoto-u.ac.jp> Message-ID: <33BBC043-1283-4C4D-8826-747C894BDE24@gmail.com> Hi, On Sep 16, 2008, at 11:13 AM, David Cournapeau wrote: > Yes, I think they package everything by themselves. I am starting to > think that we should do the same for blas/lapack, that's also how R > does > it by default, and I am so tired of seeing the same errors coming > again > and again. I think that distributions will never package blas/lapack > correctly, it is hopeless. Just out of curiosity. Humor me for a bit and let's assume that in many cases, the machines that people are building numpy/scipy on will most likely also have some install of R. Perhaps it's a compute server in your lab, or your own machine (for which pre-built versions of R most likely exist anyway). Is it possible to conjure up some special flag people can use during a numpy/scipy build/install so that they are able to link against the packaged libraries in the canonical install of R for their platform, or is that a totally bone-headed idea? -steve From charlesr.harris at gmail.com Tue Sep 16 12:05:03 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 16 Sep 2008 10:05:03 -0600 Subject: [Numpy-discussion] ValueError: setting an array element with a sequence. In-Reply-To: References: Message-ID: On Tue, Sep 16, 2008 at 9:05 AM, Lisa Delgado wrote: > Hi, > > I had previously run a program successfully when using numpy 1.1.0. I just > downloaded the Enthought Python Distribution, which has numpy 1.0.4 and am > now > having problems running my program. I get the following error. > > Traceback (most recent call last): > File "/Users/Lisa/Documents/Dissertation work/PythonPrograms/NashEq.py", > line > 57, in > EffortCosts[i,j] = buyeragents[i][j] > ValueError: setting an array element with a sequence. > What type is buyeragents? Numpy 1.1.0 fixes a number of bugs in 1.0.4 and you may have been bitten by that. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From doutriaux1 at llnl.gov Tue Sep 16 12:16:52 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Tue, 16 Sep 2008 09:16:52 -0700 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: References: Message-ID: <48CFDBF4.5070705@llnl.gov> On this system, both 1.2rc1 and rc2 hang on mtrand.c Linux thunder0 2.6.9-74chaos #1 SMP Wed Oct 24 08:41:12 PDT 2007 ia64 ia64 ia64 GNU/Linux attached is the log. -------------- next part -------------- A non-text attachment was scrubbed... Name: numpy.LOG.bz2 Type: application/x-bzip Size: 5201 bytes Desc: not available URL: From e-alerts at proteinkemi.dk Tue Sep 16 12:33:07 2008 From: e-alerts at proteinkemi.dk (Tommy Carstensen) Date: Tue, 16 Sep 2008 17:33:07 +0100 Subject: [Numpy-discussion] numpy.linalg.eig In-Reply-To: <3d375d730809150938x6bc994aai197dfa28c5e50092@mail.gmail.com> References: <003f01c91742$54b5c9d0$fe215d70$@dk> <3d375d730809150938x6bc994aai197dfa28c5e50092@mail.gmail.com> Message-ID: <00a001c91819$e6762040$b36260c0$@dk> Dear Robert Kern, Thanks for your interest in helping me. Previously I used LinearAlgebra.Heigenvectors. I should have switched to numpy.linalg.eigh instead of numpy.linalg.eig, since I work with Hermitian (non-complex and symmetric) matrices. Kind regards, Tommy -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Robert Kern Sent: 15 September 2008 17:39 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] numpy.linalg.eig On Mon, Sep 15, 2008 at 09:50, Tommy Carstensen wrote: > To the numpy mailing list, > > Previously I used python2.3 and the module LinearAlgebra for matrix > diagonalization. I recently upgraded to python2.5 and I am now using > numpy.linalg.eig for matrix diagonalization. This is approximately 3 times > slower than previously. I am using the same work station as previously and > it has all lapack libraries installed to my knowledge. Does anyone have an > explanation as to why I am experiencing the reduced speed of calculation? Although you have the same LAPACK libraries installed, Numeric and numpy may have been built differently, and numpy may not be using the same libraries. Can you check? > This is my first post to this mailing list, so I hope this mail complies > with the mailing list standards. Yes, certainly. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion From F.Drossaert at googlemail.com Tue Sep 16 12:59:57 2008 From: F.Drossaert at googlemail.com (Francis) Date: Tue, 16 Sep 2008 09:59:57 -0700 (PDT) Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: <48CFCD08.10109@ar.media.kyoto-u.ac.jp> References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <48CF56F9.9070206@ar.media.kyoto-u.ac.jp> <2a66de50-78a4-47ce-adac-611d808cfe80@a1g2000hsb.googlegroups.com> <48CF79CC.9090609@ar.media.kyoto-u.ac.jp> <07bbbcff-365f-4051-924b-dae8d353b0ed@8g2000hse.googlegroups.com> <48CF96FD.9090805@ar.media.kyoto-u.ac.jp> <70cd1d75-3a32-4140-8a72-d2cdd650ef62@m3g2000hsc.googlegroups.com> <48CFCD08.10109@ar.media.kyoto-u.ac.jp> Message-ID: On 16 Sep, 16:13, David Cournapeau wrote: > Francis wrote: > > You are right. I have recompiled everything with gfortran, but now I > > get the following error when I try to import numpy: undefined symbol: > > _gfortran_concat_string. > > This means something at some point was not built with gfortran, or that > you did not use gfortran to build numpy. Which .so file generates this > error ? Can you give us the result of ldd on that .so ? To be frank, I am bit of noob, but not completely. I use the gfortran make.inc to build lapack. The make.inc file to build and install atlas is using gfortran. I used the --fcompiler=gfortran flag. I don't think there is much else I can do. > Also, are you sure libraries paths are correctly propagated ? In > particular, you should have LD_LIBRARY_PATH set up correctly to where > you put your bustom-built libraries. Being a noob, I just followed the instructions on http://www.scipy.org/Installing_SciPy/Linux. > > I have build a while ago sage-math (where everything is build > > locally), but I delete it since it was over 1GB and I am only > > interested in numpy+scipy+sympy+matlibplot. Although I did not test it > > extensive, importing numpy and scipy was not a problem. > > Yes, I think they package everything by themselves. I am starting to > think that we should do the same for blas/lapack, that's also how R does > it by default, and I am so tired of seeing the same errors coming again > and again. I think that distributions will never package blas/lapack > correctly, it is hopeless. > > You could also try EPD, Unfortunately there is no EPD installer for Centos 5 version. There was a blog how to install it, but that look a lot more complicated than what I am trying now. I will try to build sage and look if I can build numpy against the atlas libraries in sage. Otherwise I use sage and remove all the packages that are not relevant for me, e.g. maxima, gap, singular, etc. From charlesr.harris at gmail.com Tue Sep 16 13:08:27 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 16 Sep 2008 11:08:27 -0600 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: <48CFDBF4.5070705@llnl.gov> References: <48CFDBF4.5070705@llnl.gov> Message-ID: On Tue, Sep 16, 2008 at 10:16 AM, Charles Doutriaux wrote: > On this system, both 1.2rc1 and rc2 hang on mtrand.c > > Linux thunder0 2.6.9-74chaos #1 SMP Wed Oct 24 08:41:12 PDT 2007 ia64 ia64 > ia64 GNU/Linux > > attached is the log. > > Interesting. What distribution are you running on that hardware? Any chance you could add your machine to the buildbots? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From doutriaux1 at llnl.gov Tue Sep 16 13:23:39 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Tue, 16 Sep 2008 10:23:39 -0700 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: References: <48CFDBF4.5070705@llnl.gov> Message-ID: <48CFEB9B.3010909@llnl.gov> thunder0 ~/svn >more /etc/redhat-release Red Hat Enterprise Linux AS release 4 (Nahant Update 5) I guess it's kind of old... And unfortunately, the lab won't let us add this machine to anything.... C. Charles R Harris wrote: > > > On Tue, Sep 16, 2008 at 10:16 AM, Charles Doutriaux > > wrote: > > On this system, both 1.2rc1 and rc2 hang on mtrand.c > > Linux thunder0 2.6.9-74chaos #1 SMP Wed Oct 24 08:41:12 PDT 2007 > ia64 ia64 ia64 GNU/Linux > > attached is the log. > > > Interesting. What distribution are you running on that hardware? Any > chance you could add your machine to the buildbots? > > Chuck > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http:// projects.scipy.org/mailman/listinfo/numpy-discussion > From pav at iki.fi Tue Sep 16 13:33:09 2008 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 16 Sep 2008 17:33:09 +0000 (UTC) Subject: [Numpy-discussion] math directive, rest files and sphinx References: <48CE13BC.60908@ar.media.kyoto-u.ac.jp> <9457e7c80809150232m7f81304en33f695d8c6cb4f02@mail.gmail.com> <48CF36FA.9060508@ar.media.kyoto-u.ac.jp> <9457e7c80809160143i73b41d0ci868244f2005f9120@mail.gmail.com> <48CFBE0E.3090003@ar.media.kyoto-u.ac.jp> Message-ID: Tue, 16 Sep 2008 23:09:18 +0900, David Cournapeau wrote: > St?fan van der Walt wrote: [clip] >> While I like the mathml approach, it does not render perfectly on all >> browsers and, as you've discovered, doesn't support all the constructs. >> It may be easier to switch to the math module now in Sphinx, rather >> than to update the mathml module. I'll leave the decision up to Pauli >> and yourself; > > I don't think I am entitled to make any decision here: I have no > knowledge of the stack involved here. I can only confirm than using > sphinx 0.5 (the dev version) does work for latex constructs; I could not > make the JS backend work, but again, I know nothing about web frameworks > and co, and I guess setting up a webserver for this to work to be > trivial for anyone who knows his stuff here. I believe the most forward-proof and downwards-compatible way for us is to switch to sphinx.ext.pngmath, and rely on Sphinx 0.5.dev. (0.5 also has the cool Intersphinx extension, using which we can easily link eg. to Python docs, or from scipy docs to numpy docs.) The current mathml code could perhaps also be polished a bit and contributed to sphinx.ext as an extra math back-end. The pngmath and jsmath backends share some code that do part of the work, so the current code would need to be tweaked a bit. -- Pauli Virtanen From LisaAnnDelgado at gmail.com Tue Sep 16 14:58:34 2008 From: LisaAnnDelgado at gmail.com (Lisa Delgado) Date: Tue, 16 Sep 2008 18:58:34 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?ValueError=3A_setting_an_array_eleme?= =?utf-8?q?nt_with_a=09sequence=2E?= References: Message-ID: Charles R Harris gmail.com> writes: > > > On Tue, Sep 16, 2008 at 9:05 AM, Lisa Delgado gmail.com> wrote: > Hi, > I had previously run a program successfully when using numpy 1.1.0. I just > downloaded the Enthought Python Distribution, which has numpy 1.0.4 and am now > having problems running my program. I get the following error. > Traceback (most recent call last): > ?File "/Users/Lisa/Documents/Dissertation work/PythonPrograms/NashEq.py", line > 57, in > ? ?EffortCosts[i,j] = buyeragents[i][j] > ValueError: setting an array element with a sequence. > > > What type is buyeragents? Numpy 1.1.0 fixes a number of bugs in 1.0.4 and you may have been bitten by that.Chuck > > > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > buyeragents is a list. I made no changes to the program from when it was working on numpy 1.1.0 to now with numpy 1.0.4. Thanks. Lisa PS - This is my own program for computing Nash equilibria. From millman at berkeley.edu Tue Sep 16 15:07:23 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 16 Sep 2008 12:07:23 -0700 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: <48CFEB9B.3010909@llnl.gov> References: <48CFDBF4.5070705@llnl.gov> <48CFEB9B.3010909@llnl.gov> Message-ID: Which version of gcc are you using? Do I remember correctly that you had the same failure with numpy 1.1.1 when using gcc 3.3 and that the problem went away when you used gcc 4.1. If you are using 3.3, could you try with 4.1 and let us know if you run into the same problem? Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From LisaAnnDelgado at gmail.com Tue Sep 16 15:08:55 2008 From: LisaAnnDelgado at gmail.com (Lisa Delgado) Date: Tue, 16 Sep 2008 19:08:55 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?ValueError=3A_setting_an_array_eleme?= =?utf-8?q?nt_with_a=09sequence=2E?= References: Message-ID: Charles R Harris gmail.com> writes: > > > On Tue, Sep 16, 2008 at 9:05 AM, Lisa Delgado gmail.com> wrote: > Hi, > I had previously run a program successfully when using numpy 1.1.0. I just > downloaded the Enthought Python Distribution, which has numpy 1.0.4 and am now > having problems running my program. I get the following error. > Traceback (most recent call last): > ?File "/Users/Lisa/Documents/Dissertation work/PythonPrograms/NashEq.py", line > 57, in > ? ?EffortCosts[i,j] = buyeragents[i][j] > ValueError: setting an array element with a sequence. > > > What type is buyeragents? Numpy 1.1.0 fixes a number of bugs in 1.0.4 and you may have been bitten by that.Chuck I found a way to fix my problem. I changed the following code: for line in binputfile: buyeragents.append(line.split()) to: for line in binputfile: buyerdata = map(float, line.split()) buyeragents.append(buyerdata) My program is running fine now. Thanks. Lisa From doutriaux1 at llnl.gov Tue Sep 16 15:18:20 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Tue, 16 Sep 2008 12:18:20 -0700 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: References: <48CFDBF4.5070705@llnl.gov> <48CFEB9B.3010909@llnl.gov> Message-ID: <48D0067C.7090404@llnl.gov> Thanks Jarrod it's coming back now. I thought they had updated the system... but no luck... Ok that's the issue i'm using: gcc (GCC) 3.4.6 20060404 (Red Hat 3.4.6-8) Thanks again, C. Jarrod Millman wrote: > Which version of gcc are you using? Do I remember correctly that you > had the same failure with numpy 1.1.1 when using gcc 3.3 and that the > problem went away when you used gcc 4.1. If you are using 3.3, could > you try with 4.1 and let us know if you run into the same problem? > > Thanks, > > From aisaac at american.edu Tue Sep 16 15:24:45 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 16 Sep 2008 15:24:45 -0400 Subject: [Numpy-discussion] ValueError: setting an array element with a sequence. In-Reply-To: References: Message-ID: <48D007FD.9040401@american.edu> On 9/16/2008 2:58 PM Lisa Delgado apparently wrote: > This is my own program for computing Nash equilibria. It sounds like you are working on stuff that might interest others. Please consider contributing to econpy, at least your Nash soln procedure. What exactly are you working on? Thanks, Alan From millman at berkeley.edu Tue Sep 16 15:39:20 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 16 Sep 2008 12:39:20 -0700 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: <48D0067C.7090404@llnl.gov> References: <48CFDBF4.5070705@llnl.gov> <48CFEB9B.3010909@llnl.gov> <48D0067C.7090404@llnl.gov> Message-ID: On Tue, Sep 16, 2008 at 12:18 PM, Charles Doutriaux wrote: > Thanks Jarrod it's coming back now. > > I thought they had updated the system... but no luck... > > Ok that's the issue i'm using: > gcc (GCC) 3.4.6 20060404 (Red Hat 3.4.6-8) Glad to hear it is a "known" issue. By the way, does anyone know what version of gcc we require? Where is that documented? The only occurrence of a recommended gcc version is here: http://scipy.org/Installing_SciPy/BuildingGeneral And it states "gcc 3.x compilers are recommended." -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From wnbell at gmail.com Tue Sep 16 15:49:32 2008 From: wnbell at gmail.com (Nathan Bell) Date: Tue, 16 Sep 2008 15:49:32 -0400 Subject: [Numpy-discussion] distutils chooses wrong arch flags Message-ID: Could someone with a better knowledge of distutils look over the following SciPy ticket: http://scipy.org/scipy/scipy/ticket/738 Short version: distutils compiles with -march=pentium-m on a machine that can't execute SS2 instructions. -- Nathan Bell wnbell at gmail.com http://graphics.cs.uiuc.edu/~wnbell/ From spam4bsimons at yahoo.ca Tue Sep 16 19:48:26 2008 From: spam4bsimons at yahoo.ca (Brendan Simons) Date: Tue, 16 Sep 2008 19:48:26 -0400 Subject: [Numpy-discussion] Help speeding up element-wise operations for video processing In-Reply-To: <91cf711d0809160704n48c0e3c0t389e2649317330eb@mail.gmail.com> References: <458186.45235.qm@web51601.mail.re2.yahoo.com> <9457e7c80809160150i5acb33d7t95193c58a27ab76d@mail.gmail.com> <91cf711d0809160704n48c0e3c0t389e2649317330eb@mail.gmail.com> Message-ID: Why would I need the GPU to do parallel operations? I thought most modern processors have vector units. I just don't know if there's a way to have my code use them. I tried a quick test with pyopengl (as quick as can be done in that crazy api), but I found adding two textures of the same size as ain took nearly as long as numpy (maybe twice as fast but I need it to be more like 10x faster). I can send you the code if you like, but it may be getting a bit out of scope for this forum. Thanks, Brendan On 16-Sep-08, at 10:04 AM, David Huard wrote: > Brendan, > > Not sure if I understand correctly what you want, but ... > > Numpy vector operations are performed in C, so there will be an > iteration over the array elements. > > For parallel operations over all pixels, you'd need a package that > talks to your GPU, such as pyGPU. > I've never tried it and if you do, please report your experience, > I'd be very interested to hear about it. > > HTH, > > David > > > > > On Tue, Sep 16, 2008 at 4:50 AM, St?fan van der Walt > wrote: > Hi Brendan > > 2008/9/16 brendan simons : > > #interpolate the green pixels from the bayer filter image ain > > g = greenMask * ain > > gi = g[:-2, 1:-1].astype('uint16') > > gi += g[2:, 1:-1] > > gi += g[1:-1, :-2] > > gi += g[1:-1, 2:] > > gi /= 4 > > gi += g[1:-1, 1:-1] > > return gi > > I may be completely off base here, but you should be able to do this > *very* quickly using your GPU, or even just using OpenGL. Otherwise, > coding it up in ctypes is easy as well (I can send you a code snippet, > if you need). > > > I do something similar for red and blue, then stack the > interpolated red, > > green and blue integers into an array of 24 bit integers and blit > to the > > screen. > > > > I was hoping that none of the lines would have to iterate over > pixels, and > > would instead do the adds and multiplies as single operations. > Perhaps numpy > > has to iterate when copying a subset of an array? Is there a > faster array > > "crop" ? Any hints as to how I might code this part up using > ctypes? > > Have you tried formulating this as a convolution, and using > scipy.signal's 2-d convolve or fftconvolve? > > Cheers > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From spam4bsimons at yahoo.ca Tue Sep 16 19:43:45 2008 From: spam4bsimons at yahoo.ca (Brendan Simons) Date: Tue, 16 Sep 2008 19:43:45 -0400 Subject: [Numpy-discussion] Help speeding up element-wise operations for video processing In-Reply-To: <9457e7c80809160150i5acb33d7t95193c58a27ab76d@mail.gmail.com> References: <458186.45235.qm@web51601.mail.re2.yahoo.com> <9457e7c80809160150i5acb33d7t95193c58a27ab76d@mail.gmail.com> Message-ID: <61ADA597-D8B2-48DC-A48E-EA8F2031540B@yahoo.ca> On 16-Sep-08, at 4:50 AM, St?fan van der Walt wrote: > Hi Brendan > > 2008/9/16 brendan simons : >> #interpolate the green pixels from the bayer filter image ain >> g = greenMask * ain >> gi = g[:-2, 1:-1].astype('uint16') >> gi += g[2:, 1:-1] >> gi += g[1:-1, :-2] >> gi += g[1:-1, 2:] >> gi /= 4 >> gi += g[1:-1, 1:-1] >> return gi > > I may be completely off base here, but you should be able to do this > *very* quickly using your GPU, or even just using OpenGL. Otherwise, > coding it up in ctypes is easy as well (I can send you a code snippet, > if you need). I thought so too. I briefly researched using the GPU, but I found that, surprisingly, neither multitexturing in pyopengl, nor surface blitting in pygame was any faster than the numpy code I posted above. That could be because I'm working on a machine with an integrated graphics chip. I would love a c-types code snippet. I'm not very handy in c. Since I gather numpy is row-major, I thought I up and down crops very quickly by moving the start and end pointers of the array. For cropping left and right, is there a fast c command for "copy while skipping every nth hundred bytes"? > > >> I do something similar for red and blue, then stack the >> interpolated red, >> green and blue integers into an array of 24 bit integers and blit >> to the >> screen. >> >> I was hoping that none of the lines would have to iterate over >> pixels, and >> would instead do the adds and multiplies as single operations. >> Perhaps numpy >> has to iterate when copying a subset of an array? Is there a >> faster array >> "crop" ? Any hints as to how I might code this part up using ctypes? > > Have you tried formulating this as a convolution, and using > scipy.signal's 2-d convolve or fftconvolve? I just read through the scipy tutorial on signal.convolve, and I'm a bit lost. It's been years since I've last taken a numerical methods class. Maybe there's something here - but I'm going to have to learn a bit first. Thanks for your help, Brendan From david at ar.media.kyoto-u.ac.jp Tue Sep 16 21:37:15 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 17 Sep 2008 10:37:15 +0900 Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <48CF56F9.9070206@ar.media.kyoto-u.ac.jp> <2a66de50-78a4-47ce-adac-611d808cfe80@a1g2000hsb.googlegroups.com> <48CF79CC.9090609@ar.media.kyoto-u.ac.jp> <07bbbcff-365f-4051-924b-dae8d353b0ed@8g2000hse.googlegroups.com> <48CF96FD.9090805@ar.media.kyoto-u.ac.jp> <70cd1d75-3a32-4140-8a72-d2cdd650ef62@m3g2000hsc.googlegroups.com> <48CFCD08.10109@ar.media.kyoto-u.ac.jp> Message-ID: <48D05F4B.30508@ar.media.kyoto-u.ac.jp> Francis wrote: > > To be frank, I am bit of noob, but not completely. I use the gfortran > make.inc to build lapack. The make.inc file to build and install atlas > is using gfortran. I used the --fcompiler=gfortran flag. I don't > think there is much else I can do. Yes there is :) You should use --fcompiler=gnu95, not gfortran. That would explain the problem you had, actually. There is no problem being a noob: compiling numpy itself is easy, but unfortunately, its dependencies are not, and are almost systematically badly packaged by distributions. To make the situation worse, there are still two widely used gnu fortran compilers, which are ABI incompatible... > > Unfortunately there is no EPD installer for Centos 5 version. There > was a blog how to install it, but that look a lot more complicated > than what I am trying now. > > I will try to build sage and look if I can build numpy against the > atlas libraries in sage. Otherwise I use sage and remove all the > packages that are not relevant for me, e.g. maxima, gap, singular, > etc. I am really sorry for all this trouble. The last solution you could try, with a bit old packages is my own repository, with Centos packages: http://download.opensuse.org/repositories/home:/ashigabou/CentOS_5/ Install the numpy and scipy packages from there. There is also a src.rpm for atlas, which enables you to build atlas rpm automatically optimized for your CPU using one command (look at build atlas from ashigabou repository on www.scipy.org/Installing_Scipy/Linux). I have been trying to update those packages yesterday, but some network-related problems prevented me from. cheers, David From robert.kern at gmail.com Tue Sep 16 21:56:24 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 16 Sep 2008 20:56:24 -0500 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> Message-ID: <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg wrote: > That would make me an extremely happy user, I've been looking for this for > years! > I can't imagine I'm the only one who profiles some hundred lines of code and > ends up with 90% of total time in the dot-function For the time being, you can grab it here: http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/ It requires Cython and a C compiler to build. I'm still debating myself about the desired workflow for using it, but for now, it only profiles functions which you have registered with it. I have made the profiler work as a decorator to make this easy. E.g., from line_profiler import LineProfiler profile = LineProfiler() @profile def my_slow_func(): ... profile.dump_stats('my_slow_func.lprof') This is kind of inconvenient, so I have a little script that handles everything except for the @profile itself. It started as a script to run cProfile nicely, so it actually does that by default. I took pystone.py from the Python source and added a couple of @profile decorators to demonstrate: [line_profiler]$ ./kernprof.py --help Usage: ./kernprof.py [-s setupfile] [-o output_file_path] scriptfile [arg] ... Options: -h, --help show this help message and exit -l, --line-by-line Use the line-by-line profiler from the line_profiler module instead of cProfile. Implies --builtin. -b, --builtin Put 'profile' in the builtins. Use 'profile.enable()' and 'profile.disable()' in your code to turn it on and off, or '@profile' to decorate a single function, or 'with profile:' to profile a single section of code. -o OUTFILE, --outfile=OUTFILE Save stats to -s SETUP, --setup=SETUP Code to execute before the code to profile [line_profiler]$ ./kernprof.py -l pystone.py Pystone(1.1) time for 50000 passes = 11.34 This machine benchmarks at 4409.17 pystones/second Wrote profile results to pystone.py.lprof [line_profiler]$ ./view_line_prof.py pystone.py.lprof Timer unit: 1e-06 s File: pystone.py Function: Proc0 at line 79 Total time: 8.46516 s Line # Hits Time % Time Line Contents ===================================================== 79 @profile 80 def Proc0(loops=LOOPS): 81 global IntGlob 82 global BoolGlob 83 global Char1Glob 84 global Char2Glob 85 global Array1Glob 86 global Array2Glob 87 global PtrGlb 88 global PtrGlbNext 89 90 1 11 0.0 starttime = clock() 91 50001 128471 1.5 for i in range(loops): 92 50000 125580 1.5 pass 93 1 28 0.0 nulltime = clock() - starttime 94 95 1 21 0.0 PtrGlbNext = Record() 96 1 8 0.0 PtrGlb = Record() 97 1 2 0.0 PtrGlb.PtrComp = PtrGlbNext 98 1 2 0.0 PtrGlb.Discr = Ident1 99 1 2 0.0 PtrGlb.EnumComp = Ident3 100 1 3 0.0 PtrGlb.IntComp = 40 101 1 3 0.0 PtrGlb.StringComp = "DHRYSTONE PROGRAM, SOME STRING" 102 1 2 0.0 String1Loc = "DHRYSTONE PROGRAM, 1'ST STRING" 103 1 5 0.0 Array2Glob[8][7] = 10 104 105 1 6 0.0 starttime = clock() 106 107 50001 123728 1.5 for i in range(loops): 108 50000 220345 2.6 Proc5() 109 50000 238459 2.8 Proc4() 110 50000 119789 1.4 IntLoc1 = 2 111 50000 123639 1.5 IntLoc2 = 3 112 50000 116081 1.4 String2Loc = "DHRYSTONE PROGRAM, 2'ND STRING" 113 50000 119487 1.4 EnumLoc = Ident2 114 50000 653711 7.7 BoolGlob = not Func2(String1Loc, String2Loc) 115 50000 130518 1.5 while IntLoc1 < IntLoc2: 116 50000 132916 1.6 IntLoc3 = 5 * IntLoc1 - IntLoc2 117 50000 235608 2.8 IntLoc3 = Proc7(IntLoc1, IntLoc2) 118 50000 147952 1.7 IntLoc1 = IntLoc1 + 1 119 50000 664949 7.9 Proc8(Array1Glob, Array2Glob, IntLoc1, IntLoc3) 120 50000 1710897 20.2 PtrGlb = Proc1(PtrGlb) 121 50000 130536 1.5 CharIndex = 'A' 122 50000 130006 1.5 while CharIndex <= Char2Glob: 123 100000 546206 6.5 if EnumLoc == Func1(CharIndex, 'C'): 124 EnumLoc = Proc6(Ident1) 125 100000 353517 4.2 CharIndex = chr(ord(CharIndex)+1) 126 50000 133892 1.6 IntLoc3 = IntLoc2 * IntLoc1 127 50000 126802 1.5 IntLoc2 = IntLoc3 / IntLoc1 128 50000 139932 1.7 IntLoc2 = 7 * (IntLoc3 - IntLoc2) - IntLoc1 129 50000 1912016 22.6 IntLoc1 = Proc2(IntLoc1) 130 131 1 12 0.0 benchtime = clock() - starttime - nulltime 132 1 18 0.0 return benchtime, (loops / benchtime) File: pystone.py Function: Proc2 at line 150 Total time: 0.612312 s Line # Hits Time % Time Line Contents ===================================================== 150 @profile 151 def Proc2(IntParIO): 152 50000 75147 12.3 IntLoc = IntParIO + 10 153 50000 64428 10.5 while 1: 154 50000 70023 11.4 if Char1Glob == 'A': 155 50000 69017 11.3 IntLoc = IntLoc - 1 156 50000 70189 11.5 IntParIO = IntLoc - IntGlob 157 50000 65503 10.7 EnumLoc = Ident1 158 50000 68191 11.1 if EnumLoc == Ident1: 159 50000 66168 10.8 break 160 50000 63646 10.4 return IntParIO I used to call kernprof.py lsprof.py, but then I realized that's what cProfile used to be named when it was outside of the stdlib. So I'm just going to call it kernprof until a better name comes along. So, I could use some comments on the workflow. Does this look sensible to everyone? How else would you like to use it? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Tue Sep 16 21:39:54 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 17 Sep 2008 10:39:54 +0900 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: References: <48CFDBF4.5070705@llnl.gov> <48CFEB9B.3010909@llnl.gov> <48D0067C.7090404@llnl.gov> Message-ID: <48D05FEA.9090508@ar.media.kyoto-u.ac.jp> Jarrod Millman wrote: > By the way, does anyone know what version of gcc we require? Where is > that documented? The only occurrence of a recommended gcc version is > here: http://scipy.org/Installing_SciPy/BuildingGeneral > > And it states "gcc 3.x compilers are recommended." > Those instructions are terribly outdated, but I don't know how to put a better one. The Gcc is IA-64 specific. Honestly, since this arch is hard to find and not available at many places, and that Gnu compilers are known to be bad and not really efficient on this arch, I don't feel like spending a lot of time on debugging remotely compilers bugs. cheers, David From david at ar.media.kyoto-u.ac.jp Tue Sep 16 21:47:28 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 17 Sep 2008 10:47:28 +0900 Subject: [Numpy-discussion] Help speeding up element-wise operations for video processing In-Reply-To: References: <458186.45235.qm@web51601.mail.re2.yahoo.com> <9457e7c80809160150i5acb33d7t95193c58a27ab76d@mail.gmail.com> <91cf711d0809160704n48c0e3c0t389e2649317330eb@mail.gmail.com> Message-ID: <48D061B0.4030804@ar.media.kyoto-u.ac.jp> Brendan Simons wrote: > Why would I need the GPU to do parallel operations? I thought most > modern processors have vector units. I just don't know if there's a > way to have my code use them. Yes, modern CPU have vector units, but to use them efficiently, you have to use assembly or specially written routines, which are arch dependent (compilers still do not 'auto-vectorize' automatically the code which could be autovectorized in most cases). Numpy itself does not use specially optimized routines using vector units, except for blas/lapack. You certainly can't use the vector units efficiently from simple python code. cheers, David From david at ar.media.kyoto-u.ac.jp Tue Sep 16 22:38:52 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 17 Sep 2008 11:38:52 +0900 Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: <48D05F4B.30508@ar.media.kyoto-u.ac.jp> References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <48CF56F9.9070206@ar.media.kyoto-u.ac.jp> <2a66de50-78a4-47ce-adac-611d808cfe80@a1g2000hsb.googlegroups.com> <48CF79CC.9090609@ar.media.kyoto-u.ac.jp> <07bbbcff-365f-4051-924b-dae8d353b0ed@8g2000hse.googlegroups.com> <48CF96FD.9090805@ar.media.kyoto-u.ac.jp> <70cd1d75-3a32-4140-8a72-d2cdd650ef62@m3g2000hsc.googlegroups.com> <48CFCD08.10109@ar.media.kyoto-u.ac.jp> <48D05F4B.30508@ar.media.kyoto-u.ac.jp> Message-ID: <48D06DBC.3000007@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > > I am really sorry for all this trouble. The last solution you could try, > with a bit old packages is my own repository, with Centos packages: > > http://download.opensuse.org/repositories/home:/ashigabou/CentOS_5/ I could update numpy to a recent one (1.1.1), but unfortunately, I still cannot test the built packages because of the network problems. Blas and Lapack should work correctly if you build numpy with --fcompiler=gnu95. cheers, David From simon.palmer at gmail.com Wed Sep 17 03:32:20 2008 From: simon.palmer at gmail.com (SimonPalmer) Date: Wed, 17 Sep 2008 00:32:20 -0700 (PDT) Subject: [Numpy-discussion] numpy and stackless Message-ID: <18fd4580-3e14-41bd-ba2f-af0139c6c71d@l64g2000hse.googlegroups.com> Has anyone tried mixing numpy with stackless? I am wondering if there are any obvious reasons why it would not work. I am thinking specifically of issues that may arise using the stackless microthreads. From cournape at gmail.com Wed Sep 17 05:06:30 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 17 Sep 2008 18:06:30 +0900 Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: <48D06DBC.3000007@ar.media.kyoto-u.ac.jp> References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <2a66de50-78a4-47ce-adac-611d808cfe80@a1g2000hsb.googlegroups.com> <48CF79CC.9090609@ar.media.kyoto-u.ac.jp> <07bbbcff-365f-4051-924b-dae8d353b0ed@8g2000hse.googlegroups.com> <48CF96FD.9090805@ar.media.kyoto-u.ac.jp> <70cd1d75-3a32-4140-8a72-d2cdd650ef62@m3g2000hsc.googlegroups.com> <48CFCD08.10109@ar.media.kyoto-u.ac.jp> <48D05F4B.30508@ar.media.kyoto-u.ac.jp> <48D06DBC.3000007@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220809170206p39966c1bg25dbcf207d5717b9@mail.gmail.com> On Wed, Sep 17, 2008 at 11:38 AM, David Cournapeau wrote: > David Cournapeau wrote: >> >> I am really sorry for all this trouble. The last solution you could try, >> with a bit old packages is my own repository, with Centos packages: >> >> http://download.opensuse.org/repositories/home:/ashigabou/CentOS_5/ I could test the above on a real Centos machine: it does work flawlessly. You just have to put the .repo file into /etc/yum.repo.d, and then: sudo yum install python-numpy python-scipy I was not sure it was working with centos, but now that I do, that's certainly the path to the least pain in your case, cheers, David From F.Drossaert at googlemail.com Wed Sep 17 05:27:29 2008 From: F.Drossaert at googlemail.com (Francis) Date: Wed, 17 Sep 2008 02:27:29 -0700 (PDT) Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: <48D05F4B.30508@ar.media.kyoto-u.ac.jp> References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <48CF56F9.9070206@ar.media.kyoto-u.ac.jp> <2a66de50-78a4-47ce-adac-611d808cfe80@a1g2000hsb.googlegroups.com> <48CF79CC.9090609@ar.media.kyoto-u.ac.jp> <07bbbcff-365f-4051-924b-dae8d353b0ed@8g2000hse.googlegroups.com> <48CF96FD.9090805@ar.media.kyoto-u.ac.jp> <70cd1d75-3a32-4140-8a72-d2cdd650ef62@m3g2000hsc.googlegroups.com> <48CFCD08.10109@ar.media.kyoto-u.ac.jp> <48D05F4B.30508@ar.media.kyoto-u.ac.jp> Message-ID: On Sep 17, 2:37?am, David Cournapeau wrote: > Francis wrote: > > > To be frank, I am bit of noob, but not completely. I use the gfortran > > make.inc to build lapack. The make.inc file to build and install atlas > > is using gfortran. ?I used the --fcompiler=gfortran flag. I don't > > think there is much else I can do. > > Yes there is :) You should use --fcompiler=gnu95, not gfortran. That > would explain the problem you had, actually. There is no problem being a > noob: compiling numpy itself is easy, but unfortunately, its > dependencies are not, and are almost systematically badly packaged by > distributions. To make the situation worse, there are still two widely > used gnu fortran compilers, which are ABI incompatible... Even a noob can follow simple instructions. :) Tried gnu95 first and then gfortran but both result in the same error. > > Unfortunately there is no EPD installer for Centos 5 version. There > > was a blog how to install it, but that look a lot more complicated > > than what I am trying now. > > > I will try to build sage and look if I can build numpy against the > > atlas libraries in sage. Otherwise I use sage and remove all the > > packages that are not relevant for me, e.g. maxima, gap, singular, > > etc. > > I am really sorry for all this trouble. The last solution you could try, > with a bit old packages is my own repository, with Centos packages: > > http://download.opensuse.org/repositories/home:/ashigabou/CentOS_5/ > > Install the numpy and scipy packages from there. There is also a src.rpm > for atlas, which enables you to build atlas rpm automatically optimized > for your CPU using one command (look at build atlas from ashigabou > repository onwww.scipy.org/Installing_Scipy/Linux). > > I have been trying to update those packages yesterday, but some > network-related problems prevented me from. I can't open rpm files so I opened numpy with archive manager and extracted Numpy to site-packages. Unfortunately importing numpy results in ImportError: .... undefined symbol: _PyUnicodeUCS4_IsWhitespace. Suggested solution found in wiki is to hard-install numpy. Aaaaah ... it is like running in circles. Is it possible to use refblas and lapack libraries in the ashigabou rep. to hard-install numpy. Not sure what the site.cfg should look like than. From david at ar.media.kyoto-u.ac.jp Wed Sep 17 05:16:37 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 17 Sep 2008 18:16:37 +0900 Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <48CF56F9.9070206@ar.media.kyoto-u.ac.jp> <2a66de50-78a4-47ce-adac-611d808cfe80@a1g2000hsb.googlegroups.com> <48CF79CC.9090609@ar.media.kyoto-u.ac.jp> <07bbbcff-365f-4051-924b-dae8d353b0ed@8g2000hse.googlegroups.com> <48CF96FD.9090805@ar.media.kyoto-u.ac.jp> <70cd1d75-3a32-4140-8a72-d2cdd650ef62@m3g2000hsc.googlegroups.com> <48CFCD08.10109@ar.media.kyoto-u.ac.jp> <48D05F4B.30508@ar.media.kyoto-u.ac.jp> Message-ID: <48D0CAF5.4070109@ar.media.kyoto-u.ac.jp> Francis wrote: > > I can't open rpm files so I opened numpy with archive manager and > extracted Numpy to site-packages. Can't you ask your admin to install rpm packages ? Uncompressing the rpm and putting them into random places is unlikely to work. > Unfortunately importing numpy > results in ImportError: .... undefined symbol: > _PyUnicodeUCS4_IsWhitespace. Yes, because you built your python yourself. That's why you should not do that: once you built your own python, you are effectively using a different ABI for every python package packaged for your distribution unless you take care of using the exact same options as RH did. You should not do that unless you really know what you're doing. > Suggested solution found in wiki is to > hard-install numpy. Aaaaah ... it is like running in circles. Is it > possible to use refblas and lapack libraries in the ashigabou rep. Yes, as long as you use gfortran to build numpy/scipy. But I would first ask your admin whether it is ok to install the rpms. That's really how packages should be installed, not using the sources directly. cheers, David From faltet at pytables.org Wed Sep 17 05:34:02 2008 From: faltet at pytables.org (Francesc Alted) Date: Wed, 17 Sep 2008 11:34:02 +0200 Subject: [Numpy-discussion] Help speeding up element-wise operations for video processing In-Reply-To: <61ADA597-D8B2-48DC-A48E-EA8F2031540B@yahoo.ca> References: <458186.45235.qm@web51601.mail.re2.yahoo.com> <9457e7c80809160150i5acb33d7t95193c58a27ab76d@mail.gmail.com> <61ADA597-D8B2-48DC-A48E-EA8F2031540B@yahoo.ca> Message-ID: <200809171134.03021.faltet@pytables.org> A Wednesday 17 September 2008, Brendan Simons escrigu?: [clip] > I would love a c-types code snippet. I'm not very handy in c. Since > I gather numpy is row-major, I thought I up and down crops very > quickly by moving the start and end pointers of the array. For > cropping left and right, is there a fast c command for "copy while > skipping every nth hundred bytes"? There is no such efficient "copy while skipping every nth hundred bytes" thing for C or for any other language. You are facing here a fundamental problem in the design of modern processor architectures, namely, the (large) memory latency. That means that when accessing memory in discontiguous patterns like the one you indicated, that will keep the processor waiting for the data most of the time. There are ways to give hints to compilers in order to perform a better pre-fetching of interesting data, but this is a rather complex process, and the improvements can be meager in most of cases. In brief, if you don't have much time to spend of this, my advice is to use just regular assignment or memcpy (whatever is more comfortable for your situation), because you won't be able to get more performance than what these will offer. However, if you have more time and want to look for ways on how to scratch more performance on different memory access patterns, it is always a wise thing to have a look at the excellent "What Every Programmer Should Know About Memory" report: htt://people.redhat.com/drepper/cpumemory.pdf [Incidentally, this is possible one of the best reports available on the subject of memory access on nowadays architectures (a critical thing for achieving maximum performance), and besides it is available for free, so there is no excuse to not have look at it ;-)] Cheers, -- Francesc Alted From matthieu.brucher at gmail.com Wed Sep 17 05:47:59 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 17 Sep 2008 11:47:59 +0200 Subject: [Numpy-discussion] [numpy 1.1.1] Opteron 254 has SSE 3 ??? Message-ID: Hi, I'd like to use numpy.distutils.cpuinfo to check if I can enable some flags during compilation of Fortran code. The issue I face is that an Opteron 254 does not have SSE3. So says /proc/cpuinfo. But in fact _has_sse3() returns True on this paltform. But not _has_ssse3(). What are the differences between both values ? On a side note, is testing for sse4 in the pipeline ? Thanks, Matthieu -- French PhD student Information System Engineer Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From F.Drossaert at googlemail.com Wed Sep 17 06:13:36 2008 From: F.Drossaert at googlemail.com (Francis) Date: Wed, 17 Sep 2008 03:13:36 -0700 (PDT) Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: <48D0CAF5.4070109@ar.media.kyoto-u.ac.jp> References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <48CF56F9.9070206@ar.media.kyoto-u.ac.jp> <2a66de50-78a4-47ce-adac-611d808cfe80@a1g2000hsb.googlegroups.com> <48CF79CC.9090609@ar.media.kyoto-u.ac.jp> <07bbbcff-365f-4051-924b-dae8d353b0ed@8g2000hse.googlegroups.com> <48CF96FD.9090805@ar.media.kyoto-u.ac.jp> <70cd1d75-3a32-4140-8a72-d2cdd650ef62@m3g2000hsc.googlegroups.com> <48CFCD08.10109@ar.media.kyoto-u.ac.jp> <48D05F4B.30508@ar.media.kyoto-u.ac.jp> <48D0CAF5.4070109@ar.media.kyoto-u.ac.jp> Message-ID: In general the IT dep. don't want changes since changes require weeks of testing to make sure that existing applications are not conflicting, etc. I am not sure they would fulfill my request. Especially since it has no priority at all. I am installing Sage math now. Although I would like to avoid using it, it is at this moment the simplest solution and it will do what I want it to do. Thank you for trying to help out. On 17 Sep, 10:16, David Cournapeau wrote: > Francis wrote: > > > I can't open rpm files so I opened numpy with archive manager and > > extracted Numpy to site-packages. > > Can't you ask your admin to install rpm packages ? Uncompressing the rpm > and putting them into random places is unlikely to work. > > > ?Unfortunately importing numpy > > results in ImportError: .... undefined symbol: > > _PyUnicodeUCS4_IsWhitespace. > > Yes, because you built your python yourself. That's why you should not > do that: once you built your own python, you are effectively using a > different ABI for every python package packaged for your distribution > unless you take care of using the exact same options as RH did. You > should not do that unless you really know what you're doing. > > > ?Suggested solution found in wiki is to > > hard-install numpy. Aaaaah ... it is like running in circles. Is it > > possible to use refblas and lapack libraries in the ashigabou rep. > > Yes, as long as you use gfortran to build numpy/scipy. But I would first > ask your admin whether it is ok to install the rpms. That's really how > packages should be installed, not using the sources directly. > > cheers, > > David > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion From david at ar.media.kyoto-u.ac.jp Wed Sep 17 06:08:47 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 17 Sep 2008 19:08:47 +0900 Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <48CF56F9.9070206@ar.media.kyoto-u.ac.jp> <2a66de50-78a4-47ce-adac-611d808cfe80@a1g2000hsb.googlegroups.com> <48CF79CC.9090609@ar.media.kyoto-u.ac.jp> <07bbbcff-365f-4051-924b-dae8d353b0ed@8g2000hse.googlegroups.com> <48CF96FD.9090805@ar.media.kyoto-u.ac.jp> <70cd1d75-3a32-4140-8a72-d2cdd650ef62@m3g2000hsc.googlegroups.com> <48CFCD08.10109@ar.media.kyoto-u.ac.jp> <48D05F4B.30508@ar.media.kyoto-u.ac.jp> <48D0CAF5.4070109@ar.media.kyoto-u.ac.jp> Message-ID: <48D0D72F.1080009@ar.media.kyoto-u.ac.jp> Francis wrote: > In general the IT dep. don't want changes since changes require weeks > of testing to make sure that existing applications are not > conflicting, etc. I feel your pain. In such case, I ask if it is possible to ask for vmware installation: nowadays, cpu are fast enough such as running linux inside linux is fast, and inside your vmware installation, you install your own things. Still, as a numpy/scipy developer, it is quite frustrating that a user have to install sage, a third-party software, to make numpy/scipy work... cheers, David From Richard_Martineau at xyratex.com Wed Sep 17 06:03:53 2008 From: Richard_Martineau at xyratex.com (Richie) Date: Wed, 17 Sep 2008 03:03:53 -0700 (PDT) Subject: [Numpy-discussion] Build Numpy Source using Visual Studio 6.0 Message-ID: <01ce652c-e3fe-4ffd-9c72-45629111f1d3@e53g2000hsa.googlegroups.com> Hello I wonder if anyone can advise me or has done similar to the following? Basically I've downloaded the Python 2.5.2 source code that builds with Visual Studio 6.0. I've built Python for windows. This was easy (it even came with the pcbuild.dsw workspace file). Great! Now comes the troubled bit...I now look for similar source code for Python extensions Numpy and Scipy but there appears to be no source code for vs6.0 and the source that I can find is not all obvious. Looks like these are normally built using gcc compilers and some of the code uses variable length arrays that looks gcc specific syntax? Not sure. However I need to do all my builds in VS 6.0 as I eventually want to port across from windows os to PharLap rtos. So far I've tried building Numpy (I haven't got as far as Scipy yet) and I've organised the source code into the following projects and lumped them all into my new workspace numpy.dsw: dotblas fft lib linalg multiarray numarray random scalarmath sort umath I'm obviously not sure if this is correct. As you can see from above, I've re-arranged the source code directory named core into the following projects: dotblas sort multiarray scalarmath umath Unbelievably I am having some luck and nearly all the projects compile and link (untested though). In addition I created the auto-generated files using the python scripts. Then I included the headers where needed. However the remaining problems are: umath (wont compile specific files are __ufunc_api.c and __umath_generated.c ) multiarray (wont compile) dotblas (wont link) dotblas seems to have missing dependencies on these: _cblas_sdot _cblas_ddot _cblas_cdotu_sub _cblas_zdotu_sub _cblas_cgemm _cblas_zgemm _cblas_sgemm _cblas_dgemm _cblas_cgemv _cblas_zgemv _cblas_sgemv _cblas_dgemv _cblas_caxpy _cblas_saxpy _cblas_zaxpy _cblas_daxpy _cblas_cdotc_sub _cblas_zdotc_sub Well that's snapshot of where I am today. If anyone can offer any advice (negative or positive, I don't mind) I'd really appreciate hearing from you as I'm now at a loss as how to best proceed. Thanks for your help and very happy to have joined this group Richie From felix at physik3.uni-rostock.de Wed Sep 17 06:44:51 2008 From: felix at physik3.uni-rostock.de (Felix Richter) Date: Wed, 17 Sep 2008 12:44:51 +0200 Subject: [Numpy-discussion] Python objects in Numpy: compatibility issues with methods and functions Message-ID: <200809171244.51540.felix@physik3.uni-rostock.de> Hi all, for my calculations, I need multi-precision arithmetics. For this, I use the wonderful Python-only mpmath (http://code.google.com/p/mpmath/) and store and handle my data in Numpy arrays. I thus have Numpy arrays with dtype=object holding mpmath objects. Now, some of the array operations that make Numpy such an invaluable tool work (like adding to arrays or even the dot product!), other silently fail (like .imag or .conj()). I added an example session below. What happens if .imag is evaluated on a Numpy array? How does Numpy determine the imaginary part of an object dtype? Is this propagated to the elements and are there some hooks? (The case of conj is quite confusing, as m2.conj() silently fails but np.conj(m2) works perfectly.) In other words: What interface must a Python class provide to ensure that Numpy array operations work? I guess this is somewhat close to what T.J. Alumbaugh asked some days ago. Thanks for any hints, Felix In [1]:import numpy as np In [2]:import mpmath In [3]:m2 = np.array([mpmath.mpc(1+2j),mpmath.mpc(2+3j)]) In [4]:m2 Out[4]:array([(1.0 + 2.0j), (2.0 + 3.0j)], dtype=object) In [5]:m2.imag Out[5]:array([0, 0], dtype=object) In [6]:m2[0].imag Out[6]:mpf('2.0') In [7]:m2.conj() Out[7]:array([(1.0 + 2.0j), (2.0 + 3.0j)], dtype=object) In [8]:m2[0].conj() --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) AttributeError: 'mpc' object has no attribute 'conj' In [10]:np.conj(m2[0]) Out[10]:mpc(real='1.0', imag='-2.0') In [11]:np.conj(m2) Out[11]:array([(1.0 - 2.0j), (2.0 - 3.0j)], dtype=object) From nadavh at visionsense.com Wed Sep 17 09:55:54 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed, 17 Sep 2008 16:55:54 +0300 Subject: [Numpy-discussion] numpy.linalg.eig In-Reply-To: <00a001c91819$e6762040$b36260c0$@dk> References: <003f01c91742$54b5c9d0$fe215d70$@dk> <3d375d730809150938x6bc994aai197dfa28c5e50092@mail.gmail.com> <00a001c91819$e6762040$b36260c0$@dk> Message-ID: <1221659754.7519.4.camel@nadav.envision.co.il> Did you edit site.cfg to use optimised libraries (ATLAS, MKL, whatever)? Nadav On Tue, 2008-09-16 at 17:33 +0100, Tommy Carstensen wrote: > Dear Robert Kern, > > Thanks for your interest in helping me. > > Previously I used LinearAlgebra.Heigenvectors. I should have switched to > numpy.linalg.eigh instead of numpy.linalg.eig, since I work with Hermitian > (non-complex and symmetric) matrices. > > Kind regards, > Tommy > > -----Original Message----- > From: numpy-discussion-bounces at scipy.org > [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Robert Kern > Sent: 15 September 2008 17:39 > To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion] numpy.linalg.eig > > On Mon, Sep 15, 2008 at 09:50, Tommy Carstensen > wrote: > > To the numpy mailing list, > > > > Previously I used python2.3 and the module LinearAlgebra for matrix > > diagonalization. I recently upgraded to python2.5 and I am now using > > numpy.linalg.eig for matrix diagonalization. This is approximately 3 times > > slower than previously. I am using the same work station as previously and > > it has all lapack libraries installed to my knowledge. Does anyone have an > > explanation as to why I am experiencing the reduced speed of calculation? > > Although you have the same LAPACK libraries installed, Numeric and > numpy may have been built differently, and numpy may not be using the > same libraries. Can you check? > > > This is my first post to this mailing list, so I hope this mail complies > > with the mailing list standards. > > Yes, certainly. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Wed Sep 17 07:34:00 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 17 Sep 2008 13:34:00 +0200 Subject: [Numpy-discussion] Build Numpy Source using Visual Studio 6.0 In-Reply-To: <01ce652c-e3fe-4ffd-9c72-45629111f1d3@e53g2000hsa.googlegroups.com> References: <01ce652c-e3fe-4ffd-9c72-45629111f1d3@e53g2000hsa.googlegroups.com> Message-ID: 2008/9/17 Richie : > Hello > > I wonder if anyone can advise me or has done similar to the > following? > > Basically I've downloaded the Python 2.5.2 source code that builds > with Visual Studio 6.0. I've built Python for windows. This was easy > (it even came with the pcbuild.dsw workspace file). Great! > > Now comes the troubled bit...I now look for similar source code for > Python extensions Numpy and Scipy but there appears to be no source > code for vs6.0 and the source that I can find is not all obvious. > Looks like these are normally built using gcc I'd rather say that the files should be compiled by distutils, which will call the appropriate compiler. So if you compiled the VS6.0 version of Python with VS 6.0, you should be able to just call python setup.py install. Matthieu -- French PhD student Information System Engineer Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From arnar.flatberg at gmail.com Wed Sep 17 08:12:01 2008 From: arnar.flatberg at gmail.com (Arnar Flatberg) Date: Wed, 17 Sep 2008 14:12:01 +0200 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> Message-ID: <5d3194020809170512idfa560s6548ffcca406c325@mail.gmail.com> On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern wrote: > > So, I could use some comments on the workflow. Does this look sensible > to everyone? How else would you like to use it? > Works for me. I would love to use it as a part of an ipython session. Initially, I sprinkled som $profile in a library of my own, then I thought I would use the -s (SETUP) option to initialize some data to pass into the $profiled functions. However, I have probably misunderstood the meaning of the -s option. Could you give an example of its use? I ended in doing a %edit in ipython to write a small test script with testdata and library imports, used kernprof.py and view_line_prof.py from ipython and captured the output from view_line_prof into ipython. Pheew .... now, there's perhaps an easier way? BTW, in the requirements you may want to mention argparse (view_line_prof.py) Thanks for sharing ! Arnar -------------- next part -------------- An HTML attachment was scrubbed... URL: From paultaney at yahoo.com Wed Sep 17 08:29:18 2008 From: paultaney at yahoo.com (paul taney) Date: Wed, 17 Sep 2008 05:29:18 -0700 (PDT) Subject: [Numpy-discussion] color filtering Message-ID: <664321.69222.qm@web53906.mail.re2.yahoo.com> Hi, I"m just learning numpy and I am not a math. I just need it. I need to do a color filter; I have it working in python but it runs forever on some of my images. And I dont know enuf c or c++ to use weave (or Boost or Pyrex or...) This looks like the best way, and I have somebody who is advising me, but he said to put the question to you guys, so I have an array from a wx.Image, but for testing we can use something just 5 * 5 * 3. I want the x,y coords of the blues pixel(s) per x returned as a line of this sort: [(x1, y1), (x2, y2)...(xn, yn)] # In real life we have an image: import wx import numpy as np image = wx.Image(path, wx.BITMAP_TYPE_ANY) w, h = image.GetWidth(), image.GetHeight() # Rappin page 362, Smart page 285 a = np.fromstring(image.GetData(), np.uint8) a.reshape(w, h, 3) # But while testing we can use: import numpy as np a = np.array( [[[ 64, 237, 108], # x y [185, 240, 68], # [0, 1] [ 17, 116, 55], # [0, 2] [107, 11, 185], # [0, 3] is blue [154, 30, 52]], # [0, 4] [[251, 73, 150], # x y [ 47, 225, 97], # [1, 1] [101, 91, 251], # [1, 2] is blue [ 61, 23, 79], # [1, 3] [161, 22, 165]], # [1, 4] [[ 4, 162, 188], # [2, 0] [ 25, 131, 211], # [2, 1] [162, 112, 177], # [2, 2] [169, 83, 214], # [2, 3] is blue [254, 139, 95]], [[ 2, 88, 216], # [3, 0] is blue [230, 63, 192], [ 71, 78, 250], # [3, 2] is blue [113, 151, 142], [104, 65, 127]], [[ 6, 191, 220], # [4, 0] is blue [ 38, 141, 4], [245, 41, 23], [ 23, 127, 183], # [4,] is blue [165, 24, 5]]], np.uint8) # some of these tests work... # Proceeding from the concrete to the abstract... DIM = (5, 5) def test1(a): red, grn, blu = a[0,0] print "r=%r, g=%r, b=%r" % (red, grn, blu) # prints r=64, g=237, b=108 red, grn, blu = a[4, 4] print "r=%r, g=%r, b=%r" % (red, grn, blu) # prints r=165, g=24, b=5 def test2(a): # this is a threshold filter rmin, rmax, gmin, gmax, bmin, bmax = 0,178, 0,178, 100,255 very_blue_pixels = [] w = DIM[0] x = y = 0 # searches rows first (x is turning faster, so...) for yslice in a: for xyslice in yslice: if 0: print "x=%r, y=%r, type(xyslice)= %r" % (x, y, type(xyslice)) if 0: print "a[x=%i, y=%i] = %r" % (x, y, a[x,y]) red, grn, blu = xyslice if (rmin <= red <= rmax) and \ (gmin <= grn <= gmax) and \ (bmin <= blu <= bmax): if 1: print "a very blue pixel at x=%i, y=%i, rgb=%r" % (x, y, xyslice) very_blue_pixels.append((x,y)) y += 1 if y==w: y=0 # ugly x += 1 print very_blue_pixels def test3(a, THRESH): # for my dataset I can optimize the above loop like this # at the cost of getting less output since the THRESHOLD # is only one number very_blue_pixels = [] w = w=DIM[0] x = y = 0 for yslice in a: for xyslice in yslice: red, grn, blu = xyslice if (red <= THRESH) and (grn <= THRESH) and (THRESH <= blu): very_blue_pixels.append((x, y)) if 1: print "I see blue a[x=%i, y=%i] pixel = %r" % (x, y, xyslice) y += 1 if y==w: y=0 x += 1 print very_blue_pixels # # I see blue a[x=3, y=2] pixel = array([ 71, 78, 250], dtype=uint8) # I see blue a[x=4, y=3] pixel = array([ 23, 127, 183], dtype=uint8) # [(0, 3), (1, 2), (2, 0), (2, 1), (2, 3), (3, 0), (3, 2), (4, 3)] def test4(a): RED, GRN, BLU = 0, 1, 2 rmin, rmax, gmin, gmax, bmin, bmax = 0,178, 0,178, 100,255 line_pix = ((rmin <= a[:, :, RED] <= rmax) & (gmin <= a[:, :, GRN] <= gmax) & (bmin <= a[:, :, BLU] <= bmax)) print "test4: line_pix = %r" % (line_pix) # ValueError: The truth value of an array with more than # one element is ambiguous. Use a.any() or a.all() # There is one more test in the attached file who"s # results mystify me. It goes like this: def test5(a, rmax, gmax, bmin): RED, GRN, BLU = 0, 1, 2 line_pix = ((a[:, :, RED] <= rmax) & (a[:, :, GRN] <= gmax) & (a[:, :, BLU] >= bmin)) print "test5: line_pix = %r" % (line_pix) # returns # test5: line_pix = array([ # [False, False, False, True, False], # [False, False, True, False, True], # [ True, True, True, True, False], # [ True, False, True, True, True], # [False, False, False, True, False]], dtype=bool) All I need is a line like this: [(x,y),(x,y)(x,y)...(x,y)] preferably 1 y per x at the "best" (cluster) location, without going into kmeans or anything obscure. Any suggestions? This is for a volunteer project vectorizing scanned stripcharts for http://IEDRO.org Thank you in advance. -------------- next part -------------- A non-text attachment was scrubbed... Name: test_threshold.py Type: application/octet-stream Size: 5840 bytes Desc: not available URL: From Richard_Martineau at xyratex.com Wed Sep 17 08:45:57 2008 From: Richard_Martineau at xyratex.com (Richie) Date: Wed, 17 Sep 2008 05:45:57 -0700 (PDT) Subject: [Numpy-discussion] Build Numpy Source using Visual Studio 6.0 In-Reply-To: References: <01ce652c-e3fe-4ffd-9c72-45629111f1d3@e53g2000hsa.googlegroups.com> Message-ID: <4ad8d5d5-0eaf-4cd5-af22-2e3649486a01@f36g2000hsa.googlegroups.com> Thanks for your message Matthieu I agree that would give me a build but I'd rather have Numpy and Scipy source codesets each in their own workspaces so that as we try to port across to PharLap we have full visibility. I wonder if anyone has done similar in visual studio 2003 or higher? I'm running vs2005 for the rest of my work but the initial requirement for this Python work is that we use visual studio 6. Looks like I'm doomed! Thanks again Richie On Sep 17, 12:34?pm, "Matthieu Brucher" wrote: > 2008/9/17 Richie : > > > Hello > > > I wonder if anyone can advise me or has done similar to the > > following? > > > Basically I've downloaded the Python 2.5.2 source code that builds > > with Visual Studio 6.0. I've built Python for windows. This was easy > > (it even came with the pcbuild.dsw workspace file). Great! > > > Now comes the troubled bit...I now look for similar source code for > > Python extensions Numpy and Scipy but there appears to be no source > > code for vs6.0 and the source that I can find is not all obvious. > > Looks like these are normally built using gcc > > I'd rather say that the files should be compiled by distutils, which > will call the appropriate compiler. So if you compiled the VS6.0 > version of Python with VS 6.0, you should be able to just call python > setup.py install. > > Matthieu > -- > French PhD student > Information System Engineer > Website:http://matthieu-brucher.developpez.com/ > Blogs:http://matt.eifelle.comandhttp://blog.developpez.com/?blog=92 > LinkedIn:http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > Numpy-discussion mailing list > Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion From stefan at sun.ac.za Wed Sep 17 12:37:53 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 17 Sep 2008 18:37:53 +0200 Subject: [Numpy-discussion] color filtering In-Reply-To: <664321.69222.qm@web53906.mail.re2.yahoo.com> References: <664321.69222.qm@web53906.mail.re2.yahoo.com> Message-ID: <9457e7c80809170937j506b664dn38003d0cdaf097a1@mail.gmail.com> Hey Paul Maybe this helps: # silly algorithm for guessing blue pixels >>> bluemask = (a[...,2] > 1.25*a[...,1]) & (a[...,2] > 1.25*a[...,0]) >>> a[bluemask, :] array([[107, 11, 185], [101, 91, 251], [ 61, 23, 79], [ 25, 131, 211], [169, 83, 214], [ 2, 88, 216], [ 71, 78, 250], [ 23, 127, 183]], dtype=uint8) Cheers St?fan 2008/9/17 paul taney : > I need to do a color filter; I have it working in python > but it runs forever on some of my images. And I dont know > enuf c or c++ to use weave (or Boost or Pyrex or...) > # But while testing we can use: > > import numpy as np > a = np.array( > [[[ 64, 237, 108], # x y > [185, 240, 68], # [0, 1] > [ 17, 116, 55], # [0, 2] > [107, 11, 185], # [0, 3] is blue > [154, 30, 52]], # [0, 4] > > [[251, 73, 150], # x y > [ 47, 225, 97], # [1, 1] > [101, 91, 251], # [1, 2] is blue > [ 61, 23, 79], # [1, 3] > [161, 22, 165]], # [1, 4] > > [[ 4, 162, 188], # [2, 0] > [ 25, 131, 211], # [2, 1] > [162, 112, 177], # [2, 2] > [169, 83, 214], # [2, 3] is blue > [254, 139, 95]], > > [[ 2, 88, 216], # [3, 0] is blue > [230, 63, 192], > [ 71, 78, 250], # [3, 2] is blue > [113, 151, 142], > [104, 65, 127]], > > [[ 6, 191, 220], # [4, 0] is blue > [ 38, 141, 4], > [245, 41, 23], > [ 23, 127, 183], # [4,] is blue > [165, 24, 5]]], np.uint8) From bradknox at cs.utexas.edu Wed Sep 17 12:54:55 2008 From: bradknox at cs.utexas.edu (W Bradley Knox) Date: Wed, 17 Sep 2008 11:54:55 -0500 Subject: [Numpy-discussion] installing numpy and scipy on OS X 10.4 Message-ID: <74b137730809170954l13d1e7ecned8d7ff424a47292@mail.gmail.com> Hi everyone, I'm trying to install numpy and scipy on my Mac that runs OS X 10.4. It looks like my only option is to use the subversion copies. After running "python setup.py install" in the numpy directory, it seems to install, despite some error messages tucked away in the middle of the installation output (I can send that if I need to). Trying to import numpy fails though: >>> import numpy Traceback (most recent call last): File "", line 1, in File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/__init__.py", line 116, in raise ImportError(msg) ImportError: Error importing numpy: you should not try to import numpy from its source directory; please exit the numpy source tree, and relaunch your python intepreter from there. This happens from any directory, including /. Any thoughts on how I could fix this? Or I could also try another installation method. The numpy binaries worked previously but then scipy didn't work with them, and this discussion (http://projects.scipy.org/pipermail/numpy-discussion/2008-August/036480.html) advised installing them both from svn. Thanks, Brad From bradknox at cs.utexas.edu Wed Sep 17 13:42:25 2008 From: bradknox at cs.utexas.edu (W Bradley Knox) Date: Wed, 17 Sep 2008 12:42:25 -0500 Subject: [Numpy-discussion] installing numpy and scipy on OS X 10.4 In-Reply-To: <74b137730809170954l13d1e7ecned8d7ff424a47292@mail.gmail.com> References: <74b137730809170954l13d1e7ecned8d7ff424a47292@mail.gmail.com> Message-ID: <74b137730809171042teb67466xfa294194d69bc88f@mail.gmail.com> Okay, nevermind. I was able to install using the Superpack, even though I'm using MacPython (ActivePython is supposedly required). Brad On Wed, Sep 17, 2008 at 11:54 AM, W Bradley Knox wrote: > Hi everyone, > > I'm trying to install numpy and scipy on my Mac that runs OS X 10.4. > It looks like my only option is to use the subversion copies. After > running "python setup.py install" in the numpy directory, it seems to > install, despite some error messages tucked away in the middle of the > installation output (I can send that if I need to). > > Trying to import numpy fails though: > >>>> import numpy > Traceback (most recent call last): > File "", line 1, in > File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/__init__.py", > line 116, in > raise ImportError(msg) > ImportError: Error importing numpy: you should not try to import numpy from > its source directory; please exit the numpy source tree, and relaunch > your python intepreter from there. > > This happens from any directory, including /. Any thoughts on how I > could fix this? > > Or I could also try another installation method. The numpy binaries > worked previously but then scipy didn't work with them, and this > discussion (http://projects.scipy.org/pipermail/numpy-discussion/2008-August/036480.html) > advised installing them both from svn. > > Thanks, > Brad > From paultaney at yahoo.com Wed Sep 17 14:20:03 2008 From: paultaney at yahoo.com (paul taney) Date: Wed, 17 Sep 2008 11:20:03 -0700 (PDT) Subject: [Numpy-discussion] color filtering In-Reply-To: <9457e7c80809170937j506b664dn38003d0cdaf097a1@mail.gmail.com> Message-ID: <556044.53288.qm@web53911.mail.re2.yahoo.com> > Maybe this helps: > > # silly algorithm for guessing blue pixels > >>> bluemask = (a[...,2] > 1.25*a[...,1]) & (a[...,2] > 1.25*a[...,0]) > > >>> a[bluemask, :] > array([[107, 11, 185], > [101, 91, 251], > [ 61, 23, 79], > [ 25, 131, 211], > [169, 83, 214], > [ 2, 88, 216], > [ 71, 78, 250], > [ 23, 127, 183]], dtype=uint8) > > Cheers > St?fan Thanks St?fan, that"s very cool. But the xy coords were tossed. Can you show me a cheap way to return a line: line=[(3, 0), (2, 1), (0, 2), (1, 2), (3, 2), (0, 3), (4, 3), (3, 4)] type-o-thing? I think I decyphered the boolean array I was getting from def test5(a, THRESH): RED, GRN, BLU = 0, 1, 2 line_pix = ((a[:, :, RED] <= THRESH) & (a[:, :, GRN] <= THRESH) & (a[:, :, BLU] >= THRESH)) print "test5: line_pix = %r" % (line_pix) return line_pix #test5: line_pix = array( # x=0 1 2 3 4 # y=0 [[False, False, False, True, False], # [3, 0] is blue # 1 [False, False, True, False, False], # [2, 1] is blue # 2 [ True, True, False, True, False], # [0, 2] xxx, [1, 2] is blue, [3, 2] xxx # 3 [ True, False, False, True, False], # [0, 3] is blue, [4, 3] # 4 [False, False, False, True, False]]) # [3, 4] is blue # So now I have to fawn over it for the coords. Is there a cheaper way? def dump_bool_array(b): very_blue_pixels = [] w = DIM[0] x = y = 0 for xyslice in b: for bool in xyslice: if bool: very_blue_pixels.append((x, y)) x += 1 if x==w: x=0 y += 1 print "dump_bool_array=%r" % (very_blue_pixels) # [(3, 0), (2, 1), (0, 2), (1, 2), (3, 2), (0, 3), (4, 3), (3, 4)] THrow away the other attachment. It had a very bad bug, swapping x and y. Ug. paul ---- def test7(a, fact): RED, GRN, BLU = 0, 1, 2 bluemask = (a[...,BLU] > fact*a[...,GRN]) & \ (a[...,BLU] > fact*a[...,RED]) b = a[bluemask, :] print "test 7 with factor=%f" % fact return b -------------- next part -------------- A non-text attachment was scrubbed... Name: test_minmax.py Type: application/octet-stream Size: 6947 bytes Desc: not available URL: From stefan at sun.ac.za Wed Sep 17 15:44:43 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 17 Sep 2008 21:44:43 +0200 Subject: [Numpy-discussion] color filtering In-Reply-To: <556044.53288.qm@web53911.mail.re2.yahoo.com> References: <9457e7c80809170937j506b664dn38003d0cdaf097a1@mail.gmail.com> <556044.53288.qm@web53911.mail.re2.yahoo.com> Message-ID: <9457e7c80809171244n32328409g860fec42bc2b03ae@mail.gmail.com> 2008/9/17 paul taney : >> # silly algorithm for guessing blue pixels >> >>> bluemask = (a[...,2] > 1.25*a[...,1]) & (a[...,2] > 1.25*a[...,0]) > > Thanks St?fan, that"s very cool. But the xy coords were tossed. > Can you show me a cheap way to return a line: > > line=[(3, 0), (2, 1), (0, 2), (1, 2), (3, 2), (0, 3), (4, 3), (3, 4)] > > type-o-thing? Sure, how about np.array(bluemask.nonzero()).swapaxes(0,1) Does that do what you want? Cheers St?fan From robert.kern at gmail.com Wed Sep 17 15:50:47 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 17 Sep 2008 14:50:47 -0500 Subject: [Numpy-discussion] installing numpy and scipy on OS X 10.4 In-Reply-To: <74b137730809170954l13d1e7ecned8d7ff424a47292@mail.gmail.com> References: <74b137730809170954l13d1e7ecned8d7ff424a47292@mail.gmail.com> Message-ID: <3d375d730809171250q772a92f2n29ebc7d12744c316@mail.gmail.com> On Wed, Sep 17, 2008 at 11:54, W Bradley Knox wrote: > Hi everyone, > > I'm trying to install numpy and scipy on my Mac that runs OS X 10.4. > It looks like my only option is to use the subversion copies. After > running "python setup.py install" in the numpy directory, it seems to > install, despite some error messages tucked away in the middle of the > installation output (I can send that if I need to). > > Trying to import numpy fails though: > >>>> import numpy > Traceback (most recent call last): > File "", line 1, in > File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/__init__.py", > line 116, in > raise ImportError(msg) > ImportError: Error importing numpy: you should not try to import numpy from > its source directory; please exit the numpy source tree, and relaunch > your python intepreter from there. > > This happens from any directory, including /. Any thoughts on how I > could fix this? Can we have your build log? Something went wrong there, and some files didn't get installed (which __init__ uses to determine whether it has been installed or being run from the source tree). Even though you have gotten the Superpack to work for you, this is a problem I would like to at least know about. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From paultaney at yahoo.com Wed Sep 17 16:33:26 2008 From: paultaney at yahoo.com (paul taney) Date: Wed, 17 Sep 2008 13:33:26 -0700 (PDT) Subject: [Numpy-discussion] color filtering In-Reply-To: <9457e7c80809171244n32328409g860fec42bc2b03ae@mail.gmail.com> Message-ID: <718953.50940.qm@web53907.mail.re2.yahoo.com> > coords were tossed. > > Can you show me a cheap way to return a line: > > > > line=[(3, 0), (2, 1), (0, 2), (1, 2), (3, 2), (0, > 3), (4, 3), (3, 4)] > > > > type-o-thing? > > Sure, how about > > np.array(bluemask.nonzero()).swapaxes(0,1) > > Does that do what you want? Exactly. Very cool stuff. Thanks alot! RED, GRN, BLU = 0, 1, 2 def test8(a, f): bluemask = (a[...,BLU] > f*a[...,GRN]) & \ (a[...,BLU] > f*a[...,RED]) print "test 8 with factor=%f" % f return np.array(bluemask.nonzero()).swapaxes(0,1) print test8(a, 1.4) test 8 with factor=1.400000 [[0 3] [1 2] [2 1] [3 0] [3 4] [4 3]] paul From robert.kern at gmail.com Wed Sep 17 16:33:43 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 17 Sep 2008 15:33:43 -0500 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <5d3194020809170512idfa560s6548ffcca406c325@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <5d3194020809170512idfa560s6548ffcca406c325@mail.gmail.com> Message-ID: <3d375d730809171333p654090c9p28aa83a12d6e919a@mail.gmail.com> On Wed, Sep 17, 2008 at 07:12, Arnar Flatberg wrote: > > On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern wrote: >> >> So, I could use some comments on the workflow. Does this look sensible >> to everyone? How else would you like to use it? > > Works for me. I would love to use it as a part of an ipython session. > Initially, I sprinkled som $profile in > a library of my own, then I thought I would use the -s (SETUP) option to > initialize some data to pass into the $profiled functions. However, I have > probably misunderstood the meaning of the -s option. Could you give an > example of its use? It's mostly for using cProfile on a whole script. Since line_profiler is already only active for certain functions, it's probably not all that useful. At the time I added the feature, I was trying to profile (part of) a GUI program using wx and VTK, both of which take a long time to load and taking up a good chunk of my profile time. If I had a script, load_wx_vtk.py: import wx from enthought.tvtk import tvtk Then I could just do: $ kernprof.py -s load_wx_vtk.py my_script.py The script at the end still needs to be the main thing that executes. > I ended in doing a %edit in ipython to write a small > test script with testdata and library imports, used kernprof.py and > view_line_prof.py from ipython and captured the output from view_line_prof > into ipython. Pheew .... now, there's perhaps an easier way? It should be straightforward to make an ipython %magic, yes. There are some questions as to how that would work, though. Do you want the top-level code in the file to be profiled? Or do you just want functions defined inside to be profiled? Do you want to be able to specify functions defined outside of the code to be profiled? > BTW, in the requirements you may want to mention argparse > (view_line_prof.py) Oops! Yes. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ondrej at certik.cz Wed Sep 17 19:09:39 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 18 Sep 2008 01:09:39 +0200 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> Message-ID: <85b5c3130809171609g263ab76bj1a3afe493c352278@mail.gmail.com> On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern wrote: > On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg wrote: >> That would make me an extremely happy user, I've been looking for this for >> years! >> I can't imagine I'm the only one who profiles some hundred lines of code and >> ends up with 90% of total time in the dot-function > > For the time being, you can grab it here: > > http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/ > > It requires Cython and a C compiler to build. I'm still debating > myself about the desired workflow for using it, but for now, it only > profiles functions which you have registered with it. I have made the > profiler work as a decorator to make this easy. E.g., > > from line_profiler import LineProfiler > > profile = LineProfiler() > > @profile > def my_slow_func(): > ... > > profile.dump_stats('my_slow_func.lprof') > > This is kind of inconvenient, so I have a little script that handles > everything except for the @profile itself. It started as a script to > run cProfile nicely, so it actually does that by default. > > I took pystone.py from the Python source and added a couple of > @profile decorators to demonstrate: > > > [line_profiler]$ ./kernprof.py --help > Usage: ./kernprof.py [-s setupfile] [-o output_file_path] scriptfile [arg] ... > > Options: > -h, --help show this help message and exit > -l, --line-by-line Use the line-by-line profiler from the line_profiler > module instead of cProfile. Implies --builtin. > -b, --builtin Put 'profile' in the builtins. Use 'profile.enable()' > and 'profile.disable()' in your code to turn it on and > off, or '@profile' to decorate a single function, or > 'with profile:' to profile a single section of code. > -o OUTFILE, --outfile=OUTFILE > Save stats to > -s SETUP, --setup=SETUP > Code to execute before the code to profile > > [line_profiler]$ ./kernprof.py -l pystone.py > Pystone(1.1) time for 50000 passes = 11.34 > This machine benchmarks at 4409.17 pystones/second > Wrote profile results to pystone.py.lprof > > [line_profiler]$ ./view_line_prof.py pystone.py.lprof > Timer unit: 1e-06 s > > File: pystone.py > Function: Proc0 at line 79 > Total time: 8.46516 s [...] This is what I am getting: $ ./kernprof.py -l pystone.py Wrote profile results to pystone.py.lprof $ ./view_line_prof.py pystone.py.lprof Timer unit: 1e-06 s $ So I think you meant: $ ./kernprof.py -l mystone.py 20628 Wrote profile results to mystone.py.lprof $ ./view_line_prof.py mystone.py.lprof Timer unit: 1e-06 s File: pystone.py Function: Proc0 at line 79 Total time: 13.0803 s [...] Now it works. This is an excellent piece of software! Nice job. Just today I needed such a thing! How do you easily install it? I usually do "python setup.py install --home=~/lib" and I have the PYTHONPATH + PATH setup in my .bashrc, but I then need to manually remove the stuff from my ~/lib if I want to uninstall, which sucks. So this time I just did "python setup.py build" and moved the .so file manually to the current dir. But there must be a better way. What is your workflow? Anyway, so I used it on my code and here is what I got: File: hermes1d/mesh.py Function: integrate_function at line 119 Total time: 0.647412 s Line # Hits Time % Time Line Contents ===================================================== 119 @profile 120 def integrate_function(self, f): 121 """ 122 Integrate the function "f" on the element. 123 """ 124 96 1091 0.2 from numpy import array 125 96 461070 71.2 from scipy.integrate import quadrature 126 96 496 0.1 a, b = self.nodes[0].x, self.nodes[1].x 127 96 418 0.1 def func(x): 128 #print x 129 #print array([f.f(y) for y in x]) 130 return array([f.f(y) for y in x]) 131 96 183903 28.4 val, err = quadrature(func, a, b) 132 #print val, a, b 133 #stop 134 96 434 0.1 return val This is interesting, because 70% of time is spent on "from scipy.integrate import quadrature". So this is not good. So I moved it out and voila! My code is 3.4x faster. I really didn't know that importing (scipy.integrate) is *so* slow. File: hermes1d/mesh.py Function: integrate_function at line 121 Total time: 0.189429 s Line # Hits Time % Time Line Contents ===================================================== 121 @profile 122 def integrate_function(self, f): 123 """ 124 Integrate the function "f" on the element. 125 """ 126 96 1080 0.6 from numpy import array 127 96 499 0.3 a, b = self.nodes[0].x, self.nodes[1].x 128 96 408 0.2 def func(x): 129 #print x 130 #print array([f.f(y) for y in x]) 131 return array([f.f(y) for y in x]) 132 96 187020 98.7 val, err = quadrature(func, a, b) 133 #print val, a, b 134 #stop 135 96 422 0.2 return val Btw, after profiling quadrature in scipy: 110 384 1241 0.1 err = 100.0 111 384 1184 0.1 val = err 112 384 1167 0.1 n = 1 113 384 4086 0.5 vfunc = vectorize1(func, args, vec_func=vec_func) 114 1152 4509 0.5 while (err > tol) and (n < maxiter): 115 768 879911 97.4 newval = fixed_quad(vfunc, a, b, (), n)[0] 116 768 4072 0.5 err = abs(newval-val) 117 768 2470 0.3 val = newval 118 768 2513 0.3 n = n + 1 119 384 1381 0.2 if n == maxiter: 120 print "maxiter (%d) exceeded. Latest difference = %e" % (n,err) 121 384 1225 0.1 return val, err And fixed_quad: 46 768 630714 74.4 [x,w] = p_roots(n) 47 768 8757 1.0 x = real(x) 48 768 13312 1.6 ainf, binf = map(isinf,(a,b)) 49 768 2912 0.3 if ainf or binf: 50 raise ValueError, "Gaussian quadrature is only available for " \ 51 "finite limits." 52 768 22106 2.6 y = (b-a)*(x+1)/2.0 + a 53 768 170229 20.1 return (b-a)/2.0*sum(w*func(y,*args),0), None it gives interesting results. For example in my case, where I need to call this often, I should cache the p_roots results. Anyways, this profiler is exciting, it makes things easy to profile. Ondrej From robert.kern at gmail.com Wed Sep 17 19:29:09 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 17 Sep 2008 18:29:09 -0500 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <85b5c3130809171609g263ab76bj1a3afe493c352278@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <85b5c3130809171609g263ab76bj1a3afe493c352278@mail.gmail.com> Message-ID: <3d375d730809171629i2e29ea69weed51abd0c2a533d@mail.gmail.com> On Wed, Sep 17, 2008 at 18:09, Ondrej Certik wrote: > On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern wrote: >> On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg wrote: >>> That would make me an extremely happy user, I've been looking for this for >>> years! >>> I can't imagine I'm the only one who profiles some hundred lines of code and >>> ends up with 90% of total time in the dot-function >> >> For the time being, you can grab it here: >> >> http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/ >> >> It requires Cython and a C compiler to build. I'm still debating >> myself about the desired workflow for using it, but for now, it only >> profiles functions which you have registered with it. I have made the >> profiler work as a decorator to make this easy. E.g., >> >> from line_profiler import LineProfiler >> >> profile = LineProfiler() >> >> @profile >> def my_slow_func(): >> ... >> >> profile.dump_stats('my_slow_func.lprof') >> >> This is kind of inconvenient, so I have a little script that handles >> everything except for the @profile itself. It started as a script to >> run cProfile nicely, so it actually does that by default. >> >> I took pystone.py from the Python source and added a couple of >> @profile decorators to demonstrate: >> >> >> [line_profiler]$ ./kernprof.py --help >> Usage: ./kernprof.py [-s setupfile] [-o output_file_path] scriptfile [arg] ... >> >> Options: >> -h, --help show this help message and exit >> -l, --line-by-line Use the line-by-line profiler from the line_profiler >> module instead of cProfile. Implies --builtin. >> -b, --builtin Put 'profile' in the builtins. Use 'profile.enable()' >> and 'profile.disable()' in your code to turn it on and >> off, or '@profile' to decorate a single function, or >> 'with profile:' to profile a single section of code. >> -o OUTFILE, --outfile=OUTFILE >> Save stats to >> -s SETUP, --setup=SETUP >> Code to execute before the code to profile >> >> [line_profiler]$ ./kernprof.py -l pystone.py >> Pystone(1.1) time for 50000 passes = 11.34 >> This machine benchmarks at 4409.17 pystones/second >> Wrote profile results to pystone.py.lprof >> >> [line_profiler]$ ./view_line_prof.py pystone.py.lprof >> Timer unit: 1e-06 s >> >> File: pystone.py >> Function: Proc0 at line 79 >> Total time: 8.46516 s > [...] > > This is what I am getting: > > $ ./kernprof.py -l pystone.py > Wrote profile results to pystone.py.lprof > $ ./view_line_prof.py pystone.py.lprof > Timer unit: 1e-06 s > > $ > > So I think you meant: > > $ ./kernprof.py -l mystone.py > 20628 > Wrote profile results to mystone.py.lprof > $ ./view_line_prof.py mystone.py.lprof > Timer unit: 1e-06 s > > File: pystone.py > Function: Proc0 at line 79 > Total time: 13.0803 s > [...] > > Now it works. No, I meant pystone.py. My script-finding code may have (incorrectly) found a different, uninstrumented pystone.py file somewhere else, though. Try with "./pystone.py". > This is an excellent piece of software! Nice job. Just today I needed > such a thing! > > > How do you easily install it? "python setup.py install" should have installed the module. I haven't done anything with the scripts, yet. > I usually do "python setup.py install > --home=~/lib" and I have the PYTHONPATH + PATH setup in my .bashrc, > but I then need to manually remove the stuff from my ~/lib if I want > to uninstall, which sucks. So this time I just did "python setup.py > build" and moved the .so file manually to the current dir. But there > must be a better way. What is your workflow? For things I am developing on, I build them in-place, using the setuptools "develop" command to add the appropriate path to the easy-install.pth file. To remove, I would just edit that file. For thing I'm not developing on, I usually build and install an egg if at all possible. But then, I'm typically on a single-user box where I'm root, so I sometimes do nasty and unsanitary things like chown -R rkern:rkern /usr/local/. > Anyway, so I used it on my code and here is what I got: > > File: hermes1d/mesh.py > Function: integrate_function at line 119 > Total time: 0.647412 s > > Line # Hits Time % Time Line Contents > ===================================================== > 119 @profile > 120 def integrate_function(self, f): > 121 """ > 122 Integrate the function > "f" on the element. > 123 """ > 124 96 1091 0.2 from numpy import array > 125 96 461070 71.2 from scipy.integrate > import quadrature > 126 96 496 0.1 a, b = > self.nodes[0].x, self.nodes[1].x > 127 96 418 0.1 def func(x): > 128 #print x > 129 #print > array([f.f(y) for y in x]) > 130 return > array([f.f(y) for y in x]) > 131 96 183903 28.4 val, err = > quadrature(func, a, b) > 132 #print val, a, b > 133 #stop > 134 96 434 0.1 return val > > > > This is interesting, because 70% of time is spent on "from > scipy.integrate import quadrature". So this is not good. > > So I moved it out and voila! My code is 3.4x faster. I really didn't > know that importing (scipy.integrate) is *so* slow. It should only be slow the first time. If it's not, then there's a problem somewhere (and I wouldn't put it past being in the profiler; it might screw up something in the import mechanism). Can you do the import both outside the function *and* inside to see if that changes the result? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From spam4bsimons at yahoo.ca Wed Sep 17 22:54:12 2008 From: spam4bsimons at yahoo.ca (Brendan Simons) Date: Wed, 17 Sep 2008 22:54:12 -0400 Subject: [Numpy-discussion] Help speeding up element-wise operations for video processing In-Reply-To: <200809171134.03021.faltet@pytables.org> References: <458186.45235.qm@web51601.mail.re2.yahoo.com> <9457e7c80809160150i5acb33d7t95193c58a27ab76d@mail.gmail.com> <61ADA597-D8B2-48DC-A48E-EA8F2031540B@yahoo.ca> <200809171134.03021.faltet@pytables.org> Message-ID: Thanks for the cogent response Fransesc. It's nice to know when you're up against a "Really Hard Problem", rather than just missing something obvious. I've started reading Ulrich Drepper report, and it looks like a great place to start. Cheers, Brendan On 17-Sep-08, at 5:34 AM, Francesc Alted wrote: > A Wednesday 17 September 2008, Brendan Simons escrigu?: > [clip] >> I would love a c-types code snippet. I'm not very handy in c. Since >> I gather numpy is row-major, I thought I up and down crops very >> quickly by moving the start and end pointers of the array. For >> cropping left and right, is there a fast c command for "copy while >> skipping every nth hundred bytes"? > > There is no such efficient "copy while skipping every nth hundred > bytes" > thing for C or for any other language. You are facing here a > fundamental problem in the design of modern processor architectures, > namely, the (large) memory latency. That means that when accessing > memory in discontiguous patterns like the one you indicated, that will > keep the processor waiting for the data most of the time. There are > ways to give hints to compilers in order to perform a better > pre-fetching of interesting data, but this is a rather complex > process, > and the improvements can be meager in most of cases. > > In brief, if you don't have much time to spend of this, my advice is > to > use just regular assignment or memcpy (whatever is more comfortable > for > your situation), because you won't be able to get more performance > than > what these will offer. > > However, if you have more time and want to look for ways on how to > scratch more performance on different memory access patterns, it is > always a wise thing to have a look at the excellent "What Every > Programmer Should Know About Memory" report: > > htt://people.redhat.com/drepper/cpumemory.pdf > > [Incidentally, this is possible one of the best reports available on > the > subject of memory access on nowadays architectures (a critical thing > for achieving maximum performance), and besides it is available for > free, so there is no excuse to not have look at it ;-)] > > Cheers, > > -- > Francesc Alted > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From cournape at gmail.com Thu Sep 18 01:29:58 2008 From: cournape at gmail.com (David Cournapeau) Date: Thu, 18 Sep 2008 14:29:58 +0900 Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <07bbbcff-365f-4051-924b-dae8d353b0ed@8g2000hse.googlegroups.com> <48CF96FD.9090805@ar.media.kyoto-u.ac.jp> <70cd1d75-3a32-4140-8a72-d2cdd650ef62@m3g2000hsc.googlegroups.com> <48CFCD08.10109@ar.media.kyoto-u.ac.jp> <48D05F4B.30508@ar.media.kyoto-u.ac.jp> <48D0CAF5.4070109@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220809172229t53f0a590p22fee11b7411882d@mail.gmail.com> On Wed, Sep 17, 2008 at 7:13 PM, Francis wrote: > In general the IT dep. don't want changes since changes require weeks > of testing to make sure that existing applications are not > conflicting, etc. I am not sure they would fulfill my request. > Especially since it has no priority at all. > Ok, this really bothered that you had to install sage to use numpy. You don't have to do it since you already wasted quite some time, but I updated a small project of mine that I almost forgot: garnumpy. It is a set of makefiles to build numpy, scipy and all its dependencies (blas, lapack, fftw, etc...) automatically: - It download the tarballs for you. - It installs everything in a configurable directory, so that it is self-contained It is there; http://github.com/cournape/garnumpy/ You can get the sources with git or just the tarball generated by github if you don't have/don't want git. Basic usage: set up main_prefix in the file garnumpy.conf.mk (or keep the default value: it installs in $HOME/garnumpyinstall) cd platform/scipy && make install source startgarnumpy.sh # You now have your env set up for numpy and scipy python -c "import scipy" If you want to remove it, just rm -rf the path set up in main_prefix. cheers, David From stefan at sun.ac.za Thu Sep 18 03:25:08 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 18 Sep 2008 09:25:08 +0200 Subject: [Numpy-discussion] Help speeding up element-wise operations for video processing In-Reply-To: <61ADA597-D8B2-48DC-A48E-EA8F2031540B@yahoo.ca> References: <458186.45235.qm@web51601.mail.re2.yahoo.com> <9457e7c80809160150i5acb33d7t95193c58a27ab76d@mail.gmail.com> <61ADA597-D8B2-48DC-A48E-EA8F2031540B@yahoo.ca> Message-ID: <9457e7c80809180025l651afb5au8637a01e08621b3a@mail.gmail.com> Hey Brendan 2008/9/17 Brendan Simons : > I would love a c-types code snippet. I'm not very handy in c. Since > I gather numpy is row-major, I thought I up and down crops very > quickly by moving the start and end pointers of the array. For > cropping left and right, is there a fast c command for "copy while > skipping every nth hundred bytes"? Not sure which way you decided to go, but here are some code snippets: http://mentat.za.net/hg/graycomatrix and http://mentat.za.net/source/connected_components.tar.bz2 Also take a look at http://scipy.org/Cookbook/Ctypes Good luck! St?fan From nadavh at visionsense.com Thu Sep 18 03:37:30 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Thu, 18 Sep 2008 10:37:30 +0300 Subject: [Numpy-discussion] Help speeding up element-wise operations forvideo processing References: <458186.45235.qm@web51601.mail.re2.yahoo.com><9457e7c80809160150i5acb33d7t95193c58a27ab76d@mail.gmail.com><61ADA597-D8B2-48DC-A48E-EA8F2031540B@yahoo.ca> <9457e7c80809180025l651afb5au8637a01e08621b3a@mail.gmail.com> Message-ID: <710F2847B0018641891D9A216027636029C285@ex3.envision.co.il> Sorry for not relating to the details of the problem, but, did you take a look pygpu? It intends to be able to enable image processing as video rate. Nadav -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? St?fan van der Walt ????: ? 18-??????-08 10:25 ??: Discussion of Numerical Python ????: Re: [Numpy-discussion] Help speeding up element-wise operations forvideo processing Hey Brendan 2008/9/17 Brendan Simons : > I would love a c-types code snippet. I'm not very handy in c. Since > I gather numpy is row-major, I thought I up and down crops very > quickly by moving the start and end pointers of the array. For > cropping left and right, is there a fast c command for "copy while > skipping every nth hundred bytes"? Not sure which way you decided to go, but here are some code snippets: http://mentat.za.net/hg/graycomatrix and http://mentat.za.net/source/connected_components.tar.bz2 Also take a look at http://scipy.org/Cookbook/Ctypes Good luck! St?fan _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3728 bytes Desc: not available URL: From ondrej at certik.cz Thu Sep 18 06:46:29 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 18 Sep 2008 12:46:29 +0200 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <3d375d730809171629i2e29ea69weed51abd0c2a533d@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <85b5c3130809171609g263ab76bj1a3afe493c352278@mail.gmail.com> <3d375d730809171629i2e29ea69weed51abd0c2a533d@mail.gmail.com> Message-ID: <85b5c3130809180346k34810615ie043e5609aa7cf74@mail.gmail.com> On Thu, Sep 18, 2008 at 1:29 AM, Robert Kern wrote: > On Wed, Sep 17, 2008 at 18:09, Ondrej Certik wrote: >> On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern wrote: >>> On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg wrote: >>>> That would make me an extremely happy user, I've been looking for this for >>>> years! >>>> I can't imagine I'm the only one who profiles some hundred lines of code and >>>> ends up with 90% of total time in the dot-function >>> >>> For the time being, you can grab it here: >>> >>> http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/ >>> >>> It requires Cython and a C compiler to build. I'm still debating >>> myself about the desired workflow for using it, but for now, it only >>> profiles functions which you have registered with it. I have made the >>> profiler work as a decorator to make this easy. E.g., >>> >>> from line_profiler import LineProfiler >>> >>> profile = LineProfiler() >>> >>> @profile >>> def my_slow_func(): >>> ... >>> >>> profile.dump_stats('my_slow_func.lprof') >>> >>> This is kind of inconvenient, so I have a little script that handles >>> everything except for the @profile itself. It started as a script to >>> run cProfile nicely, so it actually does that by default. >>> >>> I took pystone.py from the Python source and added a couple of >>> @profile decorators to demonstrate: >>> >>> >>> [line_profiler]$ ./kernprof.py --help >>> Usage: ./kernprof.py [-s setupfile] [-o output_file_path] scriptfile [arg] ... >>> >>> Options: >>> -h, --help show this help message and exit >>> -l, --line-by-line Use the line-by-line profiler from the line_profiler >>> module instead of cProfile. Implies --builtin. >>> -b, --builtin Put 'profile' in the builtins. Use 'profile.enable()' >>> and 'profile.disable()' in your code to turn it on and >>> off, or '@profile' to decorate a single function, or >>> 'with profile:' to profile a single section of code. >>> -o OUTFILE, --outfile=OUTFILE >>> Save stats to >>> -s SETUP, --setup=SETUP >>> Code to execute before the code to profile >>> >>> [line_profiler]$ ./kernprof.py -l pystone.py >>> Pystone(1.1) time for 50000 passes = 11.34 >>> This machine benchmarks at 4409.17 pystones/second >>> Wrote profile results to pystone.py.lprof >>> >>> [line_profiler]$ ./view_line_prof.py pystone.py.lprof >>> Timer unit: 1e-06 s >>> >>> File: pystone.py >>> Function: Proc0 at line 79 >>> Total time: 8.46516 s >> [...] >> >> This is what I am getting: >> >> $ ./kernprof.py -l pystone.py >> Wrote profile results to pystone.py.lprof >> $ ./view_line_prof.py pystone.py.lprof >> Timer unit: 1e-06 s >> >> $ >> >> So I think you meant: >> >> $ ./kernprof.py -l mystone.py >> 20628 >> Wrote profile results to mystone.py.lprof >> $ ./view_line_prof.py mystone.py.lprof >> Timer unit: 1e-06 s >> >> File: pystone.py >> Function: Proc0 at line 79 >> Total time: 13.0803 s >> [...] >> >> Now it works. > > No, I meant pystone.py. My script-finding code may have (incorrectly) > found a different, uninstrumented pystone.py file somewhere else, > though. Try with "./pystone.py". > >> This is an excellent piece of software! Nice job. Just today I needed >> such a thing! >> >> >> How do you easily install it? > > "python setup.py install" should have installed the module. I haven't > done anything with the scripts, yet. > >> I usually do "python setup.py install >> --home=~/lib" and I have the PYTHONPATH + PATH setup in my .bashrc, >> but I then need to manually remove the stuff from my ~/lib if I want >> to uninstall, which sucks. So this time I just did "python setup.py >> build" and moved the .so file manually to the current dir. But there >> must be a better way. What is your workflow? > > For things I am developing on, I build them in-place, using the > setuptools "develop" command to add the appropriate path to the > easy-install.pth file. To remove, I would just edit that file. > > For thing I'm not developing on, I usually build and install an egg if > at all possible. > > But then, I'm typically on a single-user box where I'm root, so I > sometimes do nasty and unsanitary things like chown -R rkern:rkern > /usr/local/. > >> Anyway, so I used it on my code and here is what I got: >> >> File: hermes1d/mesh.py >> Function: integrate_function at line 119 >> Total time: 0.647412 s >> >> Line # Hits Time % Time Line Contents >> ===================================================== >> 119 @profile >> 120 def integrate_function(self, f): >> 121 """ >> 122 Integrate the function >> "f" on the element. >> 123 """ >> 124 96 1091 0.2 from numpy import array >> 125 96 461070 71.2 from scipy.integrate >> import quadrature >> 126 96 496 0.1 a, b = >> self.nodes[0].x, self.nodes[1].x >> 127 96 418 0.1 def func(x): >> 128 #print x >> 129 #print >> array([f.f(y) for y in x]) >> 130 return >> array([f.f(y) for y in x]) >> 131 96 183903 28.4 val, err = >> quadrature(func, a, b) >> 132 #print val, a, b >> 133 #stop >> 134 96 434 0.1 return val >> >> >> >> This is interesting, because 70% of time is spent on "from >> scipy.integrate import quadrature". So this is not good. >> >> So I moved it out and voila! My code is 3.4x faster. I really didn't >> know that importing (scipy.integrate) is *so* slow. > > It should only be slow the first time. If it's not, then there's a > problem somewhere (and I wouldn't put it past being in the profiler; > it might screw up something in the import mechanism). Can you do the > import both outside the function *and* inside to see if that changes > the result? Sure. "b.py": from scipy.integrate import quadrature --------------- def a(): b = 1 for x in range(1000000): a() --------------- "a.py": --------------- def a(): from scipy.integrate import quadrature b = 1 for x in range(1000000): a() --------------- and: ondra at pc232:~/repos/hermes1d$ time python a.py real 0m2.655s user 0m2.612s sys 0m0.044s ondra at pc232:~/repos/hermes1d$ time python b.py real 0m0.620s user 0m0.540s sys 0m0.080s So I guess that speaks for itself. Don't have time to dig more, but I think it's this line in scipy/integrate/__init__: __all__ = filter(lambda s:not s.startswith('_'),dir()) But maybe also something else, I don't know. Ondrej Ondrej From cimrman3 at ntc.zcu.cz Thu Sep 18 07:01:26 2008 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Thu, 18 Sep 2008 13:01:26 +0200 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> Message-ID: <48D23506.7080109@ntc.zcu.cz> Hi Robert, Robert Kern wrote: > On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg wrote: >> That would make me an extremely happy user, I've been looking for this for >> years! >> I can't imagine I'm the only one who profiles some hundred lines of code and >> ends up with 90% of total time in the dot-function > > For the time being, you can grab it here: > > http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/ > > It requires Cython and a C compiler to build. I'm still debating > myself about the desired workflow for using it, but for now, it only > profiles functions which you have registered with it. I have made the > profiler work as a decorator to make this easy. E.g., many thanks for this! I have wanted to try out the profiler but failed to build it (changeset 6 0de294aa75bf): $ python setup.py install --root=/home/share/software/ running install running build running build_py creating build creating build/lib.linux-i686-2.4 copying line_profiler.py -> build/lib.linux-i686-2.4 running build_ext cythoning _line_profiler.pyx to _line_profiler.c building '_line_profiler' extension creating build/temp.linux-i686-2.4 i486-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC -I/usr/include/python2.4 -c -I/usr/include/python2.4 -c _line_profiler.c -o build/temp.linux-i686-2.4/_line_profiler.o _line_profiler.c:1614: error: 'T_LONGLONG' undeclared here (not in a function) error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1 I have cython-0.9.8.1 and GCC 4.1.2, 32-bit machine. r. From F.Drossaert at googlemail.com Thu Sep 18 07:26:26 2008 From: F.Drossaert at googlemail.com (Francis) Date: Thu, 18 Sep 2008 04:26:26 -0700 (PDT) Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: <5b8d13220809172229t53f0a590p22fee11b7411882d@mail.gmail.com> References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <07bbbcff-365f-4051-924b-dae8d353b0ed@8g2000hse.googlegroups.com> <48CF96FD.9090805@ar.media.kyoto-u.ac.jp> <70cd1d75-3a32-4140-8a72-d2cdd650ef62@m3g2000hsc.googlegroups.com> <48CFCD08.10109@ar.media.kyoto-u.ac.jp> <48D05F4B.30508@ar.media.kyoto-u.ac.jp> <48D0CAF5.4070109@ar.media.kyoto-u.ac.jp> <5b8d13220809172229t53f0a590p22fee11b7411882d@mail.gmail.com> Message-ID: <9106f5a1-0288-4801-bd2e-089038def377@z72g2000hsb.googlegroups.com> Thank you for your effort. I guess garnumpy reflects the idea in this Pylab discussion: http://www.scipy.org/PyLab Again I get errors in libblas/lapack related to gfortran (local variable problems). I replaced the libblas.a and the liblaplack.a by the ones of sage. And started make install again. It seems to work until it tries to configure/install umfpack. /users/francisd/garnumpy/lib/libblas.a(xerbla.o): In function `xerbla_': xerbla.f:(.text+0x1b): undefined reference to `_g95_get_ioparm' xerbla.f:(.text+0x22): undefined reference to `_g95_filename' xerbla.f:(.text+0x33): undefined reference to `_g95_line' xerbla.f:(.text+0x47): undefined reference to `_g95_ioparm' xerbla.f:(.text+0x58): undefined reference to `_g95_ioparm' xerbla.f:(.text+0x69): undefined reference to `_g95_ioparm' xerbla.f:(.text+0x81): undefined reference to `_g95_ioparm' xerbla.f:(.text+0x93): undefined reference to `_g95_st_write' xerbla.f:(.text+0xa1): undefined reference to `_g95_transfer_character' xerbla.f:(.text+0xaf): undefined reference to `_g95_transfer_integer' xerbla.f:(.text+0xb4): undefined reference to `_g95_st_write_done' xerbla.f:(.text+0xbb): undefined reference to `_g95_filename' xerbla.f:(.text+0xcc): undefined reference to `_g95_line' xerbla.f:(.text+0xd7): undefined reference to `_g95_stop_blank' collect2: ld returned 1 exit status make[4]: *** [umfpack_di_demo] Error 1 Perhaps it possible to skip the installation of the umfpack since I probably won't need it or it requires the other libblas.a. Just wildly guessing here. PS Tried to work with sage but numpy is troublesome. Lacks intuitiveness, don't like it. From pzs at dcs.gla.ac.uk Thu Sep 18 07:27:49 2008 From: pzs at dcs.gla.ac.uk (Peter Saffrey) Date: Thu, 18 Sep 2008 12:27:49 +0100 Subject: [Numpy-discussion] Medians that ignore values Message-ID: <48D23B35.8040902@dcs.gla.ac.uk> I have data from biological experiments that is represented as a list of about 5000 triples. I would like to convert this to a list of the median of each triple. I did some profiling and found that numpy was much about 12 times faster for this application than using regular Python lists and a list median implementation. I'll be performing quite a few mathematical operations on these values, so using numpy arrays seems sensible. The only problem is that my data has gaps in it - where an experiment failed, a "triple" will not have three values. Some will have 2, 1 or even no values. To keep the arrays regular so that they can be used by numpy, is there some dummy value I can use to fill these gaps that will be ignored by the median routine? I tried NaN for this, but as far as median is concerned, it counts as infinity: >>> from numpy import * >>> median(array([1,3,nan])) 3.0 >>> median(array([1,nan,nan])) nan Is this the correct behavior for median with nan? Is there a fix for this or am I going to have to settle with using lists? Thanks, Peter From nadavh at visionsense.com Thu Sep 18 07:35:14 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Thu, 18 Sep 2008 14:35:14 +0300 Subject: [Numpy-discussion] Medians that ignore values References: <48D23B35.8040902@dcs.gla.ac.uk> Message-ID: <710F2847B0018641891D9A216027636029C289@ex3.envision.co.il> I think you need to use masked arrays. Nadav -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Peter Saffrey ????: ? 18-??????-08 14:27 ??: numpy-discussion at scipy.org ????: [Numpy-discussion] Medians that ignore values I have data from biological experiments that is represented as a list of about 5000 triples. I would like to convert this to a list of the median of each triple. I did some profiling and found that numpy was much about 12 times faster for this application than using regular Python lists and a list median implementation. I'll be performing quite a few mathematical operations on these values, so using numpy arrays seems sensible. The only problem is that my data has gaps in it - where an experiment failed, a "triple" will not have three values. Some will have 2, 1 or even no values. To keep the arrays regular so that they can be used by numpy, is there some dummy value I can use to fill these gaps that will be ignored by the median routine? I tried NaN for this, but as far as median is concerned, it counts as infinity: >>> from numpy import * >>> median(array([1,3,nan])) 3.0 >>> median(array([1,nan,nan])) nan Is this the correct behavior for median with nan? Is there a fix for this or am I going to have to settle with using lists? Thanks, Peter _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3462 bytes Desc: not available URL: From ondrej at certik.cz Thu Sep 18 07:48:24 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 18 Sep 2008 13:48:24 +0200 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <48D23506.7080109@ntc.zcu.cz> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <48D23506.7080109@ntc.zcu.cz> Message-ID: <85b5c3130809180448o4dd12f94n5b6ebc94e0c1a8e3@mail.gmail.com> On Thu, Sep 18, 2008 at 1:01 PM, Robert Cimrman wrote: > Hi Robert, > > Robert Kern wrote: >> On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg wrote: >>> That would make me an extremely happy user, I've been looking for this for >>> years! >>> I can't imagine I'm the only one who profiles some hundred lines of code and >>> ends up with 90% of total time in the dot-function >> >> For the time being, you can grab it here: >> >> http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/ >> >> It requires Cython and a C compiler to build. I'm still debating >> myself about the desired workflow for using it, but for now, it only >> profiles functions which you have registered with it. I have made the >> profiler work as a decorator to make this easy. E.g., > > many thanks for this! I have wanted to try out the profiler but failed > to build it (changeset 6 0de294aa75bf): > > $ python setup.py install --root=/home/share/software/ > running install > running build > running build_py > creating build > creating build/lib.linux-i686-2.4 > copying line_profiler.py -> build/lib.linux-i686-2.4 > running build_ext > cythoning _line_profiler.pyx to _line_profiler.c > building '_line_profiler' extension > creating build/temp.linux-i686-2.4 > i486-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC > -I/usr/include/python2.4 -c -I/usr/include/python2.4 -c _line_profiler.c > -o build/temp.linux-i686-2.4/_line_profiler.o > _line_profiler.c:1614: error: 'T_LONGLONG' undeclared here (not in a > function) > error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1 > > I have cython-0.9.8.1 and GCC 4.1.2, 32-bit machine. I am telling you all the time Robert to use Debian that it just works and you say, no no, gentoo is the best. :) Ondrej From bsouthey at gmail.com Thu Sep 18 09:28:45 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 18 Sep 2008 08:28:45 -0500 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <710F2847B0018641891D9A216027636029C289@ex3.envision.co.il> References: <48D23B35.8040902@dcs.gla.ac.uk> <710F2847B0018641891D9A216027636029C289@ex3.envision.co.il> Message-ID: <48D2578D.9060106@gmail.com> Nadav Horesh wrote: > I think you need to use masked arrays. > > Nadav > > > -----????? ??????----- > ???: numpy-discussion-bounces at scipy.org ??? Peter Saffrey > ????: ? 18-??????-08 14:27 > ??: numpy-discussion at scipy.org > ????: [Numpy-discussion] Medians that ignore values > > I have data from biological experiments that is represented as a list of > about 5000 triples. I would like to convert this to a list of the median > of each triple. I did some profiling and found that numpy was much about > 12 times faster for this application than using regular Python lists and > a list median implementation. I'll be performing quite a few > mathematical operations on these values, so using numpy arrays seems > sensible. > > The only problem is that my data has gaps in it - where an experiment > failed, a "triple" will not have three values. Some will have 2, 1 or > even no values. To keep the arrays regular so that they can be used by > numpy, is there some dummy value I can use to fill these gaps that will > be ignored by the median routine? > > I tried NaN for this, but as far as median is concerned, it counts as > infinity: > > >>> from numpy import * > >>> median(array([1,3,nan])) > 3.0 > >>> median(array([1,nan,nan])) > nan > > Is this the correct behavior for median with nan? Is there a fix for > this or am I going to have to settle with using lists? > > Thanks, > > Peter > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > Hi, The counting of infinity is correct due to the implementation of IEEE Standard for Binary Floating-Point for Arithmetic (IEEE 754). You might want to try isfinite() to first remove nan, +/- infinity before doing that. numpy.median(a[numpy.isfinite(a)]) Bruce From rmay31 at gmail.com Thu Sep 18 10:12:58 2008 From: rmay31 at gmail.com (Ryan May) Date: Thu, 18 Sep 2008 09:12:58 -0500 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <85b5c3130809180448o4dd12f94n5b6ebc94e0c1a8e3@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <48D23506.7080109@ntc.zcu.cz> <85b5c3130809180448o4dd12f94n5b6ebc94e0c1a8e3@mail.gmail.com> Message-ID: <48D261EA.7020404@gmail.com> Ondrej Certik wrote: > On Thu, Sep 18, 2008 at 1:01 PM, Robert Cimrman wrote: >>> >>> It requires Cython and a C compiler to build. I'm still debating >>> myself about the desired workflow for using it, but for now, it only >>> profiles functions which you have registered with it. I have made the >>> profiler work as a decorator to make this easy. E.g., >> many thanks for this! I have wanted to try out the profiler but failed >> to build it (changeset 6 0de294aa75bf): >> >> $ python setup.py install --root=/home/share/software/ >> running install >> running build >> running build_py >> creating build >> creating build/lib.linux-i686-2.4 >> copying line_profiler.py -> build/lib.linux-i686-2.4 >> running build_ext >> cythoning _line_profiler.pyx to _line_profiler.c >> building '_line_profiler' extension >> creating build/temp.linux-i686-2.4 >> i486-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC >> -I/usr/include/python2.4 -c -I/usr/include/python2.4 -c _line_profiler.c >> -o build/temp.linux-i686-2.4/_line_profiler.o >> _line_profiler.c:1614: error: 'T_LONGLONG' undeclared here (not in a >> function) >> error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1 >> >> I have cython-0.9.8.1 and GCC 4.1.2, 32-bit machine. > > I am telling you all the time Robert to use Debian that it just works > and you say, no no, gentoo is the best. :) And what's wrong with that? :) Once you get over the learning curve, Gentoo works just fine. Must be Robert K.'s fault. :) Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From jh at physics.ucf.edu Thu Sep 18 10:32:21 2008 From: jh at physics.ucf.edu (jh at physics.ucf.edu) Date: Thu, 18 Sep 2008 10:32:21 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: (numpy-discussion-request@scipy.org) References: Message-ID: > You might want to try isfinite() to first remove nan, +/- infinity > before doing that. > numpy.median(a[numpy.isfinite(a)]) We just had this discussion a month or two ago, I think even on this list, and continued it at the SciPy conference. The problem with numpy.median(a[numpy.isfinite(a)]) is that it breaks when you have a multi-dimensional array, such as an array of 5000x3 as in this case, and take median down an axis. The example above flattens the array and eliminates the possibility of taking the median down an axis in a single call, as the poster desires. Currently the only way you can handle NaNs is by using masked arrays. Create a mask by doing isfinite(a), then call the masked array median(). There's an example here: http://sd-2116.dedibox.fr/pydocweb/doc/numpy.ma/ Note that our competitor language IDL does have a /nan flag to its single median routine, making this common task much easier in that language than ours. --jh-- From pzs at dcs.gla.ac.uk Thu Sep 18 10:59:12 2008 From: pzs at dcs.gla.ac.uk (Peter Saffrey) Date: Thu, 18 Sep 2008 15:59:12 +0100 Subject: [Numpy-discussion] Medians that ignore values Message-ID: <48D26CC0.3010806@dcs.gla.ac.uk> physics.ucf.edu> writes: > Currently the only way you can handle NaNs is by using masked arrays. > Create a mask by doing isfinite(a), then call the masked array > median(). There's an example here: > > http://sd-2116.dedibox.fr/pydocweb/doc/numpy.ma/ > I had looked at masked arrays, but couldn't quite get them to work. Generating them is fine (I've randomly introduced a few nan values into this array): >>> from numeric import * >>> from pylab import rand >>> a = rand(10,3) >>> a[a > 0.8] = nan >>> m = ma.masked_array(a, isnan(a)) >>> m array(data = [[ 5.97400164e-01 1.00000000e+20 1.00000000e+20] [ 3.34623242e-01 6.53582662e-02 2.12298948e-01] [ 2.11879853e-01 1.00000000e+20 3.57822574e-01] [ 6.06911592e-01 1.96229341e-01 5.49953059e-02] [ 1.00000000e+20 2.75493584e-01 4.70929957e-01] [ 2.92845118e-01 2.11261529e-02 3.49211381e-02] [ 7.11963636e-01 2.17277855e-01 5.45487384e-02] [ 5.20995579e-01 7.57676845e-01 1.00000000e+20] [ 1.84189196e-01 7.58291436e-02 6.26567116e-01] [ 2.42083978e-01 1.00000000e+20 2.30202562e-02]], mask = [[False True True] [False False False] [False True False] [False False False] [ True False False] [False False False] [False False False] [False False True] [False False False] [False True False]], fill_value=1e+20) Remember I want medians of each triple, so I need to median the transposed matrix: >>> median(m.T) array([ 1.00000000e+20, 2.12298948e-01, 3.57822574e-01, 1.96229341e-01, 4.70929957e-01, 3.49211381e-02, 2.17277855e-01, 7.57676845e-01, 1.84189196e-01, 2.42083978e-01]) The first value is NaN, indicating that the median routine has failed to ignore the masked values. What have I missed? Thanks, Peter From ondrej at certik.cz Thu Sep 18 11:03:04 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 18 Sep 2008 17:03:04 +0200 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <48D261EA.7020404@gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <48D23506.7080109@ntc.zcu.cz> <85b5c3130809180448o4dd12f94n5b6ebc94e0c1a8e3@mail.gmail.com> <48D261EA.7020404@gmail.com> Message-ID: <85b5c3130809180803r5b12897an759c82d40d1def9d@mail.gmail.com> On Thu, Sep 18, 2008 at 4:12 PM, Ryan May wrote: > Ondrej Certik wrote: >> On Thu, Sep 18, 2008 at 1:01 PM, Robert Cimrman wrote: >>>> >>>> It requires Cython and a C compiler to build. I'm still debating >>>> myself about the desired workflow for using it, but for now, it only >>>> profiles functions which you have registered with it. I have made the >>>> profiler work as a decorator to make this easy. E.g., >>> many thanks for this! I have wanted to try out the profiler but failed >>> to build it (changeset 6 0de294aa75bf): >>> >>> $ python setup.py install --root=/home/share/software/ >>> running install >>> running build >>> running build_py >>> creating build >>> creating build/lib.linux-i686-2.4 >>> copying line_profiler.py -> build/lib.linux-i686-2.4 >>> running build_ext >>> cythoning _line_profiler.pyx to _line_profiler.c >>> building '_line_profiler' extension >>> creating build/temp.linux-i686-2.4 >>> i486-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC >>> -I/usr/include/python2.4 -c -I/usr/include/python2.4 -c _line_profiler.c >>> -o build/temp.linux-i686-2.4/_line_profiler.o >>> _line_profiler.c:1614: error: 'T_LONGLONG' undeclared here (not in a >>> function) >>> error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1 >>> >>> I have cython-0.9.8.1 and GCC 4.1.2, 32-bit machine. >> >> I am telling you all the time Robert to use Debian that it just works >> and you say, no no, gentoo is the best. :) > > And what's wrong with that? :) Once you get over the learning curve, > Gentoo works just fine. Must be Robert K.'s fault. :) Well, I think if Robert C. hasn't yet get over the learning curve after so many years of hard work, maybe the learning curve is too steep. :) Anyway, back to work: Robert K., I noticed that if I profile some function, I get results like this for example: ---- 40 3072 46952 6.6 [x,w] = p_roots(n) 41 3072 24192 3.4 x = real(x) 42 3072 34470 4.8 ainf, binf = map(isinf,(a,b)) 43 3072 6565 0.9 if ainf or binf: 44 raise ValueError, "Gaussian quadrature is only available for " \ 45 "finite limits." 46 3072 5093 0.7 if not reference: 47 x = (b-a)*(x+1)/2.0 + a 48 3072 594190 83.5 return (b-a)/2.0*sum(w*func(x,*args),0) --- Then if I turn profiling of the func() method, I get this: -------------------- 40 3072 46999 4.6 [x,w] = p_roots(n) 41 3072 24313 2.4 x = real(x) 42 3072 34327 3.4 ainf, binf = map(isinf,(a,b)) 43 3072 6190 0.6 if ainf or binf: 44 raise ValueError, "Gaussian quadrature is only available for " \ 45 "finite limits." 46 3072 4918 0.5 if not reference: 47 x = (b-a)*(x+1)/2.0 + a 48 3072 906876 88.6 return (b-a)/2.0*sum(w*func(x,*args),0) ----------- So the timing raises a lot. For obvious reasons, that's the overhead of the profiler. But the problem is that then the timings just don't fit, e.g. if I sum the total time spent in subfunctions, it doesn't account for all the time printed on the respective line in the parent function. I don't know if there is any way to fix it, or even worth fixing. So I guess one should just use the profiling info to determine which line to fix. Do you think it's worthy to implement line profiling for all lines and then make sure that all timings match? What is your experience. Ondrej From ondrej at certik.cz Thu Sep 18 11:22:57 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 18 Sep 2008 17:22:57 +0200 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <85b5c3130809180803r5b12897an759c82d40d1def9d@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <48D23506.7080109@ntc.zcu.cz> <85b5c3130809180448o4dd12f94n5b6ebc94e0c1a8e3@mail.gmail.com> <48D261EA.7020404@gmail.com> <85b5c3130809180803r5b12897an759c82d40d1def9d@mail.gmail.com> Message-ID: <85b5c3130809180822k14c7c991o158c73f459ced288@mail.gmail.com> > So the timing raises a lot. For obvious reasons, that's the overhead > of the profiler. But the problem is that then the timings just don't > fit, e.g. if I sum the total time spent in subfunctions, it doesn't > account for all the time printed on the respective line in the parent > function. > > I don't know if there is any way to fix it, or even worth fixing. So I > guess one should just use the profiling info to determine which line > to fix. > > Do you think it's worthy to implement line profiling for all lines and > then make sure that all timings match? What is your experience. The reason I want this is so that I can determine just by looking at the timing how much time is spent at the line and how much time is spent at the functions that are being called at the line. I think it is doable, what do you think? One would trace how much time was wasted in python_trace_callback and then just substract this time from all parent function's lines timings. Btw Robin, how is Matlab doing it? Ondrej From paultaney at yahoo.com Thu Sep 18 12:36:27 2008 From: paultaney at yahoo.com (paul taney) Date: Thu, 18 Sep 2008 09:36:27 -0700 (PDT) Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D26CC0.3010806@dcs.gla.ac.uk> Message-ID: <814146.56047.qm@web53911.mail.re2.yahoo.com> Hi, > >>> median(m.T) > array([ 1.00000000e+20, 2.12298948e-01, 3.57822574e-01, I believe 1.00000000e+20 is a reserved value and signifys the missing value or NaN in your case. That"s the way it was in a Fortran77 package I worked with ten years ago... --- On Thu, 9/18/08, Peter Saffrey wrote: > From: Peter Saffrey > Subject: Re: [Numpy-discussion] Medians that ignore values > To: numpy-discussion at scipy.org > Date: Thursday, September 18, 2008, 10:59 AM > physics.ucf.edu> writes: > > > Currently the only way you can handle NaNs is by > using masked arrays. > > Create a mask by doing isfinite(a), then call the > masked array > > median(). There's an example here: > > > > http://sd-2116.dedibox.fr/pydocweb/doc/numpy.ma/ > > > > I had looked at masked arrays, but couldn't quite get > them to work. > Generating them is fine (I've randomly introduced a few > nan values into > this array): > > >>> from numeric import * > >>> from pylab import rand > >>> a = rand(10,3) > >>> a[a > 0.8] = nan > >>> m = ma.masked_array(a, isnan(a)) > >>> m > array(data = > [[ 5.97400164e-01 1.00000000e+20 1.00000000e+20] > [ 3.34623242e-01 6.53582662e-02 2.12298948e-01] > [ 2.11879853e-01 1.00000000e+20 3.57822574e-01] > [ 6.06911592e-01 1.96229341e-01 5.49953059e-02] > [ 1.00000000e+20 2.75493584e-01 4.70929957e-01] > [ 2.92845118e-01 2.11261529e-02 3.49211381e-02] > [ 7.11963636e-01 2.17277855e-01 5.45487384e-02] > [ 5.20995579e-01 7.57676845e-01 1.00000000e+20] > [ 1.84189196e-01 7.58291436e-02 6.26567116e-01] > [ 2.42083978e-01 1.00000000e+20 2.30202562e-02]], > mask = > [[False True True] > [False False False] > [False True False] > [False False False] > [ True False False] > [False False False] > [False False False] > [False False True] > [False False False] > [False True False]], > fill_value=1e+20) > > > Remember I want medians of each triple, so I need to median > the > transposed matrix: > > >>> median(m.T) > array([ 1.00000000e+20, 2.12298948e-01, > 3.57822574e-01, > 1.96229341e-01, 4.70929957e-01, > 3.49211381e-02, > 2.17277855e-01, 7.57676845e-01, > 1.84189196e-01, > 2.42083978e-01]) > > The first value is NaN, indicating that the median routine > has failed to > ignore the masked values. What have I missed? > > Thanks, > > Peter > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From pgmdevlist at gmail.com Thu Sep 18 12:48:51 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 18 Sep 2008 12:48:51 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D26CC0.3010806@dcs.gla.ac.uk> References: <48D26CC0.3010806@dcs.gla.ac.uk> Message-ID: <200809181248.51981.pgmdevlist@gmail.com> On Thursday 18 September 2008 10:59:12 Peter Saffrey wrote: > I had looked at masked arrays, but couldn't quite get them to work. That's unfortunate. > >>> from numeric import * Mmh, typo? > >>> from pylab import rand > >>> a = rand(10,3) > >>> a[a > 0.8] = nan > >>> m = ma.masked_array(a, isnan(a)) > >>> m Another way would be m = ma.masked_where(a>0.8,a) > Remember I want medians of each triple, so I need to median the > transposed matrix: > >>> median(m.T) Ohoh. What version of numpy are you using ? if you don't give an axis parameter, you should get the median of the flattened array, therefore a scalar, not an array. Anyway: you should use ma.median for masked arrays. Else, you're just keeping the NaNs where they were. From arnar.flatberg at gmail.com Thu Sep 18 12:51:36 2008 From: arnar.flatberg at gmail.com (Arnar Flatberg) Date: Thu, 18 Sep 2008 18:51:36 +0200 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <3d375d730809171333p654090c9p28aa83a12d6e919a@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <5d3194020809170512idfa560s6548ffcca406c325@mail.gmail.com> <3d375d730809171333p654090c9p28aa83a12d6e919a@mail.gmail.com> Message-ID: <5d3194020809180951x7669b5cfsf2c2a97be339d16b@mail.gmail.com> On Wed, Sep 17, 2008 at 10:33 PM, Robert Kern wrote: > On Wed, Sep 17, 2008 at 07:12, Arnar Flatberg > wrote: > > > > On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern > wrote: > > It should be straightforward to make an ipython %magic, yes. There are > some questions as to how that would work, though. Do you want the > top-level code in the file to be profiled? > I dont think that will be my primary use. > Or do you just want > functions defined inside to be profiled? That was my initial thought, yes > Do you want to be able to > specify functions defined outside of the code to be profiled? Sounds complicated, but that would be nice. I'll play around with ipythons %magic to see if I can get a workflow Im comfortable with. I was thinking something in the line of ipython's %prun for a quick looksy, and the possibility to easily write small scripts to test different inputs (small/large arrays etc.) to a specific function. Any recommendations are very welcome. I do not have much experience in profiling code. Arnar -------------- next part -------------- An HTML attachment was scrubbed... URL: From pzs at dcs.gla.ac.uk Thu Sep 18 13:31:18 2008 From: pzs at dcs.gla.ac.uk (Peter Saffrey) Date: Thu, 18 Sep 2008 17:31:18 +0000 (UTC) Subject: [Numpy-discussion] Medians that ignore values References: <48D26CC0.3010806@dcs.gla.ac.uk> <200809181248.51981.pgmdevlist@gmail.com> Message-ID: Pierre GM gmail.com> writes: > Mmh, typo? > Yes, apologies. I was aiming for thorough, but ended up just careless. It's been a long day. > Ohoh. What version of numpy are you using ? The version in the Ubuntu package repository. It says 1:1.0.4-6ubuntu3. > if you don't give an axis > parameter, you should get the median of the flattened array, therefore a > scalar, not an array. Not for my version. >>> a = rand(10,3) >>> a array([[ 0.1269796 , 0.43003978, 0.4700416 ], [ 0.28867077, 0.85265318, 0.35908364], [ 0.72967127, 0.41856028, 0.54724918], [ 0.28821876, 0.69684144, 0.54647616], [ 0.09592476, 0.83704808, 0.52425368], [ 0.743552 , 0.44433314, 0.7362179 ], [ 0.4283931 , 0.13305385, 0.68422292], [ 0.68860674, 0.15057373, 0.99206493], [ 0.31846329, 0.77237046, 0.986883 ], [ 0.4578616 , 0.4580833 , 0.97754176]]) >>> median(a.T) array([ 0.43003978, 0.35908364, 0.54724918, 0.54647616, 0.52425368, 0.7362179 , 0.4283931 , 0.68860674, 0.77237046, 0.4580833 ]) > Anyway: you should use ma.median for masked arrays. Else, you're just keeping > the NaNs where they were. > That will be the problem. My version does not have median or mean methods for masked arrays, only the average() method. According to this page: http://www.scipy.org/Download 1.1.0 is the latest release. Do I need to use an SVN build to get the ma.median functionality? Peter From charlesr.harris at gmail.com Thu Sep 18 14:23:34 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 18 Sep 2008 12:23:34 -0600 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: References: <48D26CC0.3010806@dcs.gla.ac.uk> <200809181248.51981.pgmdevlist@gmail.com> Message-ID: On Thu, Sep 18, 2008 at 11:31 AM, Peter Saffrey wrote: > Pierre GM gmail.com> writes: > > > Mmh, typo? > > > > Yes, apologies. I was aiming for thorough, but ended up just careless. It's > been > a long day. > > > Ohoh. What version of numpy are you using ? > > The version in the Ubuntu package repository. It says 1:1.0.4-6ubuntu3. > > > if you don't give an axis > > parameter, you should get the median of the flattened array, therefore a > > scalar, not an array. > > Not for my version. > > >>> a = rand(10,3) > >>> a > array([[ 0.1269796 , 0.43003978, 0.4700416 ], > [ 0.28867077, 0.85265318, 0.35908364], > [ 0.72967127, 0.41856028, 0.54724918], > [ 0.28821876, 0.69684144, 0.54647616], > [ 0.09592476, 0.83704808, 0.52425368], > [ 0.743552 , 0.44433314, 0.7362179 ], > [ 0.4283931 , 0.13305385, 0.68422292], > [ 0.68860674, 0.15057373, 0.99206493], > [ 0.31846329, 0.77237046, 0.986883 ], > [ 0.4578616 , 0.4580833 , 0.97754176]]) > >>> median(a.T) > array([ 0.43003978, 0.35908364, 0.54724918, 0.54647616, 0.52425368, > 0.7362179 , 0.4283931 , 0.68860674, 0.77237046, 0.4580833 ]) > > > Anyway: you should use ma.median for masked arrays. Else, you're just > keeping > > the NaNs where they were. > > > > That will be the problem. My version does not have median or mean methods > for > masked arrays, only the average() method. > > According to this page: > > http://www.scipy.org/Download > > 1.1.0 is the latest release. Do I need to use an SVN build to get the > ma.median > functionality? > 1.1.1 is the latest release and 1.2 is coming out shortly. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Thu Sep 18 14:23:25 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 18 Sep 2008 14:23:25 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: References: <48D26CC0.3010806@dcs.gla.ac.uk> <200809181248.51981.pgmdevlist@gmail.com> Message-ID: <200809181423.26158.pgmdevlist@gmail.com> On Thursday 18 September 2008 13:31:18 Peter Saffrey wrote: > The version in the Ubuntu package repository. It says 1:1.0.4-6ubuntu3. So it's 1.0 ? It's fairly old, that would explain. > > if you don't give an axis > > parameter, you should get the median of the flattened array, therefore a > > scalar, not an array. > > Not for my version. Indeed. Looks like the default axis changed from 0 in 1.0 to None in the incoming 1.2. But that's a detail at this point. > > Anyway: you should use ma.median for masked arrays. Else, you're just > > keeping the NaNs where they were. > > That will be the problem. My version does not have median or mean methods > for masked arrays, only the average() method. The method mean has always been around for masked arrays, so has the corresponding function. But I'm surprised, median has been in numpy.ma.extras for a while. Maybe not 1.0... > According to this page: > > http://www.scipy.org/Download > > 1.1.0 is the latest release. You need to update your internet ;) 1.1.1 was released 6 weeks ago. > Do I need to use an SVN build to get the > ma.median functionality? No, you can install 1.1.1, that should work. Note that I just fixed a bug in median in SVN (it would fail when trying to get the median of a 2D array with axis=1), so you may want to check this one instead if you feel like it. You can still use 1.1.1 : as a quick workaround the forementioned bug, use ma.median(a.T, axis=0) instead of ma.median(a,axis=1) when working w/ 2D arrays. From charlesr.harris at gmail.com Thu Sep 18 14:40:28 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 18 Sep 2008 12:40:28 -0600 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <200809181423.26158.pgmdevlist@gmail.com> References: <48D26CC0.3010806@dcs.gla.ac.uk> <200809181248.51981.pgmdevlist@gmail.com> <200809181423.26158.pgmdevlist@gmail.com> Message-ID: On Thu, Sep 18, 2008 at 12:23 PM, Pierre GM wrote: > On Thursday 18 September 2008 13:31:18 Peter Saffrey wrote: > > The version in the Ubuntu package repository. It says 1:1.0.4-6ubuntu3. > > So it's 1.0 ? It's fairly old, that would explain. > > > > if you don't give an axis > > > parameter, you should get the median of the flattened array, therefore > a > > > scalar, not an array. > > > > Not for my version. > > Indeed. Looks like the default axis changed from 0 in 1.0 to None in the > incoming 1.2. But that's a detail at this point. > > > > Anyway: you should use ma.median for masked arrays. Else, you're just > > > keeping the NaNs where they were. > > > > That will be the problem. My version does not have median or mean methods > > for masked arrays, only the average() method. > > The method mean has always been around for masked arrays, so has the > corresponding function. But I'm surprised, median has been in > numpy.ma.extras > for a while. Maybe not 1.0... > > > According to this page: > > > > http://www.scipy.org/Download > > > > 1.1.0 is the latest release. > > You need to update your internet ;) 1.1.1 was released 6 weeks ago. > The page had a typo, I've fixed it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Thu Sep 18 14:41:06 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Thu, 18 Sep 2008 11:41:06 -0700 Subject: [Numpy-discussion] ready to tag 1.2.0 Message-ID: Hey, I would like to tag 1.2.0 from the 1.2.x branch. Are there any problems with this? In particular, are there any known problems that would require us having another release candidate? As soon as we get this release out we can start back-porting bugfixes from the trunk to the 1.2.x branch in preparation for a 1.2.1 release. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From robert.kern at gmail.com Thu Sep 18 15:37:09 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 18 Sep 2008 14:37:09 -0500 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <48D23506.7080109@ntc.zcu.cz> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <48D23506.7080109@ntc.zcu.cz> Message-ID: <3d375d730809181237x43ce1610r6dccb081d6698f5d@mail.gmail.com> On Thu, Sep 18, 2008 at 06:01, Robert Cimrman wrote: > Hi Robert, > > Robert Kern wrote: >> On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg wrote: >>> That would make me an extremely happy user, I've been looking for this for >>> years! >>> I can't imagine I'm the only one who profiles some hundred lines of code and >>> ends up with 90% of total time in the dot-function >> >> For the time being, you can grab it here: >> >> http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/ >> >> It requires Cython and a C compiler to build. I'm still debating >> myself about the desired workflow for using it, but for now, it only >> profiles functions which you have registered with it. I have made the >> profiler work as a decorator to make this easy. E.g., > > many thanks for this! I have wanted to try out the profiler but failed > to build it (changeset 6 0de294aa75bf): > > $ python setup.py install --root=/home/share/software/ > running install > running build > running build_py > creating build > creating build/lib.linux-i686-2.4 > copying line_profiler.py -> build/lib.linux-i686-2.4 > running build_ext > cythoning _line_profiler.pyx to _line_profiler.c > building '_line_profiler' extension > creating build/temp.linux-i686-2.4 > i486-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC > -I/usr/include/python2.4 -c -I/usr/include/python2.4 -c _line_profiler.c > -o build/temp.linux-i686-2.4/_line_profiler.o > _line_profiler.c:1614: error: 'T_LONGLONG' undeclared here (not in a > function) > error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1 > > I have cython-0.9.8.1 and GCC 4.1.2, 32-bit machine. It uses the #define'd macro PY_LONG_LONG. Go through your Python headers to see what this gets expanded to. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Thu Sep 18 15:54:13 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 18 Sep 2008 14:54:13 -0500 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <85b5c3130809180803r5b12897an759c82d40d1def9d@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <48D23506.7080109@ntc.zcu.cz> <85b5c3130809180448o4dd12f94n5b6ebc94e0c1a8e3@mail.gmail.com> <48D261EA.7020404@gmail.com> <85b5c3130809180803r5b12897an759c82d40d1def9d@mail.gmail.com> Message-ID: <3d375d730809181254r270fa98fve07bfe5b5051d410@mail.gmail.com> On Thu, Sep 18, 2008 at 10:03, Ondrej Certik wrote: > Anyway, back to work: Robert K., I noticed that if I profile some > function, I get results like this for example: > > ---- > 40 3072 46952 6.6 [x,w] = p_roots(n) > 41 3072 24192 3.4 x = real(x) > 42 3072 34470 4.8 ainf, binf = map(isinf,(a,b)) > 43 3072 6565 0.9 if ainf or binf: > 44 raise ValueError, > "Gaussian quadrature is only available for " \ > 45 "finite limits." > 46 3072 5093 0.7 if not reference: > 47 x = (b-a)*(x+1)/2.0 + a > 48 3072 594190 83.5 return > (b-a)/2.0*sum(w*func(x,*args),0) > --- > > Then if I turn profiling of the func() method, I get this: > > -------------------- > 40 3072 46999 4.6 [x,w] = p_roots(n) > 41 3072 24313 2.4 x = real(x) > 42 3072 34327 3.4 ainf, binf = map(isinf,(a,b)) > 43 3072 6190 0.6 if ainf or binf: > 44 raise ValueError, > "Gaussian quadrature is only available for " \ > 45 "finite limits." > 46 3072 4918 0.5 if not reference: > 47 x = (b-a)*(x+1)/2.0 + a > 48 3072 906876 88.6 return > (b-a)/2.0*sum(w*func(x,*args),0) > ----------- > > So the timing raises a lot. For obvious reasons, that's the overhead > of the profiler. But the problem is that then the timings just don't > fit, e.g. if I sum the total time spent in subfunctions, it doesn't > account for all the time printed on the respective line in the parent > function. > > I don't know if there is any way to fix it, or even worth fixing. So I > guess one should just use the profiling info to determine which line > to fix. So here's what going on: I'm being clever (and possibly too clever). When tracing is enabled, Python will call my tracing function just before each new line gets executed. If tracing isn't enabled for this function, I return. Otherwise, I grab the current time. Then, I look for the last line and time I recorded for this function. I look up the accumulator for the (code, old line) pair and record the time delta. Then I grab the current time *again*, and store the current line and this new time for the next go 'round. This way, I exclude most of the time spent inside the profiler itself and just record the time being spent in the code. The total time reported is just a sum of the recorded times, not the sum of wall-clock times spent in the function. Now, this does break down in your use case where you are profiling both the caller and callee and trying to determine how much of a line's time is being spent just by calling the function. I could record wall-clock times between the start and end of a function call, but I think that's fragile. For example, suppose you are profiling A() and B() but not C(), and both A() and C() call B(). Using the wall-clock time spent in B() will tell you that you spent more time in B() than the appropriate line (or lines!) that called it in A(). I think the most robust way to figure this out is to rewrite your code to pull out such calls onto their own lines. This is like breaking up your functions into tiny little one-liners in order to use cProfile only it doesn't suck *nearly* as much. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Thu Sep 18 15:56:18 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 18 Sep 2008 14:56:18 -0500 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <85b5c3130809180346k34810615ie043e5609aa7cf74@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <85b5c3130809171609g263ab76bj1a3afe493c352278@mail.gmail.com> <3d375d730809171629i2e29ea69weed51abd0c2a533d@mail.gmail.com> <85b5c3130809180346k34810615ie043e5609aa7cf74@mail.gmail.com> Message-ID: <3d375d730809181256m783483d4l7d5d6c80177c7980@mail.gmail.com> On Thu, Sep 18, 2008 at 05:46, Ondrej Certik wrote: > On Thu, Sep 18, 2008 at 1:29 AM, Robert Kern wrote: >> On Wed, Sep 17, 2008 at 18:09, Ondrej Certik wrote: >>> Anyway, so I used it on my code and here is what I got: >>> >>> File: hermes1d/mesh.py >>> Function: integrate_function at line 119 >>> Total time: 0.647412 s >>> >>> Line # Hits Time % Time Line Contents >>> ===================================================== >>> 119 @profile >>> 120 def integrate_function(self, f): >>> 121 """ >>> 122 Integrate the function >>> "f" on the element. >>> 123 """ >>> 124 96 1091 0.2 from numpy import array >>> 125 96 461070 71.2 from scipy.integrate >>> import quadrature >>> 126 96 496 0.1 a, b = >>> self.nodes[0].x, self.nodes[1].x >>> 127 96 418 0.1 def func(x): >>> 128 #print x >>> 129 #print >>> array([f.f(y) for y in x]) >>> 130 return >>> array([f.f(y) for y in x]) >>> 131 96 183903 28.4 val, err = >>> quadrature(func, a, b) >>> 132 #print val, a, b >>> 133 #stop >>> 134 96 434 0.1 return val >>> >>> >>> >>> This is interesting, because 70% of time is spent on "from >>> scipy.integrate import quadrature". So this is not good. >>> >>> So I moved it out and voila! My code is 3.4x faster. I really didn't >>> know that importing (scipy.integrate) is *so* slow. >> >> It should only be slow the first time. If it's not, then there's a >> problem somewhere (and I wouldn't put it past being in the profiler; >> it might screw up something in the import mechanism). Can you do the >> import both outside the function *and* inside to see if that changes >> the result? > > Sure. "b.py": > > from scipy.integrate import quadrature > > --------------- > def a(): > b = 1 > > > for x in range(1000000): > a() > --------------- > > "a.py": > > --------------- > def a(): > from scipy.integrate import quadrature > b = 1 > > > for x in range(1000000): > a() > --------------- > > and: > > ondra at pc232:~/repos/hermes1d$ time python a.py > > real 0m2.655s > user 0m2.612s > sys 0m0.044s > > ondra at pc232:~/repos/hermes1d$ time python b.py > > real 0m0.620s > user 0m0.540s > sys 0m0.080s > > > So I guess that speaks for itself. Don't have time to dig more, but I > think it's this line in scipy/integrate/__init__: > > __all__ = filter(lambda s:not s.startswith('_'),dir()) I don't see how. I can verify that it only gets executed once. import sys print 'scipy.integrate.__init__' __all__ = filter(lambda s:sys.stdout.write('.') or not s.startswith('_'),dir()) $ time python a.py scipy.integrate.__init__ .............................python a.py 2.26s user 0.72s system 98% cpu 3.009 total Anyways, a million repetitions adds about 2s to your time (oh, you may also want to do this a couple of times and wait for the times to settle down in order get the same effect from the disk caches). That's about 2 microseconds per import. The cost you are seeing in your profile is much, much greater than 2 us * 96, so I think it really is the first import that makes up the bulk of it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From f.yw at hotmail.com Thu Sep 18 17:27:49 2008 From: f.yw at hotmail.com (frank wang) Date: Thu, 18 Sep 2008 15:27:49 -0600 Subject: [Numpy-discussion] A bug in loadtxt and how to convert a string array (hex data) to decimal? Message-ID: Hi, All, I have found a bug in the loadtxt function. Here is the example. The file name is test.txt and contains: "Thist is test" 3FF 3fE 3Ef 3e8 3Df 3d9 3cF 3c7 In the Python 2.5.2, I type: test=loadtxt('test.txt',comments='"',dtype='string',converters={0:lambda s:int(s,16)}) test will contain array([['102', '3fE'], ['100', '3e8'], ['991', '3d9'], ['975', '3c7']], dtype='|S3') The first two values 102 and 100 are wrong. The reason I am doing this because that I have to process a large amount of data from a file. The data is in hex format. This is only way I found that I can efficiently convert the hex to decimal. Anyone has a good solution? Thanks Frank _________________________________________________________________ Get more out of the Web. Learn 10 hidden secrets of Windows Live. http://windowslive.com/connect/post/jamiethomson.spaces.live.com-Blog-cns!550F681DAD532637!5295.entry?ocid=TXT_TAGLM_WL_domore_092008 -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Sep 18 17:37:57 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 18 Sep 2008 16:37:57 -0500 Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: <9106f5a1-0288-4801-bd2e-089038def377@z72g2000hsb.googlegroups.com> References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <70cd1d75-3a32-4140-8a72-d2cdd650ef62@m3g2000hsc.googlegroups.com> <48CFCD08.10109@ar.media.kyoto-u.ac.jp> <48D05F4B.30508@ar.media.kyoto-u.ac.jp> <48D0CAF5.4070109@ar.media.kyoto-u.ac.jp> <5b8d13220809172229t53f0a590p22fee11b7411882d@mail.gmail.com> <9106f5a1-0288-4801-bd2e-089038def377@z72g2000hsb.googlegroups.com> Message-ID: <3d375d730809181437m24cf8074yba83b8ae9d8b8afa@mail.gmail.com> On Thu, Sep 18, 2008 at 06:26, Francis wrote: > Thank you for your effort. I guess garnumpy reflects the idea in this > Pylab discussion: http://www.scipy.org/PyLab > > Again I get errors in libblas/lapack related to gfortran (local > variable problems). I replaced the libblas.a and the liblaplack.a by > the ones of sage. And started make install again. It seems to work > until it tries to configure/install umfpack. > > /users/francisd/garnumpy/lib/libblas.a(xerbla.o): In function > `xerbla_': > xerbla.f:(.text+0x1b): undefined reference to `_g95_get_ioparm' Yes, SAGE uses g95, not gfortran. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From rmay31 at gmail.com Thu Sep 18 17:39:47 2008 From: rmay31 at gmail.com (Ryan May) Date: Thu, 18 Sep 2008 16:39:47 -0500 Subject: [Numpy-discussion] A bug in loadtxt and how to convert a string array (hex data) to decimal? In-Reply-To: References: Message-ID: <48D2CAA3.7080605@gmail.com> frank wang wrote: > > > > Hi, All, > > > > > > I have found a bug in the loadtxt function. Here is the example. The > file name is test.txt and contains: > > > "Thist is test" > > > 3FF 3fE > > > 3Ef 3e8 > > > 3Df 3d9 > > > 3cF 3c7 > > > > > > In the Python 2.5.2, I type: > > > > > > test=loadtxt('test.txt',comments='"',dtype='string',converters={0:lambda > s:int(s,16)}) > > > > > > test will contain > > > > > > array([['102', '3fE'], > > > ['100', '3e8'], > > > ['991', '3d9'], > > > ['975', '3c7']], > > > dtype='|S3') > It's because of how numpy handles strings arrays (which I admit I don't understand very well.) Basically, it's converting the numbers properly, but truncating them to 3 characters. Try this, which just forces it to expand to strings 4 characters wide: test=loadtxt('test.txt',comments='"',dtype='|S4',converters={0:lambda s:int(s,16)}) HTH, Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From pf_moore at yahoo.co.uk Thu Sep 18 17:55:11 2008 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Thu, 18 Sep 2008 22:55:11 +0100 Subject: [Numpy-discussion] Generating random samples without repeats Message-ID: I want to generate a series of random samples, to do simulations based on them. Essentially, I want to be able to produce a SAMPLESIZE * N matrix, where each row of N values consists of either 1. Integers between 1 and M (simulating M rolls of an N-sided die), or 2. A sample of N numbers between 1 and M without repeats (simulating deals of N cards from an M-card deck). Example (1) is easy, numpy.random.random_integers(1, M, (SAMPLESIZE, N)) But I can't find an obvious equivalent for (2). Am I missing something glaringly obvious? I'm using numpy - is there maybe something in scipy I should be looking at? Also, in evaluating samples, I'm likely to want to calculate combinatorial functions, such as the list of all pairs of items from a sample (imagine looking at how many pairs add up to 15 in a cribbage hand). Clearly, I can write a normal Python function which does this for one row, and use apply_along_axis - but that's *slow*. I'm looking for a function that, given an N*M array and a sample size S, gives a C(N,S)*S*M array of all the combinations, which runs at array-processing speeds (preferably without having to code it in C myself!!) Is there anywhere with this type of function available? This type of combinatorial simulation seems to me to be a fairly good fit for numpy's capabilities, and yet I can't seem to find things that seem relevant. Is it simly not something that people use numpy for? Or am I looking in the wrong places in the documentation? Thanks for any help, Paul. From robert.kern at gmail.com Thu Sep 18 18:00:12 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 18 Sep 2008 17:00:12 -0500 Subject: [Numpy-discussion] Generating random samples without repeats In-Reply-To: References: Message-ID: <3d375d730809181500p225e42b0u6bf92645141b35d5@mail.gmail.com> On Thu, Sep 18, 2008 at 16:55, Paul Moore wrote: > I want to generate a series of random samples, to do simulations based > on them. Essentially, I want to be able to produce a SAMPLESIZE * N > matrix, where each row of N values consists of either > > 1. Integers between 1 and M (simulating M rolls of an N-sided die), or > 2. A sample of N numbers between 1 and M without repeats (simulating > deals of N cards from an M-card deck). > > Example (1) is easy, numpy.random.random_integers(1, M, (SAMPLESIZE, N)) > > But I can't find an obvious equivalent for (2). Am I missing something > glaringly obvious? I'm using numpy - is there maybe something in scipy I > should be looking at? numpy.array([(numpy.random.permutation(M) + 1)[:N] for i in range(SAMPLESIZE)]) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From f.yw at hotmail.com Thu Sep 18 18:00:58 2008 From: f.yw at hotmail.com (frank wang) Date: Thu, 18 Sep 2008 16:00:58 -0600 Subject: [Numpy-discussion] A bug in loadtxt and how to convert a string array (hex data) to decimal? In-Reply-To: <48D2CAA3.7080605@gmail.com> References: <48D2CAA3.7080605@gmail.com> Message-ID: Hi, Ryan, Thank you very much. It solves my problem. I have struggled with this for long time. Frank> Date: Thu, 18 Sep 2008 16:39:47 -0500> From: rmay31 at gmail.com> To: numpy-discussion at scipy.org> Subject: Re: [Numpy-discussion] A bug in loadtxt and how to convert a string array (hex data) to decimal?> > frank wang wrote:> > > > > > > > Hi, All,> > > > > > > > > > > > I have found a bug in the loadtxt function. Here is the example. The > > file name is test.txt and contains:> > > > > > "Thist is test"> > > > > > 3FF 3fE> > > > > > 3Ef 3e8> > > > > > 3Df 3d9> > > > > > 3cF 3c7> > > > > > > > > > > > In the Python 2.5.2, I type:> > > > > > > > > > > > test=loadtxt('test.txt',comments='"',dtype='string',converters={0:lambda > > s:int(s,16)})> > > > > > > > > > > > test will contain> > > > > > > > > > > > array([['102', '3fE'],> > > > > > ['100', '3e8'],> > > > > > ['991', '3d9'],> > > > > > ['975', '3c7']],> > > > > > dtype='|S3')> > > > It's because of how numpy handles strings arrays (which I admit I don't > understand very well.) Basically, it's converting the numbers properly, > but truncating them to 3 characters. Try this, which just forces it to > expand to strings 4 characters wide:> test=loadtxt('test.txt',comments='"',dtype='|S4',converters={0:lambda> s:int(s,16)})> > HTH,> > Ryan> > -- > Ryan May> Graduate Research Assistant> School of Meteorology> University of Oklahoma> _______________________________________________> Numpy-discussion mailing list> Numpy-discussion at scipy.org> http://projects.scipy.org/mailman/listinfo/numpy-discussion _________________________________________________________________ See how Windows connects the people, information, and fun that are part of your life. http://clk.atdmt.com/MRT/go/msnnkwxp1020093175mrt/direct/01/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.abshoff at googlemail.com Thu Sep 18 19:20:04 2008 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Thu, 18 Sep 2008 16:20:04 -0700 Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: <3d375d730809181437m24cf8074yba83b8ae9d8b8afa@mail.gmail.com> References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <70cd1d75-3a32-4140-8a72-d2cdd650ef62@m3g2000hsc.googlegroups.com> <48CFCD08.10109@ar.media.kyoto-u.ac.jp> <48D05F4B.30508@ar.media.kyoto-u.ac.jp> <48D0CAF5.4070109@ar.media.kyoto-u.ac.jp> <5b8d13220809172229t53f0a590p22fee11b7411882d@mail.gmail.com> <9106f5a1-0288-4801-bd2e-089038def377@z72g2000hsb.googlegroups.com> <3d375d730809181437m24cf8074yba83b8ae9d8b8afa@mail.gmail.com> Message-ID: <48D2E224.3070908@gmail.com> Robert Kern wrote: > On Thu, Sep 18, 2008 at 06:26, Francis wrote: >> Thank you for your effort. I guess garnumpy reflects the idea in this >> Pylab discussion: http://www.scipy.org/PyLab >> >> Again I get errors in libblas/lapack related to gfortran (local >> variable problems). I replaced the libblas.a and the liblaplack.a by >> the ones of sage. And started make install again. It seems to work >> until it tries to configure/install umfpack. >> >> /users/francisd/garnumpy/lib/libblas.a(xerbla.o): In function >> `xerbla_': >> xerbla.f:(.text+0x1b): undefined reference to `_g95_get_ioparm' > > Yes, SAGE uses g95, not gfortran. > It depends :) Per default on OSX and Linux x86 and x86-64 Sage provides a binary g95. You can use the environment variable SAGE_FORTRAN to set it to gfortran. Additionally there is some hacked in code in various places in setup.py of Scipy that autodetects which Fortran libraries (g95 or gfortran) ATLAS was build with and adds those to the LDFLAGS dynamically. That code it butt ugly (and it is Perl, too :)), but it does the job. Obviously it will not help if mix and match Fortran compilers. Cheers, Michael From gael.varoquaux at normalesup.org Thu Sep 18 20:07:32 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 19 Sep 2008 02:07:32 +0200 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <3d375d730809181254r270fa98fve07bfe5b5051d410@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <48D23506.7080109@ntc.zcu.cz> <85b5c3130809180448o4dd12f94n5b6ebc94e0c1a8e3@mail.gmail.com> <48D261EA.7020404@gmail.com> <85b5c3130809180803r5b12897an759c82d40d1def9d@mail.gmail.com> <3d375d730809181254r270fa98fve07bfe5b5051d410@mail.gmail.com> Message-ID: <20080919000732.GA5125@phare.normalesup.org> On Thu, Sep 18, 2008 at 02:54:13PM -0500, Robert Kern wrote: > So here's what going on: I'm being clever (and possibly too clever). Oh no. Robert K. is too clever. We knew that, right ;). Ga?l From david at ar.media.kyoto-u.ac.jp Thu Sep 18 21:22:11 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 19 Sep 2008 10:22:11 +0900 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D23B35.8040902@dcs.gla.ac.uk> References: <48D23B35.8040902@dcs.gla.ac.uk> Message-ID: <48D2FEC3.3040400@ar.media.kyoto-u.ac.jp> Peter Saffrey wrote: > > Is this the correct behavior for median with nan? That's the expected behavior, at least :) (this is also the expected behavior of most math packages I know, including matlab and R, so this should not be too surprising if you have used those). > Is there a fix for > this or am I going to have to settle with using lists? You can use nanmean (from scipy.stats): >>> stats.nanmedian(np.array([1, np.nan, 3, 9])) 3 cheers, David From david at ar.media.kyoto-u.ac.jp Thu Sep 18 21:25:17 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 19 Sep 2008 10:25:17 +0900 Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: <9106f5a1-0288-4801-bd2e-089038def377@z72g2000hsb.googlegroups.com> References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <07bbbcff-365f-4051-924b-dae8d353b0ed@8g2000hse.googlegroups.com> <48CF96FD.9090805@ar.media.kyoto-u.ac.jp> <70cd1d75-3a32-4140-8a72-d2cdd650ef62@m3g2000hsc.googlegroups.com> <48CFCD08.10109@ar.media.kyoto-u.ac.jp> <48D05F4B.30508@ar.media.kyoto-u.ac.jp> <48D0CAF5.4070109@ar.media.kyoto-u.ac.jp> <5b8d13220809172229t53f0a590p22fee11b7411882d@mail.gmail.com> <9106f5a1-0288-4801-bd2e-089038def377@z72g2000hsb.googlegroups.com> Message-ID: <48D2FF7D.8090709@ar.media.kyoto-u.ac.jp> Francis wrote: > Thank you for your effort. I guess garnumpy reflects the idea in this > Pylab discussion: http://www.scipy.org/PyLab > > Again I get errors in libblas/lapack related to gfortran (local > variable problems). I replaced the libblas.a and the liblaplack.a by > the ones of sage. And started make install again. It seems to work > until it tries to configure/install umfpack. If you install blas/lapack from sage, it kind of defeats the whole purpose of garnumpy. The goal is to have a unified set of options to build. It is likely that sage uses different options than the ones from garnumpy. If you use garnumpy, you should use it for everything: it is an all for nothing, intended to replace broken distributions/workstations where you don't have admin rights. When I updated a bit garnumpy yesterday, I tested it on CENTOS 5, which is the distribution you are using, right ? > > xerbla.f:(.text+0xd7): undefined reference to `_g95_stop_blank' > collect2: ld returned 1 exit status > make[4]: *** [umfpack_di_demo] Error 1 > Yes, that's expected: you are using blas from sage, this cannot work. > Perhaps it possible to skip the installation of the umfpack since I > probably won't need it or it requires the other libblas.a. > Just wildly guessing here. garnumpy by default does not install umfpack (scipy only requires blas/lapack, and by default, garnumpy only install what is needed; you can configure it to build umfpack, atlas, whatever, but by default, only the minimal stuff), so I am not sure why you have umfpack errors. cheers, David From david at ar.media.kyoto-u.ac.jp Thu Sep 18 21:32:05 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 19 Sep 2008 10:32:05 +0900 Subject: [Numpy-discussion] building numpy locally but get error: undefined symbol: zgesdd_ In-Reply-To: <48D2FF7D.8090709@ar.media.kyoto-u.ac.jp> References: <2c758b440809152358r47186702i290bffce3a1594d9@mail.gmail.com> <07bbbcff-365f-4051-924b-dae8d353b0ed@8g2000hse.googlegroups.com> <48CF96FD.9090805@ar.media.kyoto-u.ac.jp> <70cd1d75-3a32-4140-8a72-d2cdd650ef62@m3g2000hsc.googlegroups.com> <48CFCD08.10109@ar.media.kyoto-u.ac.jp> <48D05F4B.30508@ar.media.kyoto-u.ac.jp> <48D0CAF5.4070109@ar.media.kyoto-u.ac.jp> <5b8d13220809172229t53f0a590p22fee11b7411882d@mail.gmail.com> <9106f5a1-0288-4801-bd2e-089038def377@z72g2000hsb.googlegroups.com> <48D2FF7D.8090709@ar.media.kyoto-u.ac.jp> Message-ID: <48D30115.8010502@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > If you install blas/lapack from sage, it kind of defeats the whole > purpose of garnumpy. The goal is to have a unified set of options to > build. It is likely that sage uses different options than the ones from > garnumpy. If you use garnumpy, you should use it for everything: it is > an all for nothing ^^^^ should read all or nothing. David From pgmdevlist at gmail.com Thu Sep 18 22:21:34 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 18 Sep 2008 22:21:34 -0400 Subject: [Numpy-discussion] Suggestion for recarray.view Message-ID: <200809182221.35193.pgmdevlist@gmail.com> All, I'd like to submit the following suggestion for recarray.view, so that it could accept two keywords like standard ndarrays do. As a change in records.py can potentially affect a lot of people (probably more than a change in numpy.ma), I'm not confident enough to commit it. Consider that as an attempt of having it peer-reviewed before inclusion. Cheers P. #--------------- def view(self, dtype=None, type=None): if dtype is None: return ndarray.view(self, type) elif type is None: try: if issubclass(dtype, ndarray): return ndarray.view(self, dtype) except TypeError: pass dtype = sb.dtype(dtype) if dtype.fields is None: return self.__array__().view(dtype) return ndarray.view(self, dtype) else: return ndarray.view(self, dtype, type) #--------------- From peridot.faceted at gmail.com Thu Sep 18 22:54:49 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 18 Sep 2008 22:54:49 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D2FEC3.3040400@ar.media.kyoto-u.ac.jp> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D2FEC3.3040400@ar.media.kyoto-u.ac.jp> Message-ID: 2008/9/18 David Cournapeau : > Peter Saffrey wrote: >> >> Is this the correct behavior for median with nan? > > That's the expected behavior, at least :) (this is also the expected > behavior of most math packages I know, including matlab and R, so this > should not be too surprising if you have used those). I don't think I agree: In [4]: np.median([1,3,nan]) Out[4]: 3.0 In [5]: np.median([1,nan,3]) Out[5]: nan In [6]: np.median([nan,1,3]) Out[6]: 1.0 I think the expected behaviour would be for all of these to return nan. Anne From david at ar.media.kyoto-u.ac.jp Thu Sep 18 22:45:56 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 19 Sep 2008 11:45:56 +0900 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: References: <48D23B35.8040902@dcs.gla.ac.uk> <48D2FEC3.3040400@ar.media.kyoto-u.ac.jp> Message-ID: <48D31264.2040503@ar.media.kyoto-u.ac.jp> Anne Archibald wrote: > > I don't think I agree: > > In [4]: np.median([1,3,nan]) > Out[4]: 3.0 > > In [5]: np.median([1,nan,3]) > Out[5]: nan > > In [6]: np.median([nan,1,3]) > Out[6]: 1.0 > I was referring to the fact that if you have nan in your array, you should use nanmean if you want to ignore them correctly. Now, the different behavior depending on the order of items in the arrays is indeed buggy, I thought this was fixed. cheers, David From oliphant at enthought.com Thu Sep 18 23:13:46 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 18 Sep 2008 22:13:46 -0500 Subject: [Numpy-discussion] Suggestion for recarray.view In-Reply-To: <200809182221.35193.pgmdevlist@gmail.com> References: <200809182221.35193.pgmdevlist@gmail.com> Message-ID: <48D318EA.70003@enthought.com> Pierre GM wrote: > All, > I'd like to submit the following suggestion for recarray.view, so that it > could accept two keywords like standard ndarrays do. > As a change in records.py can potentially affect a lot of people (probably > more than a change in numpy.ma), I'm not confident enough to commit it. > Consider that as an attempt of having it peer-reviewed before inclusion. > Cheers > P. > #--------------- > def view(self, dtype=None, type=None): > if dtype is None: > return ndarray.view(self, type) > elif type is None: > try: > if issubclass(dtype, ndarray): > return ndarray.view(self, dtype) > except TypeError: > pass > dtype = sb.dtype(dtype) > if dtype.fields is None: > return self.__array__().view(dtype) > return ndarray.view(self, dtype) > else: > return ndarray.view(self, dtype, type) > #--------------- > This looks pretty good to me. +1 for adding it. -Travis From johann.cohentanugi at gmail.com Thu Sep 18 23:34:18 2008 From: johann.cohentanugi at gmail.com (JohannCT) Date: Thu, 18 Sep 2008 20:34:18 -0700 (PDT) Subject: [Numpy-discussion] PyUnicodeUCS2 issue with numpy revision 5833 Message-ID: <19564937.post@talk.nabble.com> hello, I just updated my svn repository, but I am now unable anymore to import numpy : In [1]: import numpy as np --------------------------------------------------------------------------- ImportError Traceback (most recent call last) /home/cohen/ in () /usr/lib/python2.5/site-packages/numpy/__init__.py in () 123 return loader(*packages, **options) 124 --> 125 import add_newdocs 126 __all__ = ['add_newdocs'] 127 /usr/lib/python2.5/site-packages/numpy/add_newdocs.py in () 7 # core/fromnumeric.py, core/defmatrix.py up-to-date. 8 ----> 9 from lib import add_newdoc 10 11 add_newdoc('numpy.core', 'dtype', /usr/lib/python2.5/site-packages/numpy/lib/__init__.py in () 2 from numpy.version import version as __version__ 3 ----> 4 from type_check import * 5 from index_tricks import * 6 from function_base import * /usr/lib/python2.5/site-packages/numpy/lib/type_check.py in () 6 'common_type'] 7 ----> 8 import numpy.core.numeric as _nx 9 from numpy.core.numeric import asarray, asanyarray, array, isnan, \ 10 obj2sctype, zeros /usr/lib/python2.5/site-packages/numpy/core/__init__.py in () 3 from numpy.version import version as __version__ 4 ----> 5 import multiarray 6 import umath 7 import _internal # for freeze programs ImportError: /usr/lib/python2.5/site-packages/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_FromUnicode I did not change anything to my python setup, and I did remove the build directory before recompiling. Anyone has an idea? thanks in advance, Johann -- View this message in context: http://www.nabble.com/PyUnicodeUCS2-issue-with-numpy-revision-5833-tp19564937p19564937.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From johann.cohentanugi at gmail.com Thu Sep 18 23:52:52 2008 From: johann.cohentanugi at gmail.com (JohannCT) Date: Thu, 18 Sep 2008 20:52:52 -0700 (PDT) Subject: [Numpy-discussion] PyUnicodeUCS2 issue with numpy revision 5833 In-Reply-To: <19564937.post@talk.nabble.com> References: <19564937.post@talk.nabble.com> Message-ID: <19565823.post@talk.nabble.com> sorry my bad.... wrong python executable in ly config. Johann JohannCT wrote: > > hello, I just updated my svn repository, but I am now unable anymore to > import numpy : > In [1]: import numpy as np > --------------------------------------------------------------------------- > ImportError Traceback (most recent call > last) > > /home/cohen/ in () > > /usr/lib/python2.5/site-packages/numpy/__init__.py in () > 123 return loader(*packages, **options) > 124 > --> 125 import add_newdocs > 126 __all__ = ['add_newdocs'] > 127 > > /usr/lib/python2.5/site-packages/numpy/add_newdocs.py in () > 7 # core/fromnumeric.py, core/defmatrix.py up-to-date. > 8 > ----> 9 from lib import add_newdoc > 10 > 11 add_newdoc('numpy.core', 'dtype', > > /usr/lib/python2.5/site-packages/numpy/lib/__init__.py in () > 2 from numpy.version import version as __version__ > 3 > ----> 4 from type_check import * > 5 from index_tricks import * > 6 from function_base import * > > /usr/lib/python2.5/site-packages/numpy/lib/type_check.py in () > 6 'common_type'] > 7 > ----> 8 import numpy.core.numeric as _nx > 9 from numpy.core.numeric import asarray, asanyarray, array, isnan, > \ > 10 obj2sctype, zeros > > /usr/lib/python2.5/site-packages/numpy/core/__init__.py in () > 3 from numpy.version import version as __version__ > 4 > ----> 5 import multiarray > 6 import umath > 7 import _internal # for freeze programs > > ImportError: /usr/lib/python2.5/site-packages/numpy/core/multiarray.so: > undefined symbol: PyUnicodeUCS2_FromUnicode > > > I did not change anything to my python setup, and I did remove the build > directory before recompiling. > Anyone has an idea? > thanks in advance, > Johann > -- View this message in context: http://www.nabble.com/PyUnicodeUCS2-issue-with-numpy-revision-5833-tp19564937p19565823.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From peridot.faceted at gmail.com Fri Sep 19 00:18:15 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 19 Sep 2008 00:18:15 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D31264.2040503@ar.media.kyoto-u.ac.jp> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D2FEC3.3040400@ar.media.kyoto-u.ac.jp> <48D31264.2040503@ar.media.kyoto-u.ac.jp> Message-ID: 2008/9/18 David Cournapeau : > Anne Archibald wrote: >> >> I don't think I agree: >> >> In [4]: np.median([1,3,nan]) >> Out[4]: 3.0 >> >> In [5]: np.median([1,nan,3]) >> Out[5]: nan >> >> In [6]: np.median([nan,1,3]) >> Out[6]: 1.0 >> > > I was referring to the fact that if you have nan in your array, you > should use nanmean if you want to ignore them correctly. Now, the > different behavior depending on the order of items in the arrays is > indeed buggy, I thought this was fixed. That was in amax/amin. Pretty much every other function that does comparisons needs to be fixed to work with nans. In some cases it's not even clear how: where should a sort put the nans in an array? I suppose some enterprising soul should write up a fileful of tests making sure that all numpy's functions do something sane with arrays containing nans... Anne From david at ar.media.kyoto-u.ac.jp Fri Sep 19 00:16:20 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 19 Sep 2008 13:16:20 +0900 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: References: <48D23B35.8040902@dcs.gla.ac.uk> <48D2FEC3.3040400@ar.media.kyoto-u.ac.jp> <48D31264.2040503@ar.media.kyoto-u.ac.jp> Message-ID: <48D32794.6030804@ar.media.kyoto-u.ac.jp> Anne Archibald wrote: > > That was in amax/amin. Pretty much every other function that does > comparisons needs to be fixed to work with nans. In some cases it's > not even clear how: where should a sort put the nans in an array? The problem is more on how the functions use sort than sort itself in the case of median. There can't be a 'good' way to put nan in soft, for example, since nans cannot be ordered. I don't know about the best strategy: either we fix every function using comparison, handling nan as a special case as you mentioned, or there may be a more clever thing to do to avoid special casing everywhere. I don't have a clear idea of how many functions rely on ordering in numpy. cheers, David From peridot.faceted at gmail.com Fri Sep 19 02:56:12 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 19 Sep 2008 02:56:12 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D32794.6030804@ar.media.kyoto-u.ac.jp> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D2FEC3.3040400@ar.media.kyoto-u.ac.jp> <48D31264.2040503@ar.media.kyoto-u.ac.jp> <48D32794.6030804@ar.media.kyoto-u.ac.jp> Message-ID: 2008/9/19 David Cournapeau : > Anne Archibald wrote: >> >> That was in amax/amin. Pretty much every other function that does >> comparisons needs to be fixed to work with nans. In some cases it's >> not even clear how: where should a sort put the nans in an array? > > The problem is more on how the functions use sort than sort itself in > the case of median. There can't be a 'good' way to put nan in soft, for > example, since nans cannot be ordered. Well, for example, you might ask that all the non-nan elements be in order, even if you don't specify where the nan goes. > I don't know about the best strategy: either we fix every function using > comparison, handling nan as a special case as you mentioned, or there > may be a more clever thing to do to avoid special casing everywhere. I > don't have a clear idea of how many functions rely on ordering in numpy. You can always just set numpy to raise an exception whenever it comes across a nan. In fact, apart from the difficulty of correctly frobbing numpy's floating-point handling, how reasonable is it for (say) median to just run as it is now, but if an exception is thrown, fall back to a nan-aware version? Anne From david at ar.media.kyoto-u.ac.jp Fri Sep 19 03:11:05 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 19 Sep 2008 16:11:05 +0900 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: References: <48D23B35.8040902@dcs.gla.ac.uk> <48D2FEC3.3040400@ar.media.kyoto-u.ac.jp> <48D31264.2040503@ar.media.kyoto-u.ac.jp> <48D32794.6030804@ar.media.kyoto-u.ac.jp> Message-ID: <48D35089.30202@ar.media.kyoto-u.ac.jp> Anne Archibald wrote: > > Well, for example, you might ask that all the non-nan elements be in > order, even if you don't specify where the nan goes. Ah, there are two problems, then: - sort - how median use sort. For sort, I don't know how sort speed would be influenced by treating nan. In a way, calling sort with nan inside is a user error (if you take the POV nan are not comparable), but nan are used for all kind of purpose, hence maybe having a nansort would be nice. OTOH (I took a look at this when I fixed nanmean and co a while ago in scipy), matlab and R treat sort differently than mean and co. I am puzzled by this: - R sort arrays with nan as you want by default (nan can be ignored, put in front or at the end of the array). - R max does not ignore nan by default. - R median does not ignore median by default. I don't know how to set a consistency here. I don't think we are consistent by having max/amax/etc... ignoring nan but sort not ignoring it. OTOH, R is not consistent either. > > You can always just set numpy to raise an exception whenever it comes > across a nan. In fact, apart from the difficulty of correctly frobbing > numpy's floating-point handling, how reasonable is it for (say) median > to just run as it is now, but if an exception is thrown, fall back to > a nan-aware version? It would be different from the current nan vs usual function behavior for median/mean/etc...: why should sort handle nan by default, but not the other functions ? For mean/std/variance/median, if having nan is an error, you see it in the result (once we fix our median), but not with sort. Hm, I am always puzzled when I think about nan handling :) It always seem there is not good answer. David From pgmdevlist at gmail.com Fri Sep 19 03:29:14 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 19 Sep 2008 03:29:14 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D35089.30202@ar.media.kyoto-u.ac.jp> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> Message-ID: <200809190329.14131.pgmdevlist@gmail.com> On Friday 19 September 2008 03:11:05 David Cournapeau wrote: > Hm, I am always puzzled when I think about nan handling :) It always > seem there is not good answer. Which is why we have masked arrays, of course ;) From peridot.faceted at gmail.com Fri Sep 19 04:10:24 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 19 Sep 2008 04:10:24 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <200809190329.14131.pgmdevlist@gmail.com> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> Message-ID: 2008/9/19 Pierre GM : > On Friday 19 September 2008 03:11:05 David Cournapeau wrote: > >> Hm, I am always puzzled when I think about nan handling :) It always >> seem there is not good answer. > > Which is why we have masked arrays, of course ;) I think the numpy attitude to nans should be that they are unexpected bogus values that signify that something went wrong with the calculation somewhere. They can be left in place for most operations, but any operation that depends on the value should (ideally) return nan, or failing that, raise an exception. (If users want exceptions all the time, that's what seterr is for.) If people want to flag bad data, let's tell them to use masked arrays. So by this rule amax/maximum/mean/median should all return nan when there's a nan in their input; I don't think it's reasonable for sort to return an array full of nans, so I think its default behaviour should be to raise an exception if there's a nan. It's valuable (for example in median) to be able to sort them all to the end, but I don't think this should be the default. If people want nanmin, I would be tempted to tell them to use masked arrays (is there a convenience function that makes a masked array with a mask everywhere the data is nan?). I am assuming that appropriate masked sort/amax/maximum/mean/median exist already. They're definitely needed, so how much effort is it worth putting in to duplicate that functionality with nans instead of masked elements? Anne From stefan at sun.ac.za Fri Sep 19 04:13:39 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 19 Sep 2008 10:13:39 +0200 Subject: [Numpy-discussion] Suggestion for recarray.view In-Reply-To: <48D318EA.70003@enthought.com> References: <200809182221.35193.pgmdevlist@gmail.com> <48D318EA.70003@enthought.com> Message-ID: <9457e7c80809190113x557890aehf15bdadf7f9762a2@mail.gmail.com> 2008/9/19 Travis E. Oliphant : >> #--------------- >> def view(self, dtype=None, type=None): >> if dtype is None: >> return ndarray.view(self, type) >> elif type is None: >> try: >> if issubclass(dtype, ndarray): >> return ndarray.view(self, dtype) >> except TypeError: >> pass >> dtype = sb.dtype(dtype) >> if dtype.fields is None: >> return self.__array__().view(dtype) >> return ndarray.view(self, dtype) >> else: >> return ndarray.view(self, dtype, type) >> #--------------- >> > This looks pretty good to me. > > +1 for adding it. +1 and another +1 to your karma for requesting peer review. Let me know if you need me to whip up a couple of tests for verifying the different usage cases. Cheers St?fan From cimrman3 at ntc.zcu.cz Fri Sep 19 04:33:33 2008 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Fri, 19 Sep 2008 10:33:33 +0200 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <3d375d730809181237x43ce1610r6dccb081d6698f5d@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <48D23506.7080109@ntc.zcu.cz> <3d375d730809181237x43ce1610r6dccb081d6698f5d@mail.gmail.com> Message-ID: <48D363DD.6040609@ntc.zcu.cz> Robert Kern wrote: > On Thu, Sep 18, 2008 at 06:01, Robert Cimrman wrote: >> Hi Robert, >> >> Robert Kern wrote: >>> On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg wrote: >>>> That would make me an extremely happy user, I've been looking for this for >>>> years! >>>> I can't imagine I'm the only one who profiles some hundred lines of code and >>>> ends up with 90% of total time in the dot-function >>> For the time being, you can grab it here: >>> >>> http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/ >>> >>> It requires Cython and a C compiler to build. I'm still debating >>> myself about the desired workflow for using it, but for now, it only >>> profiles functions which you have registered with it. I have made the >>> profiler work as a decorator to make this easy. E.g., >> many thanks for this! I have wanted to try out the profiler but failed >> to build it (changeset 6 0de294aa75bf): >> >> $ python setup.py install --root=/home/share/software/ >> running install >> running build >> running build_py >> creating build >> creating build/lib.linux-i686-2.4 >> copying line_profiler.py -> build/lib.linux-i686-2.4 >> running build_ext >> cythoning _line_profiler.pyx to _line_profiler.c >> building '_line_profiler' extension >> creating build/temp.linux-i686-2.4 >> i486-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC >> -I/usr/include/python2.4 -c -I/usr/include/python2.4 -c _line_profiler.c >> -o build/temp.linux-i686-2.4/_line_profiler.o >> _line_profiler.c:1614: error: 'T_LONGLONG' undeclared here (not in a >> function) >> error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1 >> >> I have cython-0.9.8.1 and GCC 4.1.2, 32-bit machine. > > It uses the #define'd macro PY_LONG_LONG. Go through your Python > headers to see what this gets expanded to. > I have Python 2.4.4 in "pyconfig.h" #define HAVE_LONG_LONG 1 in "pyport.h": #ifdef HAVE_LONG_LONG #ifndef PY_LONG_LONG #define PY_LONG_LONG long long #endif #endif /* HAVE_LONG_LONG */ so it seems compatible with 'ctypedef long long PY_LONG_LONG' r. From stefan at sun.ac.za Fri Sep 19 04:36:43 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 19 Sep 2008 10:36:43 +0200 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> Message-ID: <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> 2008/9/19 Anne Archibald : > I think the numpy attitude to nans should be that they are unexpected > bogus values that signify that something went wrong with the > calculation somewhere. They can be left in place for most operations, > but any operation that depends on the value should (ideally) return > nan, or failing that, raise an exception. I agree completely. > I am assuming that appropriate masked sort/amax/maximum/mean/median > exist already. They're definitely needed, so how much effort is it > worth putting in to duplicate that functionality with nans instead of > masked elements? Unfortunately, this needs to happen at the C level. Is anyone reading this willing to spend some time taking care of the issue? It's an important one. St?fan From cimrman3 at ntc.zcu.cz Fri Sep 19 04:37:45 2008 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Fri, 19 Sep 2008 10:37:45 +0200 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <85b5c3130809180803r5b12897an759c82d40d1def9d@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <48D23506.7080109@ntc.zcu.cz> <85b5c3130809180448o4dd12f94n5b6ebc94e0c1a8e3@mail.gmail.com> <48D261EA.7020404@gmail.com> <85b5c3130809180803r5b12897an759c82d40d1def9d@mail.gmail.com> Message-ID: <48D364D9.6000605@ntc.zcu.cz> Ondrej Certik wrote: > On Thu, Sep 18, 2008 at 4:12 PM, Ryan May wrote: >> Ondrej Certik wrote: >>> On Thu, Sep 18, 2008 at 1:01 PM, Robert Cimrman wrote: >>>>> It requires Cython and a C compiler to build. I'm still debating >>>>> myself about the desired workflow for using it, but for now, it only >>>>> profiles functions which you have registered with it. I have made the >>>>> profiler work as a decorator to make this easy. E.g., >>>> many thanks for this! I have wanted to try out the profiler but failed >>>> to build it (changeset 6 0de294aa75bf): >>>> >>>> $ python setup.py install --root=/home/share/software/ >>>> running install >>>> running build >>>> running build_py >>>> creating build >>>> creating build/lib.linux-i686-2.4 >>>> copying line_profiler.py -> build/lib.linux-i686-2.4 >>>> running build_ext >>>> cythoning _line_profiler.pyx to _line_profiler.c >>>> building '_line_profiler' extension >>>> creating build/temp.linux-i686-2.4 >>>> i486-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC >>>> -I/usr/include/python2.4 -c -I/usr/include/python2.4 -c _line_profiler.c >>>> -o build/temp.linux-i686-2.4/_line_profiler.o >>>> _line_profiler.c:1614: error: 'T_LONGLONG' undeclared here (not in a >>>> function) >>>> error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1 >>>> >>>> I have cython-0.9.8.1 and GCC 4.1.2, 32-bit machine. >>> I am telling you all the time Robert to use Debian that it just works >>> and you say, no no, gentoo is the best. :) >> And what's wrong with that? :) Once you get over the learning curve, >> Gentoo works just fine. Must be Robert K.'s fault. :) > > Well, I think if Robert C. hasn't yet get over the learning curve > after so many years of hard work, maybe the learning curve is too > steep. :) This is most probably not related to Gentoo at all and certainly not related to me knowing Gentoo or not :) (and no, learning Gentoo is not that hard.) r. From pgmdevlist at gmail.com Fri Sep 19 04:32:23 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 19 Sep 2008 04:32:23 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: References: <48D23B35.8040902@dcs.gla.ac.uk> <200809190329.14131.pgmdevlist@gmail.com> Message-ID: <200809190432.23535.pgmdevlist@gmail.com> On Friday 19 September 2008 04:10:24 Anne Archibald wrote: > (is there a convenience > function that makes a masked array with a mask everywhere the data is > nan?). numpy.ma.fix_invalid, that masks your Nans and Infs and sets the underlying data to some filling value. That way, you don't carry NaNs/Infs along. > I am assuming that appropriate masked sort/amax/maximum/mean/median > exist already. They're definitely needed, so how much effort is it > worth putting in to duplicate that functionality with nans instead of > masked elements? My opinion indeed. The MaskedArray.sort method has an extra flag that lets you decide whether you want masked data at the beginning or the end of your array. That said, numpy.nanmin, numpy.nansum... don't come with the heavy machinery of numpy.ma, and are therefore faster. I'm really going to have to learn C. From pgmdevlist at gmail.com Fri Sep 19 04:34:44 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 19 Sep 2008 04:34:44 -0400 Subject: [Numpy-discussion] Suggestion for recarray.view In-Reply-To: <9457e7c80809190113x557890aehf15bdadf7f9762a2@mail.gmail.com> References: <200809182221.35193.pgmdevlist@gmail.com> <48D318EA.70003@enthought.com> <9457e7c80809190113x557890aehf15bdadf7f9762a2@mail.gmail.com> Message-ID: <200809190434.44365.pgmdevlist@gmail.com> On Friday 19 September 2008 04:13:39 St?fan van der Walt wrote: > +1 and another +1 to your karma for requesting peer review. Let me > know if you need me to whip up a couple of tests for verifying the > different usage cases. That'd be lovely. I'm a bit swamped with tricky issues in mrecords and dependents... From david at ar.media.kyoto-u.ac.jp Fri Sep 19 04:31:38 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 19 Sep 2008 17:31:38 +0900 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <200809190432.23535.pgmdevlist@gmail.com> References: <48D23B35.8040902@dcs.gla.ac.uk> <200809190329.14131.pgmdevlist@gmail.com> <200809190432.23535.pgmdevlist@gmail.com> Message-ID: <48D3636A.6070908@ar.media.kyoto-u.ac.jp> Pierre GM wrote: > That said, numpy.nanmin, numpy.nansum... don't come with the heavy machinery > of numpy.ma, and are therefore faster. > I'm really going to have to learn C. > FWIW, nanmean/nanmean/etc... are written in python, cheers, David From david at ar.media.kyoto-u.ac.jp Fri Sep 19 04:35:01 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 19 Sep 2008 17:35:01 +0900 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> Message-ID: <48D36435.5050709@ar.media.kyoto-u.ac.jp> St?fan van der Walt wrote: > > I agree completely. Me too, but I am extremely biased toward nan is always bogus by my own usage of numpy/scipy (I never use NaN as missing value, and nan is always caused by divide by 0 and co). I like that sort raise an exception by default with NaN: it breaks the API, OTOH, I can't see a good use of sort with NaN since sort does not sort values in that case: we would break the API of a broken function. > > Unfortunately, this needs to happen at the C level. Why ? cheers, David From stefan at sun.ac.za Fri Sep 19 04:52:01 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 19 Sep 2008 10:52:01 +0200 Subject: [Numpy-discussion] A bug in loadtxt and how to convert a string array (hex data) to decimal? In-Reply-To: <48D2CAA3.7080605@gmail.com> References: <48D2CAA3.7080605@gmail.com> Message-ID: <9457e7c80809190152p2e5fbab7r4b7644abcc0521b9@mail.gmail.com> 2008/9/18 Ryan May : > It's because of how numpy handles strings arrays (which I admit I don't > understand very well.) Basically, it's converting the numbers properly, > but truncating them to 3 characters. Try this, which just forces it to > expand to strings 4 characters wide: > test=loadtxt('test.txt',comments='"',dtype='|S4',converters={0:lambda > s:int(s,16)}) Here's what happens in the background: >>> data = [(1023, '3fE'), (1007, '3e8'), (991, '3d9'), (975, '3c7')] >>> np.array(data, np.dtype('string')) array([['102', '3fE'], ['100', '3e8'], ['991', '3d9'], ['975', '3c7']], dtype='|S3') Why? Because >>> np.dtype('string') dtype('|S0') So, it grabs the width from the first string it sees. A clean workaround then: test = np.loadtxt('/tmp/data.txt', comments='"', dtype='string', converters={0:lambda s: str(int(s,16))}) Cheers St?fan From pgmdevlist at gmail.com Fri Sep 19 04:54:40 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 19 Sep 2008 04:54:40 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D3636A.6070908@ar.media.kyoto-u.ac.jp> References: <48D23B35.8040902@dcs.gla.ac.uk> <200809190432.23535.pgmdevlist@gmail.com> <48D3636A.6070908@ar.media.kyoto-u.ac.jp> Message-ID: <200809190454.40685.pgmdevlist@gmail.com> On Friday 19 September 2008 04:31:38 David Cournapeau wrote: > Pierre GM wrote: > > That said, numpy.nanmin, numpy.nansum... don't come with the heavy > > machinery of numpy.ma, and are therefore faster. > > I'm really going to have to learn C. > > FWIW, nanmean/nanmean/etc... are written in python, I know. I was more dreading the time when MaskedArrays would have to be ported to C. In a way, that would probably simplify a few issues. OTOH, I don't really see it happening any time soon. From pf_moore at yahoo.co.uk Fri Sep 19 05:08:20 2008 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Fri, 19 Sep 2008 09:08:20 +0000 (UTC) Subject: [Numpy-discussion] Generating random samples without repeats References: <3d375d730809181500p225e42b0u6bf92645141b35d5@mail.gmail.com> Message-ID: Robert Kern gmail.com> writes: > On Thu, Sep 18, 2008 at 16:55, Paul Moore yahoo.co.uk> wrote: > > I want to generate a series of random samples, to do simulations based > > on them. Essentially, I want to be able to produce a SAMPLESIZE * N > > matrix, where each row of N values consists of either > > > > 1. Integers between 1 and M (simulating M rolls of an N-sided die), or > > 2. A sample of N numbers between 1 and M without repeats (simulating > > deals of N cards from an M-card deck). > > > > Example (1) is easy, numpy.random.random_integers(1, M, (SAMPLESIZE, N)) > > > > But I can't find an obvious equivalent for (2). Am I missing something > > glaringly obvious? I'm using numpy - is there maybe something in scipy I > > should be looking at? > > numpy.array([(numpy.random.permutation(M) + 1)[:N] > for i in range(SAMPLESIZE)]) > Thanks. And yet, this takes over 70s and peaks at around 400M memory use, whereas the equivalent for (1) numpy.random.random_integers(1,M,(SAMPLESIZE,N)) takes less than half a second, and negligible working memory (both end up allocating an array of the same size, but your suggestion consumes temporary working memory - I suspect, but can't prove, that the time taken comes from memory allocations rather than computation. As a one-off cost initialising my data, it's not a disaster, but I anticipate using idioms like this later in my calculations as well, where the costs could hurt more. If I'm going to need to write C code, are there any good examples of this? (I guess the source for numpy.random is a good place to start). Paul From stefan at sun.ac.za Fri Sep 19 05:10:15 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 19 Sep 2008 11:10:15 +0200 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D36435.5050709@ar.media.kyoto-u.ac.jp> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> Message-ID: <9457e7c80809190210t44025279y57e7ea916fe232d5@mail.gmail.com> 2008/9/19 David Cournapeau : > St?fan van der Walt wrote: >> >> I agree completely. > > Me too, but I am extremely biased toward nan is always bogus by my own > usage of numpy/scipy (I never use NaN as missing value, and nan is > always caused by divide by 0 and co). So am I. In all my use cases, NaNs indicate trouble. > Why ? Because we have x.max() silently ignoring NaNs, which causes a lot of head-scratching, swearing and failed experiments. Cheers St?fan From pgmdevlist at gmail.com Fri Sep 19 05:17:05 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 19 Sep 2008 05:17:05 -0400 Subject: [Numpy-discussion] Generating random samples without repeats In-Reply-To: References: <3d375d730809181500p225e42b0u6bf92645141b35d5@mail.gmail.com> Message-ID: <200809190517.06709.pgmdevlist@gmail.com> On Friday 19 September 2008 05:08:20 Paul Moore wrote: > Robert Kern gmail.com> writes: > > On Thu, Sep 18, 2008 at 16:55, Paul Moore yahoo.co.uk> wrote: > > > I want to generate a series of random samples, to do simulations based > > > on them. Essentially, I want to be able to produce a SAMPLESIZE * N > > > matrix, where each row of N values consists of either > > > 2. A sample of N numbers between 1 and M without repeats (simulating > > > deals of N cards from an M-card deck). Have you considered numpy.random.shuffle ? a = np.arange(1, M+1) result = np.random.shuffle(a)[:N] From robert.kern at gmail.com Fri Sep 19 05:26:42 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 19 Sep 2008 04:26:42 -0500 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <48D363DD.6040609@ntc.zcu.cz> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <48D23506.7080109@ntc.zcu.cz> <3d375d730809181237x43ce1610r6dccb081d6698f5d@mail.gmail.com> <48D363DD.6040609@ntc.zcu.cz> Message-ID: <3d375d730809190226g361d5855o1c1475da0379706f@mail.gmail.com> On Fri, Sep 19, 2008 at 03:33, Robert Cimrman wrote: > I have Python 2.4.4 > > in "pyconfig.h" > > #define HAVE_LONG_LONG 1 > > in "pyport.h": > > #ifdef HAVE_LONG_LONG > #ifndef PY_LONG_LONG > #define PY_LONG_LONG long long > #endif > #endif /* HAVE_LONG_LONG */ > > so it seems compatible with 'ctypedef long long PY_LONG_LONG' Ah, found it. T_LONGLONG is a #define from structmember.h which is used to describe the types of attributes. Apparently, this was not added until Python 2.5. That particular member didn't actually need to be long long, so I've fixed that. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Fri Sep 19 05:28:04 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 19 Sep 2008 04:28:04 -0500 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <3d375d730809171629i2e29ea69weed51abd0c2a533d@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <85b5c3130809171609g263ab76bj1a3afe493c352278@mail.gmail.com> <3d375d730809171629i2e29ea69weed51abd0c2a533d@mail.gmail.com> Message-ID: <3d375d730809190228v426f31fcnac6412f436300d51@mail.gmail.com> On Wed, Sep 17, 2008 at 18:29, Robert Kern wrote: > On Wed, Sep 17, 2008 at 18:09, Ondrej Certik wrote: >> This is what I am getting: >> >> $ ./kernprof.py -l pystone.py >> Wrote profile results to pystone.py.lprof >> $ ./view_line_prof.py pystone.py.lprof >> Timer unit: 1e-06 s >> >> $ >> >> So I think you meant: >> >> $ ./kernprof.py -l mystone.py >> 20628 >> Wrote profile results to mystone.py.lprof >> $ ./view_line_prof.py mystone.py.lprof >> Timer unit: 1e-06 s >> >> File: pystone.py >> Function: Proc0 at line 79 >> Total time: 13.0803 s >> [...] >> >> Now it works. > > No, I meant pystone.py. My script-finding code may have (incorrectly) > found a different, uninstrumented pystone.py file somewhere else, > though. Try with "./pystone.py". There was a bug in how I was constructing the munged namespaces. Fixed now. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ondrej at certik.cz Fri Sep 19 05:29:24 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 19 Sep 2008 11:29:24 +0200 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <48D364D9.6000605@ntc.zcu.cz> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <48D23506.7080109@ntc.zcu.cz> <85b5c3130809180448o4dd12f94n5b6ebc94e0c1a8e3@mail.gmail.com> <48D261EA.7020404@gmail.com> <85b5c3130809180803r5b12897an759c82d40d1def9d@mail.gmail.com> <48D364D9.6000605@ntc.zcu.cz> Message-ID: <85b5c3130809190229m2206ff16l22a16cc544f3b04f@mail.gmail.com> On Fri, Sep 19, 2008 at 10:37 AM, Robert Cimrman wrote: > Ondrej Certik wrote: >> On Thu, Sep 18, 2008 at 4:12 PM, Ryan May wrote: >>> Ondrej Certik wrote: >>>> On Thu, Sep 18, 2008 at 1:01 PM, Robert Cimrman wrote: >>>>>> It requires Cython and a C compiler to build. I'm still debating >>>>>> myself about the desired workflow for using it, but for now, it only >>>>>> profiles functions which you have registered with it. I have made the >>>>>> profiler work as a decorator to make this easy. E.g., >>>>> many thanks for this! I have wanted to try out the profiler but failed >>>>> to build it (changeset 6 0de294aa75bf): >>>>> >>>>> $ python setup.py install --root=/home/share/software/ >>>>> running install >>>>> running build >>>>> running build_py >>>>> creating build >>>>> creating build/lib.linux-i686-2.4 >>>>> copying line_profiler.py -> build/lib.linux-i686-2.4 >>>>> running build_ext >>>>> cythoning _line_profiler.pyx to _line_profiler.c >>>>> building '_line_profiler' extension >>>>> creating build/temp.linux-i686-2.4 >>>>> i486-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC >>>>> -I/usr/include/python2.4 -c -I/usr/include/python2.4 -c _line_profiler.c >>>>> -o build/temp.linux-i686-2.4/_line_profiler.o >>>>> _line_profiler.c:1614: error: 'T_LONGLONG' undeclared here (not in a >>>>> function) >>>>> error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1 >>>>> >>>>> I have cython-0.9.8.1 and GCC 4.1.2, 32-bit machine. >>>> I am telling you all the time Robert to use Debian that it just works >>>> and you say, no no, gentoo is the best. :) >>> And what's wrong with that? :) Once you get over the learning curve, >>> Gentoo works just fine. Must be Robert K.'s fault. :) >> >> Well, I think if Robert C. hasn't yet get over the learning curve >> after so many years of hard work, maybe the learning curve is too >> steep. :) > > This is most probably not related to Gentoo at all and certainly not > related to me knowing Gentoo or not :) (and no, learning Gentoo is not > that hard.) Let us know where the problem was. :) I am just using common sense, if something works on Debian and macosx and doesn't work on gentoo, I thought it was safe to say it was gentoo related, but I may well be wrong. :)) Ondrej From david at ar.media.kyoto-u.ac.jp Fri Sep 19 05:20:56 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 19 Sep 2008 18:20:56 +0900 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <9457e7c80809190210t44025279y57e7ea916fe232d5@mail.gmail.com> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <9457e7c80809190210t44025279y57e7ea916fe232d5@mail.gmail.com> Message-ID: <48D36EF8.9030901@ar.media.kyoto-u.ac.jp> St?fan van der Walt wrote: > > So am I. In all my use cases, NaNs indicate trouble. Yes, so I would like to have the opinion of people with other usage than ours. > > Because we have x.max() silently ignoring NaNs, which causes a lot of > head-scratching, swearing and failed experiments. But cannot this be fixed at the python level of the max function ? I think it is expected to have the low level C functions to ignore/be bogus if you have Nan. After all, if you use sort of the libc with nan, or sort in C++ for a vector of double, it will not work either. But on my numpy, it looks like nan breaks min/max, they are not ignored: np.min(np.array([0, np.nan, 1])) -> 1.0 # bogus np.min(np.array([0, np.nan, 2])) -> 2.0 # ok np.min(np.array([0, np.nan, -1])) -> -1.0 # ok np.max(np.array([0, np.nan, -1])) > -1.0 # bogus Which only makes sense when you guess how they are implemented in C... cheers, David From peridot.faceted at gmail.com Fri Sep 19 05:40:04 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 19 Sep 2008 05:40:04 -0400 Subject: [Numpy-discussion] Generating random samples without repeats In-Reply-To: References: <3d375d730809181500p225e42b0u6bf92645141b35d5@mail.gmail.com> Message-ID: 2008/9/19 Paul Moore : > Robert Kern gmail.com> writes: >> On Thu, Sep 18, 2008 at 16:55, Paul Moore yahoo.co.uk> wrote: >> > I want to generate a series of random samples, to do simulations based >> > on them. Essentially, I want to be able to produce a SAMPLESIZE * N >> > matrix, where each row of N values consists of either >> > >> > 1. Integers between 1 and M (simulating M rolls of an N-sided die), or >> > 2. A sample of N numbers between 1 and M without repeats (simulating >> > deals of N cards from an M-card deck). >> > >> > Example (1) is easy, numpy.random.random_integers(1, M, (SAMPLESIZE, N)) >> > >> > But I can't find an obvious equivalent for (2). Am I missing something >> > glaringly obvious? I'm using numpy - is there maybe something in scipy I >> > should be looking at? >> >> numpy.array([(numpy.random.permutation(M) + 1)[:N] >> for i in range(SAMPLESIZE)]) >> > > Thanks. > > And yet, this takes over 70s and peaks at around 400M memory use, whereas the > equivalent for (1) > > numpy.random.random_integers(1,M,(SAMPLESIZE,N)) > > takes less than half a second, and negligible working memory (both end up > allocating an array of the same size, but your suggestion consumes temporary > working memory - I suspect, but can't prove, that the time taken comes from > memory allocations rather than computation. > > As a one-off cost initialising my data, it's not a disaster, but I anticipate > using idioms like this later in my calculations as well, where the costs could > hurt more. > > If I'm going to need to write C code, are there any good examples of this? (I > guess the source for numpy.random is a good place to start). This was discussed on one of the mailing lists several months ago. It turns out that there is no simple way to efficiently choose without replacement in numpy/scipy. I posted a hack that does this somewhat efficiently (if SAMPLESIZE>M/2, choose the first SAMPLESIZE of a permutation; if SAMPLESIZE <48D2FEC3.3040400@ar.media.kyoto-u.ac.jp> Message-ID: David Cournapeau ar.media.kyoto-u.ac.jp> writes: > You can use nanmean (from scipy.stats): > I rejoiced when I saw this answer, because it looks like a function I can just drop in and it works. Unfortunately, nanmedian seems to be quite a bit slower than just using lists (ignoring nan values from my experiments) and a home-brew implementation of median. I was mostly using numpy for speed... I would like to try the masked array approach, but the Ubuntu packages for scipy and matplotlib depend on numpy. Does anybody know whether I can naively do "sudo python setup.py install" on a more modern numpy without disturbing scipy and matplotlib, or do I need to uninstall all three packages and install them manually from source? On my 64 bit machine, the Ubuntu numpy package is even more out of date: $ dpkg -l | grep numpy ii python-numpy 1:1.0.4-6ubuntu3 Does anybody know why this is? I might be willing to help bring the repository up to date, if anybody can give me pointers on how to do this. Peter From pgmdevlist at gmail.com Fri Sep 19 05:58:58 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 19 Sep 2008 05:58:58 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: References: <48D23B35.8040902@dcs.gla.ac.uk> <48D2FEC3.3040400@ar.media.kyoto-u.ac.jp> Message-ID: <200809190558.58925.pgmdevlist@gmail.com> On Friday 19 September 2008 05:51:55 Peter Saffrey wrote: > I would like to try the masked array approach, but the Ubuntu packages for > scipy and matplotlib depend on numpy. Does anybody know whether I can > naively do "sudo python setup.py install" on a more modern numpy without > disturbing scipy and matplotlib, or do I need to uninstall all three > packages and install them manually from source? I think there were some changes on the C side of numpy between 1.0 and 1.1, you may have to recompile scipy and matplotlib from sources. What versions are you using for those 2 packages ? From david at ar.media.kyoto-u.ac.jp Fri Sep 19 05:48:13 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 19 Sep 2008 18:48:13 +0900 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: References: <48D23B35.8040902@dcs.gla.ac.uk> <48D2FEC3.3040400@ar.media.kyoto-u.ac.jp> Message-ID: <48D3755D.8060905@ar.media.kyoto-u.ac.jp> Peter Saffrey wrote: > > I rejoiced when I saw this answer, because it looks like a function I can just > drop in and it works. Unfortunately, nanmedian seems to be quite a bit slower > than just using lists (ignoring nan values from my experiments) and a home-brew > implementation of median. I was mostly using numpy for speed... It may be that nanmedian is slow. But I would sincerly be surprised if it were slower than python list, except for some pathological cases, or maybe a bug in nanmedian. What do your data look like ? (size, number of nan, etc...) I quickly benchmarked on relatively small dataset (a few thousand samples with a few random nan), and nanmedian is "only" a few times slower than median. > > I would like to try the masked array approach, but the Ubuntu packages for scipy > and matplotlib depend on numpy. Does anybody know whether I can naively do "sudo > python setup.py install" on a more modern numpy without disturbing scipy and > matplotlib, or do I need to uninstall all three packages and install them > manually from source? My advice would be to never ever install a package from source into /usr. This will cause trouble. The way I do it is to install everything from sources into $HOME/local (of course, any directory you have regular write access to will do). > > On my 64 bit machine, the Ubuntu numpy package is even more out of date: > > $ dpkg -l | grep numpy > ii python-numpy 1:1.0.4-6ubuntu3 > > Does anybody know why this is? Yes, ubuntu updates every 6 months, the last time in last April. Numpy 1.1.0 (the first version after 1.0.4) was released in May. Also, Ubuntu updates from debian, general 4-5 months before ubuntu release data. So even if debian were to release a package the day we release a new package, Ubuntu will be one year late. I personally think that the solution would be to provide our own .deb up to date, but this is a lot of work. I think Ondrej did some work related to that; recent tools like opensuse build service and launchpad ppa makes it somewhat a bit easier, too (for the build part, at least; you still need to know how to build rpm/deb). cheers, David From pf_moore at yahoo.co.uk Fri Sep 19 06:17:51 2008 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Fri, 19 Sep 2008 10:17:51 +0000 (UTC) Subject: [Numpy-discussion] Generating random samples without repeats References: <3d375d730809181500p225e42b0u6bf92645141b35d5@mail.gmail.com> Message-ID: Anne Archibald gmail.com> writes: > This was discussed on one of the mailing lists several months ago. It > turns out that there is no simple way to efficiently choose without > replacement in numpy/scipy. That reassures me that I'm not missing something obvious! I'm pretty new with numpy (I've lurked here for a number of years, but never had a real-life need to use numpy until now). > I posted a hack that does this somewhat > efficiently (if SAMPLESIZE>M/2, choose the first SAMPLESIZE of a > permutation; if SAMPLESIZE duplicates) but it's not vectorized across many sample sets. Is your > problem large M or large N? what is SAMPLESIZE/M? It's actually large SAMPLESIZE. As an example, I'm simulating repeated deals of poker hands from a deck of cards: M=52, N=5, SAMPLESIZE=1000000. For now, Robert's approach will work, but it will start blowing up when I want 100 million samples - I don't have the memory to hold all the data (4 bytes for an int * N=5 * 100000000 = 2GB plus change). So I'll need to allocate (say) 1 million at a time in a loop and accumulate my results. That's when 70- second costs to allocate start to hurt. (After all, this is just the setup - I've got my actual calculations to do as well!!!) I'll stick with Robert's approach for now, and see if I can knock up something using Cython once I really need the speed. Paul From pzs at dcs.gla.ac.uk Fri Sep 19 07:21:12 2008 From: pzs at dcs.gla.ac.uk (Peter Saffrey) Date: Fri, 19 Sep 2008 11:21:12 +0000 (UTC) Subject: [Numpy-discussion] Medians that ignore values References: <48D23B35.8040902@dcs.gla.ac.uk> <48D2FEC3.3040400@ar.media.kyoto-u.ac.jp> <48D3755D.8060905@ar.media.kyoto-u.ac.jp> Message-ID: David Cournapeau ar.media.kyoto-u.ac.jp> writes: > It may be that nanmedian is slow. But I would sincerly be surprised if > it were slower than python list, except for some pathological cases, or > maybe a bug in nanmedian. What do your data look like ? (size, number of > nan, etc...) > I've posted my test code below, which gives me the results: $ ./arrayspeed3.py list build time: 0.01 list median time: 0.01 array nanmedian time: 0.36 I must have done something wrong to hobble nanmedian in this way... I'm quite new to numpy, so feel free to point out any obviously egregious errors. Peter === from numpy import array, nan, inf from pylab import rand from time import clock from scipy.stats.stats import nanmedian import pdb _pdb = pdb.Pdb() breakpoint = _pdb.set_trace def my_median(vallist): num_vals = len(vallist) vallist.sort() if num_vals % 2 == 1: # odd index = (num_vals - 1) / 2 return vallist[index] else: # even index = num_vals / 2 return (vallist[index] + vallist[index - 1]) / 2 numtests = 100 testsize = 100 pointlen = 3 t0 = clock() natests = rand(numtests,testsize,pointlen) # have to start with inf because list.remove(nan) doesn't remove nan natests[natests > 0.9] = inf tests = natests.tolist() natests[natests==inf] = nan for test in tests: for point in test: if inf in point: point.remove(inf) t1 = clock() print "list build time:", t1-t0 t0 = clock() allmedians = [] for test in tests: medians = [ my_median(x) for x in test ] allmedians.append(medians) t1 = clock() print "list median time:", t1-t0 t0 = clock() namedians = [] for natest in natests: thismed = nanmedian(natest, axis=1) namedians.append(thismed) t1 = clock() print "array nanmedian time:", t1-t0 From pzs at dcs.gla.ac.uk Fri Sep 19 07:23:28 2008 From: pzs at dcs.gla.ac.uk (Peter Saffrey) Date: Fri, 19 Sep 2008 11:23:28 +0000 (UTC) Subject: [Numpy-discussion] Medians that ignore values References: <48D23B35.8040902@dcs.gla.ac.uk> <48D2FEC3.3040400@ar.media.kyoto-u.ac.jp> <200809190558.58925.pgmdevlist@gmail.com> Message-ID: Pierre GM gmail.com> writes: > I think there were some changes on the C side of numpy between 1.0 and 1.1, > you may have to recompile scipy and matplotlib from sources. What versions > are you using for those 2 packages ? > $ dpkg -l | grep scipy ii python-scipy 0.6.0-8ubuntu1 scientific tools for Python $ dpkg -l | grep matplotlib ii python-matplotlib 0.91.2-0ubuntu1 Python based plotting system in a style simi ii python-matplotlib-data 0.91.2-0ubuntu1 Python based plotting system (data package) ii python-matplotlib-doc 0.91.2-0ubuntu1 Python based plotting system (documentation Peter From david at ar.media.kyoto-u.ac.jp Fri Sep 19 07:26:04 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 19 Sep 2008 20:26:04 +0900 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: References: <48D23B35.8040902@dcs.gla.ac.uk> <48D2FEC3.3040400@ar.media.kyoto-u.ac.jp> <200809190558.58925.pgmdevlist@gmail.com> Message-ID: <48D38C4C.6020703@ar.media.kyoto-u.ac.jp> Peter Saffrey wrote: > Pierre GM gmail.com> writes: > >> I think there were some changes on the C side of numpy between 1.0 and 1.1, >> you may have to recompile scipy and matplotlib from sources. What versions >> are you using for those 2 packages ? >> > > $ dpkg -l | grep scipy > ii python-scipy 0.6.0-8ubuntu1 > scientific tools for Python > > $ dpkg -l | grep matplotlib > ii python-matplotlib 0.91.2-0ubuntu1 > Python based plotting system in a style simi > ii python-matplotlib-data 0.91.2-0ubuntu1 > Python based plotting system (data package) > ii python-matplotlib-doc 0.91.2-0ubuntu1 > Python based plotting system (documentation If you build numpy from sources, please don't install it into /usr ! It will more than likely break everything which depends on numpy, as well as your debian installation (because you will overwrite packages handled by dpkg). You should really install in a local directory, outside /usr. You will have to install scipy and matplotlib in any case, too. cheers, David From david at ar.media.kyoto-u.ac.jp Fri Sep 19 07:28:35 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 19 Sep 2008 20:28:35 +0900 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: References: <48D23B35.8040902@dcs.gla.ac.uk> <48D2FEC3.3040400@ar.media.kyoto-u.ac.jp> <48D3755D.8060905@ar.media.kyoto-u.ac.jp> Message-ID: <48D38CE3.2030408@ar.media.kyoto-u.ac.jp> Peter Saffrey wrote: > > I've posted my test code below, which gives me the results: > > $ ./arrayspeed3.py > list build time: 0.01 > list median time: 0.01 > array nanmedian time: 0.36 > > I must have done something wrong to hobble nanmedian in this way... I'm quite > new to numpy, so feel free to point out any obviously egregious errors. Ok: it is "pathological", and can be done better :) First: > for natest in natests: > thismed = nanmedian(natest, axis=1) > namedians.append(thismed) ^^^ Here, you are doing nanmedian on a direction with 3 elements: this will be slow in numpy, because numpy involves some relatively heavy machinery to run on arrays. The machinery pays off for 'big' arrays, but for really small arrays like here, list can (and often are) be faster. Still, it is indeed really slow for your case; when I fixed nanmean and co, I did not know much about numpy, I just wanted them to give the right answer :) I think this can be made faster, specially for your case (where the axis along which the median is computed is really small). I opened a bug: http://scipy.org/scipy/scipy/ticket/740 cheers, David From cimrman3 at ntc.zcu.cz Fri Sep 19 08:00:22 2008 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Fri, 19 Sep 2008 14:00:22 +0200 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <3d375d730809190226g361d5855o1c1475da0379706f@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <48D23506.7080109@ntc.zcu.cz> <3d375d730809181237x43ce1610r6dccb081d6698f5d@mail.gmail.com> <48D363DD.6040609@ntc.zcu.cz> <3d375d730809190226g361d5855o1c1475da0379706f@mail.gmail.com> Message-ID: <48D39456.80505@ntc.zcu.cz> Robert Kern wrote: > Ah, found it. T_LONGLONG is a #define from structmember.h which is > used to describe the types of attributes. Apparently, this was not > added until Python 2.5. That particular member didn't actually need to > be long long, so I've fixed that. Great, I will try it after it appears on the web page. Thank you, r. From markbak at gmail.com Fri Sep 19 08:40:35 2008 From: markbak at gmail.com (mark) Date: Fri, 19 Sep 2008 05:40:35 -0700 (PDT) Subject: [Numpy-discussion] which one is best? Message-ID: I need to multiply items in a list and need a list back. Which one of the four options is best (I thought in Python there was only one way to do something???) >>> a = [1,2,3,4] >>> b = [10,20,30,40] >>> c = [] # method 1 >>> for i in range(len(a)): c.append(a[i]*b[i]) >>> c [10, 40, 90, 160] >>> array(a)*array(b) # method 2, but need to convert back to list array([ 10, 40, 90, 160]) >>> [ a[i] * b[i] for i in range(len(a)) ] # method 3 [10, 40, 90, 160] >>> [ x * y for x,y in zip(a,b) ] # method 4 [10, 40, 90, 160] Thanks for any insights, Mark From stefan at sun.ac.za Fri Sep 19 08:55:12 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 19 Sep 2008 14:55:12 +0200 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D36EF8.9030901@ar.media.kyoto-u.ac.jp> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <9457e7c80809190210t44025279y57e7ea916fe232d5@mail.gmail.com> <48D36EF8.9030901@ar.media.kyoto-u.ac.jp> Message-ID: <9457e7c80809190555k7de9e49cjb2325f70b0c7693e@mail.gmail.com> 2008/9/19 David Cournapeau : > But cannot this be fixed at the python level of the max function ? I Why shouldn't we have "nanmin"-like behaviour for the C min itself? I'd rather have a specialised function to deal with the rare kinds of datasets where NaNs are guaranteed never to occur. > But on my numpy, it looks like nan breaks min/max, they are not ignored: Yes, that's the problem. Cheers St?fan From rlw at stsci.edu Fri Sep 19 09:00:36 2008 From: rlw at stsci.edu (Rick White) Date: Fri, 19 Sep 2008 09:00:36 -0400 Subject: [Numpy-discussion] Generating random samples without repeats In-Reply-To: References: Message-ID: <6AEF99C7-675D-40C3-A318-D0E17C8E430F@stsci.edu> Paul Moore yahoo.co.uk> writes: > Robert Kern gmail.com> writes: >> On Thu, Sep 18, 2008 at 16:55, Paul Moore >> yahoo.co.uk> wrote: >>> I want to generate a series of random samples, to do simulations >>> based >>> on them. Essentially, I want to be able to produce a SAMPLESIZE * N >>> matrix, where each row of N values consists of either >>> >>> 1. Integers between 1 and M (simulating M rolls of an N-sided >>> die), or >>> 2. A sample of N numbers between 1 and M without repeats (simulating >>> deals of N cards from an M-card deck). >>> >>> Example (1) is easy, numpy.random.random_integers(1, M, >>> (SAMPLESIZE, N)) >>> >>> But I can't find an obvious equivalent for (2). Am I missing >>> something >>> glaringly obvious? I'm using numpy - is there maybe something in >>> scipy I >>> should be looking at? >> >> numpy.array([(numpy.random.permutation(M) + 1)[:N] >> for i in range(SAMPLESIZE)]) >> > > Thanks. > > And yet, this takes over 70s and peaks at around 400M memory use, > whereas the > equivalent for (1) > > numpy.random.random_integers(1,M,(SAMPLESIZE,N)) > > takes less than half a second, and negligible working memory (both > end up > allocating an array of the same size, but your suggestion consumes > temporary > working memory - I suspect, but can't prove, that the time taken > comes from > memory allocations rather than computation. It seems like numpy.random.permutation is pretty suboptimal in its speed. Here's a Python 1-liner that does the same thing (I think) but is a lot faster: a = 1+numpy.random.rand(M).argsort()[0:N-1] This still has the the problem that it generates a size N array to start with. But at least it is fast compared with permutation: >>> t1 = timeit.Timer("a = 1+numpy.random.permutation(M)[0:N-1]", "import numpy; M=1000; N=100") >>> t1.repeat(3,1000) [1.3792998790740967, 1.3387739658355713, 1.3446521759033203] >>> t2 = timeit.Timer("a = 1+numpy.random.rand(M).argsort()[0:N-1]", "import numpy; M=1000; N=100") >>> t2.repeat(3,1000) [0.17597699165344238, 0.16013598442077637, 0.16066503524780273] From stefan at sun.ac.za Fri Sep 19 09:09:00 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 19 Sep 2008 15:09:00 +0200 Subject: [Numpy-discussion] which one is best? In-Reply-To: References: Message-ID: <9457e7c80809190609l7f953524i1b309d017ba6d9d5@mail.gmail.com> 2008/9/19 mark : > I need to multiply items in a list and need a list back. Which one of > the four options is best (I thought in Python there was only one way > to do something???) With the emphasis on "preferably" and "obvious" :) "There should be one-- and preferably only one --obvious way to do it." The modern idiom is the list comprehension, rather than the for-loop. Of those options, I personally prefer using "zip". >>>> [ x * y for x,y in zip(a,b) ] # method 4 > [10, 40, 90, 160] If you have very large arrays, you can also consider (np.array(x) * np.array(y)).tolist() Cheers St?fan From didier.croutz at univ-nancy2.fr Fri Sep 19 09:27:34 2008 From: didier.croutz at univ-nancy2.fr (Didier CROUTZ) Date: Fri, 19 Sep 2008 15:27:34 +0200 Subject: [Numpy-discussion] Numpy 1.1.0 Silent Install Message-ID: <48D3A8C6.7010003@univ-nancy2.fr> Hello, I have to deploy Numpy 1.1.0 on a set of XP PRO SP3 computers (with Pytho 2.5). I'd like to know if I can silently install this product by the way of a DOS command. Thanks for your response, Regards, D. Croutz. From arnar.flatberg at gmail.com Fri Sep 19 09:50:49 2008 From: arnar.flatberg at gmail.com (Arnar Flatberg) Date: Fri, 19 Sep 2008 15:50:49 +0200 Subject: [Numpy-discussion] which one is best? In-Reply-To: <9457e7c80809190609l7f953524i1b309d017ba6d9d5@mail.gmail.com> References: <9457e7c80809190609l7f953524i1b309d017ba6d9d5@mail.gmail.com> Message-ID: <5d3194020809190650l28452702oe82d7fc92a9e3198@mail.gmail.com> On Fri, Sep 19, 2008 at 3:09 PM, St?fan van der Walt wrote: > 2008/9/19 mark : > > I need to multiply items in a list and need a list back. Which one of > > the four options is best (I thought in Python there was only one way > > to do something???) > > With the emphasis on "preferably" and "obvious" :) > > "There should be one-- and preferably only one --obvious way to do it." > > The modern idiom is the list comprehension, rather than the for-loop. > Of those options, > I personally prefer using "zip". > > >>>> [ x * y for x,y in zip(a,b) ] # method 4 > > [10, 40, 90, 160] > > If you have very large arrays, you can also consider > > (np.array(x) * np.array(y)).tolist() > > Cheers > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > I think [x*y for x in a for y in b] feels pythonic, however it has a surprisingly lousy performance. In [30]: %timeit [ x * y for x,y in zip(a,b) ] 100000 loops, best of 3: 3.96 ?s per loop In [31]: %timeit [ i*j for i in a for j in b ] 100000 loops, best of 3: 6.53 ?s per loop In [32]: a = range(100) In [33]: b = range(100) In [34]: %timeit [ x * y for x,y in zip(a,b) ] 10000 loops, best of 3: 51.9 ?s per loop In [35]: %timeit [ i*j for i in a for j in b ] 100 loops, best of 3: 2.78 ms per loop Arnar -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Fri Sep 19 10:07:56 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 19 Sep 2008 10:07:56 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D36435.5050709@ar.media.kyoto-u.ac.jp> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> Message-ID: <48D3B23C.8030100@american.edu> On 9/19/2008 4:35 AM David Cournapeau apparently wrote: > I never use NaN as missing value What do you use? Recently I needed to fill a 2d array with values from computations that could "go wrong". I created an array of NaN and then replaced the elements where the computation produced a useful value. I then applied ``nanmax``, to get the maximum of the useful values. What should I have done? Thanks, Alan Isaac From lbolla at gmail.com Fri Sep 19 10:09:11 2008 From: lbolla at gmail.com (lorenzo) Date: Fri, 19 Sep 2008 15:09:11 +0100 Subject: [Numpy-discussion] which one is best? In-Reply-To: <5d3194020809190650l28452702oe82d7fc92a9e3198@mail.gmail.com> References: <9457e7c80809190609l7f953524i1b309d017ba6d9d5@mail.gmail.com> <5d3194020809190650l28452702oe82d7fc92a9e3198@mail.gmail.com> Message-ID: <80c99e790809190709j5f141e93w2e3ccfe16ab0af1b@mail.gmail.com> On Fri, Sep 19, 2008 at 2:50 PM, Arnar Flatberg wrote: > > > I think > [x*y for x in a for y in b] > feels pythonic, however it has a surprisingly lousy performance. > > This returns a len(x)*len(y) long list, which is not what you want. This two methods seem equivalent: In [16]: x = 100000 * [1] In [17]: y = 100000 * [2] In [18]: %timeit map(lambda a,b: a*b, x, y) 10 loops, best of 3: 31.4 ms per loop In [19]: %timeit [xx*yy for xx,yy in zip(x,y)] 10 loops, best of 3: 30.4 ms per loop L. -------------- next part -------------- An HTML attachment was scrubbed... URL: From arnar.flatberg at gmail.com Fri Sep 19 10:14:38 2008 From: arnar.flatberg at gmail.com (Arnar Flatberg) Date: Fri, 19 Sep 2008 16:14:38 +0200 Subject: [Numpy-discussion] which one is best? In-Reply-To: <80c99e790809190709j5f141e93w2e3ccfe16ab0af1b@mail.gmail.com> References: <9457e7c80809190609l7f953524i1b309d017ba6d9d5@mail.gmail.com> <5d3194020809190650l28452702oe82d7fc92a9e3198@mail.gmail.com> <80c99e790809190709j5f141e93w2e3ccfe16ab0af1b@mail.gmail.com> Message-ID: <5d3194020809190714h1775c89fn2d73c6ce8ba4895c@mail.gmail.com> On Fri, Sep 19, 2008 at 4:09 PM, lorenzo wrote: > > > On Fri, Sep 19, 2008 at 2:50 PM, Arnar Flatberg wrote: > >> >> >> I think >> [x*y for x in a for y in b] >> feels pythonic, however it has a surprisingly lousy performance. >> >> > This returns a len(x)*len(y) long list, which is not what you want. > My bad, Its friday afternoon, I'll go home now :-) Arnar -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Fri Sep 19 10:24:14 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 19 Sep 2008 10:24:14 -0400 Subject: [Numpy-discussion] which one is best? In-Reply-To: References: Message-ID: <48D3B60E.7080100@american.edu> Given your restrictions I'd probably pick [ x * y for x,y in izip(a,b) ] (note the change to ``izip``). But maybe you do not really need lists at all... You can also do c = (array(a)*array(b)).tolist() which is nice and explicit but builds 3 unnecessary arrays. Alan Isaac From David.Kaplan at ird.fr Fri Sep 19 10:24:48 2008 From: David.Kaplan at ird.fr (David M. Kaplan) Date: Fri, 19 Sep 2008 16:24:48 +0200 Subject: [Numpy-discussion] which one is best? In-Reply-To: References: Message-ID: <1221834288.7162.93.camel@localhost> Hi Arnar, Your two commands below aren't doing the same thing - one is doing a[i]*b[i] and the other is doing a[i]*b[j] for all i and j. As the second is harder, it takes longer. Cheers, David On Fri, 2008-09-19 at 09:08 -0500, numpy-discussion-request at scipy.org wrote: > I think > [x*y for x in a for y in b] > feels pythonic, however it has a surprisingly lousy performance. > > In [30]: %timeit [ x * y for x,y in zip(a,b) ] > 100000 loops, best of 3: 3.96 ?s per loop > > In [31]: %timeit [ i*j for i in a for j in b ] > 100000 loops, best of 3: 6.53 ?s per loop > > In [32]: a = range(100) > > In [33]: b = range(100) > > In [34]: %timeit [ x * y for x,y in zip(a,b) ] > 10000 loops, best of 3: 51.9 ?s per loop > > In [35]: %timeit [ i*j for i in a for j in b ] > 100 loops, best of 3: 2.78 ms per loop > > Arnar -- ********************************** David M. Kaplan Charge de Recherche 1 Institut de Recherche pour le Developpement Centre de Recherche Halieutique Mediterraneenne et Tropicale av. Jean Monnet B.P. 171 34203 Sete cedex France Phone: +33 (0)4 99 57 32 27 Fax: +33 (0)4 99 57 32 95 http://www.ur097.ird.fr/team/dkaplan/index.html ********************************** From pf_moore at yahoo.co.uk Fri Sep 19 11:08:39 2008 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Fri, 19 Sep 2008 15:08:39 +0000 (UTC) Subject: [Numpy-discussion] Generating random samples without repeats References: <6AEF99C7-675D-40C3-A318-D0E17C8E430F@stsci.edu> Message-ID: Rick White stsci.edu> writes: > It seems like numpy.random.permutation is pretty suboptimal in its > speed. Here's a Python 1-liner that does the same thing (I think) > but is a lot faster: > > a = 1+numpy.random.rand(M).argsort()[0:N-1] > > This still has the the problem that it generates a size N array to > start with. But at least it is fast compared with permutation: Interesting. For my generation of a million samples, this takes about 46 sec vs the original 75. That's a 35% increase in speed. As you mention, it doesn't help memory, which still peaks at around 450M. Interestingly, I was reminded of J (http://www.jsoftware.com/), an APL derivative, which does this in a blistering 1.3 seconds, with no detectable memory overhead. Of course, being descended from APL, the code to do this is pretty obscure: 5 ? (1000000 $ 52) (Here, ? is the "deal" operator, and $ reshapes an array - so it's "deal 5 from each item in a 1000000-long array of 52's". Everything is a primitive here, so it's not hard to see why it's fast). A Python/Numpy <-> J bridge might be a fun exercise... Paul. From stefan at sun.ac.za Fri Sep 19 11:09:07 2008 From: stefan at sun.ac.za (Stefan Van der Walt) Date: Fri, 19 Sep 2008 17:09:07 +0200 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D3B23C.8030100@american.edu> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <48D3B23C.8030100@american.edu> Message-ID: <96A799E8-0C2D-4DAB-8209-28A78B4B8D9A@sun.ac.za> On 19 Sep 2008, at 16:07 , Alan G Isaac wrote: > On 9/19/2008 4:35 AM David Cournapeau apparently wrote: >> I never use NaN as missing value > > What do you use? Masked arrays. Using NaN's for missing values is dangerous. You may do some operation, which generates invalid results, and then you have a mixed bag of missing and invalid values. Cheers St?fan From aisaac at american.edu Fri Sep 19 11:36:17 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 19 Sep 2008 11:36:17 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <96A799E8-0C2D-4DAB-8209-28A78B4B8D9A@sun.ac.za> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <48D3B23C.8030100@american.edu> <96A799E8-0C2D-4DAB-8209-28A78B4B8D9A@sun.ac.za> Message-ID: <48D3C6F1.2060500@american.edu> On 9/19/2008 11:09 AM Stefan Van der Walt apparently wrote: > Masked arrays. Using NaN's for missing values is dangerous. You may > do some operation, which generates invalid results, and then you have > a mixed bag of missing and invalid values. That rather evades my full question, I think? In the case I mentioned, I am filling an array inside a loop, and the possible fill values are not constrained. So I cannot mask based on value, and I cannot mask based on position (at least until after the computations are complete). It seems to me that there are pragmatic reasons why people work with NaNs for missing values, that perhaps shd not be dismissed so quickly. But maybe I am overlooking a simple solution. Alan PS I confess I do not understand NaNs. E.g., why could there not be a value np.miss that would be a NaN that represents a missing value? Are all NaNs already assigned standard meanings? From pgmdevlist at gmail.com Fri Sep 19 11:46:37 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 19 Sep 2008 11:46:37 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D3C6F1.2060500@american.edu> References: <48D23B35.8040902@dcs.gla.ac.uk> <96A799E8-0C2D-4DAB-8209-28A78B4B8D9A@sun.ac.za> <48D3C6F1.2060500@american.edu> Message-ID: <200809191146.37228.pgmdevlist@gmail.com> On Friday 19 September 2008 11:36:17 Alan G Isaac wrote: > On 9/19/2008 11:09 AM Stefan Van der Walt apparently wrote: > > Masked arrays. Using NaN's for missing values is dangerous. You may > > do some operation, which generates invalid results, and then you have > > a mixed bag of missing and invalid values. > > That rather evades my full question, I think? > > In the case I mentioned, > I am filling an array inside a loop, > and the possible fill values are not constrained. > So I cannot mask based on value, > and I cannot mask based on position > (at least until after the computations are complete). No, but you may do the opposite: just start with an array completely masked, and unmasked it as you need: Say, you have 4x5 array, and want to unmask (0,0), (1,2), (3,4) >>> a = ma.empty((4,5), dtype=float) >>> a.mask=True >>> a[0,0] = 0 >>> a[1,2]=1 >>> a[3,4]=3 >>>a masked_array(data = [[0.0 -- -- -- --] [-- -- 1.0 -- --] [-- -- -- -- --] [-- -- -- -- 3.0]], mask = [[False True True True True] [ True True False True True] [ True True True True True] [ True True True True False]], fill_value=1e+20) >>>a.max(axis=0) masked_array(data = [0.0 -- 1.0 -- 3.0], mask = [False True False True False], fill_value=1e+20) > It seems to me that there are pragmatic reasons > why people work with NaNs for missing values, > that perhaps shd not be dismissed so quickly. > But maybe I am overlooking a simple solution. nansomething solutions tend to be considerably faster, that might be one reason. A lack of visibility of numpy.ma could be a second. In any case, I can't but agree with other posters: a NaN in an array usually means something went astray. > PS I confess I do not understand NaNs. > E.g., why could there not be a value np.miss > that would be a NaN that represents a missing value? You can't compare NaNs to anything. How do you know this np.miss is a masked value, when np.sqrt(-1.) is NaN ? From David.Kaplan at ird.fr Fri Sep 19 11:49:08 2008 From: David.Kaplan at ird.fr (David M. Kaplan) Date: Fri, 19 Sep 2008 17:49:08 +0200 Subject: [Numpy-discussion] New patch for new mgrid / ogrid functionality Message-ID: <1221839348.7162.139.camel@localhost> Hi all, Attached is a newer version of my patch that adds new mgrid / ogrid functionality for working with arrays in addition to slices. In fact, I have attached two versions of the patch: index_tricks.patch, that is just the last version of the patch I sent, and index_tricks.new.patch, that has been modified so that it is backward compatible. In the last version, mgrid calls where all arguments are slices will return an array, otherwise it returns a list as ogrid does. This is the only reasonable way to have the new functionality and maintain backwards compatibility. My 2 cents - I personally think the version that always returns a list will ultimately be more transparent and cause fewer problems than the newer version. In either case, the plan should be to eventually have it always return a list as that is the only fully consistent option, the question is just when that switch should be made and by who. If it is done at the next major release, someone else will have to remember to ax the additional code and correct the documentation.... Other changes that would be nice: add a __call__ method, create an instance called ndgrid for matlab compatibility, and have meshgrid be reimplimented using an nd_grid instance. Cheers, David -- ********************************** David M. Kaplan Charge de Recherche 1 Institut de Recherche pour le Developpement Centre de Recherche Halieutique Mediterraneenne et Tropicale av. Jean Monnet B.P. 171 34203 Sete cedex France Phone: +33 (0)4 99 57 32 27 Fax: +33 (0)4 99 57 32 95 http://www.ur097.ird.fr/team/dkaplan/index.html ********************************** -------------- next part -------------- A non-text attachment was scrubbed... Name: index_tricks.new.patch Type: text/x-patch Size: 8848 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: index_tricks.patch Type: text/x-patch Size: 8077 bytes Desc: not available URL: From pzs at dcs.gla.ac.uk Fri Sep 19 12:02:08 2008 From: pzs at dcs.gla.ac.uk (Peter Saffrey) Date: Fri, 19 Sep 2008 16:02:08 +0000 (UTC) Subject: [Numpy-discussion] Medians that ignore values References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <48D3B23C.8030100@american.edu> Message-ID: Alan G Isaac american.edu> writes: > Recently I needed to fill a 2d array with values > from computations that could "go wrong". > I created an array of NaN and then replaced > the elements where the computation produced > a useful value. I then applied ``nanmax``, > to get the maximum of the useful values. > I'm glad you posted this, because this is exactly the method I'm using. How do you detect whether there are still any missing spots in your array? nan has some rather unfortunate properties: >>> from numpy import * >>> a = array([1,2,nan]) >>> nan in a False >>> nan == nan False Should I take the earlier advice and switch to masked arrays? Peter From pgmdevlist at gmail.com Fri Sep 19 12:05:29 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 19 Sep 2008 12:05:29 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: References: <48D23B35.8040902@dcs.gla.ac.uk> <48D3B23C.8030100@american.edu> Message-ID: <200809191205.29895.pgmdevlist@gmail.com> On Friday 19 September 2008 12:02:08 Peter Saffrey wrote: > Alan G Isaac american.edu> writes: > > Recently I needed to fill a 2d array with values > > from computations that could "go wrong". > Should I take the earlier advice and switch to masked arrays? > > Peter Yes. As you've noticed, you can't compare nans (after all, nans are not numbers...), which limits their use. From aisaac at american.edu Fri Sep 19 12:22:05 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 19 Sep 2008 12:22:05 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <200809191146.37228.pgmdevlist@gmail.com> References: <48D23B35.8040902@dcs.gla.ac.uk> <96A799E8-0C2D-4DAB-8209-28A78B4B8D9A@sun.ac.za> <48D3C6F1.2060500@american.edu> <200809191146.37228.pgmdevlist@gmail.com> Message-ID: <48D3D1AD.6060803@american.edu> On 9/19/2008 11:46 AM Pierre GM apparently wrote: > You can't compare NaNs to anything. How do you know this np.miss is a masked > value, when np.sqrt(-1.) is NaN ? I thought you could use ``is``. E.g., >>> np.nan == np.nan False >>> np.nan is np.nan True Alan From aisaac at american.edu Fri Sep 19 12:23:16 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 19 Sep 2008 12:23:16 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <200809191146.37228.pgmdevlist@gmail.com> References: <48D23B35.8040902@dcs.gla.ac.uk> <96A799E8-0C2D-4DAB-8209-28A78B4B8D9A@sun.ac.za> <48D3C6F1.2060500@american.edu> <200809191146.37228.pgmdevlist@gmail.com> Message-ID: <48D3D1F4.9060202@american.edu> On 9/19/2008 11:46 AM Pierre GM apparently wrote: > No, but you may do the opposite: just start with an array completely masked, > and unmasked it as you need: Very useful example. I did not understand this possibility. Alan From charlesr.harris at gmail.com Fri Sep 19 12:25:07 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 19 Sep 2008 10:25:07 -0600 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D35089.30202@ar.media.kyoto-u.ac.jp> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D2FEC3.3040400@ar.media.kyoto-u.ac.jp> <48D31264.2040503@ar.media.kyoto-u.ac.jp> <48D32794.6030804@ar.media.kyoto-u.ac.jp> <48D35089.30202@ar.media.kyoto-u.ac.jp> Message-ID: On Fri, Sep 19, 2008 at 1:11 AM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Anne Archibald wrote: > > > > Well, for example, you might ask that all the non-nan elements be in > > order, even if you don't specify where the nan goes. > > > Ah, there are two problems, then: > - sort > - how median use sort. > > For sort, I don't know how sort speed would be influenced by treating > nan. In a way, calling sort with nan inside is a user error (if you take > the POV nan are not comparable), but nan are used for all kind of > purpose, used <- misused. Using nan to flag anything but a numerical error is going to cause problems. It wouldn't be too hard to implement nansorts, they just need a real comparison function so that all the nans end up at on end or the other. I don't know that that would make medians any easier, though. Are the nans part of the data set? A nansearchsorted would probably be needed also. If this functionality is added, the best way might be something like kind='nanquicksort'. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Fri Sep 19 12:34:46 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 19 Sep 2008 12:34:46 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <48D3B23C.8030100@american.edu> Message-ID: <48D3D4A6.8040709@american.edu> On 9/19/2008 12:02 PM Peter Saffrey apparently wrote: > >>> a = array([1,2,nan]) > >>> nan in a > False Huh. I'm inclined to call this a bug, since normal Python behavior is that ``in`` should check for identity:: >>> xl = [1.,np.nan] >>> np.nan in xl True Alan From aisaac at american.edu Fri Sep 19 12:59:57 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 19 Sep 2008 12:59:57 -0400 Subject: [Numpy-discussion] np.nan and ``is`` Message-ID: <48D3DA8D.70404@american.edu> Might someone explain this to me? >>> x = [1.,np.nan] >>> np.nan in x True >>> np.nan in np.array(x) False >>> np.nan in np.array(x).tolist() False >>> np.nan is float(np.nan) True Thank you, Alan Isaac From dalcinl at gmail.com Fri Sep 19 13:15:37 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 19 Sep 2008 14:15:37 -0300 Subject: [Numpy-discussion] np.nan and ``is`` In-Reply-To: <48D3DA8D.70404@american.edu> References: <48D3DA8D.70404@american.edu> Message-ID: You, know, float are inmutable objects, and then 'float(f)' just returns a new reference to 'f' is 'f' is (exactly) of type 'float' In [1]: f = 1.234 In [2]: f is float(f) Out[2]: True I do not remember right now the implementations of comparisons in core Python, but I believe the 'in' operator is testing first for object identity, and then 'np.nan in [np.nan]' then returns True, and then the fact that 'np.nan==np.nan' returns False is never considered. On Fri, Sep 19, 2008 at 1:59 PM, Alan G Isaac wrote: > Might someone explain this to me? > > >>> x = [1.,np.nan] > >>> np.nan in x > True > >>> np.nan in np.array(x) > False > >>> np.nan in np.array(x).tolist() > False > >>> np.nan is float(np.nan) > True > > Thank you, > Alan Isaac > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From aisaac at american.edu Fri Sep 19 13:34:51 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 19 Sep 2008 13:34:51 -0400 Subject: [Numpy-discussion] np.nan and ``is`` In-Reply-To: References: <48D3DA8D.70404@american.edu> Message-ID: <48D3E2BB.1090004@american.edu> > On Fri, Sep 19, 2008 at 1:59 PM, Alan G Isaac wrote: >> Might someone explain this to me? >> >> >>> x = [1.,np.nan] >> >>> np.nan in x >> True >> >>> np.nan in np.array(x) >> False >> >>> np.nan in np.array(x).tolist() >> False >> >>> np.nan is float(np.nan) >> True On 9/19/2008 1:15 PM Lisandro Dalcin apparently wrote: > I do not remember right now the implementations of comparisons in core > Python, but I believe the 'in' operator is testing first for object > identity, and then 'np.nan in [np.nan]' then returns True, and then > the fact that 'np.nan==np.nan' returns False is never considered. Sure. All evaluations to True make sense to me. I am asking about the ones that evaluate to False. Thanks, Alan From Chris.Barker at noaa.gov Fri Sep 19 13:52:17 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 19 Sep 2008 10:52:17 -0700 Subject: [Numpy-discussion] np.nan and ``is`` In-Reply-To: <48D3DA8D.70404@american.edu> References: <48D3DA8D.70404@american.edu> Message-ID: <48D3E6D1.8060704@noaa.gov> Alan G Isaac wrote: > Might someone explain this to me? > > >>> x = [1.,np.nan] > >>> np.nan in x > True > >>> np.nan in np.array(x) > False > >>> np.nan in np.array(x).tolist() > False > >>> np.nan is float(np.nan) > True not quite -- but I do know that "is" is tricky -- it tests object identity. I think it actually compares the pointer to the object. What makes this tricky is that python interns some objects, so that when you create two that have the same value, they may actually be the same object: >>> s1 = "this" >>> s2 = "this" >>> s1 is s2 True So short strings are interned, as are small integers and maybe floats? However, longer strings are not: >>> s1 = "A much longer string" >>> s2 = "A much longer string" >>> s1 is s2 False I don't know the interning rules, but I do know that you should never count on them, then may not be consistent between implementations, or even different runs. NaN is a floating point number with a specific value. np.nan is particular instance of that, but not all nans will be the same instance: >>> np.array(0.0) / 0 nan >>> np.array(0.0) / 0 is np.nan False So you can't use "is" to check. >>> np.array(0.0) / 0 == np.nan False and you can't use "==" The only way to do it reliably is: >>> np.isnan(np.array(0.0) / 0) True So, the short answer is that the only way to deal with NaNs properly is to have NaN-aware functions, like nanmin() and friends. Regardless of how man nan* functions get written, or what exactly they do, we really do need to make sure that no numpy function gives bogus results in the presence of NaNs, which doesn't appear to be the case now. I also think I see a consensus building that non-nan-specific numpy functions should either preserve NaN's or raise exceptions, rather than ignoring them. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Fri Sep 19 13:58:15 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 19 Sep 2008 12:58:15 -0500 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D3D4A6.8040709@american.edu> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <48D3B23C.8030100@american.edu> <48D3D4A6.8040709@american.edu> Message-ID: <3d375d730809191058r79741a09w62238631fce5ab62@mail.gmail.com> On Fri, Sep 19, 2008 at 11:34, Alan G Isaac wrote: > On 9/19/2008 12:02 PM Peter Saffrey apparently wrote: >> >>> a = array([1,2,nan]) >> >>> nan in a >> False > > Huh. I'm inclined to call this a bug, > since normal Python behavior is that > ``in`` should check for identity:: > > >>> xl = [1.,np.nan] > >>> np.nan in xl > True Except that there are no objects inside non-object arrays. There is nothing with identity inside the arrays to compare against. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From brad.malone at gmail.com Fri Sep 19 13:59:22 2008 From: brad.malone at gmail.com (Brad Malone) Date: Fri, 19 Sep 2008 10:59:22 -0700 Subject: [Numpy-discussion] Understanding mgrid Message-ID: Hi, I was wondering if someone could englighten me on what the geometrical significance of numpy.mgrid is. I can play around with it and see trends in the sizes and number of arrays, but why does it give the output that it does? Looking at the example shown below, why does it return a matrix and its transpose? Is this a representation of some geometrical grid? Does the output imply some sort of connectivity? If so, how do you see it? >>> mgrid[0:5,0:5] array([[[0, 0, 0, 0, 0], [1, 1, 1, 1, 1], [2, 2, 2, 2, 2], [3, 3, 3, 3, 3], [4, 4, 4, 4, 4]], [[0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4]]]) I have a cubic grid in 3D space that is spanned by 3 orthogonal vectors. Am I able to generate this equivalent grid with mgrid somehow? If so, how is it done? I am using mayavi and I need to be able to construct some arrays in the same way that mgrid would have constructed them, so this is why I ask. Thanks for the help, Brad -------------- next part -------------- An HTML attachment was scrubbed... URL: From efiring at hawaii.edu Fri Sep 19 14:01:13 2008 From: efiring at hawaii.edu (Eric Firing) Date: Fri, 19 Sep 2008 08:01:13 -1000 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <200809191146.37228.pgmdevlist@gmail.com> References: <48D23B35.8040902@dcs.gla.ac.uk> <96A799E8-0C2D-4DAB-8209-28A78B4B8D9A@sun.ac.za> <48D3C6F1.2060500@american.edu> <200809191146.37228.pgmdevlist@gmail.com> Message-ID: <48D3E8E9.1080406@hawaii.edu> Pierre GM wrote: >> It seems to me that there are pragmatic reasons >> why people work with NaNs for missing values, >> that perhaps shd not be dismissed so quickly. >> But maybe I am overlooking a simple solution. > > nansomething solutions tend to be considerably faster, that might be one > reason. A lack of visibility of numpy.ma could be a second. In any case, I > can't but agree with other posters: a NaN in an array usually means something > went astray. Additional reasons for using nans: 1) years of experience with Matlab, in which using nan for missing values is the standard idiom. 2) convenient interfacing with extension code in C or C++. The latter is a factor in the present use of nan in matplotlib; using nan for missing values in an array passed into extension code saves having to pass and process a second (mask) array. It is fast and simple. Eric From robert.kern at gmail.com Fri Sep 19 14:22:29 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 19 Sep 2008 13:22:29 -0500 Subject: [Numpy-discussion] Understanding mgrid In-Reply-To: References: Message-ID: <3d375d730809191122n1699c066h183d84cc8b6769b3@mail.gmail.com> On Fri, Sep 19, 2008 at 12:59, Brad Malone wrote: > Hi, I was wondering if someone could englighten me on what the geometrical > significance of numpy.mgrid is. I can play around with it and see trends in > the sizes and number of arrays, but why does it give the output that it > does? Looking at the example shown below, why does it return a matrix and > its transpose? Well, it returns one array. In your example, there is a (2,5,5) array, which is basically the concatenation of two arrays which *happen* to be transposes of each other. If you had chosen differently sized axes, they wouldn't be transposes. In [14]: mgrid[0:2,0:3] Out[14]: array([[[0, 0, 0], [1, 1, 1]], [[0, 1, 2], [0, 1, 2]]]) > Is this a representation of some geometrical grid? It can be. There are other uses for it. > Does the > output imply some sort of connectivity? It describes an orthogonal grid. > If so, how do you see it? > > >>> mgrid[0:5,0:5] > array([[[0, 0, 0, 0, 0], > [1, 1, 1, 1, 1], > [2, 2, 2, 2, 2], > > [3, 3, 3, 3, 3], > > [4, 4, 4, 4, 4]], > > [[0, 1, 2, 3, 4], > > [0, 1, 2, 3, 4], > [0, 1, 2, 3, 4], > [0, 1, 2, 3, 4], > > > [0, 1, 2, 3, 4]]]) > > > > I have a cubic grid in 3D space that is spanned by 3 orthogonal vectors. Am > I able to generate this equivalent grid with mgrid somehow? If so, how is it > done? I am using mayavi and I need to be able to construct some arrays in > the same way that mgrid would have constructed them, so this is why I ask. I would probably use indices() instead of mgrid if you are just given the x, y, and z vectors. indices([n,m,k]) is equivalent to mgrid[0:n,0:m,0:k]: In [19]: x = linspace(0, 1, 3) In [20]: x Out[20]: array([ 0. , 0.5, 1. ]) In [21]: y = linspace(1, 2.5, 4) In [22]: y Out[22]: array([ 1. , 1.5, 2. , 2.5]) In [23]: z = linspace(3, 5, 5) In [24]: z Out[24]: array([ 3. , 3.5, 4. , 4.5, 5. ]) In [25]: ix, iy, iz = indices([len(x), len(y), len(z)]) In [26]: x[ix] Out[26]: array([[[ 0. , 0. , 0. , 0. , 0. ], [ 0. , 0. , 0. , 0. , 0. ], [ 0. , 0. , 0. , 0. , 0. ], [ 0. , 0. , 0. , 0. , 0. ]], [[ 0.5, 0.5, 0.5, 0.5, 0.5], [ 0.5, 0.5, 0.5, 0.5, 0.5], [ 0.5, 0.5, 0.5, 0.5, 0.5], [ 0.5, 0.5, 0.5, 0.5, 0.5]], [[ 1. , 1. , 1. , 1. , 1. ], [ 1. , 1. , 1. , 1. , 1. ], [ 1. , 1. , 1. , 1. , 1. ], [ 1. , 1. , 1. , 1. , 1. ]]]) In [27]: y[iy] Out[27]: array([[[ 1. , 1. , 1. , 1. , 1. ], [ 1.5, 1.5, 1.5, 1.5, 1.5], [ 2. , 2. , 2. , 2. , 2. ], [ 2.5, 2.5, 2.5, 2.5, 2.5]], [[ 1. , 1. , 1. , 1. , 1. ], [ 1.5, 1.5, 1.5, 1.5, 1.5], [ 2. , 2. , 2. , 2. , 2. ], [ 2.5, 2.5, 2.5, 2.5, 2.5]], [[ 1. , 1. , 1. , 1. , 1. ], [ 1.5, 1.5, 1.5, 1.5, 1.5], [ 2. , 2. , 2. , 2. , 2. ], [ 2.5, 2.5, 2.5, 2.5, 2.5]]]) In [28]: z[iz] Out[28]: array([[[ 3. , 3.5, 4. , 4.5, 5. ], [ 3. , 3.5, 4. , 4.5, 5. ], [ 3. , 3.5, 4. , 4.5, 5. ], [ 3. , 3.5, 4. , 4.5, 5. ]], [[ 3. , 3.5, 4. , 4.5, 5. ], [ 3. , 3.5, 4. , 4.5, 5. ], [ 3. , 3.5, 4. , 4.5, 5. ], [ 3. , 3.5, 4. , 4.5, 5. ]], [[ 3. , 3.5, 4. , 4.5, 5. ], [ 3. , 3.5, 4. , 4.5, 5. ], [ 3. , 3.5, 4. , 4.5, 5. ], [ 3. , 3.5, 4. , 4.5, 5. ]]]) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pgmdevlist at gmail.com Fri Sep 19 14:23:02 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 19 Sep 2008 14:23:02 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D3E8E9.1080406@hawaii.edu> References: <48D23B35.8040902@dcs.gla.ac.uk> <200809191146.37228.pgmdevlist@gmail.com> <48D3E8E9.1080406@hawaii.edu> Message-ID: <200809191423.02668.pgmdevlist@gmail.com> On Friday 19 September 2008 14:01:13 Eric Firing wrote: > Pierre GM wrote: > 2) convenient interfacing with extension code in C or C++. > > The latter is a factor in the present use of nan in matplotlib; using > nan for missing values in an array passed into extension code saves > having to pass and process a second (mask) array. It is fast and simple. As long as you deal with floats. From aisaac at american.edu Fri Sep 19 14:34:02 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 19 Sep 2008 14:34:02 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <3d375d730809191058r79741a09w62238631fce5ab62@mail.gmail.com> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <48D3B23C.8030100@american.edu> <48D3D4A6.8040709@american.edu> <3d375d730809191058r79741a09w62238631fce5ab62@mail.gmail.com> Message-ID: <48D3F09A.3040000@american.edu> On 9/19/2008 1:58 PM Robert Kern apparently wrote: > there are no objects inside non-object arrays. There is > nothing with identity inside the arrays to compare against. Got it. Thanks. Alan From dalke at dalkescientific.com Fri Sep 19 14:39:32 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Fri, 19 Sep 2008 20:39:32 +0200 Subject: [Numpy-discussion] np.nan and ``is`` In-Reply-To: <48D3E6D1.8060704@noaa.gov> References: <48D3DA8D.70404@american.edu> <48D3E6D1.8060704@noaa.gov> Message-ID: <91C39661-D647-4B11-B752-D48B3CE5FF0F@dalkescientific.com> On Sep 19, 2008, at 7:52 PM, Christopher Barker wrote: > I don't know the interning rules, but I do know that you should never > count on them, then may not be consistent between implementations, or > even different runs. There are a few things that Python-the-language guarantees are singleton objects which can be compared correctly with "is". Those are: True, False, None Otherwise there is no guarantee that two objects of a given type which are "equal" in some sense of the word, are actually the same object. As Chris pointed out, the C implementation does (as a performance matter) have additional singletons. For example, the integers between -5 to 257 are also singletons #ifndef NSMALLPOSINTS #define NSMALLPOSINTS 257 #endif #ifndef NSMALLNEGINTS #define NSMALLNEGINTS 5 #endif /* References to small integers are saved in this array so that they can be shared. The integers that are saved are those in the range -NSMALLNEGINTS (inclusive) to NSMALLPOSINTS (not inclusive). */ static PyIntObject *small_ints[NSMALLNEGINTS + NSMALLPOSINTS]; This used to be -1 to 100 but some testing showed it was better to extend the range somewhat. There was also some performance testing about special-casing 0.0 and +/- 1.0 but I think it showed the results weren't worthwhile. So, back to NaN. There's no guarantee NaN is a singleton object, so testing with "is" almost certainly is wrong. In fact, at the bit-level there are multiple NaNs. A NaN (according to Wikipedia) fits the following bit pattern. NaN: x11111111axxxxxxxxxxxxxxxxxxxxxx. x = undefined. If a = 1, it is a quiet NaN, otherwise it is a signalling NaN. So 11111111111111111111111111111111 and 11111111111111111111111111111110 and 11111111111111111111110000000000 are all NaN values. > Andrew dalke at dalkescientific.com From robert.kern at gmail.com Fri Sep 19 14:46:40 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 19 Sep 2008 13:46:40 -0500 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <48D39456.80505@ntc.zcu.cz> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <48D23506.7080109@ntc.zcu.cz> <3d375d730809181237x43ce1610r6dccb081d6698f5d@mail.gmail.com> <48D363DD.6040609@ntc.zcu.cz> <3d375d730809190226g361d5855o1c1475da0379706f@mail.gmail.com> <48D39456.80505@ntc.zcu.cz> Message-ID: <3d375d730809191146k7425c569i2450e17e8d51c588@mail.gmail.com> On Fri, Sep 19, 2008 at 07:00, Robert Cimrman wrote: > Robert Kern wrote: >> Ah, found it. T_LONGLONG is a #define from structmember.h which is >> used to describe the types of attributes. Apparently, this was not >> added until Python 2.5. That particular member didn't actually need to >> be long long, so I've fixed that. > > Great, I will try it after it appears on the web page. Oops! It's now pushed. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From brad.malone at gmail.com Fri Sep 19 15:13:25 2008 From: brad.malone at gmail.com (Brad Malone) Date: Fri, 19 Sep 2008 12:13:25 -0700 Subject: [Numpy-discussion] Understanding mgrid In-Reply-To: <3d375d730809191122n1699c066h183d84cc8b6769b3@mail.gmail.com> References: <3d375d730809191122n1699c066h183d84cc8b6769b3@mail.gmail.com> Message-ID: Thanks for the response Robert. So, at least in this case, the results of mgrid (or indices) only provides information about the spacing of the grid and not on the absolute value of the point coordinates? In your example, is there a way to see within your x[ix], y[iy], and z[iz] matrices the same collection of points that you would see if you did something like the following? points=[] x=linspace(0,1,3) y=linspace(1,2.5,4) z=linspace(3,5,5) for k in z.tolist(): for j in y.tolist(): for i in x.tolist(): point=array([i,j,k]) points.append(point) Thanks, Brad On Fri, Sep 19, 2008 at 11:22 AM, Robert Kern wrote: > On Fri, Sep 19, 2008 at 12:59, Brad Malone wrote: > > Hi, I was wondering if someone could englighten me on what the > geometrical > > significance of numpy.mgrid is. I can play around with it and see trends > in > > the sizes and number of arrays, but why does it give the output that it > > does? Looking at the example shown below, why does it return a matrix and > > its transpose? > > Well, it returns one array. In your example, there is a (2,5,5) array, > which is basically the concatenation of two arrays which *happen* to > be transposes of each other. If you had chosen differently sized axes, > they wouldn't be transposes. > > In [14]: mgrid[0:2,0:3] > Out[14]: > array([[[0, 0, 0], > [1, 1, 1]], > > [[0, 1, 2], > [0, 1, 2]]]) > > > Is this a representation of some geometrical grid? > > It can be. There are other uses for it. > > > Does the > > output imply some sort of connectivity? > > It describes an orthogonal grid. > > > If so, how do you see it? > > > > >>> mgrid[0:5,0:5] > > array([[[0, 0, 0, 0, 0], > > [1, 1, 1, 1, 1], > > [2, 2, 2, 2, 2], > > > > [3, 3, 3, 3, 3], > > > > [4, 4, 4, 4, 4]], > > > > [[0, 1, 2, 3, 4], > > > > [0, 1, 2, 3, 4], > > [0, 1, 2, 3, 4], > > [0, 1, 2, 3, 4], > > > > > > [0, 1, 2, 3, 4]]]) > > > > > > > > I have a cubic grid in 3D space that is spanned by 3 orthogonal vectors. > Am > > I able to generate this equivalent grid with mgrid somehow? If so, how is > it > > done? I am using mayavi and I need to be able to construct some arrays in > > the same way that mgrid would have constructed them, so this is why I > ask. > > I would probably use indices() instead of mgrid if you are just given > the x, y, and z vectors. indices([n,m,k]) is equivalent to > mgrid[0:n,0:m,0:k]: > > > In [19]: x = linspace(0, 1, 3) > > In [20]: x > Out[20]: array([ 0. , 0.5, 1. ]) > > In [21]: y = linspace(1, 2.5, 4) > > In [22]: y > Out[22]: array([ 1. , 1.5, 2. , 2.5]) > > In [23]: z = linspace(3, 5, 5) > > In [24]: z > Out[24]: array([ 3. , 3.5, 4. , 4.5, 5. ]) > > In [25]: ix, iy, iz = indices([len(x), len(y), len(z)]) > > In [26]: x[ix] > Out[26]: > array([[[ 0. , 0. , 0. , 0. , 0. ], > [ 0. , 0. , 0. , 0. , 0. ], > [ 0. , 0. , 0. , 0. , 0. ], > [ 0. , 0. , 0. , 0. , 0. ]], > > [[ 0.5, 0.5, 0.5, 0.5, 0.5], > [ 0.5, 0.5, 0.5, 0.5, 0.5], > [ 0.5, 0.5, 0.5, 0.5, 0.5], > [ 0.5, 0.5, 0.5, 0.5, 0.5]], > > [[ 1. , 1. , 1. , 1. , 1. ], > [ 1. , 1. , 1. , 1. , 1. ], > [ 1. , 1. , 1. , 1. , 1. ], > [ 1. , 1. , 1. , 1. , 1. ]]]) > > In [27]: y[iy] > Out[27]: > array([[[ 1. , 1. , 1. , 1. , 1. ], > [ 1.5, 1.5, 1.5, 1.5, 1.5], > [ 2. , 2. , 2. , 2. , 2. ], > [ 2.5, 2.5, 2.5, 2.5, 2.5]], > > [[ 1. , 1. , 1. , 1. , 1. ], > [ 1.5, 1.5, 1.5, 1.5, 1.5], > [ 2. , 2. , 2. , 2. , 2. ], > [ 2.5, 2.5, 2.5, 2.5, 2.5]], > > [[ 1. , 1. , 1. , 1. , 1. ], > [ 1.5, 1.5, 1.5, 1.5, 1.5], > [ 2. , 2. , 2. , 2. , 2. ], > [ 2.5, 2.5, 2.5, 2.5, 2.5]]]) > > In [28]: z[iz] > Out[28]: > array([[[ 3. , 3.5, 4. , 4.5, 5. ], > [ 3. , 3.5, 4. , 4.5, 5. ], > [ 3. , 3.5, 4. , 4.5, 5. ], > [ 3. , 3.5, 4. , 4.5, 5. ]], > > [[ 3. , 3.5, 4. , 4.5, 5. ], > [ 3. , 3.5, 4. , 4.5, 5. ], > [ 3. , 3.5, 4. , 4.5, 5. ], > [ 3. , 3.5, 4. , 4.5, 5. ]], > > [[ 3. , 3.5, 4. , 4.5, 5. ], > [ 3. , 3.5, 4. , 4.5, 5. ], > [ 3. , 3.5, 4. , 4.5, 5. ], > [ 3. , 3.5, 4. , 4.5, 5. ]]]) > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Fri Sep 19 15:19:09 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 19 Sep 2008 15:19:09 -0400 Subject: [Numpy-discussion] Generating random samples without repeats In-Reply-To: References: <3d375d730809181500p225e42b0u6bf92645141b35d5@mail.gmail.com> Message-ID: 2008/9/19 Paul Moore : > Anne Archibald gmail.com> writes: > >> This was discussed on one of the mailing lists several months ago. It >> turns out that there is no simple way to efficiently choose without >> replacement in numpy/scipy. > > That reassures me that I'm not missing something obvious! I'm pretty new with > numpy (I've lurked here for a number of years, but never had a real-life need > to use numpy until now). > >> I posted a hack that does this somewhat >> efficiently (if SAMPLESIZE>M/2, choose the first SAMPLESIZE of a >> permutation; if SAMPLESIZE> duplicates) but it's not vectorized across many sample sets. Is your >> problem large M or large N? what is SAMPLESIZE/M? > > It's actually large SAMPLESIZE. As an example, I'm simulating repeated deals > of poker hands from a deck of cards: M=52, N=5, SAMPLESIZE=1000000. Er. I must have got my variable names wrong. Anyway, since you are choosing 5 from 52, repeats should be quite rare, and generating a (52,1000000) temporary array is expensive. So I would generate a (5,1000000) array with replacement, then use sort and diff (or unique) to look for any duplicates along the short axis; then fill those in with random values drawn with replacement. Repeat until there aren't any more duplicates. This is roughly the approach my code took, but my code did it for a single draw, and iterating it a million times will be expensive. If you redraw for the whole large array at once, it should be fairly cheap. The only cost will be re-sorting and re-diffing all those already-sorted hands that didn't need replacement. Even there you should be able to speed things with judicious use of fancy indexing. Anne P.S. here's an implementation, seems to work all right: def choose_without_replacement(m,n,repeats=None): """Choose n nonnegative integers less than m without replacement Returns an array of shape n, or (n,repeats). """ if repeats is None: r = 1 else: r = repeats if n>m: raise ValueError, "Cannot find %d nonnegative integers less than %d" % (n,m) if n>m/2: res = np.sort(np.random.rand(m,r).argsort(axis=0)[:n,:],axis=0) else: res = np.random.random_integers(m,size=(n,r)) while True: res = np.sort(res,axis=0) w = np.nonzero(np.diff(res,axis=0)==0) nr = len(w[0]) if nr==0: break res[w] = np.random.random_integers(m,size=nr) if repeats is None: return res[:,0] else: return res For really large values of repeats it does too much sorting; I didn't have the energy to make it pull all the ones with repeats to the beginning so that only they need to be re-sorted the next time through. Still, the expected number of trips through the while loop grows only logarithmically with repeats, so it shouldn't be too bad. -A From robert.kern at gmail.com Fri Sep 19 15:20:06 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 19 Sep 2008 14:20:06 -0500 Subject: [Numpy-discussion] Understanding mgrid In-Reply-To: References: <3d375d730809191122n1699c066h183d84cc8b6769b3@mail.gmail.com> Message-ID: <3d375d730809191220r2e3a29f6pa577d7f45476fae6@mail.gmail.com> On Fri, Sep 19, 2008 at 14:13, Brad Malone wrote: > Thanks for the response Robert. > > So, at least in this case, the results of mgrid (or indices) only provides > information about the spacing of the grid and not on the absolute value of > the point coordinates? No, they give indices. You can use those indices in a variety of ways. In my example, I used them to index into vectors which gave the absolute positions of the grid lines. That turned into bricks giving the absolute coordinates for each 3D grid point. > In your example, is there a way to see within your x[ix], y[iy], and z[iz] > matrices the same collection of points that you would see if you did > something like the following? > > points=[] > x=linspace(0,1,3) > y=linspace(1,2.5,4) > z=linspace(3,5,5) > for k in z.tolist(): > for j in y.tolist(): > for i in x.tolist(): > point=array([i,j,k]) > points.append(point) points = column_stack([x[ix].flat, y[iy].flat, z[iz].flat]) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From lists at cheimes.de Fri Sep 19 16:04:19 2008 From: lists at cheimes.de (Christian Heimes) Date: Fri, 19 Sep 2008 22:04:19 +0200 Subject: [Numpy-discussion] np.nan and ``is`` In-Reply-To: <91C39661-D647-4B11-B752-D48B3CE5FF0F@dalkescientific.com> References: <48D3DA8D.70404@american.edu> <48D3E6D1.8060704@noaa.gov> <91C39661-D647-4B11-B752-D48B3CE5FF0F@dalkescientific.com> Message-ID: Andrew Dalke wrote: > There are a few things that Python-the-language guarantees are singleton > objects which can be compared correctly with "is". Those are: > > True, False, None The empty tuple () and all interned strings are also guaranteed to be singletons. String interning is used to optimize code on C level. It's much faster to compare memory addresses than objects. All strings can be interned through the builtin function intern like s = intern(s). For Python 3.x the function was moved in the the sys module and changed to support str which are PyUnicode objects. > So, back to NaN. There's no guarantee NaN is a singleton > object, so testing with "is" almost certainly is wrong. > In fact, at the bit-level there are multiple NaNs. A > NaN (according to Wikipedia) fits the following bit pattern. > > NaN: x11111111axxxxxxxxxxxxxxxxxxxxxx. x = undefined. If a = 1, > > it is a quiet NaN, otherwise it is a signalling NaN. The definition is correct for all doubles on IEEE 754 aware platforms. Python's float type uses the double C type. Almost all modern computers have either hardware IEEE 754 support or software support for embedded devices (some mobile phones and PDAs). http://en.wikipedia.org/wiki/IEEE_754-1985 The Python core makes no difference between quiet NaNs and signaling NaNs. Only errno, input and output values are checked to raise an exception. We were discussion the possibility of a NaN singleton during our revamp of Python's IEEE 754 and math support for Python 2.6 and 3.0. But we decided against it because the extra code and cost wasn't worth the risks. Instead I added isnan() and isinf() to the math module. All checks for NaN, inf and the sign bit of a float must be made through the appropriate APIs - either the NumPy API or the new APIs for floats. Hope to shed some light on things Christian From aisaac at american.edu Fri Sep 19 16:28:34 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 19 Sep 2008 16:28:34 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <200809191146.37228.pgmdevlist@gmail.com> References: <48D23B35.8040902@dcs.gla.ac.uk> <96A799E8-0C2D-4DAB-8209-28A78B4B8D9A@sun.ac.za> <48D3C6F1.2060500@american.edu> <200809191146.37228.pgmdevlist@gmail.com> Message-ID: <48D40B72.5030205@american.edu> On 9/19/2008 11:46 AM Pierre GM apparently wrote: >>>> a.mask=True This is great, but is apparently new behavior as of NumPy 1.2? Alan Isaac From aisaac at american.edu Fri Sep 19 16:35:23 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 19 Sep 2008 16:35:23 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <200809190454.40685.pgmdevlist@gmail.com> References: <48D23B35.8040902@dcs.gla.ac.uk> <200809190432.23535.pgmdevlist@gmail.com> <48D3636A.6070908@ar.media.kyoto-u.ac.jp> <200809190454.40685.pgmdevlist@gmail.com> Message-ID: <48D40D0B.6030603@american.edu> On 9/19/2008 4:54 AM Pierre GM apparently wrote: > I know. I was more dreading the time when MaskedArrays would have to be ported > to C. In a way, that would probably simplify a few issues. OTOH, I don't > really see it happening any time soon. Is this possibly a GSoC sized project? Alan Isaac From pgmdevlist at gmail.com Fri Sep 19 16:54:05 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 19 Sep 2008 16:54:05 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D40B72.5030205@american.edu> References: <48D23B35.8040902@dcs.gla.ac.uk> <200809191146.37228.pgmdevlist@gmail.com> <48D40B72.5030205@american.edu> Message-ID: <200809191654.06438.pgmdevlist@gmail.com> On Friday 19 September 2008 16:28:34 Alan G Isaac wrote: > On 9/19/2008 11:46 AM Pierre GM apparently wrote: > >>>> a.mask=True > This is great, but is apparently > new behavior as of NumPy 1.2? I'm not sure, sorry. Another way is ma.array(np.empty(yourshape,yourdtype), mask=True) which should work with earlier versions. From pgmdevlist at gmail.com Fri Sep 19 17:02:17 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 19 Sep 2008 17:02:17 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D40D0B.6030603@american.edu> References: <48D23B35.8040902@dcs.gla.ac.uk> <200809190454.40685.pgmdevlist@gmail.com> <48D40D0B.6030603@american.edu> Message-ID: <200809191702.17204.pgmdevlist@gmail.com> On Friday 19 September 2008 16:35:23 Alan G Isaac wrote: > On 9/19/2008 4:54 AM Pierre GM apparently wrote: > > I know. I was more dreading the time when MaskedArrays would have to be > > ported to C. In a way, that would probably simplify a few issues. OTOH, I > > don't really see it happening any time soon. > > Is this possibly a GSoC sized project? > Alan Isaac If we can find someone who knows C and masked arrays well, that could be. From aisaac at american.edu Fri Sep 19 17:25:53 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 19 Sep 2008 17:25:53 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <200809191654.06438.pgmdevlist@gmail.com> References: <48D23B35.8040902@dcs.gla.ac.uk> <200809191146.37228.pgmdevlist@gmail.com> <48D40B72.5030205@american.edu> <200809191654.06438.pgmdevlist@gmail.com> Message-ID: <48D418E1.1030905@american.edu> On 9/19/2008 4:54 PM Pierre GM apparently wrote: > Another way is > ma.array(np.empty(yourshape,yourdtype), mask=True) > which should work with earlier versions. Seems like ``mask`` would be a natural keyword for ``ma.empty``? Thanks, Alan Isaac From pgmdevlist at gmail.com Fri Sep 19 17:40:53 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 19 Sep 2008 17:40:53 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D418E1.1030905@american.edu> References: <48D23B35.8040902@dcs.gla.ac.uk> <200809191654.06438.pgmdevlist@gmail.com> <48D418E1.1030905@american.edu> Message-ID: <200809191740.54031.pgmdevlist@gmail.com> On Friday 19 September 2008 17:25:53 Alan G Isaac wrote: > On 9/19/2008 4:54 PM Pierre GM apparently wrote: > > Another way is > > ma.array(np.empty(yourshape,yourdtype), mask=True) > > which should work with earlier versions. > > Seems like ``mask`` would be a natural > keyword for ``ma.empty``? Not a bad idea. I'll plug that in. From paultaney at yahoo.com Fri Sep 19 20:47:12 2008 From: paultaney at yahoo.com (paul taney) Date: Fri, 19 Sep 2008 17:47:12 -0700 (PDT) Subject: [Numpy-discussion] newb question In-Reply-To: <200809191740.54031.pgmdevlist@gmail.com> Message-ID: <187177.1286.qm@web53911.mail.re2.yahoo.com> Hi, What am I doing wrong here? The reshape doesn"t take. % cat test1.py import numpy as np a = np.uint8([39, 39, 231, 239, 39, 231, 39, 39, 231, 39, 39, 231, 239, 39, 231, 39, 39, 231, 39, 39, 231, 239, 39, 231, 39, 39, 231, 39, 39, 231, 239, 39, 231, 39, 39, 231,]) a.reshape(3, 4, 3) print "a = %r" % (a) % % python test1.py a = array([ 39, 39, 231, 239, 39, 231, 39, 39, 231, 39, 39, 231, 239, 39, 231, 39, 39, 231, 39, 39, 231, 239, 39, 231, 39, 39, 231, 39, 39, 231, 239, 39, 231, 39, 39, 231], dtype=uint8) ---- I am expecting: a = array([[[39, 39, 231], [239, 39, 231], [39, 39, 231]], [[39, 39, 231], [239, 39, 231], [39, 39, 231]], [[39, 39, 231], [239, 39, 231], [39, 39, 231]], [[39, 39, 231], [239, 39, 231], [39, 39, 231]]], \ dtype=np.uint8) paul ---- def vanderWalt(a, f): """thanks Stefan""" RED, GRN, BLU = 0, 1, 2 bluemask = (a[...,BLU] > f*a[...,GRN]) & \ (a[...,BLU] > f*a[...,RED]) return np.array(bluemask.nonzero()).swapaxes(0,1) From efiring at hawaii.edu Fri Sep 19 20:59:17 2008 From: efiring at hawaii.edu (Eric Firing) Date: Fri, 19 Sep 2008 14:59:17 -1000 Subject: [Numpy-discussion] newb question In-Reply-To: <187177.1286.qm@web53911.mail.re2.yahoo.com> References: <187177.1286.qm@web53911.mail.re2.yahoo.com> Message-ID: <48D44AE5.2040508@hawaii.edu> paul taney wrote: > Hi, > > What am I doing wrong here? The reshape doesn"t take. Reshape does not act in place, it returns either a new view or a copy. To reshape in place, you can assign to the shape attribute: In [13]:a = np.arange(10) In [14]:a.shape = (2,5) In [15]:a Out[15]: array([[0, 1, 2, 3, 4], [5, 6, 7, 8, 9]]) Eric > > % cat test1.py > import numpy as np > > a = np.uint8([39, 39, 231, 239, 39, 231, 39, 39, 231, > 39, 39, 231, 239, 39, 231, 39, 39, 231, > 39, 39, 231, 239, 39, 231, 39, 39, 231, > 39, 39, 231, 239, 39, 231, 39, 39, 231,]) > a.reshape(3, 4, 3) > print "a = %r" % (a) > % > % python test1.py > a = array([ 39, 39, 231, 239, 39, 231, 39, 39, 231, > 39, 39, 231, 239, 39, 231, 39, 39, 231, > 39, 39, 231, 239, 39, 231, 39, 39, 231, > 39, 39, 231, 239, 39, 231, 39, 39, 231], dtype=uint8) > > ---- > > I am expecting: > > a = array([[[39, 39, 231], [239, 39, 231], [39, 39, 231]], > [[39, 39, 231], [239, 39, 231], [39, 39, 231]], > [[39, 39, 231], [239, 39, 231], [39, 39, 231]], > [[39, 39, 231], [239, 39, 231], [39, 39, 231]]], \ > dtype=np.uint8) > > > paul > > ---- > > def vanderWalt(a, f): > """thanks Stefan""" > RED, GRN, BLU = 0, 1, 2 > bluemask = (a[...,BLU] > f*a[...,GRN]) & \ > (a[...,BLU] > f*a[...,RED]) > return np.array(bluemask.nonzero()).swapaxes(0,1) > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From pgmdevlist at gmail.com Fri Sep 19 20:54:34 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 19 Sep 2008 20:54:34 -0400 Subject: [Numpy-discussion] newb question In-Reply-To: <187177.1286.qm@web53911.mail.re2.yahoo.com> References: <187177.1286.qm@web53911.mail.re2.yahoo.com> Message-ID: <200809192054.34903.pgmdevlist@gmail.com> On Friday 19 September 2008 20:47:12 paul taney wrote: > Hi, > > What am I doing wrong here? The reshape doesn"t take. help(reshape) a.reshape(shape, order='C') Returns an array containing the data of a, but with a new shape. Refer to `numpy.reshape` for full documentation. You see that you're not modifying in place. Instead, you should use a.shape = (3,4,3) Play with the tuple to find what you want -- (4,3,3) seems to meet your expectations. From dalke at dalkescientific.com Fri Sep 19 22:51:44 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Sat, 20 Sep 2008 04:51:44 +0200 Subject: [Numpy-discussion] np.nan and ``is`` In-Reply-To: References: <48D3DA8D.70404@american.edu> <48D3E6D1.8060704@noaa.gov> <91C39661-D647-4B11-B752-D48B3CE5FF0F@dalkescientific.com> Message-ID: <2CADA6FF-DCF9-4B43-9E13-146FD5F4C3D3@dalkescientific.com> On Sep 19, 2008, at 10:04 PM, Christian Heimes wrote: > Andrew Dalke wrote: >> There are a few things that Python-the-language guarantees are >> singleton >> objects which can be compared correctly with "is". > The empty tuple () and all interned strings are also guaranteed to be > singletons. Where's the guarantee? As far as I know it's not part of Python-the-language, and I thought it was only an implementation detail of CPython. tupleobject.c says: PyTuple_Fini(void) { #if PyTuple_MAXSAVESIZE > 0 /* empty tuples are used all over the place and applications may * rely on the fact that an empty tuple is a singleton. */ Py_XDECREF(free_list[0]); free_list[0] = NULL; (void)PyTuple_ClearFreeList(); #endif } but that doesn't hold under Jython 2.2a1: Jython 2.2a1 on java1.4.2_16 (JIT: null) Type "copyright", "credits" or "license" for more information. >>> () is () 0 >>> 1 is 1 1 > String interning is used to optimize code on C level. It's > much faster to compare memory addresses than objects. All strings > can be > interned through the builtin function intern like s = intern(s). For > Python 3.x the function was moved in the the sys module and changed to > support str which are PyUnicode objects. "intern" being listed in the documentation under http://docs.python.org/lib/non-essential-built-in-funcs.html 2.2 Non-essential Built-in Functions There are several built-in functions that are no longer essential to learn, know or use in modern Python programming. They have been kept here to maintain backwards compatibility with programs written for older versions of Python. Again, I think this is only an aspect of the CPython implementation. > The Python core makes no difference between quiet NaNs and signaling > NaNs. Based on my limited readings just now, it seems that that's the general consensus: http://www.open-std.org/jtc1/sc22/wg14/www/docs/n965.htm """Standard C only adopted Quiet NaNs. It did not adopt Signaling NaNs because it was believed that they are of too limited utility for the amount of work required.""" http://www.digitalmars.com/d/archives/digitalmars/D/ signaling_NaNs_and_quiet_NaNs_75844.html "Signaling NaNs have fallen out of favor. No exceptions get raised for them." http://en.wikipedia.org/wiki/NaN """There were questions about if signalling NaNs should continue to be required in the revised standard. In the end it appears they will be left in.""" > We were discussion the possibility of a NaN singleton during > our revamp of Python's IEEE 754 and math support for Python 2.6 and > 3.0. > But we decided against it because the extra code and cost wasn't worth > the risks. Instead I added isnan() and isinf() to the math module. I couldn't find that thread. What are the advantages of converting all NaNs to a singleton? All I can come up with are disadvantages. BTW, another place to look is the Decimal module >>> import decimal >>> decimal.Decimal("nan") Decimal("NaN") >>> Looking at the decimal docs now I see a "canonical()" method which The result has the same value as the operand but always uses a canonical encoding. The definition of canonical is implementation-defined; if more than one internal encoding for a given NaN, Infinity, or finite number is possible then one ?preferred? encoding is deemed canonical. This operation then returns the value using that preferred encoding. Andrew dalke at dalkescientific.com From david at ar.media.kyoto-u.ac.jp Fri Sep 19 23:25:50 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 20 Sep 2008 12:25:50 +0900 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <9457e7c80809190555k7de9e49cjb2325f70b0c7693e@mail.gmail.com> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <9457e7c80809190210t44025279y57e7ea916fe232d5@mail.gmail.com> <48D36EF8.9030901@ar.media.kyoto-u.ac.jp> <9457e7c80809190555k7de9e49cjb2325f70b0c7693e@mail.gmail.com> Message-ID: <48D46D3E.4060801@ar.media.kyoto-u.ac.jp> St?fan van der Walt wrote: > > Why shouldn't we have "nanmin"-like behaviour for the C min itself? > Ah, I was not arguing we should not do it in C, but rather we did not have to do in C. The current behavior for nan with functions relying on ordering is broken; if someone prefer fixing it in C, great. But I was guessing more people could fix it using python, that's all. I opened a bug for min/max and nan, this should be fixed for 1.3.0, maybe 1.2.1 too. cheers, David From robert.kern at gmail.com Fri Sep 19 23:50:41 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 19 Sep 2008 22:50:41 -0500 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D46D3E.4060801@ar.media.kyoto-u.ac.jp> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <9457e7c80809190210t44025279y57e7ea916fe232d5@mail.gmail.com> <48D36EF8.9030901@ar.media.kyoto-u.ac.jp> <9457e7c80809190555k7de9e49cjb2325f70b0c7693e@mail.gmail.com> <48D46D3E.4060801@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730809192050q7a39c1cbr1da29655bc1cd679@mail.gmail.com> On Fri, Sep 19, 2008 at 22:25, David Cournapeau wrote: > St?fan van der Walt wrote: >> >> Why shouldn't we have "nanmin"-like behaviour for the C min itself? >> > > Ah, I was not arguing we should not do it in C, but rather we did not > have to do in C. The current behavior for nan with functions relying on > ordering is broken; if someone prefer fixing it in C, great. But I was > guessing more people could fix it using python, that's all. How, exactly? ndarray.min() is the where the implementation is. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Fri Sep 19 23:44:00 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 20 Sep 2008 12:44:00 +0900 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D3B23C.8030100@american.edu> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <48D3B23C.8030100@american.edu> Message-ID: <48D47180.3020101@ar.media.kyoto-u.ac.jp> Alan G Isaac wrote: > On 9/19/2008 4:35 AM David Cournapeau apparently wrote: >> I never use NaN as missing value > > What do you use? > > Recently I needed to fill a 2d array with values > from computations that could "go wrong". > I created an array of NaN and then replaced > the elements where the computation produced > a useful value. I then applied ``nanmax``, > to get the maximum of the useful values. > > What should I have done? I guess my formulation was poor: I never use NaN as missing values because I never use missing values, which is why I wanted the opinion of people who use NaN in a different manner (because I don't have a good idea on how those people would like to see numpy behave). I was certainly not arguing they should not be use for the purpose of missing value. The problem with NaN is that you cannot mix the missing value behavior and the error behavior. Dealing with them in a consistent manner is difficult. Because numpy is a general numerical computation tool, I think that NaN should be propagated and never ignored *by default*. If you have NaN because of divide by 0, etc... it should not be ignored at all. But if you want it to ignore, then numpy should make it possible: - max, min: should return NaN if NaN is in the array, or maybe even fail ? - argmax, argmin ? - sort: should fail ? - mean, std, variance: should return Nan - median: should fail (to be consistent if sort fails) ? Should return NaN ? We could then add an argument to failing functions to tell them either to ignore NaN/put them at some special location (like R does, for example). The ones I am not sure are median and argmax/argmin. For median, failing when sort does is consistent; but this can break a lot of code. For argmin/argmax, failing is the most logical, but OTOH, making argmin/argmax failing and not max/min is not consistent either. Breaking the code is maybe not that bad because currently, neither max/min nor argmax/argmin nor sort does return a meaningful function. Does that sound reasonable to you ? cheer, David From david at ar.media.kyoto-u.ac.jp Sat Sep 20 00:08:13 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 20 Sep 2008 13:08:13 +0900 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <3d375d730809192050q7a39c1cbr1da29655bc1cd679@mail.gmail.com> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <9457e7c80809190210t44025279y57e7ea916fe232d5@mail.gmail.com> <48D36EF8.9030901@ar.media.kyoto-u.ac.jp> <9457e7c80809190555k7de9e49cjb2325f70b0c7693e@mail.gmail.com> <48D46D3E.4060801@ar.media.kyoto-u.ac.jp> <3d375d730809192050q7a39c1cbr1da29655bc1cd679@mail.gmail.com> Message-ID: <48D4772D.2030208@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > On Fri, Sep 19, 2008 at 22:25, David Cournapeau > wrote: > > > How, exactly? ndarray.min() is the where the implementation is. > Ah, I keep forgetting those are implemented in the array object, sorry for that. Now I understand Stefan point. Do I understand correctly that we should then do: - implement a min/max NaN aware for every float type (real and complex) in umathmodule.c, which ignores nan (called @TYPE at _nanmin, etc...) - fix the current min/max to propagate NaN instead of giving broken result - How to do the dispatching ? Having PyArray_Min and PyArray_NanMin sounds the easiest (we don't change any C api, only add an argument to the python-callable function min, in array_min method ?) Or am I missing something ? If this is the right way to fix it I am willing to do it (we still have to agree on the default behavior first). I am not really familiar with sort module, but maybe it is really similar to min/max case. cheers, David From peridot.faceted at gmail.com Sat Sep 20 01:02:16 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sat, 20 Sep 2008 01:02:16 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D47180.3020101@ar.media.kyoto-u.ac.jp> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <48D3B23C.8030100@american.edu> <48D47180.3020101@ar.media.kyoto-u.ac.jp> Message-ID: 2008/9/19 David Cournapeau : > I guess my formulation was poor: I never use NaN as missing values > because I never use missing values, which is why I wanted the opinion of > people who use NaN in a different manner (because I don't have a good > idea on how those people would like to see numpy behave). I was > certainly not arguing they should not be use for the purpose of missing > value. I, on the other hand, was making specifically that suggestion: users should not use nans to indicate missing values. Users should use masked arrays to indicate missing values. > The problem with NaN is that you cannot mix the missing value behavior > and the error behavior. Dealing with them in a consistent manner is > difficult. Because numpy is a general numerical computation tool, I > think that NaN should be propagated and never ignored *by default*. If > you have NaN because of divide by 0, etc... it should not be ignored at > all. But if you want it to ignore, then numpy should make it possible: > > - max, min: should return NaN if NaN is in the array, or maybe even > fail ? > - argmax, argmin ? > - sort: should fail ? > - mean, std, variance: should return Nan > - median: should fail (to be consistent if sort fails) ? Should > return NaN ? This part I pretty much agree with. > We could then add an argument to failing functions to tell them either > to ignore NaN/put them at some special location (like R does, for > example). The ones I am not sure are median and argmax/argmin. For > median, failing when sort does is consistent; but this can break a lot > of code. For argmin/argmax, failing is the most logical, but OTOH, > making argmin/argmax failing and not max/min is not consistent either. > Breaking the code is maybe not that bad because currently, neither > max/min nor argmax/argmin nor sort does return a meaningful function. > Does that sound reasonable to you ? The problem with this approach is that all those decisions need to be made and all that code needs to be implemented for masked arrays. In fact I suspect that it has already been done in that case. So really what you are suggesting here is that we duplicate all this effort to implement the same functions for nans as we have for masked arrays. It's important, too, that the masked array implementation and the nan implementation behave the same way, or users will become badly confused. Who gets the task of keeping the two implementations in sync? The current situation is that numpy has two ways to indicate bad data for floating-point arrays: nans and masked arrays. We can't get rid of either: nans appear on their own, and masked arrays are the only way to mark bad data in non-floating-point arrays. We can try to make them behave the same, which will be a lot of work to provide redundant capabilities. Or we can make them behave drastically differently. Masked arrays clearly need to be able to handle masked values flexibly and explicitly. So I think nans should be handled simply and conservatively: propagate them if possible, raise if not. If users are concerned about performance, it's worth noting that on some machines nans force a fallback to software floating-point handling, with a corresponding very large performance hit. This includes some but not all x86 (and I think x86-64) CPUs. How this compares to the performance of masked arrays is not clear. Anne From dblubaugh at belcan.com Sat Sep 20 01:07:20 2008 From: dblubaugh at belcan.com (Blubaugh, David A.) Date: Sat, 20 Sep 2008 01:07:20 -0400 Subject: [Numpy-discussion] NEW GENERATED DLL ERROR FOUND WITHIN f2PY.py References: <27CC3060AF71DA40A5DC85F7D5B70F3802840466@AWMAIL04.belcan.com> Message-ID: <27CC3060AF71DA40A5DC85F7D5B70F38051C09EC@AWMAIL04.belcan.com> To All, I have now been able to generate a .pyd file from a FORTRAN file that I am trying to interface with python. I was able to execute this with an additional insight into how f2py operates. It seems as though the documentation requires an upgrade, since there appears to be missing information that might misdirect a f2py newcomer, such as myself. However, I am now facing the following new error: ImportError: DLL load with error code 193 The python script is as follows: import hello print hello.__doc__ print hello.foo.__doc__ hello.foo(4) The Fortran code is as follows: ! -*- f90 -*- subroutine foo(a) integer a print*, "Hello from Fortran!" print*, "a=", a end I was wondering as to what I should now try in order to finally produce a python sending and receiving information from a FORTRAN .pyd file. Any Suggestions??? Do I have to recompile Python with mingw32 in order to finally resolve this issue?? Thanks, David Blubaugh This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated. From david at ar.media.kyoto-u.ac.jp Sat Sep 20 01:41:25 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 20 Sep 2008 14:41:25 +0900 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <48D3B23C.8030100@american.edu> <48D47180.3020101@ar.media.kyoto-u.ac.jp> Message-ID: <48D48D05.1020401@ar.media.kyoto-u.ac.jp> Anne Archibald wrote: > > I, on the other hand, was making specifically that suggestion: users > should not use nans to indicate missing values. Users should use > masked arrays to indicate missing values. I agree it is the nicest solution in theory, but I think it is impractical (as mentioned by Eric Firing in his email). > > This part I pretty much agree with. I can't really see which one is better (failing or returning NaN for sort/min/max and their sort counterpat), or if we should let the choice be left to the user. I am fine with both, and they both require the same amount of work. > Or we can make them behave drastically differently. > Masked arrays clearly need to be able to handle masked values flexibly > and explicitly. So I think nans should be handled simply and > conservatively: propagate them if possible, raise if not. I agree about this behavior being the default. I just think that for a couple of functions, we could we give either separate functions, or additional arguments to existing functions to ignore them: I am thinking about min/max/sort and their arg* counterpart, because those are really basic, and because we already have nanmean/nanstd/nanmedian (e.g. having a nansort would help for nanmean to be much faster). > > If users are concerned about performance, it's worth noting that on > some machines nans force a fallback to software floating-point > handling, with a corresponding very large performance hit. I was more concerned with the cost of treating NaN when you do not have NaN in your array when you have to treat for NaN explicitely (everything involving comparison). But I don't see any obvious way to avoid that cost, David From charlesr.harris at gmail.com Sat Sep 20 02:15:50 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 20 Sep 2008 00:15:50 -0600 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D48D05.1020401@ar.media.kyoto-u.ac.jp> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <48D3B23C.8030100@american.edu> <48D47180.3020101@ar.media.kyoto-u.ac.jp> <48D48D05.1020401@ar.media.kyoto-u.ac.jp> Message-ID: On Fri, Sep 19, 2008 at 11:41 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Anne Archibald wrote: > > > > I, on the other hand, was making specifically that suggestion: users > > should not use nans to indicate missing values. Users should use > > masked arrays to indicate missing values. > > I agree it is the nicest solution in theory, but I think it is > impractical (as mentioned by Eric Firing in his email). > > > > > This part I pretty much agree with. > > I can't really see which one is better (failing or returning NaN for > sort/min/max and their sort counterpat), or if we should let the choice > be left to the user. I am fine with both, and they both require the same > amount of work. > > > Or we can make them behave drastically differently. > > Masked arrays clearly need to be able to handle masked values flexibly > > and explicitly. So I think nans should be handled simply and > > conservatively: propagate them if possible, raise if not. > > I agree about this behavior being the default. I just think that for a > couple of functions, we could we give either separate functions, or > additional arguments to existing functions to ignore them: I am thinking > about min/max/sort and their arg* counterpart, because those are really > basic, and because we already have nanmean/nanstd/nanmedian (e.g. having > a nansort would help for nanmean to be much faster). > I would be happy to implement nan sorts if someone can provide me with a portable and easy way to detect nans for single, double, and long double floats. And not have it fail if the architecture doesn't support nans. I think getting all the needed nan detection and setup in place is the first step for anything else. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Sep 20 02:24:20 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 20 Sep 2008 01:24:20 -0500 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: References: <48D23B35.8040902@dcs.gla.ac.uk> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <48D3B23C.8030100@american.edu> <48D47180.3020101@ar.media.kyoto-u.ac.jp> <48D48D05.1020401@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730809192324p64dd1816g4676a975ada99c78@mail.gmail.com> On Sat, Sep 20, 2008 at 01:15, Charles R Harris wrote: > I would be happy to implement nan sorts if someone can provide me with a > portable and easy way to detect nans for single, double, and long double > floats. And not have it fail if the architecture doesn't support nans. I > think getting all the needed nan detection and setup in place is the first > step for anything else. We explicitly only support IEEE-754 architectures, so we are always on an architecture that supports NaNs. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From peridot.faceted at gmail.com Sat Sep 20 02:51:26 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sat, 20 Sep 2008 02:51:26 -0400 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D3E8E9.1080406@hawaii.edu> References: <48D23B35.8040902@dcs.gla.ac.uk> <96A799E8-0C2D-4DAB-8209-28A78B4B8D9A@sun.ac.za> <48D3C6F1.2060500@american.edu> <200809191146.37228.pgmdevlist@gmail.com> <48D3E8E9.1080406@hawaii.edu> Message-ID: 2008/9/19 Eric Firing : > Pierre GM wrote: > >>> It seems to me that there are pragmatic reasons >>> why people work with NaNs for missing values, >>> that perhaps shd not be dismissed so quickly. >>> But maybe I am overlooking a simple solution. >> >> nansomething solutions tend to be considerably faster, that might be one >> reason. A lack of visibility of numpy.ma could be a second. In any case, I >> can't but agree with other posters: a NaN in an array usually means something >> went astray. > > Additional reasons for using nans: > > 1) years of experience with Matlab, in which using nan for missing > values is the standard idiom. Users are already retraining to use zero-based indexing; I don't think asking them to use a full-featured masked array package is an unreasonable retraining burden, particularly since this idiom breaks as soon as they want to work with arrays of integers or records. > 2) convenient interfacing with extension code in C or C++. > > The latter is a factor in the present use of nan in matplotlib; using > nan for missing values in an array passed into extension code saves > having to pass and process a second (mask) array. It is fast and simple. How hard is it to pass an array where the masked values have been filled with nans? It's certainly easy to go the other way (mask all nans). I think this is less painful than supporting two differently-featured sets of functions for dealing with arrays containing some invalid values. Anne From david at ar.media.kyoto-u.ac.jp Sat Sep 20 03:13:36 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 20 Sep 2008 16:13:36 +0900 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <48D3B23C.8030100@american.edu> <48D47180.3020101@ar.media.kyoto-u.ac.jp> <48D48D05.1020401@ar.media.kyoto-u.ac.jp> Message-ID: <48D4A2A0.4090405@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > > > I would be happy to implement nan sorts if someone can provide me with > a portable and easy way to detect nans for single, double, and long > double floats. And not have it fail if the architecture doesn't > support nans. I think getting all the needed nan detection and setup > in place is the first step for anything else. I guess you mean when isnan is available but broken, since we do not support platforms without IEEE 754 support ? I want to take care of this for my umathmodule cleaning (all the configuration checks/replacements are in place; if we want to be paranoid, we could check whether isnan works for all types if found on the system). cheers, David From david at ar.media.kyoto-u.ac.jp Sat Sep 20 03:25:00 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 20 Sep 2008 16:25:00 +0900 Subject: [Numpy-discussion] NEW GENERATED DLL ERROR FOUND WITHIN f2PY.py In-Reply-To: <27CC3060AF71DA40A5DC85F7D5B70F38051C09EC@AWMAIL04.belcan.com> References: <27CC3060AF71DA40A5DC85F7D5B70F3802840466@AWMAIL04.belcan.com> <27CC3060AF71DA40A5DC85F7D5B70F38051C09EC@AWMAIL04.belcan.com> Message-ID: <48D4A54C.6060303@ar.media.kyoto-u.ac.jp> Blubaugh, David A. wrote: > ImportError: DLL load with error code 193 > Likely to be a build error: how did you build the .pyd file ? cheers, David From dblubaugh at belcan.com Sat Sep 20 13:40:56 2008 From: dblubaugh at belcan.com (Blubaugh, David A.) Date: Sat, 20 Sep 2008 13:40:56 -0400 Subject: [Numpy-discussion] NEW GENERATED DLL ERROR FOUND WITHIN Message-ID: <27CC3060AF71DA40A5DC85F7D5B70F38051C09EE@AWMAIL04.belcan.com> David, Thanks for your reply. This is as to how I developed my .ptd file. I entered the following commands within this MS-DOS prompt within Windows XP: C:\python25\Scripts> C:\python25\python f2py.py -c --fcompiler=gnu95 --compiler=mingw32 -m hello hello.f90 I am using the gfortran compiler, that was prescribed to use on the following website: http://www.scipy.org/F2PY_Window I am also wondering if I have set my environment variables incorrectly? David, is this ms-dos the correct method to execute within MS-DOS? Thanks, Dsvid Message: 4 Date: Sat, 20 Sep 2008 16:25:00 +0900 From: David Cournapeau Subject: Re: [Numpy-discussion] NEW GENERATED DLL ERROR FOUND WITHIN f2PY.py To: Discussion of Numerical Python Message-ID: <48D4A54C.6060303 at ar.media.kyoto-u.ac.jp> Content-Type: text/plain; charset=ISO-8859-1 Blubaugh, David A. wrote: > ImportError: DLL load with error code 193 > Likely to be a build error: how did you build the .pyd file ? cheers, David This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated. From jah.mailinglist at gmail.com Sat Sep 20 14:02:13 2008 From: jah.mailinglist at gmail.com (Jake Harris) Date: Sat, 20 Sep 2008 11:02:13 -0700 Subject: [Numpy-discussion] Medians that ignore values Message-ID: (sorry for starting a new thread...I wasn't subscribed yet) St?fan van der Walt wrote the following on 09/19/2008 02:10 AM: > > So am I. In all my use cases, NaNs indicate trouble. > I can provide a use case where NaNs do not indicate trouble. In fact, they need to be treated as 0. For example, As x->0 in y(x) = x log x, it is traditional (eg in information theory) to take y(0) = 0. So if one is multiplying arrays and 0 * -inf is encountered, the desirable behavior is that we get 0. Because you're always working with probabilities, there is almost always no ambiguity...whenever NaN is encounter, 0 is what is desired. Perhaps numpy can have some method by which a user can specify how NaNis treated (in addition to ignore, raise, etc). Good idea? Bad idea? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jah.mailinglist at gmail.com Sat Sep 20 14:17:44 2008 From: jah.mailinglist at gmail.com (Jake Harris) Date: Sat, 20 Sep 2008 11:17:44 -0700 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: References: Message-ID: On Sat, Sep 20, 2008 at 11:02 AM, Jake Harris wrote: > > Because you're always working with probabilities, there is almost always no > ambiguity...whenever NaN is encounter, 0 is what is desired. > ...of course, division presents a good counterexample. > Bad idea? > So probably. -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Sun Sep 21 02:49:45 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sun, 21 Sep 2008 15:49:45 +0900 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <48D3B23C.8030100@american.edu> <48D47180.3020101@ar.media.kyoto-u.ac.jp> Message-ID: <48D5EE89.3010500@ar.media.kyoto-u.ac.jp> Anne Archibald wrote: > > If users are concerned about performance, it's worth noting that on > some machines nans force a fallback to software floating-point > handling, with a corresponding very large performance hit. This > includes some but not all x86 (and I think x86-64) CPUs. How this > compares to the performance of masked arrays is not clear. I spent some time on this. In particular, for max.min, I did the following for the core loop (always return nan if nan is in the array): /* nan + x and x + nan are nan, where x can be anything: normal, * denormal, nan, infinite */ tmp = *((@typ@ *)i1) + *((@typ@ *)i2); if(isnan(tmp)) { *((@typ@ *)op) = tmp; } else { *((@typ@ *)op)=*((@typ@ *)i1) @OP@ *((@typ@ *)i2) ? *((@typ@ *)i1) : *((@typ@ *)i2); } For large arrays (on my CPU, it is around 10000 items), the function is 3x slower than the original one. I think the main cost is the isnan. 3x is quite expensive, so I tested a bit isnan on Linux, and it is surprisingly slow. If I use my own, trivial @define isnan(x) ((x) != (x)), it is twice faster than the glibc isnan, and then max/min are as fast as before, except they are working :) The isnan thing is surprising, because the whole point to have a isnan is that you can do it without branching. I checked, and numpy does use the macro of isnan, not the function (glibc has both). cheers, David From cimrman3 at ntc.zcu.cz Sun Sep 21 03:09:25 2008 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Sun, 21 Sep 2008 09:09:25 +0200 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <3d375d730809191146k7425c569i2450e17e8d51c588@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <48D23506.7080109@ntc.zcu.cz> <3d375d730809181237x43ce1610r6dccb081d6698f5d@mail.gmail.com> <48D363DD.6040609@ntc.zcu.cz> <3d375d730809190226g361d5855o1c1475da0379706f@mail.gmail.com> <48D39456.80505@ntc.zcu.cz> <3d375d730809191146k7425c569i2450e17e8d51c588@mail.gmail.com> Message-ID: <48D5F325.7080104@ntc.zcu.cz> Robert Kern wrote: > On Fri, Sep 19, 2008 at 07:00, Robert Cimrman wrote: >> Robert Kern wrote: >>> Ah, found it. T_LONGLONG is a #define from structmember.h which is >>> used to describe the types of attributes. Apparently, this was not >>> added until Python 2.5. That particular member didn't actually need to >>> be long long, so I've fixed that. >> Great, I will try it after it appears on the web page. > > Oops! It's now pushed. Thanks, it builds ok now! But then cProfile is also new in 2.5. - I should probably upgrade and stop bothering you :-) r. From david at ar.media.kyoto-u.ac.jp Sun Sep 21 02:56:38 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sun, 21 Sep 2008 15:56:38 +0900 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D5EE89.3010500@ar.media.kyoto-u.ac.jp> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <48D3B23C.8030100@american.edu> <48D47180.3020101@ar.media.kyoto-u.ac.jp> <48D5EE89.3010500@ar.media.kyoto-u.ac.jp> Message-ID: <48D5F026.9090809@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > Anne Archibald wrote: >> If users are concerned about performance, it's worth noting that on >> some machines nans force a fallback to software floating-point >> handling, with a corresponding very large performance hit. This >> includes some but not all x86 (and I think x86-64) CPUs. How this >> compares to the performance of masked arrays is not clear. > > I spent some time on this. In particular, for max.min, I did the > following for the core loop (always return nan if nan is in the array): > > /* nan + x and x + nan are nan, where x can be anything: > normal, > * denormal, nan, infinite > */ > tmp = *((@typ@ *)i1) + *((@typ@ > *)i2); > if(isnan(tmp)) > { > *((@typ@ *)op) = > tmp; > } else > { > *((@typ@ *)op)=*((@typ@ *)i1) @OP@ *((@typ@ *)i2) ? *((@typ@ > *)i1) : *((@typ@ *)i2); > } Grr, sorry for the mangling: /* nan + x and x + nan are nan, where x can be anything: normal, * denormal, nan, infinite */ tmp = *((@typ@ *)i1) + *((@typ@*)i2); if(isnan(tmp)) { *((@typ@ *)op) = tmp; } else { *((@typ@ *)op) = *((@typ@ *)i1) @OP@ *((@typ@ *)i2) ? *((@typ@*)i1) : *((@typ@ *)i2); } cheers, David From robert.kern at gmail.com Sun Sep 21 03:17:05 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 21 Sep 2008 02:17:05 -0500 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <48D5F325.7080104@ntc.zcu.cz> References: <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <48D23506.7080109@ntc.zcu.cz> <3d375d730809181237x43ce1610r6dccb081d6698f5d@mail.gmail.com> <48D363DD.6040609@ntc.zcu.cz> <3d375d730809190226g361d5855o1c1475da0379706f@mail.gmail.com> <48D39456.80505@ntc.zcu.cz> <3d375d730809191146k7425c569i2450e17e8d51c588@mail.gmail.com> <48D5F325.7080104@ntc.zcu.cz> Message-ID: <3d375d730809210017s43030527n584792ada09044a6@mail.gmail.com> On Sun, Sep 21, 2008 at 02:09, Robert Cimrman wrote: > Robert Kern wrote: >> On Fri, Sep 19, 2008 at 07:00, Robert Cimrman wrote: >>> Robert Kern wrote: >>>> Ah, found it. T_LONGLONG is a #define from structmember.h which is >>>> used to describe the types of attributes. Apparently, this was not >>>> added until Python 2.5. That particular member didn't actually need to >>>> be long long, so I've fixed that. >>> Great, I will try it after it appears on the web page. >> >> Oops! It's now pushed. > > Thanks, it builds ok now! But then cProfile is also new in 2.5. - I > should probably upgrade and stop bothering you :-) Actually, if you pull again, I've just pushed my changes to remove the hard dependency on cProfile. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cimrman3 at ntc.zcu.cz Sun Sep 21 03:26:15 2008 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Sun, 21 Sep 2008 09:26:15 +0200 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <3d375d730809210017s43030527n584792ada09044a6@mail.gmail.com> References: <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <48D23506.7080109@ntc.zcu.cz> <3d375d730809181237x43ce1610r6dccb081d6698f5d@mail.gmail.com> <48D363DD.6040609@ntc.zcu.cz> <3d375d730809190226g361d5855o1c1475da0379706f@mail.gmail.com> <48D39456.80505@ntc.zcu.cz> <3d375d730809191146k7425c569i2450e17e8d51c588@mail.gmail.com> <48D5F325.7080104@ntc.zcu.cz> <3d375d730809210017s43030527n584792ada09044a6@mail.gmail.com> Message-ID: <48D5F717.5070602@ntc.zcu.cz> Robert Kern wrote: > On Sun, Sep 21, 2008 at 02:09, Robert Cimrman wrote: >> Robert Kern wrote: >>> On Fri, Sep 19, 2008 at 07:00, Robert Cimrman wrote: >>>> Robert Kern wrote: >>>>> Ah, found it. T_LONGLONG is a #define from structmember.h which is >>>>> used to describe the types of attributes. Apparently, this was not >>>>> added until Python 2.5. That particular member didn't actually need to >>>>> be long long, so I've fixed that. >>>> Great, I will try it after it appears on the web page. >>> Oops! It's now pushed. >> Thanks, it builds ok now! But then cProfile is also new in 2.5. - I >> should probably upgrade and stop bothering you :-) > > Actually, if you pull again, I've just pushed my changes to remove the > hard dependency on cProfile. Cool, then I have another one: $ ./kernprof.py -l pystone.py Traceback (most recent call last): File "./kernprof.py", line 173, in ? sys.exit(main(sys.argv)) File "./kernprof.py", line 138, in main import line_profiler File "/home/share/software/packages/line_profiler-2ea515d0cbc1/line_profiler.py", line 9, in ? from _line_profiler import LineProfiler as CLineProfiler File "_line_profiler.pyx", line 3, in _line_profiler (_line_profiler.c:2246) from cProfile import label ImportError: No module named cProfile Or do I call the profiler in a wrong way? r. From charlesr.harris at gmail.com Sun Sep 21 03:28:21 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 21 Sep 2008 01:28:21 -0600 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D5F026.9090809@ar.media.kyoto-u.ac.jp> References: <48D23B35.8040902@dcs.gla.ac.uk> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <48D3B23C.8030100@american.edu> <48D47180.3020101@ar.media.kyoto-u.ac.jp> <48D5EE89.3010500@ar.media.kyoto-u.ac.jp> <48D5F026.9090809@ar.media.kyoto-u.ac.jp> Message-ID: On Sun, Sep 21, 2008 at 12:56 AM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > David Cournapeau wrote: > > Anne Archibald wrote: > >> If users are concerned about performance, it's worth noting that on > >> some machines nans force a fallback to software floating-point > >> handling, with a corresponding very large performance hit. This > >> includes some but not all x86 (and I think x86-64) CPUs. How this > >> compares to the performance of masked arrays is not clear. > > > > I spent some time on this. In particular, for max.min, I did the > > following for the core loop (always return nan if nan is in the array): > > > > /* nan + x and x + nan are nan, where x can be anything: > > normal, > > * denormal, nan, infinite > > */ > > tmp = *((@typ@ *)i1) + *((@typ@ > > *)i2); > > if(isnan(tmp)) > > { > > *((@typ@ *)op) = > > tmp; > > } else > > { > > *((@typ@ *)op)=*((@typ@ *)i1) @OP@ *((@typ@ *)i2) ? *((@typ@ > > *)i1) : *((@typ@ *)i2); > > } > > Grr, sorry for the mangling: > > /* nan + x and x + nan are nan, where x can be anything: normal, > * denormal, nan, infinite */ > tmp = *((@typ@ *)i1) + *((@typ@*)i2); > if(isnan(tmp)) { > *((@typ@ *)op) = tmp; > } else { > *((@typ@ *)op) = *((@typ@ *)i1) @OP@ *((@typ@ *)i2) ? *((@typ@*)i1) : > *((@typ@ *)i2); > } > You can use type instead of typ so the code is a bit easier to read. It's one of the changes I've made. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Sep 21 03:35:04 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 21 Sep 2008 02:35:04 -0500 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <48D5F717.5070602@ntc.zcu.cz> References: <48D23506.7080109@ntc.zcu.cz> <3d375d730809181237x43ce1610r6dccb081d6698f5d@mail.gmail.com> <48D363DD.6040609@ntc.zcu.cz> <3d375d730809190226g361d5855o1c1475da0379706f@mail.gmail.com> <48D39456.80505@ntc.zcu.cz> <3d375d730809191146k7425c569i2450e17e8d51c588@mail.gmail.com> <48D5F325.7080104@ntc.zcu.cz> <3d375d730809210017s43030527n584792ada09044a6@mail.gmail.com> <48D5F717.5070602@ntc.zcu.cz> Message-ID: <3d375d730809210035t7be721ffx2658f5d8d3e93e1d@mail.gmail.com> On Sun, Sep 21, 2008 at 02:26, Robert Cimrman wrote: > Cool, then I have another one: > > $ ./kernprof.py -l pystone.py > Traceback (most recent call last): > File "./kernprof.py", line 173, in ? > sys.exit(main(sys.argv)) > File "./kernprof.py", line 138, in main > import line_profiler > File > "/home/share/software/packages/line_profiler-2ea515d0cbc1/line_profiler.py", > line 9, in ? > from _line_profiler import LineProfiler as CLineProfiler > File "_line_profiler.pyx", line 3, in _line_profiler > (_line_profiler.c:2246) > from cProfile import label > ImportError: No module named cProfile > > Or do I call the profiler in a wrong way? Nope, I just had a brainfart and put the implementation of label() in the wrong file. Pull again. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cimrman3 at ntc.zcu.cz Sun Sep 21 03:41:18 2008 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Sun, 21 Sep 2008 09:41:18 +0200 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <3d375d730809210035t7be721ffx2658f5d8d3e93e1d@mail.gmail.com> References: <48D23506.7080109@ntc.zcu.cz> <3d375d730809181237x43ce1610r6dccb081d6698f5d@mail.gmail.com> <48D363DD.6040609@ntc.zcu.cz> <3d375d730809190226g361d5855o1c1475da0379706f@mail.gmail.com> <48D39456.80505@ntc.zcu.cz> <3d375d730809191146k7425c569i2450e17e8d51c588@mail.gmail.com> <48D5F325.7080104@ntc.zcu.cz> <3d375d730809210017s43030527n584792ada09044a6@mail.gmail.com> <48D5F717.5070602@ntc.zcu.cz> <3d375d730809210035t7be721ffx2658f5d8d3e93e1d@mail.gmail.com> Message-ID: <48D5FA9E.3040400@ntc.zcu.cz> Robert Kern wrote: > On Sun, Sep 21, 2008 at 02:26, Robert Cimrman wrote: >> Cool, then I have another one: >> >> $ ./kernprof.py -l pystone.py >> Traceback (most recent call last): >> File "./kernprof.py", line 173, in ? >> sys.exit(main(sys.argv)) >> File "./kernprof.py", line 138, in main >> import line_profiler >> File >> "/home/share/software/packages/line_profiler-2ea515d0cbc1/line_profiler.py", >> line 9, in ? >> from _line_profiler import LineProfiler as CLineProfiler >> File "_line_profiler.pyx", line 3, in _line_profiler >> (_line_profiler.c:2246) >> from cProfile import label >> ImportError: No module named cProfile >> >> Or do I call the profiler in a wrong way? > > Nope, I just had a brainfart and put the implementation of label() in > the wrong file. Pull again. :-) The example works now in Python 2.4. r. From david at ar.media.kyoto-u.ac.jp Sun Sep 21 05:30:13 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sun, 21 Sep 2008 18:30:13 +0900 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: <48D5EE89.3010500@ar.media.kyoto-u.ac.jp> References: <48D23B35.8040902@dcs.gla.ac.uk> <48D35089.30202@ar.media.kyoto-u.ac.jp> <200809190329.14131.pgmdevlist@gmail.com> <9457e7c80809190136k531ea534r99789ac04c6b80a0@mail.gmail.com> <48D36435.5050709@ar.media.kyoto-u.ac.jp> <48D3B23C.8030100@american.edu> <48D47180.3020101@ar.media.kyoto-u.ac.jp> <48D5EE89.3010500@ar.media.kyoto-u.ac.jp> Message-ID: <48D61425.7010404@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > > The isnan thing is surprising, because the whole point to have a isnan > is that you can do it without branching. I checked, and numpy does use > the macro of isnan, not the function (glibc has both). Ok, see my patch #913 for this. The slowdown is actually specific to one tested machine (my P4). On my macbook (running Mac os X) and another linux machine running a core 2 duo, the performances are the same before and after the patch. I have not tested on windows, though. I also saw this mentioned: http://projects.scipy.org/scipy/numpy/ticket/241 Where Travis made the same argument as me concerning NaN. It seems that the slowdowns are not so significant, at least on the dataset I tested (isnan is actually quite fast on my core 2 duo: 10 cycles / double for large arrays on average, compared to the 60 / double on my P4 for the exact same binary). Travis, if you are reading this, would you reconsider your position on nan handling for min/max/co if we can keep reasonable speed ? cheers, David From dineshbvadhia at hotmail.com Sun Sep 21 13:28:35 2008 From: dineshbvadhia at hotmail.com (Dinesh B Vadhia) Date: Sun, 21 Sep 2008 10:28:35 -0700 Subject: [Numpy-discussion] array indices and dictionary Message-ID: Hi! Say, I've got a numpy array/matrix of the form: A = [[1 6 1 2 3] [4 5 4 7 0] [2 0 8 0 2] [0 0 0 3 7] [0 7 0 3 5] [8 0 3 0 6] [8 0 0 2 2] [3 1 0 4 0] [5 0 8 0 0] [2 1 0 5 6]] I want to create a dictionary of row indexes (as the keys) mapped to lists of the column indexes of non-zero numbers in that row ie. dictionary_non-zeros = { 0: [0 1 2 3 4] 1: [0 1 2 3] 2: [0 2 4] 3: [3 4] ... 9: [0 1 3 4] } How can I do this? This code ... d = {} for number, row in enumerate(A): d[number] = [value for value in row if value !=0] ... picks up the non-zero values in each row eg. d = {0: [1, 6, 1, 2, 3], 1: [4, 5, 4, 7], 2: [2, 8, 2], 3: [3, 7], ... 9: [2, 1, 5, 6]} But, I want to pick up the column index of non-zero elements per row. Thanks! Dinesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From dineshbvadhia at hotmail.com Sun Sep 21 13:51:53 2008 From: dineshbvadhia at hotmail.com (Dinesh B Vadhia) Date: Sun, 21 Sep 2008 10:51:53 -0700 Subject: [Numpy-discussion] Fw: array indices and dictionary Message-ID: Ooops, I should have said that this is easy to do with 2 for loops ie. import numpy from collections import defaultdict A = [[1 6 1 2 3] [4 5 4 7 0] [2 0 8 0 2] [0 0 0 3 7] [0 7 0 3 5] [8 0 3 0 6] [8 0 0 2 2] [3 1 0 4 0] [5 0 8 0 0] [2 1 0 5 6]] dict = defaultdict(list) I = A.shape[0] J = A.shape[1] for i in xrange(0, I, 1): for j in xrange(0, J, 1): if a[i,j] > 0: dict[i].append(j) I want to find a faster/efficient way to do this without using the 2 for loops. Thanks! Dinesh From: Dinesh B Vadhia Sent: Sunday, September 21, 2008 10:28 AM To: numpy-discussion at scipy.org Subject: array indices and dictionary Hi! Say, I've got a numpy array/matrix of the form: A = [[1 6 1 2 3] [4 5 4 7 0] [2 0 8 0 2] [0 0 0 3 7] [0 7 0 3 5] [8 0 3 0 6] [8 0 0 2 2] [3 1 0 4 0] [5 0 8 0 0] [2 1 0 5 6]] I want to create a dictionary of row indexes (as the keys) mapped to lists of the column indexes of non-zero numbers in that row ie. dictionary_non-zeros = { 0: [0 1 2 3 4] 1: [0 1 2 3] 2: [0 2 4] 3: [3 4] ... 9: [0 1 3 4] } How can I do this? This code ... d = {} for number, row in enumerate(A): d[number] = [value for value in row if value !=0] ... picks up the non-zero values in each row eg. d = {0: [1, 6, 1, 2, 3], 1: [4, 5, 4, 7], 2: [2, 8, 2], 3: [3, 7], ... 9: [2, 1, 5, 6]} But, I want to pick up the column index of non-zero elements per row. Thanks! Dinesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From wnbell at gmail.com Sun Sep 21 15:14:29 2008 From: wnbell at gmail.com (Nathan Bell) Date: Sun, 21 Sep 2008 15:14:29 -0400 Subject: [Numpy-discussion] array indices and dictionary In-Reply-To: References: Message-ID: On Sun, Sep 21, 2008 at 1:28 PM, Dinesh B Vadhia wrote: > > But, I want to pick up the column index of non-zero elements per row. > http://www.scipy.org/Numpy_Example_List_With_Doc#nonzero -- Nathan Bell wnbell at gmail.com http://graphics.cs.uiuc.edu/~wnbell/ From arnar.flatberg at gmail.com Sun Sep 21 15:28:45 2008 From: arnar.flatberg at gmail.com (Arnar Flatberg) Date: Sun, 21 Sep 2008 21:28:45 +0200 Subject: [Numpy-discussion] Fw: array indices and dictionary In-Reply-To: References: Message-ID: <5d3194020809211228p315ec3e2mc8290196b4ad4c24@mail.gmail.com> On Sun, Sep 21, 2008 at 7:51 PM, Dinesh B Vadhia wrote: > Ooops, I should have said that this is easy to do with 2 for loops ie. > > import numpy > from collections import defaultdict > A = > [[1 6 1 2 3] > [4 5 4 7 0] > [2 0 8 0 2] > [0 0 0 3 7] > [0 7 0 3 5] > [8 0 3 0 6] > [8 0 0 2 2] > [3 1 0 4 0] > [5 0 8 0 0] > [2 1 0 5 6]] > > dict = defaultdict(list) > I = A.shape[0] > J = A.shape[1] > for i in xrange(0, I, 1): > for j in xrange(0, J, 1): > if a[i,j] > 0: > dict[i].append(j) > > I want to find a faster/efficient way to do this without using the 2 for > loops. Thanks! > > Dinesh > > > *From:* Dinesh B Vadhia > *Sent:* Sunday, September 21, 2008 10:28 AM > *To:* numpy-discussion at scipy.org > *Subject:* array indices and dictionary > > Hi! Say, I've got a numpy array/matrix of the form: > > A = > [[1 6 1 2 3] > [4 5 4 7 0] > [2 0 8 0 2] > [0 0 0 3 7] > [0 7 0 3 5] > [8 0 3 0 6] > [8 0 0 2 2] > [3 1 0 4 0] > [5 0 8 0 0] > [2 1 0 5 6]] > > I want to create a dictionary of row indexes (as the keys) mapped to lists > of the column indexes of non-zero numbers in that row ie. > > dictionary_non-zeros = { > 0: [0 1 2 3 4] > 1: [0 1 2 3] > 2: [0 2 4] > 3: [3 4] > ... > 9: [0 1 3 4] > } > > How can I do this? > > This code ... > > d = {} > for number, row in enumerate(A): > d[number] = [value for value in row if value !=0] > ... picks up the non-zero values in each row eg. > > d = {0: [1, 6, 1, 2, 3], 1: [4, 5, 4, 7], 2: [2, 8, 2], 3: [3, 7], ... 9: > [2, 1, 5, 6]} > > But, I want to pick up the column index of non-zero elements per row. > > Thanks! > > Dinesh > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > Hi Perhaps, dict(enumerate([row.nonzero()[0] for row in A])) An empty index will be the output if a row is all zeros. I dont know if that would be an issue Arnar -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Mon Sep 22 03:40:42 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 22 Sep 2008 09:40:42 +0200 Subject: [Numpy-discussion] New patch for new mgrid / ogrid functionality In-Reply-To: <1221839348.7162.139.camel@localhost> References: <1221839348.7162.139.camel@localhost> Message-ID: <9457e7c80809220040w2b6eacddia1d275627beb0a47@mail.gmail.com> 2008/9/19 David M. Kaplan : > My 2 cents - I personally think the version that always returns a list > will ultimately be more transparent and cause fewer problems than the > newer version. In either case, the plan should be to eventually have it > always return a list as that is the only fully consistent option, the > question is just when that switch should be made and by who. If it is > done at the next major release, someone else will have to remember to ax > the additional code and correct the documentation.... I think this change is worth making, because: - It provides a consistent API for gridding. - It optimises the most frequenty used situation: x, y, z = ogrid[...] - Someone already took the trouble of writing the patch. If the (minor) API breakage is considered a problem, could we go through the standard deprecation process and have this included? St?fan From gael.varoquaux at normalesup.org Mon Sep 22 04:53:29 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 22 Sep 2008 10:53:29 +0200 Subject: [Numpy-discussion] New patch for new mgrid / ogrid functionality In-Reply-To: <9457e7c80809220040w2b6eacddia1d275627beb0a47@mail.gmail.com> References: <1221839348.7162.139.camel@localhost> <9457e7c80809220040w2b6eacddia1d275627beb0a47@mail.gmail.com> Message-ID: <20080922085329.GD24279@phare.normalesup.org> On Mon, Sep 22, 2008 at 09:40:42AM +0200, St?fan van der Walt wrote: > 2008/9/19 David M. Kaplan : > > My 2 cents - I personally think the version that always returns a list > > will ultimately be more transparent and cause fewer problems than the > > newer version. In either case, the plan should be to eventually have it > > always return a list as that is the only fully consistent option, the > > question is just when that switch should be made and by who. If it is > > done at the next major release, someone else will have to remember to ax > > the additional code and correct the documentation.... > I think this change is worth making, because: > - It provides a consistent API for gridding. > - It optimises the most frequenty used situation: > x, y, z = ogrid[...] > - Someone already took the trouble of writing the patch. > If the (minor) API breakage is considered a problem, could we go > through the standard deprecation process and have this included? I must admit I don't really like the API breakage. People will find out that some code that used to work doesn't work anymore. In addition there is some real usecase behind the mgrid behavior: X = mgrid[0:4, 0:4, 0:4] X = X.reshape((3, -1)) scatter(X[0, :], X[1, :]) R = matrix(((cos(0.5), sin(0.5), 0), (-sin(0.5), cos(0.5), 0), (0, 0, 1))) Y = array(R*X) scatter(Y[0, :], Y[1, :], color='r') Ga?l From pzs at dcs.gla.ac.uk Mon Sep 22 06:23:59 2008 From: pzs at dcs.gla.ac.uk (Peter Saffrey) Date: Mon, 22 Sep 2008 10:23:59 +0000 (UTC) Subject: [Numpy-discussion] Medians that ignore values References: <48D23B35.8040902@dcs.gla.ac.uk> <48D2FEC3.3040400@ar.media.kyoto-u.ac.jp> <48D3755D.8060905@ar.media.kyoto-u.ac.jp> <48D38CE3.2030408@ar.media.kyoto-u.ac.jp> Message-ID: David Cournapeau ar.media.kyoto-u.ac.jp> writes: > Still, it is indeed really slow for your case; when I fixed nanmean and > co, I did not know much about numpy, I just wanted them to give the > right answer :) I think this can be made faster, specially for your case > (where the axis along which the median is computed is really small). > I've found that if I just cut nans from the list and use regular numpy median, it is quicker - 10 times slower than list median, rather than 35 times slower. Could you just wire nanmedian to do it this way? The only difference is that on an empty list, nanmedian gives nan, but median throws an IndexError. Below is my profiling code with this change. Sample output: $ ./arrayspeed3.py list build time: 0.16 list median time: 0.08 array nanmedian time: 0.98 Peter === from numpy import * from pylab import rand from time import clock from scipy.stats.stats import nanmedian def my_median(vallist): num_vals = len(vallist) if num_vals == 0: return nan vallist.sort() if num_vals % 2 == 1: # odd index = (num_vals - 1) / 2 return vallist[index] else: # even index = num_vals / 2 return (vallist[index] + vallist[index - 1]) / 2 numtests = 100 testsize = 1000 pointlen = 3 t0 = clock() natests = rand(numtests,testsize,pointlen) # have to start with inf because list.remove(nan) doesn't remove nan natests[natests > 0.9] = inf tests = natests.tolist() natests[natests==inf] = nan for test in tests: for point in test: while inf in point: point.remove(inf) t1 = clock() print "list build time:", t1-t0 allmedians = [] t0 = clock() for test in tests: medians = [ my_median(x) for x in test ] allmedians.append(medians) t1 = clock() print "list median time:", t1-t0 t0 = clock() namedians = [] for natest in natests: thismed = [] for point in natest: maskpoint = point[negative(isnan(point))] if len(maskpoint) > 0: med = median(maskpoint) else: med = nan thismed.append(med) namedians.append(thismed) t1 = clock() print "array nanmedian time:", t1-t0 From numpy-discussion at maubp.freeserve.co.uk Mon Sep 22 07:06:20 2008 From: numpy-discussion at maubp.freeserve.co.uk (Peter) Date: Mon, 22 Sep 2008 12:06:20 +0100 Subject: [Numpy-discussion] numpy.scipy.org and Travis' book Message-ID: <320fb6e00809220406q255879e2oefad214a5aadccb5@mail.gmail.com> Hi all, Now that Travis' book is freely available, it is great to see the NumPy section of http://www.scipy.org/Documentation has been updated. However, could someone update the main numpy webpage (numpy.scipy.org) too? Quoting http://numpy.scipy.org/ > Much of the documentation for Numeric and Numarray is applicable to the new > NumPy package. However, there are significant feature improvements. A > complete guide to the new system has been written by the primary developer, > Travis Oliphant. If you want to fully understand the new system, or you just > want to encourage further development on NumPy (or SciPy), you should > purchase the documentation which is being sold for a relatively brief period > of time to help offset the cost of writing the book and producing the > Numeric/numarray hybrid, and to help raise money for future development. > If the existence of this fee-based book concerns you, Travis has written some > responses to FAQs. This should say the book is now freely available at http://www.tramy.us/numpybook.pdf but could also point out that people can make a donation via http://www.tramy.us/ It would also be nice to update http://www.tramy.us/FAQ.html and add a link to the free download, Thanks, Peter From david at ar.media.kyoto-u.ac.jp Mon Sep 22 08:05:12 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 22 Sep 2008 21:05:12 +0900 Subject: [Numpy-discussion] Medians that ignore values In-Reply-To: References: <48D23B35.8040902@dcs.gla.ac.uk> <48D2FEC3.3040400@ar.media.kyoto-u.ac.jp> <48D3755D.8060905@ar.media.kyoto-u.ac.jp> <48D38CE3.2030408@ar.media.kyoto-u.ac.jp> Message-ID: <48D789F8.2000906@ar.media.kyoto-u.ac.jp> Peter Saffrey wrote: > > I've found that if I just cut nans from the list and use regular numpy median, > it is quicker - 10 times slower than list median, rather than 35 times slower. > Could you just wire nanmedian to do it this way? Unfortunately, we can't, because we would loose generality: we need to compute median on any axis, not only the last one. The proper solution would be to have a sort/max/min/etc... which knows about nan in numpy, which is what Chuck and I are working on ATM, cheers, David From pzs at dcs.gla.ac.uk Mon Sep 22 08:59:27 2008 From: pzs at dcs.gla.ac.uk (Peter Saffrey) Date: Mon, 22 Sep 2008 12:59:27 +0000 (UTC) Subject: [Numpy-discussion] Medians that ignore values References: <48D23B35.8040902@dcs.gla.ac.uk> <48D2FEC3.3040400@ar.media.kyoto-u.ac.jp> <48D3755D.8060905@ar.media.kyoto-u.ac.jp> <48D38CE3.2030408@ar.media.kyoto-u.ac.jp> <48D789F8.2000906@ar.media.kyoto-u.ac.jp> Message-ID: David Cournapeau ar.media.kyoto-u.ac.jp> writes: > Unfortunately, we can't, because we would loose generality: we need to > compute median on any axis, not only the last one. The proper solution > would be to have a sort/max/min/etc... which knows about nan in numpy, > which is what Chuck and I are working on ATM, > Of course - thanks for looking at this. Peter From gael.varoquaux at normalesup.org Mon Sep 22 10:08:34 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 22 Sep 2008 16:08:34 +0200 Subject: [Numpy-discussion] Proceedings of the SciPy conference. Message-ID: <20080922140834.GA5840@phare.normalesup.org> The SciPy conference proceedings are finally available online: http://conference.scipy.org/proceedings/SciPy2008 . I hope you enjoy them. I find it great to have this set of excellent articles talking about works done with, or for, Python in science. For me, it is a reference to remember what was said at the conference. I hope it can also be interesting for people who were not present at the conference. I apologize for being so slow at publishing them. In addition to the round trip between authors and editors taking a while, I have been travelling back home and spent way too much time last week finishing off administrative duties in the US. Ga?l From jdh2358 at gmail.com Mon Sep 22 10:41:30 2008 From: jdh2358 at gmail.com (John Hunter) Date: Mon, 22 Sep 2008 09:41:30 -0500 Subject: [Numpy-discussion] efficient way to do this? Message-ID: <88e473830809220741x5ad009a3y1ca96a9b6b456735@mail.gmail.com> I have a an array of indices into a larger array where some condition is satisfied. I want to create a larger set of indices which *mark* all the indicies following the condition over some Nmark length window. In code: import numpy as np N = 1000 Nmark = 20 ind = np.nonzero(np.random.rand(N)<0.01)[0] marked = np.zeros(N, bool) for i in ind: marked[i:i+Nmark] = True I am going to have to do this over many arrays, and so I want to do it efficiently. Is there a way to do the above more efficiently, eg w/o the loop. In the real use case, there will be significant auto-correlation among the places where the condition is satisfied. Eg, if it is satisfied at some index, it is likely that it will be satisfied for many of its neighbors. Eg, the real case looks more like y = np.sin(2*np.pi*np.linspace(0, 2, N)) ind = np.nonzero(y>0.95)[0] marked2 = np.zeros(N, bool) for i in ind: marked2[i:i+Nmark] = True Thanks in advance for any hints, JDH From silva at lma.cnrs-mrs.fr Mon Sep 22 10:31:08 2008 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Mon, 22 Sep 2008 16:31:08 +0200 Subject: [Numpy-discussion] efficient way to do this? In-Reply-To: <88e473830809220741x5ad009a3y1ca96a9b6b456735@mail.gmail.com> References: <88e473830809220741x5ad009a3y1ca96a9b6b456735@mail.gmail.com> Message-ID: <1222093868.2820.26.camel@localhost> Le lundi 22 septembre 2008 ? 09:41 -0500, John Hunter a ?crit : > I have a an array of indices into a larger array where some condition > is satisfied. I want to create a larger set of indices which *mark* > all the indicies following the condition over some Nmark length > window. A = np.random.rand(N) What about that: marked = (A<0.01) > In the real use case, there will be significant auto-correlation among > the places where the condition is satisfied. Eg, if it is satisfied > at some index, it is likely that it will be satisfied for many of its > neighbors. Eg, the real case looks more like > > y = np.sin(2*np.pi*np.linspace(0, 2, N)) > > ind = np.nonzero(y>0.95)[0] > marked2 = np.zeros(N, bool) > for i in ind: > marked2[i:i+Nmark] = True I do not understand what you do expect here to code... -- Fabrice Silva LMA UPR CNRS 7051 From robert.kern at gmail.com Mon Sep 22 11:13:19 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 22 Sep 2008 10:13:19 -0500 Subject: [Numpy-discussion] efficient way to do this? In-Reply-To: <88e473830809220741x5ad009a3y1ca96a9b6b456735@mail.gmail.com> References: <88e473830809220741x5ad009a3y1ca96a9b6b456735@mail.gmail.com> Message-ID: <3d375d730809220813w2a8c4010t952eb468e0be75ca@mail.gmail.com> On Mon, Sep 22, 2008 at 09:41, John Hunter wrote: > I have a an array of indices into a larger array where some condition > is satisfied. I want to create a larger set of indices which *mark* > all the indicies following the condition over some Nmark length > window. In code: > > import numpy as np > > N = 1000 > Nmark = 20 > ind = np.nonzero(np.random.rand(N)<0.01)[0] > > > marked = np.zeros(N, bool) > for i in ind: > marked[i:i+Nmark] = True > > I am going to have to do this over many arrays, and so I want to do it > efficiently. Is there a way to do the above more efficiently, eg w/o > the loop. marked[ind + np.arange(Nmark)] = True -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From jdh2358 at gmail.com Mon Sep 22 11:18:19 2008 From: jdh2358 at gmail.com (John Hunter) Date: Mon, 22 Sep 2008 10:18:19 -0500 Subject: [Numpy-discussion] efficient way to do this? In-Reply-To: <3d375d730809220813w2a8c4010t952eb468e0be75ca@mail.gmail.com> References: <88e473830809220741x5ad009a3y1ca96a9b6b456735@mail.gmail.com> <3d375d730809220813w2a8c4010t952eb468e0be75ca@mail.gmail.com> Message-ID: <88e473830809220818sce67745h6b0e7584907ad710@mail.gmail.com> On Mon, Sep 22, 2008 at 10:13 AM, Robert Kern wrote: > marked[ind + np.arange(Nmark)] = True That triggers a broadcasting error: Traceback (most recent call last): File "/home/titan/johnh/test.py", line 13, in ? marked3[ind + np.arange(Nmark)] = True ValueError: shape mismatch: objects cannot be broadcast to a single shape I am hoping there is some clever way to do this with broadcasting, I am just not that clever... JDH From robert.kern at gmail.com Mon Sep 22 11:20:30 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 22 Sep 2008 10:20:30 -0500 Subject: [Numpy-discussion] New patch for new mgrid / ogrid functionality In-Reply-To: <20080922085329.GD24279@phare.normalesup.org> References: <1221839348.7162.139.camel@localhost> <9457e7c80809220040w2b6eacddia1d275627beb0a47@mail.gmail.com> <20080922085329.GD24279@phare.normalesup.org> Message-ID: <3d375d730809220820g41af84f9p9f7ebd561210d4be@mail.gmail.com> On Mon, Sep 22, 2008 at 03:53, Gael Varoquaux wrote: > On Mon, Sep 22, 2008 at 09:40:42AM +0200, St?fan van der Walt wrote: >> 2008/9/19 David M. Kaplan : >> > My 2 cents - I personally think the version that always returns a list >> > will ultimately be more transparent and cause fewer problems than the >> > newer version. In either case, the plan should be to eventually have it >> > always return a list as that is the only fully consistent option, the >> > question is just when that switch should be made and by who. If it is >> > done at the next major release, someone else will have to remember to ax >> > the additional code and correct the documentation.... > >> I think this change is worth making, because: > >> - It provides a consistent API for gridding. >> - It optimises the most frequenty used situation: > >> x, y, z = ogrid[...] > >> - Someone already took the trouble of writing the patch. > >> If the (minor) API breakage is considered a problem, could we go >> through the standard deprecation process and have this included? > > I must admit I don't really like the API breakage. People will find out > that some code that used to work doesn't work anymore. > > In addition there is some real usecase behind the mgrid behavior: > > > X = mgrid[0:4, 0:4, 0:4] > X = X.reshape((3, -1)) > > scatter(X[0, :], X[1, :]) > > R = matrix(((cos(0.5), sin(0.5), 0), (-sin(0.5), cos(0.5), 0), (0, 0, 1))) > > Y = array(R*X) > > scatter(Y[0, :], Y[1, :], color='r') Agreed. Please don't change this behavior even through a deprecation. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Mon Sep 22 11:22:15 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 22 Sep 2008 10:22:15 -0500 Subject: [Numpy-discussion] efficient way to do this? In-Reply-To: <88e473830809220818sce67745h6b0e7584907ad710@mail.gmail.com> References: <88e473830809220741x5ad009a3y1ca96a9b6b456735@mail.gmail.com> <3d375d730809220813w2a8c4010t952eb468e0be75ca@mail.gmail.com> <88e473830809220818sce67745h6b0e7584907ad710@mail.gmail.com> Message-ID: <3d375d730809220822gc05cdfbnf7c09a5970b16b45@mail.gmail.com> On Mon, Sep 22, 2008 at 10:18, John Hunter wrote: > On Mon, Sep 22, 2008 at 10:13 AM, Robert Kern wrote: > >> marked[ind + np.arange(Nmark)] = True > > That triggers a broadcasting error: > > Traceback (most recent call last): > File "/home/titan/johnh/test.py", line 13, in ? > marked3[ind + np.arange(Nmark)] = True > ValueError: shape mismatch: objects cannot be broadcast to a single shape > > I am hoping there is some clever way to do this with broadcasting, I > am just not that clever... Oh, right. Sorry. ind2mark = np.asarray((ind[:,np.newaxis] + np.arange(Nmark).flat).clip(0, N-1) marked[ind2mark] = True -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Mon Sep 22 11:23:06 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 22 Sep 2008 10:23:06 -0500 Subject: [Numpy-discussion] efficient way to do this? In-Reply-To: <3d375d730809220822gc05cdfbnf7c09a5970b16b45@mail.gmail.com> References: <88e473830809220741x5ad009a3y1ca96a9b6b456735@mail.gmail.com> <3d375d730809220813w2a8c4010t952eb468e0be75ca@mail.gmail.com> <88e473830809220818sce67745h6b0e7584907ad710@mail.gmail.com> <3d375d730809220822gc05cdfbnf7c09a5970b16b45@mail.gmail.com> Message-ID: <3d375d730809220823k76ae2488r5126ba121d39ce58@mail.gmail.com> On Mon, Sep 22, 2008 at 10:22, Robert Kern wrote: > ind2mark = np.asarray((ind[:,np.newaxis] + np.arange(Nmark).flat).clip(0, N-1) > marked[ind2mark] = True Missing parenthesis: ind2mark = np.asarray((ind[:,np.newaxis] + np.arange(Nmark)).flat).clip(0, N-1) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From jdh2358 at gmail.com Mon Sep 22 11:25:15 2008 From: jdh2358 at gmail.com (John Hunter) Date: Mon, 22 Sep 2008 10:25:15 -0500 Subject: [Numpy-discussion] efficient way to do this? In-Reply-To: <3d375d730809220823k76ae2488r5126ba121d39ce58@mail.gmail.com> References: <88e473830809220741x5ad009a3y1ca96a9b6b456735@mail.gmail.com> <3d375d730809220813w2a8c4010t952eb468e0be75ca@mail.gmail.com> <88e473830809220818sce67745h6b0e7584907ad710@mail.gmail.com> <3d375d730809220822gc05cdfbnf7c09a5970b16b45@mail.gmail.com> <3d375d730809220823k76ae2488r5126ba121d39ce58@mail.gmail.com> Message-ID: <88e473830809220825l66111192pbe63477905e49d63@mail.gmail.com> On Mon, Sep 22, 2008 at 10:23 AM, Robert Kern wrote: > On Mon, Sep 22, 2008 at 10:22, Robert Kern wrote: > >> ind2mark = np.asarray((ind[:,np.newaxis] + np.arange(Nmark).flat).clip(0, N-1) >> marked[ind2mark] = True > > Missing parenthesis: > > ind2mark = np.asarray((ind[:,np.newaxis] + np.arange(Nmark)).flat).clip(0, N-1) Excellent, thanks! Note to self, must become a newaxis guru... JDH From f.yw at hotmail.com Mon Sep 22 15:19:16 2008 From: f.yw at hotmail.com (frank wang) Date: Mon, 22 Sep 2008 13:19:16 -0600 Subject: [Numpy-discussion] loadtxt error In-Reply-To: <3d375d730809181500p225e42b0u6bf92645141b35d5@mail.gmail.com> References: <3d375d730809181500p225e42b0u6bf92645141b35d5@mail.gmail.com> Message-ID: Hi, All, I am struggling to make the loadtxt works. In my file, I have several colunms of data, say I have two. When I use the following command to load the data, fid=loadtxt('filename.csv',comments='"',dtype='|S4',converters={0:lambda s:int(s,16)}) It will load an array has two columns. However, I only want to load one column to save the memory. If I type fid=loadtxt('filename.csv',comments='"',dtype='|S4',converters={0:lambda s:int(s,16)},usecols=(1,)) I got error: 337 for i, conv in (user_converters or {}).iteritems(): 338 if usecols:--> 339 i = usecols.find(i) 340 converters[i] = conv 341 AttributeError: 'tuple' object has no attribute 'find'> c:\python25\lib\site-packages\numpy\lib\io.py(339)loadtxt() 338 if usecols:--> 339 i = usecols.find(i) 340 converters[i] = conv Can someone help to solve this problem? Or is this a bug in loadtxt? Here is a test file with contains: "ADCOUT" "Nyquist"29F 23529F 397 Thanks Frank _________________________________________________________________ See how Windows Mobile brings your life together?at home, work, or on the go. http://clk.atdmt.com/MRT/go/msnnkwxp1020093182mrt/direct/01/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Mon Sep 22 16:13:18 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 22 Sep 2008 22:13:18 +0200 Subject: [Numpy-discussion] loadtxt error In-Reply-To: References: <3d375d730809181500p225e42b0u6bf92645141b35d5@mail.gmail.com> Message-ID: <9457e7c80809221313gd8baa5bi8cad6e988d8dfabe@mail.gmail.com> 2008/9/22 frank wang : > It will load an array has two columns. However, I only want to load one > column to save the memory. If I type > > fid=loadtxt('filename.csv',comments='"',dtype='|S4',converters={0:lambda > s:int(s,16)},usecols=(1,)) You are specifying a converter that won't be in the output. I can fix that, but in the meantime do: np.loadtxt('dat.dat',comments='"',dtype='|S4',converters={1:lambda s:int(s,16)},usecols=(1,)) [(565,), (919,)] |S4 Regards St?fan From travis at enthought.com Mon Sep 22 16:18:17 2008 From: travis at enthought.com (Travis Vaught) Date: Mon, 22 Sep 2008 15:18:17 -0500 Subject: [Numpy-discussion] ANN: Enthought Python Distribution 4.0.300 Beta 2 available Message-ID: <75D8561B-D264-4F32-AE0B-75CF0C074600@enthought.com> Greetings, We've recently posted the second beta release of the Enthought Python Distribution (EPD) for our upcoming general release of version 4.0.300 with Python 2.5. You may download the beta from here: http://www.enthought.com/products/epdbeta.php Please feel free to test it out and provide feedback on the EPD Trac instance: https://svn.enthought.com/epd You can check out the release notes here: http://www.enthought.com/products/epdbetareleasenotes.php About EPD --------- The Enthought Python Distribution (EPD) is a "kitchen-sink-included" distribution of the Python? Programming Language, including over 60 additional tools and libraries. The EPD bundle includes NumPy, SciPy, IPython, 2D and 3D visualization, database adapters, and a lot of other tools right out of the box. http://www.enthought.com/products/epd.php It is currently available as a single-click installer for Windows XP (x86), Mac OS X (a universal binary for Intel 10.4 and above) and RedHat EL3 (x86 and amd64). EPD is free for academic use. An annual Subscription and installation support are available for individual commercial use (http://www.enthought.com/products/epddownload.php ). An Enterprise Subscription with support for particular deployment environments is also available for commercial purchase (http://www.enthought.com/products/enterprise.php ). The Beta versions of EPD are available for indefinite free trial. Thanks, Travis From stefan at sun.ac.za Mon Sep 22 16:34:12 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 22 Sep 2008 22:34:12 +0200 Subject: [Numpy-discussion] New patch for new mgrid / ogrid functionality In-Reply-To: <3d375d730809220820g41af84f9p9f7ebd561210d4be@mail.gmail.com> References: <1221839348.7162.139.camel@localhost> <9457e7c80809220040w2b6eacddia1d275627beb0a47@mail.gmail.com> <20080922085329.GD24279@phare.normalesup.org> <3d375d730809220820g41af84f9p9f7ebd561210d4be@mail.gmail.com> Message-ID: <9457e7c80809221334u358b426drd9e9d8f900571d0f@mail.gmail.com> 2008/9/22 Robert Kern : > Agreed. Please don't change this behavior even through a deprecation. Are you OK with the functionality in the patch that returns the mixed grid as a tuple but keeps the heterogeneous grid as is? St?fan From gael.varoquaux at normalesup.org Mon Sep 22 17:16:03 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 22 Sep 2008 23:16:03 +0200 Subject: [Numpy-discussion] New patch for new mgrid / ogrid functionality In-Reply-To: <9457e7c80809221334u358b426drd9e9d8f900571d0f@mail.gmail.com> References: <1221839348.7162.139.camel@localhost> <9457e7c80809220040w2b6eacddia1d275627beb0a47@mail.gmail.com> <20080922085329.GD24279@phare.normalesup.org> <3d375d730809220820g41af84f9p9f7ebd561210d4be@mail.gmail.com> <9457e7c80809221334u358b426drd9e9d8f900571d0f@mail.gmail.com> Message-ID: <20080922211602.GA10852@phare.normalesup.org> On Mon, Sep 22, 2008 at 10:34:12PM +0200, St?fan van der Walt wrote: > 2008/9/22 Robert Kern : > > Agreed. Please don't change this behavior even through a deprecation. > Are you OK with the functionality in the patch that returns the mixed > grid as a tuple but keeps the heterogeneous grid as is? I am. This is not a backward-incompatible API breakage. Gael From f.yw at hotmail.com Mon Sep 22 17:27:13 2008 From: f.yw at hotmail.com (frank wang) Date: Mon, 22 Sep 2008 15:27:13 -0600 Subject: [Numpy-discussion] loadtxt error In-Reply-To: <9457e7c80809221313gd8baa5bi8cad6e988d8dfabe@mail.gmail.com> References: <3d375d730809181500p225e42b0u6bf92645141b35d5@mail.gmail.com> <9457e7c80809221313gd8baa5bi8cad6e988d8dfabe@mail.gmail.com> Message-ID: Hi, Stefan, Thanks for your help. First it is my typing error. My command is:fid=loadtxt('filename.csv',comments='"',dtype='|S4',converters={1:lambdas:int(s,16)},usecols=(1,)) It is not the zero. This will causes the same error as: In [218]: fid=loadtxt('test.dat',comments='"',dtype='|S4',converters={1:lambda s:int(s,16)},usecols=(1,))---------------------------------------------------------------------------AttributeError Traceback (most recent call last) C:\DHG\DOCSIS\lab_test\parseadc.py in ()----> 1 2 3 4 5 C:\Python25\lib\site-packages\numpy\lib\io.pyc in loadtxt(fname, dtype, comments, delimiter, converters, skiprows, usecols, unpack) 337 for i, conv in (user_converters or {}).iteritems(): 338 if usecols:--> 339 i = usecols.find(i) 340 converters[i] = conv 341 AttributeError: 'tuple' object has no attribute 'find'> c:\python25\lib\site-packages\numpy\lib\io.py(339)loadtxt() 338 if usecols:--> 339 i = usecols.find(i) 340 converters[i] = conv This command is the same as yours. I do not know why it is working for you.> Date: Mon, 22 Sep 2008 22:13:18 +0200> From: stefan at sun.ac.za> To: numpy-discussion at scipy.org> Subject: Re: [Numpy-discussion] loadtxt error> > 2008/9/22 frank wang :> > It will load an array has two columns. However, I only want to load one> > column to save the memory. If I type> >> > fid=loadtxt('filename.csv',comments='"',dtype='|S4',converters={0:lambda> > s:int(s,16)},usecols=(1,))> > You are specifying a converter that won't be in the output. I can fix> that, but in the meantime do:> > np.loadtxt('dat.dat',comments='"',dtype='|S4',converters={1:lambda> s:int(s,16)},usecols=(1,))> [(565,), (919,)]> |S4> > Regards> St?fan> _______________________________________________> Numpy-discussion mailing list> Numpy-discussion at scipy.org> http://projects.scipy.org/mailman/listinfo/numpy-discussion _________________________________________________________________ See how Windows connects the people, information, and fun that are part of your life. http://clk.atdmt.com/MRT/go/msnnkwxp1020093175mrt/direct/01/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonyyu at MIT.EDU Mon Sep 22 19:25:16 2008 From: tonyyu at MIT.EDU (Tony S Yu) Date: Mon, 22 Sep 2008 19:25:16 -0400 Subject: [Numpy-discussion] array with named columns (or record arrays with homogenous types) Message-ID: <03E79269-F7E0-4330-8810-15B37D3FB102@mit.edu> Is there any way to name the columns of an array so that they can be called with a string, but still used like an ordinary array? What I have in mind is very similar to a record array, but with homogenous types. Having different types implies that certain operations (e.g., mean, sum) don't make sense, in general. For example, I want to do something like: #~~~~~~~ >>> import numpy as np >>> color = np.array([(1, 2, 3), (4, 5, 6)], ... dtype=[('r', '>> color[0].mean() TypeError: cannot perform reduce with flexible type #~~~~~~~ BTW, I actually want add the fields (columns) for all records (rows), not just each row individually, but `color.mean(axis=1)` doesn't make sense because `color` is actually a 1D array of records. Thanks, -Tony From pgmdevlist at gmail.com Mon Sep 22 19:28:50 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 22 Sep 2008 19:28:50 -0400 Subject: [Numpy-discussion] array with named columns (or record arrays with homogenous types) In-Reply-To: <03E79269-F7E0-4330-8810-15B37D3FB102@mit.edu> References: <03E79269-F7E0-4330-8810-15B37D3FB102@mit.edu> Message-ID: <200809221928.50535.pgmdevlist@gmail.com> On Monday 22 September 2008 19:25:16 Tony S Yu wrote: > >>> import numpy as np > >>> > >>> color = np.array([(1, 2, 3), (4, 5, 6)], > > ... dtype=[('r', ' # get the average (grayscale) of the first row > > >>> color[0].mean() What about color.view((float,3))[0].mean() ? From f.yw at hotmail.com Mon Sep 22 19:56:47 2008 From: f.yw at hotmail.com (frank wang) Date: Mon, 22 Sep 2008 17:56:47 -0600 Subject: [Numpy-discussion] loadtxt error In-Reply-To: References: <3d375d730809181500p225e42b0u6bf92645141b35d5@mail.gmail.com> <9457e7c80809221313gd8baa5bi8cad6e988d8dfabe@mail.gmail.com> Message-ID: This error is caused that the usecols is a tuple and it does not have find command. I do not know how to fix this problem. Thanks Frank From: f.yw at hotmail.comTo: numpy-discussion at scipy.orgDate: Mon, 22 Sep 2008 15:27:13 -0600Subject: Re: [Numpy-discussion] loadtxt error Hi, Stefan, Thanks for your help. First it is my typing error. My command is:fid=loadtxt('filename.csv',comments='"',dtype='|S4',converters={1:lambdas:int(s,16)},usecols=(1,)) It is not the zero. This will causes the same error as:In [218]: fid=loadtxt('test.dat',comments='"',dtype='|S4',converters={1:lambda s:int(s,16)},usecols=(1,))---------------------------------------------------------------------------AttributeError Traceback (most recent call last)C:\DHG\DOCSIS\lab_test\parseadc.py in ()----> 1 2 3 4 5C:\Python25\lib\site-packages\numpy\lib\io.pyc in loadtxt(fname, dtype, comments, delimiter, converters, skiprows, usecols, unpack) 337 for i, conv in (user_converters or {}).iteritems(): 338 if usecols:--> 339 i = usecols.find(i) 340 converters[i] = conv 341AttributeError: 'tuple' object has no attribute 'find'> c:\python25\lib\site-packages\numpy\lib\io.py(339)loadtxt() 338 if usecols:--> 339 i = usecols.find(i) 340 converters[i] = conv This command is the same as yours. I do not know why it is working for you.> Date: Mon, 22 Sep 2008 22:13:18 +0200> From: stefan at sun.ac.za> To: numpy-discussion at scipy.org> Subject: Re: [Numpy-discussion] loadtxt error> > 2008/9/22 frank wang :> > It will load an array has two columns. However, I only want to load one> > column to save the memory. If I type> >> > fid=loadtxt('filename.csv',comments='"',dtype='|S4',converters={0:lambda> > s:int(s,16)},usecols=(1,))> > You are specifying a converter that won't be in the output. I can fix> that, but in the meantime do:> > np.loadtxt('dat.dat',comments='"',dtype='|S4',converters={1:lambda> s:int(s,16)},usecols=(1,))> [(565,), (919,)]> |S4> > Regards> St?fan> _______________________________________________> Numpy-discussion mailing list> Numpy-discussion at scipy.org> http://projects.scipy.org/mailman/listinfo/numpy-discussion See how Windows connects the people, information, and fun that are part of your life. See Now _________________________________________________________________ See how Windows Mobile brings your life together?at home, work, or on the go. http://clk.atdmt.com/MRT/go/msnnkwxp1020093182mrt/direct/01/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Mon Sep 22 20:10:11 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 22 Sep 2008 20:10:11 -0400 Subject: [Numpy-discussion] loadtxt error In-Reply-To: References: Message-ID: <200809222010.12025.pgmdevlist@gmail.com> On Monday 22 September 2008 19:56:47 frank wang wrote: > This error is caused that the usecols is a tuple and it does not have find > command. I do not know how to fix this problem. Try to use a list instead of a tuple as a quick-fix. Anyway, Frank, you should try to give us the version of numpy you're using. Obviously, it's not the latest. From tsyu80 at gmail.com Tue Sep 23 00:06:14 2008 From: tsyu80 at gmail.com (Tony Yu) Date: Tue, 23 Sep 2008 00:06:14 -0400 Subject: [Numpy-discussion] array with named columns (or record arrays with homogenous types) In-Reply-To: References: Message-ID: <32A99673-8CD6-4DF2-A7DB-FE4C022B19F9@gmail.com> > > Message: 8 > Date: Mon, 22 Sep 2008 19:28:50 -0400 > From: Pierre GM > Subject: Re: [Numpy-discussion] array with named columns (or record > arrays with homogenous types) > To: Discussion of Numerical Python > Message-ID: <200809221928.50535.pgmdevlist at gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > On Monday 22 September 2008 19:25:16 Tony S Yu wrote: > >>>>> import numpy as np >>>>> >>>>> color = np.array([(1, 2, 3), (4, 5, 6)], >> >> ... dtype=[('r', '> '> # get the average (grayscale) of the first row >> >>>>> color[0].mean() > > What about > color.view((float,3))[0].mean() ? Yeah, that works perfectly; by specifying the axis and dropping the index (`color.view((' References: <32A99673-8CD6-4DF2-A7DB-FE4C022B19F9@gmail.com> Message-ID: <200809230016.31438.pgmdevlist@gmail.com> On Tuesday 23 September 2008 00:06:14 Tony Yu wrote: > BTW, is the tuple argument you use with `view` documented anywhere; I > haven't seen it before and a quick search doesn't give any results. http://www.scipy.org/RecordArrays + Travis O.'s numpy book, the chapter on dtype (pp. 137-138) Note that we should think about organizing www.scipy.org a bit here: we have a page on recarray (search for 'recarray'), one on RecordArrays (search for 'record'), none for RecordArray, and a search on 'records' yields nothing). From michael.abshoff at googlemail.com Tue Sep 23 00:56:32 2008 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Mon, 22 Sep 2008 21:56:32 -0700 Subject: [Numpy-discussion] complex roots() segfault on Solaris 10/x86 with numpy 1.2.rc1 using python 2.5.2 Message-ID: <48D87700.8070607@gmail.com> Hi, I have been having trouble with the following computation with numpy 1.2.rc1: Python 2.5.2 (r252:60911, Jul 14 2008, 15:20:38) [GCC 4.2.4] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> from numpy import * >>> a = array([1,0,0,0,0,-1],dtype=complex) array([ 1.+0.j, 0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j, -1.+0.j]) >>> roots(a) Segmentation Fault (core dumped) python "$@" This is python 2.5.2 build with gcc 4.2.4, numpy itself is build with "-O0", i.e. this is unlikely to be a compiler bug IMHO. This bug has been present in 1.0.4, 1.1.0 and it seems unfixed in 1.2.rc1. The numpy 1.1 test suite passed with that install, I did not run the 1.2.rc1 one yet since I do not have nose installed. gdb says nothing particularly helpful: -bash-3.00$ gdb python GNU gdb 6.6 Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-pc-solaris2.10"... (gdb) r Starting program: /home/mabshoff/sage-3.0.5-x86-solaris/local/bin/python warning: rw_common (): unable to read at addr 0xd3f08 warning: sol_thread_new_objfile: td_ta_new: Debugger service failed warning: Unable to find dynamic linker breakpoint function. GDB will be unable to debug shared library initializers and track explicitly loaded dynamic code. Python 2.5.2 (r252:60911, Jul 14 2008, 15:20:38) [GCC 4.2.4] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> from numpy import * >>> a = array([1,0,0,0,0,-1],dtype=complex) >>> a array([ 1.+0.j, 0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j, -1.+0.j]) >>> roots(a) Program received signal SIGSEGV, Segmentation fault. 0xfed622b3 in csqrt () from /lib/libm.so.2 (gdb) bt #0 0xfed622b3 in csqrt () from /lib/libm.so.2 #1 0xfe681d5a in ?? () #2 0x00000000 in ?? () (gdb) quit Any pointers? Cheers, Michael From david at ar.media.kyoto-u.ac.jp Tue Sep 23 00:56:00 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 23 Sep 2008 13:56:00 +0900 Subject: [Numpy-discussion] complex roots() segfault on Solaris 10/x86 with numpy 1.2.rc1 using python 2.5.2 In-Reply-To: <48D87700.8070607@gmail.com> References: <48D87700.8070607@gmail.com> Message-ID: <48D876E0.2000909@ar.media.kyoto-u.ac.jp> Michael Abshoff wrote: > This is python 2.5.2 build with gcc 4.2.4, numpy itself is build with > "-O0", i.e. this is unlikely to be a compiler bug IMHO. This bug has > been present in 1.0.4, 1.1.0 and it seems unfixed in 1.2.rc1. The numpy > 1.1 test suite passed with that install, I did not run the 1.2.rc1 one > yet since I do not have nose installed. > > gdb says nothing particularly helpful: Yes, because you built python with -O0; you should add at least -g to the cflags, because here, you don't have debugging symbols, meaning the gdb won't be of much help unfortunately. Would it be possible for you to rebuild python and numpy with debugging info ? I tried to look into that bug, but I can't reproduce it on opensolaris with gcc and python 2.4 cheers, David From michael.abshoff at googlemail.com Tue Sep 23 01:21:27 2008 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Mon, 22 Sep 2008 22:21:27 -0700 Subject: [Numpy-discussion] complex roots() segfault on Solaris 10/x86 with numpy 1.2.rc1 using python 2.5.2 In-Reply-To: <48D876E0.2000909@ar.media.kyoto-u.ac.jp> References: <48D87700.8070607@gmail.com> <48D876E0.2000909@ar.media.kyoto-u.ac.jp> Message-ID: <48D87CD7.9090303@gmail.com> David Cournapeau wrote: > Michael Abshoff wrote: Hi David, >> This is python 2.5.2 build with gcc 4.2.4, numpy itself is build with >> "-O0", i.e. this is unlikely to be a compiler bug IMHO. This bug has >> been present in 1.0.4, 1.1.0 and it seems unfixed in 1.2.rc1. The numpy >> 1.1 test suite passed with that install, I did not run the 1.2.rc1 one >> yet since I do not have nose installed. >> >> gdb says nothing particularly helpful: > > Yes, because you built python with -O0; you should add at least -g to > the cflags, because here, you don't have debugging symbols, meaning the > gdb won't be of much help unfortunately. > > Would it be possible for you to rebuild python and numpy with debugging > info ? Sorry for not being precise: Both python and numpy have been build with OPT=-DNDEBUG -g -O0 -fwrapv -Wall -Wstrict-prototypes i.e. "-O0" instead of "-O3". I am using ATLAS and netlib.org Lapack, so I will rebuild everything and run the BLAS as well as Lapack testers to make 100% sure everything is on working correctly. Since the above python is part of a Sage build and passes 99% of Sage's doctests I am pretty sure that ATLAS and Lapack do work, but one never knows. Since I am sleepy I will do all that tomorrow. > I tried to look into that bug, but I can't reproduce it on > opensolaris with gcc and python 2.4 Ok. Which compiler did you use? > cheers, > > David Cheers, Michael > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From david at ar.media.kyoto-u.ac.jp Tue Sep 23 01:31:51 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 23 Sep 2008 14:31:51 +0900 Subject: [Numpy-discussion] complex roots() segfault on Solaris 10/x86 with numpy 1.2.rc1 using python 2.5.2 In-Reply-To: <48D87CD7.9090303@gmail.com> References: <48D87700.8070607@gmail.com> <48D876E0.2000909@ar.media.kyoto-u.ac.jp> <48D87CD7.9090303@gmail.com> Message-ID: <48D87F47.6030108@ar.media.kyoto-u.ac.jp> Michael Abshoff wrote: > > Sorry for not being precise: Both python and numpy have been build with > > OPT=-DNDEBUG -g -O0 -fwrapv -Wall -Wstrict-prototypes Hm, strange. I don't know why you can't get any debug info, then. > > i.e. "-O0" instead of "-O3". I am using ATLAS and netlib.org Lapack, so > I will rebuild everything and run the BLAS as well as Lapack testers to > make 100% sure everything is on working correctly. Since the above > python is part of a Sage build and passes 99% of Sage's doctests I am > pretty sure that ATLAS and Lapack do work, but one never knows. Since I > am sleepy I will do all that tomorrow. It is indeed very likely the problem is there: typically, problem when passing complex number at the C/Fortran, this kind of thing. > > Ok. Which compiler did you use? gcc 3.4 (the one give by open solaris by default), no blas/lapack. I actually would try this configuration first (no blas/lapack at all), to be sure it is blas/lapack or us. cheers, David From David.Kaplan at ird.fr Tue Sep 23 03:39:53 2008 From: David.Kaplan at ird.fr (David Kaplan) Date: Tue, 23 Sep 2008 09:39:53 +0200 Subject: [Numpy-discussion] New patch for new mgrid / ogrid functionality In-Reply-To: References: Message-ID: <1222155593.6544.15.camel@piura> Hi, I would note that there is nothing in the API breakage that prevents doing what Gael mentions. The only change is that: > > X = mgrid[0:4, 0:4, 0:4] would be replaced with: X = asarray( mgrid[ 0:4, 0:4, 0:4 ] ) On the other hand, not making this change makes for a strange and ugly implementation. > Agreed. Please don't change this behavior even through a deprecation. > Long term it is bound to cause problems to have nd_grid return a list in 3 out of 4 use cases, but an array in other cases. This function is already quite confusing for users because it uses indexing instead of a function call. Adding this nuance might be great for the power user, but will confuse most. I am convinced that eventually the API should be broken, the question is just when. If you want to put it off till "2.0", that is fine, but it puts more onus on the release manager to do the cleanup necessary. Cheers, David -- ********************************** David M. Kaplan Charge de Recherche 1 Institut de Recherche pour le Developpement Centre de Recherche Halieutique Mediterraneenne et Tropicale av. Jean Monnet B.P. 171 34203 Sete cedex France Phone: +33 (0)4 99 57 32 27 Fax: +33 (0)4 99 57 32 95 http://www.ur097.ird.fr/team/dkaplan/index.html ********************************** From michael.abshoff at googlemail.com Tue Sep 23 04:37:57 2008 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Tue, 23 Sep 2008 01:37:57 -0700 Subject: [Numpy-discussion] complex roots() segfault on Solaris 10/x86 with numpy 1.2.rc1 using python 2.5.2 In-Reply-To: <48D87F47.6030108@ar.media.kyoto-u.ac.jp> References: <48D87700.8070607@gmail.com> <48D876E0.2000909@ar.media.kyoto-u.ac.jp> <48D87CD7.9090303@gmail.com> <48D87F47.6030108@ar.media.kyoto-u.ac.jp> Message-ID: <48D8AAE5.5030802@gmail.com> David Cournapeau wrote: > Michael Abshoff wrote: Hi David, >> Sorry for not being precise: Both python and numpy have been build with >> >> OPT=-DNDEBUG -g -O0 -fwrapv -Wall -Wstrict-prototypes > > Hm, strange. I don't know why you can't get any debug info, then. well, it looks like some sort of stack corruption. >> i.e. "-O0" instead of "-O3". I am using ATLAS and netlib.org Lapack, so >> I will rebuild everything and run the BLAS as well as Lapack testers to >> make 100% sure everything is on working correctly. Since the above >> python is part of a Sage build and passes 99% of Sage's doctests I am >> pretty sure that ATLAS and Lapack do work, but one never knows. Since I >> am sleepy I will do all that tomorrow. > > It is indeed very likely the problem is there: typically, problem when > passing complex number at the C/Fortran, this kind of thing. > >> Ok. Which compiler did you use? > > gcc 3.4 (the one give by open solaris by default), no blas/lapack. I > actually would try this configuration first (no blas/lapack at all), to > be sure it is blas/lapack or us. I had a second wind and I just build python+numpy without blas and lapack and low and behold it works: Python 2.5.2 (r252:60911, Sep 23 2008, 04:25:09) [GCC 4.2.4] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> from numpy import * >>> a = array([1,0,0,0,0,-1],dtype=complex) >>> a array([ 1.+0.j, 0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j, -1.+0.j]) >>> roots(a) array([-0.80901699+0.58778525j, -0.80901699-0.58778525j, 0.30901699+0.95105652j, 0.30901699-0.95105652j, 1.00000000+0.j ]) >>> ^D So it looks like gfortran 4.2.4 on Solaris + "-O3" miscompiles lapack, which is honestly unbelievable considering netlib.org lapack+BLAS are the yard stick for any Fortran compiler. Sorry for the noise and thanks for your help. > cheers, > > David Cheers, Michael > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From david at ar.media.kyoto-u.ac.jp Tue Sep 23 04:39:11 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 23 Sep 2008 17:39:11 +0900 Subject: [Numpy-discussion] complex roots() segfault on Solaris 10/x86 with numpy 1.2.rc1 using python 2.5.2 In-Reply-To: <48D8AAE5.5030802@gmail.com> References: <48D87700.8070607@gmail.com> <48D876E0.2000909@ar.media.kyoto-u.ac.jp> <48D87CD7.9090303@gmail.com> <48D87F47.6030108@ar.media.kyoto-u.ac.jp> <48D8AAE5.5030802@gmail.com> Message-ID: <48D8AB2F.8020505@ar.media.kyoto-u.ac.jp> Michael Abshoff wrote: > > So it looks like gfortran 4.2.4 on Solaris + "-O3" miscompiles lapack, > which is honestly unbelievable considering netlib.org lapack+BLAS are > the yard stick for any Fortran compiler. > Well, not many people use gfortran on Solaris, I would guess. Another problem with BLAS/LAPACK is that the testsuite is not really usable: it breaks on every platform I have used it (tens if not hundred of tests are broken). cheers, David From david.huard at gmail.com Tue Sep 23 08:42:00 2008 From: david.huard at gmail.com (David Huard) Date: Tue, 23 Sep 2008 08:42:00 -0400 Subject: [Numpy-discussion] loadtxt error In-Reply-To: <200809222010.12025.pgmdevlist@gmail.com> References: <200809222010.12025.pgmdevlist@gmail.com> Message-ID: <91cf711d0809230542l3c5227b3ja89479ec1295e1af@mail.gmail.com> This bug has been fixed in the trunk a couple of weeks ago. Cheers, David On Mon, Sep 22, 2008 at 8:10 PM, Pierre GM wrote: > On Monday 22 September 2008 19:56:47 frank wang wrote: > > This error is caused that the usecols is a tuple and it does not have > find > > command. I do not know how to fix this problem. > > Try to use a list instead of a tuple as a quick-fix. > Anyway, Frank, you should try to give us the version of numpy you're using. > Obviously, it's not the latest. > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Sep 23 13:33:16 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 23 Sep 2008 12:33:16 -0500 Subject: [Numpy-discussion] New patch for new mgrid / ogrid functionality In-Reply-To: <1222155593.6544.15.camel@piura> References: <1222155593.6544.15.camel@piura> Message-ID: <3d375d730809231033g69db0da5nb85f42a793d14ed@mail.gmail.com> On Tue, Sep 23, 2008 at 02:39, David Kaplan wrote: > Hi, > > I would note that there is nothing in the API breakage that prevents > doing what Gael mentions. The only change is that: > >> > X = mgrid[0:4, 0:4, 0:4] > > would be replaced with: > > X = asarray( mgrid[ 0:4, 0:4, 0:4 ] ) > > On the other hand, not making this change makes for a strange and ugly > implementation. > >> Agreed. Please don't change this behavior even through a deprecation. > > Long term it is bound to cause problems to have nd_grid return a list in > 3 out of 4 use cases, but an array in other cases. This function is > already quite confusing for users because it uses indexing instead of a > function call. Adding this nuance might be great for the power user, > but will confuse most. The only time it would need to return a list is with the new functionality. So expose the new functionality with a different object instead of cramming it into an existing object with an incompatible API. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stefan at sun.ac.za Tue Sep 23 13:41:17 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 23 Sep 2008 19:41:17 +0200 Subject: [Numpy-discussion] New patch for new mgrid / ogrid functionality In-Reply-To: <3d375d730809231033g69db0da5nb85f42a793d14ed@mail.gmail.com> References: <1222155593.6544.15.camel@piura> <3d375d730809231033g69db0da5nb85f42a793d14ed@mail.gmail.com> Message-ID: <9457e7c80809231041m3304735exe2522cf2df6ace4c@mail.gmail.com> 2008/9/23 Robert Kern : > The only time it would need to return a list is with the new > functionality. So expose the new functionality with a different object > instead of cramming it into an existing object with an incompatible > API. That's a great way to fix the problem: if we call it `grid`, we can turn it into a function and accept the more generic arguments, whilst having keywords to choose between open and meshed grids. Cheers St?fan From gael.varoquaux at normalesup.org Tue Sep 23 14:37:34 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 23 Sep 2008 20:37:34 +0200 Subject: [Numpy-discussion] New patch for new mgrid / ogrid functionality In-Reply-To: <1222155593.6544.15.camel@piura> References: <1222155593.6544.15.camel@piura> Message-ID: <20080923183734.GA20503@phare.normalesup.org> On Tue, Sep 23, 2008 at 09:39:53AM +0200, David Kaplan wrote: > I would note that there is nothing in the API breakage that prevents > doing what Gael mentions. The only change is that: > > > X = mgrid[0:4, 0:4, 0:4] > would be replaced with: > X = asarray( mgrid[ 0:4, 0:4, 0:4 ] ) Doesn't this force a data copy? Ga?l From gael.varoquaux at normalesup.org Tue Sep 23 14:38:37 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 23 Sep 2008 20:38:37 +0200 Subject: [Numpy-discussion] New patch for new mgrid / ogrid functionality In-Reply-To: <9457e7c80809231041m3304735exe2522cf2df6ace4c@mail.gmail.com> References: <1222155593.6544.15.camel@piura> <3d375d730809231033g69db0da5nb85f42a793d14ed@mail.gmail.com> <9457e7c80809231041m3304735exe2522cf2df6ace4c@mail.gmail.com> Message-ID: <20080923183837.GB20503@phare.normalesup.org> On Tue, Sep 23, 2008 at 07:41:17PM +0200, St?fan van der Walt wrote: > That's a great way to fix the problem: if we call it `grid`, we can > turn it into a function and accept the more generic arguments, whilst > having keywords to choose between open and meshed grids. If you make it a function you cannot pass it slices. Appart from this I am +1. Ga?l From josef.pktd at gmail.com Tue Sep 23 16:18:40 2008 From: josef.pktd at gmail.com (joep) Date: Tue, 23 Sep 2008 13:18:40 -0700 (PDT) Subject: [Numpy-discussion] python 2.4 incompatibility in numpy.lookfor in numpy 1.2.0rc2 Message-ID: Hi, I was trying to use lookfor on python 2.4 and got an exception with np.lookfor because of a missing method in pkgutil that was added in python 2.5. see the following session >>> np.lookfor('range') Traceback (most recent call last): File "", line 1, in ? np.lookfor('range') File "C:\Programs\Python24\Lib\site-packages\numpy\lib\utils.py", line 556, in lookfor cache = _lookfor_generate_cache(module, import_modules, regenerate) File "C:\Programs\Python24\Lib\site-packages\numpy\lib\utils.py", line 687, in _lookfor_generate_cache for m in pkgutil.iter_modules(item.__path__): AttributeError: 'module' object has no attribute 'iter_modules' >>> import pkgutil >>> dir(pkgutil) ['__builtins__', '__doc__', '__file__', '__name__', 'extend_path', 'os', 'sys'] >>> pkgutil.__file__ 'C:\\Programs\\Python24\\lib\\pkgutil.pyc' >>> np.version.version '1.2.0rc2' Josef From josef.pktd at gmail.com Tue Sep 23 16:34:20 2008 From: josef.pktd at gmail.com (joep) Date: Tue, 23 Sep 2008 13:34:20 -0700 (PDT) Subject: [Numpy-discussion] np.min(np.inf,1000) == np.inf Message-ID: I just fell over this: >>> np.min(np.inf,1000) 1.#INF >>> min(np.inf,1000) 1000 >>> np.max(-np.inf,1000) -1.#INF >>> max(-np.inf,1000) 1000 Is this known behavior? I finally realized, that I have to do this (the doc string says array like) >>> np.min([np.inf,1000]) 1000.0 >>> np.max([-np.inf,1000]) 1000.0 >>> np.version.version '1.2.0rc2' Josef From robert.kern at gmail.com Tue Sep 23 16:35:29 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 23 Sep 2008 15:35:29 -0500 Subject: [Numpy-discussion] python 2.4 incompatibility in numpy.lookfor in numpy 1.2.0rc2 In-Reply-To: References: Message-ID: <3d375d730809231335o4d2a996fh78c1a1ae8bc7ccc3@mail.gmail.com> On Tue, Sep 23, 2008 at 15:18, joep wrote: > Hi, > I was trying to use lookfor on python 2.4 and got an exception with > np.lookfor because of a missing method in pkgutil that was added in > python 2.5. Hmmm. Unfortunately, that's an intrinsic part of the lookfor() functionality. We could just bring in all of pkgutil.py. Or we remove lookfor(). Or we document it as working with Python 2.5 only. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From millman at berkeley.edu Tue Sep 23 16:55:56 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 23 Sep 2008 13:55:56 -0700 Subject: [Numpy-discussion] python 2.4 incompatibility in numpy.lookfor in numpy 1.2.0rc2 In-Reply-To: <3d375d730809231335o4d2a996fh78c1a1ae8bc7ccc3@mail.gmail.com> References: <3d375d730809231335o4d2a996fh78c1a1ae8bc7ccc3@mail.gmail.com> Message-ID: On Tue, Sep 23, 2008 at 1:35 PM, Robert Kern wrote: > On Tue, Sep 23, 2008 at 15:18, joep wrote: >> Hi, >> I was trying to use lookfor on python 2.4 and got an exception with >> np.lookfor because of a missing method in pkgutil that was added in >> python 2.5. > > Hmmm. Unfortunately, that's an intrinsic part of the lookfor() > functionality. We could just bring in all of pkgutil.py. Or we remove > lookfor(). Or we document it as working with Python 2.5 only. For 1.2.0, let's add a note to the docstring stating it currently works for 2.5 and make it raise a more informative exception. lookfor is in 1.1.x as well. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From robert.kern at gmail.com Tue Sep 23 17:05:11 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 23 Sep 2008 16:05:11 -0500 Subject: [Numpy-discussion] np.min(np.inf,1000) == np.inf In-Reply-To: References: Message-ID: <3d375d730809231405g1721ae52rb44fc75eee7ad37d@mail.gmail.com> On Tue, Sep 23, 2008 at 15:34, joep wrote: > I just fell over this: > >>>> np.min(np.inf,1000) > 1.#INF >>>> min(np.inf,1000) > 1000 > >>>> np.max(-np.inf,1000) > -1.#INF >>>> max(-np.inf,1000) > 1000 > > Is this known behavior? It is known that that's not the API of numpy.min(). It probably should raise an exception since 1000 is not a valid axis for a rank-0 array (which is what the first argument gets turned into under the covers). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From josef.pktd at gmail.com Tue Sep 23 18:48:22 2008 From: josef.pktd at gmail.com (joep) Date: Tue, 23 Sep 2008 15:48:22 -0700 (PDT) Subject: [Numpy-discussion] python 2.4 incompatibility in numpy.lookfor in numpy 1.2.0rc2 In-Reply-To: References: <3d375d730809231335o4d2a996fh78c1a1ae8bc7ccc3@mail.gmail.com> Message-ID: <3a807c70-ce35-4fd1-8911-91d0baa3d290@79g2000hsk.googlegroups.com> A possible solution would be in http://bazaar.launchpad.net/~pauli-virtanen/scipy/pydocweb/revision/386 It seems to be possible to be used in the same way when iter_modules is not available. The usage in ``_lookfor_generate_cache`` in lib/utils.py looks almost exactly the same as in the above revision Josef On Sep 23, 4:55?pm, "Jarrod Millman" wrote: > On Tue, Sep 23, 2008 at 1:35 PM, Robert Kern wrote: > > On Tue, Sep 23, 2008 at 15:18, joep wrote: > >> Hi, > >> I was trying to use lookfor on python 2.4 and got an exception with > >> np.lookfor because of a missing method in pkgutil that was added in > >> python 2.5. > > > Hmmm. Unfortunately, that's an intrinsic part of the lookfor() > > functionality. We could just bring in all of pkgutil.py. Or we remove > > lookfor(). Or we document it as working with Python 2.5 only. > > For 1.2.0, let's add a note to the docstring stating it currently > works for 2.5 and make it raise a more informative exception. > > lookfor is in 1.1.x as well. > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014http://cirl.berkeley.edu/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion From josef.pktd at gmail.com Tue Sep 23 21:54:36 2008 From: josef.pktd at gmail.com (joep) Date: Tue, 23 Sep 2008 18:54:36 -0700 (PDT) Subject: [Numpy-discussion] python 2.4 incompatibility in numpy.lookfor in numpy 1.2.0rc2 In-Reply-To: <3a807c70-ce35-4fd1-8911-91d0baa3d290@79g2000hsk.googlegroups.com> References: <3d375d730809231335o4d2a996fh78c1a1ae8bc7ccc3@mail.gmail.com> <3a807c70-ce35-4fd1-8911-91d0baa3d290@79g2000hsk.googlegroups.com> Message-ID: <7da9c004-cd02-4cf3-9c81-95f46a2dda57@k7g2000hsd.googlegroups.com> On Sep 23, 6:48 pm, joep wrote: > A possible solution would be in > > http://bazaar.launchpad.net/~pauli-virtanen/scipy/pydocweb/revision/386 > > It seems to be possible to be used in the same way when iter_modules > is not available. > The usage in ``_lookfor_generate_cache`` in lib/utils.py looks almost > exactly the same > as in the above revision > > Josef this works for me and leaves the module unchanged if iter_modules is available python 2.4 compatible version is copied from http://bazaar.launchpad.net/~pauli-virtanen/scipy/pydocweb/revision/386 with adjustment for variable names, and it works under python 2.4 windowsXP in Python24\Lib\site-packages\numpy\lib\utils.py def _lookfor_generate_cache starting at line 685 # import sub-packages if import_modules and hasattr(item, '__path__'): if hasattr(pkgutil,'iter_modules'): # line added for m in pkgutil.iter_modules(item.__path__): if _all is not None and m[1] not in _all: continue try: __import__("%s.%s" % (name, m[1])) except ImportError: continue else: # else branch added for pth in item.__path__: for mod_path in os.listdir(pth): init_py = os.path.join(pth, mod_path, '__init__.py') if not os.path.isfile(init_py): continue nm = "%s.%s" % (name, mod_path) try: __import__(nm) except: print >> sys.stderr, "Failed to import module %s" % nm pass example: C:\Josef\easyeclipseworkspace\sandbox\playground\numpyscipy \teststats>ipython Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)] Type "copyright", "credits" or "license" for more information. IPython 0.9.1 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object'. ?object also works, ?? prints more. In [1]: import numpy as np In [2]: np.lookfor('norm') Search results for 'norm' ------------------------- numpy.frexp y1,y2 = frexp(x) numpy.linalg.norm Matrix or vector norm. numpy.ma.cov Estimates the covariance matrix. numpy.random.normal Draw random samples from a normal (Gaussian) distribution. numpy.linalg.qr Compute QR decomposition of a matrix. ....... In [14]: np.version.version Out[14]: '1.2.0rc2' also tested with IDLE same result as with Python 2.5 Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32 Type "copyright", "credits" or "license()" for more information. **************************************************************** Personal firewall software may warn about the connection IDLE makes to its subprocess using this computer's internal loopback interface. This connection is not visible on any external interface and no data is sent to or received from the Internet. **************************************************************** IDLE 1.2.2 ==== No Subprocess ==== >>> import numpy as np >>> np.version.version '1.2.0rc2' >>> np.lookfor('norm') Search results for 'norm' ------------------------- numpy.frexp y1,y2 = frexp(x) numpy.linalg.norm Matrix or vector norm. numpy.ma.cov Estimates the covariance matrix. numpy.random.normal Draw random samples from a normal (Gaussian) distribution. numpy.linalg.qr Compute QR decomposition of a matrix. are there any tests for lookfor? Josef From f.yw at hotmail.com Wed Sep 24 01:11:01 2008 From: f.yw at hotmail.com (frank wang) Date: Tue, 23 Sep 2008 23:11:01 -0600 Subject: [Numpy-discussion] loadtxt error In-Reply-To: <91cf711d0809230542l3c5227b3ja89479ec1295e1af@mail.gmail.com> References: <200809222010.12025.pgmdevlist@gmail.com> <91cf711d0809230542l3c5227b3ja89479ec1295e1af@mail.gmail.com> Message-ID: My numpy version is numpy-1.1.1-win32-superpack-python2.5.exe. It is the newest release. By the way, in Python, how can I check numpy version? Thanks Frank Date: Tue, 23 Sep 2008 08:42:00 -0400From: david.huard at gmail.comTo: numpy-discussion at scipy.orgSubject: Re: [Numpy-discussion] loadtxt error This bug has been fixed in the trunk a couple of weeks ago. Cheers, David On Mon, Sep 22, 2008 at 8:10 PM, Pierre GM wrote: On Monday 22 September 2008 19:56:47 frank wang wrote:> This error is caused that the usecols is a tuple and it does not have find> command. I do not know how to fix this problem.Try to use a list instead of a tuple as a quick-fix.Anyway, Frank, you should try to give us the version of numpy you're using.Obviously, it's not the latest. _______________________________________________Numpy-discussion mailing listNumpy-discussion at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion _________________________________________________________________ Get more out of the Web. Learn 10 hidden secrets of Windows Live. http://windowslive.com/connect/post/jamiethomson.spaces.live.com-Blog-cns!550F681DAD532637!5295.entry?ocid=TXT_TAGLM_WL_domore_092008 -------------- next part -------------- An HTML attachment was scrubbed... URL: From f.yw at hotmail.com Wed Sep 24 01:14:59 2008 From: f.yw at hotmail.com (frank wang) Date: Tue, 23 Sep 2008 23:14:59 -0600 Subject: [Numpy-discussion] loadtxt error In-Reply-To: <200809222010.12025.pgmdevlist@gmail.com> References: <200809222010.12025.pgmdevlist@gmail.com> Message-ID: My numpy version is 1.1.1. I just downloaded and installed. It is the same result. Also, when I use the list, I got similar error said list does not have the find command. ---> 14 fid=loadtxt(fname,comments='"',dtype='|S4',converters={cols:lambda s:int(s,16)},usecols=[cols]) 15 16 #fid=loadtxt('ww36_5adcoutputsin45mhznotuner-0dbm_mux_adc_ddc_rmR.csv',comments='"',dtype='string',usecols=(0,)) C:\Python25\lib\site-packages\numpy\lib\io.pyc in loadtxt(fname, dtype, comments, delimiter, converters, skiprows, usecols, unpack) 337 for i, conv in (user_converters or {}).iteritems(): 338 if usecols:--> 339 i = usecols.find(i) 340 converters[i] = conv 341 AttributeError: 'list' object has no attribute 'find'> c:\python25\lib\site-packages\numpy\lib\io.py(339)loadtxt() 338 if usecols:--> 339 i = usecols.find(i) 340 converters[i] = conv Thanks Frank> From: pgmdevlist at gmail.com> To: numpy-discussion at scipy.org> Date: Mon, 22 Sep 2008 20:10:11 -0400> Subject: Re: [Numpy-discussion] loadtxt error> > On Monday 22 September 2008 19:56:47 frank wang wrote:> > This error is caused that the usecols is a tuple and it does not have find> > command. I do not know how to fix this problem.> > Try to use a list instead of a tuple as a quick-fix.> Anyway, Frank, you should try to give us the version of numpy you're using. > Obviously, it's not the latest.> _______________________________________________> Numpy-discussion mailing list> Numpy-discussion at scipy.org> http://projects.scipy.org/mailman/listinfo/numpy-discussion _________________________________________________________________ See how Windows connects the people, information, and fun that are part of your life. http://clk.atdmt.com/MRT/go/msnnkwxp1020093175mrt/direct/01/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From f.yw at hotmail.com Wed Sep 24 01:21:15 2008 From: f.yw at hotmail.com (frank wang) Date: Tue, 23 Sep 2008 23:21:15 -0600 Subject: [Numpy-discussion] performance of the numpy In-Reply-To: References: <200809222010.12025.pgmdevlist@gmail.com> Message-ID: Hi, I am new user from matlab world. So I always compare with matlab. in my code, i have a large array with about 100000 data. when I perform c=fft(b) d=abs(c) plot(d) I got good response. however, when i try to execute: plot(abs(fft(b))) it takes forever and i have to kill the operation. In another situation, i want debug my code, so i use pdb.set_trace() to break and then execute the next command: c=fft(b) Then the python keeps running and never got out. I have to kiill it too. i posts this and hope the expert will fix this in future. Thanks for the great work. Frank _________________________________________________________________ Get more out of the Web. Learn 10 hidden secrets of Windows Live. http://windowslive.com/connect/post/jamiethomson.spaces.live.com-Blog-cns!550F681DAD532637!5295.entry?ocid=TXT_TAGLM_WL_domore_092008 -------------- next part -------------- An HTML attachment was scrubbed... URL: From nadavh at visionsense.com Wed Sep 24 01:38:56 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed, 24 Sep 2008 08:38:56 +0300 Subject: [Numpy-discussion] performance of the numpy References: <200809222010.12025.pgmdevlist@gmail.com> Message-ID: <710F2847B0018641891D9A216027636029C29F@ex3.envision.co.il> You should use absolute (a ufunc) and not abs (internal python function): >>> plot(absolute(fft(b))) Nadav. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? frank wang ????: ? 24-??????-08 08:21 ??: Discussion of Numerical Python ????: [Numpy-discussion] performance of the numpy Hi, I am new user from matlab world. So I always compare with matlab. in my code, i have a large array with about 100000 data. when I perform c=fft(b) d=abs(c) plot(d) I got good response. however, when i try to execute: plot(abs(fft(b))) it takes forever and i have to kill the operation. In another situation, i want debug my code, so i use pdb.set_trace() to break and then execute the next command: c=fft(b) Then the python keeps running and never got out. I have to kiill it too. i posts this and hope the expert will fix this in future. Thanks for the great work. Frank _________________________________________________________________ Get more out of the Web. Learn 10 hidden secrets of Windows Live. http://windowslive.com/connect/post/jamiethomson.spaces.live.com-Blog-cns!550F681DAD532637!5295.entry?ocid=TXT_TAGLM_WL_domore_092008 -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3544 bytes Desc: not available URL: From david at ar.media.kyoto-u.ac.jp Wed Sep 24 01:57:35 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 24 Sep 2008 14:57:35 +0900 Subject: [Numpy-discussion] performance of the numpy In-Reply-To: References: <200809222010.12025.pgmdevlist@gmail.com> Message-ID: <48D9D6CF.5070806@ar.media.kyoto-u.ac.jp> frank wang wrote: > Hi, I am new user from matlab world. So I always compare with matlab. > in my code, i have a large array with about 100000 data. when I perform > > c=fft(b) > d=abs(c) > plot(d) > > I got good response. however, when i try to execute: > plot(abs(fft(b))) Can you post the exact script you used ? To be sure you are using the fft/abs/plot you think you are using (does the abs comes from python, or is it numpy.abs ? Without the imports, we can't tell). My first wild guess that your size is not factorisable in small numbers. Are you always using the same dataset (more exactly, the same number of data ?). Because numpy uses (light version of ) fftpack, which does not handle well numbers which are not factorisable in small numbers. In particular, if your size is a prime number, fftpack effectively is O(N^2). In General, it is a good idea to use the next power of 2 if at all possible; another solution is to use scipy.fftpack, which if you have for example fftw installed, deals with prime numbers. Otherwise, I don't see what could be different between doing each step after the other, and everything at once. cheers, David From discerptor at gmail.com Wed Sep 24 02:36:54 2008 From: discerptor at gmail.com (Joshua Lippai) Date: Tue, 23 Sep 2008 23:36:54 -0700 Subject: [Numpy-discussion] loadtxt error In-Reply-To: References: <200809222010.12025.pgmdevlist@gmail.com> Message-ID: <9911419a0809232336n89d6e4djcc59ad71f28732f5@mail.gmail.com> David said the bug was fixed in the trunk, which you don't have; the development state of the main source code, or the trunk, is the latest state available, and releases are always a bit behind trunk since it would be kind of ridiculous to make a new release every time someone commits a change. You can download the current NumPy source code via subversion with svn co http://svn.scipy.org/svn/numpy/trunk numpy While the most recent release version number is 1.1.1, the trunk is at something more along the lines of 1.3.0dev5861 at the moment. You can check your numpy version from the Python shell by importing numpy and then calling the __version__ function. So you would type: import numpy numpy.__version__ And the version would be displayed on screen. Bear in mind that unlike the release, which installs via an installer file you double click, you will have to compile numpy from the downloaded source code yourself. Detailed instructions for doing this (and also installing SciPy from source) are available here: http://www.scipy.org/Installing_SciPy/Windows Josh On Tue, Sep 23, 2008 at 10:14 PM, frank wang wrote: > My numpy version is 1.1.1. I just downloaded and installed. It is the same > result. Also, when I use the list, I got similar error said list does not > have the find command. > > ---> 14 > fid=loadtxt(fname,comments='"',dtype='|S4',converters={cols:lambda s > :int(s,16)},usecols=[cols]) > 15 > 16 > #fid=loadtxt('ww36_5adcoutputsin45mhznotuner-0dbm_mux_adc_ddc_rmR.cs > v',comments='"',dtype='string',usecols=(0,)) > C:\Python25\lib\site-packages\numpy\lib\io.pyc in loadtxt(fname, dtype, > comments > , delimiter, converters, skiprows, usecols, unpack) > 337 for i, conv in (user_converters or {}).iteritems(): > 338 if usecols: > --> 339 i = usecols.find(i) > 340 converters[i] = conv > 341 > AttributeError: 'list' object has no attribute 'find' >> c:\python25\lib\site-packages\numpy\lib\io.py(339)loadtxt() > 338 if usecols: > --> 339 i = usecols.find(i) > 340 converters[i] = conv > > > Thanks > > Frank > >> From: pgmdevlist at gmail.com >> To: numpy-discussion at scipy.org >> Date: Mon, 22 Sep 2008 20:10:11 -0400 >> Subject: Re: [Numpy-discussion] loadtxt error >> >> On Monday 22 September 2008 19:56:47 frank wang wrote: >> > This error is caused that the usecols is a tuple and it does not have >> > find >> > command. I do not know how to fix this problem. >> >> Try to use a list instead of a tuple as a quick-fix. >> Anyway, Frank, you should try to give us the version of numpy you're >> using. >> Obviously, it's not the latest. >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > ________________________________ > See how Windows connects the people, information, and fun that are part of > your life. See Now > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From David.Kaplan at ird.fr Wed Sep 24 03:57:13 2008 From: David.Kaplan at ird.fr (David Kaplan) Date: Wed, 24 Sep 2008 09:57:13 +0200 Subject: [Numpy-discussion] New patch for new mgrid / ogrid functionality In-Reply-To: References: Message-ID: <1222243033.6540.22.camel@piura> Hi, On Tue, 2008-09-23 at 20:54 -0500, numpy-discussion-request at scipy.org wrote: > On Tue, Sep 23, 2008 at 09:39:53AM +0200, David Kaplan wrote: > > I would note that there is nothing in the API breakage that prevents > > doing what Gael mentions. The only change is that: > > > > > X = mgrid[0:4, 0:4, 0:4] > > > would be replaced with: > > > X = asarray( mgrid[ 0:4, 0:4, 0:4 ] ) > > Doesn't this force a data copy? > I am not sure if this forces a copy (asarray shouldn't for arrays, unlike array, but I am not sure what it does for lists that can trivially become arrays), but if it does, you are getting it one way or another - in the non-API-breakage patch I sent, the asarray is incorporated into the class method itself. For Robert, the use cases I was talking about included ogrid - ogrid with or without non-slice arguments always returns a list, mgrid with non-slices returns a list, but mgrid with all slices returns an array. I find this funky, but I guess I am going upwind. I personally find adding another object that does stuff like mgrid and ogrid, but not exactly the same, hard to swallow. mgrid and ogrid are already pretty mysterious, adding another is going to make things worse. I was initially trying to make things easier for the new user by providing one function that does all of what you want in a transparent way; this solution seems to be going in the opposite direction. If you do end up adding one, I would call it ndgrid as this is what the matlab equivalent is called. The idea of making it a function eliminates the use of slices, which is where the real power of nd_grid comes in, so I would not go that route. If you want to go with a new object, I would add a "force_array" option to nd_grid that would be checked before executing the asarray code - this option would be true for mgrid, but false for ogrid and "ndgrid". Cheers, David > Ga?l > > -- ********************************** David M. Kaplan Charge de Recherche 1 Institut de Recherche pour le Developpement Centre de Recherche Halieutique Mediterraneenne et Tropicale av. Jean Monnet B.P. 171 34203 Sete cedex France Phone: +33 (0)4 99 57 32 27 Fax: +33 (0)4 99 57 32 95 http://www.ur097.ird.fr/team/dkaplan/index.html ********************************** From robert.kern at gmail.com Wed Sep 24 04:13:06 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 24 Sep 2008 03:13:06 -0500 Subject: [Numpy-discussion] New patch for new mgrid / ogrid functionality In-Reply-To: <1222243033.6540.22.camel@piura> References: <1222243033.6540.22.camel@piura> Message-ID: <3d375d730809240113k58e138e6pc0609ffe3a509d50@mail.gmail.com> On Wed, Sep 24, 2008 at 02:57, David Kaplan wrote: > Hi, > > On Tue, 2008-09-23 at 20:54 -0500, numpy-discussion-request at scipy.org > wrote: >> On Tue, Sep 23, 2008 at 09:39:53AM +0200, David Kaplan wrote: >> > I would note that there is nothing in the API breakage that prevents >> > doing what Gael mentions. The only change is that: >> >> > > > X = mgrid[0:4, 0:4, 0:4] >> >> > would be replaced with: >> >> > X = asarray( mgrid[ 0:4, 0:4, 0:4 ] ) >> >> Doesn't this force a data copy? >> > > I am not sure if this forces a copy (asarray shouldn't for arrays, > unlike array, but I am not sure what it does for lists that can > trivially become arrays), It does because the individual arrays are usually in different parts of memory. You can't just make a single array that is the concatenation of them without copying their contents to a new contiguous block of memory. > but if it does, you are getting it one way or > another - in the non-API-breakage patch I sent, the asarray is > incorporated into the class method itself. Okay, then we shouldn't implement it that way. > For Robert, the use cases I was talking about included ogrid - ogrid > with or without non-slice arguments always returns a list, mgrid with > non-slices returns a list, but mgrid with all slices returns an array. > I find this funky, but I guess I am going upwind. I don't particularly want mgrid to return sometimes an array and sometimes a list, either. I want that functionality under a different name. The sine qua non here is that mgrid[slice,slice,slice] needs to continue to behave as it currently does. We're simply not going to break code for this. > I personally find adding another object that does stuff like mgrid and > ogrid, but not exactly the same, hard to swallow. mgrid and ogrid are > already pretty mysterious, adding another is going to make things worse. > I was initially trying to make things easier for the new user by > providing one function that does all of what you want in a transparent > way; this solution seems to be going in the opposite direction. If you > do end up adding one, I would call it ndgrid as this is what the matlab > equivalent is called. > > The idea of making it a function eliminates the use of slices, which is > where the real power of nd_grid comes in, so I would not go that route. > If you want to go with a new object, I would add a "force_array" option > to nd_grid that would be checked before executing the asarray code - > this option would be true for mgrid, but false for ogrid and "ndgrid". +1. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From theller at ctypes.org Wed Sep 24 04:30:24 2008 From: theller at ctypes.org (Thomas Heller) Date: Wed, 24 Sep 2008 10:30:24 +0200 Subject: [Numpy-discussion] inserting indexes into an array Message-ID: I have a numpy array comtaining numbers (a, b, ...), like this: array([a, b, c, d, e, f]) What is the fastest way to create another array that has the index of the element before each element of the original array? The result should look like this: array([0, a, 1, b, 2, c, 3, d, 4, e, 5, f]) -- Thanks, Thomas From nadavh at visionsense.com Wed Sep 24 04:47:07 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed, 24 Sep 2008 11:47:07 +0300 Subject: [Numpy-discussion] inserting indexes into an array References: Message-ID: <710F2847B0018641891D9A216027636029C2A3@ex3.envision.co.il> Possibilities: column_stack((arange(len(ar)),ar)) or transpose((arange(len(ar)),ar)).ravel() Nadav. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Thomas Heller ????: ? 24-??????-08 11:30 ??: numpy-discussion at scipy.org ????: [Numpy-discussion] inserting indexes into an array I have a numpy array comtaining numbers (a, b, ...), like this: array([a, b, c, d, e, f]) What is the fastest way to create another array that has the index of the element before each element of the original array? The result should look like this: array([0, a, 1, b, 2, c, 3, d, 4, e, 5, f]) -- Thanks, Thomas _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3015 bytes Desc: not available URL: From schut at sarvision.nl Wed Sep 24 06:25:57 2008 From: schut at sarvision.nl (Vincent Schut) Date: Wed, 24 Sep 2008 12:25:57 +0200 Subject: [Numpy-discussion] maskedarray: how to force mask to expand Message-ID: Probably I'm just overlooking something obvious, but I'm having problems with maskedarrays (numpy.ma from svn: '1.3.0.dev5861'), the mask by default being a single bool value ('False') instead of a properly sized bool array. If I then try to mask one value by assigning values to certain mask positions (a.mask[0,0]=True) I get an error, logically. I know I can use mask_where, but I like the mask[...] idiom. And I have to expand the mask anyway, as I'm gonna write it to a file at the end. 1) Is there a way to have ma always use properly expanded masks (bool arrays instead of single bool values)? I tried the shrink=False keyword, but that does not do what I want, and is not available for numpy.ma.zeros, which I conveniently use a lot. 2) Is there a method/function to request the mask, be it a single bool value or an array, as a properly sized array? I found shrink_mask but no opposite method, and shrink_mask seems to do something subtly different even. Regards, Vincent. From gael.varoquaux at normalesup.org Wed Sep 24 06:35:48 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 24 Sep 2008 12:35:48 +0200 Subject: [Numpy-discussion] New patch for new mgrid / ogrid functionality In-Reply-To: <1222243033.6540.22.camel@piura> References: <1222243033.6540.22.camel@piura> Message-ID: <20080924103548.GB1535@phare.normalesup.org> On Wed, Sep 24, 2008 at 09:57:13AM +0200, David Kaplan wrote: > > > X = asarray( mgrid[ 0:4, 0:4, 0:4 ] ) > > Doesn't this force a data copy? > I am not sure if this forces a copy (asarray shouldn't for arrays, > unlike array, but I am not sure what it does for lists that can > trivially become arrays), I cannot really see how avoiding a copy would be possible: the data is not garantied to be organized in a structured between the different arrays of the list. If there is a way to avoid a copy, then it means my knowledge of numpy is really lacking. Ga?l From pav at iki.fi Wed Sep 24 07:18:43 2008 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 24 Sep 2008 11:18:43 +0000 (UTC) Subject: [Numpy-discussion] python 2.4 incompatibility in numpy.lookfor in numpy 1.2.0rc2 References: <3d375d730809231335o4d2a996fh78c1a1ae8bc7ccc3@mail.gmail.com> <3a807c70-ce35-4fd1-8911-91d0baa3d290@79g2000hsk.googlegroups.com> Message-ID: Hi, Tue, 23 Sep 2008 15:48:22 -0700, joep wrote: > A possible solution would be in > > http://bazaar.launchpad.net/~pauli-virtanen/scipy/pydocweb/revision/386 > > It seems to be possible to be used in the same way when iter_modules is > not available. > The usage in ``_lookfor_generate_cache`` in lib/utils.py looks almost > exactly the same > as in the above revision Yes, sorry about this; obviously lookfor wasn't tested with Python2.4. I'll commit and backport a fix for this today, if that's OK. -- Pauli Virtanen From pav at iki.fi Wed Sep 24 07:55:43 2008 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 24 Sep 2008 11:55:43 +0000 (UTC) Subject: [Numpy-discussion] python 2.4 incompatibility in numpy.lookfor in numpy 1.2.0rc2 References: <3d375d730809231335o4d2a996fh78c1a1ae8bc7ccc3@mail.gmail.com> <3a807c70-ce35-4fd1-8911-91d0baa3d290@79g2000hsk.googlegroups.com> Message-ID: Wed, 24 Sep 2008 11:18:43 +0000, Pauli Virtanen wrote: > Tue, 23 Sep 2008 15:48:22 -0700, joep wrote: >> A possible solution would be in >> >> http://bazaar.launchpad.net/~pauli-virtanen/scipy/pydocweb/revision/386 >> >> It seems to be possible to be used in the same way when iter_modules >> is not available. >> The usage in ``_lookfor_generate_cache`` in lib/utils.py looks almost >> exactly the same >> as in the above revision > > Yes, sorry about this; obviously lookfor wasn't tested with Python2.4. > I'll commit and backport a fix for this today, if that's OK. In trunk now, r5862. -- Pauli Virtanen From michel.dupront at hotmail.fr Wed Sep 24 07:59:58 2008 From: michel.dupront at hotmail.fr (Michel Dupront) Date: Wed, 24 Sep 2008 13:59:58 +0200 Subject: [Numpy-discussion] A basic question about swig and Numeric Message-ID: Hello, I am trying to use Numeric and swig but it seems that there are few points that I don't understand. The only excuse I have is that I am new to these tools. I have a simple example that I cannot make work the way I would like. I have a c++ function that take as argument a std::vector. >From python I want to call the c++ function with an array object. For that purpose I want to write a typemap. Here is what was supposed to be a very simple example. example.h: #include #include void globale_print(std::vector vint) { std::cout<<"globale print vecteur: "; for (size_t i=0; i { PyArrayObject* aa=(PyArrayObject *) $input; $1.resize(aa->dimensions[0]); for (unsigned int i=0; idimensions[0]; i++) { $1[i]= *(int *) (aa->data + i* aa->strides[0] ); } }; %include "example.h" runme.py: import example import Numeric array=Numeric.array(range(5)) print array print type(array) example.globale_print(array) Like it is, it works. But there are a lot of things that I don't understand. For example if I want to check aa, like: """ if (!PyArray_Check(aa)) { return NULL; } """ the compilation of the wrapper is ok but when I run the python script runme.py, I have a segmentation fault because of the first line. Here I really need help. Please keep in mind that it is my first try to make a typemap .... Thanks a lot Michel _________________________________________________________________ Installez gratuitement les 20 ?m?ticones Windows Live Messenger les plus fous ! Cliquez ici ! http://www.emoticones-messenger.fr/ From david.huard at gmail.com Wed Sep 24 11:07:38 2008 From: david.huard at gmail.com (David Huard) Date: Wed, 24 Sep 2008 11:07:38 -0400 Subject: [Numpy-discussion] loadtxt error In-Reply-To: <9911419a0809232336n89d6e4djcc59ad71f28732f5@mail.gmail.com> References: <200809222010.12025.pgmdevlist@gmail.com> <9911419a0809232336n89d6e4djcc59ad71f28732f5@mail.gmail.com> Message-ID: <91cf711d0809240807h6b54e38fhe475af6eabeca098@mail.gmail.com> Note that the fix was also backported to 1.2, for which binary builds are available: David [ copied from a recent thread ] The 1.2.0rc2 is now available: http://svn.scipy.org/svn/numpy/tags/1.2.0rc2 The source tarball is here: https://cirl.berkeley.edu/numpy/numpy-1.2.0rc2.tar.gz Here is the universal Mac binary: https://cirl.berkeley.edu/numpy/numpy-1.2.0rc2-py2.5-macosx10.5.dmg Here are the Window's binaries: http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.2 .0rc2-win32-superpack-python2.4.exe http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.2 .0rc2-win32-superpack-python2.5.exe Here are the release notes: http://scipy.org/scipy/numpy/milestone/1.2.0 On Wed, Sep 24, 2008 at 2:36 AM, Joshua Lippai wrote: > David said the bug was fixed in the trunk, which you don't have; the > development state of the main source code, or the trunk, is the latest > state available, and releases are always a bit behind trunk since it > would be kind of ridiculous to make a new release every time someone > commits a change. You can download the current NumPy source code via > subversion with > > svn co http://svn.scipy.org/svn/numpy/trunk numpy > > While the most recent release version number is 1.1.1, the trunk is at > something more along the lines of 1.3.0dev5861 at the moment. You can > check your numpy version from the Python shell by importing numpy and > then calling the __version__ function. So you would type: > > import numpy > numpy.__version__ > > And the version would be displayed on screen. Bear in mind that unlike > the release, which installs via an installer file you double click, > you will have to compile numpy from the downloaded source code > yourself. Detailed instructions for doing this (and also installing > SciPy from source) are available here: > > http://www.scipy.org/Installing_SciPy/Windows > > > Josh > > On Tue, Sep 23, 2008 at 10:14 PM, frank wang wrote: > > My numpy version is 1.1.1. I just downloaded and installed. It is the > same > > result. Also, when I use the list, I got similar error said list does not > > have the find command. > > > > ---> 14 > > fid=loadtxt(fname,comments='"',dtype='|S4',converters={cols:lambda s > > :int(s,16)},usecols=[cols]) > > 15 > > 16 > > #fid=loadtxt('ww36_5adcoutputsin45mhznotuner-0dbm_mux_adc_ddc_rmR.cs > > v',comments='"',dtype='string',usecols=(0,)) > > C:\Python25\lib\site-packages\numpy\lib\io.pyc in loadtxt(fname, dtype, > > comments > > , delimiter, converters, skiprows, usecols, unpack) > > 337 for i, conv in (user_converters or {}).iteritems(): > > 338 if usecols: > > --> 339 i = usecols.find(i) > > 340 converters[i] = conv > > 341 > > AttributeError: 'list' object has no attribute 'find' > >> c:\python25\lib\site-packages\numpy\lib\io.py(339)loadtxt() > > 338 if usecols: > > --> 339 i = usecols.find(i) > > 340 converters[i] = conv > > > > > > Thanks > > > > Frank > > > >> From: pgmdevlist at gmail.com > >> To: numpy-discussion at scipy.org > >> Date: Mon, 22 Sep 2008 20:10:11 -0400 > >> Subject: Re: [Numpy-discussion] loadtxt error > >> > >> On Monday 22 September 2008 19:56:47 frank wang wrote: > >> > This error is caused that the usecols is a tuple and it does not have > >> > find > >> > command. I do not know how to fix this problem. > >> > >> Try to use a list instead of a tuple as a quick-fix. > >> Anyway, Frank, you should try to give us the version of numpy you're > >> using. > >> Obviously, it's not the latest. > >> _______________________________________________ > >> Numpy-discussion mailing list > >> Numpy-discussion at scipy.org > >> http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > ________________________________ > > See how Windows connects the people, information, and fun that are part > of > > your life. See Now > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Wed Sep 24 11:10:45 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 24 Sep 2008 17:10:45 +0200 Subject: [Numpy-discussion] loadtxt error In-Reply-To: <9911419a0809232336n89d6e4djcc59ad71f28732f5@mail.gmail.com> References: <200809222010.12025.pgmdevlist@gmail.com> <9911419a0809232336n89d6e4djcc59ad71f28732f5@mail.gmail.com> Message-ID: <9457e7c80809240810l6bf656aey70434d893200aa98@mail.gmail.com> 2008/9/24 Joshua Lippai : > And the version would be displayed on screen. Bear in mind that unlike > the release, which installs via an installer file you double click, > you will have to compile numpy from the downloaded source code > yourself. Detailed instructions for doing this (and also installing > SciPy from source) are available here: In this case, you can simply grab the io.py file and replace your version. Good luck, St?fan From tom.denniston at alum.dartmouth.org Wed Sep 24 11:21:04 2008 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Wed, 24 Sep 2008 10:21:04 -0500 Subject: [Numpy-discussion] inserting indexes into an array In-Reply-To: <710F2847B0018641891D9A216027636029C2A3@ex3.envision.co.il> References: <710F2847B0018641891D9A216027636029C2A3@ex3.envision.co.il> Message-ID: You could try something like: In [15]: arr = numpy.array([100,200,300]) In [16]: arr2 = numpy.empty((len(arr)*2,)) In [17]: arr2[::2]=arr In [18]: arr2[1::2]=numpy.arange(len(arr)) In [20]: arr2 Out[20]: array([ 100., 0., 200., 1., 300., 2.]) 2008/9/24 Nadav Horesh : > Possibilities: > > column_stack((arange(len(ar)),ar)) > or > transpose((arange(len(ar)),ar)).ravel() > > Nadav. > > > -----????? ??????----- > ???: numpy-discussion-bounces at scipy.org ??? Thomas Heller > ????: ? 24-??????-08 11:30 > ??: numpy-discussion at scipy.org > ????: [Numpy-discussion] inserting indexes into an array > > I have a numpy array comtaining numbers (a, b, ...), like this: > > array([a, b, c, d, e, f]) > > What is the fastest way to create another array that has the index > of the element before each element of the original array? > > The result should look like this: > > array([0, a, 1, b, 2, c, 3, d, 4, e, 5, f]) > > -- > Thanks, > Thomas > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From pgmdevlist at gmail.com Wed Sep 24 12:37:21 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 24 Sep 2008 12:37:21 -0400 Subject: [Numpy-discussion] maskedarray: how to force mask to expand In-Reply-To: References: Message-ID: <200809241237.21287.pgmdevlist@gmail.com> Vincent, You should really consider putting an example next time. I must admit that I'm not sure what you're trying to do, and where/why it fails. Yes, by default, the mask of a new MaskedArray is set to the value 'nomask', which is the boolean False. Directly setting an element of the mask in that condition fails of course. The reasons behind using this behavior are (1) backward compatibility and (2) speed, as you can bypass a lot of operations on the mask when it is empty. If you need to mask one or several elements, the easiest is not to modify the mask itself, but to use the the special value `masked`: >>>a = ma.array(np.arange(6).reshape(3,2)) masked_array(data = [[0 1] [2 3] [4 5]], mask = False, fill_value=999999) >>> # Mask the first element. >>> a[0,0] = ma.masked >>> a masked_array(data = [[-- 1] [2 3] [4 5]], mask = [[ True False] [False False] [False False]], fill_value=999999) This value, `masked`, is also useful to check whether one particular element is masked: >>> a[0,0] is ma.masked True >>> a[0,1] is ma.masked False You can also force the mask to be full of False with the proper shape by that way: >>> a = ma.array(np.arange(6).reshape(3,2) >>> # Force the mask to have the proper shape and be full of False: >>> a.mask = False masked_array(data = [[0 1] [2 3] [4 5]], mask = [[False False] [False False] [False False]], fill_value=999999) The shrink argument of ma.array collapses amask full of False to nomask, once again for speed reasons. So no, it won't do what you look like to want. I agree that having to deal with nomask is not completely intuitive. However, it is required for backward compatibility. One day, the class will be ported to C, and then I'll push to have the mask set to the proper shape ab initio, because then speed will be less of an issue. In the meantime, I hope I answered your question. On Wednesday 24 September 2008 06:25:57 Vincent Schut wrote: > Probably I'm just overlooking something obvious, but I'm having problems > with maskedarrays (numpy.ma from svn: '1.3.0.dev5861'), the mask by > default being a single bool value ('False') instead of a properly sized > bool array. If I then try to mask one value by assigning values to > certain mask positions (a.mask[0,0]=True) I get an error, logically. I > know I can use mask_where, but I like the mask[...] idiom. And I have to > expand the mask anyway, as I'm gonna write it to a file at the end. > > 1) Is there a way to have ma always use properly expanded masks (bool > arrays instead of single bool values)? I tried the shrink=False keyword, > but that does not do what I want, and is not available for > numpy.ma.zeros, which I conveniently use a lot. > > 2) Is there a method/function to request the mask, be it a single bool > value or an array, as a properly sized array? I found shrink_mask but no > opposite method, and shrink_mask seems to do something subtly different > even. > > Regards, > Vincent. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From bsouthey at gmail.com Wed Sep 24 14:13:16 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 24 Sep 2008 13:13:16 -0500 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: References: Message-ID: <48DA833C.8000402@gmail.com> Hi, What is the status of NumPy 1.2 and when can we expect the final version? Thanks Bruce Jarrod Millman wrote: > Hello, > > The 1.2.0rc2 is now available: > http://svn.scipy.org/svn/numpy/tags/1.2.0rc2 > > The source tarball is here: > https://cirl.berkeley.edu/numpy/numpy-1.2.0rc2.tar.gz > > Here is the universal Mac binary: > https://cirl.berkeley.edu/numpy/numpy-1.2.0rc2-py2.5-macosx10.5.dmg > > Here are the Window's binaries: > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.2.0rc2-win32-superpack-python2.4.exe > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.2.0rc2-win32-superpack-python2.5.exe > > Here are the release notes: > http://scipy.org/scipy/numpy/milestone/1.2.0 > > Please test this release ASAP and let us know if there are any > problems. If there are no show stoppers, this will become the > 1.2.0 release. > > Thanks, > > From millman at berkeley.edu Wed Sep 24 14:33:47 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 24 Sep 2008 11:33:47 -0700 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: <48DA833C.8000402@gmail.com> References: <48DA833C.8000402@gmail.com> Message-ID: On Wed, Sep 24, 2008 at 11:13 AM, Bruce Southey wrote: > What is the status of NumPy 1.2 and when can we expect the final version? I hope to announce it on Friday or Saturday. The only thing I am looking into now is whether we should back port the fix to lookfor or not: http://projects.scipy.org/scipy/numpy/changeset/5862 Either way, I plan to tag later today. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From f.yw at hotmail.com Wed Sep 24 15:25:59 2008 From: f.yw at hotmail.com (frank wang) Date: Wed, 24 Sep 2008 13:25:59 -0600 Subject: [Numpy-discussion] loadtxt error In-Reply-To: <9457e7c80809240810l6bf656aey70434d893200aa98@mail.gmail.com> References: <200809222010.12025.pgmdevlist@gmail.com> <9911419a0809232336n89d6e4djcc59ad71f28732f5@mail.gmail.com> <9457e7c80809240810l6bf656aey70434d893200aa98@mail.gmail.com> Message-ID: Thank you very much for all of you. I have downloaded the binary version 1.2rc and it fixed the problem. My special thanks to the person who created the window binary version for users who do not know or do not have the capacity to build the numpy from source. Frank> Date: Wed, 24 Sep 2008 17:10:45 +0200> From: stefan at sun.ac.za> To: numpy-discussion at scipy.org> Subject: Re: [Numpy-discussion] loadtxt error> > 2008/9/24 Joshua Lippai :> > And the version would be displayed on screen. Bear in mind that unlike> > the release, which installs via an installer file you double click,> > you will have to compile numpy from the downloaded source code> > yourself. Detailed instructions for doing this (and also installing> > SciPy from source) are available here:> > In this case, you can simply grab the io.py file and replace your version.> > Good luck,> St?fan> _______________________________________________> Numpy-discussion mailing list> Numpy-discussion at scipy.org> http://projects.scipy.org/mailman/listinfo/numpy-discussion _________________________________________________________________ See how Windows connects the people, information, and fun that are part of your life. http://clk.atdmt.com/MRT/go/msnnkwxp1020093175mrt/direct/01/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Wed Sep 24 17:48:44 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 24 Sep 2008 18:48:44 -0300 Subject: [Numpy-discussion] A basic question about swig and Numeric In-Reply-To: References: Message-ID: I believe you should look at the SWIG docs and then write a typecheck typemap. Checking foir the type of and array and returning NULL is not fair play for SWIG, nor for Python. Before returning NULL, and exception should be set. For this, SWIG provides some 'SWIG_xxx_fail' macros. Typemaps and fragments are powerfull features, but it takes some time to learn the right way to write them. On Wed, Sep 24, 2008 at 8:59 AM, Michel Dupront wrote: > > > > > Hello, > > I am trying to use Numeric and swig but it seems that > there are few points that I don't understand. > The only excuse I have is that I am new to these tools. > I have a simple example that I cannot make work the > way I would like. > I have a c++ function that take as argument a std::vector. > >From python I want to call the c++ function with an array > object. For that purpose I want to write a typemap. > > Here is what was supposed to be a very simple example. > > example.h: > #include > #include > > void globale_print(std::vector vint) > { > std::cout<<"globale print vecteur: "; > for (size_t i=0; i > { > PyArrayObject* aa=(PyArrayObject *) $input; > $1.resize(aa->dimensions[0]); > for (unsigned int i=0; idimensions[0]; i++) > { > $1[i]= *(int *) (aa->data + i* aa->strides[0] ); > } > }; > > %include "example.h" > > > runme.py: > import example > import Numeric > > array=Numeric.array(range(5)) > print array > print type(array) > > example.globale_print(array) > > > > Like it is, it works. But there are a lot of things that I don't > understand. > For example if I want to check aa, like: > """ > if (!PyArray_Check(aa)) { > return NULL; > } > """ > > the compilation of the wrapper is ok but when I run the python > script runme.py, I have a segmentation fault because of the first line. > > Here I really need help. > Please keep in mind that it is my first try to make a typemap .... > > Thanks a lot > Michel > _________________________________________________________________ > Installez gratuitement les 20 ?m?ticones Windows Live Messenger les plus fous ! Cliquez ici ! > http://www.emoticones-messenger.fr/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Wed Sep 24 17:51:05 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 24 Sep 2008 18:51:05 -0300 Subject: [Numpy-discussion] A basic question about swig and Numeric In-Reply-To: References: Message-ID: Ups! Since I?ve started to use Cython, it seems I?m starting to forget things about SWIG. Mi comments about a typecheck typemaps were a nonsese (they have another pourpose). Look at the SWIG docs, you need to use something like SWIG_arg_fail macro. On Wed, Sep 24, 2008 at 6:48 PM, Lisandro Dalcin wrote: > I believe you should look at the SWIG docs and then write a typecheck > typemap. Checking foir the type of and array and returning NULL is not > fair play for SWIG, nor for Python. Before returning NULL, and > exception should be set. For this, SWIG provides some 'SWIG_xxx_fail' > macros. Typemaps and fragments are powerfull features, but it takes > some time to learn the right way to write them. > > On Wed, Sep 24, 2008 at 8:59 AM, Michel Dupront > wrote: >> >> >> >> >> Hello, >> >> I am trying to use Numeric and swig but it seems that >> there are few points that I don't understand. >> The only excuse I have is that I am new to these tools. >> I have a simple example that I cannot make work the >> way I would like. >> I have a c++ function that take as argument a std::vector. >> >From python I want to call the c++ function with an array >> object. For that purpose I want to write a typemap. >> >> Here is what was supposed to be a very simple example. >> >> example.h: >> #include >> #include >> >> void globale_print(std::vector vint) >> { >> std::cout<<"globale print vecteur: "; >> for (size_t i=0; i >> { >> PyArrayObject* aa=(PyArrayObject *) $input; >> $1.resize(aa->dimensions[0]); >> for (unsigned int i=0; idimensions[0]; i++) >> { >> $1[i]= *(int *) (aa->data + i* aa->strides[0] ); >> } >> }; >> >> %include "example.h" >> >> >> runme.py: >> import example >> import Numeric >> >> array=Numeric.array(range(5)) >> print array >> print type(array) >> >> example.globale_print(array) >> >> >> >> Like it is, it works. But there are a lot of things that I don't >> understand. >> For example if I want to check aa, like: >> """ >> if (!PyArray_Check(aa)) { >> return NULL; >> } >> """ >> >> the compilation of the wrapper is ok but when I run the python >> script runme.py, I have a segmentation fault because of the first line. >> >> Here I really need help. >> Please keep in mind that it is my first try to make a typemap .... >> >> Thanks a lot >> Michel >> _________________________________________________________________ >> Installez gratuitement les 20 ?m?ticones Windows Live Messenger les plus fous ! Cliquez ici ! >> http://www.emoticones-messenger.fr/ >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion >> > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From Chris.Barker at noaa.gov Wed Sep 24 18:34:10 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 24 Sep 2008 15:34:10 -0700 Subject: [Numpy-discussion] A basic question about swig and Numeric In-Reply-To: References: Message-ID: <48DAC062.4000208@noaa.gov> Michel Dupront wrote: > I have a c++ function that take as argument a std::vector. >>From python I want to call the c++ function with an array > object. For that purpose I want to write a typemap. Have you seen the SWIG typemaps that come with numpy?: ...\site-packages\numpy\doc\swig They are designed to work with plain old fashioned C arrays(pointers), but you can either learn form them, or write a little C++ that takes a C pointer and size, creates a vector, and calls your existing functions. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception From Chris.Barker at noaa.gov Wed Sep 24 18:37:03 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 24 Sep 2008 15:37:03 -0700 Subject: [Numpy-discussion] performance of the numpy In-Reply-To: <710F2847B0018641891D9A216027636029C29F@ex3.envision.co.il> References: <200809222010.12025.pgmdevlist@gmail.com> <710F2847B0018641891D9A216027636029C29F@ex3.envision.co.il> Message-ID: <48DAC10F.4040801@noaa.gov> Nadav Horesh wrote: > You should use absolute (a ufunc) and not abs (internal python function): > >>>> plot(absolute(fft(b))) another reason why "import *" is a bad idea: import numpy as np import pylab as plot #(what is the convention for this now?) pylab.plot(np.absolute(np.fft(b))) yes, it's more typing, but you'll never get confused as to what module functions come from. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception From f.yw at hotmail.com Wed Sep 24 20:36:31 2008 From: f.yw at hotmail.com (frank wang) Date: Wed, 24 Sep 2008 18:36:31 -0600 Subject: [Numpy-discussion] performance of the numpy In-Reply-To: <48DAC10F.4040801@noaa.gov> References: <200809222010.12025.pgmdevlist@gmail.com> <710F2847B0018641891D9A216027636029C29F@ex3.envision.co.il> <48DAC10F.4040801@noaa.gov> Message-ID: Hi, All, I am using ipython with --pylab flag. ipython loads the numpy into the workspace, so I do not know abs is from python or numpy. The weird thing is if I execute the code line by line, I do not have any speed problem. But when I combine them together into one command, it slowdonws the computer significantly. >From my understanding, using the modulename.functionname will slow down the python performance. For a big simulation, it may not be a good idear. Are there any suggestion for the matlab uses who want to use numpy/scipy how to setup their working environment? Thanks Frank> Date: Wed, 24 Sep 2008 15:37:03 -0700> From: Chris.Barker at noaa.gov> To: numpy-discussion at scipy.org> Subject: Re: [Numpy-discussion] performance of the numpy> > Nadav Horesh wrote:> > You should use absolute (a ufunc) and not abs (internal python function):> > > >>>> plot(absolute(fft(b)))> > another reason why "import *" is a bad idea:> > import numpy as np> import pylab as plot #(what is the convention for this now?)> > pylab.plot(np.absolute(np.fft(b)))> > yes, it's more typing, but you'll never get confused as to what module > functions come from.> > -Chris> > -- > Christopher Barker, Ph.D.> Oceanographer> > NOAA/OR&R/HAZMAT (206) 526-6959 voice> 7600 Sand Point Way NE (206) 526-6329 fax> Seattle, WA 98115 (206) 526-6317 main reception> _______________________________________________> Numpy-discussion mailing list> Numpy-discussion at scipy.org> http://projects.scipy.org/mailman/listinfo/numpy-discussion _________________________________________________________________ See how Windows connects the people, information, and fun that are part of your life. http://clk.atdmt.com/MRT/go/msnnkwxp1020093175mrt/direct/01/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dineshbvadhia at hotmail.com Wed Sep 24 22:29:53 2008 From: dineshbvadhia at hotmail.com (Dinesh B Vadhia) Date: Wed, 24 Sep 2008 19:29:53 -0700 Subject: [Numpy-discussion] take a numpy Message-ID: I'm using numpy take() to pickout elements from a (1xM) array A indicated by the index numbers in indices ie. B = A.take(indices = list_indexes) It work perfectly but as A is large the performance isn't great and was wondering if there are faster methods available or ways to improve the use of take()? Dinesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Sep 24 23:21:47 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 24 Sep 2008 22:21:47 -0500 Subject: [Numpy-discussion] take a numpy In-Reply-To: References: Message-ID: <3d375d730809242021y754606b9v87e39ec743bf7078@mail.gmail.com> On Wed, Sep 24, 2008 at 21:29, Dinesh B Vadhia wrote: > I'm using numpy take() to pickout elements from a (1xM) array A indicated by > the index numbers in indices ie. > > B = A.take(indices = list_indexes) > > It work perfectly but as A is large the performance isn't great and was > wondering if there are faster methods available or ways to improve the use > of take()? If list_indexes is actually a Python list, then most of your time is probably getting spent in converting the list to an array. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From jason-sage at creativetrax.com Wed Sep 24 23:25:01 2008 From: jason-sage at creativetrax.com (jason-sage at creativetrax.com) Date: Wed, 24 Sep 2008 22:25:01 -0500 Subject: [Numpy-discussion] PyArray_SETITEM macro ends in semicolon Message-ID: <48DB048D.8030203@creativetrax.com> I'm working on getting the Sage matrices for real/complex doubles to use numpy as a backend. In this, I'm using the PyArray_SETITEM macro from within Cython. However, Cython wraps the macro in a function call to convert the output to a Python value: __pyx_1 = PyInt_FromLong(PyArray_SETITEM(__pyx_v_self->_matrix_numpy, PyArray_GETPTR2(__pyx_v_self->_matrix_numpy, __pyx_v_i, __pyx_v_j), __pyx_v_value)); However, after preprocessing, because of the semicolon at the end of the SETITEM macro, we get: PyInt_FromLong(((PyArrayObject *)(__pyx_v_self->_matrix_numpy))->descr->f->setitem((PyObject *)(__pyx_v_value), (char *)(((void *)((((PyArrayObject *)(__pyx_v_self->_matrix_numpy))->data) + (__pyx_v_i)*(((PyArrayObject *)(__pyx_v_self->_matrix_numpy))->strides)[0] + (__pyx_v_j)*(((PyArrayObject *)(__pyx_v_self->_matrix_numpy))->strides)[1]))), (PyArrayObject *)(__pyx_v_self->_matrix_numpy));); Note that at the end, we have a "););". The file refuses to compile. Presumably, since SETITEM returns a value, wrapping the return value in a function call seems to be a reasonable thing to do. Would there be a problem in eliminating the semicolon and instead wrapping the entire function body in parenthesis? I noticed that GETITEM also ended in a semicolon, though I didn't have the same problem as above since Cython didn't automatically wrap it in a function call (I'm not even sure if it returns something). On a side note, is the above the best way (i.e., fastest way given an arbitrary numpy array) to set/get an element? Thanks, Jason From charlesr.harris at gmail.com Thu Sep 25 00:38:45 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 24 Sep 2008 22:38:45 -0600 Subject: [Numpy-discussion] PyArray_SETITEM macro ends in semicolon In-Reply-To: <48DB048D.8030203@creativetrax.com> References: <48DB048D.8030203@creativetrax.com> Message-ID: On Wed, Sep 24, 2008 at 9:25 PM, wrote: > I'm working on getting the Sage matrices for real/complex doubles to use > numpy as a backend. In this, I'm using the PyArray_SETITEM macro from > within Cython. However, Cython wraps the macro in a function call to > convert the output to a Python value: > > __pyx_1 = PyInt_FromLong(PyArray_SETITEM(__pyx_v_self->_matrix_numpy, > PyArray_GETPTR2(__pyx_v_self->_matrix_numpy, __pyx_v_i, __pyx_v_j), > __pyx_v_value)); > > However, after preprocessing, because of the semicolon at the end of the > SETITEM macro, we get: > Hmm, not good. IMHO, macros should be ended with semicolons *when used*, not in themselves. > > > PyInt_FromLong(((PyArrayObject > *)(__pyx_v_self->_matrix_numpy))->descr->f->setitem((PyObject > *)(__pyx_v_value), (char *)(((void *)((((PyArrayObject > *)(__pyx_v_self->_matrix_numpy))->data) + (__pyx_v_i)*(((PyArrayObject > *)(__pyx_v_self->_matrix_numpy))->strides)[0] + > (__pyx_v_j)*(((PyArrayObject > *)(__pyx_v_self->_matrix_numpy))->strides)[1]))), (PyArrayObject > *)(__pyx_v_self->_matrix_numpy));); > > Note that at the end, we have a "););". The file refuses to compile. > Presumably, since SETITEM returns a value, wrapping the return value in > a function call seems to be a reasonable thing to do. Would there be a > problem in eliminating the semicolon and instead wrapping the entire > function body in parenthesis? > We should do this if possible. Open a ticket. > > I noticed that GETITEM also ended in a semicolon, though I didn't have > the same problem as above since Cython didn't automatically wrap it in a > function call (I'm not even sure if it returns something). > > On a side note, is the above the best way (i.e., fastest way given an > arbitrary numpy array) to set/get > an element? > Don't know. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Thu Sep 25 02:01:41 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 25 Sep 2008 08:01:41 +0200 Subject: [Numpy-discussion] performance of the numpy In-Reply-To: References: <200809222010.12025.pgmdevlist@gmail.com> <710F2847B0018641891D9A216027636029C29F@ex3.envision.co.il> <48DAC10F.4040801@noaa.gov> Message-ID: > >From my understanding, using the modulename.functionname will slow down the > python performance. For a big simulation, it may not be a good idear. The slowdown is very small (I'd say inferior to a ms), it's nothing compared to the way you're doing your computations. Matthieu -- French PhD student Information System Engineer Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From jason-sage at creativetrax.com Thu Sep 25 02:03:27 2008 From: jason-sage at creativetrax.com (jason-sage at creativetrax.com) Date: Thu, 25 Sep 2008 01:03:27 -0500 Subject: [Numpy-discussion] numpy.complex* functions do not call the __complex__ method Message-ID: <48DB29AF.6010208@creativetrax.com> In creating an array of type numpy.complex128, I'm having problems passing in Sage types that should be considered complex numbers since they implement the standard __complex__ method. However, numpy doesn't recognize that. Here's a minimal example: In [1]: class MyNum: ...: def __complex__(self): ...: return complex(1,3) ...: ...: In [2]: import numpy In [3]: a=MyNum() In [4]: complex(a) Out[4]: (1+3j) In [5]: numpy.complex128(a) --------------------------------------------------------------------------- Traceback (most recent call last) /home/jason/download/numpy/numpy-1.1.0/src/numpy/ in () : MyNum instance has no attribute '__float__' This is with numpy 1.1. Is this a bug, or is this by design? Thanks, Jason From cournape at gmail.com Thu Sep 25 02:19:04 2008 From: cournape at gmail.com (David Cournapeau) Date: Thu, 25 Sep 2008 15:19:04 +0900 Subject: [Numpy-discussion] performance of the numpy In-Reply-To: References: <200809222010.12025.pgmdevlist@gmail.com> <710F2847B0018641891D9A216027636029C29F@ex3.envision.co.il> <48DAC10F.4040801@noaa.gov> Message-ID: <5b8d13220809242319x48820291rc11eccbe82a0b0d1@mail.gmail.com> On Thu, Sep 25, 2008 at 9:36 AM, frank wang wrote: > Hi, All, > > I am using ipython with --pylab flag. ipython loads the numpy into the > workspace, so I do not know abs is from python or numpy. The weird thing is > if I execute the code line by line, I do not have any speed problem. But > when I combine them together into one command, it slowdonws the computer > significantly. Yes, this is the strange part. Isn't it possible to reproduce this in a self contained script, so that we can reproduce the bug ? cheers, David From stefan at sun.ac.za Thu Sep 25 03:51:46 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 25 Sep 2008 09:51:46 +0200 Subject: [Numpy-discussion] loadtxt error In-Reply-To: References: <200809222010.12025.pgmdevlist@gmail.com> <9911419a0809232336n89d6e4djcc59ad71f28732f5@mail.gmail.com> <9457e7c80809240810l6bf656aey70434d893200aa98@mail.gmail.com> Message-ID: <9457e7c80809250051y755e232bl8ca79847db186b2@mail.gmail.com> 2008/9/24 frank wang : > Thank you very much for all of you. I have downloaded the binary version > 1.2rc and it fixed the problem. > > My special thanks to the person who created the window binary version for > users who do not know or do not have the capacity to build the numpy from > source. That would be David Cournapeau -- one of the few brave enough to venture there. Cheers St?fan From schut at sarvision.nl Thu Sep 25 04:37:56 2008 From: schut at sarvision.nl (Vincent Schut) Date: Thu, 25 Sep 2008 10:37:56 +0200 Subject: [Numpy-discussion] maskedarray: how to force mask to expand In-Reply-To: <200809241237.21287.pgmdevlist@gmail.com> References: <200809241237.21287.pgmdevlist@gmail.com> Message-ID: Pierre GM wrote: > Vincent, > > You should really consider putting an example next time. I must admit that I'm > not sure what you're trying to do, and where/why it fails. Pierre, sorry for that, I was posting hastily before leaving work, and was myself pretty confused about ma's behaviour on this too, so it was hard for me to explain or phrase my question clearly. It just feels a bit strange that ma.array by default gives a mask without shape and of False. I mean, what's the difference then between that and a normal numpy array? If I did not want a mask, I'd use numpy.array. I do want a mask, so I'd expect ma to give me a mask, which it in fact does not (or does, on which we can have different opinions, but a default mask of False imho == nomask == no mask). OK, that being said, I understand the argument of backwards compatibility. I disagree on the argument of speed, because for that the same applies: if I were really concerned about speed, I'd use numpy arrays, keep a separate mask myself, and before any operation I'd get a flattened copy of all my data that is not masked and run the operation on that. IMHO masked arrays are there to trade speed for convenience, so that's what I expect. Just for clarity, to rephrase my question: how do I force ma to give me (always/by default/by some method on a maskedarray) a full shaped mask instead of 'False' or nomask? Because I am sure from the beginning that I'll need this mask in full shape, I want it, and I want to be able to treat it like any normal bool array :-) > > Yes, by default, the mask of a new MaskedArray is set to the value 'nomask', > which is the boolean False. Directly setting an element of the mask in that > condition fails of course. The reasons behind using this behavior are (1) > backward compatibility and (2) speed, as you can bypass a lot of operations > on the mask when it is empty. 1) is clear 2) seems unintuitive to me. I'd say, use numpy arrays then, use .filled() before you do something, or use a flag 'bypass_mask=True', etc. Any of these seem more intuitive to me that what is does now. No offence, I really appreciate your work, just my 2c for a possible future... > > If you need to mask one or several elements, the easiest is not to modify the > mask itself, but to use the the special value `masked`: > >>>> a = ma.array(np.arange(6).reshape(3,2)) > masked_array(data = > [[0 1] > [2 3] > [4 5]], > mask = > False, > fill_value=999999) >>>> # Mask the first element. >>>> a[0,0] = ma.masked Ah, I did not know that one. Does that always work, I mean, with slices, fancy indexing, etc.? Like 'a[a<0 | a>100] = ma.masked'? It's kind of clean to fiddle with the mask of the array without really interacting with the mask itself, if you understand what I mean... :) And is there also a complement, like ma.unmasked? I could not find it (very quick search, I admit)... Or can I use !ma.masked? >>>> a > masked_array(data = > [[-- 1] > [2 3] > [4 5]], > mask = > [[ True False] > [False False] > [False False]], > fill_value=999999) > > This value, `masked`, is also useful to check whether one particular element > is masked: >>>> a[0,0] is ma.masked > True >>>> a[0,1] is ma.masked > False > > You can also force the mask to be full of False with the proper shape by that > way: >>>> a = ma.array(np.arange(6).reshape(3,2) >>>> # Force the mask to have the proper shape and be full of False: >>>> a.mask = False > masked_array(data = > [[0 1] > [2 3] > [4 5]], > mask = > [[False False] > [False False] > [False False]], > fill_value=999999) Ah, now the magic starts... (normal user cap on head, beware): In [9]: am.mask Out[9]: False In [10]: am.mask = False In [11]: am.mask Out[11]: array([[False, False], [False, False]], dtype=bool) while (with the same am as before [9], with am.mask == False): In [15]: am.mask = am.mask In [16]: am.mask Out[16]: False Do you see (and agree with me about) the inconsistency? Setting am.mask with its own value changes that same value of am.mask. While am.mask = am.mask, which on first sight should be the same as am.mask = False, as am.mask==False is True, does *not* change the value of am.mask... > > > The shrink argument of ma.array collapses amask full of False to nomask, once > again for speed reasons. So no, it won't do what you look like to want. I already supposed so... > > I agree that having to deal with nomask is not completely intuitive. However, > it is required for backward compatibility. One day, the class will be ported > to C, and then I'll push to have the mask set to the proper shape ab initio, > because then speed will be less of an issue. Glad that we share opinions about the unintuitiveness... Eagerly awaiting the port to C, not (only) for speed, but mainly for consistency. > > In the meantime, I hope I answered your question. Well, yes and no. To resume: by default, the mask of a masked array (if not given at creation as a bool array) is always 'False'. There is no keyword to force the mask at creation to full shape, and there is no method on a maskedarray to change the mask to full shape. However, one can apply some magic and use 'a.mask' = False directly after creation to force the mask to full shape. This of course only works when the mask already *was* False, otherwise you'll be effectively changing your mask. So we presume ma never by default returns a mask of 'True', and then this works. The obvious trick to workaround this remote possibility of a mask of 'True' would be a.mask = a.mask, but that does not work. Hey, sorry about starting a discussion about this, while I meant to ask just a simple question (and really assumed I had overlooked something, it seemed so simple...). Again, no offence meant, and your work on ma is really appreciated. I hope this discussion will result in more intuitiveness in a future (C?) implementation of ma. Regards, Vincent. > > > On Wednesday 24 September 2008 06:25:57 Vincent Schut wrote: >> Probably I'm just overlooking something obvious, but I'm having problems >> with maskedarrays (numpy.ma from svn: '1.3.0.dev5861'), the mask by >> default being a single bool value ('False') instead of a properly sized >> bool array. If I then try to mask one value by assigning values to >> certain mask positions (a.mask[0,0]=True) I get an error, logically. I >> know I can use mask_where, but I like the mask[...] idiom. And I have to >> expand the mask anyway, as I'm gonna write it to a file at the end. >> >> 1) Is there a way to have ma always use properly expanded masks (bool >> arrays instead of single bool values)? I tried the shrink=False keyword, >> but that does not do what I want, and is not available for >> numpy.ma.zeros, which I conveniently use a lot. >> >> 2) Is there a method/function to request the mask, be it a single bool >> value or an array, as a properly sized array? I found shrink_mask but no >> opposite method, and shrink_mask seems to do something subtly different >> even. >> >> Regards, >> Vincent. >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion From pzs at dcs.gla.ac.uk Thu Sep 25 04:46:25 2008 From: pzs at dcs.gla.ac.uk (Peter Saffrey) Date: Thu, 25 Sep 2008 08:46:25 +0000 (UTC) Subject: [Numpy-discussion] Standard functions (z-score) on nan (again) Message-ID: I've bodged my way through my median problems (see previous postings). Now I need to take a z-score of an array that might contain nans. At the moment, if the array, which is 7000 elements, contains 1 nan or more, all the results come out as nan. My other problem is that my array is indexed from somewhere else - position 0 holds the value for a particular id, position 1 holds the value for a particular id and so on, so if I remove the nans, all the indices are messed up. Can anybody suggest a sensible fix for this problem? I know I should really be using masked arrays, but I can't seem to find a masked-array version of zs. Is there one? Peter From dagss at student.matnat.uio.no Thu Sep 25 07:10:10 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 25 Sep 2008 13:10:10 +0200 Subject: [Numpy-discussion] PyArray_SETITEM macro ends in semicolon In-Reply-To: <48DB048D.8030203@creativetrax.com> References: <48DB048D.8030203@creativetrax.com> Message-ID: <48DB7192.6030208@student.matnat.uio.no> jason-sage at creativetrax.com wrote: > I'm working on getting the Sage matrices for real/complex doubles to use > numpy as a backend. In this, I'm using the PyArray_SETITEM macro from > within Cython. However, Cython wraps the macro in a function call to > convert the output to a Python value: > > __pyx_1 = PyInt_FromLong(PyArray_SETITEM(__pyx_v_self->_matrix_numpy, > PyArray_GETPTR2(__pyx_v_self->_matrix_numpy, __pyx_v_i, __pyx_v_j), > __pyx_v_value)); > > However, after preprocessing, because of the semicolon at the end of the > SETITEM macro, we get: > > > PyInt_FromLong(((PyArrayObject > *)(__pyx_v_self->_matrix_numpy))->descr->f->setitem((PyObject > *)(__pyx_v_value), (char *)(((void *)((((PyArrayObject > *)(__pyx_v_self->_matrix_numpy))->data) + (__pyx_v_i)*(((PyArrayObject > *)(__pyx_v_self->_matrix_numpy))->strides)[0] + > (__pyx_v_j)*(((PyArrayObject > *)(__pyx_v_self->_matrix_numpy))->strides)[1]))), (PyArrayObject > *)(__pyx_v_self->_matrix_numpy));); > > Note that at the end, we have a "););". The file refuses to compile. > Presumably, since SETITEM returns a value, wrapping the return value in > a function call seems to be a reasonable thing to do. Would there be a > problem in eliminating the semicolon and instead wrapping the entire > function body in parenthesis? > > I noticed that GETITEM also ended in a semicolon, though I didn't have > the same problem as above since Cython didn't automatically wrap it in a > function call (I'm not even sure if it returns something). > > On a side note, is the above the best way (i.e., fastest way given an > arbitrary numpy array) to set/get > an element? > If I get to where I want to be with Cython/NumPy integration, there definitely shouldn't be a need to call PyArray_SETITEM from Cython. As you already use Cython, and if you know that your array is real/complex doubles (as indicated by your post), and also know that you have two dimensions (as indicated by your code), then there are definitely faster methods. Have you looked at the Cython numpy array support, as explained in http://wiki.cython.org/tutorials/numpy ? (This was mentored by Robert Bradshaw of Sage too). Your usecase looks like a place where it can be used. I am actively developing this, so if you wish you can send me proposals for changes to the Cython NumPy support which fixes whatever issues you have with it. (Support for access to complex numbers is an obvious one and that is worked on now.) (As an aside, the int returned from PyArray_SETITEM is likely an error return value (though I'm not 100% sure). Converting it to a Python long is really inefficient, type the variable that is assigned to as an int. But that really belongs on the Cython list.) Dag Sverre From ndbecker2 at gmail.com Thu Sep 25 08:51:35 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 25 Sep 2008 08:51:35 -0400 Subject: [Numpy-discussion] numpy.complex* functions do not call the __complex__ method References: <48DB29AF.6010208@creativetrax.com> Message-ID: jason-sage at creativetrax.com wrote: > In creating an array of type numpy.complex128, I'm having problems > passing in Sage types that should be considered complex numbers since > they implement the standard __complex__ method. However, numpy doesn't > recognize that. Here's a minimal example: > I had tried to add my own complex_int class (using c++ code). Although partly successful, I was not able to get x.real to work. The reason is that PyArray_ISCOMPLEX is used in various places, and this is a hard-coded macro. There is no way to extend numpy's complex behavior to support user added types. I wish there was. From haase at msg.ucsf.edu Thu Sep 25 09:56:15 2008 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 25 Sep 2008 15:56:15 +0200 Subject: [Numpy-discussion] how to test if index-tuple is "contained" in tuple of slice objects Message-ID: Hi, Is there a good way to test that e.g. (2,3) is "contained" in [:,3] { == (slice(None), 3) } but not in [:,2] ? Background: I have an nd-image "2d section" viewer that supports line graphics overlays. Currently I can assign a tuple of ints to a given graphic, so that the lines are only shown when the respective 2d-section is displayed. I would like to extend this to support tuples such as (2,slice(None)) to show the graphics for "all z sections" at time 2, assuming the axis order to be time,z,y,x. Thanks for any help, Sebastian Haase From jh at physics.ucf.edu Thu Sep 25 10:19:39 2008 From: jh at physics.ucf.edu (jh at physics.ucf.edu) Date: Thu, 25 Sep 2008 10:19:39 -0400 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: (numpy-discussion-request@scipy.org) References: Message-ID: I hope you'll grab the updated docstrings before tagging. At least grab numpy.__doc__. The main numpy help string didn't adequately point users to alternatives to help() for the ufuncs. --jh-- From nadavh at visionsense.com Thu Sep 25 11:12:05 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Thu, 25 Sep 2008 18:12:05 +0300 Subject: [Numpy-discussion] performance of the numpy References: <200809222010.12025.pgmdevlist@gmail.com><710F2847B0018641891D9A216027636029C29F@ex3.envision.co.il><48DAC10F.4040801@noaa.gov> <5b8d13220809242319x48820291rc11eccbe82a0b0d1@mail.gmail.com> Message-ID: <710F2847B0018641891D9A216027636029C2A9@ex3.envision.co.il> I might bot be a bug, since in numpy abs is a synonym to absolute. So it is possible that in one case the python's abs is called, and in the other case it's the numpy's. An explicit example should help to clarify the problem. Nadav. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? David Cournapeau ????: ? 25-??????-08 09:19 ??: Discussion of Numerical Python ????: Re: [Numpy-discussion] performance of the numpy On Thu, Sep 25, 2008 at 9:36 AM, frank wang wrote: > Hi, All, > > I am using ipython with --pylab flag. ipython loads the numpy into the > workspace, so I do not know abs is from python or numpy. The weird thing is > if I execute the code line by line, I do not have any speed problem. But > when I combine them together into one command, it slowdonws the computer > significantly. Yes, this is the strange part. Isn't it possible to reproduce this in a self contained script, so that we can reproduce the bug ? cheers, David _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3708 bytes Desc: not available URL: From thorsten at partita.dartmouth.edu Thu Sep 25 10:37:40 2008 From: thorsten at partita.dartmouth.edu (thorsten at partita.dartmouth.edu) Date: Thu, 25 Sep 2008 10:37:40 -0400 (EDT) Subject: [Numpy-discussion] lsame_ undefined ... Message-ID: I've been trying to install numpy on my linux system and get the following error message whey I try to import it: Python 2.4.3 (#1, Apr 22 2006, 18:02:44) [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-54)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy Traceback (most recent call last): File "", line 1, in ? File "/usr/local/lib/python2.4/site-packages/numpy/__init__.py", line 103, in ? import linalg File "/usr/local/lib/python2.4/site-packages/numpy/linalg/__init__.py", line 4, in ? from linalg import * File "/usr/local/lib/python2.4/site-packages/numpy/linalg/linalg.py", line 29, in ? from numpy.linalg import lapack_lite ImportError: /usr/local/lib/python2.4/site-packages/numpy/linalg/lapack_lite.so: undefined symbol: lsame_ >>> Is there a way to fix this? Thanks, John Thorstensen -- ---------------------------------------------------------------------------- John R. Thorstensen Department of Physics and Astronomy Dartmouth College 6127 Wilder Laboratory Hanover, NH 03755 Office: 239 Wilder Hall Phone: voice 603-646-2869 FAX 603-646-1446 Hinman box 6127 ---------------------------------------------------------------------------- From david at ar.media.kyoto-u.ac.jp Thu Sep 25 11:16:06 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 26 Sep 2008 00:16:06 +0900 Subject: [Numpy-discussion] lsame_ undefined ... In-Reply-To: References: Message-ID: <48DBAB36.3090502@ar.media.kyoto-u.ac.jp> thorsten at partita.dartmouth.edu wrote: > > Is there a way to fix this? > Which blas/lapack did you use ? Did you built it by yourself, or are you using the one packaged by your OS vendor ? cheers, David From thorsten at partita.dartmouth.edu Thu Sep 25 11:34:32 2008 From: thorsten at partita.dartmouth.edu (thorsten at partita.dartmouth.edu) Date: Thu, 25 Sep 2008 11:34:32 -0400 (EDT) Subject: [Numpy-discussion] lsame_ undefined ... In-Reply-To: <48DBAB36.3090502@ar.media.kyoto-u.ac.jp> Message-ID: David, I built it myself from source about 2 years ago. Has it changed much? Thanks for answering so quickly! John On Fri, 26 Sep 2008, David Cournapeau wrote: > thorsten at partita.dartmouth.edu wrote: > > > > Is there a way to fix this? > > > > Which blas/lapack did you use ? Did you built it by yourself, or are you > using the one packaged by your OS vendor ? > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- ---------------------------------------------------------------------------- John R. Thorstensen Department of Physics and Astronomy Dartmouth College 6127 Wilder Laboratory Hanover, NH 03755 Office: 239 Wilder Hall Phone: voice 603-646-2869 FAX 603-646-1446 Hinman box 6127 ---------------------------------------------------------------------------- From david at ar.media.kyoto-u.ac.jp Thu Sep 25 11:28:38 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 26 Sep 2008 00:28:38 +0900 Subject: [Numpy-discussion] lsame_ undefined ... In-Reply-To: References: Message-ID: <48DBAE26.5040602@ar.media.kyoto-u.ac.jp> thorsten at partita.dartmouth.edu wrote: > David, > > I built it myself from source about 2 years ago. Has it > changed much? It has not changed at all, but your compiler has, and they are not ABI compatible, or maybe you made an error when installing numpy. Which compiler did you use to build BLAS/LAPACK (g77 or gfortran) ? Which version of BLAS/LAPACK: did you use LAPACK 3.0 or LAPACK 3.1.1 ? LAPACK 3.0 BLAS is incomplete, and should not be used (but your error is not a symptom of this problem. cheers, David From thorsten at partita.dartmouth.edu Thu Sep 25 11:47:06 2008 From: thorsten at partita.dartmouth.edu (thorsten at partita.dartmouth.edu) Date: Thu, 25 Sep 2008 11:47:06 -0400 (EDT) Subject: [Numpy-discussion] lsame_ undefined ... In-Reply-To: <48DBAE26.5040602@ar.media.kyoto-u.ac.jp> Message-ID: On Fri, 26 Sep 2008, David Cournapeau wrote: > thorsten at partita.dartmouth.edu wrote: > > David, > > > > I built it myself from source about 2 years ago. Has it > > changed much? > > It has not changed at all, but your compiler has, and they are not ABI > compatible, or maybe you made an error when installing numpy. > > Which compiler did you use to build BLAS/LAPACK (g77 or gfortran) ? g77, it appears. This is a somewhat older Red Hat Enterprise Linux system, before they switched over to gfortran. > Which version of BLAS/LAPACK: did you use LAPACK 3.0 or LAPACK 3.1.1 ? > LAPACK 3.0 BLAS is incomplete, and should not be used (but your error is > not a symptom of this problem. I used 3.1.1. Thanks, John From david at ar.media.kyoto-u.ac.jp Thu Sep 25 11:41:04 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 26 Sep 2008 00:41:04 +0900 Subject: [Numpy-discussion] lsame_ undefined ... In-Reply-To: References: Message-ID: <48DBB110.9080500@ar.media.kyoto-u.ac.jp> thorsten at partita.dartmouth.edu wrote: > > g77, it appears. This is a somewhat older Red Hat Enterprise Linux > system, before they switched over to gfortran. You can't mix both compilers. You have to use either one of them to build *everything*: blas, lapack, numpy (scipy, etc...). g77 and gfortran are not ABI compatible. How did you build numpy (did you give any option to python setup.py build ?) > > I used 3.1.1. Note that 3.1.1 is from February 2007, so if you literally meant two years in your former email, you could not use 3.1.1. I can't remember if lsame is one function which was missing in previous versions of lapack, though. David From pgmdevlist at gmail.com Thu Sep 25 13:02:51 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 25 Sep 2008 13:02:51 -0400 Subject: [Numpy-discussion] maskedarray: how to force mask to expand In-Reply-To: References: <200809241237.21287.pgmdevlist@gmail.com> Message-ID: <200809251302.52864.pgmdevlist@gmail.com> Vincent, The argument of speed (having a default mask of nomask) applies only to the computations inside MaskedArray. Of course, it is still far faster not to use masks but only ndarrays. > Just for clarity, to rephrase my question: how do I force ma to give me > (always/by default/by some method on a maskedarray) a full shaped mask > instead of 'False' or nomask? Because I am sure from the beginning that > I'll need this mask in full shape, I want it, and I want to be able to > treat it like any normal bool array :-) Easy: >>> a = ma.array([1,2,3,4], mask=False) masked_array(data = [1 2 3 4], mask = [False False False False], fill_value=999999) Puzzling ? Not so much. See, by default, the value of the mask parameter is `nomask`. nomask is in fact a 0-size boolean ndarray with value 0 (False). At the creation of the masked array, we check whether a value was given to the mask parameter. If no value is given, we default to `nomask`, and we end up with `a.mask is nomask`. If you force the mask parameter to the boolean False, you're not using `nomask`: in that case, the full mask is created. That won't work with ma.zeros or ma.ones. I could add an extra keyword to deal witht that, but is it really needed when you can simply do a.mask=False ? Note: >>> a=ma.array([1,2,3,]) >>> a.mask is False False >>> a.mask is ma.nomask True >>> a.mask == False True > > If you need to mask one or several elements, the easiest is not to modify > > the mask itself, but to use the the special value `masked`: > Ah, I did not know that one. Does that always work, I mean, with slices, > fancy indexing, etc.? Like 'a[a<0 | a>100] = ma.masked'? It's kind of > clean to fiddle with the mask of the array without really interacting > with the mask itself, if you understand what I mean... :) It does work, try it for yourself. It's actually *the* recommended way to set values to the mask. > And is there also a complement, like ma.unmasked? I could not find it > (very quick search, I admit)... Or can I use !ma.masked? No, there isn't and no you can't. ma.masked is actually a constant defined module-wide and independent of any array. That way, you can test whether an element is masked with `a[..] is masked`. > Ah, now the magic starts... (normal user cap on head, beware): > > In [9]: am.mask > Out[9]: False > > In [10]: am.mask = False > > In [11]: am.mask > Out[11]: > array([[False, False], > [False, False]], dtype=bool) > > while (with the same am as before [9], with am.mask == False): > > In [15]: am.mask = am.mask > > In [16]: am.mask > Out[16]: False > > Do you see (and agree with me about) the inconsistency? No. I'm afraid you're confusing `nomask` and `False`. Once again, nomask is NOT the same thing as False. It's the same value, but not the same object. From thorsten at partita.dartmouth.edu Thu Sep 25 13:42:07 2008 From: thorsten at partita.dartmouth.edu (thorsten at partita.dartmouth.edu) Date: Thu, 25 Sep 2008 13:42:07 -0400 (EDT) Subject: [Numpy-discussion] lsame_ undefined ... In-Reply-To: <48DBB110.9080500@ar.media.kyoto-u.ac.jp> Message-ID: On Fri, 26 Sep 2008, David Cournapeau wrote: > thorsten at partita.dartmouth.edu wrote: > > > > g77, it appears. This is a somewhat older Red Hat Enterprise Linux > > system, before they switched over to gfortran. > > You can't mix both compilers. You have to use either one of them to > build *everything*: blas, lapack, numpy (scipy, etc...). g77 and > gfortran are not ABI compatible. Definitely g77, then -- gfortran doesn't exist on this machine. > How did you build numpy (did you give any option to python setup.py build ?) No options, it was entirely the defaults. > > > > I used 3.1.1. > > Note that 3.1.1 is from February 2007, so if you literally meant two > years in your former email, you could not use 3.1.1. I can't remember if > lsame is one function which was missing in previous versions of lapack, > though. The "two years" was a guess -- it must have been early 2007. I do see "lsame" in the lapack that's on the system. Looking inside /usr/local/lib/python2.4/site-packages/numpy/linalg/lapack_lite.so, I do find the string "lsame_" in a few spots in among the binary. So it looks for all the world like lsame_ should be available, but for some reason it's not being found ("undefined symbol: lsame_"). Thanks again, John -- ---------------------------------------------------------------------------- John R. Thorstensen Department of Physics and Astronomy Dartmouth College 6127 Wilder Laboratory Hanover, NH 03755 Office: 239 Wilder Hall Phone: voice 603-646-2869 FAX 603-646-1446 Hinman box 6127 ---------------------------------------------------------------------------- From charlesr.harris at gmail.com Thu Sep 25 14:58:54 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 25 Sep 2008 12:58:54 -0600 Subject: [Numpy-discussion] lsame_ undefined ... In-Reply-To: References: <48DBB110.9080500@ar.media.kyoto-u.ac.jp> Message-ID: On Thu, Sep 25, 2008 at 11:42 AM, wrote: > On Fri, 26 Sep 2008, David Cournapeau wrote: > > > thorsten at partita.dartmouth.edu wrote: > > > > > > g77, it appears. This is a somewhat older Red Hat Enterprise Linux > > > system, before they switched over to gfortran. > > > > You can't mix both compilers. You have to use either one of them to > > build *everything*: blas, lapack, numpy (scipy, etc...). g77 and > > gfortran are not ABI compatible. > > Definitely g77, then -- gfortran doesn't exist on this machine. > > > How did you build numpy (did you give any option to python setup.py build > ?) > > No options, it was entirely the defaults. > > > > > > > I used 3.1.1. > > > > Note that 3.1.1 is from February 2007, so if you literally meant two > > years in your former email, you could not use 3.1.1. I can't remember if > > lsame is one function which was missing in previous versions of lapack, > > though. > > The "two years" was a guess -- it must have been early 2007. > > I do see "lsame" in the lapack that's on the system. Looking > inside /usr/local/lib/python2.4/site-packages/numpy/linalg/lapack_lite.so, > I do find the string "lsame_" in a few spots in among the binary. > So it looks for all the world like lsame_ should be available, > but for some reason it's not being found ("undefined symbol: lsame_"). > Is there more than one version of blas/lapack on the system? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Thu Sep 25 15:25:55 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Thu, 25 Sep 2008 12:25:55 -0700 Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST-- In-Reply-To: References: Message-ID: On Thu, Sep 25, 2008 at 7:19 AM, wrote: > I hope you'll grab the updated docstrings before tagging. At least > grab numpy.__doc__. The main numpy help string didn't adequately > point users to alternatives to help() for the ufuncs. That will have to wait until 1.2.1, which we will put out shortly. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From rowen at u.washington.edu Thu Sep 25 15:45:49 2008 From: rowen at u.washington.edu (Russell E. Owen) Date: Thu, 25 Sep 2008 12:45:49 -0700 Subject: [Numpy-discussion] A basic question about swig and Numeric References: Message-ID: In article , Michel Dupront wrote: > > Hello, > > I am trying to use Numeric and swig but it seems that > there are few points that I don't understand. > The only excuse I have is that I am new to these tools. > I have a simple example that I cannot make work the > way I would like. > I have a c++ function that take as argument a std::vector. > >From python I want to call the c++ function with an array > object. For that purpose I want to write a typemap. Would this suffice: %include "std_vector.i" %template(vectorF) std::vector; %template(vectorD) std::vector; This will certainly make a SWIGged function accept a list where a vector of floats or doubles is expected (and you can expand the types of course). I'm not sure it'll take a numpy or Numeric array of float or double, but if not then perhaps you can read the code in std_vector.i and see how it works. -- Russell From josef.pktd at gmail.com Thu Sep 25 16:18:06 2008 From: josef.pktd at gmail.com (joep) Date: Thu, 25 Sep 2008 13:18:06 -0700 (PDT) Subject: [Numpy-discussion] numpy.random.hypergeometric - strange results Message-ID: In my fuzz testing of scipy stats, I get sometimes a test failure. I think there is something wrong with numpy.random.hypergeometric for some cases: Josef >>> import numpy.random as mtrand >>> mtrand.hypergeometric(3,17,12,size=10) # there are only 3 good balls in urn array([16, 17, 16, 16, 15, 16, 17, 16, 17, 16]) >>> mtrand.hypergeometric(17,3,12,size=10) #negative result array([-3, -4, -3, -4, -3, -3, -4, -4, -5, -4]) >>> mtrand.hypergeometric(4,3,12,size=10) >>> np.version.version '1.2.0rc2' I did not find any clear pattern when trying out different parameter values: >>> mtrand.hypergeometric(10,10,12,size=10) array([5, 6, 4, 4, 8, 5, 4, 6, 7, 4]) >>> mtrand.hypergeometric(10,10,20,size=10) array([10, 10, 10, 10, 10, 10, 10, 10, 10, 10]) >>> mtrand.hypergeometric(10,10,19,size=10) array([10, 9, 9, 9, 9, 9, 10, 9, 9, 9]) >>> mtrand.hypergeometric(10,10,5,size=10) array([3, 5, 2, 2, 1, 2, 2, 4, 3, 1]) >>> mtrand.hypergeometric(10,2,5,size=10) array([4, 5, 4, 5, 5, 5, 4, 3, 4, 4]) >>> mtrand.hypergeometric(2,10,5,size=10) array([0, 2, 1, 0, 2, 2, 1, 1, 1, 1]) >>> mtrand.hypergeometric(17,3,12,size=10) array([-5, -3, -4, -4, -4, -3, -4, -4, -3, -3]) >>> mtrand.hypergeometric(3,17,12,size=10) array([15, 16, 17, 16, 15, 16, 15, 15, 17, 17]) >>> mtrand.hypergeometric(18,3,12,size=10) array([-5, -6, -6, -4, -4, -4, -5, -3, -5, -5]) >>> mtrand.hypergeometric(18,3,5,size=10) array([4, 5, 5, 5, 5, 5, 4, 5, 4, 3]) >>> mtrand.hypergeometric(18,3,19,size=10) array([1, 1, 2, 1, 1, 1, 1, 3, 1, 1]) From jason-sage at creativetrax.com Thu Sep 25 16:28:50 2008 From: jason-sage at creativetrax.com (jason-sage at creativetrax.com) Date: Thu, 25 Sep 2008 15:28:50 -0500 Subject: [Numpy-discussion] numpy.complex* functions do not call the __complex__ method In-Reply-To: References: <48DB29AF.6010208@creativetrax.com> Message-ID: <48DBF482.4090805@creativetrax.com> Neal Becker wrote: > jason-sage at creativetrax.com wrote: > > >> In creating an array of type numpy.complex128, I'm having problems >> passing in Sage types that should be considered complex numbers since >> they implement the standard __complex__ method. However, numpy doesn't >> recognize that. Here's a minimal example: >> >> > > I had tried to add my own complex_int class (using c++ code). Although partly successful, I was not able to get x.real to work. The reason is that PyArray_ISCOMPLEX is used in various places, and this is a hard-coded macro. There is no way to extend numpy's complex behavior to support user added types. I wish there was. > Is there any possibility of redefining the macro to first call the Python function that converts to a python complex? (I don't know much about the python C API, just throwing an idea out). My example works if I do: numpy.complex128(complex(my_complex_number)), but not if I do numpy.complex128(my_complex_number). Can we make the call to python's complex (or maybe numpy's copy of it) automatic? Thanks, Jason From dblubaugh at belcan.com Thu Sep 25 17:44:12 2008 From: dblubaugh at belcan.com (Blubaugh, David A.) Date: Thu, 25 Sep 2008 17:44:12 -0400 Subject: [Numpy-discussion] F2PY errors still exist Message-ID: <27CC3060AF71DA40A5DC85F7D5B70F38052B29F4@AWMAIL04.belcan.com> To All, I am now trying to use the visual compaq compiler. I am legally required to utilize this compiler for my employment. I have tried the following: C:\Python25\Scripts>C:\python25\python f2py.py -c --fcompiler=compaqv -m hello hello.f90 I was wondering as to if I should input --compiler=msvc. If so, I am also obtaining an error with that as well. I will then obtain the following error!! I am now starting to becomed deeply concerned about F2PY's inability to generate any working executable or DLL files in capacity. The errors are as follows: running build running scons No module named msvccompiler in numpy.distutils; trying from distutils customize MSVCCompiler Found executable C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.e xe customize GnuFCompiler Could not locate executable g77 Found executable C:\Program Files\Microsoft Visual Studio\DF98\BIN\f77.exe gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found Found executable C:\Program Files\Microsoft Visual Studio\DF98\BIN\f77.exe customize IntelVisualFCompiler Could not locate executable ifl customize AbsoftFCompiler Found executable C:\Program Files\Microsoft Visual Studio\DF98\BIN\f90.exe Found executable C:\Program Files\Microsoft Visual Studio\DF98\BIN\f90.exe customize CompaqVisualFCompiler Found executable C:\Program Files\Microsoft Visual Studio\DF98\BIN\DF.exe Found executable C:\Program Files\Microsoft Visual Studio\DF98\BIN\DF.exe customize CompaqVisualFCompiler No module named msvccompiler in numpy.distutils; trying from distutils customize MSVCCompiler Missing compiler_cxx fix for MSVCCompiler customize MSVCCompiler using scons running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler opti ons running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler opt ions running build_src building extension "hello" sources f2py options: [] f2py:> c:\docume~1\david\locals~1\temp\tmpmladol\src.win32-2.5\hellomodule.c creating c:\docume~1\david\locals~1\temp\tmpmladol creating c:\docume~1\david\locals~1\temp\tmpmladol\src.win32-2.5 Reading fortran codes... Reading file 'hello.f90' (format:free) Post-processing... Block: hello Block: foo Post-processing (stage 2)... Building modules... Building module "hello"... Constructing wrapper function "foo"... foo(a) Wrote C/API module "hello" to file "c:\docume~1\david\locals~1\temp\tmpm ladol\src.win32-2.5/hellomodule.c" adding 'c:\docume~1\david\locals~1\temp\tmpmladol\src.win32-2.5\fortranobject. c' to sources. adding 'c:\docume~1\david\locals~1\temp\tmpmladol\src.win32-2.5' to include_di rs. copying C:\python25\lib\site-packages\numpy\f2py\src\fortranobject.c -> c:\docum e~1\david\locals~1\temp\tmpmladol\src.win32-2.5 copying C:\python25\lib\site-packages\numpy\f2py\src\fortranobject.h -> c:\docum e~1\david\locals~1\temp\tmpmladol\src.win32-2.5 running build_ext No module named msvccompiler in numpy.distutils; trying from distutils customize MSVCCompiler customize MSVCCompiler using build_ext customize CompaqVisualFCompiler customize CompaqVisualFCompiler using build_ext building 'hello' extension compiling C sources creating c:\docume~1\david\locals~1\temp\tmpmladol\Release creating c:\docume~1\david\locals~1\temp\tmpmladol\Release\docume~1 creating c:\docume~1\david\locals~1\temp\tmpmladol\Release\docume~1\david creating c:\docume~1\david\locals~1\temp\tmpmladol\Release\docume~1\david\locals ~1 creating c:\docume~1\david\locals~1\temp\tmpmladol\Release\docume~1\david\locals ~1\temp creating c:\docume~1\david\locals~1\temp\tmpmladol\Release\docume~1\david\locals ~1\temp\tmpmladol creating c:\docume~1\david\locals~1\temp\tmpmladol\Release\docume~1\david\locals ~1\temp\tmpmladol\src.win32-2.5 C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c /nologo /Ox /MD /W3 /GX /DNDEBUG -Ic:\docume~1\david\locals~1\temp\tmpmladol\src.win32-2.5 -IC:\python25\lib\site-packages\numpy\core\include -IC:\python25\include -IC:\py thon25\PC /Tcc:\docume~1\david\locals~1\temp\tmpmladol\src.win32-2.5\hellomodule .c /Foc:\docume~1\david\locals~1\temp\tmpmladol\Release\docume~1\david\loca ls~1\ temp\tmpmladol\src.win32-2.5\hellomodule.obj Found executable C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.e xe C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c /nologo /Ox /MD /W3 /GX /DNDEBUG -Ic:\docume~1\david\locals~1\temp\tmpmladol\src.win32-2.5 -IC:\python25\lib\site-packages\numpy\core\include -IC:\python25\include -IC:\py thon25\PC /Tcc:\docume~1\david\locals~1\temp\tmpmladol\src.win32-2.5\fortranobje ct.c /Foc:\docume~1\david\locals~1\temp\tmpmladol\Release\docume~1\david\loca ls~ 1\temp\tmpmladol\src.win32-2.5\fortranobject.obj compiling Fortran sources Fortran f77 compiler: C:\Program Files\Microsoft Visual Studio\DF98\BIN\DF.exe / f77rtl /fixed /nologo /MD /WX /iface=(cref,nomixed_str_len_arg) /names:lowercase /assume:underscore /Ox /fast /optimize:5 /unroll:0 /math_library:fast /threads Fortran f90 compiler: C:\Program Files\Microsoft Visual Studio\DF98\BIN\DF.exe / nologo /MD /WX /iface=(cref,nomixed_str_len_arg) /names:lowercase /assume:unders core /Ox /fast /optimize:5 /unroll:0 /math_library:fast /threads Fortran fix compiler: C:\Program Files\Microsoft Visual Studio\DF98\BIN\DF.exe / fixed /nologo /MD /WX /iface=(cref,nomixed_str_len_arg) /names:lowercase /assume :underscore /Ox /fast /optimize:5 /unroll:0 /math_library:fast /threads compile options: '-Ic:\docume~1\david\locals~1\temp\tmpmladol\src.win32-2.5 -IC: \python25\lib\site-packages\numpy\core\include -IC:\python25\include -IC:\python 25\PC -c' DF.exe:f90: hello.f90 C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\python25\libs /LIBPATH:C:\python25\PCBuild /LIBPATH :C:\python25\libs /LIBPATH:C:\python25\PCBuild /EXPORT:inithello c:\docume~1\dav id\locals~1\temp\tmpmladol\Release\docume~1\david\locals~1\temp\tmpmlado l\src.wi n32-2.5\hellomodule.obj c:\docume~1\david\locals~1\temp\tmpmladol\Release\docume ~1\david\locals~1\temp\tmpmladol\src.win32-2.5\fortranobject.obj c:\docume~1\dav id\locals~1\temp\tmpmladol\Release\hello.o /OUT:.\hello.pyd /IMPLIB:c:\docume~1\ david\locals~1\temp\tmpmladol\Release\docume~1\david\locals~1\temp\tmpml adol\src .win32-2.5\hello.lib Found executable C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\link .exe LINK : fatal error LNK1104: cannot open file 'dformd.lib' error: Command "C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\link. exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\python25\libs /LIBPATH:C:\python25\ PCBuild /LIBPATH:C:\python25\libs /LIBPATH:C:\python25\PCBuild /EXPORT:inithello c:\docume~1\david\locals~1\temp\tmpmladol\Release\docume~1\david\locals~ 1\temp\ tmpmladol\src.win32-2.5\hellomodule.obj c:\docume~1\david\locals~1\temp\tmpmlado l\Release\docume~1\david\locals~1\temp\tmpmladol\src.win32-2.5\fortranob ject.obj c:\docume~1\david\locals~1\temp\tmpmladol\Release\hello.o /OUT:.\hello.pyd /IMP LIB:c:\docume~1\david\locals~1\temp\tmpmladol\Release\docume~1\david\loc als~1\te mp\tmpmladol\src.win32-2.5\hello.lib" failed with exit status 1104 C:\Python25\Scripts> This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated. From peridot.faceted at gmail.com Thu Sep 25 17:59:13 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 25 Sep 2008 17:59:13 -0400 Subject: [Numpy-discussion] Standard functions (z-score) on nan (again) In-Reply-To: References: Message-ID: 2008/9/25 Peter Saffrey : > I've bodged my way through my median problems (see previous postings). Now I > need to take a z-score of an array that might contain nans. At the moment, if > the array, which is 7000 elements, contains 1 nan or more, all the results come > out as nan. > > My other problem is that my array is indexed from somewhere else - position 0 > holds the value for a particular id, position 1 holds the value for a particular > id and so on, so if I remove the nans, all the indices are messed up. Can > anybody suggest a sensible fix for this problem? > > I know I should really be using masked arrays, but I can't seem to find a > masked-array version of zs. Is there one? Just keep an array of indices: c = ~np.isnan(A) Aclean = A[c] indices = np.arange(len(A))[c] Then the original index of Aclean[i] is indices[i]. Anne From dblubaugh at belcan.com Thu Sep 25 18:16:08 2008 From: dblubaugh at belcan.com (Blubaugh, David A.) Date: Thu, 25 Sep 2008 18:16:08 -0400 Subject: [Numpy-discussion] F2PY errors still exist (possible solution) In-Reply-To: <27CC3060AF71DA40A5DC85F7D5B70F380521CBA0@AWMAIL04.belcan.com> References: <27CC3060AF71DA40A5DC85F7D5B70F380521CBA0@AWMAIL04.belcan.com> Message-ID: <27CC3060AF71DA40A5DC85F7D5B70F38052B2A09@AWMAIL04.belcan.com> To All, I can now see as to where the error lies. I apologize for my poor eye sight. The error lies in me using Compaq Visual Fortran 6.0. The error listed from f2py.py after I enter the following commands are the following: Commands Entered >>> C:\Python25\Scripts>C:\python25\python f2py.py -c --fcompiler=compaqv -m hello hello.f90 Error obtained >>> LINK: fatal error LNK1104: cannot open file "dformd.lib" Possible Solution >>> This error results from the Compaq Visual Fortran compiler not being installed with the proper file folder arrangement. The recommended file arrangement for the Compaq Visual Fortran compiler is as follows: Developer Studio Files: C:\Program Files\Microsoft Visual Studio\Common Visual Fortran Files: C:\Program Files\Microsoft Visual Studio\Df98 Shared Visual C++ Files: C:\Program Files\Microsoft Visual Studio\VC98 Should I just uninstall and then reinstall Compaq Visual Fortran 6.) to resolve this issue? Thanks, David Blubaugh -----Original Message----- From: Blubaugh, David A. Sent: Thursday, September 25, 2008 5:44 PM To: 'numpy-discussion at scipy.org' Subject: F2PY errors still exist To All, I am now trying to use the visual compaq compiler. I am legally required to utilize this compiler for my employment. I have tried the following: C:\Python25\Scripts>C:\python25\python f2py.py -c --fcompiler=compaqv -m hello hello.f90 I was wondering as to if I should input --compiler=msvc. If so, I am also obtaining an error with that as well. I will then obtain the following error!! I am now starting to becomed deeply concerned about F2PY's inability to generate any working executable or DLL files in capacity. The errors are as follows: running build running scons No module named msvccompiler in numpy.distutils; trying from distutils customize MSVCCompiler Found executable C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.e xe customize GnuFCompiler Could not locate executable g77 Found executable C:\Program Files\Microsoft Visual Studio\DF98\BIN\f77.exe gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found Found executable C:\Program Files\Microsoft Visual Studio\DF98\BIN\f77.exe customize IntelVisualFCompiler Could not locate executable ifl customize AbsoftFCompiler Found executable C:\Program Files\Microsoft Visual Studio\DF98\BIN\f90.exe Found executable C:\Program Files\Microsoft Visual Studio\DF98\BIN\f90.exe customize CompaqVisualFCompiler Found executable C:\Program Files\Microsoft Visual Studio\DF98\BIN\DF.exe Found executable C:\Program Files\Microsoft Visual Studio\DF98\BIN\DF.exe customize CompaqVisualFCompiler No module named msvccompiler in numpy.distutils; trying from distutils customize MSVCCompiler Missing compiler_cxx fix for MSVCCompiler customize MSVCCompiler using scons running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler opti ons running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler opt ions running build_src building extension "hello" sources f2py options: [] f2py:> c:\docume~1\david\locals~1\temp\tmpmladol\src.win32-2.5\hellomodule.c creating c:\docume~1\david\locals~1\temp\tmpmladol creating c:\docume~1\david\locals~1\temp\tmpmladol\src.win32-2.5 Reading fortran codes... Reading file 'hello.f90' (format:free) Post-processing... Block: hello Block: foo Post-processing (stage 2)... Building modules... Building module "hello"... Constructing wrapper function "foo"... foo(a) Wrote C/API module "hello" to file "c:\docume~1\david\locals~1\temp\tmpm ladol\src.win32-2.5/hellomodule.c" adding 'c:\docume~1\david\locals~1\temp\tmpmladol\src.win32-2.5\fortranobject. c' to sources. adding 'c:\docume~1\david\locals~1\temp\tmpmladol\src.win32-2.5' to include_di rs. copying C:\python25\lib\site-packages\numpy\f2py\src\fortranobject.c -> c:\docum e~1\david\locals~1\temp\tmpmladol\src.win32-2.5 copying C:\python25\lib\site-packages\numpy\f2py\src\fortranobject.h -> c:\docum e~1\david\locals~1\temp\tmpmladol\src.win32-2.5 running build_ext No module named msvccompiler in numpy.distutils; trying from distutils customize MSVCCompiler customize MSVCCompiler using build_ext customize CompaqVisualFCompiler customize CompaqVisualFCompiler using build_ext building 'hello' extension compiling C sources creating c:\docume~1\david\locals~1\temp\tmpmladol\Release creating c:\docume~1\david\locals~1\temp\tmpmladol\Release\docume~1 creating c:\docume~1\david\locals~1\temp\tmpmladol\Release\docume~1\david creating c:\docume~1\david\locals~1\temp\tmpmladol\Release\docume~1\david\locals ~1 creating c:\docume~1\david\locals~1\temp\tmpmladol\Release\docume~1\david\locals ~1\temp creating c:\docume~1\david\locals~1\temp\tmpmladol\Release\docume~1\david\locals ~1\temp\tmpmladol creating c:\docume~1\david\locals~1\temp\tmpmladol\Release\docume~1\david\locals ~1\temp\tmpmladol\src.win32-2.5 C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c /nologo /Ox /MD /W3 /GX /DNDEBUG -Ic:\docume~1\david\locals~1\temp\tmpmladol\src.win32-2.5 -IC:\python25\lib\site-packages\numpy\core\include -IC:\python25\include -IC:\py thon25\PC /Tcc:\docume~1\david\locals~1\temp\tmpmladol\src.win32-2.5\hellomodule .c /Foc:\docume~1\david\locals~1\temp\tmpmladol\Release\docume~1\david\loca ls~1\ temp\tmpmladol\src.win32-2.5\hellomodule.obj Found executable C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.e xe C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c /nologo /Ox /MD /W3 /GX /DNDEBUG -Ic:\docume~1\david\locals~1\temp\tmpmladol\src.win32-2.5 -IC:\python25\lib\site-packages\numpy\core\include -IC:\python25\include -IC:\py thon25\PC /Tcc:\docume~1\david\locals~1\temp\tmpmladol\src.win32-2.5\fortranobje ct.c /Foc:\docume~1\david\locals~1\temp\tmpmladol\Release\docume~1\david\loca ls~ 1\temp\tmpmladol\src.win32-2.5\fortranobject.obj compiling Fortran sources Fortran f77 compiler: C:\Program Files\Microsoft Visual Studio\DF98\BIN\DF.exe / f77rtl /fixed /nologo /MD /WX /iface=(cref,nomixed_str_len_arg) /names:lowercase /assume:underscore /Ox /fast /optimize:5 /unroll:0 /math_library:fast /threads Fortran f90 compiler: C:\Program Files\Microsoft Visual Studio\DF98\BIN\DF.exe / nologo /MD /WX /iface=(cref,nomixed_str_len_arg) /names:lowercase /assume:unders core /Ox /fast /optimize:5 /unroll:0 /math_library:fast /threads Fortran fix compiler: C:\Program Files\Microsoft Visual Studio\DF98\BIN\DF.exe / fixed /nologo /MD /WX /iface=(cref,nomixed_str_len_arg) /names:lowercase /assume :underscore /Ox /fast /optimize:5 /unroll:0 /math_library:fast /threads compile options: '-Ic:\docume~1\david\locals~1\temp\tmpmladol\src.win32-2.5 -IC: \python25\lib\site-packages\numpy\core\include -IC:\python25\include -IC:\python 25\PC -c' DF.exe:f90: hello.f90 C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\python25\libs /LIBPATH:C:\python25\PCBuild /LIBPATH :C:\python25\libs /LIBPATH:C:\python25\PCBuild /EXPORT:inithello c:\docume~1\dav id\locals~1\temp\tmpmladol\Release\docume~1\david\locals~1\temp\tmpmlado l\src.wi n32-2.5\hellomodule.obj c:\docume~1\david\locals~1\temp\tmpmladol\Release\docume ~1\david\locals~1\temp\tmpmladol\src.win32-2.5\fortranobject.obj c:\docume~1\dav id\locals~1\temp\tmpmladol\Release\hello.o /OUT:.\hello.pyd /IMPLIB:c:\docume~1\ david\locals~1\temp\tmpmladol\Release\docume~1\david\locals~1\temp\tmpml adol\src .win32-2.5\hello.lib Found executable C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\link .exe LINK : fatal error LNK1104: cannot open file 'dformd.lib' error: Command "C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\link. exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\python25\libs /LIBPATH:C:\python25\ PCBuild /LIBPATH:C:\python25\libs /LIBPATH:C:\python25\PCBuild /EXPORT:inithello c:\docume~1\david\locals~1\temp\tmpmladol\Release\docume~1\david\locals~ 1\temp\ tmpmladol\src.win32-2.5\hellomodule.obj c:\docume~1\david\locals~1\temp\tmpmlado l\Release\docume~1\david\locals~1\temp\tmpmladol\src.win32-2.5\fortranob ject.obj c:\docume~1\david\locals~1\temp\tmpmladol\Release\hello.o /OUT:.\hello.pyd /IMP LIB:c:\docume~1\david\locals~1\temp\tmpmladol\Release\docume~1\david\loc als~1\te mp\tmpmladol\src.win32-2.5\hello.lib" failed with exit status 1104 C:\Python25\Scripts> This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated. From thorsten at partita.dartmouth.edu Thu Sep 25 18:37:44 2008 From: thorsten at partita.dartmouth.edu (thorsten at partita.dartmouth.edu) Date: Thu, 25 Sep 2008 18:37:44 -0400 (EDT) Subject: [Numpy-discussion] lsame_ undefined ... In-Reply-To: Message-ID: On Thu, 25 Sep 2008, Charles R Harris wrote: > On Thu, Sep 25, 2008 at 11:42 AM, wrote: > > > On Fri, 26 Sep 2008, David Cournapeau wrote: > > > > > thorsten at partita.dartmouth.edu wrote: > > > > > > > > g77, it appears. This is a somewhat older Red Hat Enterprise Linux > > > > system, before they switched over to gfortran. > > > > > > You can't mix both compilers. You have to use either one of them to > > > build *everything*: blas, lapack, numpy (scipy, etc...). g77 and > > > gfortran are not ABI compatible. > > > > Definitely g77, then -- gfortran doesn't exist on this machine. > > > > > How did you build numpy (did you give any option to python setup.py build > > ?) > > > > No options, it was entirely the defaults. > > > > > > > > > > I used 3.1.1. > > > > > > Note that 3.1.1 is from February 2007, so if you literally meant two > > > years in your former email, you could not use 3.1.1. I can't remember if > > > lsame is one function which was missing in previous versions of lapack, > > > though. > > > > The "two years" was a guess -- it must have been early 2007. > > > > I do see "lsame" in the lapack that's on the system. Looking > > inside /usr/local/lib/python2.4/site-packages/numpy/linalg/lapack_lite.so, > > I do find the string "lsame_" in a few spots in among the binary. > > So it looks for all the world like lsame_ should be available, > > but for some reason it's not being found ("undefined symbol: lsame_"). > > > > Is there more than one version of blas/lapack on the system? There appears to be only one. I had attempted an install of scipy at one point, which might have brought something else along, but running "find" from the top of the directory tree shows only a single copy as far as I can tell. John From charlesr.harris at gmail.com Thu Sep 25 20:50:20 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 25 Sep 2008 18:50:20 -0600 Subject: [Numpy-discussion] Bug fix or behavior change? Message-ID: Hi All, Currently subtract for boolean arrays is defined in /**begin repeat * Arithmetic operators * * # OP = ||, ^, &&# * #kind = add, subtract, multiply# */ static void BOOL_ at kind@(char **args, intp *dimensions, intp *steps, void *func) { register intp i; intp is1=steps[0],is2=steps[1],os=steps[2], n=dimensions[0]; char *i1=args[0], *i2=args[1], *op=args[2]; for(i=0; i From jason-sage at creativetrax.com Thu Sep 25 21:07:05 2008 From: jason-sage at creativetrax.com (jason-sage at creativetrax.com) Date: Thu, 25 Sep 2008 20:07:05 -0500 Subject: [Numpy-discussion] PyArray_SETITEM macro ends in semicolon In-Reply-To: <48DB7192.6030208@student.matnat.uio.no> References: <48DB048D.8030203@creativetrax.com> <48DB7192.6030208@student.matnat.uio.no> Message-ID: <48DC35B9.70409@creativetrax.com> Dag Sverre Seljebotn wrote: > jason-sage at creativetrax.com wrote: > >> I'm working on getting the Sage matrices for real/complex doubles to use >> numpy as a backend. In this, I'm using the PyArray_SETITEM macro from >> within Cython. However, Cython wraps the macro in a function call to >> convert the output to a Python value: >> >> __pyx_1 = PyInt_FromLong(PyArray_SETITEM(__pyx_v_self->_matrix_numpy, >> PyArray_GETPTR2(__pyx_v_self->_matrix_numpy, __pyx_v_i, __pyx_v_j), >> __pyx_v_value)); >> >> However, after preprocessing, because of the semicolon at the end of the >> SETITEM macro, we get: >> >> >> PyInt_FromLong(((PyArrayObject >> *)(__pyx_v_self->_matrix_numpy))->descr->f->setitem((PyObject >> *)(__pyx_v_value), (char *)(((void *)((((PyArrayObject >> *)(__pyx_v_self->_matrix_numpy))->data) + (__pyx_v_i)*(((PyArrayObject >> *)(__pyx_v_self->_matrix_numpy))->strides)[0] + >> (__pyx_v_j)*(((PyArrayObject >> *)(__pyx_v_self->_matrix_numpy))->strides)[1]))), (PyArrayObject >> *)(__pyx_v_self->_matrix_numpy));); >> >> Note that at the end, we have a "););". The file refuses to compile. >> Presumably, since SETITEM returns a value, wrapping the return value in >> a function call seems to be a reasonable thing to do. Would there be a >> problem in eliminating the semicolon and instead wrapping the entire >> function body in parenthesis? >> >> I noticed that GETITEM also ended in a semicolon, though I didn't have >> the same problem as above since Cython didn't automatically wrap it in a >> function call (I'm not even sure if it returns something). >> >> On a side note, is the above the best way (i.e., fastest way given an >> arbitrary numpy array) to set/get >> an element? >> >> > > If I get to where I want to be with Cython/NumPy integration, there > definitely shouldn't be a need to call PyArray_SETITEM from Cython. > > As you already use Cython, and if you know that your array is > real/complex doubles (as indicated by your post), and also know that you > have two dimensions (as indicated by your code), then there are > definitely faster methods. Have you looked at the Cython numpy array > support, as explained in > > http://wiki.cython.org/tutorials/numpy > > ? (This was mentored by Robert Bradshaw of Sage too). Your usecase looks > like a place where it can be used. I am actively developing this, so if > you wish you can send me proposals for changes to the Cython NumPy > support which fixes whatever issues you have with it. (Support for > access to complex numbers is an obvious one and that is worked on now.) > > Yes, I'm excited about this. In fact, I held off making the transition to numpy for a while thinking that the buffer interface would be in Sage's Cython any day. However, I realized that the biggest thing this would affect would be the getter and setter function for an array entry (most things are just offloaded to numpy/scipy functions), so I decided to make a push for it and convert the double and complex double matrices over to numpy arrays anyway. I'm really looking forward to seeing the cython buffer access, though. Thanks for all your work; I've been lurking on the mailing list archives and Cython website to track the progress on occasion during the summer. > (As an aside, the int returned from PyArray_SETITEM is likely an error > return value (though I'm not 100% sure). Converting it to a Python long > is really inefficient, type the variable that is assigned to as an int. > But that really belongs on the Cython list.) > That's a really good point. Thanks. It is an error value; I should type it to prevent it from being converted to a python long. Thanks, Jason From david at ar.media.kyoto-u.ac.jp Thu Sep 25 23:34:33 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 26 Sep 2008 12:34:33 +0900 Subject: [Numpy-discussion] lsame_ undefined ... In-Reply-To: References: Message-ID: <48DC5849.5050307@ar.media.kyoto-u.ac.jp> thorsten at partita.dartmouth.edu wrote: > > There appears to be only one. I had attempted an install of scipy > at one point, which might have brought something else along, but > running "find" from the top of the directory tree shows only a single > copy as far as I can tell. > Something went wrong somewhere, but I can't see what. If you are willing to rebuild, I have a small project of mine to build numpy from scratch, including blas/lapack (and ATLAS, FFTW, etc... if you want), with the right options: http://www.ar.media.kyoto-u.ac.jp/members/david/softwares/garnumpy/index.html This should work, and was done precisely for your case (when blas/lapack is not provided by your OS vendor). cheers, David From david at ar.media.kyoto-u.ac.jp Fri Sep 26 00:15:00 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 26 Sep 2008 13:15:00 +0900 Subject: [Numpy-discussion] Proper NaN handling in max and co: a redux Message-ID: <48DC61C4.6020307@ar.media.kyoto-u.ac.jp> Hi, We started a small document to gather all information given in previous threads about NaN handling in max and co in numpy. Thanks to Anne (Archibald) for useful comments/additions/typo corrections: http://projects.scipy.org/scipy/numpy/wiki/ProperNanHandling We describe approaches taken by R and matlab for the relevant functions, and possible approaches to take in numpy. I have an patch almost ready (with tests) for the approach "Returning NaN" (Chuck said he would be willing to implement the sort/argsort part, which is the part missing in my current implementation). I think it would be nice for 1.3.0, cheers, David From charlesr.harris at gmail.com Fri Sep 26 00:53:53 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 25 Sep 2008 22:53:53 -0600 Subject: [Numpy-discussion] Proper NaN handling in max and co: a redux In-Reply-To: <48DC61C4.6020307@ar.media.kyoto-u.ac.jp> References: <48DC61C4.6020307@ar.media.kyoto-u.ac.jp> Message-ID: On Thu, Sep 25, 2008 at 10:15 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Hi, > > We started a small document to gather all information given in > previous threads about NaN handling in max and co in numpy. Thanks to > Anne (Archibald) for useful comments/additions/typo corrections: > > http://projects.scipy.org/scipy/numpy/wiki/ProperNanHandling > > We describe approaches taken by R and matlab for the relevant functions, > and possible approaches to take in numpy. I have an patch almost ready > (with tests) for the approach "Returning NaN" (Chuck said he would be > willing to implement the sort/argsort part, which is the part missing in > my current implementation). > > I think it would be nice for 1.3.0, > I'm also wondering about the sign ufunc. It should probably return nan for nans, but -1,0,1 are the current values. We also need to decide which end of the sorted values the nans should go to. I'm a bit partial to the beginning but the end would be fine with me, it might even be a bit more natural as searchsorted would find the beginning of the nans by default. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Fri Sep 26 00:56:34 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 26 Sep 2008 13:56:34 +0900 Subject: [Numpy-discussion] Proper NaN handling in max and co: a redux In-Reply-To: References: <48DC61C4.6020307@ar.media.kyoto-u.ac.jp> Message-ID: <48DC6B82.9080201@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > I'm also wondering about the sign ufunc. It should probably return nan > for nans, but -1,0,1 are the current values. We also need to decide > which end of the sorted values the nans should go to. I'm a bit > partial to the beginning but the end would be fine with me, it might > even be a bit more natural as searchsorted would find the beginning of > the nans by default. Note that in both suggest approaches, sort with NaN would raise an error. We could then provide a nansort, which ignore the NaN, and deal with your case; would it be hard to have an option for the beginning vs the end, or would that be difficult ? cheers, David From millman at berkeley.edu Fri Sep 26 02:07:41 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Thu, 25 Sep 2008 23:07:41 -0700 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 Message-ID: I'm pleased to announce the release of NumPy 1.2.0. NumPy is the fundamental package needed for scientific computing with Python. It contains: * a powerful N-dimensional array object * sophisticated (broadcasting) functions * basic linear algebra functions * basic Fourier transforms * sophisticated random number capabilities * tools for integrating Fortran code. Besides it's obvious scientific uses, NumPy can also be used as an efficient multi-dimensional container of generic data. Arbitrary data-types can be defined. This allows NumPy to seamlessly and speedily integrate with a wide-variety of databases. This minor release comes almost four months after the 1.1.0 release. The major features of this release are a new testing framework and huge amount of documentation work. It also includes a some minor API breakage scheduled in the 1.1 release. Please note that NumPy 1.2.0 requires Python 2.4 or greater. For information, please see the release notes: http://sourceforge.net/project/shownotes.php?group_id=1369&release_id=628858 You can download the release from here: http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 Thank you to everybody who contributed to this release. Enjoy, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From david at ar.media.kyoto-u.ac.jp Fri Sep 26 02:33:03 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 26 Sep 2008 15:33:03 +0900 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: Message-ID: <48DC821F.7020401@ar.media.kyoto-u.ac.jp> Jarrod Millman wrote: > For information, please see the release notes: > http://sourceforge.net/project/shownotes.php?group_id=1369&release_id=628858 > > You can download the release from here: > http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 Note that updated packages for various linux distributions (Fedora, Centos/RHEL, OpenSuse) are available: http://download.opensuse.org/repositories/home:/ashigabou/ Please consider using this before building your own if you are not confortable with building numpy and more importantly blas/lapack from sources, thanks, David P.S Some repositories are broken (Fedora 9, etc... the packages are not built, so there is no chance of breaking anything); I unfortunately can't fix them ATM, but I will as soon as the issues preventing me from fixing it will be solved. From schut at sarvision.nl Fri Sep 26 03:45:55 2008 From: schut at sarvision.nl (Vincent Schut) Date: Fri, 26 Sep 2008 09:45:55 +0200 Subject: [Numpy-discussion] maskedarray: how to force mask to expand In-Reply-To: <200809251302.52864.pgmdevlist@gmail.com> References: <200809241237.21287.pgmdevlist@gmail.com> <200809251302.52864.pgmdevlist@gmail.com> Message-ID: Pierre, Thanks for your explanations. It still seems a little (too) complicated, but from a backwards-compatibility pov combined with your 'nomask is not False' implementation detail, I can understand mostly :-) I think the idea that when a.mask returns False, that actually means nomask instead of the False I'm used to, is what caused a major part of my confusion. It might actually be nice to give you some context of why I asked this: during my (satellite image) processing, I use maskedarrays by default for each step in the processing chain, and I need to save the result of each step to disk. That means saving the array and its mask. I save both of them as tiff files (because these can include all other info that is nice for satellite imagery, like coordinates and projection). When saving the mask, I'm creating a tiff file and pushing the .mask array into it. Therefore, I obviously need the .mask not to be nomask, but to be a full shaped array. That's the context you need to see my confusion in. Because speed and memory don't matter that much to me (well, they to matter, but I'm processing huge amounts of data anyways, and using parallel/clustered processing anyways, so I can take those masks...), I thought it might be easiest to make sure my data always has a full shaped mask. But, of course, perfomance-wise it would be best to be able to work with nomasks, and only expand these to full shaped masks when writing to disk. That's why I asked for a possible method on an ma to force expanding a mask, or e.g. an ma.mask.as_full_shaped_mask() method that returns either the mask, or (if nomask) a new array of Falses. I just supposed it existed and I could not find it, but now I understand it does not exist. But I could easily write something that checks for nomask and always returns an expanded mask. The 'trick' to create ma's with the mask=False keyword is neat, I had not thought about that. Same applies for masking values using ma[idx] = ma.masked. Just for completeness (in case someone else is reading this and wondering how to *unmask* values): just setting ma[idx] to some valid number will unset the mask for that index. No need to do ma[idx] = ma.unmask or whatever, just ma[idx] = v. OK, top posting is bad :) Further comments inline. Pierre GM wrote: > Vincent, > > The argument of speed (having a default mask of nomask) applies only to the > computations inside MaskedArray. Of course, it is still far faster not to use > masks but only ndarrays. > >> Just for clarity, to rephrase my question: how do I force ma to give me >> (always/by default/by some method on a maskedarray) a full shaped mask >> instead of 'False' or nomask? Because I am sure from the beginning that >> I'll need this mask in full shape, I want it, and I want to be able to >> treat it like any normal bool array :-) > > Easy: >>>> a = ma.array([1,2,3,4], mask=False) > masked_array(data = [1 2 3 4], > mask = [False False False False], > fill_value=999999) > > Puzzling ? Not so much. See, by default, the value of the mask parameter is > `nomask`. nomask is in fact a 0-size boolean ndarray with value 0 (False). At > the creation of the masked array, we check whether a value was given to the > mask parameter. If no value is given, we default to `nomask`, and we end up > with `a.mask is nomask`. If you force the mask parameter to the boolean > False, you're not using `nomask`: in that case, the full mask is created. I understand that now. > > That won't work with ma.zeros or ma.ones. I could add an extra keyword to deal > witht that, but is it really needed when you can simply do a.mask=False ? No real need for that, then. Would be just conveniencewise sugar-addition on (especially my) cake. > > Note: >>>> a=ma.array([1,2,3,]) >>>> a.mask is False > False >>>> a.mask is ma.nomask > True >>>> a.mask == False > True > Btw, in future versions, would it be an idea to separate 'nomask' and 'False' a little more? I always assumed (correctly?) that in python, True and False are singletons (as far as that is possible in python), just like None. 'False is False' should always compare to True, then. In this case (a.mask is False) it at least *seems* to break that 'rule'... >> And is there also a complement, like ma.unmasked? I could not find it >> (very quick search, I admit)... Or can I use !ma.masked? Can just set elements to unmask them, found out that. No need for ma.unmasked. > > No, there isn't and no you can't. ma.masked is actually a constant defined > module-wide and independent of any array. That way, you can test whether an > element is masked with `a[..] is masked`. > >> Ah, now the magic starts... (normal user cap on head, beware): >> >> In [9]: am.mask >> Out[9]: False >> >> In [10]: am.mask = False >> >> In [11]: am.mask >> Out[11]: >> array([[False, False], >> [False, False]], dtype=bool) >> >> while (with the same am as before [9], with am.mask == False): >> >> In [15]: am.mask = am.mask >> >> In [16]: am.mask >> Out[16]: False >> >> Do you see (and agree with me about) the inconsistency? > > No. I'm afraid you're confusing `nomask` and `False`. Once again, nomask is > NOT the same thing as False. It's the same value, but not the same object. Exactly. It might not be inconsistent, but imho it does a lot of effort to /feel/ inconsistent to Joe Average. And that's what was causing a lot of confusion. For example, why doesn't a.mask return 'nomask' instead of 'False'? That would have saved me some head-scratching... See also my comment earlier on assuming False to be a python-wide constant (singleton). "It's the same value, but not the same object" imho breaks this python guarantee (though I admit that I'll have to look that up, this guarantee might just be there only in my head...) Well, so far so good. My problems have been solved largely. The philosophical discussion could go on... Vincent. From meine at informatik.uni-hamburg.de Fri Sep 26 05:50:56 2008 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Fri, 26 Sep 2008 11:50:56 +0200 Subject: [Numpy-discussion] Help speeding up element-wise operations for video processing In-Reply-To: <61ADA597-D8B2-48DC-A48E-EA8F2031540B@yahoo.ca> References: <458186.45235.qm@web51601.mail.re2.yahoo.com> <9457e7c80809160150i5acb33d7t95193c58a27ab76d@mail.gmail.com> <61ADA597-D8B2-48DC-A48E-EA8F2031540B@yahoo.ca> Message-ID: <200809261150.57982.meine@informatik.uni-hamburg.de> Am Mittwoch, 17. September 2008 01:43:45 schrieb Brendan Simons: > On 16-Sep-08, at 4:50 AM, St?fan van der Walt wrote: > > I may be completely off base here, but you should be able to do this > > *very* quickly using your GPU, or even just using OpenGL. Otherwise, > > coding it up in ctypes is easy as well (I can send you a code snippet, > > if you need). > > I thought so too. I briefly researched using the GPU, but I found > that, surprisingly, neither multitexturing in pyopengl, nor surface > blitting in pygame was any faster than the numpy code I posted above. It should be really fast and your algorithm is very suited because you need the input matrix(texture) multiple times. This is very similar to the approach of this article: Random Walks for Interactive Alpha-Matting L. Grady, T. Schiwietz, S. Aharon (USA), and R. Westermann (Germany) which appeared in the Proc. VIIP 2005 (IASTED) > That could be because I'm working on a machine with an integrated > graphics chip. I guess so. -- Ciao, / / /--/ / / ANS From jrs65 at cam.ac.uk Fri Sep 26 06:33:10 2008 From: jrs65 at cam.ac.uk (Richard Shaw) Date: Fri, 26 Sep 2008 11:33:10 +0100 Subject: [Numpy-discussion] Cython: Passing numpy array into C via buffer? Message-ID: <48DCBA66.1080008@cam.ac.uk> Hello, Forgive me if this is a stupid question, I've been looking around all the Cython documentation and I can't find out if this is possible. What I would like to do is generally is wrap a C function that takes a double array, and be able to pass in a numpy array, I was wondering if it's possible to do this using the buffer interface? I understand I could do it using this method (http://wiki.cython.org/WrappingNumpy), but the buffer interface seems much neater. Specifically, I have a C function that I would like to wrap with signature: #### cfile.h double do_something(double * array, int len); I was hoping with the buffer interface, I could do something like this: #### pyfile.pyx import numpy cimport numpy cdef extern from "cfile.h" double do_something(double * array, int len) cdef do_something_python(numpy.ndarray[numpy.float64, ndim = 1] numarr): cdef double x cdef double * data cdef int l data = ... # Not sure what to do here l = numarr.shape[0] x = do_something(data, l) The above probably doesn't work for many reasons (I've just jotted it down), but the think I can't figure out is what to do at the "data = ..." line. Any help would be greatly appreciated! Thanks, Richard From dagss at student.matnat.uio.no Fri Sep 26 06:51:43 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 26 Sep 2008 12:51:43 +0200 Subject: [Numpy-discussion] Cython: Passing numpy array into C via buffer? In-Reply-To: <48DCBA66.1080008@cam.ac.uk> References: <48DCBA66.1080008@cam.ac.uk> Message-ID: <48DCBEBF.3040701@student.matnat.uio.no> Richard Shaw wrote: > Hello, > > Forgive me if this is a stupid question, I've been looking around all > the Cython documentation and I can't find out if this is possible. > > What I would like to do is generally is wrap a C function that takes a > double array, and be able to pass in a numpy array, I was wondering if > it's possible to do this using the buffer interface? I understand I > could do it using this method (http://wiki.cython.org/WrappingNumpy), > but the buffer interface seems much neater. Questions like this should be posted to the Cython list (otherwise us Cython developers might easily ignore it, we do not follow everything that goes on in numpy-discussion). > Specifically, I have a C function that I would like to wrap with signature: > > #### cfile.h > double do_something(double * array, int len); > > I was hoping with the buffer interface, I could do something like this: > > #### pyfile.pyx > import numpy > cimport numpy > > cdef extern from "cfile.h" > double do_something(double * array, int len) > > cdef do_something_python(numpy.ndarray[numpy.float64, ndim = 1] numarr): numpy.float64_t, currently at least. > cdef double x > cdef double * data > cdef int l > > data = ... # Not sure what to do here > l = numarr.shape[0] Yes, I understand this. I definitely plan to make this easier in a coming Cython release, i.e. something like cimport cython ... data = cython.buffer.bufptr(array) I just haven't got there yet. For now, there's an ugly hack you can do (untested though): your_header.h: #define GET_BUFPTR_HACK(x) __pyx_bstruct_#x.buf yourfile.pyx: cdef extern from "your_header.h": void* GET_BUFPTR_HACK(x) ... data = GET_BUFPTR_HACK(buf) (The reason this is so horribly ugly is the "#x" in the macro, meaning this depends very much of the particular mangling scheme used. This will likely break in a future version of Cython, but then cython.buffer.butptr should be available.) -- Dag Sverre From dagss at student.matnat.uio.no Fri Sep 26 06:57:44 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 26 Sep 2008 12:57:44 +0200 Subject: [Numpy-discussion] Cython: Passing numpy array into C via buffer? In-Reply-To: <48DCBA66.1080008@cam.ac.uk> References: <48DCBA66.1080008@cam.ac.uk> Message-ID: <48DCC028.1010507@student.matnat.uio.no> Richard Shaw wrote: > Hello, > > Forgive me if this is a stupid question, I've been looking around all > the Cython documentation and I can't find out if this is possible. > > What I would like to do is generally is wrap a C function that takes a > double array, and be able to pass in a numpy array, I was wondering if > it's possible to do this using the buffer interface? I understand I > could do it using this method (http://wiki.cython.org/WrappingNumpy), > but the buffer interface seems much neater. > > Specifically, I have a C function that I would like to wrap with signature: > > #### cfile.h > double do_something(double * array, int len); > > I was hoping with the buffer interface, I could do something like this: > > #### pyfile.pyx > import numpy > cimport numpy > > cdef extern from "cfile.h" > double do_something(double * array, int len) > > cdef do_something_python(numpy.ndarray[numpy.float64, ndim = 1] numarr): > cdef double x > cdef double * data > cdef int l > > data = ... # Not sure what to do here > l = numarr.shape[0] > > x = do_something(data, l) I'm sorry, my previous answer was a really, really bad one. Here's a better one: data = numarr.data :-) With the addition of the _t on float64 this should just work. Please insert an extra check to make sure that your data is contiguous though! Otherwise you need to make a copy to pass the data to an external lib (one way is by using numpy.array with a contiguouos argument to construct a copy, and then copy the data back again afterwards). -- Dag Sverre From jrs65 at cam.ac.uk Fri Sep 26 07:01:29 2008 From: jrs65 at cam.ac.uk (Richard Shaw) Date: Fri, 26 Sep 2008 12:01:29 +0100 Subject: [Numpy-discussion] Cython: Passing numpy array into C via buffer? In-Reply-To: <48DCC028.1010507@student.matnat.uio.no> References: <48DCBA66.1080008@cam.ac.uk> <48DCC028.1010507@student.matnat.uio.no> Message-ID: <48DCC109.4030104@cam.ac.uk> Dag Sverre Seljebotn wrote: > I'm sorry, my previous answer was a really, really bad one. Here's a > better one: > > data = numarr.data > > :-) With the addition of the _t on float64 this should just work. That sounds great. I thought there must be something like that, it's glad to see that Cython doesn't disappoint. I'll get straight to trying it out. > Please insert an extra check to make sure that your data is contiguous > though! Otherwise you need to make a copy to pass the data to an > external lib (one way is by using numpy.array with a contiguouos > argument to construct a copy, and then copy the data back again afterwards). That make sense. I'll be sure to do check that one. Thanks for your very swift response, it's really helpful. I'll be sure to post to the Cython-dev list in future if that's more appropriate. Cheers, Richard From charlesr.harris at gmail.com Fri Sep 26 08:29:02 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 26 Sep 2008 06:29:02 -0600 Subject: [Numpy-discussion] Proper NaN handling in max and co: a redux In-Reply-To: <48DC6B82.9080201@ar.media.kyoto-u.ac.jp> References: <48DC61C4.6020307@ar.media.kyoto-u.ac.jp> <48DC6B82.9080201@ar.media.kyoto-u.ac.jp> Message-ID: On Thu, Sep 25, 2008 at 10:56 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > I'm also wondering about the sign ufunc. It should probably return nan > > for nans, but -1,0,1 are the current values. We also need to decide > > which end of the sorted values the nans should go to. I'm a bit > > partial to the beginning but the end would be fine with me, it might > > even be a bit more natural as searchsorted would find the beginning of > > the nans by default. > > Note that in both suggest approaches, sort with NaN would raise an > error. We could then provide a nansort, which ignore the NaN, and deal > with your case; would it be hard to have an option for the beginning vs > the end, or would that be difficult ? > It shouldn't be any more difficult to do either based on a keyword. Argsorts shouldn't be a problem either. I'm thinking that the most flexible way to handle the sorts is to make a preliminary pass through the data and collect all the nans at one end or the other and sort the remainder. That would also make it easy to raise an error if we wanted to and avoid an extra compare in all the other sorting passes. That approach could probably be extended to masked arrays also. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Sep 26 11:35:46 2008 From: josef.pktd at gmail.com (joep) Date: Fri, 26 Sep 2008 08:35:46 -0700 (PDT) Subject: [Numpy-discussion] vectorize in new.instancemethod and *args Message-ID: <92bb090b-b44a-4c77-a5be-161bacd2f1d8@d45g2000hsc.googlegroups.com> I have a question about the use of vectorize in new.instancemethod: Using vectorize with *args requires adjustment to the number of arguments, nin = nargs. This works when it is used with a function. However, I don't manage to set nin when using vectorize with a method created with new.instancemethod. I would like to do: def funcm(self,x,*args): ... class D(object): def __init__(self): vecfunc = vectorize(funcm) vecfunc.nin = 2 self.funcm = new.instancemethod(vecfunc,self,A) But, calling D().funcm still causes a value error. I managed to do it by setting vecfunc.nin = None. What is the correct or best way to do this? I needed it when I wanted to correct scipy.stats.distributions. Below (and as attachment) is a script that summarizes what I figured out about the use of vectorize with *args. I did not find any documentation for this case, I got the information by trial and error and hints in the bug reports. Josef ''' Nin adjustment in np.vectorize with *args ========================================= * Problem: simple use of vectorize with *args raises exception:: File "C:\Programs\Python24\Lib\site-packages\numpy\lib \function_base.py", line 1636, in __call__ raise ValueError, "mismatch between python function inputs"\ ValueError: mismatch between python function inputs and received arguments * solution adjust nin directly, either set nin to the correct number of arguments or set nin to None * with functions both ways of adjusting nin work, (case 2 and 3) * with new.instance method, the only way, I managed to get it to work is by setting nin = None (case E). Setting nin = nargs did not work (case C and D) with functions -------------- * case 0: function without vectorize * case 1: function with vectorize -> broken * case 2: function with vectorize, with nin adjustment -> works * case 3: function with vectorize, with nin adjustment, nin = None -> works with class and new.instancemethod --------------------------------- * case A: class without vectorize -> works * case B: class with vectorize -> broken: argument miss match * case C: class with vectorize with nin adjustment -> broken * case D: class with vectorize with nin adjustment -> broken * case E: class with vectorize with nin adjustment, nin = None -> works Motivation: ----------- vectorize (sgf) in scipy.stats.distribution does not work in cases where there is no correct nin adjustment. Example for new.instancemethod, that looks broken, is self._ppf in class rv_discrete. ''' import new import numpy as np from numpy import vectorize def func1(x,*args): print 'args = ', args print 'x = ', x return np.sum(x) def funcm(self,x,*args): ''' function for use as instance method''' print 'args = ', args print 'x = ', x return np.sum(x) # case 0: function without vectorize print 'func1(1,*(2,))' func1(1,*(2,)) # case 1: function with vectorize -> broken vecfunc = vectorize(func1) print 'vecfunc(1):' vecfunc(1) print 'vecfunc(1,3):' try: vecfunc(1,3) except ValueError,e: print e print 'vecfunc(1,*(2,))' try: vecfunc(1,*(2,)) except ValueError,e: print e # case 2: function with vectorize, with nin adjustment -> works vecfunc2 = vectorize(func1) vecfunc2.nin = 2 print 'vecfunc2(1):' vecfunc2(1) print 'vecfunc2(1,3):' vecfunc2(1,3) print 'vecfunc2(1,*(2,))' vecfunc2(1,*(2,)) # case 3: function with vectorize, with nin adjustment, nin = None -> works vecfunc3 = vectorize(func1) vecfunc3.nin = None print 'vecfunc3(1):' vecfunc3(1) print 'vecfunc3(1,3):' vecfunc3(1,3) print 'vecfunc3(1,*(2,))' vecfunc3(1,*(2,)) print 'vecfunc3(1,*(2,5))' vecfunc3(1,*(2,5)) # with class and new.instancemethod # case A: class without vectorize -> works class A(object): def __init__(self): self.funcm = new.instancemethod(funcm,self,A) aa = A() #print dir(aav) print 'A: aav.funcm(5)' aa.funcm(5) print 'A: aav.funcm(5,2)' aa.funcm(5,2) print 'A: aav.funcm(5,*(2,))' aa.funcm(5,*(2,)) # case B: class with vectorize -> broken: argument miss match class B(object): def __init__(self): self.funcm = new.instancemethod(vectorize(funcm),self,A) aav = B() #print dir(aav) print 'B: aav.funcm(5)' aav.funcm(5) print 'B: aav.funcm(5,2)' try: aav.funcm(5,2) except ValueError,e: print e print 'B: aav.funcm(5,*(2,))' try: aav.funcm(5,*(2,)) except ValueError,e: print e # case C: class with vectorize with nin adjustment -> broken # AttributeError: 'instancemethod' object has no attribute 'nin' class C(object): def __init__(self): self.funcm = new.instancemethod(vectorize(funcm),self,A) #self.funcm.nin = 2 try: self.funcm.nin = 2 except AttributeError,e: print e aav = C() #print dir(aav) print 'C: aav.funcm(5)' aav.funcm(5) print 'C: aav.funcm(5,2)' try: aav.funcm(5,2) except ValueError,e: print e print 'C: aav.funcm(5,*(2,))' try: aav.funcm(5,*(2,)) except ValueError,e: print e # case D: class with vectorize with nin adjustment -> broken # nin is not correctly used by vectorize class D(object): def __init__(self): # define the vectorized function vecfunc = vectorize(funcm) vecfunc.nin = 2 self.funcm = new.instancemethod(vecfunc,self,A) #self.funcm.nin = 2 try: self.funcm.nin = 2 except AttributeError,e: print e aav = D() #print dir(aav) print 'D: aav.funcm(5)' aav.funcm(5) print 'D: aav.funcm(5,2)' try: aav.funcm(5,2) except ValueError,e: print e print 'D: aav.funcm(5,*(2,))' try: aav.funcm(5,*(2,)) except ValueError,e: print e # case E: class with vectorize with nin adjustment, nin = None -> works # nin is calculated by vectorize class E(object): def __init__(self): vecfunc = vectorize(funcm,otypes='d') vecfunc.nin = None # remove nin at let vectorize do the work self.funcm = new.instancemethod(vecfunc,self,A) #self.funcm.nin = 2 try: self.funcm.nin = 2 except AttributeError,e: print e aav = E() #print dir(aav) print 'E: aav.funcm(5)' aav.funcm(5) print 'E: aav.funcm(5,2)' try: aav.funcm(5,2) except ValueError,e: print e print 'E: aav.funcm(5,*(2,))' try: aav.funcm(5,*(2,)) except ValueError,e: print e aav.funcm([1,2,5.2]) aav.funcm([1,2,5.2],*(2,)) res = aav.funcm(np.array([1,2,5.2]),*(2,)) print res From pgmdevlist at gmail.com Fri Sep 26 11:45:27 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 26 Sep 2008 11:45:27 -0400 Subject: [Numpy-discussion] maskedarray: how to force mask to expand In-Reply-To: References: <200809251302.52864.pgmdevlist@gmail.com> Message-ID: <200809261145.27139.pgmdevlist@gmail.com> Vincent, > I think the > idea that when a.mask returns False, that actually means nomask instead > of the False I'm used to, is what caused a major part of my confusion. Indeed. > It might actually be nice to give you some context of why I asked this: > during my (satellite image) processing, I use maskedarrays by default > for each step in the processing chain Ah. You'll probably notice it might be faster to process the array and its mask in parallel, and then recreate a masked array at the end. If you need to save a mask that hasn't been set yet, you could use the `ma.getmaskarray` command instead of trying to access the mask as an attribute, that is, use: >>> mask=ma.getmaskarray(a) instead of >>> mask=a.mask or >>> mask=ma.getmask(a) `ma.getmaskarray(a)` always return an array with the same shape as `a`, even if full of False. If you need to create a new mask from the shape and dtype of `a`, you can use `ma.make_mask_none(a)`. Both functions are documented. > force expanding a mask, or e.g. an ma.mask.as_full_shaped_mask() method > that returns either the mask, or (if nomask) a new array of Falses. I > just supposed it existed and I could not find it, but now I understand > it does not exist. Actually, it does, that's the ma.getmaskarray() I was just telling you about. > Just for completeness (in case someone else is reading this and > wondering how to *unmask* values): just setting ma[idx] to some valid > number will unset the mask for that index. No need to do ma[idx] = > ma.unmask or whatever, just ma[idx] = v. Exactly. Except that for the sake of completeness, there's a little known attribute (_hardmask) that prevents you to unmask some data by mistake. By default, `_hardmask` is False, which means that if you set a masked value to a non-masked one, you update the mask as well. If you set a._hardmask to True or use the `harden_mask` method, you won't be able to unmask the data that way, you'll have to modify the mask directly. It's useful when the mask cannot or should not be changed. > Btw, in future versions, would it be an idea to separate 'nomask' and > 'False' a little more? I always assumed (correctly?) that in python, > True and False are singletons (as far as that is possible in python), > just like None. 'False is False' should always compare to True, then. In > this case (a.mask is False) it at least *seems* to break that 'rule'... You mean, returning something that says 'nomask' instead of 'False' ? I'm not sure how I can do that. Sure, it'd be nice. > > No. I'm afraid you're confusing `nomask` and `False`. Once again, nomask > > is NOT the same thing as False. It's the same value, but not the same > > object. > > Exactly. It might not be inconsistent, but imho it does a lot of effort > to /feel/ inconsistent to Joe Average. It's not intentional. I guess most of your problems would never have come if we had a proper documentation. As you're starting to use MaskedArrays, you could take note and that would provide us with a basis we could improve on... From pgmdevlist at gmail.com Fri Sep 26 11:47:18 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 26 Sep 2008 11:47:18 -0400 Subject: [Numpy-discussion] Proper NaN handling in max and co: a redux In-Reply-To: References: <48DC61C4.6020307@ar.media.kyoto-u.ac.jp> <48DC6B82.9080201@ar.media.kyoto-u.ac.jp> Message-ID: <200809261147.18562.pgmdevlist@gmail.com> On Friday 26 September 2008 08:29:02 Charles R Harris wrote: > It shouldn't be any more difficult to do either based on a keyword. > Argsorts shouldn't be a problem either. I'm thinking that the most flexible > way to handle the sorts is to make a preliminary pass through the data and > collect all the nans at one end or the other and sort the remainder. That > would also make it easy to raise an error if we wanted to and avoid an > extra compare in all the other sorting passes. That approach could probably > be extended to masked arrays also. Note that MaskedArray.sort has already an extra keyword 'end_with', that lets you decide whether missing data should be at the beginning or the end of the array. From peridot.faceted at gmail.com Fri Sep 26 12:00:57 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 26 Sep 2008 12:00:57 -0400 Subject: [Numpy-discussion] Proper NaN handling in max and co: a redux In-Reply-To: <48DC6B82.9080201@ar.media.kyoto-u.ac.jp> References: <48DC61C4.6020307@ar.media.kyoto-u.ac.jp> <48DC6B82.9080201@ar.media.kyoto-u.ac.jp> Message-ID: 2008/9/26 David Cournapeau : > Charles R Harris wrote: >> >> I'm also wondering about the sign ufunc. It should probably return nan >> for nans, but -1,0,1 are the current values. We also need to decide >> which end of the sorted values the nans should go to. I'm a bit >> partial to the beginning but the end would be fine with me, it might >> even be a bit more natural as searchsorted would find the beginning of >> the nans by default. I would think sign should return NaN (does it not now?) unless its return type is integer, in which case I can't see a better answer than raising an exception (we certainly don't want it silently swallowing NaNs). > Note that in both suggest approaches, sort with NaN would raise an > error. We could then provide a nansort, which ignore the NaN, and deal > with your case; would it be hard to have an option for the beginning vs > the end, or would that be difficult ? Is it really a good idea to duplicate the maskedarray sorting code? Anne From josef.pktd at gmail.com Fri Sep 26 12:39:59 2008 From: josef.pktd at gmail.com (joep) Date: Fri, 26 Sep 2008 09:39:59 -0700 (PDT) Subject: [Numpy-discussion] vectorize in new.instancemethod and *args In-Reply-To: <92bb090b-b44a-4c77-a5be-161bacd2f1d8@d45g2000hsc.googlegroups.com> References: <92bb090b-b44a-4c77-a5be-161bacd2f1d8@d45g2000hsc.googlegroups.com> Message-ID: <968a09f9-070b-436d-b8bd-cbe9c77af0f4@y21g2000hsf.googlegroups.com> On Sep 26, 11:35?am, joep wrote: > I have a question about the use of vectorize in new.instancemethod: > > # case D: class with vectorize with nin adjustment -> broken > # nin is not correctly used by vectorize > > class D(object): > ? ? def __init__(self): > ? ? ? ? # define the vectorized function > ? ? ? ? vecfunc = vectorize(funcm) > ? ? ? ? vecfunc.nin = 2 > ? ? ? ? self.funcm = new.instancemethod(vecfunc,self,A) correction case D works vecfunc.nin = 3 works my silly mistake, not counting self as an argument Josef From david at ar.media.kyoto-u.ac.jp Fri Sep 26 13:20:28 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 27 Sep 2008 02:20:28 +0900 Subject: [Numpy-discussion] Proper NaN handling in max and co: a redux In-Reply-To: References: <48DC61C4.6020307@ar.media.kyoto-u.ac.jp> <48DC6B82.9080201@ar.media.kyoto-u.ac.jp> Message-ID: <48DD19DC.6070400@ar.media.kyoto-u.ac.jp> Anne Archibald wrote: > > I would think sign should return NaN (does it not now?) unless its > return type is integer, in which case I can't see a better answer than > raising an exception (we certainly don't want it silently swallowing > NaNs). > signbit (the C99 macro) returns an integer. So we have to check for nan I think. > > Is it really a good idea to duplicate the maskedarray sorting code? > This should be done in C (it is not difficult to do so, has a clear speed advantage, and is the only solution for numpy arrays themselves) and then, masked arrays can reuse this code. cheers, David From theller at ctypes.org Fri Sep 26 13:45:18 2008 From: theller at ctypes.org (Thomas Heller) Date: Fri, 26 Sep 2008 19:45:18 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: Message-ID: Jarrod Millman schrieb: > I'm pleased to announce the release of NumPy 1.2.0. > I see that the windows installer is now a superkac installer, which AFAIU selects the 'right' components for the target computer at install time. If I now build applications using numpy with py2exe for distribution, what will happen on the target computers if the components are not 'right' for the actual machine type? -- Thanks, Thomas From david at ar.media.kyoto-u.ac.jp Fri Sep 26 13:37:44 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 27 Sep 2008 02:37:44 +0900 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: Message-ID: <48DD1DE8.4000904@ar.media.kyoto-u.ac.jp> Thomas Heller wrote: > > I see that the windows installer is now a superkac installer, > which AFAIU selects the 'right' components for the target computer > at install time. Yes, it is the case since 1.1.0. It is not ideal, but it effectively solved a common problem of people not having the right architecture (AFAICS, the support emails on that point drop from several /months to 0). > > If I now build applications using numpy with py2exe for distribution, > what will happen on the target computers if the components are not 'right' > for the actual machine type? Note that the superpack is actually quite primitive: it is a nsis installer which encompasses actual installers built with bdist_wininst, the nsis being used only to detect CPU arch automatically. So if you need the installers as before, you can just ask nsis to extract all three installers I believe, from the command line. If it is a problem, or if we can make it easier for you, please tell us. I am the one who did this scheme, and I do not use py2exe (or windows for that matter), so I would be happy to solve the problems if you tell me how :) cheers, David From theller at ctypes.org Fri Sep 26 14:24:00 2008 From: theller at ctypes.org (Thomas Heller) Date: Fri, 26 Sep 2008 20:24:00 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: <48DD1DE8.4000904@ar.media.kyoto-u.ac.jp> References: <48DD1DE8.4000904@ar.media.kyoto-u.ac.jp> Message-ID: [Thomas] >> If I now build applications using numpy with py2exe for distribution, >> what will happen on the target computers if the components are not 'right' >> for the actual machine type? [David] > Note that the superpack is actually quite primitive: it is a nsis > installer which encompasses actual installers built with bdist_wininst, > the nsis being used only to detect CPU arch automatically. So if you > need the installers as before, you can just ask nsis to extract all > three installers I believe, from the command line. If it is a problem, > or if we can make it easier for you, please tell us. I am the one who > did this scheme, and I do not use py2exe (or windows for that matter), > so I would be happy to solve the problems if you tell me how :) Well, the first question is: What does happen when I install the SSE3 version (or how it's called) on my machine, use py2exe to build an app, and this app runs on a SSE2 machine - degraded performance, or hard crashes? I could probably live with the least performance, but not with the crashes ;-). So, maybe the gui could allow to select whether to install the high-performance version specialized for the current cpu, or a more portable but a little bit slower version (I assume there is one included) that can be safely used for py2exe. -- Thanks, Thomas From charlesr.harris at gmail.com Fri Sep 26 18:33:24 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 26 Sep 2008 16:33:24 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: <48DD1DE8.4000904@ar.media.kyoto-u.ac.jp> Message-ID: On Fri, Sep 26, 2008 at 12:24 PM, Thomas Heller wrote: > [Thomas] > >> If I now build applications using numpy with py2exe for distribution, > >> what will happen on the target computers if the components are not > 'right' > >> for the actual machine type? > > [David] > > Note that the superpack is actually quite primitive: it is a nsis > > installer which encompasses actual installers built with bdist_wininst, > > the nsis being used only to detect CPU arch automatically. So if you > > need the installers as before, you can just ask nsis to extract all > > three installers I believe, from the command line. If it is a problem, > > or if we can make it easier for you, please tell us. I am the one who > > did this scheme, and I do not use py2exe (or windows for that matter), > > so I would be happy to solve the problems if you tell me how :) > > Well, the first question is: What does happen when I install the SSE3 > version > (or how it's called) on my machine, use py2exe to build an app, and this > app runs on a SSE2 machine - degraded performance, or hard crashes? > > I could probably live with the least performance, but not with the crashes > ;-). > Probably hard crashes. > > So, maybe the gui could allow to select whether to install the > high-performance > version specialized for the current cpu, or a more portable but a little > bit > slower version (I assume there is one included) that can be safely used for > py2exe. > Compiling without Atlas would probably be the safest. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmoses at engineering.ucsb.edu Fri Sep 26 19:07:40 2008 From: pgmoses at engineering.ucsb.edu (Poul Georg Moses) Date: Fri, 26 Sep 2008 16:07:40 -0700 (PDT) Subject: [Numpy-discussion] numpy installation as non root using site.cfg Message-ID: <19697706.post@talk.nabble.com> Dear mailing list I am trying to install numpy-1.1.1 on a cluster where I do not have root permission. I have succesfully installed \ python2.5.2. I am using the cite.cfg file to include the blas and atlas libraries. I have done that by setting the\ defaults to: library_dirs = /usr/lib include_dirs = /usr/include where blas and atlas is located. Then I type: python setup.py install --home=$HOME and get the following output Running from numpy source directory. F2PY Version 2_5585 Traceback (most recent call last): File "setup.py", line 96, in setup_package() File "setup.py", line 89, in setup_package configuration=configuration ) File "/data/pgmoses/SourceCode/numpy-1.1.1/numpy/distutils/core.py", line 150, in setup config = configuration() File "setup.py", line 56, in configuration config.add_subpackage('numpy') File "/data/pgmoses/SourceCode/numpy-1.1.1/numpy/distutils/misc_util.py", line 850, in add_subpackage caller_level = 2) File "/data/pgmoses/SourceCode/numpy-1.1.1/numpy/distutils/misc_util.py", line 833, in get_subpackage caller_level = caller_level + 1) File "/data/pgmoses/SourceCode/numpy-1.1.1/numpy/distutils/misc_util.py", line 780, in _get_configuration_from_s\ etup_py config = setup_module.configuration(*args) File "numpy/setup.py", line 9, in configuration config.add_subpackage('core') File "/data/pgmoses/SourceCode/numpy-1.1.1/numpy/distutils/misc_util.py", line 850, in add_subpackage caller_level = 2) File "/data/pgmoses/SourceCode/numpy-1.1.1/numpy/distutils/misc_util.py", line 833, in get_subpackage caller_level = caller_level + 1) File "/data/pgmoses/SourceCode/numpy-1.1.1/numpy/distutils/misc_util.py", line 780, in _get_configuration_from_s\ etup_py config = setup_module.configuration(*args) File "numpy/core/setup.py", line 299, in configuration blas_info = get_info('blas_opt',0) File "/data/pgmoses/SourceCode/numpy-1.1.1/numpy/distutils/system_info.py", line 267, in get_info return cl().get_info(notfound_action) File "/data/pgmoses/SourceCode/numpy-1.1.1/numpy/distutils/system_info.py", line 366, in __init__ self.parse_config_files() File "/data/pgmoses/SourceCode/numpy-1.1.1/numpy/distutils/system_info.py", line 372, in parse_config_files self.cp.read(self.files) File "/data/pgmoses/lib/python2.5/ConfigParser.py", line 267, in read self._read(fp, filename) File "/data/pgmoses/lib/python2.5/ConfigParser.py", line 462, in _read raise MissingSectionHeaderError(fpname, lineno, line) ConfigParser.MissingSectionHeaderError: File contains no section headers. file: /data/pgmoses/SourceCode/numpy-1.1.1/site.cfg, line: 60 'library_dirs = /usr/lib\n' Do you have any suggestion to what I might be doing wrong Best regards, Poul Georg Moses -- View this message in context: http://www.nabble.com/numpy-installation-as-non-root-using-site.cfg-tp19697706p19697706.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From robert.kern at gmail.com Fri Sep 26 19:15:44 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 26 Sep 2008 18:15:44 -0500 Subject: [Numpy-discussion] numpy installation as non root using site.cfg In-Reply-To: <19697706.post@talk.nabble.com> References: <19697706.post@talk.nabble.com> Message-ID: <3d375d730809261615oada4b56h2eb42428aac51b43@mail.gmail.com> On Fri, Sep 26, 2008 at 18:07, Poul Georg Moses wrote: > > Dear mailing list > > I am trying to install numpy-1.1.1 on a cluster where I do not have root > permission. I have succesfully installed \ > python2.5.2. I am using the cite.cfg file to include the blas and atlas > libraries. I have done that by setting the\ > defaults to: > library_dirs = /usr/lib > include_dirs = /usr/include You need [DEFAULT] library_dirs = /usr/lib include_dirs = /usr/include just like it is in the site.cfg.example -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Fri Sep 26 20:36:47 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 26 Sep 2008 19:36:47 -0500 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <5d3194020809170512idfa560s6548ffcca406c325@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <5d3194020809170512idfa560s6548ffcca406c325@mail.gmail.com> Message-ID: <3d375d730809261736lf472811t6f34934d40d4ba55@mail.gmail.com> On Wed, Sep 17, 2008 at 07:12, Arnar Flatberg wrote: > > On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern wrote: >> >> So, I could use some comments on the workflow. Does this look sensible >> to everyone? How else would you like to use it? > > Works for me. I would love to use it as a part of an ipython session. > Initially, I sprinkled som $profile in > a library of my own, then I thought I would use the -s (SETUP) option to > initialize some data to pass into the $profiled functions. However, I have > probably misunderstood the meaning of the -s option. Could you give an > example of its use? I ended in doing a %edit in ipython to write a small > test script with testdata and library imports, used kernprof.py and > view_line_prof.py from ipython and captured the output from view_line_prof > into ipython. Pheew .... now, there's perhaps an easier way? There is now! Add import line_profiler ip.expose_magic('lprun', line_profiler.magic_lprun) to your ipy_user_conf.py . In [1]: %lprun? Type: Magic function Base Class: Namespace: IPython internal File: /Users/rkern/hg/line_profiler/line_profiler.py Definition: %lprun(self, parameter_s='') Docstring: Execute a statement under the line-by-line profiler from the line_profiler module. Usage: %lprun -f func1 -f func2 The given statement (which doesn't require quote marks) is run via the LineProfiler. Profiling is enabled for the functions specified by the -f options. The statistics will be shown side-by-side with the code through the pager once the statement has completed. Options: -f : LineProfiler only profiles functions and methods it is told to profile. This option tells the profiler about these functions. Multiple -f options may be used. The argument may be any expression that gives a Python function or method object. However, one must be careful to avoid spaces that may confuse the option parser. Additionally, functions defined in the interpreter at the In[] prompt or via %run currently cannot be displayed. Write these functions out to a separate file and import them. One or more -f options are required to get any useful results. -D : dump the raw statistics out to a marshal file on disk. The usual extension for this is ".lprof". These statistics may be viewed later by running line_profiler.py as a script. -T : dump the text-formatted statistics with the code side-by-side out to a text file. -r: return the LineProfiler object after it has completed profiling. > BTW, in the requirements you may want to mention argparse > (view_line_prof.py) 'tis gone now. The script and the requirement for argparse. Now you just use line_profiler.py as a script. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From russel at appliedminds.com Fri Sep 26 20:33:40 2008 From: russel at appliedminds.com (Russel Howe) Date: Fri, 26 Sep 2008 17:33:40 -0700 Subject: [Numpy-discussion] bug in fromiter - with patch Message-ID: <48DD7F64.3030904@appliedminds.com> As of SVN from this morning, fromiter cannot create an array whose dtype is a byte type In [1]: np.fromiter(range(10), dtype='b') --------------------------------------------------------------------------- MemoryError Traceback (most recent call last) /Users/baxter/ in () MemoryError: cannot allocate array memory I am not sure how to include a test whose failure condition is the raising of an exception, but the patch includes the above test. Russel -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: fromiter.patch URL: From russel at appliedminds.com Fri Sep 26 20:50:43 2008 From: russel at appliedminds.com (Russel Howe) Date: Fri, 26 Sep 2008 17:50:43 -0700 Subject: [Numpy-discussion] bug in fromiter - with patch In-Reply-To: <48DD7F64.3030904@appliedminds.com> References: <48DD7F64.3030904@appliedminds.com> Message-ID: <48DD8363.6010706@appliedminds.com> Oops. I made a last minute change without re-running the tests. The tests refer to np.range, that should just be range. Corrected test attached. Russel Howe wrote: > As of SVN from this morning, fromiter cannot create an array whose dtype > is a byte type > > In [1]: np.fromiter(range(10), dtype='b') > --------------------------------------------------------------------------- > MemoryError Traceback (most recent call last) > > /Users/baxter/ in () > > MemoryError: cannot allocate array memory > > > I am not sure how to include a test whose failure condition is the > raising of an exception, but the patch includes the above test. > > Russel > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: fromiter.patch URL: From fperez.net at gmail.com Sat Sep 27 01:50:25 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 26 Sep 2008 22:50:25 -0700 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <3d375d730809261736lf472811t6f34934d40d4ba55@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <5d3194020809170512idfa560s6548ffcca406c325@mail.gmail.com> <3d375d730809261736lf472811t6f34934d40d4ba55@mail.gmail.com> Message-ID: Hey, On Fri, Sep 26, 2008 at 5:36 PM, Robert Kern wrote: > There is now! Add > > import line_profiler > ip.expose_magic('lprun', line_profiler.magic_lprun) > > to your ipy_user_conf.py . Mmh, should we ship this out of the box in ipython? The C dependency is the only thing that gives me a bit of pause. There goes our pure C, easy to build binaries for... On the other hand, me wants :) What's your take on that? f From dpeterson at enthought.com Sat Sep 27 02:09:23 2008 From: dpeterson at enthought.com (Dave Peterson) Date: Sat, 27 Sep 2008 01:09:23 -0500 Subject: [Numpy-discussion] ANNOUNCE: EPD (Enthought Python Distribution) 4.0.300 Beta 3 available Message-ID: <48DDCE13.60107@enthought.com> Hello, We've recently posted the third beta release of EPD (the Enthought Python Distribution) with Python 2.5 version 4.0.300. You may download the beta from here: http://www.enthought.com/products/epdbeta.php Please help us test it out and provide feedback on the EPD Trac instance: https://svn.enthought.com/epd You can check out the release notes here: http://www.enthought.com/products/epdbetareleasenotes.php About EPD --------- The Enthought Python Distribution (EPD) is a "kitchen-sink-included" distribution of the Python? Programming Language, including over 60 additional tools and libraries. The EPD bundle includes NumPy, SciPy, IPython, 2D and 3D visualization, database adapters, and a lot of other tools right out of the box. http://www.enthought.com/products/epd.php It is currently available as a single-click installer for Windows XP (x86), Mac OS X (a universal binary for OS X 10.4 and above), and RedHat 3 and 4 (x86 and amd64). EPD is free for academic use. An annual subscription and installation support are available for individual commercial use. An enterprise subscription with support for particular deployment environments is also available for commercial purchase. The beta versions of EPD are available for indefinite free trial. -- Dave From oliphant at enthought.com Sat Sep 27 02:51:19 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sat, 27 Sep 2008 01:51:19 -0500 Subject: [Numpy-discussion] Bug fix or behavior change? In-Reply-To: References: Message-ID: <48DDD7E7.3060506@enthought.com> Charles R Harris wrote: > Hi All, > > Currently subtract for boolean arrays is defined in > > /**begin repeat > * Arithmetic operators > * > * # OP = ||, ^, &&# > * #kind = add, subtract, multiply# > */ > static void > BOOL_ at kind@(char **args, intp *dimensions, intp *steps, void *func) > { > register intp i; > intp is1=steps[0],is2=steps[1],os=steps[2], n=dimensions[0]; > char *i1=args[0], *i2=args[1], *op=args[2]; > for(i=0; i *((Bool *)op)=*((Bool *)i1) @OP@ *((Bool *)i2); > } > } > /**end repeat**/ > > > Note that this might yield unexpected results if the boolean value is > not 0 or 1, which is possible using views. Note also that bitwise_xor > converts the boolean to 0 or 1 before using ^. I think subtract should > work the same way, but that would change current behavior. So... do I > make the change and call it a bug fix or leave it as it is? I think it's a bugfix and so am +1 on the change. -Travis From charlesr.harris at gmail.com Sat Sep 27 03:18:09 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 27 Sep 2008 01:18:09 -0600 Subject: [Numpy-discussion] Bug fix or behavior change? In-Reply-To: <48DDD7E7.3060506@enthought.com> References: <48DDD7E7.3060506@enthought.com> Message-ID: On Sat, Sep 27, 2008 at 12:51 AM, Travis E. Oliphant wrote: > Charles R Harris wrote: > > Hi All, > > > > Currently subtract for boolean arrays is defined in > > > > /**begin repeat > > * Arithmetic operators > > * > > * # OP = ||, ^, &&# > > * #kind = add, subtract, multiply# > > */ > > static void > > BOOL_ at kind@(char **args, intp *dimensions, intp *steps, void *func) > > { > > register intp i; > > intp is1=steps[0],is2=steps[1],os=steps[2], n=dimensions[0]; > > char *i1=args[0], *i2=args[1], *op=args[2]; > > for(i=0; i > *((Bool *)op)=*((Bool *)i1) @OP@ *((Bool *)i2); > > } > > } > > /**end repeat**/ > > > > > > Note that this might yield unexpected results if the boolean value is > > not 0 or 1, which is possible using views. Note also that bitwise_xor > > converts the boolean to 0 or 1 before using ^. I think subtract should > > work the same way, but that would change current behavior. So... do I > > make the change and call it a bug fix or leave it as it is? > > I think it's a bugfix and so am +1 on the change. > Done. I've made all the boolean ufuncs convert args to 0 or 1 before applying the operators. Next question, what about logical_and applied to complex numbers. Currently it gives In [16]: a = complex(1,0) In [17]: b = complex(0,1) In [18]: logical_and(a,b) Out[18]: False i.e., it && all the components together. I suspect what it should be doing is anding first and second arguments being non-zero. Also, there is this bit for absolute of signed numbers static void @TYPE at _absolute(char **args, intp *dimensions, intp *steps, void *func) { UNARY_LOOP { const @type@ in1 = *((@type@ *)ip1); *((@type@ *)op) = (in1 < 0) ? -in1 : in1; *((@type@ *)op) += 0; /* clear sign-bit */ } } I don't understand the clearing the sign-bit line, it looks unnecessary. Was there some problem that this once fixed? Also, the signature for the ones_like ufuncs has 1 input argument, but only the output argument is used. I suppose changing this would be a pain, but it's a bit odd. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Sep 27 03:25:10 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 27 Sep 2008 01:25:10 -0600 Subject: [Numpy-discussion] Bug fix or behavior change? In-Reply-To: References: <48DDD7E7.3060506@enthought.com> Message-ID: On Sat, Sep 27, 2008 at 1:18 AM, Charles R Harris wrote: > > > On Sat, Sep 27, 2008 at 12:51 AM, Travis E. Oliphant < > oliphant at enthought.com> wrote: > >> Charles R Harris wrote: >> > Hi All, >> > >> > Currently subtract for boolean arrays is defined in >> > >> > /**begin repeat >> > * Arithmetic operators >> > * >> > * # OP = ||, ^, &&# >> > * #kind = add, subtract, multiply# >> > */ >> > static void >> > BOOL_ at kind@(char **args, intp *dimensions, intp *steps, void *func) >> > { >> > register intp i; >> > intp is1=steps[0],is2=steps[1],os=steps[2], n=dimensions[0]; >> > char *i1=args[0], *i2=args[1], *op=args[2]; >> > for(i=0; i> > *((Bool *)op)=*((Bool *)i1) @OP@ *((Bool *)i2); >> > } >> > } >> > /**end repeat**/ >> > >> > >> > Note that this might yield unexpected results if the boolean value is >> > not 0 or 1, which is possible using views. Note also that bitwise_xor >> > converts the boolean to 0 or 1 before using ^. I think subtract should >> > work the same way, but that would change current behavior. So... do I >> > make the change and call it a bug fix or leave it as it is? >> >> I think it's a bugfix and so am +1 on the change. >> > > Done. I've made all the boolean ufuncs convert args to 0 or 1 before > applying the operators. Next question, what about logical_and applied to > complex numbers. Currently it gives > I'm also thinking that bitwise operators should not be allowed for booleans. But that might break some code... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Sat Sep 27 03:55:59 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 27 Sep 2008 16:55:59 +0900 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: <48DD1DE8.4000904@ar.media.kyoto-u.ac.jp> Message-ID: <48DDE70F.8050507@ar.media.kyoto-u.ac.jp> Thomas Heller wrote: > > Well, the first question is: What does happen when I install the SSE3 version > (or how it's called) on my machine, use py2exe to build an app, and this > app runs on a SSE2 machine - degraded performance, or hard crashes? Hard crash. That's the whole point of the installer, actually: install the right version. The problem is two fold: - it is practically impossible to force ATLAS to configure itself to use an instruction set different than the one used to build ATLAS. - we don't have a system to dynamically load the right ATLAS when importing numpy. Note that this is not new: you had the problem before, because before, numpy was *only* built with SSE2 support, and any machine wo SSE2 would crash when using numpy. > > So, maybe the gui could allow to select whether to install the high-performance > version specialized for the current cpu, or a more portable but a little bit > slower version (I assume there is one included) that can be safely used for py2exe. Is it really complicated to decompress the .exe to get the installers and choose the one you want ? I am reluctant to add a GUI option because nsis is primitive, and adding gui is no fun (I use it only because it is open source and has a plug-in system; the scripting language to build the binary is awful). Also, the nsis installer itself has no GUI on purpose, to avoid confusing people with a two stages installer. cheers, David From robert.kern at gmail.com Sat Sep 27 04:31:43 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 27 Sep 2008 03:31:43 -0500 Subject: [Numpy-discussion] profiling line by line In-Reply-To: References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <5d3194020809170512idfa560s6548ffcca406c325@mail.gmail.com> <3d375d730809261736lf472811t6f34934d40d4ba55@mail.gmail.com> Message-ID: <3d375d730809270131n842e003t81a391fa5a959abb@mail.gmail.com> On Sat, Sep 27, 2008 at 00:50, Fernando Perez wrote: > Hey, > > On Fri, Sep 26, 2008 at 5:36 PM, Robert Kern wrote: > >> There is now! Add >> >> import line_profiler >> ip.expose_magic('lprun', line_profiler.magic_lprun) >> >> to your ipy_user_conf.py . > > Mmh, should we ship this out of the box in ipython? The C dependency > is the only thing that gives me a bit of pause. There goes our pure > C, easy to build binaries for... On the other hand, me wants :) > > What's your take on that? I'd prefer not to. The exclusion of C modules is an eminently reasonable one. You would inevitably make it optional, and my opinion is that optional functionality is best handled by separate packages. Feel free to add it as "highly recommended" in the IPython installation instructions, though. :-) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From andrea.gavana at gmail.com Sat Sep 27 04:50:32 2008 From: andrea.gavana at gmail.com (Andrea Gavana) Date: Sat, 27 Sep 2008 09:50:32 +0100 Subject: [Numpy-discussion] Minimum distance between 2 paths in 3D Message-ID: Hi All, I was wondering if someone had any suggestions/references/snippets of code on how to find the minimum distance between 2 paths in 3D. Basically, for every path, I have I series of points (x, y, z) and I would like to know if there is a better way, other than comparing point by point the 2 paths, to find the minimum distance between them. Any suggestion is welcome :-) Thank you. Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://xoomer.alice.it/infinity77/ From rob.clewley at gmail.com Sat Sep 27 11:05:07 2008 From: rob.clewley at gmail.com (Rob Clewley) Date: Sat, 27 Sep 2008 11:05:07 -0400 Subject: [Numpy-discussion] Minimum distance between 2 paths in 3D In-Reply-To: References: Message-ID: Hi Andrea, > I was wondering if someone had any suggestions/references/snippets > of code on how to find the minimum distance between 2 paths in 3D. > Basically, for every path, I have I series of points (x, y, z) and I > would like to know if there is a better way, other than comparing > point by point the 2 paths, to find the minimum distance between them. In 2D there would be a few tricks you could use, but in higher dimensions anything smart that you could attempt might cost you more computation time than just comparing the points (unless N is huge). At least make sure to put the "looping" over points into a vectorized form to avoid python for loops. e.g. two curves given by 3xN arrays c and d: from numpy import concatenate, argmin from numpy.linalg import norm distvec = concatenate([c[:,i]-d.T for i in range(N)]) # all N**2 distance vectors ns = [norm(a) for a in distvec] # all N**2 norms of the distance vectors cix, dix = divmod(argmin(ns), N) # find the index of the minimum norm from [0 .. N**2] and decode which point this corresponds to The minimum is between the points c[:,cix] and d[:,dix]. A numpy wonk might be able to squeeze a bit more optimization out of this, but I think this code works OK. Unfortunately, unless you know more mathematical properties of your curves in advance (info about their maximum curvature, for instance) you'll end up needing to check every pair of points. If N is really big, like over 10**4 maybe, it might be worth trying to break the curves up into pieces contained in bounding boxes which you can eliminate from a full search if they don't intersect. -Rob From josef.pktd at gmail.com Sat Sep 27 14:12:14 2008 From: josef.pktd at gmail.com (joep) Date: Sat, 27 Sep 2008 11:12:14 -0700 (PDT) Subject: [Numpy-discussion] numpy.random.logseries - incorrect convergence for k=1, k=2 Message-ID: random numbers generated by numpy.random.logseries do not converge to theoretical distribution: for probability paramater pr = 0.8, the random number generator converges to a frequency for k=1 at 39.8 %, while the theoretical probability mass is 49.71 k=2 is oversampled, other k's look ok check frequency of k=1 and k=2 at N = 1000000 0.398406 0.296465 pmf at k = 1 and k=2 with formula [ 0.4971 0.1988] for probability paramater pr = 0.3, the results are not as bad, but still off: frequency for k=1 at 82.6 %, while the theoretical probability mass is 84.11 check frequency of k=1 and k=2 at N = 1000000 0.826006 0.141244 pmf at k = 1 and k=2 with formula [ 0.8411 0.1262] below is a quick script for checking this Josef {{{ import numpy as np from scipy import stats pr = 0.8 np.set_printoptions(precision=2, suppress=True) # calculation for N=1million takes some time for N in [1000, 10000, 10000, 1000000]: rvsn=np.random.logseries(pr,size=N) fr=stats.itemfreq(rvsn) pmfs=stats.logser.pmf(fr[:,0],pr)*100 print 'log series sample frequency and pmf (in %) with N = ', N print np.column_stack((fr[:,0],fr[:,1]*100.0/N,pmfs)) np.set_printoptions(precision=4, suppress=True) print 'check frequency of k=1 and k=2 at N = ', N print np.sum(rvsn==1)/float(N), print np.sum(rvsn==2)/float(N) k = np.array([1,2]) print 'pmf at k = 1 and k=2 with formula' print -pr**k * 1.0 / k / np.log(1-pr) }}} From fperez.net at gmail.com Sat Sep 27 14:47:48 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 27 Sep 2008 11:47:48 -0700 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <3d375d730809270131n842e003t81a391fa5a959abb@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <5d3194020809170512idfa560s6548ffcca406c325@mail.gmail.com> <3d375d730809261736lf472811t6f34934d40d4ba55@mail.gmail.com> <3d375d730809270131n842e003t81a391fa5a959abb@mail.gmail.com> Message-ID: On Sat, Sep 27, 2008 at 1:31 AM, Robert Kern wrote: >> Mmh, should we ship this out of the box in ipython? The C dependency >> is the only thing that gives me a bit of pause. There goes our pure >> C, easy to build binaries for... On the other hand, me wants :) s/pure C/pure Python/ grr... >> What's your take on that? > > I'd prefer not to. The exclusion of C modules is an eminently > reasonable one. You would inevitably make it optional, and my opinion > is that optional functionality is best handled by separate packages. Agreed. Not having C dependencies simplifies lots of things for us. If we ever gain an unavoidable one, we can then revisit this. > Feel free to add it as "highly recommended" in the IPython > installation instructions, though. :-) Most certainly will. It would be better to have a more stable release/download link than your hg repo. Do you have a page or a Pypi link for it yet? A little readme? Cheers, f From pgmdevlist at gmail.com Sat Sep 27 14:49:22 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Sat, 27 Sep 2008 14:49:22 -0400 Subject: [Numpy-discussion] Interesting bug with numpy.ma Message-ID: <200809271449.22356.pgmdevlist@gmail.com> All, I've just been pointed to a strange bug in numpy.ma import numpy as np import numpy.ma as ma assert(ma.masked*float(1) is ma.masked) assert(float(1)*ma.masked is ma.masked) assert(ma.masked*np.float(1) is ma.masked) assert(np.float(1)*ma.masked is ma.masked) assert(ma.masked*np.float_(1) is ma.masked) assert(np.float_(1)*ma.masked is ma.masked) The last assertion fails, and I have no idea why. What's the difference between assertions 3 and 5 ? FYI, ma.masked=ma.MaskedArray(0.,dtype=np.float_,mask=True), and MaskedArray has a __array_priority__ of 15. Implementing a __rmul__ doesn't work either. Any idea welcome. From fperez.net at gmail.com Sat Sep 27 15:30:32 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 27 Sep 2008 12:30:32 -0700 Subject: [Numpy-discussion] Cython docs including numpy info Message-ID: Hi all, I just found out something that might interest others on this list. Gabriel Gellner (forgive me if I'm not duly crediting others, I don't know the detailed history of this) has been leading the effort to produce self-contained Cython documentation, in sphix, that includes the new numpy support added recently to Cython: http://www.mudskipper.ca/cython-doc/ I'm looking at these docs right now, but I'm sure Gabriel and the rest of the Cython team would benefit from more feedback. Cheers, f From peridot.faceted at gmail.com Sat Sep 27 16:23:06 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sat, 27 Sep 2008 16:23:06 -0400 Subject: [Numpy-discussion] Minimum distance between 2 paths in 3D In-Reply-To: References: Message-ID: 2008/9/27 Andrea Gavana : > I was wondering if someone had any suggestions/references/snippets > of code on how to find the minimum distance between 2 paths in 3D. > Basically, for every path, I have I series of points (x, y, z) and I > would like to know if there is a better way, other than comparing > point by point the 2 paths, to find the minimum distance between them. If you treat the points as simply two clouds of points (that is, ignore the fact that they represent points on paths), spatial data structures can make this kind of question faster. For example a kd-tree can be constructed in something like O(n log n) time, and you can answer questions like "what is the closest point in this set to (a,b,c)?" in something like O(log n) time. So you could do this by putting one set of points in a kd-tree and just running queries against each point in the other. There exist other spatial data structures - octrees, voxel grids, bounding-volume hierarchies, other binary space partitioning trees, as well as various more specialized ones. As the number of dimensions becomes large they become substantially more difficult to work with, and you do need to balance construction time against lookup time, but they are definitely worth thinking about in your problem. Sadly, no such data structure exists in either numpy or scipy, though biopython apparently has an implementation of kd-trees. Good luck, Anne From theller at ctypes.org Sat Sep 27 16:38:33 2008 From: theller at ctypes.org (Thomas Heller) Date: Sat, 27 Sep 2008 22:38:33 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: <48DDE70F.8050507@ar.media.kyoto-u.ac.jp> References: <48DD1DE8.4000904@ar.media.kyoto-u.ac.jp> <48DDE70F.8050507@ar.media.kyoto-u.ac.jp> Message-ID: David Cournapeau schrieb: > Thomas Heller wrote: >> >> Well, the first question is: What does happen when I install the SSE3 version >> (or how it's called) on my machine, use py2exe to build an app, and this >> app runs on a SSE2 machine - degraded performance, or hard crashes? > > Hard crash. That's the whole point of the installer, actually: install > the right version. The problem is two fold: > - it is practically impossible to force ATLAS to configure itself to > use an instruction set different than the one used to build ATLAS. > - we don't have a system to dynamically load the right ATLAS when > importing numpy. > > Note that this is not new: you had the problem before, because before, > numpy was *only* built with SSE2 support, and any machine wo SSE2 would > crash when using numpy. I see. >> So, maybe the gui could allow to select whether to install the high-performance >> version specialized for the current cpu, or a more portable but a little bit >> slower version (I assume there is one included) that can be safely used for py2exe. > > Is it really complicated to decompress the .exe to get the installers > and choose the one you want ? I am reluctant to add a GUI option because > nsis is primitive, and adding gui is no fun (I use it only because it is > open source and has a plug-in system; the scripting language to build > the binary is awful). Also, the nsis installer itself has no GUI on > purpose, to avoid confusing people with a two stages installer. No, it isn't complicated. I searched a little about command line options, and didn't find any. But 7-zip did unpack the installers. I can live with that. -- Thanks, Thomas From dalke at dalkescientific.com Sat Sep 27 16:59:15 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Sat, 27 Sep 2008 22:59:15 +0200 Subject: [Numpy-discussion] Minimum distance between 2 paths in 3D In-Reply-To: References: Message-ID: On Sep 27, 2008, at 10:23 PM, Anne Archibald wrote: > Sadly, no such data structure exists > in either numpy or scipy, though biopython apparently has an > implementation of kd-trees. It's not part of scipy but I saw there was a scikit which supported ANN: http://scipy.org/scipy/scikits/wiki/AnnWrapper scikits.ann exposes a single class, kdtree that wraps the Approximate Nearest Neighbor library's kd-tree implementation. kdtree has a single (non-constructor) method, knn that finds the indecies (of the points used to construct the kdtree) of the k-nearest neighbors and the squared distances to those points Does anyone have any experience with it? Though perhaps I should ask on the scipy list instead of numpy .... Andrew dalke at dalkescientific.com From robert.kern at gmail.com Sat Sep 27 18:18:31 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 27 Sep 2008 17:18:31 -0500 Subject: [Numpy-discussion] Minimum distance between 2 paths in 3D In-Reply-To: References: Message-ID: <3d375d730809271518y7204f39er2309697a2d002717@mail.gmail.com> On Sat, Sep 27, 2008 at 15:23, Anne Archibald wrote: > 2008/9/27 Andrea Gavana : > >> I was wondering if someone had any suggestions/references/snippets >> of code on how to find the minimum distance between 2 paths in 3D. >> Basically, for every path, I have I series of points (x, y, z) and I >> would like to know if there is a better way, other than comparing >> point by point the 2 paths, to find the minimum distance between them. > > If you treat the points as simply two clouds of points (that is, > ignore the fact that they represent points on paths), spatial data > structures can make this kind of question faster. For example a > kd-tree can be constructed in something like O(n log n) time, and you > can answer questions like "what is the closest point in this set to > (a,b,c)?" in something like O(log n) time. So you could do this by > putting one set of points in a kd-tree and just running queries > against each point in the other. There exist other spatial data > structures - octrees, voxel grids, bounding-volume hierarchies, other > binary space partitioning trees, as well as various more specialized > ones. As the number of dimensions becomes large they become > substantially more difficult to work with, and you do need to balance > construction time against lookup time, but they are definitely worth > thinking about in your problem. Sadly, no such data structure exists > in either numpy or scipy, though biopython apparently has an > implementation of kd-trees. Sturla Molden has just contributed a pure Python+numpy implementation on the Scipy Cookbook. http://www.scipy.org/Cookbook/KDTree -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From peridot.faceted at gmail.com Sat Sep 27 23:18:46 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sat, 27 Sep 2008 23:18:46 -0400 Subject: [Numpy-discussion] Minimum distance between 2 paths in 3D In-Reply-To: <3d375d730809271518y7204f39er2309697a2d002717@mail.gmail.com> References: <3d375d730809271518y7204f39er2309697a2d002717@mail.gmail.com> Message-ID: 2008/9/27 Robert Kern : > On Sat, Sep 27, 2008 at 15:23, Anne Archibald wrote: >> 2008/9/27 Andrea Gavana : >> >>> I was wondering if someone had any suggestions/references/snippets >>> of code on how to find the minimum distance between 2 paths in 3D. >>> Basically, for every path, I have I series of points (x, y, z) and I >>> would like to know if there is a better way, other than comparing >>> point by point the 2 paths, to find the minimum distance between them. >> >> If you treat the points as simply two clouds of points (that is, >> ignore the fact that they represent points on paths), spatial data >> structures can make this kind of question faster. For example a >> kd-tree can be constructed in something like O(n log n) time, and you >> can answer questions like "what is the closest point in this set to >> (a,b,c)?" in something like O(log n) time. So you could do this by >> putting one set of points in a kd-tree and just running queries >> against each point in the other. There exist other spatial data >> structures - octrees, voxel grids, bounding-volume hierarchies, other >> binary space partitioning trees, as well as various more specialized >> ones. As the number of dimensions becomes large they become >> substantially more difficult to work with, and you do need to balance >> construction time against lookup time, but they are definitely worth >> thinking about in your problem. Sadly, no such data structure exists >> in either numpy or scipy, though biopython apparently has an >> implementation of kd-trees. > > Sturla Molden has just contributed a pure Python+numpy implementation > on the Scipy Cookbook. > > http://www.scipy.org/Cookbook/KDTree I think a kd-tree implementation would be a valuable addition to scipy, perhaps in a submodule scipy.spatial that might eventually contain other spatial data structures and algorithms. What do you think? Should we have one? Should it be based on Sturla Molden's code, if the license permits? I am willing to contribute one, if not. Anne From wnbell at gmail.com Sat Sep 27 23:58:32 2008 From: wnbell at gmail.com (Nathan Bell) Date: Sat, 27 Sep 2008 23:58:32 -0400 Subject: [Numpy-discussion] Minimum distance between 2 paths in 3D In-Reply-To: References: <3d375d730809271518y7204f39er2309697a2d002717@mail.gmail.com> Message-ID: On Sat, Sep 27, 2008 at 11:18 PM, Anne Archibald wrote: > > I think a kd-tree implementation would be a valuable addition to > scipy, perhaps in a submodule scipy.spatial that might eventually > contain other spatial data structures and algorithms. What do you > think? Should we have one? Should it be based on Sturla Molden's code, > if the license permits? I am willing to contribute one, if not. +1 If you're implementing one, I would highly recommend the "left-balanced" kd-tree. http://www2.imm.dtu.dk/pubdb/views/edoc_download.php/2535/pdf/imm2535.pdf Here's one implementation (undetermined license): http://www2.imm.dtu.dk/~jab/software.html#kdtrees I have a C++ implementation based on the one above that does a few more operations (like nearest-n neighbors) that you're welcome to use. The use of n_th() in the STL makes left-balancing fairly straightforward. FWIW I also have a pure python implementation here: http://code.google.com/p/pydec/source/browse/trunk/pydec/math/kd_tree.py -- Nathan Bell wnbell at gmail.com http://graphics.cs.uiuc.edu/~wnbell/ From irving at naml.us Sun Sep 28 00:34:22 2008 From: irving at naml.us (Geoffrey Irving) Date: Sat, 27 Sep 2008 21:34:22 -0700 Subject: [Numpy-discussion] nonuniform scatter operations Message-ID: <7f9d599f0809272134h5e37d0ffkd98c6dafc2b3f9a1@mail.gmail.com> Hello, Is there an efficient way to implement a nonuniform gather operation in numpy? Specifically, I want to do something like n,m = 100,1000 X = random.uniform(size=n) K = random.randint(n, size=m) Y = random.uniform(size=m) for k,y in zip(K,Y): X[k] += y but I want it to be fast. The naive attempt "X[K] += Y" does not work, since the slice assumes the indices don't repeat. Thanks, Geoffrey From wnbell at gmail.com Sun Sep 28 01:01:27 2008 From: wnbell at gmail.com (Nathan Bell) Date: Sun, 28 Sep 2008 01:01:27 -0400 Subject: [Numpy-discussion] nonuniform scatter operations In-Reply-To: <7f9d599f0809272134h5e37d0ffkd98c6dafc2b3f9a1@mail.gmail.com> References: <7f9d599f0809272134h5e37d0ffkd98c6dafc2b3f9a1@mail.gmail.com> Message-ID: On Sun, Sep 28, 2008 at 12:34 AM, Geoffrey Irving wrote: > > Is there an efficient way to implement a nonuniform gather operation > in numpy? Specifically, I want to do something like > > n,m = 100,1000 > X = random.uniform(size=n) > K = random.randint(n, size=m) > Y = random.uniform(size=m) > > for k,y in zip(K,Y): > X[k] += y > > but I want it to be fast. The naive attempt "X[K] += Y" does not > work, since the slice assumes the indices don't repeat. > I don't know of numpy solution, but in scipy you could use a sparse matrix to perform the operation. I think the following does what you want. from scipy.sparse import coo_matrix X += coo_matrix( (Y, (K,zeros(m,dtype=int)), shape=(n,1)).sum(axis=1) This reduces to a simple C++ loop, so speed should be good: http://projects.scipy.org/scipy/scipy/browser/trunk/scipy/sparse/sparsetools/coo.h#L139 -- Nathan Bell wnbell at gmail.com http://graphics.cs.uiuc.edu/~wnbell/ From david at ar.media.kyoto-u.ac.jp Sun Sep 28 01:10:17 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sun, 28 Sep 2008 14:10:17 +0900 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: <48DD1DE8.4000904@ar.media.kyoto-u.ac.jp> <48DDE70F.8050507@ar.media.kyoto-u.ac.jp> Message-ID: <48DF11B9.3080609@ar.media.kyoto-u.ac.jp> Thomas Heller wrote: > > No, it isn't complicated. I searched a little about command line options, > and didn't find any. But 7-zip did unpack the installers. I can live with > that. I will look into adding a command line switch. I don't want to add a GUI, but adding some command line switches for advanced uses and scripting makes a lot of sense, without risks of disrupting the whole install "experience". cheers, David From peridot.faceted at gmail.com Sun Sep 28 02:50:10 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sun, 28 Sep 2008 02:50:10 -0400 Subject: [Numpy-discussion] nonuniform scatter operations In-Reply-To: <7f9d599f0809272134h5e37d0ffkd98c6dafc2b3f9a1@mail.gmail.com> References: <7f9d599f0809272134h5e37d0ffkd98c6dafc2b3f9a1@mail.gmail.com> Message-ID: 2008/9/28 Geoffrey Irving : > Is there an efficient way to implement a nonuniform gather operation > in numpy? Specifically, I want to do something like > > n,m = 100,1000 > X = random.uniform(size=n) > K = random.randint(n, size=m) > Y = random.uniform(size=m) > > for k,y in zip(K,Y): > X[k] += y > > but I want it to be fast. The naive attempt "X[K] += Y" does not > work, since the slice assumes the indices don't repeat. I believe histogram can be persuaded to do this. Anne From andrea.gavana at gmail.com Sun Sep 28 05:42:25 2008 From: andrea.gavana at gmail.com (Andrea Gavana) Date: Sun, 28 Sep 2008 10:42:25 +0100 Subject: [Numpy-discussion] Minimum distance between 2 paths in 3D In-Reply-To: References: Message-ID: Hi Rob & All, On Sat, Sep 27, 2008 at 4:05 PM, Rob Clewley wrote: > Hi Andrea, > >> I was wondering if someone had any suggestions/references/snippets >> of code on how to find the minimum distance between 2 paths in 3D. >> Basically, for every path, I have I series of points (x, y, z) and I >> would like to know if there is a better way, other than comparing >> point by point the 2 paths, to find the minimum distance between them. > > In 2D there would be a few tricks you could use, but in higher > dimensions anything smart that you could attempt might cost you more > computation time than just comparing the points (unless N is huge). At > least make sure to put the "looping" over points into a vectorized > form to avoid python for loops. e.g. two curves given by 3xN arrays c > and d: > > from numpy import concatenate, argmin > from numpy.linalg import norm > > distvec = concatenate([c[:,i]-d.T for i in range(N)]) # all N**2 > distance vectors > ns = [norm(a) for a in distvec] # all N**2 norms of the distance vectors > cix, dix = divmod(argmin(ns), N) # find the index of the minimum > norm from [0 .. N**2] and decode which point this corresponds to > > The minimum is between the points c[:,cix] and d[:,dix]. A numpy wonk > might be able to squeeze a bit more optimization out of this, but I > think this code works OK. > > Unfortunately, unless you know more mathematical properties of your > curves in advance (info about their maximum curvature, for instance) > you'll end up needing to check every pair of points. If N is really > big, like over 10**4 maybe, it might be worth trying to break the > curves up into pieces contained in bounding boxes which you can > eliminate from a full search if they don't intersect. Thank you very much for your kind suggestions and the code snippet. I think it will do just fine, as my well trajectories may have from 20 to 500 points maximum. I'll try it tomorrow at work and see how it goes. Thank you again. Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://xoomer.alice.it/infinity77/ From oc-spam66 at laposte.net Sun Sep 28 11:47:27 2008 From: oc-spam66 at laposte.net (oc-spam66) Date: Sun, 28 Sep 2008 17:47:27 +0200 (CEST) Subject: [Numpy-discussion] Vectorization of the product of several matrices ? Message-ID: <31254818.516611222616846993.JavaMail.www@wwinf8401> Hello, I have two lists of numpy matrices : LM = [M_i, i=1..N] and LN = [N_i, i =1..N] and I would like to compute the list of the products : LP = [M_i * N_i, i=1..N]. I can do : P=[] for i in range(N) : P.append(LM[i]*LN[i]) But this is not vectorized. Is there a faster solution ? Can I vectorize this operation ? How ? Will it be more efficient than the code above ? Thank you for your help, -- O.C. Cr?ez votre adresse ?lectronique prenom.nom at laposte.net 1 Go d'espace de stockage, anti-spam et anti-virus int?gr?s. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Sun Sep 28 12:30:56 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Sun, 28 Sep 2008 12:30:56 -0400 Subject: [Numpy-discussion] Help needed with the priority of 0d-arrays and np.scalars Message-ID: <200809281230.57424.pgmdevlist@gmail.com> All, In a recent post, I was pointing to a bug in numpy.ma. I think I can understand what happens, but I'd need a couple of clarifications. When multiplying a 0d ndarrray or one of its subclass by a numpy scalar, we end up with a numpy scalar cast to the highest type, right ? So, np.float64(1)*np.array(0.,dtype='g') is a np.float128 object, while np.float64(1)*np.array(0,dtype=np.float32) is a np.float64 object. What surprises me is that numpy scalars have a low priority (listed as -100 or -1000000 depending on the page of the numpy book), so if I have a 0d subclass of ndarray with a higher priority, I should end up with a 0d subclass of ndarray, but it doesn't seem to be the case. The problem comes and bites me with ma.masked, defined as a 0d MaskedArray with __array_priority__ of 15 and a np.float64 dtype. Doing ma.masked*np.float64(1.) yields ma.masked, as it should. Doing np.float64(1.)*ma.masked yields np.float64(0.) but shouldn't: I should access instead the __rmul__ method of ma.masked. Any comment/suggestion would be welcome. From irving at naml.us Sun Sep 28 16:15:30 2008 From: irving at naml.us (Geoffrey Irving) Date: Sun, 28 Sep 2008 13:15:30 -0700 Subject: [Numpy-discussion] nonuniform scatter operations In-Reply-To: References: <7f9d599f0809272134h5e37d0ffkd98c6dafc2b3f9a1@mail.gmail.com> Message-ID: <7f9d599f0809281315r5237966bq48cef623c5f8d521@mail.gmail.com> On Sat, Sep 27, 2008 at 10:01 PM, Nathan Bell wrote: > On Sun, Sep 28, 2008 at 12:34 AM, Geoffrey Irving wrote: >> >> Is there an efficient way to implement a nonuniform gather operation >> in numpy? Specifically, I want to do something like >> >> n,m = 100,1000 >> X = random.uniform(size=n) >> K = random.randint(n, size=m) >> Y = random.uniform(size=m) >> >> for k,y in zip(K,Y): >> X[k] += y >> >> but I want it to be fast. The naive attempt "X[K] += Y" does not >> work, since the slice assumes the indices don't repeat. >> > > I don't know of numpy solution, but in scipy you could use a sparse > matrix to perform the operation. I think the following does what you > want. > > from scipy.sparse import coo_matrix > X += coo_matrix( (Y, (K,zeros(m,dtype=int)), shape=(n,1)).sum(axis=1) > > This reduces to a simple C++ loop, so speed should be good: > http://projects.scipy.org/scipy/scipy/browser/trunk/scipy/sparse/sparsetools/coo.h#L139 Thanks. That works great. A slightly cleaner version is X += coo_matrix((Y, (K, zeros_like(K)))).sum(axis=1) The next question is: is there a similar way that generalizes to the case where X is n by 3 and Y is m by 3 (besides the obvious loop over range(3), that is)? Geoffrey From matthieu.brucher at gmail.com Mon Sep 29 04:00:08 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 29 Sep 2008 10:00:08 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: Message-ID: I've tested the latest Numpy on a RHEL 4, and I got this result : Running unit tests for numpy NumPy version 1.2.0 NumPy is installed in /.../BRUCHER/local/lib/python2.5/site-packages/numpy Python version 2.5.2 (r252:60911, Sep 29 2008, 09:47:20) [GCC Intel(R) C++ gcc 3.4 mode] nose version 0.10.3 ...........................................................................................................................................................................................................................................................................................................................................................................................................................................FFF.............K............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ctypes is not available on this python: skipping the test (import error was: ctypes is not available.) .No distutils available, skipping test. ..... ====================================================================== FAIL: test_umath.TestComplexFunctions.test_branch_cuts(, -0.5, 1j, 1, -1, True) ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/nose/case.py", line 182, in runTest self.test(*self.arg) File "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", line 333, in _check_branch_cut assert np.all(np.absolute(y0.imag - ym.imag*im_sign) < atol), (y0, ym) AssertionError: (array([-0.69314718+3.14159265j]), array([-0.69314718+3.14159265j])) ====================================================================== FAIL: test_umath.TestComplexFunctions.test_branch_cuts(, -0.5, 1j, 1, -1, True) ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/nose/case.py", line 182, in runTest self.test(*self.arg) File "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", line 333, in _check_branch_cut assert np.all(np.absolute(y0.imag - ym.imag*im_sign) < atol), (y0, ym) AssertionError: (array([-0.30103+1.36437635j]), array([-0.30103+1.36437635j])) ====================================================================== FAIL: test_umath.TestComplexFunctions.test_branch_cuts(, -1.5, 1j, 1, -1, True) ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/nose/case.py", line 182, in runTest self.test(*self.arg) File "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", line 333, in _check_branch_cut assert np.all(np.absolute(y0.imag - ym.imag*im_sign) < atol), (y0, ym) AssertionError: (array([-0.69314718+3.14159265j]), array([-0.69314718+3.14159265j])) ---------------------------------------------------------------------- Ran 1726 tests in 7.533s FAILED (KNOWNFAIL=1, failures=3) So I have 3 failures that are unexpected. Matthieu 2008/9/26 Jarrod Millman : > I'm pleased to announce the release of NumPy 1.2.0. > > NumPy is the fundamental package needed for scientific computing with > Python. It contains: > > * a powerful N-dimensional array object > * sophisticated (broadcasting) functions > * basic linear algebra functions > * basic Fourier transforms > * sophisticated random number capabilities > * tools for integrating Fortran code. > > Besides it's obvious scientific uses, NumPy can also be used as an > efficient multi-dimensional container of generic data. Arbitrary > data-types can be defined. This allows NumPy to seamlessly and > speedily integrate with a wide-variety of databases. > > This minor release comes almost four months after the 1.1.0 > release. The major features of this release are a new > testing framework and huge amount of documentation work. It > also includes a some minor API breakage scheduled in the > 1.1 release. > > Please note that NumPy 1.2.0 requires Python 2.4 or greater. > > For information, please see the release notes: > http://sourceforge.net/project/shownotes.php?group_id=1369&release_id=628858 > > You can download the release from here: > http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 > > Thank you to everybody who contributed to this release. > > Enjoy, > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Information System Engineer Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From cournapeau at cslab.kecl.ntt.co.jp Mon Sep 29 04:19:10 2008 From: cournapeau at cslab.kecl.ntt.co.jp (David Cournapeau) Date: Mon, 29 Sep 2008 17:19:10 +0900 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: Message-ID: <1222676350.21860.13.camel@bbc8> On Mon, 2008-09-29 at 10:00 +0200, Matthieu Brucher wrote: > I've tested the latest Numpy on a RHEL 4, and I got this result : > Hi Matthieu, Are you on 32 or 64 bits ? As you can see, those errors are test errors, not errors in the function themselves (tolerance too low; maybe influenced by compilers - maybe gcc 3 ? - , etc...). Any way, those seem harmless, cheers, David From matthieu.brucher at gmail.com Mon Sep 29 04:21:31 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 29 Sep 2008 10:21:31 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: <1222676350.21860.13.camel@bbc8> References: <1222676350.21860.13.camel@bbc8> Message-ID: 2008/9/29 David Cournapeau : > On Mon, 2008-09-29 at 10:00 +0200, Matthieu Brucher wrote: >> I've tested the latest Numpy on a RHEL 4, and I got this result : >> > > Hi Matthieu, > > Are you on 32 or 64 bits ? As you can see, those errors are test > errors, not errors in the function themselves (tolerance too low; maybe > influenced by compilers - maybe gcc 3 ? - , etc...). Any way, those seem > harmless, > > cheers, > > David Hi, I compiled Python with a 64bits compiler (icc 10.1.018) as well as Numpy (that reminds me that I should link against the MKL as well). I know it's only tolerance errors, but they still show up ;) Matthieu -- French PhD student Information System Engineer Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From cournapeau at cslab.kecl.ntt.co.jp Mon Sep 29 04:30:55 2008 From: cournapeau at cslab.kecl.ntt.co.jp (David Cournapeau) Date: Mon, 29 Sep 2008 17:30:55 +0900 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: <1222676350.21860.13.camel@bbc8> Message-ID: <1222677055.21860.16.camel@bbc8> On Mon, 2008-09-29 at 10:21 +0200, Matthieu Brucher wrote: > > Hi, I compiled Python with a 64bits compiler (icc 10.1.018) as well as > Numpy (that reminds me that I should link against the MKL as well). I > know it's only tolerance errors, but they still show up ;) Did you compile with the -fast option ? Because this flag is not harmless with numpy in my experience (same for sunsstudio; in the later, numpy crashes quite quickly, actually). I don't know if this is a bug in numpy or in the compilers. cheers, David From matthieu.brucher at gmail.com Mon Sep 29 04:33:45 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 29 Sep 2008 10:33:45 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: <1222677055.21860.16.camel@bbc8> References: <1222676350.21860.13.camel@bbc8> <1222677055.21860.16.camel@bbc8> Message-ID: 2008/9/29 David Cournapeau : > On Mon, 2008-09-29 at 10:21 +0200, Matthieu Brucher wrote: >> >> Hi, I compiled Python with a 64bits compiler (icc 10.1.018) as well as >> Numpy (that reminds me that I should link against the MKL as well). I >> know it's only tolerance errors, but they still show up ;) > > Did you compile with the -fast option ? Because this flag is not > harmless with numpy in my experience (same for sunsstudio; in the later, > numpy crashes quite quickly, actually). I don't know if this is a bug in > numpy or in the compilers. No, I used -xW -axP because I don't have a Core 2 Duo computer (Opteron) Matthieu -- French PhD student Information System Engineer Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From pav at iki.fi Mon Sep 29 05:59:17 2008 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 29 Sep 2008 09:59:17 +0000 (UTC) Subject: [Numpy-discussion] ANN: NumPy 1.2.0 References: <1222676350.21860.13.camel@bbc8> Message-ID: Mon, 29 Sep 2008 17:19:10 +0900, David Cournapeau wrote: > On Mon, 2008-09-29 at 10:00 +0200, Matthieu Brucher wrote: >> I've tested the latest Numpy on a RHEL 4, and I got this result : >> >> > Hi Matthieu, > > Are you on 32 or 64 bits ? As you can see, those errors are test > errors, not errors in the function themselves (tolerance too low; maybe > influenced by compilers - maybe gcc 3 ? - , etc...). Any way, those seem > harmless, I think the errors may imply that with your compiler, numpy gives >>> np.log(-1 + 0j) -3.1415926535897931j or something similar "wrong" behavior at the branch cut of the logarithm, instead of >>> np.log(-1 + 0j) 3.1415926535897931j >>> np.log(-1 - 1e-99j) -3.1415926535897931j >>> np.log(-1 + 1e-99j) 3.1415926535897931j I'd guess this is typically harmless, but may lead to problems if your code relies on the choice of branch at the branch cut. Could you check if this is really the case? If log seems to work OK, then it's a problem with the test and not the functions. The current implementation of complex log seemed to work on other platforms, but maybe icc has some features that break it in this respect. -- Pauli Virtanen From matthieu.brucher at gmail.com Mon Sep 29 06:07:53 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 29 Sep 2008 12:07:53 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: <1222676350.21860.13.camel@bbc8> Message-ID: >>>> np.log(-1 + 0j) > 3.1415926535897931j >>>> np.log(-1 - 1e-99j) > -3.1415926535897931j >>>> np.log(-1 + 1e-99j) > 3.1415926535897931j > > I'd guess this is typically harmless, but may lead to problems if your > code relies on the choice of branch at the branch cut. > > Could you check if this is really the case? If log seems to work OK, then > it's a problem with the test and not the functions. The result is what you expected. BTW, I can confirm that the latest official MKL does not work with numpy, as it is explained on the Intel forum (http://software.intel.com/en-us/forums/intel-math-kernel-library/topic/60460). I get the i_free not defined issue. Matthieu -- French PhD student Information System Engineer Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From pav at iki.fi Mon Sep 29 06:28:28 2008 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 29 Sep 2008 10:28:28 +0000 (UTC) Subject: [Numpy-discussion] ANN: NumPy 1.2.0 References: <1222676350.21860.13.camel@bbc8> Message-ID: Mon, 29 Sep 2008 12:07:53 +0200, Matthieu Brucher wrote: >>>>> np.log(-1 + 0j) >> 3.1415926535897931j >>>>> np.log(-1 - 1e-99j) >> -3.1415926535897931j >>>>> np.log(-1 + 1e-99j) >> 3.1415926535897931j >> >> I'd guess this is typically harmless, but may lead to problems if your >> code relies on the choice of branch at the branch cut. >> >> Could you check if this is really the case? If log seems to work OK, >> then it's a problem with the test and not the functions. > > The result is what you expected. Do you mean that the problem is with the test, or with the branch cut of log? -- Pauli Virtanen From matthieu.brucher at gmail.com Mon Sep 29 06:33:32 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 29 Sep 2008 12:33:32 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: <1222676350.21860.13.camel@bbc8> Message-ID: 2008/9/29 Pauli Virtanen : > Mon, 29 Sep 2008 12:07:53 +0200, Matthieu Brucher wrote: > >>>>>> np.log(-1 + 0j) >>> 3.1415926535897931j >>>>>> np.log(-1 - 1e-99j) >>> -3.1415926535897931j >>>>>> np.log(-1 + 1e-99j) >>> 3.1415926535897931j >>> >>> I'd guess this is typically harmless, but may lead to problems if your >>> code relies on the choice of branch at the branch cut. >>> >>> Could you check if this is really the case? If log seems to work OK, >>> then it's a problem with the test and not the functions. >> >> The result is what you expected. > > Do you mean that the problem is with the test, or with the branch cut of > log? I don't know, but the results are : >>> np.log(-1 + 0j) 3.1415926535897931j >>> np.log(-1 - 1e-99j) -3.1415926535897931j >>> np.log(-1 + 1e-99j) 3.1415926535897931j Matthieu -- French PhD student Information System Engineer Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From pav at iki.fi Mon Sep 29 06:44:17 2008 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 29 Sep 2008 10:44:17 +0000 (UTC) Subject: [Numpy-discussion] ANN: NumPy 1.2.0 References: <1222676350.21860.13.camel@bbc8> Message-ID: Mon, 29 Sep 2008 12:33:32 +0200, Matthieu Brucher wrote: > 2008/9/29 Pauli Virtanen : >> Mon, 29 Sep 2008 12:07:53 +0200, Matthieu Brucher wrote: >> >>>>>>> np.log(-1 + 0j) >>>> 3.1415926535897931j >>>>>>> np.log(-1 - 1e-99j) >>>> -3.1415926535897931j >>>>>>> np.log(-1 + 1e-99j) >>>> 3.1415926535897931j >>>> >>>> I'd guess this is typically harmless, but may lead to problems if >>>> your code relies on the choice of branch at the branch cut. >>>> >>>> Could you check if this is really the case? If log seems to work OK, >>>> then it's a problem with the test and not the functions. >>> >>> The result is what you expected. >> >> Do you mean that the problem is with the test, or with the branch cut >> of log? > > I don't know, but the results are : > >>>> np.log(-1 + 0j) > 3.1415926535897931j >>>> np.log(-1 - 1e-99j) > -3.1415926535897931j >>>> np.log(-1 + 1e-99j) > 3.1415926535897931j Ok, then the problem must be that with ICC, numpy's log does not respect the negative zero in the imaginary part. On my platform, it does: >>> import numpy as np >>> x = 1 + 0j >>> np.log(-x) -3.1415926535897931j >>> np.log(-1 - 0j) 3.1415926535897931j So probably also this test should be marked as a known failure (for some platforms). But this is probably harmless anyway. -- Pauli Virtanen From arnar.flatberg at gmail.com Mon Sep 29 09:29:16 2008 From: arnar.flatberg at gmail.com (Arnar Flatberg) Date: Mon, 29 Sep 2008 15:29:16 +0200 Subject: [Numpy-discussion] profiling line by line In-Reply-To: <3d375d730809261736lf472811t6f34934d40d4ba55@mail.gmail.com> References: <3d375d730809150818m384460c9i5e7bd4b622feeaeb@mail.gmail.com> <5d3194020809150913k179156b9g263754b5836787f1@mail.gmail.com> <3d375d730809161856x5403d44eu89d2be6cd8c1204f@mail.gmail.com> <5d3194020809170512idfa560s6548ffcca406c325@mail.gmail.com> <3d375d730809261736lf472811t6f34934d40d4ba55@mail.gmail.com> Message-ID: <5d3194020809290629i6fbdf891wdca0974a2b8ec56c@mail.gmail.com> On Sat, Sep 27, 2008 at 2:36 AM, Robert Kern wrote: > On Wed, Sep 17, 2008 at 07:12, Arnar Flatberg > wrote: > > > > On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern > wrote: > > There is now! Excellent work! I've had no problem with using the ipython-support (lprun), and it works for all my usecases, so far. > >Feel free to add it as "highly recommended" in the IPython > >installation instructions, though. :-) >Most certainly will. It would be better to have a more stable >release/download link than your hg repo. Do you have a page or a >Pypi link for it yet? A little readme? I thought the hg-repos distribution was painless. The doc string says it all for usage, however a note on the ipy_user_conf setup should probably be documented some other place than just this thread. Thanks, Arnar -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmoses at engineering.ucsb.edu Mon Sep 29 12:52:47 2008 From: pgmoses at engineering.ucsb.edu (pgmoses) Date: Mon, 29 Sep 2008 09:52:47 -0700 Subject: [Numpy-discussion] numpy installation as non root using site.cfg In-Reply-To: <3d375d730809261615oada4b56h2eb42428aac51b43@mail.gmail.com> References: <19697706.post@talk.nabble.com> <3d375d730809261615oada4b56h2eb42428aac51b43@mail.gmail.com> Message-ID: <2796ce999f10ffd49733e9ece6a7b7e1@engineering.ucsb.edu> Thank you Robert It is now possible for me to install numpy. I get an error with the defaults libraries, which I have reported in a pending mail to the mailen list. Then I tried to use the intel mkl libraries in this way: [mkl] library_dirs = /opt/intel/mkl70/lib/32 mkl_libs = mkl, guide lapack_libs = mkl_lapack include_dirs = /opt/intel/mkl70/include I can still install numpy but when I import numpy I get an error. pgmoses at head ~]$ python Python 2.5.2 (r252:60911, Sep 26 2008, 14:15:17) [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-53)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy Traceback (most recent call last): File "", line 1, in File "/data/pgmoses/lib/python/numpy/__init__.py", line 103, in import linalg File "/data/pgmoses/lib/python/numpy/linalg/__init__.py", line 4, in from linalg import * File "/data/pgmoses/lib/python/numpy/linalg/linalg.py", line 29, in from numpy.linalg import lapack_lite ImportError: /data/pgmoses/lib/python/numpy/linalg/lapack_lite.so: undefined symbol: dgesdd_ >>> It seem like my inclusion of the mkl libraries did not work, since it is trying to use lapack_lite.so. I was my impression that it should work any way. What is the underlying problem here? Best regards, Poul Georg Moses On 4:15 pm 09/26/08 "Robert Kern" wrote: > On Fri, Sep 26, 2008 at 18:07, Poul Georg Moses > wrote: > > > > Dear mailing list > > > > I am trying to install numpy-1.1.1 on a cluster where I do not have > > root permission. I have succesfully installed \ > > python2.5.2. I am using the cite.cfg file to include the blas and > > atlas libraries. I have done that by setting the\ > > defaults to: > > library_dirs = /usr/lib > > include_dirs = /usr/include > > You need > > [DEFAULT] > library_dirs = /usr/lib > include_dirs = /usr/include > > just like it is in the site.cfg.example > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Mon Sep 29 13:05:09 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 11:05:09 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: Message-ID: On Mon, Sep 29, 2008 at 2:00 AM, Matthieu Brucher < matthieu.brucher at gmail.com> wrote: > I've tested the latest Numpy on a RHEL 4, and I got this result : > > Running unit tests for numpy > NumPy version 1.2.0 > NumPy is installed in /.../BRUCHER/local/lib/python2.5/site-packages/numpy > Python version 2.5.2 (r252:60911, Sep 29 2008, 09:47:20) [GCC Intel(R) > C++ gcc 3.4 mode] > nose version 0.10.3 > > ...........................................................................................................................................................................................................................................................................................................................................................................................................................................FFF.............K.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................. > > ..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ctypes > is not available on this python: skipping the test (import error was: > ctypes is not available.) > .No distutils available, skipping test. > ..... > ====================================================================== > FAIL: test_umath.TestComplexFunctions.test_branch_cuts(, > -0.5, 1j, 1, -1, True) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/nose/case.py", > line 182, in runTest > self.test(*self.arg) > File > "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", > line 333, in _check_branch_cut > assert np.all(np.absolute(y0.imag - ym.imag*im_sign) < atol), (y0, ym) > AssertionError: (array([-0.69314718+3.14159265j]), > array([-0.69314718+3.14159265j])) > > ====================================================================== > FAIL: test_umath.TestComplexFunctions.test_branch_cuts( 'log10'>, -0.5, 1j, 1, -1, True) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/nose/case.py", > line 182, in runTest > self.test(*self.arg) > File > "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", > line 333, in _check_branch_cut > assert np.all(np.absolute(y0.imag - ym.imag*im_sign) < atol), (y0, ym) > AssertionError: (array([-0.30103+1.36437635j]), > array([-0.30103+1.36437635j])) > > ====================================================================== > FAIL: test_umath.TestComplexFunctions.test_branch_cuts( 'log1p'>, -1.5, 1j, 1, -1, True) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/nose/case.py", > line 182, in runTest > self.test(*self.arg) > File > "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", > line 333, in _check_branch_cut > assert np.all(np.absolute(y0.imag - ym.imag*im_sign) < atol), (y0, ym) > AssertionError: (array([-0.69314718+3.14159265j]), > array([-0.69314718+3.14159265j])) > Using assert for this sort of thing is a bug, assert goes away with the -OO options. This needs to be fixed. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 29 13:09:10 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Sep 2008 12:09:10 -0500 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: Message-ID: <3d375d730809291009j1db8550cv182d7f4a1776cf0b@mail.gmail.com> On Mon, Sep 29, 2008 at 12:05, Charles R Harris wrote: > > > On Mon, Sep 29, 2008 at 2:00 AM, Matthieu Brucher > wrote: >> >> I've tested the latest Numpy on a RHEL 4, and I got this result : >> >> Running unit tests for numpy >> NumPy version 1.2.0 >> NumPy is installed in /.../BRUCHER/local/lib/python2.5/site-packages/numpy >> Python version 2.5.2 (r252:60911, Sep 29 2008, 09:47:20) [GCC Intel(R) >> C++ gcc 3.4 mode] >> nose version 0.10.3 >> >> ...........................................................................................................................................................................................................................................................................................................................................................................................................................................FFF.............K.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................. >> >> ..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ctypes >> is not available on this python: skipping the test (import error was: >> ctypes is not available.) >> .No distutils available, skipping test. >> ..... >> ====================================================================== >> FAIL: test_umath.TestComplexFunctions.test_branch_cuts(, >> -0.5, 1j, 1, -1, True) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File >> "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/nose/case.py", >> line 182, in runTest >> self.test(*self.arg) >> File >> "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", >> line 333, in _check_branch_cut >> assert np.all(np.absolute(y0.imag - ym.imag*im_sign) < atol), (y0, ym) >> AssertionError: (array([-0.69314718+3.14159265j]), >> array([-0.69314718+3.14159265j])) >> >> ====================================================================== >> FAIL: test_umath.TestComplexFunctions.test_branch_cuts(> 'log10'>, -0.5, 1j, 1, -1, True) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File >> "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/nose/case.py", >> line 182, in runTest >> self.test(*self.arg) >> File >> "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", >> line 333, in _check_branch_cut >> assert np.all(np.absolute(y0.imag - ym.imag*im_sign) < atol), (y0, ym) >> AssertionError: (array([-0.30103+1.36437635j]), >> array([-0.30103+1.36437635j])) >> >> ====================================================================== >> FAIL: test_umath.TestComplexFunctions.test_branch_cuts(> 'log1p'>, -1.5, 1j, 1, -1, True) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File >> "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/nose/case.py", >> line 182, in runTest >> self.test(*self.arg) >> File >> "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", >> line 333, in _check_branch_cut >> assert np.all(np.absolute(y0.imag - ym.imag*im_sign) < atol), (y0, ym) >> AssertionError: (array([-0.69314718+3.14159265j]), >> array([-0.69314718+3.14159265j])) > > Using assert for this sort of thing is a bug, assert goes away with the -OO > options. This needs to be fixed. These asserts are in tests. That is exactly where we are supposed to use them. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pav at iki.fi Mon Sep 29 13:14:38 2008 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 29 Sep 2008 17:14:38 +0000 (UTC) Subject: [Numpy-discussion] ANN: NumPy 1.2.0 References: Message-ID: Mon, 29 Sep 2008 11:05:09 -0600, Charles R Harris wrote: [clip: assert in tests] > Using assert for this sort of thing is a bug, assert goes away with the > -OO options. This needs to be fixed. "assert FOO" is used in quite many of Numpy's test cases, and it appears Nose endorses using assert (and uses it in its own tests). Should we go ahead and change all "assert FOO" in Numpy's tests to something like 'self.failUnless'? -- Pauli Virtanen From charlesr.harris at gmail.com Mon Sep 29 13:12:58 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 11:12:58 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: <3d375d730809291009j1db8550cv182d7f4a1776cf0b@mail.gmail.com> References: <3d375d730809291009j1db8550cv182d7f4a1776cf0b@mail.gmail.com> Message-ID: On Mon, Sep 29, 2008 at 11:09 AM, Robert Kern wrote: > On Mon, Sep 29, 2008 at 12:05, Charles R Harris > wrote: > > > > > > On Mon, Sep 29, 2008 at 2:00 AM, Matthieu Brucher > > wrote: > >> > >> I've tested the latest Numpy on a RHEL 4, and I got this result : > >> > >> Running unit tests for numpy > >> NumPy version 1.2.0 > >> NumPy is installed in > /.../BRUCHER/local/lib/python2.5/site-packages/numpy > >> Python version 2.5.2 (r252:60911, Sep 29 2008, 09:47:20) [GCC Intel(R) > >> C++ gcc 3.4 mode] > >> nose version 0.10.3 > >> > >> > ...........................................................................................................................................................................................................................................................................................................................................................................................................................................FFF.............K............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... > ... > >> > >> > ..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ctypes > >> is not available on this python: skipping the test (import error was: > >> ctypes is not available.) > >> .No distutils available, skipping test. > >> ..... > >> ====================================================================== > >> FAIL: test_umath.TestComplexFunctions.test_branch_cuts(, > >> -0.5, 1j, 1, -1, True) > >> ---------------------------------------------------------------------- > >> Traceback (most recent call last): > >> File > >> > "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/nose/case.py", > >> line 182, in runTest > >> self.test(*self.arg) > >> File > >> > "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", > >> line 333, in _check_branch_cut > >> assert np.all(np.absolute(y0.imag - ym.imag*im_sign) < atol), (y0, > ym) > >> AssertionError: (array([-0.69314718+3.14159265j]), > >> array([-0.69314718+3.14159265j])) > >> > >> ====================================================================== > >> FAIL: test_umath.TestComplexFunctions.test_branch_cuts( >> 'log10'>, -0.5, 1j, 1, -1, True) > >> ---------------------------------------------------------------------- > >> Traceback (most recent call last): > >> File > >> > "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/nose/case.py", > >> line 182, in runTest > >> self.test(*self.arg) > >> File > >> > "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", > >> line 333, in _check_branch_cut > >> assert np.all(np.absolute(y0.imag - ym.imag*im_sign) < atol), (y0, > ym) > >> AssertionError: (array([-0.30103+1.36437635j]), > >> array([-0.30103+1.36437635j])) > >> > >> ====================================================================== > >> FAIL: test_umath.TestComplexFunctions.test_branch_cuts( >> 'log1p'>, -1.5, 1j, 1, -1, True) > >> ---------------------------------------------------------------------- > >> Traceback (most recent call last): > >> File > >> > "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/nose/case.py", > >> line 182, in runTest > >> self.test(*self.arg) > >> File > >> > "/data/pau112/INNO/BRUCHER/local/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", > >> line 333, in _check_branch_cut > >> assert np.all(np.absolute(y0.imag - ym.imag*im_sign) < atol), (y0, > ym) > >> AssertionError: (array([-0.69314718+3.14159265j]), > >> array([-0.69314718+3.14159265j])) > > > > Using assert for this sort of thing is a bug, assert goes away with the > -OO > > options. This needs to be fixed. > > These asserts are in tests. That is exactly where we are supposed to use > them. > Wrong, asserts are for debugging, not for general tests. Using them will give incorrect results when running the tests. This tends to be a problem on windows platforms. It needs to be fixed. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 29 13:23:41 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 11:23:41 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: Message-ID: On Mon, Sep 29, 2008 at 11:14 AM, Pauli Virtanen wrote: > Mon, 29 Sep 2008 11:05:09 -0600, Charles R Harris wrote: > [clip: assert in tests] > > Using assert for this sort of thing is a bug, assert goes away with the > > -OO options. This needs to be fixed. > > "assert FOO" is used in quite many of Numpy's test cases, and it appears > Nose endorses using assert (and uses it in its own tests). > > Should we go ahead and change all "assert FOO" in Numpy's tests to > something like 'self.failUnless'? > I fixed the np.testing functions to use if (foo) : raise AssertionError("Da Bomb") That way no changes to exception catching are needed, but the error will still be raised when the -OO option is used. Strictly speaking, it isn't quite right, but so it goes... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 29 13:27:46 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Sep 2008 12:27:46 -0500 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: Message-ID: <3d375d730809291027n3fa901b8m22f00c83a3a98822@mail.gmail.com> On Mon, Sep 29, 2008 at 12:23, Charles R Harris wrote: > > > On Mon, Sep 29, 2008 at 11:14 AM, Pauli Virtanen wrote: >> >> Mon, 29 Sep 2008 11:05:09 -0600, Charles R Harris wrote: >> [clip: assert in tests] >> > Using assert for this sort of thing is a bug, assert goes away with the >> > -OO options. This needs to be fixed. >> >> "assert FOO" is used in quite many of Numpy's test cases, and it appears >> Nose endorses using assert (and uses it in its own tests). >> >> Should we go ahead and change all "assert FOO" in Numpy's tests to >> something like 'self.failUnless'? > > I fixed the np.testing functions to use > > if (foo) : > raise AssertionError("Da Bomb") > > That way no changes to exception catching are needed, but the error will > still be raised when the -OO option is used. Strictly speaking, it isn't > quite right, but so it goes... When is -OO ever used for running tests? The assert statement is used for tests under nose. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Mon Sep 29 13:30:10 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 11:30:10 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: <3d375d730809291027n3fa901b8m22f00c83a3a98822@mail.gmail.com> References: <3d375d730809291027n3fa901b8m22f00c83a3a98822@mail.gmail.com> Message-ID: On Mon, Sep 29, 2008 at 11:27 AM, Robert Kern wrote: > On Mon, Sep 29, 2008 at 12:23, Charles R Harris > wrote: > > > > > > On Mon, Sep 29, 2008 at 11:14 AM, Pauli Virtanen wrote: > >> > >> Mon, 29 Sep 2008 11:05:09 -0600, Charles R Harris wrote: > >> [clip: assert in tests] > >> > Using assert for this sort of thing is a bug, assert goes away with > the > >> > -OO options. This needs to be fixed. > >> > >> "assert FOO" is used in quite many of Numpy's test cases, and it appears > >> Nose endorses using assert (and uses it in its own tests). > >> > >> Should we go ahead and change all "assert FOO" in Numpy's tests to > >> something like 'self.failUnless'? > > > > I fixed the np.testing functions to use > > > > if (foo) : > > raise AssertionError("Da Bomb") > > > > That way no changes to exception catching are needed, but the error will > > still be raised when the -OO option is used. Strictly speaking, it isn't > > quite right, but so it goes... > > When is -OO ever used for running tests? The assert statement is used > for tests under nose. When it is the default on windows platforms. There was a ticket.. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 29 13:35:27 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Sep 2008 12:35:27 -0500 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: <3d375d730809291027n3fa901b8m22f00c83a3a98822@mail.gmail.com> Message-ID: <3d375d730809291035y11a6e68cmb275beec4cd93e1@mail.gmail.com> On Mon, Sep 29, 2008 at 12:30, Charles R Harris wrote: > > On Mon, Sep 29, 2008 at 11:27 AM, Robert Kern wrote: >> >> On Mon, Sep 29, 2008 at 12:23, Charles R Harris >> wrote: >> > >> > >> > On Mon, Sep 29, 2008 at 11:14 AM, Pauli Virtanen wrote: >> >> >> >> Mon, 29 Sep 2008 11:05:09 -0600, Charles R Harris wrote: >> >> [clip: assert in tests] >> >> > Using assert for this sort of thing is a bug, assert goes away with >> >> > the >> >> > -OO options. This needs to be fixed. >> >> >> >> "assert FOO" is used in quite many of Numpy's test cases, and it >> >> appears >> >> Nose endorses using assert (and uses it in its own tests). >> >> >> >> Should we go ahead and change all "assert FOO" in Numpy's tests to >> >> something like 'self.failUnless'? >> > >> > I fixed the np.testing functions to use >> > >> > if (foo) : >> > raise AssertionError("Da Bomb") >> > >> > That way no changes to exception catching are needed, but the error will >> > still be raised when the -OO option is used. Strictly speaking, it isn't >> > quite right, but so it goes... >> >> When is -OO ever used for running tests? The assert statement is used >> for tests under nose. > > When it is the default on windows platforms. There was a ticket.. I see #893, which talks about when -OO is used explicitly and when numpy is part of a py2exe application. This does not refer to any default configuration on Windows platforms. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Mon Sep 29 13:40:26 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 11:40:26 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: <3d375d730809291035y11a6e68cmb275beec4cd93e1@mail.gmail.com> References: <3d375d730809291027n3fa901b8m22f00c83a3a98822@mail.gmail.com> <3d375d730809291035y11a6e68cmb275beec4cd93e1@mail.gmail.com> Message-ID: On Mon, Sep 29, 2008 at 11:35 AM, Robert Kern wrote: > On Mon, Sep 29, 2008 at 12:30, Charles R Harris > wrote: > > > > On Mon, Sep 29, 2008 at 11:27 AM, Robert Kern > wrote: > >> > >> On Mon, Sep 29, 2008 at 12:23, Charles R Harris > >> wrote: > >> > > >> > > >> > On Mon, Sep 29, 2008 at 11:14 AM, Pauli Virtanen wrote: > >> >> > >> >> Mon, 29 Sep 2008 11:05:09 -0600, Charles R Harris wrote: > >> >> [clip: assert in tests] > >> >> > Using assert for this sort of thing is a bug, assert goes away with > >> >> > the > >> >> > -OO options. This needs to be fixed. > >> >> > >> >> "assert FOO" is used in quite many of Numpy's test cases, and it > >> >> appears > >> >> Nose endorses using assert (and uses it in its own tests). > >> >> > >> >> Should we go ahead and change all "assert FOO" in Numpy's tests to > >> >> something like 'self.failUnless'? > >> > > >> > I fixed the np.testing functions to use > >> > > >> > if (foo) : > >> > raise AssertionError("Da Bomb") > >> > > >> > That way no changes to exception catching are needed, but the error > will > >> > still be raised when the -OO option is used. Strictly speaking, it > isn't > >> > quite right, but so it goes... > >> > >> When is -OO ever used for running tests? The assert statement is used > >> for tests under nose. > > > > When it is the default on windows platforms. There was a ticket.. > > I see #893, which talks about when -OO is used explicitly and when > numpy is part of a py2exe application. This does not refer to any > default configuration on Windows platforms. > Let's put it this way, if the tests were consistent with assert then they couldn't be run with the -OO option. If they *can* be run with the -OO option then they had better run correctly. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Sep 29 13:47:39 2008 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 29 Sep 2008 17:47:39 +0000 (UTC) Subject: [Numpy-discussion] ANN: NumPy 1.2.0 References: <3d375d730809291027n3fa901b8m22f00c83a3a98822@mail.gmail.com> <3d375d730809291035y11a6e68cmb275beec4cd93e1@mail.gmail.com> Message-ID: Mon, 29 Sep 2008 11:40:26 -0600, Charles R Harris wrote: [clip] > Let's put it this way, if the tests were consistent with assert then > they couldn't be run with the -OO option. If they *can* be run with the > -OO option then they had better run correctly. Perhaps the Nose author would have input on this? At least Nose seems to be the place where any warnings saying that tests won't work with -OO should go. -- Pauli Virtanen From robert.kern at gmail.com Mon Sep 29 13:50:55 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Sep 2008 12:50:55 -0500 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: <3d375d730809291027n3fa901b8m22f00c83a3a98822@mail.gmail.com> <3d375d730809291035y11a6e68cmb275beec4cd93e1@mail.gmail.com> Message-ID: <3d375d730809291050u30435533rb58a4549a3176c77@mail.gmail.com> On Mon, Sep 29, 2008 at 12:40, Charles R Harris wrote: > > On Mon, Sep 29, 2008 at 11:35 AM, Robert Kern wrote: >> >> On Mon, Sep 29, 2008 at 12:30, Charles R Harris >> wrote: >> > >> > On Mon, Sep 29, 2008 at 11:27 AM, Robert Kern >> > wrote: >> >> When is -OO ever used for running tests? The assert statement is used >> >> for tests under nose. >> > >> > When it is the default on windows platforms. There was a ticket.. >> >> I see #893, which talks about when -OO is used explicitly and when >> numpy is part of a py2exe application. This does not refer to any >> default configuration on Windows platforms. > > Let's put it this way, if the tests were consistent with assert then they > couldn't be run with the -OO option. If they *can* be run with the -OO > option then they had better run correctly. I'm afraid that I don't understand what you are referring to. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Mon Sep 29 14:07:43 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 12:07:43 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: <3d375d730809291050u30435533rb58a4549a3176c77@mail.gmail.com> References: <3d375d730809291027n3fa901b8m22f00c83a3a98822@mail.gmail.com> <3d375d730809291035y11a6e68cmb275beec4cd93e1@mail.gmail.com> <3d375d730809291050u30435533rb58a4549a3176c77@mail.gmail.com> Message-ID: On Mon, Sep 29, 2008 at 11:50 AM, Robert Kern wrote: > On Mon, Sep 29, 2008 at 12:40, Charles R Harris > wrote: > > > > On Mon, Sep 29, 2008 at 11:35 AM, Robert Kern > wrote: > >> > >> On Mon, Sep 29, 2008 at 12:30, Charles R Harris > >> wrote: > >> > > >> > On Mon, Sep 29, 2008 at 11:27 AM, Robert Kern > >> > wrote: > > >> >> When is -OO ever used for running tests? The assert statement is used > >> >> for tests under nose. > >> > > >> > When it is the default on windows platforms. There was a ticket.. > >> > >> I see #893, which talks about when -OO is used explicitly and when > >> numpy is part of a py2exe application. This does not refer to any > >> default configuration on Windows platforms. > > > > Let's put it this way, if the tests were consistent with assert then they > > couldn't be run with the -OO option. If they *can* be run with the -OO > > option then they had better run correctly. > > I'm afraid that I don't understand what you are referring to. > Asserts are transparent. If they go away, programs containing asserts should still run correctly. It's like bounds checking in array references, say in the std::vector class. If compiled with -DNODEBUG, the programs still run. Programs shouldn't use asserts for non-debugging purposes, they should raise real exceptions. If you think the numpy tests are essentially debugging tests, then they should all disappear when run with the -OO option, i.e., nose should do nothing. If nose wants to make that change, that is fine with me. But as is, incorrect failures may be reported and that is an error. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 29 14:17:03 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Sep 2008 13:17:03 -0500 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: <3d375d730809291027n3fa901b8m22f00c83a3a98822@mail.gmail.com> <3d375d730809291035y11a6e68cmb275beec4cd93e1@mail.gmail.com> <3d375d730809291050u30435533rb58a4549a3176c77@mail.gmail.com> Message-ID: <3d375d730809291117wa83dd1lfa7bcc5c04bd9601@mail.gmail.com> On Mon, Sep 29, 2008 at 13:07, Charles R Harris wrote: > > On Mon, Sep 29, 2008 at 11:50 AM, Robert Kern wrote: >> >> On Mon, Sep 29, 2008 at 12:40, Charles R Harris >> wrote: >> > >> > On Mon, Sep 29, 2008 at 11:35 AM, Robert Kern >> > wrote: >> >> >> >> On Mon, Sep 29, 2008 at 12:30, Charles R Harris >> >> wrote: >> >> > >> >> > On Mon, Sep 29, 2008 at 11:27 AM, Robert Kern >> >> > wrote: >> >> >> >> When is -OO ever used for running tests? The assert statement is >> >> >> used >> >> >> for tests under nose. >> >> > >> >> > When it is the default on windows platforms. There was a ticket.. >> >> >> >> I see #893, which talks about when -OO is used explicitly and when >> >> numpy is part of a py2exe application. This does not refer to any >> >> default configuration on Windows platforms. >> > >> > Let's put it this way, if the tests were consistent with assert then >> > they >> > couldn't be run with the -OO option. If they *can* be run with the -OO >> > option then they had better run correctly. >> >> I'm afraid that I don't understand what you are referring to. > > Asserts are transparent. If they go away, programs containing asserts should > still run correctly. It's like bounds checking in array references, say in > the std::vector class. If compiled with -DNODEBUG, the programs still run. > Programs shouldn't use asserts for non-debugging purposes, they should raise > real exceptions. If you think the numpy tests are essentially debugging > tests, then they should all disappear when run with the -OO option, i.e., > nose should do nothing. If nose wants to make that change, that is fine with > me. But as is, incorrect failures may be reported and that is an error. What failures? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Mon Sep 29 14:22:55 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 12:22:55 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: <3d375d730809291117wa83dd1lfa7bcc5c04bd9601@mail.gmail.com> References: <3d375d730809291027n3fa901b8m22f00c83a3a98822@mail.gmail.com> <3d375d730809291035y11a6e68cmb275beec4cd93e1@mail.gmail.com> <3d375d730809291050u30435533rb58a4549a3176c77@mail.gmail.com> <3d375d730809291117wa83dd1lfa7bcc5c04bd9601@mail.gmail.com> Message-ID: On Mon, Sep 29, 2008 at 12:17 PM, Robert Kern wrote: > On Mon, Sep 29, 2008 at 13:07, Charles R Harris > wrote: > > > > On Mon, Sep 29, 2008 at 11:50 AM, Robert Kern > wrote: > >> > >> On Mon, Sep 29, 2008 at 12:40, Charles R Harris > >> wrote: > >> > > >> > On Mon, Sep 29, 2008 at 11:35 AM, Robert Kern > >> > wrote: > >> >> > >> >> On Mon, Sep 29, 2008 at 12:30, Charles R Harris > >> >> wrote: > >> >> > > >> >> > On Mon, Sep 29, 2008 at 11:27 AM, Robert Kern < > robert.kern at gmail.com> > >> >> > wrote: > >> > >> >> >> When is -OO ever used for running tests? The assert statement is > >> >> >> used > >> >> >> for tests under nose. > >> >> > > >> >> > When it is the default on windows platforms. There was a ticket.. > >> >> > >> >> I see #893, which talks about when -OO is used explicitly and when > >> >> numpy is part of a py2exe application. This does not refer to any > >> >> default configuration on Windows platforms. > >> > > >> > Let's put it this way, if the tests were consistent with assert then > >> > they > >> > couldn't be run with the -OO option. If they *can* be run with the -OO > >> > option then they had better run correctly. > >> > >> I'm afraid that I don't understand what you are referring to. > > > > Asserts are transparent. If they go away, programs containing asserts > should > > still run correctly. It's like bounds checking in array references, say > in > > the std::vector class. If compiled with -DNODEBUG, the programs still > run. > > Programs shouldn't use asserts for non-debugging purposes, they should > raise > > real exceptions. If you think the numpy tests are essentially debugging > > tests, then they should all disappear when run with the -OO option, i.e., > > nose should do nothing. If nose wants to make that change, that is fine > with > > me. But as is, incorrect failures may be reported and that is an error. > > What failures? > Read the ticket. And if a test incorrectly passes, that is also an error. Wrong is wrong, programming ain't philosophy. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 29 14:32:37 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Sep 2008 13:32:37 -0500 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: <3d375d730809291027n3fa901b8m22f00c83a3a98822@mail.gmail.com> <3d375d730809291035y11a6e68cmb275beec4cd93e1@mail.gmail.com> <3d375d730809291050u30435533rb58a4549a3176c77@mail.gmail.com> <3d375d730809291117wa83dd1lfa7bcc5c04bd9601@mail.gmail.com> Message-ID: <3d375d730809291132r4be63199r6a779c84a4288b35@mail.gmail.com> On Mon, Sep 29, 2008 at 13:22, Charles R Harris wrote: > > On Mon, Sep 29, 2008 at 12:17 PM, Robert Kern wrote: >> >> On Mon, Sep 29, 2008 at 13:07, Charles R Harris >> wrote: >> > >> > On Mon, Sep 29, 2008 at 11:50 AM, Robert Kern >> > wrote: >> >> >> >> On Mon, Sep 29, 2008 at 12:40, Charles R Harris >> >> wrote: >> >> > >> >> > On Mon, Sep 29, 2008 at 11:35 AM, Robert Kern >> >> > wrote: >> >> >> >> >> >> On Mon, Sep 29, 2008 at 12:30, Charles R Harris >> >> >> wrote: >> >> >> > >> >> >> > On Mon, Sep 29, 2008 at 11:27 AM, Robert Kern >> >> >> > >> >> >> > wrote: >> >> >> >> >> >> When is -OO ever used for running tests? The assert statement is >> >> >> >> used >> >> >> >> for tests under nose. >> >> >> > >> >> >> > When it is the default on windows platforms. There was a ticket.. >> >> >> >> >> >> I see #893, which talks about when -OO is used explicitly and when >> >> >> numpy is part of a py2exe application. This does not refer to any >> >> >> default configuration on Windows platforms. >> >> > >> >> > Let's put it this way, if the tests were consistent with assert then >> >> > they >> >> > couldn't be run with the -OO option. If they *can* be run with the >> >> > -OO >> >> > option then they had better run correctly. >> >> >> >> I'm afraid that I don't understand what you are referring to. >> > >> > Asserts are transparent. If they go away, programs containing asserts >> > should >> > still run correctly. It's like bounds checking in array references, say >> > in >> > the std::vector class. If compiled with -DNODEBUG, the programs still >> > run. >> > Programs shouldn't use asserts for non-debugging purposes, they should >> > raise >> > real exceptions. If you think the numpy tests are essentially debugging >> > tests, then they should all disappear when run with the -OO option, >> > i.e., >> > nose should do nothing. If nose wants to make that change, that is fine >> > with >> > me. But as is, incorrect failures may be reported and that is an error. >> >> What failures? > > Read the ticket. And if a test incorrectly passes, that is also an error. > Wrong is wrong, programming ain't philosophy. And practicality beats purity. Just don't run the damn tests with -OO. Yes, it's an abuse of the assert statement which was intended for inline debugging inside production code. Unit tests are not production code. They don't have the same constraints or requirements. In particular, unit tests written for the nose framework use (and are encouraged to use) assert statements. Deal with it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From faltet at pytables.org Mon Sep 29 14:43:39 2008 From: faltet at pytables.org (Francesc Alted) Date: Mon, 29 Sep 2008 20:43:39 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: <3d375d730809291050u30435533rb58a4549a3176c77@mail.gmail.com> Message-ID: <200809292043.40547.faltet@pytables.org> A Monday 29 September 2008, Charles R Harris escrigu?: > On Mon, Sep 29, 2008 at 11:50 AM, Robert Kern wrote: > > On Mon, Sep 29, 2008 at 12:40, Charles R Harris > > > > wrote: > > > On Mon, Sep 29, 2008 at 11:35 AM, Robert Kern > > > > > > > wrote: > > >> On Mon, Sep 29, 2008 at 12:30, Charles R Harris > > >> > > >> wrote: > > >> > On Mon, Sep 29, 2008 at 11:27 AM, Robert Kern > > >> > > > >> > > > >> > wrote: > > >> >> When is -OO ever used for running tests? The assert statement > > >> >> is used for tests under nose. > > >> > > > >> > When it is the default on windows platforms. There was a > > >> > ticket.. > > >> > > >> I see #893, which talks about when -OO is used explicitly and > > >> when numpy is part of a py2exe application. This does not refer > > >> to any default configuration on Windows platforms. > > > > > > Let's put it this way, if the tests were consistent with assert > > > then they couldn't be run with the -OO option. If they *can* be > > > run with the -OO option then they had better run correctly. > > > > I'm afraid that I don't understand what you are referring to. > > Asserts are transparent. If they go away, programs containing asserts > should still run correctly. It's like bounds checking in array > references, say in the std::vector class. If compiled with -DNODEBUG, > the programs still run. Programs shouldn't use asserts for > non-debugging purposes, they should raise real exceptions. If you > think the numpy tests are essentially debugging tests, then they > should all disappear when run with the -OO option, i.e., nose should > do nothing. If nose wants to make that change, that is fine with me. > But as is, incorrect failures may be reported and that is an error. Plain assert clauses used to be the most common way to check for test units in the original "unittest" module. Later on, the authors started to introduce things like self.assert_() and family and warned that the use of the "assert" clause should be avoided (maybe thinking about possible changes in semantics inside Python itself, maybe thinking about the use of -OO). But I agree with Robert Kern that tests *should* never be run under -OO because they are *tests*, not production code, and using the -OO for running the test units is simply a wrong practice. However, in the long term it should be useful to get rid of all the plain asserts in the test suite. My two cents, -- Francesc Alted From charlesr.harris at gmail.com Mon Sep 29 14:44:13 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 12:44:13 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: <3d375d730809291132r4be63199r6a779c84a4288b35@mail.gmail.com> References: <3d375d730809291027n3fa901b8m22f00c83a3a98822@mail.gmail.com> <3d375d730809291035y11a6e68cmb275beec4cd93e1@mail.gmail.com> <3d375d730809291050u30435533rb58a4549a3176c77@mail.gmail.com> <3d375d730809291117wa83dd1lfa7bcc5c04bd9601@mail.gmail.com> <3d375d730809291132r4be63199r6a779c84a4288b35@mail.gmail.com> Message-ID: On Mon, Sep 29, 2008 at 12:32 PM, Robert Kern wrote: > On Mon, Sep 29, 2008 at 13:22, Charles R Harris > wrote: > > > > On Mon, Sep 29, 2008 at 12:17 PM, Robert Kern > wrote: > >> > >> On Mon, Sep 29, 2008 at 13:07, Charles R Harris > >> wrote: > >> > > >> > On Mon, Sep 29, 2008 at 11:50 AM, Robert Kern > >> > wrote: > >> >> > >> >> On Mon, Sep 29, 2008 at 12:40, Charles R Harris > >> >> wrote: > >> >> > > >> >> > On Mon, Sep 29, 2008 at 11:35 AM, Robert Kern < > robert.kern at gmail.com> > >> >> > wrote: > >> >> >> > >> >> >> On Mon, Sep 29, 2008 at 12:30, Charles R Harris > >> >> >> wrote: > >> >> >> > > >> >> >> > On Mon, Sep 29, 2008 at 11:27 AM, Robert Kern > >> >> >> > > >> >> >> > wrote: > >> >> > >> >> >> >> When is -OO ever used for running tests? The assert statement > is > >> >> >> >> used > >> >> >> >> for tests under nose. > >> >> >> > > >> >> >> > When it is the default on windows platforms. There was a > ticket.. > >> >> >> > >> >> >> I see #893, which talks about when -OO is used explicitly and when > >> >> >> numpy is part of a py2exe application. This does not refer to any > >> >> >> default configuration on Windows platforms. > >> >> > > >> >> > Let's put it this way, if the tests were consistent with assert > then > >> >> > they > >> >> > couldn't be run with the -OO option. If they *can* be run with the > >> >> > -OO > >> >> > option then they had better run correctly. > >> >> > >> >> I'm afraid that I don't understand what you are referring to. > >> > > >> > Asserts are transparent. If they go away, programs containing asserts > >> > should > >> > still run correctly. It's like bounds checking in array references, > say > >> > in > >> > the std::vector class. If compiled with -DNODEBUG, the programs still > >> > run. > >> > Programs shouldn't use asserts for non-debugging purposes, they should > >> > raise > >> > real exceptions. If you think the numpy tests are essentially > debugging > >> > tests, then they should all disappear when run with the -OO option, > >> > i.e., > >> > nose should do nothing. If nose wants to make that change, that is > fine > >> > with > >> > me. But as is, incorrect failures may be reported and that is an > error. > >> > >> What failures? > > > > Read the ticket. And if a test incorrectly passes, that is also an error. > > Wrong is wrong, programming ain't philosophy. > > And practicality beats purity. Just don't run the damn tests with -OO. > Yes, it's an abuse of the assert statement which was intended for > inline debugging inside production code. Unit tests are not production > code. They don't have the same constraints or requirements. In > particular, unit tests written for the nose framework use (and are > encouraged to use) assert statements. Deal with it. > Knowingly writing incorrect public code when it is easy to do it right can't be justified. If nose has a bug in this regard, nose should be fixed. And I think the numpy unit tests *are* production code, they give us information on the behavior of the different platforms and compilers out there. And what if there are *real* errors that only show up if python is run with the -OO options? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 29 14:49:26 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Sep 2008 13:49:26 -0500 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: <200809292043.40547.faltet@pytables.org> References: <3d375d730809291050u30435533rb58a4549a3176c77@mail.gmail.com> <200809292043.40547.faltet@pytables.org> Message-ID: <3d375d730809291149k7cf4b66ew57def1e91bee994a@mail.gmail.com> On Mon, Sep 29, 2008 at 13:43, Francesc Alted wrote: > Plain assert clauses used to be the most common way to check for test > units in the original "unittest" module. Later on, the authors started > to introduce things like self.assert_() and family and warned that the > use of the "assert" clause should be avoided (maybe thinking about > possible changes in semantics inside Python itself, maybe thinking > about the use of -OO). Mostly it was reporting, I think. With assertEquals(x,y), you get a nice message showing the values of x and y. This is obviated by nose, which can introspect to get the values. From the unittest documentation: """These methods are used instead of the assert statement so the test runner can accumulate all test results and produce a report.""" -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Mon Sep 29 15:01:56 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 13:01:56 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: <200809292043.40547.faltet@pytables.org> References: <3d375d730809291050u30435533rb58a4549a3176c77@mail.gmail.com> <200809292043.40547.faltet@pytables.org> Message-ID: On Mon, Sep 29, 2008 at 12:43 PM, Francesc Alted wrote: > A Monday 29 September 2008, Charles R Harris escrigu?: > > On Mon, Sep 29, 2008 at 11:50 AM, Robert Kern > wrote: > > > On Mon, Sep 29, 2008 at 12:40, Charles R Harris > > > > > > wrote: > > > > On Mon, Sep 29, 2008 at 11:35 AM, Robert Kern > > > > > > > > > > wrote: > > > >> On Mon, Sep 29, 2008 at 12:30, Charles R Harris > > > >> > > > >> wrote: > > > >> > On Mon, Sep 29, 2008 at 11:27 AM, Robert Kern > > > >> > > > > >> > > > > >> > wrote: > > > >> >> When is -OO ever used for running tests? The assert statement > > > >> >> is used for tests under nose. > > > >> > > > > >> > When it is the default on windows platforms. There was a > > > >> > ticket.. > > > >> > > > >> I see #893, which talks about when -OO is used explicitly and > > > >> when numpy is part of a py2exe application. This does not refer > > > >> to any default configuration on Windows platforms. > > > > > > > > Let's put it this way, if the tests were consistent with assert > > > > then they couldn't be run with the -OO option. If they *can* be > > > > run with the -OO option then they had better run correctly. > > > > > > I'm afraid that I don't understand what you are referring to. > > > > Asserts are transparent. If they go away, programs containing asserts > > should still run correctly. It's like bounds checking in array > > references, say in the std::vector class. If compiled with -DNODEBUG, > > the programs still run. Programs shouldn't use asserts for > > non-debugging purposes, they should raise real exceptions. If you > > think the numpy tests are essentially debugging tests, then they > > should all disappear when run with the -OO option, i.e., nose should > > do nothing. If nose wants to make that change, that is fine with me. > > But as is, incorrect failures may be reported and that is an error. > > Plain assert clauses used to be the most common way to check for test > units in the original "unittest" module. Later on, the authors started > to introduce things like self.assert_() and family and warned that the > use of the "assert" clause should be avoided (maybe thinking about > possible changes in semantics inside Python itself, maybe thinking > about the use of -OO). > > But I agree with Robert Kern that tests *should* never be run under -OO > because they are *tests*, not production code, and using the -OO for > running the test units is simply a wrong practice. > What if someone abuses assert in production code, how are you going to catch that without using -OO? Assert abuse seems to be a common bug... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at pytables.org Mon Sep 29 15:08:29 2008 From: faltet at pytables.org (Francesc Alted) Date: Mon, 29 Sep 2008 21:08:29 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: <3d375d730809291149k7cf4b66ew57def1e91bee994a@mail.gmail.com> References: <200809292043.40547.faltet@pytables.org> <3d375d730809291149k7cf4b66ew57def1e91bee994a@mail.gmail.com> Message-ID: <200809292108.29832.faltet@pytables.org> A Monday 29 September 2008, Robert Kern escrigu?: > On Mon, Sep 29, 2008 at 13:43, Francesc Alted wrote: > > Plain assert clauses used to be the most common way to check for > > test units in the original "unittest" module. Later on, the > > authors started to introduce things like self.assert_() and family > > and warned that the use of the "assert" clause should be avoided > > (maybe thinking about possible changes in semantics inside Python > > itself, maybe thinking about the use of -OO). > > Mostly it was reporting, I think. With assertEquals(x,y), you get a > nice message showing the values of x and y. This is obviated by nose, > which can introspect to get the values. From the unittest > documentation: > > """These methods are used instead of the assert statement so the > test runner can accumulate all test results and produce a report.""" Maybe. At any rate, getting rid of plain asserts in test units seems to be a good practice generally speaking. Cheers, -- Francesc Alted From faltet at pytables.org Mon Sep 29 15:13:40 2008 From: faltet at pytables.org (Francesc Alted) Date: Mon, 29 Sep 2008 21:13:40 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: <200809292043.40547.faltet@pytables.org> Message-ID: <200809292113.41050.faltet@pytables.org> A Monday 29 September 2008, Charles R Harris escrigu?: > On Mon, Sep 29, 2008 at 12:43 PM, Francesc Alted wrote: > > A Monday 29 September 2008, Charles R Harris escrigu?: > > > On Mon, Sep 29, 2008 at 11:50 AM, Robert Kern > > > > > > > wrote: > > > > On Mon, Sep 29, 2008 at 12:40, Charles R Harris > > > > > > > > wrote: > > > > > On Mon, Sep 29, 2008 at 11:35 AM, Robert Kern > > > > > > > > > > > > > wrote: > > > > >> On Mon, Sep 29, 2008 at 12:30, Charles R Harris > > > > >> > > > > >> wrote: > > > > >> > On Mon, Sep 29, 2008 at 11:27 AM, Robert Kern > > > > >> > > > > > >> > > > > > >> > wrote: > > > > >> >> When is -OO ever used for running tests? The assert > > > > >> >> statement is used for tests under nose. > > > > >> > > > > > >> > When it is the default on windows platforms. There was a > > > > >> > ticket.. > > > > >> > > > > >> I see #893, which talks about when -OO is used explicitly > > > > >> and when numpy is part of a py2exe application. This does > > > > >> not refer to any default configuration on Windows platforms. > > > > > > > > > > Let's put it this way, if the tests were consistent with > > > > > assert then they couldn't be run with the -OO option. If they > > > > > *can* be run with the -OO option then they had better run > > > > > correctly. > > > > > > > > I'm afraid that I don't understand what you are referring to. > > > > > > Asserts are transparent. If they go away, programs containing > > > asserts should still run correctly. It's like bounds checking in > > > array references, say in the std::vector class. If compiled with > > > -DNODEBUG, the programs still run. Programs shouldn't use asserts > > > for non-debugging purposes, they should raise real exceptions. If > > > you think the numpy tests are essentially debugging tests, then > > > they should all disappear when run with the -OO option, i.e., > > > nose should do nothing. If nose wants to make that change, that > > > is fine with me. But as is, incorrect failures may be reported > > > and that is an error. > > > > Plain assert clauses used to be the most common way to check for > > test units in the original "unittest" module. Later on, the > > authors started to introduce things like self.assert_() and family > > and warned that the use of the "assert" clause should be avoided > > (maybe thinking about possible changes in semantics inside Python > > itself, maybe thinking about the use of -OO). > > > > But I agree with Robert Kern that tests *should* never be run under > > -OO because they are *tests*, not production code, and using the > > -OO for running the test units is simply a wrong practice. > > What if someone abuses assert in production code, how are you going > to catch that without using -OO? Assert abuse seems to be a common > bug... It is not. "assert" is meant to serve as a *debugging* tool, and Python interpreter is clever when removing it when using the '-OO'. That's a very poweful feature of the language when correctly used. If you want persistent checks, use an "if" clause instead. Cheers, -- Francesc Alted From aisaac at american.edu Mon Sep 29 15:29:18 2008 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 29 Sep 2008 15:29:18 -0400 Subject: [Numpy-discussion] should tests use assert? In-Reply-To: References: <3d375d730809291050u30435533rb58a4549a3176c77@mail.gmail.com> <200809292043.40547.faltet@pytables.org> Message-ID: <48E12C8E.2030207@american.edu> > On Mon, Sep 29, 2008 at 12:43 PM, Francesc Alted wrote:` >> But I agree with Robert Kern that tests *should* never be >> run under -OO >> because they are *tests*, not production code, and using >> the -OO for >> running the test units is simply a wrong practice. Charles R Harris wrote: > What if someone abuses assert in production code, how are you going to > catch that without using -OO? Assert abuse seems to be a common bug... Thanks for having this discussion in public; it has been interesting. Chuck's last point above gave me my first real sense of what he might be concerned about. Until then Robert's and Francesco's responses seemed quite complete to me. Thank you, Alan Isaac From peridot.faceted at gmail.com Mon Sep 29 16:36:36 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Mon, 29 Sep 2008 16:36:36 -0400 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: <200809292043.40547.faltet@pytables.org> References: <3d375d730809291050u30435533rb58a4549a3176c77@mail.gmail.com> <200809292043.40547.faltet@pytables.org> Message-ID: 2008/9/29 Francesc Alted : > But I agree with Robert Kern that tests *should* never be run under -OO > because they are *tests*, not production code, and using the -OO for > running the test units is simply a wrong practice. I have to disagree with this. What if some piece of code misbehaves when optimization is turned on? This frequently happens with C code, and I would not be too surprised if it occurred with python code. The easiest way, which I think is what Chuck was referring to as "abuse of asserts" is for someone to write something like try: while True: do_something() assert np.all(a<17) except AssertionError: pass This is of course a bug, though it may work, more or less, in non-optimized code. But if you can't run your test suite in optimized mode, it's going to be useless finding this bug. Perhaps more plausibly, someone might use assert to check the validity of user input. Then you turn on optimization and use input becomes unchecked... I support getting rid of assert statements in tests. How hard is it to create a function assert_ in numpy.Testing? assert np.all(a<17), "oops" becomes assert_(np.all(a<17), "oops") It could even be done automatically, or nearly. Anne From f.yw at hotmail.com Mon Sep 29 17:32:06 2008 From: f.yw at hotmail.com (frank wang) Date: Mon, 29 Sep 2008 15:32:06 -0600 Subject: [Numpy-discussion] Are there command similar as Matlab find command? In-Reply-To: <9457e7c80809250051y755e232bl8ca79847db186b2@mail.gmail.com> References: <200809222010.12025.pgmdevlist@gmail.com> <9911419a0809232336n89d6e4djcc59ad71f28732f5@mail.gmail.com> <9457e7c80809240810l6bf656aey70434d893200aa98@mail.gmail.com> <9457e7c80809250051y755e232bl8ca79847db186b2@mail.gmail.com> Message-ID: Hi, I am trying to find a command in numpy or python that perform similar function as Matlab find command. It will return the indexes of array that satisfy a condition. So far I have not found anything. Thanks Frank> Date: Thu, 25 Sep 2008 09:51:46 +0200> From: stefan at sun.ac.za> To: numpy-discussion at scipy.org> Subject: Re: [Numpy-discussion] loadtxt error> > 2008/9/24 frank wang :> > Thank you very much for all of you. I have downloaded the binary version> > 1.2rc and it fixed the problem.> >> > My special thanks to the person who created the window binary version for> > users who do not know or do not have the capacity to build the numpy from> > source.> > That would be David Cournapeau -- one of the few brave enough to venture there.> > Cheers> St?fan> _______________________________________________> Numpy-discussion mailing list> Numpy-discussion at scipy.org> http://projects.scipy.org/mailman/listinfo/numpy-discussion _________________________________________________________________ Stay up to date on your PC, the Web, and your mobile phone with Windows Live. http://clk.atdmt.com/MRT/go/msnnkwxp1020093185mrt/direct/01/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wnbell at gmail.com Mon Sep 29 17:45:51 2008 From: wnbell at gmail.com (Nathan Bell) Date: Mon, 29 Sep 2008 17:45:51 -0400 Subject: [Numpy-discussion] Are there command similar as Matlab find command? In-Reply-To: References: <200809222010.12025.pgmdevlist@gmail.com> <9911419a0809232336n89d6e4djcc59ad71f28732f5@mail.gmail.com> <9457e7c80809240810l6bf656aey70434d893200aa98@mail.gmail.com> <9457e7c80809250051y755e232bl8ca79847db186b2@mail.gmail.com> Message-ID: On Mon, Sep 29, 2008 at 5:32 PM, frank wang wrote: > Hi, > > I am trying to find a command in numpy or python that perform similar > function as Matlab find command. It will return the indexes of array that > satisfy a condition. So far I have not found anything. > If you're familiar with MATLAB, look here: http://www.scipy.org/NumPy_for_Matlab_Users In the table you'll find the following equivalence: find(a>0.5) <-> where(a>0.5) -- Nathan Bell wnbell at gmail.com http://graphics.cs.uiuc.edu/~wnbell/ From oliphant at enthought.com Mon Sep 29 17:48:23 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 29 Sep 2008 16:48:23 -0500 Subject: [Numpy-discussion] Are there command similar as Matlab find command? In-Reply-To: References: <200809222010.12025.pgmdevlist@gmail.com> <9911419a0809232336n89d6e4djcc59ad71f28732f5@mail.gmail.com> <9457e7c80809240810l6bf656aey70434d893200aa98@mail.gmail.com> <9457e7c80809250051y755e232bl8ca79847db186b2@mail.gmail.com> Message-ID: <48E14D27.5040604@enthought.com> frank wang wrote: > Hi, > > I am trying to find a command in numpy or python that perform similar > function as Matlab find command. It will return the indexes of array > that satisfy a condition. So far I have not found anything. There are several ways to do this, but what are you trying to do? Non-zero on the boolean array resulting from the condition is the most direct way: (a>30).nonzero() where(a>30) This returns a tuple of indices of length nd, where nd is the number of dimensions of a. (i.e. for 1-d case you need to extract the first element of the tuple to get the indices you want). But, if you are going to use these indices to access elements of the array, there are better ways to do that: a[a>30] compress(a>30, a) etc. -Travis From charlesr.harris at gmail.com Mon Sep 29 17:52:30 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 15:52:30 -0600 Subject: [Numpy-discussion] What is the sign of nan? Message-ID: Hi All, I've been cleaning up the ufunc loops and the sign function currently doesn't have a defined behavior for nans. This makes the results depend on the order/type of comparisons in the code, which looks fragile to me. So what should it return? I vote for nan but am open for suggestions. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 29 18:13:27 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 16:13:27 -0600 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: References: Message-ID: On Mon, Sep 29, 2008 at 3:52 PM, Charles R Harris wrote: > Hi All, > > I've been cleaning up the ufunc loops and the sign function currently > doesn't have a defined behavior for nans. This makes the results depend on > the order/type of comparisons in the code, which looks fragile to me. So > what should it return? I vote for nan but am open for suggestions. > And while we're at it, lets decide how to treat max/min when nans are involved. Or should we just say the behavior is undefined. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 29 18:28:04 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Sep 2008 17:28:04 -0500 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: References: Message-ID: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> On Mon, Sep 29, 2008 at 17:13, Charles R Harris wrote: > > On Mon, Sep 29, 2008 at 3:52 PM, Charles R Harris > wrote: >> >> Hi All, >> >> I've been cleaning up the ufunc loops and the sign function currently >> doesn't have a defined behavior for nans. This makes the results depend on >> the order/type of comparisons in the code, which looks fragile to me. So >> what should it return? I vote for nan but am open for suggestions. > > And while we're at it, lets decide how to treat max/min when nans are > involved. Or should we just say the behavior is undefined. When feasible, I would like float(s)->float functions to return NaN when given a NaN as an argument. At least as the main versions of the function. Specific NaN-ignoring functions can also be introduced, but as separate functions. I don't know what exactly to do about float->int functions (e.g. argmin). I also don't know how these should interact with the current seterr() state. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Catherine.M.Moroney at jpl.nasa.gov Mon Sep 29 18:39:29 2008 From: Catherine.M.Moroney at jpl.nasa.gov (Catherine Moroney) Date: Mon, 29 Sep 2008 15:39:29 -0700 Subject: [Numpy-discussion] problem calling SD.SDim.setname in pyhdf Message-ID: <917B6947-2B77-4C35-932D-7900EAD6AB6B@jpl.nasa.gov> Hi, I recently upgraded pyhdf from 0.7-3 to 0.8-2, and have noticed some problems with the SD.SDim interface. Specifically, the short test code attached works fine with version 0.7-3 and fails with version 0.8-2 when it's trying to assign a name to the dimension. I get the following error (below). Does anybody else have similar problems while trying to attach a name to an SDim object in the most recent version of pyhdf? I'm running on a Linux box, python version 2.5. etting Dimension: Dimension0 Traceback (most recent call last): File "./Test_pyhdf.py", line 35, in main() File "./Test_pyhdf.py", line 27, in main dim.setname(dname) File "/usr/lib64/python2.5/site-packages/pyhdf/SD.py", line 2856, in setname _checkErr('setname', status, 'cannot execute') File "/usr/lib64/python2.5/site-packages/pyhdf/error.py", line 24, in _checkErr raise HDF4Error, str pyhdf.error.HDF4Error: ('s', 'e', 't', 'n', 'a', 'm', 'e', ' ', ':', ' ', 'c', 'a', 'n', 'n', 'o', 't', ' ', 'e', 'x', 'e', 'c', 'u', 't', 'e') Thanks for any help, Catherine #! /usr/bin/env python from pyhdf import HDF from pyhdf import SD def main(): rank = (10,20) flhdf = SD.SD("Test_pyhdf.hdf",HDF.HC.CREATE|HDF.HC.WRITE| HDF.HC.TRUNC) sds = flhdf.create("Test SDS",HDF.HC.INT32,rank) for idim in range(0,2): dim = sds.dim(idim) dname = "Dimension" + str(idim) print "Setting Dimension: ",dname dtype = HDF.HC.FLOAT32 scale = [i for i in range(0,rank[idim])] dim.setname(dname) <----- CRASHES ON THIS LINE dim.units = "None" dim.setscale(dtype, scale) sds.endaccess() flhdf.end() if __name__ == "__main__": main() From charlesr.harris at gmail.com Mon Sep 29 18:40:28 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 16:40:28 -0600 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> Message-ID: On Mon, Sep 29, 2008 at 4:28 PM, Robert Kern wrote: > On Mon, Sep 29, 2008 at 17:13, Charles R Harris > wrote: > > > > On Mon, Sep 29, 2008 at 3:52 PM, Charles R Harris > > wrote: > >> > >> Hi All, > >> > >> I've been cleaning up the ufunc loops and the sign function currently > >> doesn't have a defined behavior for nans. This makes the results depend > on > >> the order/type of comparisons in the code, which looks fragile to me. So > >> what should it return? I vote for nan but am open for suggestions. > > > > And while we're at it, lets decide how to treat max/min when nans are > > involved. Or should we just say the behavior is undefined. > > When feasible, I would like float(s)->float functions to return NaN > when given a NaN as an argument. At least as the main versions of the > function. Specific NaN-ignoring functions can also be introduced, but > as separate functions. I don't know what exactly to do about > float->int functions (e.g. argmin). I also don't know how these should > interact with the current seterr() state. > So the proposition is, sign, max, min return nan when any of the arguments is nan. +1 Complex numbers are more complicated because we first compare the real parts, then the imaginary. Arguably 1 > 0 + nan*1j. I propose that the sign of a complex number containing nans should be nan, but I can't decide what should happen with max/min Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 29 18:54:31 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 16:54:31 -0600 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> Message-ID: On Mon, Sep 29, 2008 at 4:40 PM, Charles R Harris wrote: > > > On Mon, Sep 29, 2008 at 4:28 PM, Robert Kern wrote: > >> On Mon, Sep 29, 2008 at 17:13, Charles R Harris >> wrote: >> > >> > On Mon, Sep 29, 2008 at 3:52 PM, Charles R Harris >> > wrote: >> >> >> >> Hi All, >> >> >> >> I've been cleaning up the ufunc loops and the sign function currently >> >> doesn't have a defined behavior for nans. This makes the results depend >> on >> >> the order/type of comparisons in the code, which looks fragile to me. >> So >> >> what should it return? I vote for nan but am open for suggestions. >> > >> > And while we're at it, lets decide how to treat max/min when nans are >> > involved. Or should we just say the behavior is undefined. >> >> When feasible, I would like float(s)->float functions to return NaN >> when given a NaN as an argument. At least as the main versions of the >> function. Specific NaN-ignoring functions can also be introduced, but >> as separate functions. I don't know what exactly to do about >> float->int functions (e.g. argmin). I also don't know how these should >> interact with the current seterr() state. >> > > So the proposition is, sign, max, min return nan when any of the arguments > is nan. > > +1 > I also propose that all logical operators involving nan return false, i.e., ==, !=, <, <=, >, >=, and, or, xor, not. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 29 19:07:13 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Sep 2008 18:07:13 -0500 Subject: [Numpy-discussion] problem calling SD.SDim.setname in pyhdf In-Reply-To: <917B6947-2B77-4C35-932D-7900EAD6AB6B@jpl.nasa.gov> References: <917B6947-2B77-4C35-932D-7900EAD6AB6B@jpl.nasa.gov> Message-ID: <3d375d730809291607o59576d57m18c91d38e415b28d@mail.gmail.com> On Mon, Sep 29, 2008 at 17:39, Catherine Moroney wrote: > Hi, > > I recently upgraded pyhdf from 0.7-3 to 0.8-2, and have noticed some > problems with the SD.SDim interface. > > Specifically, the short test code attached works fine with version 0.7-3 > and fails with version 0.8-2 when it's trying to assign a name > to the dimension. > > I get the following error (below). Does anybody else have similar > problems > while trying to attach a name to an SDim object in the most recent > version > of pyhdf? > > I'm running on a Linux box, python version 2.5. > > etting Dimension: Dimension0 > Traceback (most recent call last): > File "./Test_pyhdf.py", line 35, in > main() > File "./Test_pyhdf.py", line 27, in main > dim.setname(dname) > File "/usr/lib64/python2.5/site-packages/pyhdf/SD.py", line 2856, > in setname > _checkErr('setname', status, 'cannot execute') > File "/usr/lib64/python2.5/site-packages/pyhdf/error.py", line 24, > in _checkErr > raise HDF4Error, str > pyhdf.error.HDF4Error: ('s', 'e', 't', 'n', 'a', 'm', 'e', ' ', ':', > ' ', 'c', 'a', 'n', 'n', 'o', 't', ' ', 'e', 'x', 'e', 'c', 'u', 't', > 'e') We have noticed the same thing on 64-bit Linux platforms. We are currently tracking this bug here: https://svn.enthought.com/epd/ticket/357 Please report pyhdf problems to the enthought-dev mailing list, rather than here (as I am probably the responsible party). https://mail.enthought.com/mailman/listinfo/enthought-dev Unfortunately, I don't know what's wrong at the moment. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Mon Sep 29 19:10:03 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 17:10:03 -0600 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> Message-ID: On Mon, Sep 29, 2008 at 4:54 PM, Charles R Harris wrote: > > > On Mon, Sep 29, 2008 at 4:40 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Mon, Sep 29, 2008 at 4:28 PM, Robert Kern wrote: >> >>> On Mon, Sep 29, 2008 at 17:13, Charles R Harris >>> wrote: >>> > >>> > On Mon, Sep 29, 2008 at 3:52 PM, Charles R Harris >>> > wrote: >>> >> >>> >> Hi All, >>> >> >>> >> I've been cleaning up the ufunc loops and the sign function currently >>> >> doesn't have a defined behavior for nans. This makes the results >>> depend on >>> >> the order/type of comparisons in the code, which looks fragile to me. >>> So >>> >> what should it return? I vote for nan but am open for suggestions. >>> > >>> > And while we're at it, lets decide how to treat max/min when nans are >>> > involved. Or should we just say the behavior is undefined. >>> >>> When feasible, I would like float(s)->float functions to return NaN >>> when given a NaN as an argument. At least as the main versions of the >>> function. Specific NaN-ignoring functions can also be introduced, but >>> as separate functions. I don't know what exactly to do about >>> float->int functions (e.g. argmin). I also don't know how these should >>> interact with the current seterr() state. >>> >> >> So the proposition is, sign, max, min return nan when any of the arguments >> is nan. >> >> +1 >> > > I also propose that all logical operators involving nan return false, i.e., > ==, !=, <, <=, >, >=, and, or, xor, not. > > Currently this is so except for !=. On my machine nan != nan is true. Looks like it is being computed in C as !(nan == nan). Hmm, anyone know of a C standard on this? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 29 19:16:03 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Sep 2008 18:16:03 -0500 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> Message-ID: <3d375d730809291616r7ad9a86et92d62da281628c3b@mail.gmail.com> On Mon, Sep 29, 2008 at 18:10, Charles R Harris wrote: > > On Mon, Sep 29, 2008 at 4:54 PM, Charles R Harris > wrote: >> >> On Mon, Sep 29, 2008 at 4:40 PM, Charles R Harris >> wrote: >>> >>> On Mon, Sep 29, 2008 at 4:28 PM, Robert Kern >>> wrote: >>>> >>>> On Mon, Sep 29, 2008 at 17:13, Charles R Harris >>>> wrote: >>>> > >>>> > On Mon, Sep 29, 2008 at 3:52 PM, Charles R Harris >>>> > wrote: >>>> >> >>>> >> Hi All, >>>> >> >>>> >> I've been cleaning up the ufunc loops and the sign function currently >>>> >> doesn't have a defined behavior for nans. This makes the results >>>> >> depend on >>>> >> the order/type of comparisons in the code, which looks fragile to me. >>>> >> So >>>> >> what should it return? I vote for nan but am open for suggestions. >>>> > >>>> > And while we're at it, lets decide how to treat max/min when nans are >>>> > involved. Or should we just say the behavior is undefined. >>>> >>>> When feasible, I would like float(s)->float functions to return NaN >>>> when given a NaN as an argument. At least as the main versions of the >>>> function. Specific NaN-ignoring functions can also be introduced, but >>>> as separate functions. I don't know what exactly to do about >>>> float->int functions (e.g. argmin). I also don't know how these should >>>> interact with the current seterr() state. >>> >>> So the proposition is, sign, max, min return nan when any of the >>> arguments is nan. >>> >>> +1 >> >> I also propose that all logical operators involving nan return false, >> i.e., ==, !=, <, <=, >, >=, and, or, xor, not. >> > > Currently this is so except for !=. On my machine nan != nan is true. Looks > like it is being computed in C as !(nan == nan). Hmm, anyone know of a C > standard on this? C99 Annex F: http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1124.pdf In particular: """ F.8.3 Relational operators x != x -> false The statement x != x is true if x is a NaN. """ -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Mon Sep 29 19:44:18 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 17:44:18 -0600 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: <3d375d730809291616r7ad9a86et92d62da281628c3b@mail.gmail.com> References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <3d375d730809291616r7ad9a86et92d62da281628c3b@mail.gmail.com> Message-ID: On Mon, Sep 29, 2008 at 5:16 PM, Robert Kern wrote: > On Mon, Sep 29, 2008 at 18:10, Charles R Harris > wrote: > > > > On Mon, Sep 29, 2008 at 4:54 PM, Charles R Harris > > wrote: > >> > >> On Mon, Sep 29, 2008 at 4:40 PM, Charles R Harris > >> wrote: > >>> > >>> On Mon, Sep 29, 2008 at 4:28 PM, Robert Kern > >>> wrote: > >>>> > >>>> On Mon, Sep 29, 2008 at 17:13, Charles R Harris > >>>> wrote: > >>>> > > >>>> > On Mon, Sep 29, 2008 at 3:52 PM, Charles R Harris > >>>> > wrote: > >>>> >> > >>>> >> Hi All, > >>>> >> > >>>> >> I've been cleaning up the ufunc loops and the sign function > currently > >>>> >> doesn't have a defined behavior for nans. This makes the results > >>>> >> depend on > >>>> >> the order/type of comparisons in the code, which looks fragile to > me. > >>>> >> So > >>>> >> what should it return? I vote for nan but am open for suggestions. > >>>> > > >>>> > And while we're at it, lets decide how to treat max/min when nans > are > >>>> > involved. Or should we just say the behavior is undefined. > >>>> > >>>> When feasible, I would like float(s)->float functions to return NaN > >>>> when given a NaN as an argument. At least as the main versions of the > >>>> function. Specific NaN-ignoring functions can also be introduced, but > >>>> as separate functions. I don't know what exactly to do about > >>>> float->int functions (e.g. argmin). I also don't know how these should > >>>> interact with the current seterr() state. > >>> > >>> So the proposition is, sign, max, min return nan when any of the > >>> arguments is nan. > >>> > >>> +1 > >> > >> I also propose that all logical operators involving nan return false, > >> i.e., ==, !=, <, <=, >, >=, and, or, xor, not. > >> > > > > Currently this is so except for !=. On my machine nan != nan is true. > Looks > > like it is being computed in C as !(nan == nan). Hmm, anyone know of a C > > standard on this? > > C99 Annex F: > > http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1124.pdf > > In particular: > > """ > F.8.3 Relational operators > x != x -> false The statement x != x is true if x is a NaN. > """ > Thanks, that was very helpful. I wonder how widespread the less, lessequal, etc. macros are? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From f.yw at hotmail.com Mon Sep 29 21:16:14 2008 From: f.yw at hotmail.com (frank wang) Date: Mon, 29 Sep 2008 19:16:14 -0600 Subject: [Numpy-discussion] Are there command similar as Matlab find command? In-Reply-To: <48E14D27.5040604@enthought.com> References: <200809222010.12025.pgmdevlist@gmail.com> <9911419a0809232336n89d6e4djcc59ad71f28732f5@mail.gmail.com> <9457e7c80809240810l6bf656aey70434d893200aa98@mail.gmail.com> <9457e7c80809250051y755e232bl8ca79847db186b2@mail.gmail.com> <48E14D27.5040604@enthought.com> Message-ID: Thanks for the help. It seems that the where command has problem when I tried to run it in the debug mode. It does not return any thing such as: (Pdb) aa=array([1,2,3,4] (Pdb) where(aa>2) (1)()> c:\dhg\docsis\lab_test\parseadc.py(70)parsempeg()-> bb=array(fid).astype('int') (Pdb) It does not return any result. Frank> Date: Mon, 29 Sep 2008 16:48:23 -0500> From: oliphant at enthought.com> To: numpy-discussion at scipy.org> Subject: Re: [Numpy-discussion] Are there command similar as Matlab find command?> > frank wang wrote:> > Hi,> > > > I am trying to find a command in numpy or python that perform similar > > function as Matlab find command. It will return the indexes of array > > that satisfy a condition. So far I have not found anything.> > There are several ways to do this, but what are you trying to do? > Non-zero on the boolean array resulting from the condition is the most > direct way:> > (a>30).nonzero()> where(a>30)> > This returns a tuple of indices of length nd, where nd is the number of > dimensions of a. (i.e. for 1-d case you need to extract the first > element of the tuple to get the indices you want).> > But, if you are going to use these indices to access elements of the > array, there are better ways to do that:> > a[a>30]> compress(a>30, a)> > etc.> > -Travis> > _______________________________________________> Numpy-discussion mailing list> Numpy-discussion at scipy.org> http://projects.scipy.org/mailman/listinfo/numpy-discussion _________________________________________________________________ Want to do more with Windows Live? Learn ?10 hidden secrets? from Jamie. http://windowslive.com/connect/post/jamiethomson.spaces.live.com-Blog-cns!550F681DAD532637!5295.entry?ocid=TXT_TAGLM_WL_domore_092008 -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 29 21:22:14 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Sep 2008 20:22:14 -0500 Subject: [Numpy-discussion] Are there command similar as Matlab find command? In-Reply-To: References: <200809222010.12025.pgmdevlist@gmail.com> <9911419a0809232336n89d6e4djcc59ad71f28732f5@mail.gmail.com> <9457e7c80809240810l6bf656aey70434d893200aa98@mail.gmail.com> <9457e7c80809250051y755e232bl8ca79847db186b2@mail.gmail.com> <48E14D27.5040604@enthought.com> Message-ID: <3d375d730809291822x2c4fb34dq19aab6e9466e2d64@mail.gmail.com> On Mon, Sep 29, 2008 at 20:16, frank wang wrote: > Thanks for the help. > > It seems that the where command has problem when I tried to run it in the > debug mode. It does not return any thing such as: > > (Pdb) aa=array([1,2,3,4] > (Pdb) where(aa>2) > (1)() >> c:\dhg\docsis\lab_test\parseadc.py(70)parsempeg() > -> bb=array(fid).astype('int') > (Pdb) > > It does not return any result. It's worth noting that that is not a "debug mode" of the interpreter; it's the pdb debugger, a separate piece of software which is used in an entirely different manner. Specifically, pdb has a "where" command which is what you are getting here. (Pdb) ?where w(here) Print a stack trace, with the most recent frame at the bottom. An arrow indicates the "current frame", which determines the context of most commands. 'bt' is an alias for this command. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Mon Sep 29 21:22:41 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 19:22:41 -0600 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> Message-ID: On Mon, Sep 29, 2008 at 4:28 PM, Robert Kern wrote: > On Mon, Sep 29, 2008 at 17:13, Charles R Harris > wrote: > > > > On Mon, Sep 29, 2008 at 3:52 PM, Charles R Harris > > wrote: > >> > >> Hi All, > >> > >> I've been cleaning up the ufunc loops and the sign function currently > >> doesn't have a defined behavior for nans. This makes the results depend > on > >> the order/type of comparisons in the code, which looks fragile to me. So > >> what should it return? I vote for nan but am open for suggestions. > > > > And while we're at it, lets decide how to treat max/min when nans are > > involved. Or should we just say the behavior is undefined. > > When feasible, I would like float(s)->float functions to return NaN > when given a NaN as an argument. At least as the main versions of the > function. Specific NaN-ignoring functions can also be introduced, but > as separate functions. I don't know what exactly to do about > float->int functions (e.g. argmin). I also don't know how these should > interact with the current seterr() state. > OK, maximum, minimum, and sign now return NaN. I still don't know what to do for the complex cases, although I suspect they should do the same on the principal that if either the real or imaginary parts are NaN then the number is undefined. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jqford at hotmail.com Mon Sep 29 22:31:47 2008 From: jqford at hotmail.com (Joshua Ford) Date: Mon, 29 Sep 2008 22:31:47 -0400 Subject: [Numpy-discussion] Numpy Nu-bee: "forward fill" function Message-ID: Hello all! I'm a python and numpy newbie and I'm having difficulty doing something that seems to be relatively straightforward. So I figured I would ask the experts! I would like to conduct the equivalent of a "forward fill" function on an array, that repeats the last non-null value until the next non-null value is reached. I saw that the scipy TimeSeries has a "forward_fill" function: http://pytseries.sourceforge.net/lib/interpolation.html I would like to do exactly what the TimeSeries "forward_fill" function does - but I don't have a time series. I simply want to convert an array [1,2,3,4,5,6,-999,-999,-999,-999,7,8] to [1,2,3,4,5,6,6,6,6,6,7,8] Here's what I've tried: from numpy import * a = array([1,2,3,4,5,6,-999,-999,-999,-999,7,8]) adata = a for i in adata: if i != -999: j = i elif i == -999: j = adata[i-1] print j else: print 'end' Of course .... this doesn't work. Many thanks in advance for any help or suggestions! As most of you can probably tell from my ugly code, I'm not a programmer... -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Mon Sep 29 23:02:21 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 30 Sep 2008 12:02:21 +0900 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> Message-ID: <48E196BD.6090905@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > So the proposition is, sign, max, min return nan when any of the > arguments is nan. Note that internally, signbit (the C function) returns an integer. > > Complex numbers are more complicated because we first compare the real > parts, then the imaginary. Arguably 1 > 0 + nan*1j. Really ? Without thinking about the consequences, returning a NaN complex would be what I expect, should we go the route comparison with NaN returns a NaN. > I propose that the sign of a complex number containing nans should be > nan, but I can't decide what should happen with max/min Did you take a look at: http://projects.scipy.org/scipy/numpy/wiki/ProperNanHandling Anne and me did this, with several approaches. We did not consider interactions with the FPU error state, though, which is something which needs to be added. cheers, David From david at ar.media.kyoto-u.ac.jp Mon Sep 29 23:07:03 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 30 Sep 2008 12:07:03 +0900 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <3d375d730809291616r7ad9a86et92d62da281628c3b@mail.gmail.com> Message-ID: <48E197D7.1050203@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > Thanks, that was very helpful. I wonder how widespread the less, > lessequal, etc. macros are? If it is C99, count on some platforms (MS in particular) to not do it. Also, when doing things in C, beware that some compilers break most reasonable expectations about floating points. In particular, x - x where x is a NaN or inf will returns 0 with MS compilers with the compiler flags currently used in python (/Ox), which breaks almost any code out there relying on proper NaN/Inf handling. So if possible, special case with isnan/isinf/isfinite cheers, David From pgmdevlist at gmail.com Mon Sep 29 23:24:02 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 29 Sep 2008 23:24:02 -0400 Subject: [Numpy-discussion] Numpy Nu-bee: "forward fill" function In-Reply-To: References: Message-ID: <200809292324.03883.pgmdevlist@gmail.com> On Monday 29 September 2008 22:31:47 Joshua Ford wrote: > I would like to conduct the equivalent of a "forward fill" function on an > array, that repeats the last non-null value until the next non-null value > is reached. I saw that the scipy TimeSeries has a "forward_fill" > function: > > http://pytseries.sourceforge.net/lib/interpolation.html > > I would like to do exactly what the TimeSeries "forward_fill" function does > - but I don't have a time series. Which is quite OK, because you don't need a time series, a simple MaskedArray will do. For your example >>> import numpy.ma as ma >>> import scikits.timeseries.lib as tl >>> a= ma.masked_values(([1,2,3,4,5,6,-999,-999,-999,-999,7,8], -999) >>> a masked_array(data = [1 2 3 4 5 6 -- -- -- -- 7 8], mask = [False False False False False False True True True True False False], fill_value=-999) >>> tl.forward_fill(tl) masked_array(data = [1 2 3 4 5 6 6 6 6 6 7 8], mask = [False False False False False False False False False False False False], fill_value=-999) From peridot.faceted at gmail.com Mon Sep 29 23:24:01 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Mon, 29 Sep 2008 23:24:01 -0400 Subject: [Numpy-discussion] Proposal: scipy.spatial Message-ID: Hi, Once again there has been a thread on the numpy/scipy mailing lists requesting (essentially) some form of spatial data structure. Pointers have been posted to ANN (sadly LGPLed and in C++) as well as a handful of pure-python implementations of kd-trees. I suggest the creation of a new submodule of scipy, scipy.spatial, to contain spatial data structures and algorithms. Specifically, I propose it contain a kd-tree implementation providing nearest-neighbor, approximate nearest-neighbor, and all-points-near queries. I have a few other suggestions for things it might contain, but kd-trees seem like a good first step. 2008/9/27 Nathan Bell : > On Sat, Sep 27, 2008 at 11:18 PM, Anne Archibald > wrote: >> >> I think a kd-tree implementation would be a valuable addition to >> scipy, perhaps in a submodule scipy.spatial that might eventually >> contain other spatial data structures and algorithms. What do you >> think? Should we have one? Should it be based on Sturla Molden's code, >> if the license permits? I am willing to contribute one, if not. > > +1 Judging that your vote and mine are enough in the absence of dissenting voices, I have written an implementation based on yours, Sturla Molden's, and the algorithms described by the authors of the ANN library. Before integrating it into scipy, though, I'd like to send it around for comments. Particular issues: * It's pure python right now, but with some effort it could be partially or completely cythonized. This is probably a good idea in the long run. In the meantime I have crudely vectorized it so that users can at least avoid looping in their own code. * It is able to return the r nearest neighbors, optionally subject to a maximum distance limit, as well as approximate nearest neighbors. * It is not currently set up to conveniently return all points within some fixed distance of the target point, but this could easily be added. * It returns distances and indices of nearest neighbors in the original array. * The testing code is, frankly, a mess. I need to look into using nose in a sensible fashion. * The license is the scipy license. I am particularly concerned about providing a convenient return format. The natural return from a query is a list of neighbors, since it may have variable length (there may not be r elements in the tree, or you might have supplied a maximum distance which doesn't contain r points). For a single query, it's simple to return a python list (should it be sorted? currently it's a heap). But if you want to vectorize the process, storing the results in an array becomes cumbersome. One option is an object array full of lists; another, which, I currently use, is an array with nonexistent points marked with an infinite distance and an invalid index. A third would be to return masked arrays. How do you recommend handling this variable (or potentially-variable) sized output? > If you're implementing one, I would highly recommend the > "left-balanced" kd-tree. > http://www2.imm.dtu.dk/pubdb/views/edoc_download.php/2535/pdf/imm2535.pdf Research suggests that at least in high dimension a more geometric balancing criterion can produce vastly better results. I used the "sliding midpoint" rule, which doesn't allow such a numerical balancing but does ensure that you don't have too many long skinny cells (since a sphere tends to cut very many of these). I've also been thinking about what else would go in scipy.spatial. I think it would be valuable to have a reasonably efficient distance matrix function (we seem to get that question a lot, and the answer's not trivial) as well as a sparse distance matrix function based on the kd-trees. The module could potentially also swallow the (currently sandboxed?) delaunay code. Anne -------------- next part -------------- A non-text attachment was scrubbed... Name: kdtree.py Type: text/x-python Size: 10591 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test_kdtree.py Type: text/x-python Size: 4369 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: plot_kdtree.py Type: text/x-python Size: 1403 bytes Desc: not available URL: From charlesr.harris at gmail.com Mon Sep 29 23:36:45 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 21:36:45 -0600 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: <48E197D7.1050203@ar.media.kyoto-u.ac.jp> References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <3d375d730809291616r7ad9a86et92d62da281628c3b@mail.gmail.com> <48E197D7.1050203@ar.media.kyoto-u.ac.jp> Message-ID: Hi David, On Mon, Sep 29, 2008 at 9:07 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > Thanks, that was very helpful. I wonder how widespread the less, > > lessequal, etc. macros are? > > If it is C99, count on some platforms (MS in particular) to not do it. > Also, when doing things in C, beware that some compilers break most > reasonable expectations about floating points. In particular, x - x > where x is a NaN or inf will returns 0 with MS compilers with the > compiler flags currently used in python (/Ox), which breaks almost any > code out there relying on proper NaN/Inf handling. So if possible, > special case with isnan/isinf/isfinite > Do any of the MS compilers handle these things correctly? I am loath to load up the code with expensive macros just to deal with bass ackwards compilers. Especially if I have to special case all the infs also. However, if I can use ifdefs to compile various versions I could be talked into it. OTOH, we could just say the results are undefined when non-conforming compilers are used. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Sep 30 00:02:54 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 22:02:54 -0600 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: <48E196BD.6090905@ar.media.kyoto-u.ac.jp> References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <48E196BD.6090905@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Sep 29, 2008 at 9:02 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > So the proposition is, sign, max, min return nan when any of the > > arguments is nan. > > Note that internally, signbit (the C function) returns an integer. > That is the signature of the ufunc. It could be changed... I believe the actual signbit of nan is undefined but I suppose we could return -1 in the nan case. That would be a fairly typical error signal for integers. > > > > > Complex numbers are more complicated because we first compare the real > > parts, then the imaginary. Arguably 1 > 0 + nan*1j. > > Really ? Without thinking about the consequences, returning a NaN > complex would be what I expect, should we go the route comparison with > NaN returns a NaN. > Yeah, I'm headed that way also. > > > I propose that the sign of a complex number containing nans should be > > nan, but I can't decide what should happen with max/min > > Did you take a look at: > > http://projects.scipy.org/scipy/numpy/wiki/ProperNanHandling > Note that in my branch the current behavior is In [11]: a = np.array([0, np.nan, -1]) In [12]: np.max(a) Out[12]: nan In [13]: np.min(a) Out[13]: nan This is consistent with regarding nans as propagating errors. I can merge my branch back into yours if you want to play with these things. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Mon Sep 29 23:53:00 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 30 Sep 2008 12:53:00 +0900 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <3d375d730809291616r7ad9a86et92d62da281628c3b@mail.gmail.com> <48E197D7.1050203@ar.media.kyoto-u.ac.jp> Message-ID: <48E1A29C.7020805@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > Do any of the MS compilers handle these things correctly? Don't know. To be 100 % honest, one of the problem for MS compilers is the /Ox flag (for IEEE-754 rules). This should not be used for numpy, period (I am sure you could break numpy with gcc and -ffast-math and co; the difference being gcc by default is compliant by default and documented properly). Also, MS compiler (even the recent ones) say they are not C99 compliant, because there is no customer need for it: http://blogs.msdn.com/vcblog/archive/2007/11/05/iso-c-standard-update.aspx cheers, David From david at ar.media.kyoto-u.ac.jp Mon Sep 29 23:54:44 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 30 Sep 2008 12:54:44 +0900 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <48E196BD.6090905@ar.media.kyoto-u.ac.jp> Message-ID: <48E1A304.6090705@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > > On Mon, Sep 29, 2008 at 9:02 PM, David Cournapeau > > > wrote: > > Charles R Harris wrote: > > > > So the proposition is, sign, max, min return nan when any of the > > arguments is nan. > > Note that internally, signbit (the C function) returns an integer. > > > That is the signature of the ufunc. It could be changed... Nope, I am talking about the C99 signbit macro. man signbit tells me: NAME signbit - test sign of a real floating point number SYNOPSIS #include int signbit(x); Compile with -std=c99; link with -lm. cheers, David From Chris.Barker at noaa.gov Tue Sep 30 00:10:10 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 29 Sep 2008 21:10:10 -0700 Subject: [Numpy-discussion] Proposal: scipy.spatial In-Reply-To: References: Message-ID: <48E1A6A2.4050406@noaa.gov> Anne Archibald wrote: > I suggest the creation of > a new submodule of scipy, scipy.spatial, +1 Here's one to consider: http://pypi.python.org/pypi/Rtree and perhaps other stuff from: http://trac.gispython.org/spatialindex/wiki which I think is LGPL -- can scipy use that? By the way, a general purpose bounding box class could be kind of handy -- I have one that is a start. It's a subclass of numpy arrays. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Tue Sep 30 00:11:09 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 22:11:09 -0600 Subject: [Numpy-discussion] Proposal: scipy.spatial In-Reply-To: References: Message-ID: On Mon, Sep 29, 2008 at 9:24 PM, Anne Archibald wrote: > Hi, > > Once again there has been a thread on the numpy/scipy mailing lists > requesting (essentially) some form of spatial data structure. Pointers > have been posted to ANN (sadly LGPLed and in C++) as well as a handful > of pure-python implementations of kd-trees. I suggest the creation of > a new submodule of scipy, scipy.spatial, to contain spatial data > structures and algorithms. Specifically, I propose it contain a > kd-tree implementation providing nearest-neighbor, approximate > nearest-neighbor, and all-points-near queries. I have a few other > suggestions for things it might contain, but kd-trees seem like a good > first step. > > 2008/9/27 Nathan Bell : > > On Sat, Sep 27, 2008 at 11:18 PM, Anne Archibald > > wrote: > >> > >> I think a kd-tree implementation would be a valuable addition to > >> scipy, perhaps in a submodule scipy.spatial that might eventually > >> contain other spatial data structures and algorithms. What do you > >> think? Should we have one? Should it be based on Sturla Molden's code, > >> if the license permits? I am willing to contribute one, if not. > > > > +1 > > Judging that your vote and mine are enough in the absence of > dissenting voices, I have written an implementation based on yours, > Sturla Molden's, and the algorithms described by the authors of the > ANN library. Before integrating it into scipy, though, I'd like to > send it around for comments. > > Particular issues: > > * It's pure python right now, but with some effort it could be > partially or completely cythonized. This is probably a good idea in > the long run. In the meantime I have crudely vectorized it so that > users can at least avoid looping in their own code. > * It is able to return the r nearest neighbors, optionally subject to > a maximum distance limit, as well as approximate nearest neighbors. > * It is not currently set up to conveniently return all points within > some fixed distance of the target point, but this could easily be > added. > * It returns distances and indices of nearest neighbors in the original > array. > * The testing code is, frankly, a mess. I need to look into using nose > in a sensible fashion. > * The license is the scipy license. > > I am particularly concerned about providing a convenient return > format. The natural return from a query is a list of neighbors, since > it may have variable length (there may not be r elements in the tree, > or you might have supplied a maximum distance which doesn't contain r > points). For a single query, it's simple to return a python list > (should it be sorted? currently it's a heap). But if you want to > vectorize the process, storing the results in an array becomes > cumbersome. One option is an object array full of lists; another, > which, I currently use, is an array with nonexistent points marked > with an infinite distance and an invalid index. A third would be to > return masked arrays. How do you recommend handling this variable (or > potentially-variable) sized output? > I think lists are the way to go. They should be pretty fast down at the C level. I use them myself for collecting the elements of equivalence classes and such that can be quite dynamic when new elements/relations are added. > > > If you're implementing one, I would highly recommend the > > "left-balanced" kd-tree. > > > http://www2.imm.dtu.dk/pubdb/views/edoc_download.php/2535/pdf/imm2535.pdf > > Research suggests that at least in high dimension a more geometric > balancing criterion can produce vastly better results. I used the > "sliding midpoint" rule, which doesn't allow such a numerical > balancing but does ensure that you don't have too many long skinny > cells (since a sphere tends to cut very many of these). > > I've also been thinking about what else would go in scipy.spatial. I > think it would be valuable to have a reasonably efficient distance > matrix function (we seem to get that question a lot, and the answer's > not trivial) as well as a sparse distance matrix function based on the > kd-trees. The module could potentially also swallow the (currently > sandboxed?) delaunay code. > It would be good to define the interface separately from the algorithm so that the latter can be tweaked without affecting the API. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Sep 30 00:14:33 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 22:14:33 -0600 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: <48E1A304.6090705@ar.media.kyoto-u.ac.jp> References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <48E196BD.6090905@ar.media.kyoto-u.ac.jp> <48E1A304.6090705@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Sep 29, 2008 at 9:54 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > > > On Mon, Sep 29, 2008 at 9:02 PM, David Cournapeau > > > > > wrote: > > > > Charles R Harris wrote: > > > > > > So the proposition is, sign, max, min return nan when any of the > > > arguments is nan. > > > > Note that internally, signbit (the C function) returns an integer. > > > > > > That is the signature of the ufunc. It could be changed... > > Nope, I am talking about the C99 signbit macro. man signbit tells me: > > NAME > signbit - test sign of a real floating point number > > SYNOPSIS > #include > > int signbit(x); > > Compile with -std=c99; link with -lm. > Yes, that too. But I was thinking of the ufunc returning nan when needed. However, I think -1 is the way to go for that to get minimal breakage. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Sep 30 00:18:13 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 22:18:13 -0600 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: <48E1A29C.7020805@ar.media.kyoto-u.ac.jp> References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <3d375d730809291616r7ad9a86et92d62da281628c3b@mail.gmail.com> <48E197D7.1050203@ar.media.kyoto-u.ac.jp> <48E1A29C.7020805@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Sep 29, 2008 at 9:53 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > Do any of the MS compilers handle these things correctly? > > Don't know. To be 100 % honest, one of the problem for MS compilers is > the /Ox flag (for IEEE-754 rules). This should not be used for numpy, > period (I am sure you could break numpy with gcc and -ffast-math and co; > the difference being gcc by default is compliant by default and > documented properly). Also, MS compiler (even the recent ones) say they > are not C99 compliant, because there is no customer need for it: > > http://blogs.msdn.com/vcblog/archive/2007/11/05/iso-c-standard-update.aspx > That's why they don't support long doubles either. I wonder how they plan on moving into the HPC area with that attitude? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Tue Sep 30 00:09:37 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 30 Sep 2008 13:09:37 +0900 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <48E196BD.6090905@ar.media.kyoto-u.ac.jp> <48E1A304.6090705@ar.media.kyoto-u.ac.jp> Message-ID: <48E1A681.5050709@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > Yes, that too. But I was thinking of the ufunc returning nan when needed I think this is better for consistency, yes. NaN is not a number, so it has no sign :) More seriously, I think those features should be clearly listed and thought out before being implemented, particularly wrt error handling and testing. I would rather see umathmodule.c cleaning first, wo any functionality change, and then, once integrated in the trunk, do the NaN changes in a branch, in particular because of broken implementations which will not do what you expect. Personally, I also have changes for NaN handling, but on my private git import of numpy. Also, if we fix this, we should fix this once for all, and with a lot of tests, if only to test on buggy implementations. cheers, David From f.yw at hotmail.com Tue Sep 30 00:30:22 2008 From: f.yw at hotmail.com (frank wang) Date: Mon, 29 Sep 2008 22:30:22 -0600 Subject: [Numpy-discussion] Are there command similar as Matlab find command? In-Reply-To: <3d375d730809291822x2c4fb34dq19aab6e9466e2d64@mail.gmail.com> References: <200809222010.12025.pgmdevlist@gmail.com> <9911419a0809232336n89d6e4djcc59ad71f28732f5@mail.gmail.com> <9457e7c80809240810l6bf656aey70434d893200aa98@mail.gmail.com> <9457e7c80809250051y755e232bl8ca79847db186b2@mail.gmail.com> <48E14D27.5040604@enthought.com> <3d375d730809291822x2c4fb34dq19aab6e9466e2d64@mail.gmail.com> Message-ID: Hi, Robert, Thank you very much for clarification of this. I really do not know the difference of debug mode and the pdb debugger. To me, it seems that pdb is only way to debug the python code. How do the expert of numpy/python debug their code? Are there any more efficient way to debug the code in python world? I used to use matlab which comes with a nice debugger. But now I really want to try the open source software and I found python/numpy is a nice tool. The only lagging piece of python/numpy comparing with matlab is the powerful debuggign capability. Onece again, I really appreciate the help I got from the forum. Frank> Date: Mon, 29 Sep 2008 20:22:14 -0500> From: robert.kern at gmail.com> To: numpy-discussion at scipy.org> Subject: Re: [Numpy-discussion] Are there command similar as Matlab find command?> > On Mon, Sep 29, 2008 at 20:16, frank wang wrote:> > Thanks for the help.> >> > It seems that the where command has problem when I tried to run it in the> > debug mode. It does not return any thing such as:> >> > (Pdb) aa=array([1,2,3,4]> > (Pdb) where(aa>2)> > (1)()> >> c:\dhg\docsis\lab_test\parseadc.py(70)parsempeg()> > -> bb=array(fid).astype('int')> > (Pdb)> >> > It does not return any result.> > It's worth noting that that is not a "debug mode" of the interpreter;> it's the pdb debugger, a separate piece of software which is used in> an entirely different manner. Specifically, pdb has a "where" command> which is what you are getting here.> > (Pdb) ?where> w(here)> Print a stack trace, with the most recent frame at the bottom.> An arrow indicates the "current frame", which determines the> context of most commands. 'bt' is an alias for this command.> > -- > Robert Kern> > "I have come to believe that the whole world is an enigma, a harmless> enigma that is made terrible by our own mad attempt to interpret it as> though it had an underlying truth."> -- Umberto Eco> _______________________________________________> Numpy-discussion mailing list> Numpy-discussion at scipy.org> http://projects.scipy.org/mailman/listinfo/numpy-discussion _________________________________________________________________ Stay up to date on your PC, the Web, and your mobile phone with Windows Live. http://clk.atdmt.com/MRT/go/msnnkwxp1020093185mrt/direct/01/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Tue Sep 30 00:28:45 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 30 Sep 2008 13:28:45 +0900 Subject: [Numpy-discussion] Are there command similar as Matlab find command? In-Reply-To: References: <200809222010.12025.pgmdevlist@gmail.com> <9911419a0809232336n89d6e4djcc59ad71f28732f5@mail.gmail.com> <9457e7c80809240810l6bf656aey70434d893200aa98@mail.gmail.com> <9457e7c80809250051y755e232bl8ca79847db186b2@mail.gmail.com> <48E14D27.5040604@enthought.com> <3d375d730809291822x2c4fb34dq19aab6e9466e2d64@mail.gmail.com> Message-ID: <48E1AAFD.5090101@ar.media.kyoto-u.ac.jp> frank wang wrote: > Hi, Robert, > > Thank you very much for clarification of this. > > I really do not know the difference of debug mode and the pdb debugger. That's totally different in principle. Debug mode means your code is compiled/interpreted differently to keep as much information as possible, even if it makes your code run slower. But it can still be run under the normal running environment (python interpreter in python's case). A debugger is a *different* running environment, often specialized for introspection, to do things not normally possible. Here, pdb. Generally, for a debugger to be useful, your code has to be interpreted/run in debug mode, though (but does not have to be in principle; just less information). Running under a debugger can also mean your code run significantly slower depending on what you do. If you are familiar with C, debug mode in python is like compiling with -g -DDEBUG with gcc, and debugger is running gdb. > To me, it seems that pdb is only way to debug the python code. How do > the expert of numpy/python debug their code? Are there any more > efficient way to debug the code in python world? I used to use matlab > which comes with a nice debugger. It depends on the kind of bugs. In python, the "print method" is much more powerful than in C because you can inspect many things (you can even go into the stack to know which function called any entry point, for example, although the debugger makes it a bit easier I guess). Personally, I rarely if ever use the python debugger. cheers, David From robert.kern at gmail.com Tue Sep 30 00:50:41 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Sep 2008 23:50:41 -0500 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <48E196BD.6090905@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730809292150v39476921oe23411fd609de465@mail.gmail.com> On Mon, Sep 29, 2008 at 23:02, Charles R Harris wrote: > > > On Mon, Sep 29, 2008 at 9:02 PM, David Cournapeau > wrote: >> >> Charles R Harris wrote: >> > >> > So the proposition is, sign, max, min return nan when any of the >> > arguments is nan. >> >> Note that internally, signbit (the C function) returns an integer. > > That is the signature of the ufunc. It could be changed... I believe the > actual signbit of nan is undefined but I suppose we could return -1 in the > nan case. That would be a fairly typical error signal for integers. numpy.signbit() should work like C99 signbit() (where possible), IMO. It can only return (integer) 0 or 1, and it does differentiate between NAN and -NAN. I don't think we should invent new semantics if we can avoid it. I think we can change what the platform provides, but only in the direction of C99, IMO. I see signbit() as more along the lines of functions like isnan() than log(). There is no C99 cognate for numpy.sign(), and it is a float->float function, so I think we could make it return NAN. C99's copysign(x,y) is almost a cognate (e.g. numpy.sign(y) == copysign(1.0,y) except for y==+/-0.0), but since it does fall down on y==0, I don't think it's determinative for y==NAN. [~]$ man copysign COPYSIGN(3) BSD Library Functions Manual COPYSIGN(3) NAME copysign -- changes the sign of x to that of y SYNOPSIS #include double copysign(double x, double y); ... [~]$ gcc --version i686-apple-darwin9-gcc-4.0.1 (GCC) 4.0.1 (Apple Inc. build 5465) Copyright (C) 2005 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. [~]$ cat foo.c #include #include int main(int argc, char **argv) { printf("signbit(NAN) = %d\n", signbit(NAN)); printf("signbit(-NAN) = %d\n", signbit(-NAN)); printf("copysign(1.0, NAN) = %g\n", copysign(1.0, NAN)); printf("copysign(1.0, -NAN) = %g\n", copysign(1.0, -NAN)); return 0; } [~]$ gcc -std=c99 -o foo foo.c -lm [~]$ ./foo signbit(NAN) = 0 signbit(-NAN) = 1 copysign(1.0, NAN) = 1 copysign(1.0, -NAN) = -1 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Tue Sep 30 00:52:34 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 22:52:34 -0600 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: <48E1A681.5050709@ar.media.kyoto-u.ac.jp> References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <48E196BD.6090905@ar.media.kyoto-u.ac.jp> <48E1A304.6090705@ar.media.kyoto-u.ac.jp> <48E1A681.5050709@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Sep 29, 2008 at 10:09 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > Yes, that too. But I was thinking of the ufunc returning nan when needed > > > > I think this is better for consistency, yes. NaN is not a number, so it > has no sign :) More seriously, I think those features should be clearly > listed and thought out before being implemented, particularly wrt error > handling and testing. I would rather see umathmodule.c cleaning first, umathmodule.c is cleaned/done. Or was until I put in some of the nan handling. But some of the nan comparisons were just odd, in particular the sign function, which was effectively undefined. And since the current behavior is an accident of comparison choices and order, I consider it undefined also. So at a minimum I would fix up sign/maximum/minimum. That would be consistent with both masking and error propagation in common usage. The curiosities arise when using the reduce method. Also returning -1 for the signbit seems like the right thing to do. If the nan comparisons can't be relied on, then using the isnan macro is an option. However, I am not inclined to go on and use isfinite and all those. > wo any functionality change, and then, once integrated in the trunk, do > the NaN changes in a branch, in particular because of broken > implementations which will not do what you expect. Personally, I also > have changes for NaN handling, but on my private git import of numpy. > I'm am currently working off your branch (at your suggestion). Is it safe to use NAN? > Also, if we fix this, we should fix this once for all, and with a lot of > tests, if only to test on buggy implementations. > Yes, the tests will define the standard. There is one test for sign of complex numbers that currently breaks, but I don't consider that important as anyone currently relying on sign when the array contains nans is out of their mind. An option for making the code more portable is to make the various comparison operators macros, then worry about fixing the macros for various compilers. I would be happy to use macros such as lessthan, which in addition to other things are not supposed to raise exceptions. Of course, if MS compilers don't even get the arithmetic right... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Tue Sep 30 00:49:18 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 30 Sep 2008 13:49:18 +0900 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <48E196BD.6090905@ar.media.kyoto-u.ac.jp> <48E1A304.6090705@ar.media.kyoto-u.ac.jp> <48E1A681.5050709@ar.media.kyoto-u.ac.jp> Message-ID: <48E1AFCE.4080304@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > umathmodule.c is cleaned/done. Or was until I put in some of the nan > handling. But some of the nan comparisons were just odd, in particular > the sign function, which was effectively undefined. And since the > current behavior is an accident of comparison choices and order, I > consider it undefined also. So at a minimum I would fix up > sign/maximum/minimum. Yes, I was not arguing about making the changes, but about making them together in the same branch. Because I bet you only build on linux, and if something is broken on one platform, it will be difficult to make the difference between pure code changes and feature changes. Whereas if you have two branches (the second being on top of the other one), then we can test the two differently. This makes job of testers much easier, I believe. At least, it makes my life easier when testing with ICC, MS compilers and co. > > I'm am currently working off your branch (at your suggestion). Is it > safe to use NAN? I've just answered to your private email about that point :) > > Yes, the tests will define the standard. There is one test for sign of > complex numbers that currently breaks, but I don't consider that > important as anyone currently relying on sign when the array contains > nans is out of their mind. An option for making the code more portable > is to make the various comparison operators macros, then worry about > fixing the macros for various compilers. Yes, that's by far the best method for compatibility. Implement a layer of compatibility, and then do as if you were on a common platform. That's the whole point of my clean_math_branch, BTW. This make tracking bugs much easier, cheers, David From david at ar.media.kyoto-u.ac.jp Tue Sep 30 00:55:38 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 30 Sep 2008 13:55:38 +0900 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: <3d375d730809292150v39476921oe23411fd609de465@mail.gmail.com> References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <48E196BD.6090905@ar.media.kyoto-u.ac.jp> <3d375d730809292150v39476921oe23411fd609de465@mail.gmail.com> Message-ID: <48E1B14A.3090406@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > > numpy.signbit() should work like C99 signbit() (where possible), IMO. > It can only return (integer) 0 or 1, and it does differentiate between > NAN and -NAN. I don't think we should invent new semantics if we can > avoid it. Agreed, but for signbit case, I can't find what the semantic should be for signbit and NANs; do you know any reference ? For example, going into your example: #include #include int main(int argc, char **argv) { printf("signbit(NAN) = %d\n", signbit(NAN)); printf("signbit(-NAN) = %d\n", signbit(-NAN)); printf("signbit((-1) * NAN) = %d\n", signbit((-1) * NAN)); printf("signbit(- NAN + NAN) = %d\n", signbit(-NAN + NAN)); printf("signbit(NAN - NAN) = %d\n", signbit(NAN - NAN)); return 0; } when run tells me: signbit(NAN) = 0 signbit(-NAN) = -2147483648 signbit((-1) * NAN) = 0 signbit(- NAN + NAN) = -2147483648 signbit(NAN - NAN) = 0 Does not this indicate that signbit(NAN) is undefined ? I guess I am afraid that the glibc NAN is just one type of NAN, and is not the behavior of any NAN, cheers, David From charlesr.harris at gmail.com Tue Sep 30 01:14:12 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 23:14:12 -0600 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: <3d375d730809292150v39476921oe23411fd609de465@mail.gmail.com> References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <48E196BD.6090905@ar.media.kyoto-u.ac.jp> <3d375d730809292150v39476921oe23411fd609de465@mail.gmail.com> Message-ID: On Mon, Sep 29, 2008 at 10:50 PM, Robert Kern wrote: > On Mon, Sep 29, 2008 at 23:02, Charles R Harris > wrote: > > > > > > On Mon, Sep 29, 2008 at 9:02 PM, David Cournapeau > > wrote: > >> > >> Charles R Harris wrote: > >> > > >> > So the proposition is, sign, max, min return nan when any of the > >> > arguments is nan. > >> > >> Note that internally, signbit (the C function) returns an integer. > > > > That is the signature of the ufunc. It could be changed... I believe the > > actual signbit of nan is undefined but I suppose we could return -1 in > the > > nan case. That would be a fairly typical error signal for integers. > > numpy.signbit() should work like C99 signbit() (where possible), IMO. > It can only return (integer) 0 or 1, and it does differentiate between > NAN and -NAN. I don't think we should invent new semantics if we can > avoid it. I think we can change what the platform provides, but only > in the direction of C99, IMO. I see signbit() as more along the lines > of functions like isnan() than log(). > Sounds reasonable. > > There is no C99 cognate for numpy.sign(), and it is a float->float > function, so I think we could make it return NAN. C99's copysign(x,y) > is almost a cognate (e.g. numpy.sign(y) == copysign(1.0,y) except for > y==+/-0.0), but since it does fall down on y==0, I don't think it's > determinative for y==NAN. > Sign doesn't distinguish +/-0 . The sign bit of 0 is explicitly cleared in the current (and former) code by adding +0 to the result. > > [~]$ man copysign > COPYSIGN(3) BSD Library Functions Manual > COPYSIGN(3) > > NAME > copysign -- changes the sign of x to that of y > > SYNOPSIS > #include > > double > copysign(double x, double y); > ... > [~]$ gcc --version > i686-apple-darwin9-gcc-4.0.1 (GCC) 4.0.1 (Apple Inc. build 5465) > Copyright (C) 2005 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. There is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > [~]$ cat foo.c > #include > #include > > int main(int argc, char **argv) > { > printf("signbit(NAN) = %d\n", signbit(NAN)); > printf("signbit(-NAN) = %d\n", signbit(-NAN)); > printf("copysign(1.0, NAN) = %g\n", copysign(1.0, NAN)); > printf("copysign(1.0, -NAN) = %g\n", copysign(1.0, -NAN)); > return 0; > } > [~]$ gcc -std=c99 -o foo foo.c -lm > [~]$ ./foo > signbit(NAN) = 0 > signbit(-NAN) = 1 > copysign(1.0, NAN) = 1 > copysign(1.0, -NAN) = -1 > Hmm, signbit(NAN) = 0 signbit(-NAN) = -2147483648 copysign(1.0, NAN) = 1 copysign(1.0, -NAN) = -1 signbit(0.0) = 0 signbit(-0.0) = -2147483648 copysign(1.0, 0.0) = 1 copysign(1.0, -0.0) = -1 Looking at the standard, signbit is only required to return a non-zero value for negatives. I think we need to be more explicit for numpy. How about 1? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Sep 30 01:20:28 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 30 Sep 2008 00:20:28 -0500 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> Message-ID: <3d375d730809292220q1567da2bi8547d166bd5494c@mail.gmail.com> On Mon, Sep 29, 2008 at 20:22, Charles R Harris wrote: > > On Mon, Sep 29, 2008 at 4:28 PM, Robert Kern wrote: >> >> On Mon, Sep 29, 2008 at 17:13, Charles R Harris >> wrote: >> > >> > On Mon, Sep 29, 2008 at 3:52 PM, Charles R Harris >> > wrote: >> >> >> >> Hi All, >> >> >> >> I've been cleaning up the ufunc loops and the sign function currently >> >> doesn't have a defined behavior for nans. This makes the results depend >> >> on >> >> the order/type of comparisons in the code, which looks fragile to me. >> >> So >> >> what should it return? I vote for nan but am open for suggestions. >> > >> > And while we're at it, lets decide how to treat max/min when nans are >> > involved. Or should we just say the behavior is undefined. >> >> When feasible, I would like float(s)->float functions to return NaN >> when given a NaN as an argument. At least as the main versions of the >> function. Specific NaN-ignoring functions can also be introduced, but >> as separate functions. I don't know what exactly to do about >> float->int functions (e.g. argmin). I also don't know how these should >> interact with the current seterr() state. > > OK, maximum, minimum, and sign now return NaN. Oops! F.9.9.2 The fmax functions 1 If just one argument is a NaN, the fmax functions return the other argument (if both arguments are NaNs, the functions return a NaN). 2 The body of the fmax function might be {return (isgreaterequal(x, y) || isnan(y)) ? x : y; } If we want to follow C99 semantics rather than our own NaN-always-propagates semantics, then we should do this instead. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From peridot.faceted at gmail.com Tue Sep 30 01:25:32 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Tue, 30 Sep 2008 01:25:32 -0400 Subject: [Numpy-discussion] Are there command similar as Matlab find command? In-Reply-To: References: <9911419a0809232336n89d6e4djcc59ad71f28732f5@mail.gmail.com> <9457e7c80809240810l6bf656aey70434d893200aa98@mail.gmail.com> <9457e7c80809250051y755e232bl8ca79847db186b2@mail.gmail.com> <48E14D27.5040604@enthought.com> <3d375d730809291822x2c4fb34dq19aab6e9466e2d64@mail.gmail.com> Message-ID: 2008/9/30 frank wang : > I really do not know the difference of debug mode and the pdb debugger. To > me, it seems that pdb is only way to debug the python code. How do the > expert of numpy/python debug their code? Are there any more efficient way to > debug the code in python world? I used to use matlab which comes with a nice > debugger. But now I really want to try the open source software and I found > python/numpy is a nice tool. The only lagging piece of python/numpy > comparing with matlab is the powerful debuggign capability. I think you will find that the debugger, pdb, is a good analog to matlab's debugging mode. The way I usually debug a piece of code looks something like: [in window A] edit foomodule.py to add a function frobnicate() [in window B, which has a running ipython session] >>> import foomodule >>> reload(foomodule) >>> frobnicate() >>> %pdb >>> frobnicate() (pdb) print some_variable 47 [in window A] edit foomodule.py to fix the bug [in window B] >>> reload(foomodule) >>> frobnicate() 21 >>> Alternatively, I write a test_foomodule.py, which contains a suite of unit tests. I run these tests and get dropped into a debugger when one fails, and I try to track it down. The main problem you ran into is that the debugger is not just any old python prompt. It has its own commands. It also, confusingly, will accept python commands, as long as they don't conflict with an internal command. I got into the habit of, when I wanted to run some python thing, writing (pdb) p 17+13 rather than just (pdb) 17+13 If you had simply written (pdb) p where(a>13) everything would have been fine. Anne From charlesr.harris at gmail.com Tue Sep 30 01:31:59 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Sep 2008 23:31:59 -0600 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: <3d375d730809292220q1567da2bi8547d166bd5494c@mail.gmail.com> References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <3d375d730809292220q1567da2bi8547d166bd5494c@mail.gmail.com> Message-ID: On Mon, Sep 29, 2008 at 11:20 PM, Robert Kern wrote: > On Mon, Sep 29, 2008 at 20:22, Charles R Harris > wrote: > > > > On Mon, Sep 29, 2008 at 4:28 PM, Robert Kern > wrote: > >> > >> On Mon, Sep 29, 2008 at 17:13, Charles R Harris > >> wrote: > >> > > >> > On Mon, Sep 29, 2008 at 3:52 PM, Charles R Harris > >> > wrote: > >> >> > >> >> Hi All, > >> >> > >> >> I've been cleaning up the ufunc loops and the sign function currently > >> >> doesn't have a defined behavior for nans. This makes the results > depend > >> >> on > >> >> the order/type of comparisons in the code, which looks fragile to me. > >> >> So > >> >> what should it return? I vote for nan but am open for suggestions. > >> > > >> > And while we're at it, lets decide how to treat max/min when nans are > >> > involved. Or should we just say the behavior is undefined. > >> > >> When feasible, I would like float(s)->float functions to return NaN > >> when given a NaN as an argument. At least as the main versions of the > >> function. Specific NaN-ignoring functions can also be introduced, but > >> as separate functions. I don't know what exactly to do about > >> float->int functions (e.g. argmin). I also don't know how these should > >> interact with the current seterr() state. > > > > OK, maximum, minimum, and sign now return NaN. > > Oops! > > F.9.9.2 The fmax functions > 1 If just one argument is a NaN, the fmax functions return the other > argument (if both arguments are NaNs, the functions return a NaN). > 2 The body of the fmax function might be > {return (isgreaterequal(x, y) || > isnan(y)) ? x : y; } > > If we want to follow C99 semantics rather than our own > NaN-always-propagates semantics, then we should do this instead. > That would have the virtue that amax/amin would return the actual max and min of the non nan numbers. Unless all of them were nans, in which case we would get the starting value, which should probabably be nan in this case ;) Hey, that would work nicely. So, I think I should use the comparison macros for the floats. However, if I read correctly, they don't raise exceptions. Would that be a problem? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wnbell at gmail.com Tue Sep 30 01:38:57 2008 From: wnbell at gmail.com (Nathan Bell) Date: Tue, 30 Sep 2008 01:38:57 -0400 Subject: [Numpy-discussion] nonuniform scatter operations In-Reply-To: <7f9d599f0809281315r5237966bq48cef623c5f8d521@mail.gmail.com> References: <7f9d599f0809272134h5e37d0ffkd98c6dafc2b3f9a1@mail.gmail.com> <7f9d599f0809281315r5237966bq48cef623c5f8d521@mail.gmail.com> Message-ID: On Sun, Sep 28, 2008 at 4:15 PM, Geoffrey Irving wrote: > > Thanks. That works great. A slightly cleaner version is > > X += coo_matrix((Y, (K, zeros_like(K)))).sum(axis=1) > > The next question is: is there a similar way that generalizes to the > case where X is n by 3 and Y is m by 3 (besides the obvious loop over > range(3), that is)? > You could flatten the arrays and make a single matrix that implemented the operation. I'd stick with the loop over range(3) though, it's more readable and likely to be as fast or faster than flattening the arrays yourself. -- Nathan Bell wnbell at gmail.com http://graphics.cs.uiuc.edu/~wnbell/ From wnbell at gmail.com Tue Sep 30 01:46:43 2008 From: wnbell at gmail.com (Nathan Bell) Date: Tue, 30 Sep 2008 01:46:43 -0400 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: <3d375d730809292220q1567da2bi8547d166bd5494c@mail.gmail.com> References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <3d375d730809292220q1567da2bi8547d166bd5494c@mail.gmail.com> Message-ID: On Tue, Sep 30, 2008 at 1:20 AM, Robert Kern wrote: > > F.9.9.2 The fmax functions > 1 If just one argument is a NaN, the fmax functions return the other > argument (if both arguments are NaNs, the functions return a NaN). > 2 The body of the fmax function might be > {return (isgreaterequal(x, y) || > isnan(y)) ? x : y; } > > If we want to follow C99 semantics rather than our own > NaN-always-propagates semantics, then we should do this instead. > +1 for NaN-always-propagates since we have explicit variants for the alternative semantics. Users are more likely to remember that "NaNs always propagate" than "as stated in the C99 standard...". -- Nathan Bell wnbell at gmail.com http://graphics.cs.uiuc.edu/~wnbell/ From david at ar.media.kyoto-u.ac.jp Tue Sep 30 01:18:42 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 30 Sep 2008 14:18:42 +0900 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: <48E1B14A.3090406@ar.media.kyoto-u.ac.jp> References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <48E196BD.6090905@ar.media.kyoto-u.ac.jp> <3d375d730809292150v39476921oe23411fd609de465@mail.gmail.com> <48E1B14A.3090406@ar.media.kyoto-u.ac.jp> Message-ID: <48E1B6B2.7060106@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > > when run tells me: > > signbit(NAN) = 0 > signbit(-NAN) = -2147483648 > signbit((-1) * NAN) = 0 > signbit(- NAN + NAN) = -2147483648 > signbit(NAN - NAN) = 0 > > Does not this indicate that signbit(NAN) is undefined ? I guess I am > afraid that the glibc NAN is just one type of NAN, and is not the > behavior of any NAN, Bah, this is gibberish. signbit test for the signbit (duh), nothing else, so this is always valid, and never raises a FPE_INVALID: glibc says that copysign is always valid, but I tested with signbit as well, and it does not seem to be raised either for that macro. http://www.gnu.org/software/libc/manual/html_node/FP-Bit-Twiddling.html#FP-Bit-Twiddling Funny how far a bug in median can take us :) cheers, David From david at ar.media.kyoto-u.ac.jp Tue Sep 30 01:56:29 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 30 Sep 2008 14:56:29 +0900 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <3d375d730809292220q1567da2bi8547d166bd5494c@mail.gmail.com> Message-ID: <48E1BF8D.10305@ar.media.kyoto-u.ac.jp> Nathan Bell wrote: > > +1 for NaN-always-propagates since we have explicit variants for the > alternative semantics. > > Users are more likely to remember that "NaNs always propagate" than > "as stated in the C99 standard...". I don't know. I would like to agree with you, but OTOH, starting to go against the C99 standard may bring us quite far. FWIW, matlab has the same behavior as mandated by C99 (but not R, by default). The problem I have with fmax is that: - isgreaterequal may be slow ? May well be red-herring. - I guess isgreaterequal is not available on windows with MS compilers - we can't detect NaN with FPE_INVALID (since by definition isgreaterequal never raises it), hence can't detect it with seterr. This is starting to get mind-blowing... cheers, David From robert.kern at gmail.com Tue Sep 30 02:24:08 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 30 Sep 2008 01:24:08 -0500 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <3d375d730809292220q1567da2bi8547d166bd5494c@mail.gmail.com> Message-ID: <3d375d730809292324p563d85afoe5162bb4e3344b8c@mail.gmail.com> On Tue, Sep 30, 2008 at 00:46, Nathan Bell wrote: > On Tue, Sep 30, 2008 at 1:20 AM, Robert Kern wrote: >> >> F.9.9.2 The fmax functions >> 1 If just one argument is a NaN, the fmax functions return the other >> argument (if both arguments are NaNs, the functions return a NaN). >> 2 The body of the fmax function might be >> {return (isgreaterequal(x, y) || >> isnan(y)) ? x : y; } >> >> If we want to follow C99 semantics rather than our own >> NaN-always-propagates semantics, then we should do this instead. >> > > +1 for NaN-always-propagates since we have explicit variants for the > alternative semantics. > > Users are more likely to remember that "NaNs always propagate" than > "as stated in the C99 standard...". OTOH, Python 2.6 and up will be following to the C99 standard as closely as possible. I would prefer to keep up with them. It's true that "as stated in the C99 standard" is more difficult to remember, but "NaNs always propagate" is probably not going to be consistent with everything we actually implement, no matter how hard we try. Regardless of what policy we use, the best thing is to have a consistent implementation on every platform and a comprehensive test suite to ensure that. Then, a user doesn't have to remember; they just try it out in the interpreter and see what it does. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Tue Sep 30 02:25:48 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 30 Sep 2008 00:25:48 -0600 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: <48E1BF8D.10305@ar.media.kyoto-u.ac.jp> References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <3d375d730809292220q1567da2bi8547d166bd5494c@mail.gmail.com> <48E1BF8D.10305@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Sep 29, 2008 at 11:56 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Nathan Bell wrote: > > > > +1 for NaN-always-propagates since we have explicit variants for the > > alternative semantics. > > > > Users are more likely to remember that "NaNs always propagate" than > > "as stated in the C99 standard...". > > I don't know. I would like to agree with you, but OTOH, starting to go > against the C99 standard may bring us quite far. FWIW, matlab has the > same behavior as mandated by C99 (but not R, by default). > > The problem I have with fmax is that: > - isgreaterequal may be slow ? May well be red-herring. For gcc, isgreaterequal should be the same as >= the way numpy is set up now, i.e., no errors raised. Looking at the assembly it all looked pretty clean with both numbers stored in the FPU. What I couldn't figure is why every compare needed to be done afresh, i.e. a > b and a == b had to run two compares, this on variables declared constant. I'm kind of inclined to follow the C standard as I figure there is already a lot of discussion behind it and it would be kind of a waste to hash it all out again on this list. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Sep 30 02:34:01 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 30 Sep 2008 00:34:01 -0600 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: <3d375d730809292324p563d85afoe5162bb4e3344b8c@mail.gmail.com> References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <3d375d730809292220q1567da2bi8547d166bd5494c@mail.gmail.com> <3d375d730809292324p563d85afoe5162bb4e3344b8c@mail.gmail.com> Message-ID: On Tue, Sep 30, 2008 at 12:24 AM, Robert Kern wrote: > On Tue, Sep 30, 2008 at 00:46, Nathan Bell wrote: > > On Tue, Sep 30, 2008 at 1:20 AM, Robert Kern > wrote: > >> > >> F.9.9.2 The fmax functions > >> 1 If just one argument is a NaN, the fmax functions return the other > >> argument (if both arguments are NaNs, the functions return a NaN). > >> 2 The body of the fmax function might be > >> {return (isgreaterequal(x, y) || > >> isnan(y)) ? x : y; } > >> > >> If we want to follow C99 semantics rather than our own > >> NaN-always-propagates semantics, then we should do this instead. > >> > > > > +1 for NaN-always-propagates since we have explicit variants for the > > alternative semantics. > > > > Users are more likely to remember that "NaNs always propagate" than > > "as stated in the C99 standard...". > > OTOH, Python 2.6 and up will be following to the C99 standard as > closely as possible. I would prefer to keep up with them. It's true > that "as stated in the C99 standard" is more difficult to remember, > but "NaNs always propagate" is probably not going to be consistent > with everything we actually implement, no matter how hard we try. > I wonder how much of the Python stuff we can steal^W borrow. I assume the Python license is compatible with numpy? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Sep 30 02:36:07 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 30 Sep 2008 01:36:07 -0500 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <3d375d730809292220q1567da2bi8547d166bd5494c@mail.gmail.com> <3d375d730809292324p563d85afoe5162bb4e3344b8c@mail.gmail.com> Message-ID: <3d375d730809292336gb30a456nfb10a73f84db333d@mail.gmail.com> On Tue, Sep 30, 2008 at 01:34, Charles R Harris wrote: > > On Tue, Sep 30, 2008 at 12:24 AM, Robert Kern wrote: >> OTOH, Python 2.6 and up will be following to the C99 standard as >> closely as possible. I would prefer to keep up with them. It's true >> that "as stated in the C99 standard" is more difficult to remember, >> but "NaNs always propagate" is probably not going to be consistent >> with everything we actually implement, no matter how hard we try. > > I wonder how much of the Python stuff we can steal^W borrow. I assume the > Python license is compatible with numpy? Yeah. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From numpy-discussion at maubp.freeserve.co.uk Tue Sep 30 04:29:36 2008 From: numpy-discussion at maubp.freeserve.co.uk (Peter) Date: Tue, 30 Sep 2008 09:29:36 +0100 Subject: [Numpy-discussion] Proposal: scipy.spatial In-Reply-To: <48E1A6A2.4050406@noaa.gov> References: <48E1A6A2.4050406@noaa.gov> Message-ID: <320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com> On Tue, Sep 30, 2008 at 5:10 AM, Christopher Barker wrote: > > Anne Archibald wrote: >> I suggest the creation of >> a new submodule of scipy, scipy.spatial, > > +1 > > Here's one to consider: > http://pypi.python.org/pypi/Rtree > and perhaps other stuff from: > http://trac.gispython.org/spatialindex/wiki > which I think is LGPL -- can scipy use that? There is also a KDTree module in Biopython (which is under a BSD/MIT style licence), http://biopython.org/SRC/biopython/Bio/KDTree/ The current version is in C, there is an older version available in the CVS history in C++ too, http://cvs.biopython.org/cgi-bin/viewcvs/viewcvs.cgi/biopython/Bio/KDTree/?cvsroot=biopython Peter From bevan07 at gmail.com Tue Sep 30 05:06:57 2008 From: bevan07 at gmail.com (bevan) Date: Tue, 30 Sep 2008 09:06:57 +0000 (UTC) Subject: [Numpy-discussion] Upper and lower envelopes Message-ID: Hello, I have some XY data. I would like to generate the equations for an upper and lower envelope that excludes a percentage of the data points. I would like to define the slope of the envelope line (say 3) and then have my code find the intercept that fits my requirements (say 5% of data below the lower envelope). This would then give me the equation and I could plot the upper and lower envelopes. I hope this makes sense. Thanks for any help. From faltet at pytables.org Tue Sep 30 05:27:29 2008 From: faltet at pytables.org (Francesc Alted) Date: Tue, 30 Sep 2008 11:27:29 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.2.0 In-Reply-To: References: <200809292043.40547.faltet@pytables.org> Message-ID: <200809301127.29752.faltet@pytables.org> A Monday 29 September 2008, Anne Archibald escrigu?: > 2008/9/29 Francesc Alted : > > But I agree with Robert Kern that tests *should* never be run under > > -OO because they are *tests*, not production code, and using the > > -OO for running the test units is simply a wrong practice. > > I have to disagree with this. What if some piece of code misbehaves > when optimization is turned on? This frequently happens with C code, > and I would not be too surprised if it occurred with python code. The > easiest way, which I think is what Chuck was referring to as "abuse > of asserts" is for someone to write something like > > try: > while True: > do_something() > assert np.all(a<17) > except AssertionError: > pass > > This is of course a bug, though it may work, more or less, in > non-optimized code. But if you can't run your test suite in optimized > mode, it's going to be useless finding this bug. It is a bug, and it reveals a poor knowledge of the assert statement indeed. But the sad truth is that this sort of things *can* be done. > Perhaps more plausibly, someone might use assert to check the > validity of user input. Then you turn on optimization and use input > becomes unchecked... Using assert for validating user input is a programming error. The official library reference [1]_ is clear about the usage of the assert statement: """ Assert statements are a convenient way to insert *debugging* assertions into a program. """ (emphasis is mine) .. [1] http://docs.python.org/ref/assert.html Unfortunately, Python will not warn about the problem (even pylint neither issues a warning on that!). > I support getting rid of assert statements in tests. How hard is it > to create a function assert_ in numpy.Testing? > > assert np.all(a<17), "oops" > > becomes > > assert_(np.all(a<17), "oops") > > It could even be done automatically, or nearly. This is already provided by the unitest module (and most surely by nose too), through its self.assert_() function: self.assert_(np.all(a<17), "oops") so, changing this is probably quite easy to be done (although it might admittedly require quite a few keystrokes to be issued). Cheers, -- Francesc Alted From pete.forman at westerngeco.com Tue Sep 30 07:10:10 2008 From: pete.forman at westerngeco.com (Pete Forman) Date: Tue, 30 Sep 2008 12:10:10 +0100 Subject: [Numpy-discussion] What is the sign of nan? References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <3d375d730809292220q1567da2bi8547d166bd5494c@mail.gmail.com> <3d375d730809292324p563d85afoe5162bb4e3344b8c@mail.gmail.com> <3d375d730809292336gb30a456nfb10a73f84db333d@mail.gmail.com> Message-ID: I've seen no mention in this thread of IEEE Std 754-2008 which was published last month. minNum(x, y) and maxNum(x, y) return a floating-point number if the other argument is NaN. The OP's question is about the sign of NaN. In 754r it can be tested, copied, etc. Operations involving NaNs propagate the payload but not the sign bit AFAIK. -- Pete Forman -./\.- Disclaimer: This post is originated WesternGeco -./\.- by myself and does not represent pete.forman at westerngeco.com -./\.- the opinion of Schlumberger or http://petef.22web.net -./\.- WesternGeco. From charlesr.harris at gmail.com Tue Sep 30 08:17:10 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 30 Sep 2008 06:17:10 -0600 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <3d375d730809292220q1567da2bi8547d166bd5494c@mail.gmail.com> <3d375d730809292324p563d85afoe5162bb4e3344b8c@mail.gmail.com> <3d375d730809292336gb30a456nfb10a73f84db333d@mail.gmail.com> Message-ID: On Tue, Sep 30, 2008 at 5:10 AM, Pete Forman wrote: > I've seen no mention in this thread of IEEE Std 754-2008 which was > published last month. minNum(x, y) and maxNum(x, y) return a > floating-point number if the other argument is NaN. > > The OP's question is about the sign of NaN. In 754r it can be tested, > copied, etc. Operations involving NaNs propagate the payload but not > the sign bit AFAIK. OK, here is what is looks like to me at the moment given that numpy requires an IEEE754 machine: - We need a reliable value for NAN. This is perhaps best done by using a union and explicitly twiddling the bits depending on the endian choice of the architecture. For architectures with foobar extended precision we don't worry about supporting the extended precision. The result can be tested, perhaps when numpy is loaded. We could possibly get the value from python. What happens to PPC? - Max/min follow the IEEE standard. Given a choice of nan/non-nan, return non-nan. This can be extended to complex numbers where the choice is based on the real parts unless they are equal or both nans, in which case the decision is made on the imaginary parts. - Signbit returns the value of the signbit function, but nonzero values are set to 1. - I am unsure of sign. Should it return signed zeros? Should it return nan for nan or return the sign of the nan? I am inclined towards returning nan. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jqford at hotmail.com Tue Sep 30 08:20:08 2008 From: jqford at hotmail.com (Joshua Ford) Date: Tue, 30 Sep 2008 08:20:08 -0400 Subject: [Numpy-discussion] Numpy Nu-bee: "forward fill" function In-Reply-To: <200809292324.03883.pgmdevlist@gmail.com> References: <200809292324.03883.pgmdevlist@gmail.com> Message-ID: Thank you so very much! > From: pgmdevlist at gmail.com > To: numpy-discussion at scipy.org > Date: Mon, 29 Sep 2008 23:24:02 -0400 > Subject: Re: [Numpy-discussion] Numpy Nu-bee: "forward fill" function > > On Monday 29 September 2008 22:31:47 Joshua Ford wrote: > > I would like to conduct the equivalent of a "forward fill" function on an > > array, that repeats the last non-null value until the next non-null value > > is reached. I saw that the scipy TimeSeries has a "forward_fill" > > function: > > > > http://pytseries.sourceforge.net/lib/interpolation.html > > > > I would like to do exactly what the TimeSeries "forward_fill" function does > > - but I don't have a time series. > > Which is quite OK, because you don't need a time series, a simple MaskedArray > will do. For your example > > >>> import numpy.ma as ma > >>> import scikits.timeseries.lib as tl > >>> a= ma.masked_values(([1,2,3,4,5,6,-999,-999,-999,-999,7,8], -999) > >>> a > masked_array(data = [1 2 3 4 5 6 -- -- -- -- 7 8], > mask = [False False False False False False True True True True > False False], > fill_value=-999) > >>> tl.forward_fill(tl) > masked_array(data = [1 2 3 4 5 6 6 6 6 6 7 8], > mask = [False False False False False False False False False False > False False], > fill_value=-999) > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.huard at gmail.com Tue Sep 30 09:01:41 2008 From: david.huard at gmail.com (David Huard) Date: Tue, 30 Sep 2008 09:01:41 -0400 Subject: [Numpy-discussion] Upper and lower envelopes In-Reply-To: References: Message-ID: <91cf711d0809300601g555e0e64k58f5d82ff5775015@mail.gmail.com> Bevan, You can estimate the intercept and slope using least-squares (scipy.optimize.leastsq). Make sure though that errors in X are small compared to errors in Y, otherwise, your slope will be underestimated. Using the slope, you can write a function lower(b,a, X,Y) that will compute y=aX+b and return True if Y < y. Computing the ratio of true elements will give you the percentage of points below the curve. You can then find b such that the ratio is .5 and .95 using scipy.optimize.fmin. There are other ways to do this; Make a 2D histogram of the data (normed), compute the cumulative sum along Y and find the histogram bins (along x) such that the cumulative histogram is approximately equal to .5 and .95. Partition the data in N sets along the x-axis, fit a normal distribution to each set and compute the quantile corresponding to .5 and .95 cumulative probability density. David By the way, anonymous mails from newcomers don't get as much attention as those that are signed. Call it mailing list etiquette. On Tue, Sep 30, 2008 at 5:06 AM, bevan wrote: > Hello, > > I have some XY data. I would like to generate the equations for an upper > and > lower envelope that excludes a percentage of the data points. > > I would like to define the slope of the envelope line (say 3) and then have > my > code find the intercept that fits my requirements (say 5% of data below the > lower envelope). This would then give me the equation and I could plot the > upper and lower envelopes. > > > I hope this makes sense. Thanks for any help. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pete.forman at westerngeco.com Tue Sep 30 09:42:18 2008 From: pete.forman at westerngeco.com (Pete Forman) Date: Tue, 30 Sep 2008 14:42:18 +0100 Subject: [Numpy-discussion] What is the sign of nan? References: <3d375d730809291528s59c6705am690a17d5a543d682@mail.gmail.com> <3d375d730809292220q1567da2bi8547d166bd5494c@mail.gmail.com> <3d375d730809292324p563d85afoe5162bb4e3344b8c@mail.gmail.com> <3d375d730809292336gb30a456nfb10a73f84db333d@mail.gmail.com> Message-ID: "Charles R Harris" writes: > OK, here is what is looks like to me at the moment given that numpy > requires an IEEE754 machine: > > o We need a reliable value for NAN. [...] > > > o Max/min follow the IEEE standard. Given a choice of > nan/non-nan, return non-nan. [...] Yes, that follows 754r and C99. > o Signbit returns the value of the signbit function, but nonzero > values are set to 1. Looks okay to me. > o I am unsure of sign. Should it return signed zeros? Should it > return nan for nan or return the sign of the nan? I am > inclined towards returning nan. How is sign used? If it is in x * sign(y) then it might be better to use copysign(x, y) which is well defined even with signed zeros and NaNs. It depends on whether you want special behavior when y is zero. In copysign y being 0 or +0 is considered positive, so x is returned. So you could use this as a specification. def sign(y): if y == 0: # True for -0 and +0 too return 0 # or perhaps return y else return copysign(1, y) Your inclination leads to this. def sign(y): if y == 0 or isnan(y): return y else return copysign(1, y) The better choice will be governed by how sign is used in practice. -- Pete Forman -./\.- Disclaimer: This post is originated WesternGeco -./\.- by myself and does not represent pete.forman at westerngeco.com -./\.- the opinion of Schlumberger or http://petef.22web.net -./\.- WesternGeco. From bblais at bryant.edu Tue Sep 30 13:53:05 2008 From: bblais at bryant.edu (Brian Blais) Date: Tue, 30 Sep 2008 13:53:05 -0400 Subject: [Numpy-discussion] xml-rpc with numpy arrays Message-ID: <98DE1D44-A0B6-4CC6-8FD0-11F187D20862@bryant.edu> Hello, I am trying to use xml-rpc to be able to run some simulations remotely. I am running into a problem with the transfer of numpy arrays. my server code looks like: #!/usr/bin/env python def again(x): # test out the sending of data return [x,x] from SimpleXMLRPCServer import SimpleXMLRPCServer SimpleXMLRPCServer.allow_reuse_address = 1 server = SimpleXMLRPCServer(("", 8000)) server.register_function(again) try: print "Serving..." server.serve_forever() # Start the server finally: print "done." server.server_close() my client code looks like: import numpy from xmlrpclib import ServerProxy server=ServerProxy('http://localhost:8000') server.again(5) # this works b=numpy.random.rand(5,5) server.again(b) # this gives an error this gives the error: : cannot marshal objects which seems to be a deficiency of the marshal library, or perhaps I am doing something wrong. Is there a way to fix this? Is there another approach that I should be using? thanks, Brian Blais -- Brian Blais bblais at bryant.edu http://web.bryant.edu/~bblais -------------- next part -------------- An HTML attachment was scrubbed... URL: From rocksportrocker at googlemail.com Tue Sep 30 14:09:33 2008 From: rocksportrocker at googlemail.com (Uwe Schmitt) Date: Tue, 30 Sep 2008 11:09:33 -0700 (PDT) Subject: [Numpy-discussion] xml-rpc with numpy arrays In-Reply-To: <98DE1D44-A0B6-4CC6-8FD0-11F187D20862@bryant.edu> References: <98DE1D44-A0B6-4CC6-8FD0-11F187D20862@bryant.edu> Message-ID: <6a360b7d-bde5-4e51-8df5-b40a5e64a442@u28g2000hsc.googlegroups.com> Hi, in order to marshal numpy arrays, you can use the tostring() method. The inverse is the fromstring() function in numpy. But you must know dtype and shape in order to reconstruct your array. Greetings, Uwe On 30 Sep., 19:53, Brian Blais wrote: > Hello, > > I am trying to use xml-rpc to be able to run some simulations ? > remotely. ?I am running into a problem with the transfer of numpy ? > arrays. ?my server code looks like: > > #!/usr/bin/env python > def again(x): ?# test out the sending of data > ? ? ?return [x,x] > > from SimpleXMLRPCServer import SimpleXMLRPCServer > SimpleXMLRPCServer.allow_reuse_address = 1 > > server = SimpleXMLRPCServer(("", 8000)) > > server.register_function(again) > try: > ? ? ?print "Serving..." > ? ? ?server.serve_forever() # Start the server > finally: > ? ? ?print "done." > ? ? ?server.server_close() > > my client code looks like: > > import numpy > from xmlrpclib import ServerProxy > server=ServerProxy('http://localhost:8000') > server.again(5) ?# this works > > b=numpy.random.rand(5,5) > > server.again(b) ?# this gives an error > > this gives the error: : cannot marshal ? > objects > > which seems to be a deficiency of the marshal library, or perhaps I ? > am doing something wrong. ?Is there a way to fix this? ?Is there ? > another approach that I should be using? > > ? ? ? ? ? ? ? ? thanks, > > ? ? ? ? ? ? ? ? ? ? ? ? Brian Blais > > -- > Brian Blais > bbl... at bryant.eduhttp://web.bryant.edu/~bblais > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Tue Sep 30 15:04:41 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 30 Sep 2008 13:04:41 -0600 Subject: [Numpy-discussion] What is the sign of nan? In-Reply-To: References: <3d375d730809292220q1567da2bi8547d166bd5494c@mail.gmail.com> <3d375d730809292324p563d85afoe5162bb4e3344b8c@mail.gmail.com> <3d375d730809292336gb30a456nfb10a73f84db333d@mail.gmail.com> Message-ID: On Tue, Sep 30, 2008 at 7:42 AM, Pete Forman wrote: > "Charles R Harris" writes: > > > OK, here is what is looks like to me at the moment given that numpy > > requires an IEEE754 machine: > > > > o We need a reliable value for NAN. [...] > > > > > > o Max/min follow the IEEE standard. Given a choice of > > nan/non-nan, return non-nan. [...] > > Yes, that follows 754r and C99. > > > o Signbit returns the value of the signbit function, but nonzero > > values are set to 1. > > Looks okay to me. > > > o I am unsure of sign. Should it return signed zeros? Should it > > return nan for nan or return the sign of the nan? I am > > inclined towards returning nan. > > How is sign used? If it is in x * sign(y) then it might be better to > use copysign(x, y) which is well defined even with signed zeros and > NaNs. It depends on whether you want special behavior when y is zero. > In copysign y being 0 or +0 is considered positive, so x is returned. > > So you could use this as a specification. > > def sign(y): > if y == 0: # True for -0 and +0 too > return 0 # or perhaps return y > else > return copysign(1, y) > > Your inclination leads to this. > > def sign(y): > if y == 0 or isnan(y): > return y > else > return copysign(1, y) > I'm leaning towards the first at the moment. I would prefer the signed zero also, but that might actually break some code so probably the safe near term choice is the unsigned zero. For max/min I am going to introduce new ufuncs, fmax/fmin, which return numbers unless both arguements are nan. The current maximum/minimum functions will return nan if either arguement is a nan. How these might integrated into the max/min ndarray methods can be left to the future. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Sep 30 15:05:27 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 30 Sep 2008 14:05:27 -0500 Subject: [Numpy-discussion] xml-rpc with numpy arrays In-Reply-To: <98DE1D44-A0B6-4CC6-8FD0-11F187D20862@bryant.edu> References: <98DE1D44-A0B6-4CC6-8FD0-11F187D20862@bryant.edu> Message-ID: <3d375d730809301205l7c138d78xc4b687b7403c4df7@mail.gmail.com> On Tue, Sep 30, 2008 at 12:53, Brian Blais wrote: > Hello, > I am trying to use xml-rpc to be able to run some simulations remotely. I > am running into a problem with the transfer of numpy arrays. my server code > looks like: > #!/usr/bin/env python > def again(x): # test out the sending of data > return [x,x] > > from SimpleXMLRPCServer import SimpleXMLRPCServer > SimpleXMLRPCServer.allow_reuse_address = 1 > server = SimpleXMLRPCServer(("", 8000)) > server.register_function(again) > try: > print "Serving..." > server.serve_forever() # Start the server > finally: > print "done." > server.server_close() > > > my client code looks like: > import numpy > from xmlrpclib import ServerProxy > server=ServerProxy('http://localhost:8000') > server.again(5) # this works > b=numpy.random.rand(5,5) > server.again(b) # this gives an error > this gives the error: : cannot marshal 'numpy.ndarray'> objects > which seems to be a deficiency of the marshal library, or perhaps I am doing > something wrong. Is there a way to fix this? Is there another approach > that I should be using? The marshal module *only* handles builtin Python types. It explicitly does not handle anything from third parties like numpy. As Uwe suggests, you can use tostring()/fromstring() yourself and also pass along the dtype and shape information. With numpy 1.2.0, there is a somewhat better alternative. I have defined a small file format that should represent all numpy arrays with dtype and shape information embedded in its header. That way, you just have to pass the string. from cStringIO import StringIO from numpy.lib import format def from_string(s): f = StringIO(s) arr = format.read_array(f) return arr def to_string(arr): f = StringIO() format.write_array(f, arr) s = f.getvalue() return s -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From lists_ravi at lavabit.com Tue Sep 30 15:48:09 2008 From: lists_ravi at lavabit.com (Ravi) Date: Tue, 30 Sep 2008 15:48:09 -0400 Subject: [Numpy-discussion] PyArray_Resize reference counting Message-ID: <200809301548.10782.lists_ravi@lavabit.com> Hi, Consider the following code: PyArrayObject *array = get_me_an_array_along_with_a_reference(); /* 1 */ PyArray_Dims *dims = get_me_some_dims(); /* 2 */ array = (PyArrayObject *)PyArray_Resize( array, dims, 1 ); /* 3 */ Is the code above valid? In particular, is the assigment on line 3 valid? Travis' book says that a reference to the new array is returned. Does that mean that I need to store a pointer to the original array somewhere else and decrement its reference count after the resize call? Alternatively, if I don't need to store the pointer because the returned pointer is the same as the one passed into the resize function, do I need to decrement the reference count since I already have a reference to the array? In other words, if the reference count is, say, 16 before line 1 and 17 after line 1, will it be 18 after line 3? In that case, I need to decrement the reference once to get it back to 17 because I should not need a reference as I already have one. Regards, Ravi From hep.sebastien.binet at gmail.com Tue Sep 30 16:18:36 2008 From: hep.sebastien.binet at gmail.com (Sebastien Binet) Date: Tue, 30 Sep 2008 13:18:36 -0700 Subject: [Numpy-discussion] xml-rpc with numpy arrays In-Reply-To: <3d375d730809301205l7c138d78xc4b687b7403c4df7@mail.gmail.com> References: <98DE1D44-A0B6-4CC6-8FD0-11F187D20862@bryant.edu> <3d375d730809301205l7c138d78xc4b687b7403c4df7@mail.gmail.com> Message-ID: <200809301318.36701.binet@cern.ch> Hi, [..snip..] On Tuesday 30 September 2008 12:05:27 Robert Kern wrote: > > doing something wrong. Is there a way to fix this? Is there another > > approach that I should be using? > > The marshal module *only* handles builtin Python types. It explicitly > does not handle anything from third parties like numpy. but you can use the copy_reg module ('copyreg' in py2.6) to register pickling/unpickling functions to make 'pickle' able to handle user-types: ## test_pickling_numpy.py ## import numpy import copy_reg import cPickle as pickle def from_string(s): return numpy.fromstring(s) def to_string(o): return from_string, (o.tostring(),) a = numpy.arange (10,dtype=float) copy_reg.pickle (numpy.ndarray, to_string, from_string) s = pickle.dumps (a) b = pickle.loads (s) assert (a==b, "pickling failed !") ## EOF ## see for more informations: http://effbot.org/librarybook/copy-reg.htm cheers, sebastien. -- ################################### # Dr. Sebastien Binet # Lawrence Berkeley National Lab. # 1 Cyclotron Road # Berkeley, CA 94720 ################################### From oliphant at enthought.com Tue Sep 30 16:26:08 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 30 Sep 2008 15:26:08 -0500 Subject: [Numpy-discussion] PyArray_Resize reference counting In-Reply-To: <200809301548.10782.lists_ravi@lavabit.com> References: <200809301548.10782.lists_ravi@lavabit.com> Message-ID: <48E28B60.2030403@enthought.com> Ravi wrote: > Hi, > Consider the following code: > > PyArrayObject *array = get_me_an_array_along_with_a_reference(); /* 1 */ > PyArray_Dims *dims = get_me_some_dims(); /* 2 */ > array = (PyArrayObject *)PyArray_Resize( array, dims, 1 ); /* 3 */ > > Is the code above valid? No. You need to do something like: temp = PyArray_Resize(array,...) Py_DECREF(array) array = temp -Travis From peridot.faceted at gmail.com Tue Sep 30 16:37:04 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Tue, 30 Sep 2008 16:37:04 -0400 Subject: [Numpy-discussion] Upper and lower envelopes In-Reply-To: References: Message-ID: 2008/9/30 bevan : > Hello, > > I have some XY data. I would like to generate the equations for an upper and > lower envelope that excludes a percentage of the data points. > > I would like to define the slope of the envelope line (say 3) and then have my > code find the intercept that fits my requirements (say 5% of data below the > lower envelope). This would then give me the equation and I could plot the > upper and lower envelopes. > > > I hope this makes sense. Thanks for any help. For this particular problem - where you know the slope - it's not too hard. If the slope is b, and your points are x and y, compute y-b*x, then sort that array, and choose the 5th and 95th percentile values. Anne From charlesr.harris at gmail.com Tue Sep 30 16:49:29 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 30 Sep 2008 14:49:29 -0600 Subject: [Numpy-discussion] Vectorization of the product of several matrices ? In-Reply-To: <31254818.516611222616846993.JavaMail.www@wwinf8401> References: <31254818.516611222616846993.JavaMail.www@wwinf8401> Message-ID: On Sun, Sep 28, 2008 at 9:47 AM, oc-spam66 wrote: > > Hello, > > I have two lists of numpy matrices : LM = [M_i, i=1..N] and LN = [N_i, i > =1..N] > and I would like to compute the list of the products : LP = [M_i * N_i, > i=1..N]. > > I can do : > > P=[] > for i in range(N) : > P.append(LM[i]*LN[i]) > > But this is not vectorized. Is there a faster solution ? > Can I vectorize this operation ? How ? > Will it be more efficient than the code above ? > > Thank you for your help, > There are at least three methods I can think of, but choosing the best one requires more information. How long are the lists? Do the arrays have variable dimensions? The simplest and most adaptable method is probably something like In [1]: M = [matrix(eye(3))]*5 In [2]: N = [matrix(eye(3))]*5 In [3]: P = [m*n for m,n in zip(M,N)] In [4]: P Out[4]: [matrix([[ 1., 0., 0.], [ 0., 1., 0.], [ 0., 0., 1.]]), matrix([[ 1., 0., 0.], [ 0., 1., 0.], [ 0., 0., 1.]]), matrix([[ 1., 0., 0.], [ 0., 1., 0.], [ 0., 0., 1.]]), matrix([[ 1., 0., 0.], [ 0., 1., 0.], [ 0., 0., 1.]]), matrix([[ 1., 0., 0.], [ 0., 1., 0.], [ 0., 0., 1.]])] Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Tue Sep 30 17:04:39 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 30 Sep 2008 17:04:39 -0400 Subject: [Numpy-discussion] Help needed with the priority of 0d-arrays and np.scalars Message-ID: <200809301704.39434.pgmdevlist@gmail.com> Sorry to bump my own post, I know it's rude... However, I'm in dire need for some pointers: what are the priority rules when multiplying numpy scalars and 0d ndarrays ? Thanks a lot in advance From oliphant at enthought.com Tue Sep 30 17:18:13 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 30 Sep 2008 16:18:13 -0500 Subject: [Numpy-discussion] Help needed with the priority of 0d-arrays and np.scalars In-Reply-To: <200809301704.39434.pgmdevlist@gmail.com> References: <200809301704.39434.pgmdevlist@gmail.com> Message-ID: <48E29795.702@enthought.com> Pierre GM wrote: > Sorry to bump my own post, I know it's rude... > > However, I'm in dire need for some pointers: what are the priority rules when > multiplying numpy scalars and 0d ndarrays ? > > Thanks a lot in advance > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > Hmm... I'm not 100% sure off the top of my head, but I would say that 0d arrays should determine the type coercion if any and the returned thing should be a numpy scalar. -Travis From lists_ravi at lavabit.com Tue Sep 30 17:20:51 2008 From: lists_ravi at lavabit.com (Ravi) Date: Tue, 30 Sep 2008 17:20:51 -0400 Subject: [Numpy-discussion] PyArray_Resize reference counting In-Reply-To: <48E28B60.2030403@enthought.com> References: <200809301548.10782.lists_ravi@lavabit.com> <48E28B60.2030403@enthought.com> Message-ID: <200809301720.53958.lists_ravi@lavabit.com> On Tuesday 30 September 2008 16:26:08 Travis E. Oliphant wrote: > You need to do something like: > > temp = PyArray_Resize(array,...) > Py_DECREF(array) > array = temp In this case, since a new array may be returned, is there no way to synchronize with other objects holding a reference to the original array? In other words, let's say A, B and C hold references to X, a numpy array. Then, if C needs to resize the array, the newly resized array is not available to A and B since their references are to the old array. Since C will typically not know about A and B, should resizing be disallowed (by me as a programmer writing code for A, B and C) in order to ensure that the arrays as seen by them do not go out of sync? Regards, Ravi From peridot.faceted at gmail.com Tue Sep 30 17:31:17 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Tue, 30 Sep 2008 17:31:17 -0400 Subject: [Numpy-discussion] Proposal: scipy.spatial In-Reply-To: <320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com> References: <48E1A6A2.4050406@noaa.gov> <320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com> Message-ID: 2008/9/30 Peter : > On Tue, Sep 30, 2008 at 5:10 AM, Christopher Barker > wrote: >> >> Anne Archibald wrote: >>> I suggest the creation of >>> a new submodule of scipy, scipy.spatial, >> >> +1 >> >> Here's one to consider: >> http://pypi.python.org/pypi/Rtree >> and perhaps other stuff from: >> http://trac.gispython.org/spatialindex/wiki >> which I think is LGPL -- can scipy use that? > > There is also a KDTree module in Biopython (which is under a BSD/MIT > style licence), > http://biopython.org/SRC/biopython/Bio/KDTree/ > > The current version is in C, there is an older version available in > the CVS history in C++ too, > http://cvs.biopython.org/cgi-bin/viewcvs/viewcvs.cgi/biopython/Bio/KDTree/?cvsroot=biopython I think the profusion of different implementations is an argument for including this in scipy. I think it is also an argument for providing a standard interface with (at least potentially) several different implementations. At the moment, that proposed interface looks like: T = KDTree(data) distances, indices = T.query(xs) # single nearest neighbor distances, indices = T.query(xs, k=10) # ten nearest neighbors distances, indices = T.query(xs, k=None, distance_upper_bound=1.0) # all within 1 of x In the first two cases, missing neighbors are represented with an infinite distance and an invalid index. In the last case, distances and indices are both either lists (if there's only one query point) or object arrays of lists (if there are many query points). If only one neighbor is requested, the array does not have a dimension of length 1 in the which-neighbor position. If (potentially) many neighbors are returned, they are sorted by distance, nearest first. What do you think of this interface? It may make sense to provide additional kinds of query - nearest neighbors between two trees, for example - some of which would be available only for some implementations. Anne From gael.varoquaux at normalesup.org Tue Sep 30 17:34:23 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 30 Sep 2008 23:34:23 +0200 Subject: [Numpy-discussion] Proposal: scipy.spatial In-Reply-To: References: <48E1A6A2.4050406@noaa.gov> <320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com> Message-ID: <20080930213423.GB22173@phare.normalesup.org> On Tue, Sep 30, 2008 at 05:31:17PM -0400, Anne Archibald wrote: > T = KDTree(data) > distances, indices = T.query(xs) # single nearest neighbor > distances, indices = T.query(xs, k=10) # ten nearest neighbors > distances, indices = T.query(xs, k=None, distance_upper_bound=1.0) # > all within 1 of x > What do you think of this interface? k=None in the third call to T.query seems redundant. It should be possible do put some logics so that the call is simply distances, indices = T.query(xs, distance_upper_bound=1.0) My 2 cents, Ga?l From pgmdevlist at gmail.com Tue Sep 30 17:36:59 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 30 Sep 2008 17:36:59 -0400 Subject: [Numpy-discussion] Help needed with the priority of 0d-arrays and np.scalars In-Reply-To: <48E29795.702@enthought.com> References: <200809301704.39434.pgmdevlist@gmail.com> <48E29795.702@enthought.com> Message-ID: <200809301737.00340.pgmdevlist@gmail.com> On Tuesday 30 September 2008 17:18:13 Travis E. Oliphant wrote: > Hmm... I'm not 100% sure off the top of my head, but I would say that > 0d arrays should determine the type coercion if any and the returned > thing should be a numpy scalar. Travis, I'm afraid I don't understand. According to you, a 0d array should always have the priority on a numpy scalar, right ? That doesn't seem to be the case: ma.masked is a 0d ndarray with a np.float64 dtype and a __array__priority__ of 15. It implements special __mul__ and __rmul__ methods. The multiplication `ma.masked * np.float64(1)` calls `ma.masked.__mul__`, as it should. The multiplication `np.float64(1)*ma.masked` doesn't call `ma.masked.__mul__` nor `ma.masked.__rmul__`. However, `np.float32(1)*ma.masked` does. How could I force `np.float64(1)*ma.masked` to call the proper method ? From peridot.faceted at gmail.com Tue Sep 30 18:10:46 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Tue, 30 Sep 2008 18:10:46 -0400 Subject: [Numpy-discussion] Proposal: scipy.spatial In-Reply-To: <20080930213423.GB22173@phare.normalesup.org> References: <48E1A6A2.4050406@noaa.gov> <320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com> <20080930213423.GB22173@phare.normalesup.org> Message-ID: 2008/9/30 Gael Varoquaux : > On Tue, Sep 30, 2008 at 05:31:17PM -0400, Anne Archibald wrote: >> T = KDTree(data) > >> distances, indices = T.query(xs) # single nearest neighbor > >> distances, indices = T.query(xs, k=10) # ten nearest neighbors > >> distances, indices = T.query(xs, k=None, distance_upper_bound=1.0) # >> all within 1 of x > >> What do you think of this interface? > > k=None in the third call to T.query seems redundant. It should be > possible do put some logics so that the call is simply > > distances, indices = T.query(xs, distance_upper_bound=1.0) Well, the problem with this is that you often want to provide a distance upper bound as well as a number of nearest neighbors. For example, suppose you are trying to find the point of closest approach of a path to the set of points stored in a kd-tree. You do the first query normally, but then since the second point is close to the first, you can accelerate the search dramatically by providing an upper bound of the distance from the first point's nearest neighbor to the second point. With a little luck, this will save ever having to visit much of the tree. It is also possible, of course, to provide wrappers to query() that do just one thing; I had this in mind more for fiddling the output (returning only the distance to the nearest neighbor, for instance). Anne From wnbell at gmail.com Tue Sep 30 18:15:19 2008 From: wnbell at gmail.com (Nathan Bell) Date: Tue, 30 Sep 2008 18:15:19 -0400 Subject: [Numpy-discussion] Proposal: scipy.spatial In-Reply-To: References: <48E1A6A2.4050406@noaa.gov> <320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com> <20080930213423.GB22173@phare.normalesup.org> Message-ID: On Tue, Sep 30, 2008 at 6:10 PM, Anne Archibald wrote: > > Well, the problem with this is that you often want to provide a > distance upper bound as well as a number of nearest neighbors. For This use case is also important in scattered data interpolation, so we definitely want to support it. -- Nathan Bell wnbell at gmail.com http://graphics.cs.uiuc.edu/~wnbell/ From dalcinl at gmail.com Tue Sep 30 18:24:56 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 30 Sep 2008 19:24:56 -0300 Subject: [Numpy-discussion] xml-rpc with numpy arrays In-Reply-To: <200809301318.36701.binet@cern.ch> References: <98DE1D44-A0B6-4CC6-8FD0-11F187D20862@bryant.edu> <3d375d730809301205l7c138d78xc4b687b7403c4df7@mail.gmail.com> <200809301318.36701.binet@cern.ch> Message-ID: Sebastien, numpy arrays are picklable; so no need to register them with copy_reg. I believe the actual problem with xmlrpclib is that it uses the marshal protocol (only supports core builtin types), and not the pickle protocol. On Tue, Sep 30, 2008 at 5:18 PM, Sebastien Binet wrote: > Hi, > > [..snip..] > On Tuesday 30 September 2008 12:05:27 Robert Kern wrote: >> > doing something wrong. Is there a way to fix this? Is there another >> > approach that I should be using? >> >> The marshal module *only* handles builtin Python types. It explicitly >> does not handle anything from third parties like numpy. > > but you can use the copy_reg module ('copyreg' in py2.6) to register > pickling/unpickling functions to make 'pickle' able to handle user-types: > > ## test_pickling_numpy.py ## > import numpy > import copy_reg > import cPickle as pickle > > def from_string(s): > return numpy.fromstring(s) > > def to_string(o): > return from_string, (o.tostring(),) > > a = numpy.arange (10,dtype=float) > copy_reg.pickle (numpy.ndarray, to_string, from_string) > s = pickle.dumps (a) > b = pickle.loads (s) > assert (a==b, "pickling failed !") > ## EOF ## > > see for more informations: > http://effbot.org/librarybook/copy-reg.htm > > cheers, > sebastien. > -- > ################################### > # Dr. Sebastien Binet > # Lawrence Berkeley National Lab. > # 1 Cyclotron Road > # Berkeley, CA 94720 > ################################### > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From bevan07 at gmail.com Tue Sep 30 18:35:21 2008 From: bevan07 at gmail.com (Bevan) Date: Tue, 30 Sep 2008 22:35:21 +0000 (UTC) Subject: [Numpy-discussion] Upper and lower envelopes References: Message-ID: Anne Archibald gmail.com> writes: > > 2008/9/30 bevan gmail.com>: > > Hello, > > > > I have some XY data. I would like to generate the equations for an upper and > > lower envelope that excludes a percentage of the data points. > > > > I would like to define the slope of the envelope line (say 3) and then have my > > code find the intercept that fits my requirements (say 5% of data below the > > lower envelope). This would then give me the equation and I could plot the > > upper and lower envelopes. > > > > > > I hope this makes sense. Thanks for any help. > > For this particular problem - where you know the slope - it's not too > hard. If the slope is b, and your points are x and y, compute y-b*x, > then sort that array, and choose the 5th and 95th percentile values. > > Anne > David and Anne, Thanks for your help. I first tried your suggestion David but did not get it working by the time Anne's post arrived. I plan on trying to get the optimize.fmin to work as i can see some cool uses for it in the future. I managed to get Anne's suggestion to work relatively quickly (only slowed by the dullard operating the machine...) Apologies for the lack of signature in my first post and thanks again. My code (for what its worth) is as follows: import numpy as np import pylab from scipy import stats,polyval #If the slope is b, and your points are x and y, compute y-b*x, #then sort that array, and choose the 5th and 95th percentile values. def Envelope(x,y,slpe,percntExclude): ans = y-slpe*x ans.sort() intercpt =stats.scoreatpercentile(ans,percntExclude) return intercpt slpeUpper = 1.0 slpeLower = 3.0 percntUpper = 95 percntLower = 5 x = np.random.rand(100) y = x*3+1.5+ np.random.rand(100) #Linear regression using stats.linregress #Returns: slope, intercept, r, two-tailed prob, stderr-of-the-estimate slpe,intercpt,r,tt,stderr=stats.linregress(x,y) yRegress=polyval([slpe,intercpt],x) intercptLower = Envelope(x,y,slpeLower,percntLower) intercptUpper = Envelope(x,y,slpeUpper,percntUpper) yLower = polyval([slpeLower,intercptLower],x) yUpper = polyval([slpeUpper,intercptUpper],x) pylab.figure() pylab.plot(x,y,'k.') pylab.plot(x,yRegress,'b-') pylab.plot(x,yLower,'r--') pylab.plot(x,yUpper,'r--') pylab.show() Bevan Jenkins From hep.sebastien.binet at gmail.com Tue Sep 30 18:42:44 2008 From: hep.sebastien.binet at gmail.com (Sebastien Binet) Date: Tue, 30 Sep 2008 15:42:44 -0700 Subject: [Numpy-discussion] xml-rpc with numpy arrays In-Reply-To: References: <98DE1D44-A0B6-4CC6-8FD0-11F187D20862@bryant.edu> <200809301318.36701.binet@cern.ch> Message-ID: <200809301542.44608.binet@cern.ch> Lisandro, On Tuesday 30 September 2008 15:24:56 Lisandro Dalcin wrote: > Sebastien, numpy arrays are picklable; so no need to register them > with copy_reg. I believe the actual problem with xmlrpclib is that it > uses the marshal protocol (only supports core builtin types), and not > the pickle protocol. yeah... Robert pointed it to me that as xmlrpc is meant for cross-language RPC, sending python objects over the wire isn't so useful, hence the usage of marshal instead of pickle. cheers, sebastien. -- ################################### # Dr. Sebastien Binet # Lawrence Berkeley National Lab. # 1 Cyclotron Road # Berkeley, CA 94720 ################################### From oliphant at enthought.com Tue Sep 30 18:54:21 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 30 Sep 2008 17:54:21 -0500 Subject: [Numpy-discussion] PyArray_Resize reference counting In-Reply-To: <200809301720.53958.lists_ravi@lavabit.com> References: <200809301548.10782.lists_ravi@lavabit.com> <48E28B60.2030403@enthought.com> <200809301720.53958.lists_ravi@lavabit.com> Message-ID: <48E2AE1D.9080802@enthought.com> Ravi wrote: > On Tuesday 30 September 2008 16:26:08 Travis E. Oliphant wrote: > >> You need to do something like: >> >> temp = PyArray_Resize(array,...) >> Py_DECREF(array) >> array = temp >> > > In this case, since a new array may be returned, is there no way to > synchronize with other objects holding a reference to the original array? Hold on!! I just went to the code and noticed that PyArray_Resize returns None. So, you certainly don't want to point array to it. The array does not get any reference count changes. There are three concepts here: 1) The array object 2) The memory for the array 3) The reference count on the array object PyArray_Resize doesn't change 1) or 3), it changes 2) So, PyObject *dummy; dummy = PyArray_Resize(array, ...) if (dummy == NULL) goto fail; Py_DECREF(dummy) is what you need to do. -Travis From bblais at bryant.edu Tue Sep 30 20:27:32 2008 From: bblais at bryant.edu (Brian Blais) Date: Tue, 30 Sep 2008 20:27:32 -0400 Subject: [Numpy-discussion] xml-rpc with numpy arrays In-Reply-To: <200809301542.44608.binet@cern.ch> References: <98DE1D44-A0B6-4CC6-8FD0-11F187D20862@bryant.edu> <200809301318.36701.binet@cern.ch> <200809301542.44608.binet@cern.ch> Message-ID: <1861310C-ED7D-4554-83BF-5751549041E7@bryant.edu> On Sep 30, 2008, at 18:42 , Sebastien Binet wrote: > yeah... Robert pointed it to me that as xmlrpc is meant for cross- > language RPC, > sending python objects over the wire isn't so useful, hence the > usage of marshal > instead of pickle. > thanks for all of the help. My initial solution is to pickle my object, with the text-based version of pickle, and send it across rpc. I do this because the actual thing I am sending is a dictionary, with lots of arrays, and other things. I'll have a look at the format that Robert sent, because that looks useful for other things I am doing. Sebastien, why is sending python objects over the wire not so useful? is there a better way to do this sort of thing than xmlrpc? I thought it looked particularly simple (other than this pickling issue, of course. :) ). thanks, bb -- Brian Blais bblais at bryant.edu http://web.bryant.edu/~bblais -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Tue Sep 30 20:39:20 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 30 Sep 2008 17:39:20 -0700 Subject: [Numpy-discussion] Fwd: [sage-devel] Sage Days 11: November 7-10, Austin, Texas In-Reply-To: References: Message-ID: Hi all, I think the original announcement only went to the Sage lists, but this should be of interest to this crowd as well. Cheers, f ---------- Forwarded message ---------- From: Craig Citro Date: Sun, Sep 21, 2008 at 8:47 PM Subject: [sage-devel] Sage Days 11: November 7-10, Austin, Texas To: sage-devel at googlegroups.com, sage-support at googlegroups.com Hi all, This is a reminder that Sage Days 11 is fast approaching! The topic is "Special functions and computational number theory meet scientific computing." The plan is to bring together a bunch of number theorists and scientific computing/supercomputing experts, and Austin's incredible supercomputing facilities make this the perfect place for it. If you're new to Sage, and interested in this topic, attending a Sage Days is the perfect way to get more involved. (For those of you who've been around a while, you know the drill -- just go ahead and skip this paragraph.) Sage Days are friendly and very intensive focused development workshops, which are probably fairly different from other math-related conferences and workshops you've attended. There are usually only one to two talks per day, in order to allow for the maximum time for working discussions and coding sprints. The groups are organized on the first day, and there are regular progress reports throughout the workshop. We have a webpage: http://www.math.utexas.edu/sage and a wiki page: http://wiki.sagemath.org/days11 If you're planning on joining us in Austin, please register soon at http://www.ma.utexas.edu/sage/rform1.php, and add your name to the wiki. We do have some limited funding available, so please let us know if you'll need funding. We have a block of rooms reserved at the Doubletree Guest Suites in Austin, and we'll let you know more details on that soon. If you have any questions, feel free to email me or any of the other organizers! On behalf of the organizing committee, -Craig Citro From hep.sebastien.binet at gmail.com Tue Sep 30 20:52:40 2008 From: hep.sebastien.binet at gmail.com (Sebastien Binet) Date: Tue, 30 Sep 2008 17:52:40 -0700 Subject: [Numpy-discussion] xml-rpc with numpy arrays In-Reply-To: <1861310C-ED7D-4554-83BF-5751549041E7@bryant.edu> References: <98DE1D44-A0B6-4CC6-8FD0-11F187D20862@bryant.edu> <200809301542.44608.binet@cern.ch> <1861310C-ED7D-4554-83BF-5751549041E7@bryant.edu> Message-ID: <200809301752.41097.binet@cern.ch> Brian, On Tuesday 30 September 2008 17:27:32 Brian Blais wrote: > On Sep 30, 2008, at 18:42 , Sebastien Binet wrote: > > yeah... Robert pointed it to me that as xmlrpc is meant for cross- > > language RPC, > > sending python objects over the wire isn't so useful, hence the > > usage of marshal > > instead of pickle. > > thanks for all of the help. My initial solution is to pickle my > object, with the text-based version of pickle, and send it across > rpc. I do this because the actual thing I am sending is a > dictionary, with lots of arrays, and other things. I'll have a look > at the format that Robert sent, because that looks useful for other > things I am doing. > > Sebastien, why is sending python objects over the wire not so > useful? it isn't useful if you don't have a python interpreter instance on the other end of the wire, ie a python program on one end and a javascript one on the other end :) (well, you can always re-engineer the pickle format to unpickle what you received, etc... it is 'just' a matter of programming) cheers, sebastien. -- ################################### # Dr. Sebastien Binet # Lawrence Berkeley National Lab. # 1 Cyclotron Road # Berkeley, CA 94720 ################################### From david.huard at gmail.com Tue Sep 30 22:28:38 2008 From: david.huard at gmail.com (David Huard) Date: Tue, 30 Sep 2008 22:28:38 -0400 Subject: [Numpy-discussion] Upper and lower envelopes In-Reply-To: References: Message-ID: <91cf711d0809301928yab8f1bek29db88542abf398e@mail.gmail.com> On Tue, Sep 30, 2008 at 4:37 PM, Anne Archibald wrote: > 2008/9/30 bevan : > > Hello, > > > > I have some XY data. I would like to generate the equations for an upper > and > > lower envelope that excludes a percentage of the data points. > > > > I would like to define the slope of the envelope line (say 3) and then > have my > > code find the intercept that fits my requirements (say 5% of data below > the > > lower envelope). This would then give me the equation and I could plot > the > > upper and lower envelopes. > > > > > > I hope this makes sense. Thanks for any help. > > For this particular problem - where you know the slope - it's not too > hard. If the slope is b, and your points are x and y, compute y-b*x, > then sort that array, and choose the 5th and 95th percentile values. > That's a pretty elegant solution. Thanks for sharing, David > > Anne > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Tue Sep 30 23:16:13 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 1 Oct 2008 00:16:13 -0300 Subject: [Numpy-discussion] xml-rpc with numpy arrays In-Reply-To: <1861310C-ED7D-4554-83BF-5751549041E7@bryant.edu> References: <98DE1D44-A0B6-4CC6-8FD0-11F187D20862@bryant.edu> <200809301318.36701.binet@cern.ch> <200809301542.44608.binet@cern.ch> <1861310C-ED7D-4554-83BF-5751549041E7@bryant.edu> Message-ID: On Tue, Sep 30, 2008 at 9:27 PM, Brian Blais > thanks for all of the help. My initial solution is to pickle my object, > with the text-based version of pickle, and send it across rpc. I do this > because the actual thing I am sending is a dictionary, with lots of arrays, > and other things. I'll have a look at the format that Robert sent, because > that looks useful for other things I am doing. Did you try to send binary pickles (protocol=2)? Perhaps it works, give a try! Of course, you need the client and server machines having the same arch. > Sebastien, why is sending python objects over the wire not so useful? is > there a better way to do this sort of thing than xmlrpc? I thought it > looked particularly simple (other than this pickling issue, of course. :) > ). > I believe xmlrpclib is currently the simpler approach. Some day I'll have the time to implement something similar using MPI communication with mpi4py. However, I believe it can be done even better: local, client-side proxies should automatically provide access to all members/methods of remote, server-side instances. The registering step needed within xmlrpclib is a bit ugly ;-) -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594