From faltet at pytables.org Fri Aug 1 01:26:55 2008 From: faltet at pytables.org (Francesc Alted) Date: Fri, 1 Aug 2008 07:26:55 +0200 Subject: [Numpy-discussion] User-defined data-types Message-ID: <200808010726.57592.faltet@pytables.org> Hi, We need to make a dtype that needs to be defined in the user's application space. In the NumPy book there is a section devoted to this subject, but also warns: """ Adding data-types is one of the less well-tested areas for NumPy 1.0, so there may be bugs remaining in the approach. """ Anybody out there has tried this successfully? Thanks, -- Francesc Alted From david at ar.media.kyoto-u.ac.jp Fri Aug 1 01:30:47 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 01 Aug 2008 14:30:47 +0900 Subject: [Numpy-discussion] numpy 1.1.rc2: win32 binaries In-Reply-To: References: <488D58A3.6070800@ar.media.kyoto-u.ac.jp> <488D5984.202@ar.media.kyoto-u.ac.jp> Message-ID: <48929F87.8010806@ar.media.kyoto-u.ac.jp> Bruce Southey wrote: > Hi, > The installation worked on my old Athlon XP running Windows XP and > 'numpy.test(level=1)' gave no errors. > By old, do you mean it does not have at least SSE2 ? If so, the problem would not have happened anyway because the numpy installer does not use ATLAS there, Thanks for the report, David From charlesr.harris at gmail.com Fri Aug 1 02:22:17 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 1 Aug 2008 00:22:17 -0600 Subject: [Numpy-discussion] User-defined data-types In-Reply-To: <200808010726.57592.faltet@pytables.org> References: <200808010726.57592.faltet@pytables.org> Message-ID: On Thu, Jul 31, 2008 at 11:26 PM, Francesc Alted wrote: > Hi, > > We need to make a dtype that needs to be defined in the user's > application space. In the NumPy book there is a section devoted to > this subject, but also warns: > > """ > Adding data-types is one of the less well-tested areas for NumPy 1.0, so > there may be bugs remaining in the approach. > """ > > Anybody out there has tried this successfully? > Not me, but it is on my list. You will probably want to use at least a few ufuncs with it, no? The new type will probably also need a type number. Oh, Heck, give it a shot. Someone has to go first O_o Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Fri Aug 1 07:05:56 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 01 Aug 2008 07:05:56 -0400 Subject: [Numpy-discussion] User-defined data-types References: <200808010726.57592.faltet@pytables.org> Message-ID: Charles R Harris wrote: > On Thu, Jul 31, 2008 at 11:26 PM, Francesc Alted > wrote: > >> Hi, >> >> We need to make a dtype that needs to be defined in the user's >> application space. In the NumPy book there is a section devoted to >> this subject, but also warns: >> >> """ >> Adding data-types is one of the less well-tested areas for NumPy 1.0, so >> there may be bugs remaining in the approach. >> """ >> >> Anybody out there has tried this successfully? >> > > Not me, but it is on my list. You will probably want to use at least a few > ufuncs with it, no? The new type will probably also need a type number. > Oh, > Heck, give it a shot. Someone has to go first O_o > > Chuck Yes, I have, a few months back. Check list archives. From bsouthey at gmail.com Fri Aug 1 09:57:11 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 01 Aug 2008 08:57:11 -0500 Subject: [Numpy-discussion] numpy 1.1.rc2: win32 binaries In-Reply-To: <48929F87.8010806@ar.media.kyoto-u.ac.jp> References: <488D58A3.6070800@ar.media.kyoto-u.ac.jp> <488D5984.202@ar.media.kyoto-u.ac.jp> <48929F87.8010806@ar.media.kyoto-u.ac.jp> Message-ID: <48931637.4050804@gmail.com> David Cournapeau wrote: > Bruce Southey wrote: > >> Hi, >> The installation worked on my old Athlon XP running Windows XP and >> 'numpy.test(level=1)' gave no errors. >> >> > > By old, do you mean it does not have at least SSE2 ? If so, the problem > would not have happened anyway because the numpy installer does not use > ATLAS there, > > Thanks for the report, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > Okay, I was not aware that the ticket was SSE2 as the system is only SSE. I can get access to a system that does. Can you please me the link again because I am getting a 404 error for this: http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy-1.1.1.dev5559-win32-superpack-python2.5.exe Thanks Bruce From Chris.Barker at noaa.gov Fri Aug 1 12:18:48 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 01 Aug 2008 09:18:48 -0700 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <5b8d13220807311128w56e1996bue1ec304aca672b47@mail.gmail.com> References: <92AFDF8C-418F-4B6D-895C-8E7F0FD47D90@dalkescientific.com> <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <846D821E-BA9F-4043-AE39-D7D648013C30@dalkescientific.com> <6B6B9FA6-864D-4D31-99BB-E9AE7A93A6D3@dalkescientific.com> <9457e7c80807301359k599b9155ge296ae804485c94@mail.gmail.com> <4891EC2F.9050501@noaa.gov> <4891E996.5070000@ar.media.kyoto-u.ac.jp> <4891F276.8070404@noaa.gov> <5b8d13220807311128w56e1996bue1ec304aca672b47@mail.gmail.com> Message-ID: <48933768.2010503@noaa.gov> David Cournapeau wrote: >> time python -c "import numpy" >> >> real 0m8.383s >> user 0m0.320s >> sys 0m7.805s > What does python -c "import sys; print sys.path" say ? A lot! 41 entries, and lot's of eggs -- are eggs an issue? I'm also wondering how the order is determined -- if it looked in site-packages first, it would find numpy a whole lot faster. I also tried: python -v -v -c "import numpy" &>junk2.txt which results in: # installing zipimport hook import zipimport # builtin # installed zipimport hook # trying /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site.so # trying /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/sitemodule.so # trying /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site.py ... ... And a LOT more: $ grep "# trying" junk2.txt | wc -l 7446 For comaprison: $ python -v -v -c "import sys" &>junk3.txt $ grep "# trying" junk3.txt | wc -l 618 which still seems like a lot. So I think I've found the problem, it's looking in 7446 places ! but why? I suspect the thing to do is to re-install from scratch, and only add in packages I'm really using now. I wonder if making sure all eggs are unzipped would help, too. Thanks for all your help on what is really OT at this point. -Chris $ python -c "import sys; [sys.stdout.write(p+'\n') for p in sys.path]; print len(sys.path) " /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/altgraph-0.6.7-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Pyrex-0.9.5.1a-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Shapely-1.0-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/ipython-0.8.2-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/py2app-0.4.2-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/modulegraph-0.7.2.dev_r21-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/PyPubSub-3.0a5-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/macholib-1.2.1.dev_r23-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/matplotlib-0.98.0-py2.5-macosx-10.3-fat.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.3-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Pylons-0.9.7beta5-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Tempita-0.2-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/WebError-0.8-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/WebOb-0.9.2-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Mako-0.2.0-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/decorator-2.2.0-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/simplejson-1.8.1-py2.5-macosx-10.3-ppc.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/FormEncode-1.0.1-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/PasteScript-1.6.3-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/PasteDeploy-1.3.2-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Paste-1.6-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Beaker-0.9.5-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/WebHelpers-0.6dev_20080613-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Routes-1.9-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/bdist_mpkg-0.4.3-py2.5.egg /Users/cbarker/HAZMAT/Hazpy/trunk/lib /usr/local/lib/wxPython-unicode-2.8.8.1/lib/python2.5/site-packages /usr/local/lib/wxPython-unicode-2.8.8.1/lib/python2.5/site-packages/wx-2.8-mac-unicode /Library/Frameworks/Python.framework/Versions/2.5/lib/python25.zip /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5 /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat-darwin /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat-mac /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat-mac/lib-scriptpackages /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-tk /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-dynload /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/PIL /usr/local/lib/wxPython-unicode-2.8.8.1/lib/python2.5 41 >> Any idea what could be wrong? I have no clue where to start, though I >> suppose a complete clean out and re-install of python comes to mind. >> >> oh, and this is a dual G5 PPC (which should have a faster disk than your >> Macbook) > > disk should not matter. If hot, everything should be in the IO buffer, > opening a file is of the order of a few micro seconds (that's > certainly the order on Linux; the VM on Mac OS X is likely not as > good, but still). > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From gael.varoquaux at normalesup.org Fri Aug 1 12:53:51 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 1 Aug 2008 18:53:51 +0200 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <48933768.2010503@noaa.gov> References: <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <846D821E-BA9F-4043-AE39-D7D648013C30@dalkescientific.com> <6B6B9FA6-864D-4D31-99BB-E9AE7A93A6D3@dalkescientific.com> <9457e7c80807301359k599b9155ge296ae804485c94@mail.gmail.com> <4891EC2F.9050501@noaa.gov> <4891E996.5070000@ar.media.kyoto-u.ac.jp> <4891F276.8070404@noaa.gov> <5b8d13220807311128w56e1996bue1ec304aca672b47@mail.gmail.com> <48933768.2010503@noaa.gov> Message-ID: <20080801165351.GC24781@phare.normalesup.org> On Fri, Aug 01, 2008 at 09:18:48AM -0700, Christopher Barker wrote: > > What does python -c "import sys; print sys.path" say ? > A lot! 41 entries, and lot's of eggs -- are eggs an issue? I'm also > wondering how the order is determined -- if it looked in site-packages > first, it would find numpy a whole lot faster. AFAIK this is a setuptools issue. From what I hear, it might be fixed in the svn version of setuptools, but they still have to make a release that has this feature. The two issues I can see are: import path priority, it should be screwed up like it is, and speed. Speed is obviously a hard problem. > I suspect the thing to do is to re-install from scratch, and only add in > packages I'm really using now. Avoid eggs if you can. This has been my policy. I am not sure how much this is just superstition or a real problem, though. I realize that you are on mac, and that mac unlike some distribution of linux does not have a good dependency tracking system. Thus seutptools and eggs are a great tentation. Them come to a cost, but it can probably be improved. If you care about this problem, you could try and work with the setuptools developers to improve the situation. I must say that I am under UBuntu, and I don't have the dependency problem at all, so setuptools does not answer an important need for me. I however realize that not everybody wants to use Ubuntu and I thus care about the problem, maybe not enough to invest much time in setuptools, but at least enough to try to report problems and track solution. Do not underestimate how difficult it is to get a package-manager that works well. If you ever do verify that it is indeed eggs that I slowing down your import, I'd be interested in having the confirmation, just so that I am sure I am not blaming them for nothing. Cheers, Ga?l From cournape at gmail.com Fri Aug 1 15:46:41 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 2 Aug 2008 04:46:41 +0900 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <48933768.2010503@noaa.gov> References: <846D821E-BA9F-4043-AE39-D7D648013C30@dalkescientific.com> <6B6B9FA6-864D-4D31-99BB-E9AE7A93A6D3@dalkescientific.com> <9457e7c80807301359k599b9155ge296ae804485c94@mail.gmail.com> <4891EC2F.9050501@noaa.gov> <4891E996.5070000@ar.media.kyoto-u.ac.jp> <4891F276.8070404@noaa.gov> <5b8d13220807311128w56e1996bue1ec304aca672b47@mail.gmail.com> <48933768.2010503@noaa.gov> Message-ID: <5b8d13220808011246k33def1eeqc0c1ca316bcefc70@mail.gmail.com> On Sat, Aug 2, 2008 at 1:18 AM, Christopher Barker wrote: > > A lot! 41 entries, and lot's of eggs -- are eggs an issue? I'm also > wondering how the order is determined -- if it looked in site-packages > first, it would find numpy a whole lot faster. I don't think the number itself is an issue. Putting eggs first is the way it has to be I think, that's just how eggs are supposed to work. > I also tried: > > python -v -v -c "import numpy" &>junk2.txt > > which results in: > > # installing zipimport hook > import zipimport # builtin > # installed zipimport hook > # trying > /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site.so > # trying > /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/sitemodule.so > # trying > /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site.py > > ... > ... > > And a LOT more: > > $ grep "# trying" junk2.txt | wc -l > 7446 > > For comaprison: > $ python -v -v -c "import sys" &>junk3.txt > $ grep "# trying" junk3.txt | wc -l > 618 > > which still seems like a lot. > > So I think I've found the problem, it's looking in 7446 places ! but why? Part of it is how python looks for modules. Again, I don't think the number itself is the issue: non existing files should not impact much because python import is basically doing a stat, and a stat on a non existing file, in the hot situation, takes nothing. IOW, I don't think the problem is the numbers themselves. It has to be something else. A simple profiling like python -m cProfile -o foo.stats foo.py and then: python -c "import pstats; p = pstats.Stats("foo.stats"); p.sort_stats('cumulative').print_stats(50)" May give useful information. This and using shark as Robert suggested should point to some direction, cheers, David From robert.kern at gmail.com Fri Aug 1 15:49:54 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 1 Aug 2008 14:49:54 -0500 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <20080801165351.GC24781@phare.normalesup.org> References: <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <6B6B9FA6-864D-4D31-99BB-E9AE7A93A6D3@dalkescientific.com> <9457e7c80807301359k599b9155ge296ae804485c94@mail.gmail.com> <4891EC2F.9050501@noaa.gov> <4891E996.5070000@ar.media.kyoto-u.ac.jp> <4891F276.8070404@noaa.gov> <5b8d13220807311128w56e1996bue1ec304aca672b47@mail.gmail.com> <48933768.2010503@noaa.gov> <20080801165351.GC24781@phare.normalesup.org> Message-ID: <3d375d730808011249h29460aavd39f3b0f8874c361@mail.gmail.com> On Fri, Aug 1, 2008 at 11:53, Gael Varoquaux wrote: > On Fri, Aug 01, 2008 at 09:18:48AM -0700, Christopher Barker wrote: >> > What does python -c "import sys; print sys.path" say ? > >> A lot! 41 entries, and lot's of eggs -- are eggs an issue? I'm also >> wondering how the order is determined -- if it looked in site-packages >> first, it would find numpy a whole lot faster. > > AFAIK this is a setuptools issue. From what I hear, it might be fixed in > the svn version of setuptools, but they still have to make a release that > has this feature. > > The two issues I can see are: import path priority, it should be screwed > up like it is, and speed. Speed is obviously a hard problem. > >> I suspect the thing to do is to re-install from scratch, and only add in >> packages I'm really using now. > > Avoid eggs if you can. This has been my policy. I am not sure how much > this is just superstition or a real problem, though. Superstition. [~]$ python -c "import sys; print len(sys.path)" 269 [~]$ python -v -v -c "import numpy" 2> foo.txt [~]$ wc -l foo.txt 42500 foo.txt [~]$ time python -c "import numpy" python -c "import numpy" 0.18s user 0.46s system 88% cpu 0.716 total So cut it out. Chris, please profile your import so we actually have some real information to work with instead of prejudices. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ondrej at certik.cz Fri Aug 1 16:33:02 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 1 Aug 2008 22:33:02 +0200 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <3d375d730807311302p46138c29k1cae6a5aa8899360@mail.gmail.com> References: <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <846D821E-BA9F-4043-AE39-D7D648013C30@dalkescientific.com> <6B6B9FA6-864D-4D31-99BB-E9AE7A93A6D3@dalkescientific.com> <9457e7c80807301359k599b9155ge296ae804485c94@mail.gmail.com> <9457e7c80807310242q75d4a943wbdd1d4e2c264b613@mail.gmail.com> <3d375d730807310303q54ef94f7m3ba74b3f47f6e5ea@mail.gmail.com> <0BD87DFD-6B55-44E3-90EA-C1F83301F091@dalkescientific.com> <3d375d730807311302p46138c29k1cae6a5aa8899360@mail.gmail.com> Message-ID: <85b5c3130808011333hff71ae8j7f82741c2ecb58c4@mail.gmail.com> On Thu, Jul 31, 2008 at 10:02 PM, Robert Kern wrote: > On Thu, Jul 31, 2008 at 05:43, Andrew Dalke wrote: >> On Jul 31, 2008, at 12:03 PM, Robert Kern wrote: > >>> But you still can't remove them since they are being used inside >>> numerictypes. That's why I labeled them "internal utility functions" >>> instead of leaving them with minimal docstrings such that you would >>> have to guess. >> >> My proposal is to replace that code with a table mapping >> the type name to the uppercase/lowercase/capitalized forms, >> thus eliminating the (small) amount of time needed to >> import string. >> >> It makes adding new types slightly more difficult. >> >> I know it's a tradeoff. > > Probably not a bad one. Write up the patch, and then we'll see how > much it affects the import time. > > I would much rather that we discuss concrete changes like this rather > than rehash the justifications of old decisions. Regardless of the > merits about the old decisions (and I agreed with your position at the > time), it's a pointless and irrelevant conversation. The decisions > were made, and now we have a user base to whom we have promised not to > break their code so egregiously again. The relevant conversation is > what changes we can make now. > > Some general guidelines: > > 1) Everything exposed by "from numpy import *" still needs to work. > a) The layout of everything under numpy.core is an implementation detail. > b) _underscored functions and explicitly labeled internal functions > can probably be modified. > c) Ask about specific functions when in doubt. > > 2) The improvement in import times should be substantial. Feel free to > bundle up the optimizations for consideration. > > 3) Moving imports from module-level down into the functions where they > are used is generally okay if we get a reasonable win from it. The > local imports should be commented, explaining that they are made local > in order to improve the import times. > > 4) __import__ hacks are off the table. > > 5) Proxy objects ... I would really like to avoid proxy objects. They > have caused fragility in the past. > > 6) I'm not a fan of having environment variables control the way numpy > gets imported, but I'm willing to consider it. For example, I might go > for having proxy objects for linalg et al. *only* if a particular > environment variable were set. But there had better be a very large > improvement in import times. I just want to say that I agree with Andrew that slow imports just suck. But it's not really that bad, for example on my system: In [1]: %time import numpy CPU times: user 0.11 s, sys: 0.01 s, total: 0.12 s Wall time: 0.12 s so that's ok. For comparison: In [1]: %time import sympy CPU times: user 0.12 s, sys: 0.02 s, total: 0.14 s Wall time: 0.14 s But I am still unhappy about it, I'd like if the package could import much faster, because it adds up, when you need to import 7 packages like that, it's suddenly 1s and that's just too much. But of course everything within the constrains that Robert has outlined. From the theoretical point of view, I don't understand why python cannot just import numpy (or any other package) immediatelly, and only at the moment the user actually access something, to import it in real. Mercurial uses a lazy import module, that does exactly this. Maybe that's an option? Look into mercurial/demandimport.py. Use it like this: In [1]: import demandimport In [2]: demandimport.enable() In [3]: %time import numpy CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s Wall time: 0.00 s That's pretty good, huh? :) Unfortunately, numpy cannot work with lazy import (yet): In [5]: %time from numpy import array ERROR: An unexpected error occurred while tokenizing input The following traceback may be corrupted or invalid The error message is: ('EOF in multi-line statement', (17, 0)) --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) [skip] /usr/lib/python2.5/site-packages/numpy/lib/index_tricks.py in () 14 import function_base 15 import numpy.core.defmatrix as matrix ---> 16 makemat = matrix.matrix 17 18 # contributed by Stefan van der Walt /home/ondra/ext/sympy/demandimport.pyc in __getattribute__(self, attr) 73 return object.__getattribute__(self, attr) 74 self._load() ---> 75 return getattr(self._module, attr) 76 def __setattr__(self, attr, val): 77 self._load() AttributeError: 'module' object has no attribute 'matrix' BTW, neither can SymPy. However, maybe it shows some possibilities and maybe it's possible to fix numpy to work with such a lazy import. On the other hand, I can imagine it can bring a lot more troubles, so it should probably only be optional. Ondrej From Chris.Barker at noaa.gov Fri Aug 1 18:22:40 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 01 Aug 2008 15:22:40 -0700 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <5b8d13220808011246k33def1eeqc0c1ca316bcefc70@mail.gmail.com> References: <846D821E-BA9F-4043-AE39-D7D648013C30@dalkescientific.com> <6B6B9FA6-864D-4D31-99BB-E9AE7A93A6D3@dalkescientific.com> <9457e7c80807301359k599b9155ge296ae804485c94@mail.gmail.com> <4891EC2F.9050501@noaa.gov> <4891E996.5070000@ar.media.kyoto-u.ac.jp> <4891F276.8070404@noaa.gov> <5b8d13220807311128w56e1996bue1ec304aca672b47@mail.gmail.com> <48933768.2010503@noaa.gov> <5b8d13220808011246k33def1eeqc0c1ca316bcefc70@mail.gmail.com> Message-ID: <48938CB0.5010403@noaa.gov> David Cournapeau wrote: > IOW, I don't think the problem is the numbers themselves. It has to be > something else. A simple profiling like > > python -m cProfile -o foo.stats foo.py > > and then: > > python -c "import pstats; p = pstats.Stats("foo.stats"); > p.sort_stats('cumulative').print_stats(50)" OK, see the results -- I think (though i may be wrong) this means that the problem isn't in finding the numpy package: As for Shark, I'm sorry I missed that message, but I'm trying to see if I can do that now -- I don't seem to have Shark installed, and the ADC site doesn't seem to be working, but I'll keep looking. Thanks for all your help with this... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: ImportNumpyStats.txt URL: From robert.kern at gmail.com Fri Aug 1 18:28:48 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 1 Aug 2008 17:28:48 -0500 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <48938CB0.5010403@noaa.gov> References: <9457e7c80807301359k599b9155ge296ae804485c94@mail.gmail.com> <4891EC2F.9050501@noaa.gov> <4891E996.5070000@ar.media.kyoto-u.ac.jp> <4891F276.8070404@noaa.gov> <5b8d13220807311128w56e1996bue1ec304aca672b47@mail.gmail.com> <48933768.2010503@noaa.gov> <5b8d13220808011246k33def1eeqc0c1ca316bcefc70@mail.gmail.com> <48938CB0.5010403@noaa.gov> Message-ID: <3d375d730808011528ic8dacfbm883def8d073b4228@mail.gmail.com> On Fri, Aug 1, 2008 at 17:22, Christopher Barker wrote: > David Cournapeau wrote: >> >> IOW, I don't think the problem is the numbers themselves. It has to be >> something else. A simple profiling like >> >> python -m cProfile -o foo.stats foo.py >> >> and then: >> >> python -c "import pstats; p = pstats.Stats("foo.stats"); >> p.sort_stats('cumulative').print_stats(50)" > > OK, see the results -- I think (though i may be wrong) this means that the > problem isn't in finding the numpy package: Can you send foo.stats, too? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Chris.Barker at noaa.gov Fri Aug 1 18:50:02 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 01 Aug 2008 15:50:02 -0700 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <3d375d730808011528ic8dacfbm883def8d073b4228@mail.gmail.com> References: <9457e7c80807301359k599b9155ge296ae804485c94@mail.gmail.com> <4891EC2F.9050501@noaa.gov> <4891E996.5070000@ar.media.kyoto-u.ac.jp> <4891F276.8070404@noaa.gov> <5b8d13220807311128w56e1996bue1ec304aca672b47@mail.gmail.com> <48933768.2010503@noaa.gov> <5b8d13220808011246k33def1eeqc0c1ca316bcefc70@mail.gmail.com> <48938CB0.5010403@noaa.gov> <3d375d730808011528ic8dacfbm883def8d073b4228@mail.gmail.com> Message-ID: <4893931A.5050804@noaa.gov> Robert Kern wrote: > Can you send foo.stats, too? sure can. Also, I've got Shark up and running, and have run it on a script that does nothing but import numpy, but really have no idea what I'm looking at, or how to send it to someone that does (you?). I'll keep poking at it some more. Thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- A non-text attachment was scrubbed... Name: ImportNumpy.stats.zip Type: application/zip Size: 22348 bytes Desc: not available URL: From robert.kern at gmail.com Fri Aug 1 19:01:58 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 1 Aug 2008 18:01:58 -0500 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <4893931A.5050804@noaa.gov> References: <4891EC2F.9050501@noaa.gov> <4891E996.5070000@ar.media.kyoto-u.ac.jp> <4891F276.8070404@noaa.gov> <5b8d13220807311128w56e1996bue1ec304aca672b47@mail.gmail.com> <48933768.2010503@noaa.gov> <5b8d13220808011246k33def1eeqc0c1ca316bcefc70@mail.gmail.com> <48938CB0.5010403@noaa.gov> <3d375d730808011528ic8dacfbm883def8d073b4228@mail.gmail.com> <4893931A.5050804@noaa.gov> Message-ID: <3d375d730808011601n198943b1k30cc1423bf147bcb@mail.gmail.com> On Fri, Aug 1, 2008 at 17:50, Christopher Barker wrote: > Robert Kern wrote: >> >> Can you send foo.stats, too? > > sure can. > > Also, I've got Shark up and running, and have run it on a script that > does nothing but import numpy, but really have no idea what I'm looking > at, or how to send it to someone that does (you?). I'll keep poking at > it some more. File/Save As..., pick a file name. When asked about whether to embed source files or strip them out, choose Strip. Then email the resulting .mshark file to me. It looks like your Python just takes a truly inordinate amount of time to execute any code. Some of the problematic modules like httplib have been moved to local imports, but the time it takes for your Python to execute the code in that module is still ridiculously large. Can you profile just importing httplib instead of numpy? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Chris.Barker at noaa.gov Fri Aug 1 19:11:49 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 01 Aug 2008 16:11:49 -0700 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <3d375d730808011249h29460aavd39f3b0f8874c361@mail.gmail.com> References: <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <6B6B9FA6-864D-4D31-99BB-E9AE7A93A6D3@dalkescientific.com> <9457e7c80807301359k599b9155ge296ae804485c94@mail.gmail.com> <4891EC2F.9050501@noaa.gov> <4891E996.5070000@ar.media.kyoto-u.ac.jp> <4891F276.8070404@noaa.gov> <5b8d13220807311128w56e1996bue1ec304aca672b47@mail.gmail.com> <48933768.2010503@noaa.gov> <20080801165351.GC24781@phare.normalesup.org> <3d375d730808011249h29460aavd39f3b0f8874c361@mail.gmail.com> Message-ID: <48939835.4080801@noaa.gov> Robert Kern wrote: > So cut it out. > > Chris, please profile your import so we actually have some real > information to work with instead of prejudices. OK, while we're working on that, I've tried just re-naming my entire python install, then starting from scratch. So far, I've python re-installed, and installed just numpy ('1.1.1rc2'), and the import seems downright blazing compared to what I was getting: $ time python -c "import numpy" real 0m0.973s user 0m0.290s sys 0m0.682s much more in line with what others are getting. I'll start installing stuff that I actually am using now, and maybe I'll see when (if) it breaks down. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Fri Aug 1 19:10:34 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 1 Aug 2008 18:10:34 -0500 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <4893931A.5050804@noaa.gov> References: <4891EC2F.9050501@noaa.gov> <4891E996.5070000@ar.media.kyoto-u.ac.jp> <4891F276.8070404@noaa.gov> <5b8d13220807311128w56e1996bue1ec304aca672b47@mail.gmail.com> <48933768.2010503@noaa.gov> <5b8d13220808011246k33def1eeqc0c1ca316bcefc70@mail.gmail.com> <48938CB0.5010403@noaa.gov> <3d375d730808011528ic8dacfbm883def8d073b4228@mail.gmail.com> <4893931A.5050804@noaa.gov> Message-ID: <3d375d730808011610j77c2368ag78bfaefbfdf03b8c@mail.gmail.com> On Fri, Aug 1, 2008 at 17:50, Christopher Barker wrote: > Robert Kern wrote: >> >> Can you send foo.stats, too? > > sure can. > > Also, I've got Shark up and running, and have run it on a script that > does nothing but import numpy, but really have no idea what I'm looking > at, or how to send it to someone that does (you?). I'll keep poking at > it some more. Looking at the Shark results you sent me, it looks like all of your time is getting sucked up by the system call getdirentries(). Googling for some of the function names in that stack brings me to the message "Slow python initialization" on the Pythonmac-SIG: http://mail.python.org/pipermail/pythonmac-sig/2005-December/015542.html The ultimate resolution was that Bill Spotz, the original poster, ran Disk Utility and used the Disk Repair option to clean up a large number of unused inodes. This solved the problem for him: http://mail.python.org/pipermail/pythonmac-sig/2005-December/015548.html -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Chris.Barker at noaa.gov Fri Aug 1 19:17:07 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 01 Aug 2008 16:17:07 -0700 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <48939835.4080801@noaa.gov> References: <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <6B6B9FA6-864D-4D31-99BB-E9AE7A93A6D3@dalkescientific.com> <9457e7c80807301359k599b9155ge296ae804485c94@mail.gmail.com> <4891EC2F.9050501@noaa.gov> <4891E996.5070000@ar.media.kyoto-u.ac.jp> <4891F276.8070404@noaa.gov> <5b8d13220807311128w56e1996bue1ec304aca672b47@mail.gmail.com> <48933768.2010503@noaa.gov> <20080801165351.GC24781@phare.normalesup.org> <3d375d730808011249h29460aavd39f3b0f8874c361@mail.gmail.com> <48939835.4080801@noaa.gov> Message-ID: <48939973.4010008@noaa.gov> Christopher Barker wrote: > $ time python -c "import numpy" > > real 0m0.973s > user 0m0.290s > sys 0m0.682s > > much more in line with what others are getting. > > I'll start installing stuff that I actually am using now, and maybe I'll > see when (if) it breaks down. OK, I just installed wxPython, and whoa! time python -c "import numpy" real 0m2.793s user 0m0.294s sys 0m2.494s so it's taking almost two seconds more to import numpy, now that wxPython is installed. I haven't even imported it yet. importing wx isn't as bad: $ time python -c "import wx" real 0m1.589s user 0m0.274s sys 0m1.000s -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Fri Aug 1 19:18:24 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 01 Aug 2008 16:18:24 -0700 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <3d375d730808011601n198943b1k30cc1423bf147bcb@mail.gmail.com> References: <4891EC2F.9050501@noaa.gov> <4891E996.5070000@ar.media.kyoto-u.ac.jp> <4891F276.8070404@noaa.gov> <5b8d13220807311128w56e1996bue1ec304aca672b47@mail.gmail.com> <48933768.2010503@noaa.gov> <5b8d13220808011246k33def1eeqc0c1ca316bcefc70@mail.gmail.com> <48938CB0.5010403@noaa.gov> <3d375d730808011528ic8dacfbm883def8d073b4228@mail.gmail.com> <4893931A.5050804@noaa.gov> <3d375d730808011601n198943b1k30cc1423bf147bcb@mail.gmail.com> Message-ID: <489399C0.2060809@noaa.gov> Robert Kern wrote: > File/Save As..., pick a file name. When asked about whether to embed > source files or strip them out, choose Strip. Then email the resulting > .mshark file to me. I've done that, and sent it to you directly -- it's too big to put in the mailing list. > It looks like your Python just takes a truly inordinate amount of time > to execute any code. Some of the problematic modules like httplib have > been moved to local imports, but the time it takes for your Python to > execute the code in that module is still ridiculously large. Can you > profile just importing httplib instead of numpy? I've got to go catch a bus now, and I don't have a Mac at home, so this will have to wait 'till next Monday -- thanks for all your time on this. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From cournape at gmail.com Fri Aug 1 23:45:56 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 2 Aug 2008 12:45:56 +0900 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <85b5c3130808011333hff71ae8j7f82741c2ecb58c4@mail.gmail.com> References: <846D821E-BA9F-4043-AE39-D7D648013C30@dalkescientific.com> <6B6B9FA6-864D-4D31-99BB-E9AE7A93A6D3@dalkescientific.com> <9457e7c80807301359k599b9155ge296ae804485c94@mail.gmail.com> <9457e7c80807310242q75d4a943wbdd1d4e2c264b613@mail.gmail.com> <3d375d730807310303q54ef94f7m3ba74b3f47f6e5ea@mail.gmail.com> <0BD87DFD-6B55-44E3-90EA-C1F83301F091@dalkescientific.com> <3d375d730807311302p46138c29k1cae6a5aa8899360@mail.gmail.com> <85b5c3130808011333hff71ae8j7f82741c2ecb58c4@mail.gmail.com> Message-ID: <5b8d13220808012045o60437930r1ee9d0edb410bab1@mail.gmail.com> On Sat, Aug 2, 2008 at 5:33 AM, Ondrej Certik wrote: > But I am still unhappy about it, I'd like if the package could import > much faster, because it adds up, when you need to import 7 packages > like that, it's suddenly 1s and that's just too much. Too much for what ? We need more information on the kind of things people who complaing about numpy startup cost are doing. I suggested lazy import a few weeks ago when this discussion started (with the example of bzr instead of hg), but I am less convinced that it would be that useful, because numpy is fundamentally different than bzr/hg. As robert said, it would bring some complexity, and in an area where python is already "fishy". When you import numpy, you expect some core things to be available, and they are the ones who take the most time. In bzr/hg, you use a *program*, and you can relatively easily change the API because not many people use it. But numpy is essentially an API, not a tool, so we don't have this freedom. Also, it means it is relatively easy for bzr/hg developers to control lazy import ,because they are the users, and users of bzr/hg don't deal with python directly. If our own lazy import has some bugs, it will impact many people who will not be able to trace it. The main advantage I see with lazy imports is that it avoids someone else from breaking the speed-up work by re-importing globally a costly package. > But of course everything within the constrains that Robert has > outlined. From the theoretical point of view, I don't understand why > python cannot just import numpy (or any other package) immediatelly, > and only at the moment the user actually access something, to import > it in real. I guess because it would be complex to do everywhere while keeping all the semantics of python import. Also, like everything "lazy", it means it is more complicated to follow what's happening. Your examples show that it would be complex to do. As I see it, there are some things in numpy we could do a bit differently to cut significantly import times (a few ten ), without changing much. Let's try that first. > Mercurial uses a lazy import module, that does exactly > this. Maybe that's an option? Note that mercurial is under the GPL :) cheers, David From aisaac at american.edu Sat Aug 2 01:03:40 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 2 Aug 2008 01:03:40 -0400 Subject: [Numpy-discussion] numpy 1.1.rc2: win32 binaries In-Reply-To: References: <488D58A3.6070800@ar.media.kyoto-u.ac.jp><488D5984.202@ar.media.kyoto-u.ac.jp> Message-ID: > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy-1.1.1.dev5559-win32-superpack-python2.5.exe Still dead. Cheers, Alan Isaac From david at ar.media.kyoto-u.ac.jp Sat Aug 2 01:06:18 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 02 Aug 2008 14:06:18 +0900 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <48939973.4010008@noaa.gov> References: <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <6B6B9FA6-864D-4D31-99BB-E9AE7A93A6D3@dalkescientific.com> <9457e7c80807301359k599b9155ge296ae804485c94@mail.gmail.com> <4891EC2F.9050501@noaa.gov> <4891E996.5070000@ar.media.kyoto-u.ac.jp> <4891F276.8070404@noaa.gov> <5b8d13220807311128w56e1996bue1ec304aca672b47@mail.gmail.com> <48933768.2010503@noaa.gov> <20080801165351.GC24781@phare.normalesup.org> <3d375d730808011249h29460aavd39f3b0f8874c361@mail.gmail.com> <48939835.4080801@noaa.gov> <48939973.4010008@noaa.gov> Message-ID: <4893EB4A.6050905@ar.media.kyoto-u.ac.jp> Christopher Barker wrote: > > OK, I just installed wxPython, and whoa! > > time python -c "import numpy" > > real 0m2.793s > user 0m0.294s > sys 0m2.494s > > so it's taking almost two seconds more to import numpy, now that > wxPython is installed. I haven't even imported it yet. importing wx > isn't as bad: > > $ time python -c "import wx" > > real 0m1.589s > user 0m0.274s > sys 0m1.000s > > Since numpy wo wx + wc import times adds up to numpy import times, this suggests that numpy may import wx. Which it shouldn't, obviously. There is something strange happening here. Please check wether wx really is imported when you do import numpy: python -c "import numpy; import sys; print sys.modules" And if it is, we have to know why it is imported at all when doing import numpy. cheers, David From david at ar.media.kyoto-u.ac.jp Sat Aug 2 01:09:02 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 02 Aug 2008 14:09:02 +0900 Subject: [Numpy-discussion] numpy 1.1.rc2: win32 binaries In-Reply-To: References: <488D58A3.6070800@ar.media.kyoto-u.ac.jp><488D5984.202@ar.media.kyoto-u.ac.jp> Message-ID: <4893EBEE.4030907@ar.media.kyoto-u.ac.jp> Alan G Isaac wrote: >> http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy-1.1.1.dev5559-win32-superpack-python2.5.exe >> Yes, sorry, it should read: http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.1.1.dev5559-win32-superpack-python2.5.exe cheers, David From robert.kern at gmail.com Sat Aug 2 02:13:20 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 2 Aug 2008 01:13:20 -0500 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <4893EB4A.6050905@ar.media.kyoto-u.ac.jp> References: <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <4891E996.5070000@ar.media.kyoto-u.ac.jp> <4891F276.8070404@noaa.gov> <5b8d13220807311128w56e1996bue1ec304aca672b47@mail.gmail.com> <48933768.2010503@noaa.gov> <20080801165351.GC24781@phare.normalesup.org> <3d375d730808011249h29460aavd39f3b0f8874c361@mail.gmail.com> <48939835.4080801@noaa.gov> <48939973.4010008@noaa.gov> <4893EB4A.6050905@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730808012313t66a96137l423856f9ed5f1448@mail.gmail.com> On Sat, Aug 2, 2008 at 00:06, David Cournapeau wrote: > Christopher Barker wrote: >> >> OK, I just installed wxPython, and whoa! >> >> time python -c "import numpy" >> >> real 0m2.793s >> user 0m0.294s >> sys 0m2.494s >> >> so it's taking almost two seconds more to import numpy, now that >> wxPython is installed. I haven't even imported it yet. importing wx >> isn't as bad: >> >> $ time python -c "import wx" >> >> real 0m1.589s >> user 0m0.274s >> sys 0m1.000s > > Since numpy wo wx + wc import times adds up to numpy import times, this > suggests that numpy may import wx. Which it shouldn't, obviously. There > is something strange happening here. Please check wether wx really is > imported when you do import numpy: > > python -c "import numpy; import sys; print sys.modules" > > And if it is, we have to know why it is imported at all when doing > import numpy. It isn't. The problem is on Chris's file system. Whatever is wrong with his file system (Bill Spotz's identical problem suggests too many temporary but unused inodes) increases the traversal of the file system. wx has a .pth file which adds entries to sys.path. Every time one tries to import something, the entries on sys.path are examined for the module. So increasing the number of entries on sys.path exacerbates the problem. But the problem really is his disk; it's not a problem with numpy or Python or anything else. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Sat Aug 2 02:16:03 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 02 Aug 2008 15:16:03 +0900 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <3d375d730808012313t66a96137l423856f9ed5f1448@mail.gmail.com> References: <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <4891E996.5070000@ar.media.kyoto-u.ac.jp> <4891F276.8070404@noaa.gov> <5b8d13220807311128w56e1996bue1ec304aca672b47@mail.gmail.com> <48933768.2010503@noaa.gov> <20080801165351.GC24781@phare.normalesup.org> <3d375d730808011249h29460aavd39f3b0f8874c361@mail.gmail.com> <48939835.4080801@noaa.gov> <48939973.4010008@noaa.gov> <4893EB4A.6050905@ar.media.kyoto-u.ac.jp> <3d375d730808012313t66a96137l423856f9ed5f1448@mail.gmail.com> Message-ID: <4893FBA3.2080805@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > > It isn't. The problem is on Chris's file system. Whatever is wrong > with his file system (Bill Spotz's identical problem suggests too many > temporary but unused inodes) increases the traversal of the file > system. Ah, I did not think it could indeed affect the whole fs. This seems much more likely, then. I guess I was confused because wx caused me some problems a long time ago, with scipy, and thought maybe there were some leftovers in Chris' system. It would also explain why import numpy is still kind of slow on his machine. I don't remember the numbers, but I think it was quicker on my PPC minimac (under Mac os X) than on his computer. > wx has a .pth file which adds entries to sys.path. Every time > one tries to import something, the entries on sys.path are examined > for the module. So increasing the number of entries on sys.path > exacerbates the problem. But the problem really is his disk; it's not > a problem with numpy or Python or anything else. > It was an fs problem, after all. I am a bit surprised this can happen in such an aggravated manner, though. cheers, David From dalke at dalkescientific.com Sat Aug 2 05:19:13 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Sat, 2 Aug 2008 11:19:13 +0200 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <09EF1D94-B252-45F0-9E73-EBDC57E3993E@dalkescientific.com> References: <92AFDF8C-418F-4B6D-895C-8E7F0FD47D90@dalkescientific.com> <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <846D821E-BA9F-4043-AE39-D7D648013C30@dalkescientific.com> <6B6B9FA6-864D-4D31-99BB-E9AE7A93A6D3@dalkescientific.com> <9457e7c80807301359k599b9155ge296ae804485c94@mail.gmail.com> <9457e7c80807310242q75d4a943wbdd1d4e2c264b613@mail.gmail.com> <3d375d730807310303q54ef94f7m3ba74b3f47f6e5ea@mail.gmail.com> <0BD87DFD-6B55-44E3-90EA-C1F83301F091@dalkescientific.com> <3d375d730807311302p46138c29k1cae6a5aa8899360@mail.gmail.com> <09EF1D94-B252-45F0-9E73-EBDC57E3993E@dalkescientific.com> Message-ID: <47A17C17-9F4D-4B5A-8E02-22BB66C05642@dalkescientific.com> I've got a proof of concept that take the time on my machine to "import numpy" from 0.21 seconds down to 0.08 seconds. Doing that required some somewhat awkward things, like deferring all 'import re' statements. I don't think that's stable in the long run because people will blithely import re in the future and not care that it takes 0.02 seconds to import. I don't blame them for complaining; I was curious on how fast I could get things. Note that when I started complaining about this a month ago the import time on my machine was about 0.3 seconds. I'll work on patches within the next couple of days. Here's an outline of what I did, along with some questions about what's feasible. 1) don't import 'numpy.testing'. Savings = 0.012s. Doing so required patches like -from numpy.testing import Tester -test = Tester().test -bench = Tester().bench +def test(label='fast', verbose=1, extra_argv=None, doctests=False, + coverage=False, **kwargs): + from testing import Tester + import numpy + Tester(numpy).test(label, verbose, extra_argv, doctests, + coverage, **kwargs) +def bench(label='fast', verbose=1, extra_argv=None): + from testing import Tester + import numpy + Tester(numpy).bench(label, verbose, extra_argv) QUESTION: since numpy is moving to nose, and the documentation only describes doing 'import numpy; numpy.test()', can I remove all other definitions of "test" and "bench"? 2) removing 'import ctypeslib' in top-level -> 0.023 seconds QUESTION: is this considered part of the API that must be preserved? The primary use case is supposed to be to help interactive users. I don't think interactive users spend much time using ctypes, and those that do are also those that aren't confused about needing an extra import statement. 3) removing 'import string' in numerictypes.py -> 0.008 seconds . This requires some ugly but simple changes to the code. 4) remove the 'import re' in _internal, numpy/lib/, function_base, and other places. This reduced my overall startup cost by 0.013. 5) defer bzip and gzip imports in _datasource: 0.009 s. This will require non-trivial code changes. 6) defer 'format' from io.py: 0.007 s 7) _datasource imports shutil in order to use shutil.rmdir in a __del__. I don't think this can be deferred, because I don't want to do an import during system shutdown, which is when the __del__ might be called. It would save 0.004s. 8) If I can remove 'import doc' from the top-level numpy (is that part of the required API?) then I can save 0.004s. 9) defer urlparse in _datasource: about 0.003s 10) If I get rid of the cPickle top-level numeric.py then I can save 0.006 seconds. 11) not importing add_newdocs saves 0.005 s. This might be possible by moving all of the docstrings to the actual functions. I haven't looked into this much and it might not be possible. Those millisecond improvements add up! When I do an interactive 'import numpy' on my system I don't notice the import time like I did before. Andrew dalke at dalkescientific.com Andrew dalke at dalkescientific.com From nadavh at visionsense.com Sat Aug 2 09:15:51 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Sat, 2 Aug 2008 16:15:51 +0300 Subject: [Numpy-discussion] Bilteral filter Message-ID: <710F2847B0018641891D9A216027636029C1F4@ex3.envision.co.il> Hi all, Is there any efficient implementation of bilateral filter that works smoothly with numpy? Nadav. From zachary.pincus at yale.edu Sat Aug 2 12:14:19 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Sat, 2 Aug 2008 10:14:19 -0600 Subject: [Numpy-discussion] Bilteral filter In-Reply-To: <710F2847B0018641891D9A216027636029C1F4@ex3.envision.co.il> References: <710F2847B0018641891D9A216027636029C1F4@ex3.envision.co.il> Message-ID: > Is there any efficient implementation of bilateral filter that works > smoothly with numpy? Not that I know of... Of course, if you were to write one, I'm sure there would be some interest in it! I would recommend looking into the tools in scipy.ndimage for basic image-processing support, but there's no bilateral filtering there. Zach From millman at berkeley.edu Sat Aug 2 16:27:55 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Sat, 2 Aug 2008 13:27:55 -0700 Subject: [Numpy-discussion] ANN: NumPy 1.1.1 Message-ID: I'm pleased to announce the release of NumPy 1.1.1. NumPy is the fundamental package needed for scientific computing with Python. It contains: * a powerful N-dimensional array object * sophisticated (broadcasting) functions * basic linear algebra functions * basic Fourier transforms * sophisticated random number capabilities * tools for integrating Fortran code. Besides it's obvious scientific uses, NumPy can also be used as an efficient multi-dimensional container of generic data. Arbitrary data-types can be defined. This allows NumPy to seamlessly and speedily integrate with a wide-variety of databases. Numpy 1.1.1 is a bug fix release featuring major improvements in Python 2.3.x compatibility and masked arrays. For information, please see the release notes: http://sourceforge.net/project/shownotes.php?group_id=1369&release_id=617279 Thank you to everybody who contributed to this release. Enjoy, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From stefan at sun.ac.za Sat Aug 2 17:42:00 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sat, 2 Aug 2008 23:42:00 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.1.1 In-Reply-To: References: Message-ID: <9457e7c80808021442g77a37151pf0330585b2c02340@mail.gmail.com> 2008/8/2 Jarrod Millman : > I'm pleased to announce the release of NumPy 1.1.1. A big thank you to Charles and Jarrod for all the time they put into this release! Cheers St?fan From millman at berkeley.edu Sat Aug 2 17:47:19 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Sat, 2 Aug 2008 14:47:19 -0700 Subject: [Numpy-discussion] ANN: NumPy 1.1.1 In-Reply-To: <9457e7c80808021442g77a37151pf0330585b2c02340@mail.gmail.com> References: <9457e7c80808021442g77a37151pf0330585b2c02340@mail.gmail.com> Message-ID: On Sat, Aug 2, 2008 at 2:42 PM, St?fan van der Walt wrote: > 2008/8/2 Jarrod Millman : >> I'm pleased to announce the release of NumPy 1.1.1. > > A big thank you to Charles and Jarrod for all the time they put into > this release! And, of course, a big thanks to David Cournapeau and Chris Burns. Now it is time for us to start focusing on NumPy 1.2 and SciPy 0.7, which are both set to be released at the end of the SciPy conference. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From millman at berkeley.edu Sat Aug 2 20:36:24 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Sat, 2 Aug 2008 17:36:24 -0700 Subject: [Numpy-discussion] numpy 1.1.rc2: win32 binaries In-Reply-To: References: <488D58A3.6070800@ar.media.kyoto-u.ac.jp> <488D5984.202@ar.media.kyoto-u.ac.jp> Message-ID: On Fri, Aug 1, 2008 at 10:03 PM, Alan G Isaac wrote: >> http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy-1.1.1.dev5559-win32-superpack-python2.5.exe You can get the official binaries from here: http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103&release_id=617279 Please let us know if they work for you or not. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From nadavh at visionsense.com Sun Aug 3 05:26:10 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Sun, 3 Aug 2008 12:26:10 +0300 Subject: [Numpy-discussion] Bilteral filter References: <710F2847B0018641891D9A216027636029C1F4@ex3.envision.co.il> Message-ID: <710F2847B0018641891D9A216027636029C1F8@ex3.envision.co.il> My main disappointment from python is that I could not donate code to its libraries (beside 3 small functions) because others made it before me. I hoped that I would have the same luck this time ;) I have a prototype in cython that I am experimenting with. I'll publish it within few days unless someone would address me to an existing code. Nadav. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Zachary Pincus ????: ? 02-??????-08 19:14 ??: Discussion of Numerical Python ????: Re: [Numpy-discussion] Bilteral filter > Is there any efficient implementation of bilateral filter that works > smoothly with numpy? Not that I know of... Of course, if you were to write one, I'm sure there would be some interest in it! I would recommend looking into the tools in scipy.ndimage for basic image-processing support, but there's no bilateral filtering there. Zach _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3251 bytes Desc: not available URL: From david at ar.media.kyoto-u.ac.jp Sun Aug 3 06:25:43 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sun, 03 Aug 2008 19:25:43 +0900 Subject: [Numpy-discussion] Toward a numpy build with warning: handling unused variable Message-ID: <489587A7.7010108@ar.media.kyoto-u.ac.jp> Hi, I have recently tried various warning flags with numpy, and already discover several potential errors in the C code. Now, because of the huge amount of warning we get with e.g. -W -Wall, it would be nice to clean the code for this kind of warning 'level' because we have a huge amount of garbage right now at this level. One big proportion of warning with -W is the unused variable, which happens with the dummy/self variable in the typical C functions. I know of two ways to handle this: - the simple: use the variable in some way, e.g. Add a (void)dummy; in the function code. Problem: it is not really nice, and if later you want to use the variable, it may hide other useful warnings, although I am not sure about the later. - more sophisticated way: using compiler specific handling (gcc attribute) + mangling: http://sourcefrog.net/weblog/software/languages/C/unused.html It is more complicated, is compiler specific, but it prevents accidental use of the variable (thanks to the mangling). The compiler-specific is not that much of a problem IMHO, since this is for warnings (and most of use use gcc ?). What do people think ? If nobody has anything against it, I will prepare a patch to support this. cheers, David From bolme1234 at comcast.net Sun Aug 3 11:19:03 2008 From: bolme1234 at comcast.net (David Bolme) Date: Sun, 3 Aug 2008 09:19:03 -0600 Subject: [Numpy-discussion] Bilteral filter In-Reply-To: <710F2847B0018641891D9A216027636029C1F8@ex3.envision.co.il> References: <710F2847B0018641891D9A216027636029C1F4@ex3.envision.co.il> <710F2847B0018641891D9A216027636029C1F8@ex3.envision.co.il> Message-ID: <2914DB5E-D83E-4DA3-8469-8D69C70E41DA@comcast.net> I am building up a python Computer Vision / Image Processing library built on PIL/Scipy. If you cannot find a home for this code in scipy I think it would be a good fit for this library. http://pyvision.sourceforge.net Dave On Aug 3, 2008, at 3:26 AM, Nadav Horesh wrote: > My main disappointment from python is that I could not donate code > to its libraries (beside 3 small functions) because others made it > before me. I hoped that I would have the same luck this time ;) I > have a prototype in cython that I am experimenting with. I'll > publish it within few days unless someone would address me to an > existing code. > > Nadav. > > -----????? ??????----- > ???: numpy-discussion-bounces at scipy.org ??? Zachary Pincus > ????: ? 02-??????-08 19:14 > ??: Discussion of Numerical Python > ????: Re: [Numpy-discussion] Bilteral filter > >> Is there any efficient implementation of bilateral filter that works >> smoothly with numpy? > > Not that I know of... > > Of course, if you were to write one, I'm sure there would be some > interest in it! I would recommend looking into the tools in > scipy.ndimage for basic image-processing support, but there's no > bilateral filtering there. > > Zach > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Sun Aug 3 11:35:46 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 3 Aug 2008 10:35:46 -0500 Subject: [Numpy-discussion] Toward a numpy build with warning: handling unused variable In-Reply-To: <489587A7.7010108@ar.media.kyoto-u.ac.jp> References: <489587A7.7010108@ar.media.kyoto-u.ac.jp> Message-ID: On Sun, Aug 3, 2008 at 5:25 AM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Hi, > > I have recently tried various warning flags with numpy, and already > discover > several potential errors in the C code. Now, because of the huge amount of > warning > we get with e.g. -W -Wall, it would be nice to clean the code for this > kind of warning 'level' because we have a huge amount of garbage right now > at this level. > One big proportion of warning with -W is the unused variable, which happens > with the dummy/self variable in the > typical C functions. > > I know of two ways to handle this: > - the simple: use the variable in some way, e.g. Add a (void)dummy; > in the function code. Problem: it is not really nice, and if later you > want to use the variable, it may hide other useful warnings, although I > am not sure about the later. > - more sophisticated way: using compiler specific handling (gcc > attribute) + mangling: > > http://sourcefrog.net/weblog/software/languages/C/unused.html > > It is more complicated, is compiler specific, but it prevents accidental > use of the variable (thanks to the mangling). The compiler-specific is > not that much of a problem IMHO, since this is for warnings (and most of > use use gcc ?). > > What do people think ? If nobody has anything against it, I will prepare a > patch to support this. > I say leave these warnings alone. If nothing else, they point to possible cleanups in some future refactoring. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From novak at ucolick.org Sun Aug 3 12:42:19 2008 From: novak at ucolick.org (Greg Novak) Date: Sun, 3 Aug 2008 09:42:19 -0700 Subject: [Numpy-discussion] member1d and unique elements Message-ID: I have two arrays of integers, and would like to know _where_ they have elements in common, not just _which_ elements are in common. This is because the entries in the integer array are aligned with other arrays. This seems very close to what member1d advertises as its function. However, member1d says that it expects arrays with only unique elements. First of all, my desired operation is well-posed: I'd like f(ar1, ar2) to return something in the shape of ar1 with True if the value at that position appears anywhere in ar2 (regardless of duplication) and False otherwise. So I looked at the code and have two questions: 1) What is this code trying to achieve? aux = perm[ii+1] perm[ii+1] = perm[ii] perm[ii] = aux Here perm is the stable argsort of the two concatenated arguments: perm = concatenate((ar1, ar2)).argsort(kind='mergesort'). arr is the array of combined inputs in sorted order: arr = concatenate((ar1, ar2))[perm] and ii is a list of indices into arr where the value of arr is equal to the next value in the array (arr[ii] == arr[ii+1]) _and_ arr[ii] came from the _second_ input (ar2). Now, this last bit (looking for elements of arr that are equal and both came from the second array) is clearly trying to deal with duplication, which is why I'm interested... So, the code snippet is trying to swap perm[ii+1] with perm[ii], but I don't see why. Furthermore, there are funny results if a value is duplicated three times, not just twice -- perm is no longer a permutation vector. Eg, member1d([1], [2,2,2]) results perm=[0,1,2,3] and ii=[1,2] before the above snippet, and the above snippet makes perm into [0,2,3,2] I've commented those three lines, and I've never seen any changes to the output of member1d. The new value of perm is used to compute the expression: perm.argsort(kind='mergesort')[:len( ar1 )], but the changes to that expression as a result of the above three lines are always at the high end of the array, which is sliced off by the last [:len(ar1)]. Finally, my second question is: 2) Does anyone have a test case where member1d fails as a result of duplicates in the input? So far I haven't found any, with the above lines commented or not. Upon reflection and review of the changelog, another theory occurs to me: member1d did not originally use a stable sort. What I've written above for interpretation of the value ii (indicates duplication within ar2) is true for a stable sort, but for an unstable sort the same condition has the interpretation that ii holds the values where the sorting algorithm swapped the order of equal values unstably. Then the code snippet in question 1) looks like an attempt to swap those values in the permutation array to make the sort stable again. The attempt would fail if there was duplication in either array. So, I would propose deleting those three lines (since they seem to be a non-functional relic) and declaring in the docstring that member1d doesn't require unique elements. Also, if this is correct, then the function simplifies considerably since several values don't need to be computed anymore: def setmember1d( ar1, ar2 ): ar = nm.concatenate( (ar1, ar2 ) ) perm = ar.argsort(kind='mergesort') aux = ar[perm] flag = nm.concatenate( (aux[1:] == aux[:-1], [False] ) ) indx = perm.argsort(kind='mergesort')[:len( ar1 )] return flag[indx] Corrections to the above are welcome since I'm going to start using member1d without regard for uniqueness, and I'd like to know if I'm making a big mistake... Thanks, Greg From stefan at sun.ac.za Sun Aug 3 13:16:42 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 3 Aug 2008 19:16:42 +0200 Subject: [Numpy-discussion] member1d and unique elements In-Reply-To: References: Message-ID: <9457e7c80808031016j7403a347k79280644c07e481a@mail.gmail.com> 2008/8/3 Greg Novak : > First of all, my desired operation is well-posed: I'd like f(ar1, > ar2) to return something in the shape of ar1 with True if the value at > that position appears anywhere in ar2 (regardless of duplication) and > False otherwise. Just because one-liners are so irresistable: >>> x array([1, 1, 2, 5, 7]) >>> x2 array([1, 7]) >>> np.logical_or.reduce(x[:,None] == x2, axis=1) array([ True, True, False, False, True], dtype=bool) >>> (x[:,None] == x2).sum(axis=1) > 0 array([ True, True, False, False, True], dtype=bool) Creates an MnX temporary, but oh well. Thanks for taking a look at member1d. I'm swamped, so I cannot take part in the conversation, but I hope someone takes note. If no one does, please file a ticket so that we don't lose track of this conversation. Cheers St?fan From gael.varoquaux at normalesup.org Sun Aug 3 13:18:09 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 3 Aug 2008 19:18:09 +0200 Subject: [Numpy-discussion] member1d and unique elements In-Reply-To: <9457e7c80808031016j7403a347k79280644c07e481a@mail.gmail.com> References: <9457e7c80808031016j7403a347k79280644c07e481a@mail.gmail.com> Message-ID: <20080803171809.GA13589@phare.normalesup.org> On Sun, Aug 03, 2008 at 07:16:42PM +0200, St?fan van der Walt wrote: > 2008/8/3 Greg Novak : > > First of all, my desired operation is well-posed: I'd like f(ar1, > > ar2) to return something in the shape of ar1 with True if the value at > > that position appears anywhere in ar2 (regardless of duplication) and > > False otherwise. > Just because one-liners are so irresistable: Damn, a perl programmer has sneeked in this mailing-list. Please call security. Ga?l From pgmdevlist at gmail.com Sun Aug 3 14:29:23 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Sun, 3 Aug 2008 14:29:23 -0400 Subject: [Numpy-discussion] Renaming MaskedArray._basedict Message-ID: <200808031429.24311.pgmdevlist@gmail.com> All, I intend to rename a little-known attribute of MaskedArray, _basedict, to something more understandable such as _optinfo. This attribute is a dictionary that comes handy when subclassing MaskedArray or to save information. _basedict would still be available for a while as an alias to _optinfo, but may disappear in the long run. I intend to put a DeprecationWarning at the beginning of numpy.ma.core. A side-effect is that this warning is issued each time numpy.ma is imported, which clutters the output of the tests. Am I missing something ? P. From wnbell at gmail.com Sun Aug 3 16:31:33 2008 From: wnbell at gmail.com (Nathan Bell) Date: Sun, 3 Aug 2008 15:31:33 -0500 Subject: [Numpy-discussion] Renaming MaskedArray._basedict In-Reply-To: <200808031429.24311.pgmdevlist@gmail.com> References: <200808031429.24311.pgmdevlist@gmail.com> Message-ID: On Sun, Aug 3, 2008 at 1:29 PM, Pierre GM wrote: > > I intend to put a DeprecationWarning at the beginning of numpy.ma.core. A > side-effect is that this warning is issued each time numpy.ma is imported, > which clutters the output of the tests. > Am I missing something ? Can you could trap it in __getattr_ instead? For instance: http://projects.scipy.org/scipy/scipy/browser/trunk/scipy/sparse/csr.py#L87 -- Nathan Bell wnbell at gmail.com http://graphics.cs.uiuc.edu/~wnbell/ From pgmdevlist at gmail.com Sun Aug 3 17:50:37 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Sun, 3 Aug 2008 17:50:37 -0400 Subject: [Numpy-discussion] Renaming MaskedArray._basedict In-Reply-To: References: <200808031429.24311.pgmdevlist@gmail.com> Message-ID: <200808031750.38078.pgmdevlist@gmail.com> On Sunday 03 August 2008 16:31:33 Nathan Bell wrote: > On Sun, Aug 3, 2008 at 1:29 PM, Pierre GM wrote: > Can you could trap it in __getattr_ instead? For instance: > http://projects.scipy.org/scipy/scipy/browser/trunk/scipy/sparse/csr.py#L87 I could but I don't want to. MaskedArray should stay as low-level as possible, and overwriting __getattr__ would slow things down. Besides, it works quite well as it is, and leaves the user the possibility to modify it so that elements of _basedict/_optinfo can be accessed as properties for additional checks. From cournape at gmail.com Sun Aug 3 19:02:36 2008 From: cournape at gmail.com (David Cournapeau) Date: Mon, 4 Aug 2008 08:02:36 +0900 Subject: [Numpy-discussion] Toward a numpy build with warning: handling unused variable In-Reply-To: References: <489587A7.7010108@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220808031602x567d4855i6b961e116ff69a5b@mail.gmail.com> On Mon, Aug 4, 2008 at 12:35 AM, Charles R Harris wrote: > > I say leave these warnings alone. If nothing else, they point to possible > cleanups in some future refactoring. Not really, because they are inherent to the way the Python C API works (the first argument of any python C function, which reference to self for methods). Also, having hundred of bogus warnings mean you never see the interesting ones (building numpy core alone with -W generates > 1500 such bogus warnings...) cheers, David From stefan at sun.ac.za Sun Aug 3 19:14:36 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 4 Aug 2008 01:14:36 +0200 Subject: [Numpy-discussion] Toward a numpy build with warning: handling unused variable In-Reply-To: <5b8d13220808031602x567d4855i6b961e116ff69a5b@mail.gmail.com> References: <489587A7.7010108@ar.media.kyoto-u.ac.jp> <5b8d13220808031602x567d4855i6b961e116ff69a5b@mail.gmail.com> Message-ID: <9457e7c80808031614m70426e2emc19ae339026aaf2@mail.gmail.com> Hi David 2008/8/4 David Cournapeau : > Also, having hundred of bogus warnings mean you never see the > interesting ones (building numpy core alone with -W generates > 1500 > such bogus warnings...) I agree -- seeing the NumPy-related build warnings would be very useful. Do you know how to address these? numpy/core/src/arrayobject.c:3092: warning: format '%d' expects type 'int', but argument 4 has type 'long int' Regards St?fan From efiring at hawaii.edu Sun Aug 3 18:26:22 2008 From: efiring at hawaii.edu (Eric Firing) Date: Sun, 03 Aug 2008 12:26:22 -1000 Subject: [Numpy-discussion] Renaming MaskedArray._basedict In-Reply-To: <200808031429.24311.pgmdevlist@gmail.com> References: <200808031429.24311.pgmdevlist@gmail.com> Message-ID: <4896308E.1000104@hawaii.edu> Pierre GM wrote: > All, > I intend to rename a little-known attribute of MaskedArray, _basedict, to > something more understandable such as _optinfo. This attribute is a > dictionary that comes handy when subclassing MaskedArray or to save > information. > _basedict would still be available for a while as an alias to _optinfo, but > may disappear in the long run. > I intend to put a DeprecationWarning at the beginning of numpy.ma.core. A > side-effect is that this warning is issued each time numpy.ma is imported, > which clutters the output of the tests. > Am I missing something ? Pierre, The attribute name, _basedict, indicates that it is a private attribute, so I don't see why a deprecation warning is needed at all. Anyone using private attributes should also be reading release notes. I don't like the idea of a warning popping up upon each import of numpy.ma. Eric From cournape at gmail.com Sun Aug 3 19:30:50 2008 From: cournape at gmail.com (David Cournapeau) Date: Mon, 4 Aug 2008 08:30:50 +0900 Subject: [Numpy-discussion] Toward a numpy build with warning: handling unused variable In-Reply-To: <9457e7c80808031614m70426e2emc19ae339026aaf2@mail.gmail.com> References: <489587A7.7010108@ar.media.kyoto-u.ac.jp> <5b8d13220808031602x567d4855i6b961e116ff69a5b@mail.gmail.com> <9457e7c80808031614m70426e2emc19ae339026aaf2@mail.gmail.com> Message-ID: <5b8d13220808031630k49ee7aa9u11499a355369df14@mail.gmail.com> On Mon, Aug 4, 2008 at 8:14 AM, St?fan van der Walt wrote: > > I agree -- seeing the NumPy-related build warnings would be very useful. I did not anticipate people would disagree on the idea of removing warnings, so I guess thare are two questions: - why / why not removing bogus warnings - if removing is considered OK, what's the best method: with the compiler-specific method, it could be done automatically and safely with some regex. > Do you know how to address these? > > numpy/core/src/arrayobject.c:3092: warning: format '%d' expects type > 'int', but argument 4 has type 'long int' Yes, at least on some platforms: this is addressed by using the inttypes.h header, proof of concept here: http://projects.scipy.org/scipy/numpy/ticket/847 But it would need a configuration stage to test we have the required formats in inttypes (the patch relies on how configuration is done in the python header, which I don't like and is not really robust). cheers, David From charlesr.harris at gmail.com Sun Aug 3 19:31:20 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 3 Aug 2008 18:31:20 -0500 Subject: [Numpy-discussion] Toward a numpy build with warning: handling unused variable In-Reply-To: <5b8d13220808031602x567d4855i6b961e116ff69a5b@mail.gmail.com> References: <489587A7.7010108@ar.media.kyoto-u.ac.jp> <5b8d13220808031602x567d4855i6b961e116ff69a5b@mail.gmail.com> Message-ID: On Sun, Aug 3, 2008 at 6:02 PM, David Cournapeau wrote: > On Mon, Aug 4, 2008 at 12:35 AM, Charles R Harris > wrote: > > > > > I say leave these warnings alone. If nothing else, they point to possible > > cleanups in some future refactoring. > > Not really, because they are inherent to the way the Python C API > works (the first argument of any python C function, which reference to > self for methods). > But how many methods do arrays and types have? As many as 1500? I think it is a bad idea to work around these sort of things. If self is needed, it belongs there. If not, it should be removed. If the warnings are too numerous, filter them. And why would self go unused in any genuine method? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sun Aug 3 19:39:37 2008 From: cournape at gmail.com (David Cournapeau) Date: Mon, 4 Aug 2008 08:39:37 +0900 Subject: [Numpy-discussion] Toward a numpy build with warning: handling unused variable In-Reply-To: References: <489587A7.7010108@ar.media.kyoto-u.ac.jp> <5b8d13220808031602x567d4855i6b961e116ff69a5b@mail.gmail.com> Message-ID: <5b8d13220808031639me4cd4bak319638fafb45bbac@mail.gmail.com> On Mon, Aug 4, 2008 at 8:31 AM, Charles R Harris wrote: > > But how many methods do arrays and types have? As many as 1500? With autogenerated code, it is not difficult to imagine so many warnings. >I think it > is a bad idea to work around these sort of things. If self is needed, it > belongs there. If not, it should be removed. But that's the point: they can't be removed. The python C api force you for any function to have two arguments: static PyObject * spam_system(PyObject *self, PyObject *args) Many warnings are "warning: unused argument unused", or "warning: dummy argument unused" :) You can't remove them form the argument list. > If the warnings are too > numerous, filter them. If you filter them, what's the point of keeping them ? > And why would self go unused in any genuine method? Because at the C level, both methods and functions have exactly the signature, so for a function, you have an unused argument. AFAIK, most C functions in numpy are functions, not methods. As I said, it is inherent to the way C python functions are defined. Again, using the mangling method would mean that any attempt to use the argument would generate an error from the compiler, so you can't hide anything. cheers, David From stefan at sun.ac.za Sun Aug 3 19:50:36 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 4 Aug 2008 01:50:36 +0200 Subject: [Numpy-discussion] Toward a numpy build with warning: handling unused variable In-Reply-To: <5b8d13220808031630k49ee7aa9u11499a355369df14@mail.gmail.com> References: <489587A7.7010108@ar.media.kyoto-u.ac.jp> <5b8d13220808031602x567d4855i6b961e116ff69a5b@mail.gmail.com> <9457e7c80808031614m70426e2emc19ae339026aaf2@mail.gmail.com> <5b8d13220808031630k49ee7aa9u11499a355369df14@mail.gmail.com> Message-ID: <9457e7c80808031650j25c86125tfd5b1c4d46eda95e@mail.gmail.com> 2008/8/4 David Cournapeau : > On Mon, Aug 4, 2008 at 8:14 AM, St?fan van der Walt wrote: > >> >> I agree -- seeing the NumPy-related build warnings would be very useful. > > I did not anticipate people would disagree on the idea of removing > warnings, so I guess thare are two questions: > - why / why not removing bogus warnings > - if removing is considered OK, what's the best method: with the > compiler-specific method, it could be done automatically and safely > with some regex. Just to be clear: I meant that I agree with you that we should see NumPy related warnings that aren't smothered in bogus messages. Is there no way to do this without compiler-specific hacks? St?fan From charlesr.harris at gmail.com Sun Aug 3 20:32:07 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 3 Aug 2008 19:32:07 -0500 Subject: [Numpy-discussion] Toward a numpy build with warning: handling unused variable In-Reply-To: <5b8d13220808031639me4cd4bak319638fafb45bbac@mail.gmail.com> References: <489587A7.7010108@ar.media.kyoto-u.ac.jp> <5b8d13220808031602x567d4855i6b961e116ff69a5b@mail.gmail.com> <5b8d13220808031639me4cd4bak319638fafb45bbac@mail.gmail.com> Message-ID: On Sun, Aug 3, 2008 at 6:39 PM, David Cournapeau wrote: > On Mon, Aug 4, 2008 at 8:31 AM, Charles R Harris > wrote: > > > > But how many methods do arrays and types have? As many as 1500? > > With autogenerated code, it is not difficult to imagine so many warnings. > > >I think it > > is a bad idea to work around these sort of things. If self is needed, it > > belongs there. If not, it should be removed. > > But that's the point: they can't be removed. The python C api force > you for any function to have two arguments: > > static PyObject * > spam_system(PyObject *self, PyObject *args) > > Many warnings are "warning: unused argument unused", or "warning: > dummy argument unused" :) You can't remove them form the argument > list. > There is self and args, filter them out. If the arguments have other names, rename them. > > > If the warnings are too > > numerous, filter them. > > If you filter them, what's the point of keeping them ? > Because the compiler's inadequate determination of errors is driving the coding. That seems wrong way around. Maybe these variables can be notated somehow, i.e, __unused__(arg), which would at least document intent. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournapeau at cslab.kecl.ntt.co.jp Sun Aug 3 21:42:29 2008 From: cournapeau at cslab.kecl.ntt.co.jp (David Cournapeau) Date: Mon, 04 Aug 2008 10:42:29 +0900 Subject: [Numpy-discussion] Toward a numpy build with warning: handling unused variable In-Reply-To: <9457e7c80808031650j25c86125tfd5b1c4d46eda95e@mail.gmail.com> References: <489587A7.7010108@ar.media.kyoto-u.ac.jp> <5b8d13220808031602x567d4855i6b961e116ff69a5b@mail.gmail.com> <9457e7c80808031614m70426e2emc19ae339026aaf2@mail.gmail.com> <5b8d13220808031630k49ee7aa9u11499a355369df14@mail.gmail.com> <9457e7c80808031650j25c86125tfd5b1c4d46eda95e@mail.gmail.com> Message-ID: <1217814149.12930.20.camel@bbc8> On Mon, 2008-08-04 at 01:50 +0200, St?fan van der Walt wrote: > > Just to be clear: I meant that I agree with you that we should see > NumPy related warnings that aren't smothered in bogus messages. Is > there no way to do this without compiler-specific hacks? Yes, there is, but it is more error prone (because then you are hiding the warning instead of saying it is unused with a safeguard against any use). I would not call this an hack, because __attribute__ is a well known, well used gcc mechanism. For people not using gcc, it would not change anything. Maybe there are another more portable methods I am not aware of, though. cheers, David From cournapeau at cslab.kecl.ntt.co.jp Sun Aug 3 21:39:08 2008 From: cournapeau at cslab.kecl.ntt.co.jp (David Cournapeau) Date: Mon, 04 Aug 2008 10:39:08 +0900 Subject: [Numpy-discussion] Toward a numpy build with warning: handling unused variable In-Reply-To: References: <489587A7.7010108@ar.media.kyoto-u.ac.jp> <5b8d13220808031602x567d4855i6b961e116ff69a5b@mail.gmail.com> <5b8d13220808031639me4cd4bak319638fafb45bbac@mail.gmail.com> Message-ID: <1217813948.12930.15.camel@bbc8> On Sun, 2008-08-03 at 19:32 -0500, Charles R Harris wrote: > > > > There is self and args, filter them out. If the arguments have other > names, rename them. That's much more error prone. You would need a list of such arguments, and more importantly, it means you cannot enable warning by default (because you cannot expect the casual user to filter them out). And if you filter them out, you won't see them, meaning you hide them anyway. So I don't really see the point. > > > > If the warnings are too > > numerous, filter them. > > > If you filter them, what's the point of keeping them ? > > > > Because the compiler's inadequate determination of errors is driving > the coding. I don't understand what you mean here ? The problem I see is the following: currently, there are so many warnings that we don't enable them, and I would guess nobody currently enable them. > That seems wrong way around. Maybe these variables can be notated > somehow, i.e, __unused__(arg), which would at least document intent. Which is what I have been arguing all along :) If you look at the link I posted before, it suggests a UNUSED macro which uses gcc attribute when build with gcc, and do nothing otherwise: #ifdef __GNUCC__ #define NPY_UNUSED(x) NPY_UNUSED__ ## x __attribute__((unused)) __NPY_MANGLE_UNUSED) #else #define NPY_UNUSED(x) x #endif This makes it hard to get it wrong: - you can't use the variable by accident later: you will get a compilation error thanks to the mangling - you don't have meaningless warnings, meaning you see the real ones. - if you don't use gcc, it is exactly as before. I realize I may have not been not really clear: I am not suggesting to remove any warning about unused variable, just the ones which we cannot do anything about because they are in the argument list and we know for sure we will not use them. That include python C functions not used as a method (dummy/unused, e.g. the first argument), ufunc which do not use their func argument, etc... cheers, David From nadavh at visionsense.com Sun Aug 3 23:56:41 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Mon, 4 Aug 2008 06:56:41 +0300 Subject: [Numpy-discussion] Bilteral filter References: <710F2847B0018641891D9A216027636029C1F4@ex3.envision.co.il><710F2847B0018641891D9A216027636029C1F8@ex3.envision.co.il> <2914DB5E-D83E-4DA3-8469-8D69C70E41DA@comcast.net> Message-ID: <710F2847B0018641891D9A216027636029C1FC@ex3.envision.co.il> I'll post the code in this mailing list, and hope it would find a good home. Nadav. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? David Bolme ????: ? 03-??????-08 18:19 ??: Discussion of Numerical Python ????: Re: [Numpy-discussion] Bilteral filter I am building up a python Computer Vision / Image Processing library built on PIL/Scipy. If you cannot find a home for this code in scipy I think it would be a good fit for this library. http://pyvision.sourceforge.net Dave On Aug 3, 2008, at 3:26 AM, Nadav Horesh wrote: > My main disappointment from python is that I could not donate code > to its libraries (beside 3 small functions) because others made it > before me. I hoped that I would have the same luck this time ;) I > have a prototype in cython that I am experimenting with. I'll > publish it within few days unless someone would address me to an > existing code. > > Nadav. > > -----????? ??????----- > ???: numpy-discussion-bounces at scipy.org ??? Zachary Pincus > ????: ? 02-??????-08 19:14 > ??: Discussion of Numerical Python > ????: Re: [Numpy-discussion] Bilteral filter > >> Is there any efficient implementation of bilateral filter that works >> smoothly with numpy? > > Not that I know of... > > Of course, if you were to write one, I'm sure there would be some > interest in it! I would recommend looking into the tools in > scipy.ndimage for basic image-processing support, but there's no > bilateral filtering there. > > Zach > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3803 bytes Desc: not available URL: From pgmdevlist at gmail.com Mon Aug 4 00:03:20 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 4 Aug 2008 00:03:20 -0400 Subject: [Numpy-discussion] Renaming MaskedArray._basedict In-Reply-To: <4896308E.1000104@hawaii.edu> References: <200808031429.24311.pgmdevlist@gmail.com> <4896308E.1000104@hawaii.edu> Message-ID: <200808040003.21823.pgmdevlist@gmail.com> Eric, Good point. Off we go with the warning then. > > The attribute name, _basedict, indicates that it is a private attribute, > so I don't see why a deprecation warning is needed at all. Anyone using > private attributes should also be reading release notes. I don't like > the idea of a warning popping up upon each import of numpy.ma. > > Eric > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From thrabe at burnham.org Mon Aug 4 00:37:59 2008 From: thrabe at burnham.org (Thomas Hrabe) Date: Sun, 3 Aug 2008 21:37:59 -0700 Subject: [Numpy-discussion] arrays : c aligmen to fortran and back Message-ID: Hi all, I need to convert a 3d array from c alingment to fortran and was wandering for the simplest code available. Its all about python and an interfaced C++ program, which, however, processes fortran aligned arrays. Is there a simple code from converting an array arr = numpy.array((30,20,10)) to an array arr2 with the same shape (arr.shape == arr2.shape) = true where the data is aligned in fortran style. Furthermore, has anybody ever seen a C library which converts arrays from C alignment to Fortran without swapping the dimensions ([30,20,10] to [10,20,30]). Thank you in advance for your help, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From dagss at student.matnat.uio.no Mon Aug 4 04:26:30 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 04 Aug 2008 10:26:30 +0200 Subject: [Numpy-discussion] Toward a numpy build with warning: handling unused variable In-Reply-To: <1217814149.12930.20.camel@bbc8> References: <489587A7.7010108@ar.media.kyoto-u.ac.jp> <5b8d13220808031602x567d4855i6b961e116ff69a5b@mail.gmail.com> <9457e7c80808031614m70426e2emc19ae339026aaf2@mail.gmail.com> <5b8d13220808031630k49ee7aa9u11499a355369df14@mail.gmail.com> <9457e7c80808031650j25c86125tfd5b1c4d46eda95e@mail.gmail.com> <1217814149.12930.20.camel@bbc8> Message-ID: <4896BD36.6060603@student.matnat.uio.no> David Cournapeau wrote: > On Mon, 2008-08-04 at 01:50 +0200, St?fan van der Walt wrote: >> Just to be clear: I meant that I agree with you that we should see >> NumPy related warnings that aren't smothered in bogus messages. Is >> there no way to do this without compiler-specific hacks? > > Yes, there is, but it is more error prone (because then you are hiding > the warning instead of saying it is unused with a safeguard against any > use). I would not call this an hack, because __attribute__ is a well > known, well used gcc mechanism. For people not using gcc, it would not > change anything. > > Maybe there are another more portable methods I am not aware of, though. > > cheers, How about this: #define UNUSED(x) (x=x) void func(int a, int b, int c) { UNUSED(a); UNUSED(b); ... } Alternatively, #define UNUSED(x) ((void)x) also seems to stop the warning. I only tested this on 4.2.3, YMMV. -- Dag Sverre From stefan at sun.ac.za Mon Aug 4 04:41:07 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 4 Aug 2008 10:41:07 +0200 Subject: [Numpy-discussion] arrays : c aligmen to fortran and back In-Reply-To: References: Message-ID: <9457e7c80808040141kf8e6081vba0db8b8c50e798b@mail.gmail.com> Hi Thomas 2008/8/4 Thomas Hrabe : > I need to convert a 3d array from c alingment to fortran and was wandering > for the simplest code available. > Its all about python and an interfaced C++ program, which, however, > processes fortran aligned arrays. > > Is there a simple code from converting an array > arr = numpy.array((30,20,10)) > > to an array arr2 with the same shape > (arr.shape == arr2.shape) = true > where the data is aligned in fortran style. Does xf = np.asfortranarray(x) do what you want? St?fan From bsouthey at gmail.com Mon Aug 4 06:26:43 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 4 Aug 2008 05:26:43 -0500 Subject: [Numpy-discussion] numpy 1.1.rc2: win32 binaries In-Reply-To: References: <488D58A3.6070800@ar.media.kyoto-u.ac.jp> <488D5984.202@ar.media.kyoto-u.ac.jp> Message-ID: Hi, I installed the 'official binaries' on a Intel Celeron M 530 that supports SSE2 and SSE3 running MS Vista. All tests passed and with regards to the ticket: numpy.inner(F,F) resulted in 'array([[ Inf]])' Regards Bruce On Sat, Aug 2, 2008 at 7:36 PM, Jarrod Millman wrote: > On Fri, Aug 1, 2008 at 10:03 PM, Alan G Isaac wrote: >>> http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy-1.1.1.dev5559-win32-superpack-python2.5.exe > > You can get the official binaries from here: > http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103&release_id=617279 > > Please let us know if they work for you or not. > > Thanks, > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From david at ar.media.kyoto-u.ac.jp Mon Aug 4 07:54:46 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 04 Aug 2008 20:54:46 +0900 Subject: [Numpy-discussion] numpy 1.1.rc2: win32 binaries In-Reply-To: References: <488D58A3.6070800@ar.media.kyoto-u.ac.jp> <488D5984.202@ar.media.kyoto-u.ac.jp> Message-ID: <4896EE06.7070204@ar.media.kyoto-u.ac.jp> Bruce Southey wrote: > Hi, > I installed the 'official binaries' on a Intel Celeron M 530 that > supports SSE2 and SSE3 running MS Vista. All tests passed and with > regards to the ticket: numpy.inner(F,F) resulted in 'array([[ Inf]])' > I thought that problem had be gone for good, but you're right, I could reproduce on one machine. I have added a regression test on 1.1.x and trunk to avoid letting this slip one more time, and I am now investigating the problem itself, thanks for the bug report, David From david at ar.media.kyoto-u.ac.jp Mon Aug 4 08:07:04 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 04 Aug 2008 21:07:04 +0900 Subject: [Numpy-discussion] numpy 1.1.rc2: win32 binaries In-Reply-To: References: <488D58A3.6070800@ar.media.kyoto-u.ac.jp> <488D5984.202@ar.media.kyoto-u.ac.jp> Message-ID: <4896F0E8.40409@ar.media.kyoto-u.ac.jp> Bruce Southey wrote: > Hi, > I installed the 'official binaries' on a Intel Celeron M 530 that > supports SSE2 and SSE3 running MS Vista. All tests passed and with > regards to the ticket: numpy.inner(F,F) resulted in 'array([[ Inf]])' > Ok, it looks like this is entirely due to my own stupidity. I screwed up, and linked the binaries to an earlier version of ATLAS, the one with the problem... I will try to generate new binaries within today, cheers, David From nicolas.chopin at bris.ac.uk Mon Aug 4 10:58:00 2008 From: nicolas.chopin at bris.ac.uk (Nicolas Chopin) Date: Mon, 4 Aug 2008 14:58:00 +0000 (UTC) Subject: [Numpy-discussion] concatenate an array with a number Message-ID: Hi list, I want to do this: y = concatenate(x[0],x) where x is a 1d array. However, the only way I managed to make this work is: y = concatenate( array((x[0],),ndmin=1), x) which is kind of cryptic. (If I remove ndmin, numpy still complains.) 1. Is there a better way? 2. Would it be possible to allow numbers as arguments for concatenate? Many thanks NC From robert.kern at gmail.com Mon Aug 4 12:10:49 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 4 Aug 2008 11:10:49 -0500 Subject: [Numpy-discussion] concatenate an array with a number In-Reply-To: References: Message-ID: <3d375d730808040910y6380639fp60bb50f83569a79d@mail.gmail.com> On Mon, Aug 4, 2008 at 09:58, Nicolas Chopin wrote: > Hi list, > I want to do this: > > y = concatenate(x[0],x) > > where x is a 1d array. > However, the only way I managed to make this work is: > > y = concatenate( array((x[0],),ndmin=1), x) > > which is kind of cryptic. (If I remove ndmin, numpy still complains.) > > 1. Is there a better way? atleast_1d() -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robince at gmail.com Mon Aug 4 12:19:46 2008 From: robince at gmail.com (Robin) Date: Mon, 4 Aug 2008 17:19:46 +0100 Subject: [Numpy-discussion] concatenate an array with a number In-Reply-To: References: Message-ID: On Mon, Aug 4, 2008 at 3:58 PM, Nicolas Chopin wrote: > Hi list, > I want to do this: > > y = concatenate(x[0],x) > > where x is a 1d array. > However, the only way I managed to make this work is: > > y = concatenate( array((x[0],),ndmin=1), x) > > which is kind of cryptic. (If I remove ndmin, numpy still complains.) I think r_[x[0],x] will do you what you want... Robin From pgmdevlist at gmail.com Mon Aug 4 12:11:09 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 4 Aug 2008 12:11:09 -0400 Subject: [Numpy-discussion] concatenate an array with a number In-Reply-To: References: Message-ID: <200808041211.10083.pgmdevlist@gmail.com> On Monday 04 August 2008 10:58:00 Nicolas Chopin wrote: > Hi list, > I want to do this: > > y = concatenate(x[0],x) > > where x is a 1d array. > However, the only way I managed to make this work is: > > y = concatenate( array((x[0],),ndmin=1), x) > > which is kind of cryptic. (If I remove ndmin, numpy still complains.) > > 1. Is there a better way? y = np.concatenate([x[0]], x) From pgmdevlist at gmail.com Mon Aug 4 12:49:38 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 4 Aug 2008 12:49:38 -0400 Subject: [Numpy-discussion] What/when is the next release ? Message-ID: <200808041249.38998.pgmdevlist@gmail.com> All, * What will be the next release: 1.1.x (w/ support of Python 2.3) or 1.2.0 (w/o support to Python 2.3) ? I need to correct an elusive bug in numpy.ma.MaskedArray and wondered whether I needed to backport it or not (1.1.x would likely be affected as well). Thanks a lot in advance, P. From dalcinl at gmail.com Mon Aug 4 12:29:15 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 4 Aug 2008 13:29:15 -0300 Subject: [Numpy-discussion] Toward a numpy build with warning: handling unused variable In-Reply-To: <1217813948.12930.15.camel@bbc8> References: <489587A7.7010108@ar.media.kyoto-u.ac.jp> <5b8d13220808031602x567d4855i6b961e116ff69a5b@mail.gmail.com> <5b8d13220808031639me4cd4bak319638fafb45bbac@mail.gmail.com> <1217813948.12930.15.camel@bbc8> Message-ID: David, I second your approach. Furthermore, look how SWIG handles this, it is very similar to your proposal. The difference is that SWIG uses SWIGUNUSED for some autogenerated functions. Furthermore, it seems the SWIG developers protected the generated code taking into account GCC versions ;-) and the C++ case, and also Intel compilers. /* attribute recognised by some compilers to avoid 'unused' warnings */ #ifndef SWIGUNUSED # if defined(__GNUC__) # if !(defined(__cplusplus)) || (__GNUC__ > 3 || (__GNUC__ == 3 && __GNUC_MINOR__ >= 4)) # define SWIGUNUSED __attribute__ ((__unused__)) # else # define SWIGUNUSED # endif # elif defined(__ICC) # define SWIGUNUSED __attribute__ ((__unused__)) # else # define SWIGUNUSED # endif #endif #ifndef SWIGUNUSEDPARM # ifdef __cplusplus # define SWIGUNUSEDPARM(p) # else # define SWIGUNUSEDPARM(p) p SWIGUNUSED # endif #endif On Sun, Aug 3, 2008 at 10:39 PM, David Cournapeau wrote: > On Sun, 2008-08-03 at 19:32 -0500, Charles R Harris wrote: > >> >> >> >> There is self and args, filter them out. If the arguments have other >> names, rename them. > > That's much more error prone. You would need a list of such arguments, > and more importantly, it means you cannot enable warning by default > (because you cannot expect the casual user to filter them out). > > And if you filter them out, you won't see them, meaning you hide them > anyway. So I don't really see the point. > >> >> >> > If the warnings are too >> > numerous, filter them. >> >> >> If you filter them, what's the point of keeping them ? >> >> >> >> Because the compiler's inadequate determination of errors is driving >> the coding. > > I don't understand what you mean here ? The problem I see is the > following: currently, there are so many warnings that we don't enable > them, and I would guess nobody currently enable them. > >> That seems wrong way around. Maybe these variables can be notated >> somehow, i.e, __unused__(arg), which would at least document intent. > > Which is what I have been arguing all along :) If you look at the link I > posted before, it suggests a UNUSED macro which uses gcc attribute when > build with gcc, and do nothing otherwise: > > #ifdef __GNUCC__ > #define NPY_UNUSED(x) NPY_UNUSED__ ## x __attribute__((unused)) > __NPY_MANGLE_UNUSED) > #else > #define NPY_UNUSED(x) x > #endif > > This makes it hard to get it wrong: > - you can't use the variable by accident later: you will get a > compilation error thanks to the mangling > - you don't have meaningless warnings, meaning you see the real ones. > - if you don't use gcc, it is exactly as before. > > I realize I may have not been not really clear: I am not suggesting to > remove any warning about unused variable, just the ones which we cannot > do anything about because they are in the argument list and we know for > sure we will not use them. That include python C functions not used as a > method (dummy/unused, e.g. the first argument), ufunc which do not use > their func argument, etc... > > cheers, > > David > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From millman at berkeley.edu Mon Aug 4 13:10:48 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 4 Aug 2008 10:10:48 -0700 Subject: [Numpy-discussion] What/when is the next release ? In-Reply-To: <200808041249.38998.pgmdevlist@gmail.com> References: <200808041249.38998.pgmdevlist@gmail.com> Message-ID: On Mon, Aug 4, 2008 at 9:49 AM, Pierre GM wrote: > * What will be the next release: 1.1.x (w/ support of Python 2.3) or 1.2.0 > (w/o support to Python 2.3) ? There are currently no plans for 1.1.2. Not that it won't happen; it's just that there hasn't been any plans for the release. And I very much doubt there will be any serious discussion of releasing 1.1.2 before 1.2.0. Here is the current 1.2.0 schedule: - 8/08/08 tag the 1.2.0rc1 release and prepare packages - 8/15/08 tag the 1.2.0rc2 release and prepare packages - 8/23/08 tag the 1.2.0 release and prepare packages - 8/24/08 announce release I moved the rc dates forward a bit since I didn't get 1.1.1 out as early as I had hoped, which means I haven't had time to look over the current state of the trunk. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From charlesr.harris at gmail.com Mon Aug 4 13:17:15 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 4 Aug 2008 12:17:15 -0500 Subject: [Numpy-discussion] What/when is the next release ? In-Reply-To: <200808041249.38998.pgmdevlist@gmail.com> References: <200808041249.38998.pgmdevlist@gmail.com> Message-ID: HI Pierre, On Mon, Aug 4, 2008 at 11:49 AM, Pierre GM wrote: > All, > * What will be the next release: 1.1.x (w/ support of Python 2.3) or 1.2.0 > (w/o support to Python 2.3) ? > I need to correct an elusive bug in numpy.ma.MaskedArray and wondered > whether > I needed to backport it or not (1.1.x would likely be affected as well). > Thanks a lot in advance, > P. I don't know what the schedule is, but if you have a bug fix it is probably a good idea to back port it to 1.1.x so that any future release will be easier. Also, if you do do the backport, put an informative comment with the commit. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Mon Aug 4 13:26:56 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 4 Aug 2008 10:26:56 -0700 Subject: [Numpy-discussion] What/when is the next release ? In-Reply-To: References: <200808041249.38998.pgmdevlist@gmail.com> Message-ID: On Mon, Aug 4, 2008 at 10:10 AM, Jarrod Millman wrote: > Here is the current 1.2.0 schedule: > > - 8/08/08 tag the 1.2.0rc1 release and prepare packages > - 8/15/08 tag the 1.2.0rc2 release and prepare packages > - 8/23/08 tag the 1.2.0 release and prepare packages > - 8/24/08 announce release I was just talking to Robert and he pointed out that we should have a beta release first. So I am renaming 1.2.0rc1 to 1.2.0b1 and 1.2.0rc2 is now 1.2.0rc1. The plan is still to try and get 1.2.0 out on 8/24. Just to be clear the schedule is: - 8/08/08 tag the 1.2.0b1 release and prepare packages - 8/15/08 tag the 1.2.0rc1 release and prepare packages - 8/23/08 tag the 1.2.0 release and prepare packages - 8/24/08 announce release -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From millman at berkeley.edu Mon Aug 4 13:31:38 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 4 Aug 2008 10:31:38 -0700 Subject: [Numpy-discussion] What/when is the next release ? In-Reply-To: References: <200808041249.38998.pgmdevlist@gmail.com> Message-ID: On Mon, Aug 4, 2008 at 10:17 AM, Charles R Harris wrote: > I don't know what the schedule is, but if you have a bug fix it is probably > a good idea to back port it to 1.1.x so that any future release will be > easier. Also, if you do do the backport, put an informative comment with the > commit. I agree with the caveat that your focus should be mostly on 1.2 at this point. If your fix for 1.2 is not easy to back port to 1.1.x, I wouldn't worry about it too much. Ideally we will get all the fixes in 1.2 back to 1.1.x, but it isn't absolutely essential that every fix make it back. That said I would really like to see as many fixes as possible applied to both the trunk and the last stable release branch. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From millman at berkeley.edu Mon Aug 4 13:45:51 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 4 Aug 2008 10:45:51 -0700 Subject: [Numpy-discussion] 1.2 tasks Message-ID: Here are the remaining tasks that I am aware of that need to be done before tagging 1.2.0b1 on the 8th. Median ====== The call signature for median needs to change from def median(a, axis=0, out=None, overwrite_input=False): to def median(a, axis=None, out=None, overwrite_input=False): in both numpy/ma/extras.py and numpy/lib/function_base.py Histogram ======== The call signature for histogram needs to change from def histogram(a, bins=10, range=None, normed=False, weights=None, new=False): to def histogram(a, bins=10, range=None, normed=False, weights=None, new=True): in numpy/lib/function_base.py Documentation ============ The documentation project needs to merge in its changes. Stefan will take care of this on the 5th. Please let me know ASAP if there is anything I am missing. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From pgmdevlist at gmail.com Mon Aug 4 14:07:15 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 4 Aug 2008 14:07:15 -0400 Subject: [Numpy-discussion] What/when is the next release ? In-Reply-To: References: <200808041249.38998.pgmdevlist@gmail.com> Message-ID: <200808041407.16713.pgmdevlist@gmail.com> Charles, Jarrod, That isn't be a pb. I just ran into a pb with 1.2 that I know would show up in 1.1.x, so I just commited the changes to the 2 versions. I was just wondering for how long I would have to backport this kind of fixes. > I agree with the caveat that your focus should be mostly on 1.2 at > this point. If your fix for 1.2 is not easy to back port to 1.1.x, I > wouldn't worry about it too much. Ideally we will get all the fixes > in 1.2 back to 1.1.x, but it isn't absolutely essential that every fix > make it back. That said I would really like to see as many fixes as > possible applied to both the trunk and the last stable release branch. > > Thanks, From dsdale24 at gmail.com Mon Aug 4 14:16:33 2008 From: dsdale24 at gmail.com (Darren Dale) Date: Mon, 4 Aug 2008 14:16:33 -0400 Subject: [Numpy-discussion] 1.2 tasks In-Reply-To: References: Message-ID: <200808041416.34092.dsdale24@gmail.com> Hi Jarrod, I was wondering if someone knowledgeable would please look into the behavior of concatenate() with subclasses of ndarray. I posted at scipy-dev some work on a subclass that handles units, and when two of these are concatenated, a regular ndarray is returned rather than another instance of the subclass. This behavior was discussed a while back at http://thread.gmane.org/gmane.comp.python.numeric.general/14494 . It resurfaced on enthought-dev in Sept-2007, at which point Travis responded (I hope he doesnt mind being quoted): > there should be no array creation functions which do not > call __array_finalize__ if it is defined, and therefore, the described > behavior is a bug that should be reported. The concatentate code should > also respect the sub-type of the arrays being concatenated (it seems to > as I look at the code...) > > However, the larger question of priority occurs in mixed concatenation > cases which are probably being encountered. In such cases, the > sub-type with the largest priority will be the resulting type (and if > it's not your class then your __array_finalize__ will not be called). > I wonder if this was what was being encountered rather than a > "limitation of NumPy" I encountered the problem again over the weekend with the package I am working on, which can be checked out from: "hg clone \ http://dale.chess.cornell.edu/~darren/cgi-bin/hgwebdir.cgi/quantities \ quantities" it installs in the usual way. The concat issue can the be reproduced by doing: [~] |1> from quantities import NDQuantity udunits(3): Already initialized from file "/usr/lib64/python2.5/site-packages/quantities/quantities-data/udunits.dat" [~] |2> J=NDQuantity([1.,2.,3.],'J') [~] |3> J <3> NDQuantity([ 1., 2., 3.]), kg * m^2 / s^2 [~] |4> import numpy [~] |5> numpy.concatenate([J,J]) <5> array([ 1., 2., 3., 1., 2., 3.]) Maybe this is an issue with my code, and not numpy, but since others have reported the problem before me, perhaps someone could please have a look before releasing 1.2? I dont think array priority is the problem, since I see the issue concatenating two of the same array subtypes. Regards, Darren On Monday 04 August 2008 01:45:51 pm Jarrod Millman wrote: > Here are the remaining tasks that I am aware of that need to be done > before tagging 1.2.0b1 on the 8th. > > Median > ====== > The call signature for median needs to change from > def median(a, axis=0, out=None, overwrite_input=False): > to > def median(a, axis=None, out=None, overwrite_input=False): > in both numpy/ma/extras.py and numpy/lib/function_base.py > > Histogram > ======== > The call signature for histogram needs to change from > def histogram(a, bins=10, range=None, normed=False, weights=None, > new=False): to > def histogram(a, bins=10, range=None, normed=False, weights=None, > new=True): in numpy/lib/function_base.py > > Documentation > ============ > The documentation project needs to merge in its changes. Stefan will > take care of this on the 5th. > > Please let me know ASAP if there is anything I am missing. > > Thanks, From charlesr.harris at gmail.com Mon Aug 4 14:36:12 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 4 Aug 2008 13:36:12 -0500 Subject: [Numpy-discussion] What/when is the next release ? In-Reply-To: References: <200808041249.38998.pgmdevlist@gmail.com> Message-ID: On Mon, Aug 4, 2008 at 12:31 PM, Jarrod Millman wrote: > On Mon, Aug 4, 2008 at 10:17 AM, Charles R Harris > wrote: > > I don't know what the schedule is, but if you have a bug fix it is > probably > > a good idea to back port it to 1.1.x so that any future release will be > > easier. Also, if you do do the backport, put an informative comment with > the > > commit. > > I agree with the caveat that your focus should be mostly on 1.2 at > this point. If your fix for 1.2 is not easy to back port to 1.1.x, I > wouldn't worry about it too much. Ideally we will get all the fixes > in 1.2 back to 1.1.x, but it isn't absolutely essential that every fix > make it back. That said I would really like to see as many fixes as > possible applied to both the trunk and the last stable release branch. > > Thanks, > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Aug 4 14:40:45 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 4 Aug 2008 13:40:45 -0500 Subject: [Numpy-discussion] What/when is the next release ? In-Reply-To: <200808041407.16713.pgmdevlist@gmail.com> References: <200808041249.38998.pgmdevlist@gmail.com> <200808041407.16713.pgmdevlist@gmail.com> Message-ID: On Mon, Aug 4, 2008 at 1:07 PM, Pierre GM wrote: > Charles, Jarrod, > That isn't be a pb. I just ran into a pb with 1.2 that I know would show up > in > 1.1.x, so I just commited the changes to the 2 versions. I was just > wondering > for how long I would have to backport this kind of fixes. > > > I agree with the caveat that your focus should be mostly on 1.2 at > > this point. If your fix for 1.2 is not easy to back port to 1.1.x, I > > wouldn't worry about it too much. Ideally we will get all the fixes > > in 1.2 back to 1.1.x, but it isn't absolutely essential that every fix > > make it back. That said I would really like to see as many fixes as > > possible applied to both the trunk and the last stable release branch. > > > I think one more 1.1.x release might be appropriate, if only to have one solid version out for Python 2.3. Hopefully Scipy will also release a compatible version. And then we should put an end date on the series, say Jan 1, 2009 or maybe after the first bugfix release of 1.2. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.huard at gmail.com Mon Aug 4 14:41:18 2008 From: david.huard at gmail.com (David Huard) Date: Mon, 4 Aug 2008 14:41:18 -0400 Subject: [Numpy-discussion] 1.2 tasks In-Reply-To: References: Message-ID: <91cf711d0808041141s121a4322nf54ce41684149fc4@mail.gmail.com> On Mon, Aug 4, 2008 at 1:45 PM, Jarrod Millman wrote: > Here are the remaining tasks that I am aware of that need to be done > before tagging 1.2.0b1 on the 8th. > > Median > ====== > The call signature for median needs to change from > def median(a, axis=0, out=None, overwrite_input=False): > to > def median(a, axis=None, out=None, overwrite_input=False): > in both numpy/ma/extras.py and numpy/lib/function_base.py > > Histogram > ======== > The call signature for histogram needs to change from > def histogram(a, bins=10, range=None, normed=False, weights=None, > new=False): > to > def histogram(a, bins=10, range=None, normed=False, weights=None, > new=True): > in numpy/lib/function_base.py > Question: Should histogram raise a warning by default (new=True) to warn users that the behaviour has changed ? Or warn only if new=False to remind that the old behaviour will be deprecated in 1.3 ? I think that users will prefer being annoyed at warnings than surprised by an unexpected change, but repeated warnings can become a nuisance. To minimize annoyance, we could also offer three possibilities: new=None (default) : Equivalent to True, print warning about change. new=True : Don't print warning. new=False : Print warning about future deprecation. So those who have already set new=True don't get warnings, and all others are warned. Feedback ? David H. > Documentation > ============ > The documentation project needs to merge in its changes. Stefan will > take care of this on the 5th. > > Please let me know ASAP if there is anything I am missing. > > Thanks, > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Aug 4 14:49:42 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 4 Aug 2008 13:49:42 -0500 Subject: [Numpy-discussion] What/when is the next release ? In-Reply-To: References: <200808041249.38998.pgmdevlist@gmail.com> <200808041407.16713.pgmdevlist@gmail.com> Message-ID: <3d375d730808041149i32eaf33fn747b26be28edd3e8@mail.gmail.com> On Mon, Aug 4, 2008 at 13:40, Charles R Harris wrote: > > On Mon, Aug 4, 2008 at 1:07 PM, Pierre GM wrote: >> >> Charles, Jarrod, >> That isn't be a pb. I just ran into a pb with 1.2 that I know would show >> up in >> 1.1.x, so I just commited the changes to the 2 versions. I was just >> wondering >> for how long I would have to backport this kind of fixes. >> >> > I agree with the caveat that your focus should be mostly on 1.2 at >> > this point. If your fix for 1.2 is not easy to back port to 1.1.x, I >> > wouldn't worry about it too much. Ideally we will get all the fixes >> > in 1.2 back to 1.1.x, but it isn't absolutely essential that every fix >> > make it back. That said I would really like to see as many fixes as >> > possible applied to both the trunk and the last stable release branch. > > I think one more 1.1.x release might be appropriate, if only to have one > solid version out for Python 2.3. Hopefully Scipy will also release a > compatible version. Too late for that. SVN scipy already requires SVN numpy. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Chris.Barker at noaa.gov Mon Aug 4 15:24:21 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 04 Aug 2008 12:24:21 -0700 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <3d375d730808012313t66a96137l423856f9ed5f1448@mail.gmail.com> References: <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <4891E996.5070000@ar.media.kyoto-u.ac.jp> <4891F276.8070404@noaa.gov> <5b8d13220807311128w56e1996bue1ec304aca672b47@mail.gmail.com> <48933768.2010503@noaa.gov> <20080801165351.GC24781@phare.normalesup.org> <3d375d730808011249h29460aavd39f3b0f8874c361@mail.gmail.com> <48939835.4080801@noaa.gov> <48939973.4010008@noaa.gov> <4893EB4A.6050905@ar.media.kyoto-u.ac.jp> <3d375d730808012313t66a96137l423856f9ed5f1448@mail.gmail.com> Message-ID: <48975765.9020301@noaa.gov> Robert Kern wrote: > It isn't. The problem is on Chris's file system. Thanks for all your help, Robert. Interestingly, I haven't noticed any problems anywhere else, but who knows? I guess this is what Linux Torvalds meant when he said that OS-X's file system was "brain dead" > Whatever is wrong > with his file system (Bill Spotz's identical problem suggests too many > temporary but unused inodes) I didn't see anything about Bill having similar issues -- was it on this list? > But the problem really is his disk; it's not > a problem with numpy or Python or anything else. so the question is: what can I do about it? Do I have any other choice than wiping the disk and re-installing? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From pgmdevlist at gmail.com Mon Aug 4 16:11:03 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 4 Aug 2008 16:11:03 -0400 Subject: [Numpy-discussion] 1.2 tasks In-Reply-To: References: Message-ID: <200808041611.03790.pgmdevlist@gmail.com> On Monday 04 August 2008 13:45:51 Jarrod Millman wrote: > Here are the remaining tasks that I am aware of that need to be done > before tagging 1.2.0b1 on the 8th. > The call signature for median needs to change from > def median(a, axis=0, out=None, overwrite_input=False): > to > def median(a, axis=None, out=None, overwrite_input=False): > in both numpy/ma/extras.py and numpy/lib/function_base.py Done for numpy.ma.extras (r5607) From robert.kern at gmail.com Mon Aug 4 16:26:59 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 4 Aug 2008 15:26:59 -0500 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <48975765.9020301@noaa.gov> References: <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <5b8d13220807311128w56e1996bue1ec304aca672b47@mail.gmail.com> <48933768.2010503@noaa.gov> <20080801165351.GC24781@phare.normalesup.org> <3d375d730808011249h29460aavd39f3b0f8874c361@mail.gmail.com> <48939835.4080801@noaa.gov> <48939973.4010008@noaa.gov> <4893EB4A.6050905@ar.media.kyoto-u.ac.jp> <3d375d730808012313t66a96137l423856f9ed5f1448@mail.gmail.com> <48975765.9020301@noaa.gov> Message-ID: <3d375d730808041326sb6bece9q7fd0a8c4efe5cd08@mail.gmail.com> On Mon, Aug 4, 2008 at 14:24, Christopher Barker wrote: > Robert Kern wrote: >> It isn't. The problem is on Chris's file system. > > Thanks for all your help, Robert. Interestingly, I haven't noticed any > problems anywhere else, but who knows? > > I guess this is what Linux Torvalds meant when he said that OS-X's file > system was "brain dead" > >> Whatever is wrong >> with his file system (Bill Spotz's identical problem suggests too many >> temporary but unused inodes) > > I didn't see anything about Bill having similar issues -- was it on this > list? >From my earlier message in this thread: """ Looking at the Shark results you sent me, it looks like all of your time is getting sucked up by the system call getdirentries(). Googling for some of the function names in that stack brings me to the message "Slow python initialization" on the Pythonmac-SIG: http://mail.python.org/pipermail/pythonmac-sig/2005-December/015542.html The ultimate resolution was that Bill Spotz, the original poster, ran Disk Utility and used the Disk Repair option to clean up a large number of unused inodes. This solved the problem for him: http://mail.python.org/pipermail/pythonmac-sig/2005-December/015548.html """ -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pgmdevlist at gmail.com Mon Aug 4 16:29:57 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 4 Aug 2008 16:29:57 -0400 Subject: [Numpy-discussion] 1.2 tasks In-Reply-To: <200808041416.34092.dsdale24@gmail.com> References: <200808041416.34092.dsdale24@gmail.com> Message-ID: <200808041629.57817.pgmdevlist@gmail.com> On Monday 04 August 2008 14:16:33 Darren Dale wrote: > Hi Jarrod, > > I was wondering if someone knowledgeable would please look into the > behavior of concatenate() with subclasses of ndarray. Darren, I ran into similar problems when I started working on numpy.ma and scikits.timeseries, and I remember having conversations with Bryce Hendrix of Enthought about that on this very list. I think you have a problem in `NDQuantities.__array_finalize__`: you don't initialize `._units` if `obj` doesn't have a `_units`. At least, you should use something like self._units = getattr(obj, '_units', None), which will set `._units` to None if `obj` is not a NDQuantities, and will also ensure that your NDQuantities array always have a ._units attribute. In other words, if I take a view of a ndarray as a NDQuantities, I should have a _units defined by default (None being the most appropriate in that case). About concatenation: In a lot of cases, concatenate requires some tweaking. In scikits.timeseries, we have to decide what to do when the series being concatenated overlap. In your case, you have to decide what to do when concatenating 2 arrays A & B, for example: * if A & B have the same unit, keep it * If A (B) doesn't have a unit, keep the unit of B (A) * If the units of A and B are compatible, convert one to the other * If the units are incompatible, raise an exception. What I suggest is to implement your own concatenate: check the units, concatenate the arrays as regular ndarrays, take a view of the result, modify the unit of the result accordingly. About subclassing: * You may want to check the cookbook: http://www.scipy.org/Subclasses * You may also want to check other ndarray subclasses, such as MaskedArray, TimeSeries () for some implementation details. * You may also want add your own experience on http://www.scipy.org/Subclasses, so that we don't lose it. Drop me a line offlist if you'd like, I'd be happy to help (depending on the time at hand). Cheers. P. From jh at physics.ucf.edu Mon Aug 4 16:42:12 2008 From: jh at physics.ucf.edu (Joe Harrington) Date: Mon, 04 Aug 2008 16:42:12 -0400 Subject: [Numpy-discussion] WTFM! Message-ID: SciPy Documentation Marathon 2008 Status Report We are now nearing the end of the summer. We have a ton of great docstrings, a nice PDF and HTML reference guide, a new package with pages on general topics like slicing, and a glossary. We had hoped to have all the numpy docstrings in first-draft form in time for the pre-fall (1.2) release. The actual number of pages was more than double our quick goal-setting assessment, so we won't make it. As of this moment, we have: status % pages Needs editing 52 430 Being written / Changed 27 226 Needs review 18 152 Needs review (revised) 0 1 Needs work (reviewed) 0 3 Reviewed (needs proof) 2 19 Proofed 0 0 Unimportant 1531 Our current status can always be seen at: http://sd-2116.dedibox.fr/pydocweb/stats/ Definitions of the categories are also on the wiki, but "being written" is our first-draft category. So, we're just shy of halfway there, and since the goal more than doubled, we can say we have not failed our expectations. But, we haven't succeeded, either, and we certainly haven't finished. So, this being a marathon, we're not going to stop! Please join us if you haven't already for the... PRE-CONFERENCE DOC BLITZ! We can, quite realistically, get up to 60% for numpy 1.2. We've had several 8% weeks this summer and we've got several weeks to go. Stefan will merge docstrings into the beta for 1.2 on 5 August, and will continue merging from the wiki for the release candidates and final cut. However, Writing has slowed to a crawl in recent weeks. Please pitch in to help those who are still writing so we can get to 60% by release 1.2. Looking further ahead, I hope all the volunteers will continue writing for the rest of the summer and fall, so that we can put 100% decent drafts into 1.3, and a 100% reviewed set of docstrings into 1.4. Then we can turn our attention to scipy. Enough stick, here's some carrot: This is the design for this year's documentation prize, a T-shirt in Robert Kern black, designed by Teresa Jeffcott: http://physics.ucf.edu/~jh/scipyshirt-2008-2.png We'll hand these out at SciPy '08 to anyone who has written 1000 words or more (according to the stats page) or who has made an equivalent contribution in other ways (reviewing, wiki creation, etc.; Stefan and I will judge). So far, 11 contributors qualify, but several more could easily reach that goal in time. In fact, several of our volunteer writers have produced 1000 words in one week. The offer remains good through the first-draft phase, though you'll need to act quickly to get your docs into 1.2 and be recognized at the conference! If you won't be at SciPy '08 or if you qualify later, we'll mail one to you. As always, further discussion belongs on the scipy-dev mailing list, as does your request to be added to the doc wiki editors group, so head over to the doc wiki main page, established an account there, and give us a shout! WTFM, --jh-- From thrabe at burnham.org Mon Aug 4 17:14:51 2008 From: thrabe at burnham.org (Thomas Hrabe) Date: Mon, 4 Aug 2008 14:14:51 -0700 Subject: [Numpy-discussion] arrays : c aligmen to fortran and back References: <9457e7c80808040141kf8e6081vba0db8b8c50e798b@mail.gmail.com> Message-ID: Yeah, this function does the job. Thank you very much! -----Original Message----- From: numpy-discussion-bounces at scipy.org on behalf of St?fan van der Walt Sent: Mon 8/4/2008 1:41 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] arrays : c aligmen to fortran and back Hi Thomas 2008/8/4 Thomas Hrabe : > I need to convert a 3d array from c alingment to fortran and was wandering > for the simplest code available. > Its all about python and an interfaced C++ program, which, however, > processes fortran aligned arrays. > > Is there a simple code from converting an array > arr = numpy.array((30,20,10)) > > to an array arr2 with the same shape > (arr.shape == arr2.shape) = true > where the data is aligned in fortran style. Does xf = np.asfortranarray(x) do what you want? St?fan _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3218 bytes Desc: not available URL: From thrabe at burnham.org Mon Aug 4 17:21:54 2008 From: thrabe at burnham.org (Thomas Hrabe) Date: Mon, 4 Aug 2008 14:21:54 -0700 Subject: [Numpy-discussion] arrays : c aligmen to fortran and back References: <9457e7c80808040141kf8e6081vba0db8b8c50e798b@mail.gmail.com> Message-ID: PS: is there any public C library available which would perform the same operation? like void* swap(int numberDimensions,int sizeDimensions,int dataSize,int conversionType,void* data); where conversionType is the flag fortran to c or vice versa. As it works for numpy, there should be some public C/C++ code available... ??? I would like to have this python call in my C code (without calling an embedded function or so). xf = np.asfortranarray(x) Thank you very much, Thomas -----Original Message----- From: numpy-discussion-bounces at scipy.org on behalf of Thomas Hrabe Sent: Mon 8/4/2008 2:14 PM To: Discussion of Numerical Python; numpy-discussion-bounces at scipy.org Subject: RE: [Numpy-discussion] arrays : c aligmen to fortran and back Yeah, this function does the job. Thank you very much! -----Original Message----- From: numpy-discussion-bounces at scipy.org on behalf of St?fan van der Walt Sent: Mon 8/4/2008 1:41 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] arrays : c aligmen to fortran and back Hi Thomas 2008/8/4 Thomas Hrabe : > I need to convert a 3d array from c alingment to fortran and was wandering > for the simplest code available. > Its all about python and an interfaced C++ program, which, however, > processes fortran aligned arrays. > > Is there a simple code from converting an array > arr = numpy.array((30,20,10)) > > to an array arr2 with the same shape > (arr.shape == arr2.shape) = true > where the data is aligned in fortran style. Does xf = np.asfortranarray(x) do what you want? St?fan _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3642 bytes Desc: not available URL: From dsdale24 at gmail.com Mon Aug 4 17:53:54 2008 From: dsdale24 at gmail.com (Darren Dale) Date: Mon, 4 Aug 2008 17:53:54 -0400 Subject: [Numpy-discussion] 1.2 tasks In-Reply-To: <200808041629.57817.pgmdevlist@gmail.com> References: <200808041416.34092.dsdale24@gmail.com> <200808041629.57817.pgmdevlist@gmail.com> Message-ID: <200808041753.54348.dsdale24@gmail.com> Hi Pierre, On Monday 04 August 2008 04:29:57 pm Pierre GM wrote: > On Monday 04 August 2008 14:16:33 Darren Dale wrote: > > Hi Jarrod, > > > > I was wondering if someone knowledgeable would please look into the > > behavior of concatenate() with subclasses of ndarray. > > Darren, > I ran into similar problems when I started working on numpy.ma and > scikits.timeseries, and I remember having conversations with Bryce Hendrix > of Enthought about that on this very list. > > I think you have a problem in `NDQuantities.__array_finalize__`: you don't > initialize `._units` if `obj` doesn't have a `_units`. At least, you should > use something like > self._units = getattr(obj, '_units', None), > which will set `._units` to None if `obj` is not a NDQuantities, and will > also ensure that your NDQuantities array always have a ._units attribute. > In other words, if I take a view of a ndarray as a NDQuantities, I should > have a _units defined by default (None being the most appropriate in that > case). > > About concatenation: > In a lot of cases, concatenate requires some tweaking. In > scikits.timeseries, we have to decide what to do when the series being > concatenated overlap. In your case, you have to decide what to do when > concatenating 2 arrays A & B, for example: > * if A & B have the same unit, keep it > * If A (B) doesn't have a unit, keep the unit of B (A) > * If the units of A and B are compatible, convert one to the other > * If the units are incompatible, raise an exception. > What I suggest is to implement your own concatenate: check the units, > concatenate the arrays as regular ndarrays, take a view of the result, > modify the unit of the result accordingly. > > About subclassing: > * You may want to check the cookbook: > http://www.scipy.org/Subclasses > * You may also want to check other ndarray subclasses, such as MaskedArray, > TimeSeries () for some implementation details. > * You may also want add your own experience on > http://www.scipy.org/Subclasses, so that we don't lose it. > > Drop me a line offlist if you'd like, I'd be happy to help (depending on > the time at hand). > Cheers. Thank you for the comments. You made a number of good points and I will look into each. I agree that it looks like I need to write my own concatenate function, but judging from Travis' comment a while back, there may still be an issue with numpy's concatenate. In my example, it is not just that concat is stripping the units, it is returning a different type. Maybe that is not such a bad thing in this case, but I thought it should be brought to the numpy maintainers attention. Darren From kwgoodman at gmail.com Mon Aug 4 18:06:57 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 4 Aug 2008 15:06:57 -0700 Subject: [Numpy-discussion] boolean defaults for svd Message-ID: svd uses 1 for its defaults: svd(a, full_matrices=1, compute_uv=1) Anyone interested in changing 1 to True? It shouldn't break any code, right? From dsdale24 at gmail.com Mon Aug 4 18:43:46 2008 From: dsdale24 at gmail.com (Darren Dale) Date: Mon, 4 Aug 2008 18:43:46 -0400 Subject: [Numpy-discussion] 1.2 tasks In-Reply-To: <200808041629.57817.pgmdevlist@gmail.com> References: <200808041416.34092.dsdale24@gmail.com> <200808041629.57817.pgmdevlist@gmail.com> Message-ID: <200808041843.46628.dsdale24@gmail.com> On Monday 04 August 2008 6:27:12 pm you wrote: > On Monday 04 August 2008 17:53:54 you wrote: > In my example, it is not > > > just that concat is stripping the units, it is returning a different > > type. > > By that, you mean that the result is not a NDQuantity ? Ah. > > As you mentioned Travis' email, you must have noticed he was talking about > priorities: set a __array_priority__ argument as class variable of > NDQuantity with a value higher than 1 and hop ! You have a ndquantity at > the end. Except that in your case, you can't print it, because it doesn't > have a ._units. Not a surprise, as your __array_finalize__ doesn't define a > ._units unless the obj parameter is already a ndquantity: here, that's not > the case, there's a conversion to ndarray under the hood during > concatenate, then a call to __array_finalize__ with a ndarray as obj. > > So, in fact, all is well and works as it should on the numpy part. I stand corrected. I must have misunderstood the documentation of __array_priority__ in the numpy manual. Thank you for helping clear this up! Darren From Chris.Barker at noaa.gov Mon Aug 4 19:01:12 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 04 Aug 2008 16:01:12 -0700 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <3d375d730808041326sb6bece9q7fd0a8c4efe5cd08@mail.gmail.com> References: <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <5b8d13220807311128w56e1996bue1ec304aca672b47@mail.gmail.com> <48933768.2010503@noaa.gov> <20080801165351.GC24781@phare.normalesup.org> <3d375d730808011249h29460aavd39f3b0f8874c361@mail.gmail.com> <48939835.4080801@noaa.gov> <48939973.4010008@noaa.gov> <4893EB4A.6050905@ar.media.kyoto-u.ac.jp> <3d375d730808012313t66a96137l423856f9ed5f1448@mail.gmail.com> <48975765.9020301@noaa.gov> <3d375d730808041326sb6bece9q7fd0a8c4efe5cd08@mail.gmail.com> Message-ID: <48978A38.6010503@noaa.gov> OK, So I'm an idiot. After reading this, I thought "I haven't rebooted for a while". It turns out it's been 35 days. I think I've been having slow startup for a longer than that, but none the less, I re-booted (which took a long time), and presto: $ time python -c "import numpy" real 0m0.686s user 0m0.322s sys 0m0.363s much better! I suspect OS-X did some disk-cleaning on re-boot. Frankly, 35 days is pretty pathetic for an uptime, but as I said, I think this issue has been going on longer. Perhaps OS-X runs a disk check every n re-boots, like some linux distros do. Sorry about the noise, and thanks, particularly to Robert, for taking an interest in this. -Chris Robert Kern wrote: > On Mon, Aug 4, 2008 at 14:24, Christopher Barker wrote: >> Robert Kern wrote: >>> It isn't. The problem is on Chris's file system. >> Thanks for all your help, Robert. Interestingly, I haven't noticed any >> problems anywhere else, but who knows? >> >> I guess this is what Linux Torvalds meant when he said that OS-X's file >> system was "brain dead" >> >>> Whatever is wrong >>> with his file system (Bill Spotz's identical problem suggests too many >>> temporary but unused inodes) >> I didn't see anything about Bill having similar issues -- was it on this >> list? > >>From my earlier message in this thread: > > """ > Looking at the Shark results you sent me, it looks like all of your > time is getting sucked up by the system call getdirentries(). Googling > for some of the function names in that stack brings me to the message > "Slow python initialization" on the Pythonmac-SIG: > > http://mail.python.org/pipermail/pythonmac-sig/2005-December/015542.html > > The ultimate resolution was that Bill Spotz, the original poster, ran > Disk Utility and used the Disk Repair option to clean up a large > number of unused inodes. This solved the problem for him: > > http://mail.python.org/pipermail/pythonmac-sig/2005-December/015548.html > """ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Mon Aug 4 19:04:06 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 4 Aug 2008 18:04:06 -0500 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <48978A38.6010503@noaa.gov> References: <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <20080801165351.GC24781@phare.normalesup.org> <3d375d730808011249h29460aavd39f3b0f8874c361@mail.gmail.com> <48939835.4080801@noaa.gov> <48939973.4010008@noaa.gov> <4893EB4A.6050905@ar.media.kyoto-u.ac.jp> <3d375d730808012313t66a96137l423856f9ed5f1448@mail.gmail.com> <48975765.9020301@noaa.gov> <3d375d730808041326sb6bece9q7fd0a8c4efe5cd08@mail.gmail.com> <48978A38.6010503@noaa.gov> Message-ID: <3d375d730808041604j39c4cebfif40ca57ba803069c@mail.gmail.com> On Mon, Aug 4, 2008 at 18:01, Christopher Barker wrote: > OK, > > So I'm an idiot. After reading this, I thought "I haven't rebooted for a > while". It turns out it's been 35 days. I think I've been having slow > startup for a longer than that, but none the less, I re-booted (which > took a long time), and presto: > > $ time python -c "import numpy" > > real 0m0.686s > user 0m0.322s > sys 0m0.363s > > much better! It's still pretty bad, though. I do recommend running Disk Repair like Bill did. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dalke at dalkescientific.com Mon Aug 4 19:15:36 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Tue, 5 Aug 2008 01:15:36 +0200 Subject: [Numpy-discussion] running numpy tests In-Reply-To: <1d36917a0807301921m70107b84s3a64077f73818410@mail.gmail.com> References: <92AFDF8C-418F-4B6D-895C-8E7F0FD47D90@dalkescientific.com> <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <846D821E-BA9F-4043-AE39-D7D648013C30@dalkescientific.com> <6B6B9FA6-864D-4D31-99BB-E9AE7A93A6D3@dalkescientific.com> <1d36917a0807301351x703ee800lf96fdc94944fdd9b@mail.gmail.com> <1d36917a0807301921m70107b84s3a64077f73818410@mail.gmail.com> Message-ID: <22B8A6BB-2F49-4566-9B3C-086A4C2A0B8D@dalkescientific.com> I'm working on the patches for reducing the import overhead. I want to make sure I don't break anything. I'm trying to figure out how to run all of the tests. I expected, based on the following Alan McIntyre wrote: > They actually do two different things; numpy.test() runs test for all > of numpy, and numpy.testing.test() runs tests for numpy.testing only. > There are similar functions in numpy.lib, numpy.core, etc. Robert Kern wrote: > By now, we have most > of the denizens here trained to do numpy.test() when testing their new > installations. README: > After installation, tests can be run (from outside the source > directory) with: > > python -c 'import numpy; numpy.test()' that 'numpy.test()' runs everthing. When I run numpy.test() I don't seem to run all of the tests. That is, I don't see the output I get when I run numpy.lib.test() . Here's a copy of my output, to show you what I mean. Also, I can't figure out what when I run a test I get a new Python prompt. Python 2.5 (r25:51918, Sep 19 2006, 08:49:13) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.test() Running unit tests for numpy NumPy version 1.2.0.dev5595 NumPy is installed in /Library/Frameworks/Python.framework/Versions/ 2.5/lib/python2.5/site-packages/numpy Python version 2.5 (r25:51918, Sep 19 2006, 08:49:13) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] nose version 0.10.3 Not implemented: Defined_Binary_Op Not implemented: Defined_Binary_Op Defined_Operator not defined used by Generic_Spec Needs match implementation: Allocate_Stmt Needs match implementation: Associate_Construct .... many lines removed .... Needs match implementation: Target_Stmt Needs match implementation: Type_Bound_Procedure_Part Needs match implementation: Where_Construct ----- Nof match implementation needs: 51 out of 224 Nof tests needs: 224 out of 224 Total number of classes: 529 ----- No module named test_derived_scalar_ext , recompiling test_derived_scalar_ext. Parsing '/tmp/tmpyQVvVI.f90'.. Generating interface for test_derived_scalar_ext Subroutine: f2py_test_derived_scalar_ext_foo Generating interface for test_derived_scalar_ext.myt: f2py_type_myt_32 Generating interface for Integer: npy_int32 Generating interface for test_derived_scalar_ext Subroutine: f2py_test_derived_scalar_ext_f2pywrap_foo2 setup arguments: ' build_ext --build-temp tmp/ext_temp --build-lib tmp build_clib --build-temp tmp/clib_temp --build-clib tmp/clib_clib' running build_ext running build_src building library "test_derived_scalar_ext_fortran_f2py" sources building library "test_derived_scalar_ext_f_wrappers_f2py" sources building extension "test_derived_scalar_ext" sources running build_clib customize UnixCCompiler customize UnixCCompiler using build_clib customize NAGFCompiler ... 12 lines removed as it tries to find a compiler ... customize Gnu95FCompiler Found executable /usr/local/bin/gfortran customize Gnu95FCompiler customize Gnu95FCompiler using build_clib building 'test_derived_scalar_ext_fortran_f2py' library compiling Fortran sources Fortran f77 compiler: /usr/local/bin/gfortran -Wall -ffixed-form -fno- second-underscore -fPIC -O3 -funroll-loops Fortran f90 compiler: /usr/local/bin/gfortran -Wall -fno-second- underscore -fPIC -O3 -funroll-loops Fortran fix compiler: /usr/local/bin/gfortran -Wall -ffixed-form -fno- second-underscore -Wall -fno-second-underscore -fPIC -O3 -funroll-loops creating tmp/clib_temp creating tmp/clib_temp/tmp compile options: '-c' gfortran:f90: /tmp/tmpyQVvVI.f90 creating tmp/clib_clib ar: adding 1 object files to tmp/clib_clib/ libtest_derived_scalar_ext_fortran_f2py.a ranlib:@ tmp/clib_clib/libtest_derived_scalar_ext_fortran_f2py.a building 'test_derived_scalar_ext_f_wrappers_f2py' library compiling Fortran sources Fortran f77 compiler: /usr/local/bin/gfortran -Wall -ffixed-form -fno- second-underscore -fPIC -O3 -funroll-loops Fortran f90 compiler: /usr/local/bin/gfortran -Wall -fno-second- underscore -fPIC -O3 -funroll-loops Fortran fix compiler: /usr/local/bin/gfortran -Wall -ffixed-form -fno- second-underscore -Wall -fno-second-underscore -fPIC -O3 -funroll-loops compile options: '-c' gfortran:f90: tmp/test_derived_scalar_ext_f_wrappers_f2py.f90 ar: adding 1 object files to tmp/clib_clib/ libtest_derived_scalar_ext_f_wrappers_f2py.a ranlib:@ tmp/clib_clib/libtest_derived_scalar_ext_f_wrappers_f2py.a customize UnixCCompiler customize UnixCCompiler using build_ext ... about 15 lines removed ... customize Gnu95FCompiler customize Gnu95FCompiler using build_ext building 'test_derived_scalar_ext' extension compiling C sources C compiler: gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/ MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 creating tmp/ext_temp creating tmp/ext_temp/tmp compile options: '-I/Library/Frameworks/Python.framework/Versions/2.5/ lib/python2.5/site-packages/numpy/core/include -I/Library/Frameworks/ Python.framework/Versions/2.5/include/python2.5 -c' gcc: tmp/test_derived_scalar_extmodule.c /usr/local/bin/gfortran -Wall -Wall -undefined dynamic_lookup -bundle tmp/ext_temp/tmp/test_derived_scalar_extmodule.o -L/usr/local/lib/gcc/ i686-apple-darwin8/4.2.1 -Ltmp/clib_clib - ltest_derived_scalar_ext_f_wrappers_f2py - ltest_derived_scalar_ext_fortran_f2py -lgfortran -o tmp/ test_derived_scalar_ext.so Removing build directory tmp/ext_temp Removing build directory tmp/clib_temp Removing build directory tmp/clib_clib Python 2.5 (r25:51918, Sep 19 2006, 08:49:13) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy.lib >>> numpy.lib.test() Running unit tests for numpy.lib NumPy version 1.2.0.dev5595 NumPy is installed in /Library/Frameworks/Python.framework/Versions/ 2.5/lib/python2.5/site-packages/numpy Python version 2.5 (r25:51918, Sep 19 2006, 08:49:13) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] nose version 0.10.3 ........................................................................ ........................................................................ ........................................................................ ........................................................................ ........................................................................ ........................................................................ ........................................................................ ........................................................................ ........................................................................ ........................................................................ ........................................................................ ........................................................................ ........................................................................ ................................................................. ---------------------------------------------------------------------- Ran 1001 tests in 4.497s OK Also, when I do the suggested python -c 'import numpy; numpy.test()' I get output very much like when I did things interactively, except the last lines of the output are: ... gcc: tmp/test_derived_scalar_extmodule.c /usr/local/bin/gfortran -Wall -Wall -undefined dynamic_lookup -bundle tmp/ext_temp/tmp/test_derived_scalar_extmodule.o -L/usr/local/lib/gcc/ i686-apple-darwin8/4.2.1 -Ltmp/clib_clib - ltest_derived_scalar_ext_f_wrappers_f2py - ltest_derived_scalar_ext_fortran_f2py -lgfortran -o tmp/ test_derived_scalar_ext.so Removing build directory tmp/ext_temp Removing build directory tmp/clib_temp Removing build directory tmp/clib_clib Argument expected for the -c option usage: /Library/Frameworks/Python.framework/Versions/2.5/Resources/ Python.app/Contents/MacOS/Python [option] ... [-c cmd | -m mod | file | -] [arg] ... Try `python -h' for more information. Andrew dalke at dalkescientific.com From robert.kern at gmail.com Mon Aug 4 20:00:00 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 4 Aug 2008 19:00:00 -0500 Subject: [Numpy-discussion] running numpy tests In-Reply-To: <22B8A6BB-2F49-4566-9B3C-086A4C2A0B8D@dalkescientific.com> References: <92AFDF8C-418F-4B6D-895C-8E7F0FD47D90@dalkescientific.com> <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <846D821E-BA9F-4043-AE39-D7D648013C30@dalkescientific.com> <6B6B9FA6-864D-4D31-99BB-E9AE7A93A6D3@dalkescientific.com> <1d36917a0807301351x703ee800lf96fdc94944fdd9b@mail.gmail.com> <1d36917a0807301921m70107b84s3a64077f73818410@mail.gmail.com> <22B8A6BB-2F49-4566-9B3C-086A4C2A0B8D@dalkescientific.com> Message-ID: <3d375d730808041700k4f093122j84f895327895fb89@mail.gmail.com> On Mon, Aug 4, 2008 at 18:15, Andrew Dalke wrote: > I'm working on the patches for reducing the import overhead. I want > to make sure I don't break anything. I'm trying to figure out how to > run all of the tests. I expected, based on the following > > Alan McIntyre wrote: >> They actually do two different things; numpy.test() runs test for all >> of numpy, and numpy.testing.test() runs tests for numpy.testing only. >> There are similar functions in numpy.lib, numpy.core, etc. > > Robert Kern wrote: >> By now, we have most >> of the denizens here trained to do numpy.test() when testing their new >> installations. > > > README: >> After installation, tests can be run (from outside the source >> directory) with: >> >> python -c 'import numpy; numpy.test()' > > > that 'numpy.test()' runs everthing. > > > When I run numpy.test() I don't seem to run all of the tests. That > is, I don't see the output I get when I run numpy.lib.test() . > Here's a copy of my output, to show you what I mean. You have old stuff in your checkout/installation. Make sure you have deleted all of the *.pycs and directories which have been deleted in SVN. > Also, I can't > figure out what when I run a test I get a new Python prompt. I can't parse that sentence. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dalke at dalkescientific.com Mon Aug 4 20:56:13 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Tue, 5 Aug 2008 02:56:13 +0200 Subject: [Numpy-discussion] running numpy tests In-Reply-To: <3d375d730808041700k4f093122j84f895327895fb89@mail.gmail.com> References: <92AFDF8C-418F-4B6D-895C-8E7F0FD47D90@dalkescientific.com> <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <846D821E-BA9F-4043-AE39-D7D648013C30@dalkescientific.com> <6B6B9FA6-864D-4D31-99BB-E9AE7A93A6D3@dalkescientific.com> <1d36917a0807301351x703ee800lf96fdc94944fdd9b@mail.gmail.com> <1d36917a0807301921m70107b84s3a64077f73818410@mail.gmail.com> <22B8A6BB-2F49-4566-9B3C-086A4C2A0B8D@dalkescientific.com> <3d375d730808041700k4f093122j84f895327895fb89@mail.gmail.com> Message-ID: <9DB1589B-1C0C-4595-9EDE-0A86BB2B477D@dalkescientific.com> On Aug 5, 2008, at 2:00 AM, Robert Kern wrote: > You have old stuff in your checkout/installation. Make sure you have > deleted all of the *.pycs and directories which have been deleted in > SVN. I removed all .pyc files, wiped my installation directory, and it works now as I expect it to work. > >> Also, I can't >> figure out what when I run a test I get a new Python prompt. > > I can't parse that sentence. Mmm, I wrote "what" when I mean "why". In my earlier output from the interactive shell you could see at the end that a new Python session started. I was in the new shell, and couldn't go backwards to get to the 'numpy.test()' I had just executed. With the removal of old stuff, I no longer have that problem. Thanks for the fix, Andrew dalke at dalkescientific.com From dalke at dalkescientific.com Mon Aug 4 21:41:11 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Tue, 5 Aug 2008 03:41:11 +0200 Subject: [Numpy-discussion] running numpy tests In-Reply-To: <3d375d730808041700k4f093122j84f895327895fb89@mail.gmail.com> References: <92AFDF8C-418F-4B6D-895C-8E7F0FD47D90@dalkescientific.com> <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <846D821E-BA9F-4043-AE39-D7D648013C30@dalkescientific.com> <6B6B9FA6-864D-4D31-99BB-E9AE7A93A6D3@dalkescientific.com> <1d36917a0807301351x703ee800lf96fdc94944fdd9b@mail.gmail.com> <1d36917a0807301921m70107b84s3a64077f73818410@mail.gmail.com> <22B8A6BB-2F49-4566-9B3C-086A4C2A0B8D@dalkescientific.com> <3d375d730808041700k4f093122j84f895327895fb89@mail.gmail.com> Message-ID: On Aug 5, 2008, at 2:00 AM, Robert Kern wrote: > You have old stuff in your checkout/installation. Make sure you have > deleted all of the *.pycs and directories which have been deleted in > SVN. Now that I've fixed that, I can tell that I made a mistake related to the self-test code. I can't figure it out. Many of the modules have code which look like from testing import Tester test = Tester().test bench = Tester().bench As I understand it, this code is migrating to use nosetests. Because people expect 'import numpy; numpy.test()' to work, there will be a compatibility period where this API is unchanged. I found that importing 'testing' costs 0.013 seconds, which is 10% of my current import time. I would like to defer the import until needed, so I rewrote the 'test' code as: def test(label='fast', verbose=1, extra_argv=None, doctests=False, coverage=False, **kwargs): from testing import Tester import numpy Tester(numpy).test(label, verbose, extra_argv, doctests, coverage, **kwargs) In my view there's no difference between them, but Tester().test does introspection to figure out the module location. (In fact, if I don't pass it the module explicitly then it expects that locals() ["__file__"] for sys._getframe(-1) will exist, which is not the case with my function. The underling code should instead check for that variable in globals().) I ended up with recursion errors, and I don't know why. Any idea of what to do? [josiah:~] dalke% python -c 'import numpy; numpy.test()' Running unit tests for numpy NumPy version 1.2.0.dev5607 NumPy is installed in /Library/Frameworks/Python.framework/Versions/ 2.5/lib/python2.5/site-packages/numpy Python version 2.5 (r25:51918, Sep 19 2006, 08:49:13) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] nose version 0.10.3 Running unit tests for numpy NumPy version 1.2.0.dev5607 NumPy is installed in /Library/Frameworks/Python.framework/Versions/ 2.5/lib/python2.5/site-packages/numpy Python version 2.5 (r25:51918, Sep 19 2006, 08:49:13) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] nose version 0.10.3 Running unit tests for numpy NumPy version 1.2.0.dev5607 NumPy is installed in /Library/Frameworks/Python.framework/Versions/ 2.5/lib/python2.5/site-packages/numpy .... Python version 2.5 (r25:51918, Sep 19 2006, 08:49:13) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] nose version 0.10.3 EEEEEEEEEEE ====================================================================== ERROR: numpy.test ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/__init__.py", line 107, in test coverage, **kwargs) File "/Users/dalke/cvses/numpy/build/lib.macosx-10.3-fat-2.5/numpy/ testing/nosetester.py", line 270, in test t = NumpyTestProgram(argv=argv, exit=False, plugins=plugins) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/core.py", line 219, in __init__ argv=argv, testRunner=testRunner, testLoader=testLoader) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/unittest.py", line 758, in __init__ self.parseArgs(argv) ... return self.call(*arg, **kw) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/plugins/ manager.py", line 145, in simple result = meth(*arg, **kw) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/plugins/ attrib.py", line 214, in wantMethod return self.validateAttrib(attribs) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/plugins/ attrib.py", line 164, in validateAttrib if not value(key, attribs): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/plugins/ attrib.py", line 118, in eval_in_context return eval(expr, None, ContextHelper(attribs)) File "", line 1, in File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/plugins/ attrib.py", line 50, in __getitem__ return self.obj.get(name, False) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/plugins/ attrib.py", line 66, in get log.debug('Get %s from %s.%s', name, self.cls, self.method) RuntimeError: maximum recursion depth exceeded > Andrew dalke at dalkescientific.com From robert.kern at gmail.com Mon Aug 4 22:19:52 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 4 Aug 2008 21:19:52 -0500 Subject: [Numpy-discussion] running numpy tests In-Reply-To: References: <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <846D821E-BA9F-4043-AE39-D7D648013C30@dalkescientific.com> <6B6B9FA6-864D-4D31-99BB-E9AE7A93A6D3@dalkescientific.com> <1d36917a0807301351x703ee800lf96fdc94944fdd9b@mail.gmail.com> <1d36917a0807301921m70107b84s3a64077f73818410@mail.gmail.com> <22B8A6BB-2F49-4566-9B3C-086A4C2A0B8D@dalkescientific.com> <3d375d730808041700k4f093122j84f895327895fb89@mail.gmail.com> Message-ID: <3d375d730808041919habe60efy59b7f3afebf1a48c@mail.gmail.com> On Mon, Aug 4, 2008 at 20:41, Andrew Dalke wrote: > On Aug 5, 2008, at 2:00 AM, Robert Kern wrote: >> You have old stuff in your checkout/installation. Make sure you have >> deleted all of the *.pycs and directories which have been deleted in >> SVN. > > Now that I've fixed that, I can tell that I made a > mistake related to the self-test code. I can't figure > it out. > > Many of the modules have code which look like > > from testing import Tester > test = Tester().test > bench = Tester().bench > > As I understand it, this code is migrating to use nosetests. > Because people expect 'import numpy; numpy.test()' to work, > there will be a compatibility period where this API is > unchanged. > > I found that importing 'testing' costs 0.013 seconds, which > is 10% of my current import time. I would like to defer > the import until needed, so I rewrote the 'test' code as: > > def test(label='fast', verbose=1, extra_argv=None, doctests=False, > coverage=False, **kwargs): > from testing import Tester > import numpy > Tester(numpy).test(label, verbose, extra_argv, doctests, > coverage, **kwargs) > > > In my view there's no difference between them, but Tester().test does > introspection to figure out the module location. (In fact, if I > don't pass it the module explicitly then it expects that locals() > ["__file__"] for sys._getframe(-1) will exist, which is not the case > with my function. The underling code should instead check for that > variable in globals().) Probably. Or take a level= argument to tell _getframe() to go up an extra level. Or both. > I ended up with recursion errors, and I don't know why. Any idea of > what to do? Ah. My guess is that the test collector sees numpy.test() as a function that matches its regex for a unit test. It used to be a bound method, so I think nose ignored it, then. You should be able to tell nose not to collect it like so: def test(...): ... test.__test__ = False -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From schut at sarvision.nl Tue Aug 5 04:04:32 2008 From: schut at sarvision.nl (Vincent Schut) Date: Tue, 05 Aug 2008 10:04:32 +0200 Subject: [Numpy-discussion] 1.2 tasks In-Reply-To: <91cf711d0808041141s121a4322nf54ce41684149fc4@mail.gmail.com> References: <91cf711d0808041141s121a4322nf54ce41684149fc4@mail.gmail.com> Message-ID: David Huard wrote: > > > On Mon, Aug 4, 2008 at 1:45 PM, Jarrod Millman > wrote: > > Question: Should histogram raise a warning by default (new=True) to warn > users that the behaviour has changed ? Or warn only if new=False to > remind that > the old behaviour will be deprecated in 1.3 ? I think that users will > prefer being annoyed at warnings than surprised by an unexpected change, > but repeated warnings > can become a nuisance. > > To minimize annoyance, we could also offer three possibilities: > > new=None (default) : Equivalent to True, print warning about change. > new=True : Don't print warning. > new=False : Print warning about future deprecation. > > So those who have already set new=True don't get warnings, and all > others are warned. Feedback ? As a regular user of histogram I say: please warn! Your proposal above seems OK to me. I do have histogram in a lot of kind of old (and sometimes long-running) code of mine, and I certainly would prefer to be warned. Vincent. From cournape at gmail.com Tue Aug 5 05:28:18 2008 From: cournape at gmail.com (David Cournapeau) Date: Tue, 5 Aug 2008 18:28:18 +0900 Subject: [Numpy-discussion] Toward a numpy build with warning: handling unused variable In-Reply-To: References: <489587A7.7010108@ar.media.kyoto-u.ac.jp> <5b8d13220808031602x567d4855i6b961e116ff69a5b@mail.gmail.com> <5b8d13220808031639me4cd4bak319638fafb45bbac@mail.gmail.com> <1217813948.12930.15.camel@bbc8> Message-ID: <5b8d13220808050228p16956683x48b5c2ce505a670b@mail.gmail.com> On Tue, Aug 5, 2008 at 1:29 AM, Lisandro Dalcin wrote: > David, I second your approach. Furthermore, look how SWIG handles > this, it is very similar to your proposal. The difference is that SWIG > uses SWIGUNUSED for some autogenerated functions. Furthermore, it > seems the SWIG developers protected the generated code taking into > account GCC versions ;-) and the C++ case, and also Intel compilers. Yes, I would have been surprised if most compilers do not have a way to deal with that. I did not know icc handled __attribute__. Charles, are you still have strongly against this, since this is in the vein of what you suggested (tagging the argument) ? cheers, David From nadavh at visionsense.com Tue Aug 5 11:38:41 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Tue, 05 Aug 2008 18:38:41 +0300 Subject: [Numpy-discussion] Bilateral filter Message-ID: <1217950721.21145.26.camel@nadav.envision.co.il> Attached here my cython implementation of the bilateral filter, which is my first cython program. I would ask for the following: 1. Is there any way to speed up the code just by "cosmetic" modifications? 2. In particular I use the unportable gcc function __builtin_abs: Is there any way to access this in a portable way? 3. I would like to donate this code to scipy or any other suitable project. What can/should I do to realise it? Remarks: The code contains 3 end-user routines: 1. A pure python implementation: Easy to read and modify --- it can be cut out into a python source code. 2. A straight forward cython implementation: About 4 times as fast as the python implementation. 3. An optimised cython implementation earning another factor of 2 in speed (depends on the parameters used). I see this code as a "research grade" that would evolve in the near future, as I am working on a related project, and hopefully following your comments. Nadav -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- ''' A cython implementation of the plain (and slow) algorithm for bilateral filtering. Copyright 2008, Nadav Horesh nadavh at visionsense.com ''' from numpy.numarray.nd_image import generic_filter import numpy as np cdef extern from "arrayobject.h": ctypedef int intp ctypedef extern class numpy.ndarray [object PyArrayObject]: cdef char *data cdef int nd cdef intp *dimensions cdef intp *strides cdef int flags cdef extern from "math.h": double exp(double x) ## gcc specific (?) cdef extern: int __builtin_abs(int x) cdef int GAUSS_SAMP = 32 cdef int GAUSS_IDX_MAX = GAUSS_SAMP - 1 class _Bilat_fcn(object): ''' The class provides the bilaterl filter function to be called by generic_filter. initialization parameters: spat_sig: The sigma of the spatial Gaussian filter inten_sig: The sigma of the gray-levels Gaussian filter filter_size: (int) The size of the spatial convolution kernel. If not set, it is set to ~ 4*spat_sig. ''' def __init__(self, spat_sig, inten_sig, filter_size=None): if filter_size is not None and filter_size >= 2: self.xy_size = int(filter_size) else: self.xy_size = int(round(spat_sig*4)) # Make filter size odd self.xy_size += 1-self.xy_size%2 x = np.arange(self.xy_size, dtype=float) x = (x-x.mean())**2 #xy_ker: Spatial convolution kernel self.xy_ker = np.exp(-np.add.outer(x,x)/(2*spat_sig**2)).ravel() self.xy_ker /= self.xy_ker.sum() self.inten_sig = 2 * inten_sig**2 self.index = self.xy_size**2 // 2 ## An initialization for LUT instead of a Gaussian function call ## (for the fc_filter method) x = np.linspace(0,3.0, GAUSS_SAMP) self.gauss_lut = np.exp(-x**2/2) self.gauss_lut[-1] = 0.0 # Nullify all distant deltas self.x_quant = 3*inten_sig / GAUSS_IDX_MAX ## ## Filtering functions ## def __call__ (self, data): 'An unoptimized (pure python) implementation' weight = np.exp(-(data-data[self.index])**2/self.inten_sig) * self.xy_ker return np.dot(data, weight) / weight.sum() def cfilter(self, ndarray data): 'An optimized implementation' cdef ndarray kernel = self.xy_ker cdef double sigma = self.inten_sig cdef double weight_i, weight, result, centre, dat_i cdef double *pdata=data.data, *pker=kernel.data cdef int i, dim = data.dimensions[0] centre = pdata[self.index] weight = 0.0 result = 0.0 for i from 0 <= i < dim: dat_i = pdata[i] weight_i = exp(-(dat_i-centre)**2 / sigma) * pker[i] weight += weight_i; result += dat_i * weight_i return result / weight def fc_filter(self, ndarray data): 'Use further optimisation by replacing exp functions calls by a LUT' cdef ndarray kernel = self.xy_ker cdef ndarray gauss_lut_arr = self.gauss_lut cdef double sigma = self.inten_sig cdef double weight_i, weight, result, centre, dat_i cdef double *pdata=data.data, *pker=kernel.data cdef int i, dim = data.dimensions[0] cdef int exp_i # Entry index for the LUT cdef double x_quant = self.x_quant cdef double *gauss_lut = gauss_lut_arr.data centre = pdata[self.index] weight = 0.0 result = 0.0 for i from 0 <= i < dim: dat_i = pdata[i] exp_i = __builtin_abs(((dat_i-centre) / x_quant)) if exp_i > GAUSS_IDX_MAX: exp_i = GAUSS_IDX_MAX weight_i = gauss_lut[exp_i] * pker[i] weight += weight_i; result += dat_i * weight_i return result / weight # # Filtering functions to be called from outsize # def bilateral(mat, xy_sig, z_sig, filter_size=None, mode='reflect'): ''' Bilateral filter a 2D array. mat: A 2D array xy_sig: The sigma of the spatial Gaussian filter. z_sig: The sigma of the gray levels Gaussian filter. filter_size: Sise of the spatial filter kernel: None of values < 2 --- auto select. mode: See numpy.numarray.nd_image.generic_filter documentation output: A 2D array of the same dimensions as mat and a float64 dtype Remarks: 1. The spatial filter kernel size is ~4*xy_sig, unless specified otherwise 2. The algorithm is slow but has a minimal memory footprint ''' filter_fcn = _Bilat_fcn(xy_sig, z_sig, filter_size) size = filter_fcn.xy_size return generic_filter(mat, filter_fcn.cfilter, size =(size,size), mode=mode) def bilateral_slow(mat, xy_sig, z_sig, filter_size=None, mode='reflect'): 'A pure python implementation of the bilateral filter, for details see bilateral doc.' filter_fcn = _Bilat_fcn(xy_sig, z_sig, filter_size) size = filter_fcn.xy_size return generic_filter(mat, filter_fcn, size =(size,size), mode=mode) def bilateral_fast(mat, xy_sig, z_sig, filter_size=None, mode='reflect'): 'A fast implementation of bilateral filter, for details see bilateral doc.' filter_fcn = _Bilat_fcn(xy_sig, z_sig, filter_size) size = filter_fcn.xy_size return generic_filter(mat, filter_fcn.fc_filter, size =(size,size), mode=mode) From charlesr.harris at gmail.com Tue Aug 5 08:49:46 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 5 Aug 2008 07:49:46 -0500 Subject: [Numpy-discussion] Toward a numpy build with warning: handling unused variable In-Reply-To: <5b8d13220808050228p16956683x48b5c2ce505a670b@mail.gmail.com> References: <489587A7.7010108@ar.media.kyoto-u.ac.jp> <5b8d13220808031602x567d4855i6b961e116ff69a5b@mail.gmail.com> <5b8d13220808031639me4cd4bak319638fafb45bbac@mail.gmail.com> <1217813948.12930.15.camel@bbc8> <5b8d13220808050228p16956683x48b5c2ce505a670b@mail.gmail.com> Message-ID: On Tue, Aug 5, 2008 at 4:28 AM, David Cournapeau wrote: > On Tue, Aug 5, 2008 at 1:29 AM, Lisandro Dalcin wrote: > > David, I second your approach. Furthermore, look how SWIG handles > > this, it is very similar to your proposal. The difference is that SWIG > > uses SWIGUNUSED for some autogenerated functions. Furthermore, it > > seems the SWIG developers protected the generated code taking into > > account GCC versions ;-) and the C++ case, and also Intel compilers. > > Yes, I would have been surprised if most compilers do not have a way > to deal with that. I did not know icc handled __attribute__. > > Charles, are you still have strongly against this, since this is in > the vein of what you suggested (tagging the argument) ? > I think tagging the argument is the way to go. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalke at dalkescientific.com Tue Aug 5 09:24:17 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Tue, 5 Aug 2008 15:24:17 +0200 Subject: [Numpy-discussion] running numpy tests In-Reply-To: <3d375d730808041919habe60efy59b7f3afebf1a48c@mail.gmail.com> References: <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <846D821E-BA9F-4043-AE39-D7D648013C30@dalkescientific.com> <6B6B9FA6-864D-4D31-99BB-E9AE7A93A6D3@dalkescientific.com> <1d36917a0807301351x703ee800lf96fdc94944fdd9b@mail.gmail.com> <1d36917a0807301921m70107b84s3a64077f73818410@mail.gmail.com> <22B8A6BB-2F49-4566-9B3C-086A4C2A0B8D@dalkescientific.com> <3d375d730808041700k4f093122j84f895327895fb89@mail.gmail.com> <3d375d730808041919habe60efy59b7f3afebf1a48c@mail.gmail.com> Message-ID: On Aug 5, 2008, at 4:19 AM, Robert Kern wrote: > def test(...): > ... > test.__test__ = False That did it - thanks! Does "import numpy; numpy.bench()" work for anyone? When I try it I get [josiah:~] dalke% python -c 'import numpy; numpy.bench()' ---------------------------------------------------------------------- Ran 0 tests in 0.003s OK I can go ahead and remove those if they don't work for anyone. Andrew dalke at dalkescientific.com From alan.mcintyre at gmail.com Tue Aug 5 09:53:41 2008 From: alan.mcintyre at gmail.com (Alan McIntyre) Date: Tue, 5 Aug 2008 09:53:41 -0400 Subject: [Numpy-discussion] running numpy tests In-Reply-To: References: <6B6B9FA6-864D-4D31-99BB-E9AE7A93A6D3@dalkescientific.com> <1d36917a0807301351x703ee800lf96fdc94944fdd9b@mail.gmail.com> <1d36917a0807301921m70107b84s3a64077f73818410@mail.gmail.com> <22B8A6BB-2F49-4566-9B3C-086A4C2A0B8D@dalkescientific.com> <3d375d730808041700k4f093122j84f895327895fb89@mail.gmail.com> <3d375d730808041919habe60efy59b7f3afebf1a48c@mail.gmail.com> Message-ID: <1d36917a0808050653r71bac08au94537b50fbb4527d@mail.gmail.com> At the moment, bench() doesn't work. That's something I'll try to look at this week, but from Friday until the 15th I'm going to be driving most of the time and may not get as much done as I'd like. On 8/5/08, Andrew Dalke wrote: > On Aug 5, 2008, at 4:19 AM, Robert Kern wrote: >> def test(...): >> ... >> test.__test__ = False > > That did it - thanks! > > Does "import numpy; numpy.bench()" work for anyone? When I try it I get > > > [josiah:~] dalke% python -c 'import numpy; numpy.bench()' > > ---------------------------------------------------------------------- > Ran 0 tests in 0.003s > > OK > > I can go ahead and remove those if they don't work for anyone. > > Andrew > dalke at dalkescientific.com > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From dalcinl at gmail.com Tue Aug 5 10:10:38 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 5 Aug 2008 11:10:38 -0300 Subject: [Numpy-discussion] Toward a numpy build with warning: handling unused variable In-Reply-To: References: <489587A7.7010108@ar.media.kyoto-u.ac.jp> <5b8d13220808031602x567d4855i6b961e116ff69a5b@mail.gmail.com> <5b8d13220808031639me4cd4bak319638fafb45bbac@mail.gmail.com> <1217813948.12930.15.camel@bbc8> <5b8d13220808050228p16956683x48b5c2ce505a670b@mail.gmail.com> Message-ID: Of course, you could also call GCC like this '-Wall -Wno-unused-parameter'. Then you will only get warnings about unused functions and local variables. On Tue, Aug 5, 2008 at 9:49 AM, Charles R Harris wrote: > > > On Tue, Aug 5, 2008 at 4:28 AM, David Cournapeau wrote: >> >> On Tue, Aug 5, 2008 at 1:29 AM, Lisandro Dalcin wrote: >> > David, I second your approach. Furthermore, look how SWIG handles >> > this, it is very similar to your proposal. The difference is that SWIG >> > uses SWIGUNUSED for some autogenerated functions. Furthermore, it >> > seems the SWIG developers protected the generated code taking into >> > account GCC versions ;-) and the C++ case, and also Intel compilers. >> >> Yes, I would have been surprised if most compilers do not have a way >> to deal with that. I did not know icc handled __attribute__. >> >> Charles, are you still have strongly against this, since this is in >> the vein of what you suggested (tagging the argument) ? > > > > I think tagging the argument is the way to go. > > Chuck > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From david.huard at gmail.com Tue Aug 5 11:48:10 2008 From: david.huard at gmail.com (David Huard) Date: Tue, 5 Aug 2008 11:48:10 -0400 Subject: [Numpy-discussion] 1.2 tasks In-Reply-To: References: <91cf711d0808041141s121a4322nf54ce41684149fc4@mail.gmail.com> Message-ID: <91cf711d0808050848x140e3a4eq2c8dc1aacecb78b6@mail.gmail.com> On Tue, Aug 5, 2008 at 4:04 AM, Vincent Schut wrote: > David Huard wrote: > > > > > > On Mon, Aug 4, 2008 at 1:45 PM, Jarrod Millman > > wrote: > > > > > > Question: Should histogram raise a warning by default (new=True) to warn > > users that the behaviour has changed ? Or warn only if new=False to > > remind that > > the old behaviour will be deprecated in 1.3 ? I think that users will > > prefer being annoyed at warnings than surprised by an unexpected change, > > but repeated warnings > > can become a nuisance. > > > > To minimize annoyance, we could also offer three possibilities: > > > > new=None (default) : Equivalent to True, print warning about change. > > new=True : Don't print warning. > > new=False : Print warning about future deprecation. > > > > So those who have already set new=True don't get warnings, and all > > others are warned. Feedback ? > > As a regular user of histogram I say: please warn! Your proposal above > seems OK to me. I do have histogram in a lot of kind of old (and > sometimes long-running) code of mine, and I certainly would prefer to be > warned. > > Vincent. Thanks for the feedback. Here is what will be printed: If new=False The original semantics of histogram is scheduled to be deprecated in NumPy 1.3. The new semantics fixes long-standing issues with outliers handling. The main changes concern 1. the definition of the bin edges, now including the rightmost edge, and 2. the handling of upper outliers, now ignored rather than tallied in the rightmost bin. Please read the docstring for more information. If new=None (default) The semantics of histogram has been modified in the current release to fix long-standing issues with outliers handling. The main changes concern 1. the definition of the bin edges, now including the rightmost edge, and 2. the handling of upper outliers, now ignored rather than tallied in the rightmost bin. The previous behaviour is still accessible using `new=False`, but is scheduled to be deprecated in the next release (1.3). *This warning will not printed in the 1.3 release.* Please read the docstring for more information. I modified the docstring to put the emphasis on the new semantics, adapted the tests and updated the ticket. David > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalke at dalkescientific.com Tue Aug 5 11:51:25 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Tue, 5 Aug 2008 17:51:25 +0200 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <3d375d730808041604j39c4cebfif40ca57ba803069c@mail.gmail.com> References: <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <20080801165351.GC24781@phare.normalesup.org> <3d375d730808011249h29460aavd39f3b0f8874c361@mail.gmail.com> <48939835.4080801@noaa.gov> <48939973.4010008@noaa.gov> <4893EB4A.6050905@ar.media.kyoto-u.ac.jp> <3d375d730808012313t66a96137l423856f9ed5f1448@mail.gmail.com> <48975765.9020301@noaa.gov> <3d375d730808041326sb6bece9q7fd0a8c4efe5cd08@mail.gmail.com> <48978A38.6010503@noaa.gov> <3d375d730808041604j39c4cebfif40ca57ba803069c@mail.gmail.com> Message-ID: I just added ticket 874 http://scipy.org/scipy/numpy/ticket/874 which on my machine takes the import time from 0.15 seconds down to 0.093 seconds. A bit over a month ago it was about 0.33 seconds. :) The biggest trick I didn't apply was to defer importing the re module, and only compile the patterns when they are needed. That would save about 0.013 seconds, but the result would be rather fragile as people rarely consider deferring pattern definitions. I'll submit a patch for it, expected it to be rejected. ;) I noticed that two places import the standard Python math library, which doesn't seem necessary: numpy/lib/__init__.py: import math __all__ = ['emath','math'] 'math' is never used in that module. It does mean that 'numpy.math' is the same as Python's math module, which I didn't expect. I didn't understand why it was explicitly in the __all_ list so I didn't change it. All tests pass without it. numpy/lib/index_tricks.py: import math .. size.append(math.ceil((key[k].stop - start)/ (step*1.0))) Wouldn't numpy.ceil be okay here? All tests pass when I use numpy.ceil. This would save about 0.002 seconds, so it's a small improvement. I think there's a subtle and minor bug in _datasource. def __del__(self): # Remove temp directories if self._istmpdest: rmtree(self._destpath) Is 'rmtree' guaranteed to be present in the module scope if the object is garbage collected during Python shutdown? Andrew dalke at dalkescientific.com From dalke at dalkescientific.com Tue Aug 5 11:54:50 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Tue, 5 Aug 2008 17:54:50 +0200 Subject: [Numpy-discussion] running numpy tests In-Reply-To: <1d36917a0808050653r71bac08au94537b50fbb4527d@mail.gmail.com> References: <6B6B9FA6-864D-4D31-99BB-E9AE7A93A6D3@dalkescientific.com> <1d36917a0807301351x703ee800lf96fdc94944fdd9b@mail.gmail.com> <1d36917a0807301921m70107b84s3a64077f73818410@mail.gmail.com> <22B8A6BB-2F49-4566-9B3C-086A4C2A0B8D@dalkescientific.com> <3d375d730808041700k4f093122j84f895327895fb89@mail.gmail.com> <3d375d730808041919habe60efy59b7f3afebf1a48c@mail.gmail.com> <1d36917a0808050653r71bac08au94537b50fbb4527d@mail.gmail.com> Message-ID: <6E30CF4C-E5F8-490D-87C7-7BB263C94D40@dalkescientific.com> On Aug 5, 2008, at 3:53 PM, Alan McIntyre wrote: > At the moment, bench() doesn't work. That's something I'll try to > look at this week, but from Friday until the 15th I'm going to be > driving most of the time and may not get as much done as I'd like. Thanks for the confirmation. The import speedup patch I just submitted keeps the 'bench' definitions there (including one that was missing). But instead of defining it as a bound method, I used functions that import the testing submodule and construct/call the right objects. It should behave the same. I think. Andrew dalke at dalkescientific.com From cimrman3 at ntc.zcu.cz Tue Aug 5 12:00:08 2008 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Tue, 05 Aug 2008 18:00:08 +0200 Subject: [Numpy-discussion] member1d and unique elements In-Reply-To: References: Message-ID: <48987908.9080703@ntc.zcu.cz> Greg Novak wrote: > I have two arrays of integers, and would like to know _where_ they > have elements in common, not just _which_ elements are in common. > This is because the entries in the integer array are aligned with > other arrays. This seems very close to what member1d advertises as > its function. However, member1d says that it expects arrays with only > unique elements. > > First of all, my desired operation is well-posed: I'd like f(ar1, > ar2) to return something in the shape of ar1 with True if the value at > that position appears anywhere in ar2 (regardless of duplication) and > False otherwise. > > So I looked at the code and have two questions: > 1) What is this code trying to achieve? > aux = perm[ii+1] > perm[ii+1] = perm[ii] > perm[ii] = aux > > Here perm is the stable argsort of the two concatenated arguments: > perm = concatenate((ar1, ar2)).argsort(kind='mergesort'). > arr is the array of combined inputs in sorted order: > arr = concatenate((ar1, ar2))[perm] > and ii is a list of indices into arr where the value of arr is equal > to the next value in the array (arr[ii] == arr[ii+1]) _and_ arr[ii] > came from the _second_ input (ar2). > > Now, this last bit (looking for elements of arr that are equal and > both came from the second array) is clearly trying to deal with > duplication, which is why I'm interested... > > So, the code snippet is trying to swap perm[ii+1] with perm[ii], but I > don't see why. Furthermore, there are funny results if a value is > duplicated three times, not just twice -- perm is no longer a > permutation vector. Eg, member1d([1], [2,2,2]) results perm=[0,1,2,3] > and ii=[1,2] before the above snippet, and the above snippet makes > perm into [0,2,3,2] > > I've commented those three lines, and I've never seen any changes to > the output of member1d. The new value of perm is used to compute the > expression: perm.argsort(kind='mergesort')[:len( ar1 )], but the > changes to that expression as a result of the above three lines are > always at the high end of the array, which is sliced off by the last > [:len(ar1)]. > > Finally, my second question is: > 2) Does anyone have a test case where member1d fails as a result of > duplicates in the input? So far I haven't found any, with the above > lines commented or not. > > Upon reflection and review of the changelog, another theory occurs to > me: member1d did not originally use a stable sort. What I've written > above for interpretation of the value ii (indicates duplication within > ar2) is true for a stable sort, but for an unstable sort the same > condition has the interpretation that ii holds the values where the > sorting algorithm swapped the order of equal values unstably. Then > the code snippet in question 1) looks like an attempt to swap those > values in the permutation array to make the sort stable again. The > attempt would fail if there was duplication in either array. > > So, I would propose deleting those three lines (since they seem to be > a non-functional relic) and declaring in the docstring that member1d > doesn't require unique elements. > > Also, if this is correct, then the function simplifies considerably > since several values don't need to be computed anymore: > > def setmember1d( ar1, ar2 ): > ar = nm.concatenate( (ar1, ar2 ) ) > perm = ar.argsort(kind='mergesort') > aux = ar[perm] > flag = nm.concatenate( (aux[1:] == aux[:-1], [False] ) ) > indx = perm.argsort(kind='mergesort')[:len( ar1 )] > return flag[indx] > > Corrections to the above are welcome since I'm going to start using > member1d without regard for uniqueness, and I'd like to know if I'm > making a big mistake... Hi Greg, I do not have much time to investigate it in detail right now, but it does not work for repeated entries in ar1: In [14]: nm.setmember1d( [1,2,3,2], [1, 3] ) Out[14]: array([ True, True, True, False], dtype=bool) thanks for trying to enhance arraysetops! r. From zachary.pincus at yale.edu Tue Aug 5 12:29:53 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Tue, 5 Aug 2008 10:29:53 -0600 Subject: [Numpy-discussion] Bilateral filter In-Reply-To: <1217950721.21145.26.camel@nadav.envision.co.il> References: <1217950721.21145.26.camel@nadav.envision.co.il> Message-ID: > Attached here my cython implementation of the bilateral filter, > which is my first cython program. I would ask for the following: Thanks for the code! I know it will be of use to me. (Do you place any particular license on it?) Zach On Aug 5, 2008, at 9:38 AM, Nadav Horesh wrote: > Attached here my cython implementation of the bilateral filter, > which is my first cython program. I would ask for the following: > > ? Is there any way to speed up the code just by "cosmetic" > modifications? > ? In particular I use the unportable gcc function __builtin_abs: Is > there any way to access this in a portable way? > ? I would like to donate this code to scipy or any other suitable > project. What can/should I do to realise it? > > Remarks: > > The code contains 3 end-user routines: > ? A pure python implementation: Easy to read and modify --- it can > be cut out into a python source code. > ? A straight forward cython implementation: About 4 times as fast > as the python implementation. > ? An optimised cython implementation earning another factor of 2 in > speed (depends on the parameters used). > I see this code as a "research grade" that would evolve in the near > future, as I am working on a related project, and hopefully > following your comments. > > Nadav > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From millman at berkeley.edu Tue Aug 5 13:18:27 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 5 Aug 2008 10:18:27 -0700 Subject: [Numpy-discussion] 1.2 tasks In-Reply-To: <91cf711d0808050848x140e3a4eq2c8dc1aacecb78b6@mail.gmail.com> References: <91cf711d0808041141s121a4322nf54ce41684149fc4@mail.gmail.com> <91cf711d0808050848x140e3a4eq2c8dc1aacecb78b6@mail.gmail.com> Message-ID: On Tue, Aug 5, 2008 at 8:48 AM, David Huard wrote: > Thanks for the feedback. Here is what will be printed: > > If new=False > > The original semantics of histogram is scheduled to be > deprecated in NumPy 1.3. The new semantics fixes > long-standing issues with outliers handling. The main > changes concern > 1. the definition of the bin edges, > now including the rightmost edge, and > 2. the handling of upper outliers, > now ignored rather than tallied in the rightmost bin. > > Please read the docstring for more information. > > > > If new=None (default) > > The semantics of histogram has been modified in > the current release to fix long-standing issues with > outliers handling. The main changes concern > 1. the definition of the bin edges, > now including the rightmost edge, and > 2. the handling of upper outliers, now ignored rather > than tallied in the rightmost bin. > The previous behaviour is still accessible using > `new=False`, but is scheduled to be deprecated in the > next release (1.3). > > *This warning will not printed in the 1.3 release.* > > Please read the docstring for more information. Thanks for taking care of this. I thought that we were going to remove the new parameter in the 1.3 release. Is that still the plan? If so, shouldn't the warning state "will be removed in the next minor release (1.3)" rather than "is scheduled to be deprecated in the next release (1.3)"? In my mind the old behavior is deprecated in this release (1.2). The 1.2 release will be longer lived (~6 months) than the 1.1 release and I anticipate several bugfix releases (1.2.1, 1.2.2, 1.2.3, etc). So I think it is reasonable to just remove the old behavior in the 1.3 release. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From doutriaux1 at llnl.gov Tue Aug 5 13:20:10 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Tue, 05 Aug 2008 10:20:10 -0700 Subject: [Numpy-discussion] 64bit issue? Message-ID: <48988BCA.9030203@llnl.gov> Hello I'm running into some strange error on a 64bit machine, I tracked it down to this line returning a NULL pointer, any idea why is this? I tried both numpy1.1.1 and what in trunk, numpy.test() passes for both Ok here's the uname of the machine and the offending line: Linux quartic.txcorp.com 2.6.20-1.2320.fc5 #1 SMP Tue Jun 12 18:50:49 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux array = (PyArrayObject *)PyArray_SimpleNew(d, dims, self->type); where d is 3 and dims: 120,46,72 self->type is 11 it seems to pass with d=1, but i'm not 100% positive. Any idea on why it could be failing? Thanks, C. PS: just in case here are numpy.test results: >>> import numpy >>> numpy.test() Numpy is installed in /home/facets/doutriaux1/cdat/latest/lib/python2.5/site-packages/numpy Numpy version 1.1.1 Python version 2.5.2 (r252:60911, Aug 4 2008, 13:47:12) [GCC 4.1.1 20070105 (Red Hat 4.1.1-51)] Found 3/3 tests for numpy.core.tests.test_memmap Found 145/145 tests for numpy.core.tests.test_regression Found 12/12 tests for numpy.core.tests.test_records Found 3/3 tests for numpy.core.tests.test_errstate Found 72/72 tests for numpy.core.tests.test_numeric Found 36/36 tests for numpy.core.tests.test_numerictypes Found 290/290 tests for numpy.core.tests.test_multiarray Found 18/18 tests for numpy.core.tests.test_defmatrix Found 63/63 tests for numpy.core.tests.test_unicode Found 16/16 tests for numpy.core.tests.test_umath Found 7/7 tests for numpy.core.tests.test_scalarmath Found 2/2 tests for numpy.core.tests.test_ufunc Found 5/5 tests for numpy.distutils.tests.test_misc_util Found 4/4 tests for numpy.distutils.tests.test_fcompiler_gnu Found 2/2 tests for numpy.fft.tests.test_fftpack Found 3/3 tests for numpy.fft.tests.test_helper Found 15/15 tests for numpy.lib.tests.test_twodim_base Found 1/1 tests for numpy.lib.tests.test_machar Found 1/1 tests for numpy.lib.tests.test_regression Found 43/43 tests for numpy.lib.tests.test_type_check Found 1/1 tests for numpy.lib.tests.test_ufunclike Found 15/15 tests for numpy.lib.tests.test_io Found 25/25 tests for numpy.lib.tests.test__datasource Found 10/10 tests for numpy.lib.tests.test_arraysetops Found 1/1 tests for numpy.lib.tests.test_financial Found 4/4 tests for numpy.lib.tests.test_polynomial Found 6/6 tests for numpy.lib.tests.test_index_tricks Found 49/49 tests for numpy.lib.tests.test_shape_base Found 55/55 tests for numpy.lib.tests.test_function_base Found 5/5 tests for numpy.lib.tests.test_getlimits Found 3/3 tests for numpy.linalg.tests.test_regression Found 89/89 tests for numpy.linalg.tests.test_linalg Found 96/96 tests for numpy.ma.tests.test_core Found 19/19 tests for numpy.ma.tests.test_mrecords Found 15/15 tests for numpy.ma.tests.test_extras Found 4/4 tests for numpy.ma.tests.test_subclassing Found 36/36 tests for numpy.ma.tests.test_old_ma Found 1/1 tests for numpy.oldnumeric.tests.test_oldnumeric Found 7/7 tests for numpy.tests.test_random Found 16/16 tests for numpy.testing.tests.test_utils Found 6/6 tests for numpy.tests.test_ctypeslib ............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ ---------------------------------------------------------------------- Ran 1292 tests in 1.237s OK and: >>> import numpy >>> numpy.test() Running unit tests for numpy NumPy version 1.2.0.dev5611 NumPy is installed in /home/facets/doutriaux1/cdat/latest/lib/python2.5/site-packages/numpy Python version 2.5.2 (r252:60911, Aug 4 2008, 13:47:12) [GCC 4.1.1 20070105 (Red Hat 4.1.1-51)] nose version 0.10.3 ................................................................................................................................................................................................................................................................................................................................................................................................................................S.SSS.SSSSSS.SS..S.SSSSSS.SS.S.SSSS.SS.SSS.SSS.SS.S..........SS.SSS.S..S..................S......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... ---------------------------------------------------------------------- Ran 1924 tests in 7.046s OK (SKIP=45) From millman at berkeley.edu Tue Aug 5 13:20:25 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 5 Aug 2008 10:20:25 -0700 Subject: [Numpy-discussion] Bilateral filter In-Reply-To: References: <1217950721.21145.26.camel@nadav.envision.co.il> Message-ID: On Tue, Aug 5, 2008 at 9:29 AM, Zachary Pincus wrote: >> Attached here my cython implementation of the bilateral filter, >> which is my first cython program. I would ask for the following: > > Thanks for the code! I know it will be of use to me. (Do you place any > particular license on it?) It would be great if you would use the new BSD license. If you want it included in SciPy, you will need to use that license anyway. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From stefan at sun.ac.za Tue Aug 5 13:24:36 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 5 Aug 2008 19:24:36 +0200 Subject: [Numpy-discussion] 1.2 tasks In-Reply-To: References: <91cf711d0808041141s121a4322nf54ce41684149fc4@mail.gmail.com> <91cf711d0808050848x140e3a4eq2c8dc1aacecb78b6@mail.gmail.com> Message-ID: <9457e7c80808051024u1a9fe26axcb2865beed869c52@mail.gmail.com> 2008/8/5 Jarrod Millman : >> If new=None (default) Could you put in a check for new=True, and suppress those messages? A user that knows about the changes wouldn't want to see anything. Regards St?fan From millman at berkeley.edu Tue Aug 5 13:36:28 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 5 Aug 2008 10:36:28 -0700 Subject: [Numpy-discussion] 1.2 tasks In-Reply-To: <9457e7c80808051024u1a9fe26axcb2865beed869c52@mail.gmail.com> References: <91cf711d0808041141s121a4322nf54ce41684149fc4@mail.gmail.com> <91cf711d0808050848x140e3a4eq2c8dc1aacecb78b6@mail.gmail.com> <9457e7c80808051024u1a9fe26axcb2865beed869c52@mail.gmail.com> Message-ID: On Tue, Aug 5, 2008 at 10:24 AM, St?fan van der Walt wrote: > Could you put in a check for new=True, and suppress those messages? A > user that knows about the changes wouldn't want to see anything. Yes, that is all ready available. Maybe the warning message for 'new=None' should mention this, though. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From novak at ucolick.org Tue Aug 5 13:46:32 2008 From: novak at ucolick.org (Greg Novak) Date: Tue, 5 Aug 2008 10:46:32 -0700 Subject: [Numpy-discussion] member1d and unique elements In-Reply-To: <48987908.9080703@ntc.zcu.cz> References: <48987908.9080703@ntc.zcu.cz> Message-ID: Argh. I could swear that yesterday I typed test cases just like the one you provide, and it behaved correctly. Nevertheless, it clearly fails in spite of my memory, so attached is a version which I believe gives the correct behavior. Greg On Tue, Aug 5, 2008 at 9:00 AM, Robert Cimrman wrote: > I do not have much time to investigate it in detail right now, but it > does not work for repeated entries in ar1: > > In [14]: nm.setmember1d( [1,2,3,2], [1, 3] ) > Out[14]: array([ True, True, True, False], dtype=bool) -------------- next part -------------- A non-text attachment was scrubbed... Name: setmember.py Type: text/x-python Size: 2459 bytes Desc: not available URL: From nadavh at visionsense.com Tue Aug 5 13:42:50 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Tue, 5 Aug 2008 20:42:50 +0300 Subject: [Numpy-discussion] Bilateral filter References: <1217950721.21145.26.camel@nadav.envision.co.il> Message-ID: <710F2847B0018641891D9A216027636029C204@ex3.envision.co.il> As much as I know, the python's community is bound to the BSD style licence. I was hoping that the code would be included in scipy, after I would attach the required declaration (I do not know which), and maybe with some adjustment to the coding style. In short, for the next few days I am waiting for comments and responses that would enable me to donate the code to an existing project (scipy? pyvision?). Meanwhile I'll improve the co a bit. If you can't wait, and not willing to use the code without a licence statement, I'll repost the code with the scipy's licence declaration. Nadav. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Zachary Pincus ????: ? 05-??????-08 19:29 ??: Discussion of Numerical Python ????: Re: [Numpy-discussion] Bilateral filter > Attached here my cython implementation of the bilateral filter, > which is my first cython program. I would ask for the following: Thanks for the code! I know it will be of use to me. (Do you place any particular license on it?) Zach On Aug 5, 2008, at 9:38 AM, Nadav Horesh wrote: > Attached here my cython implementation of the bilateral filter, > which is my first cython program. I would ask for the following: > > ? Is there any way to speed up the code just by "cosmetic" > modifications? > ? In particular I use the unportable gcc function __builtin_abs: Is > there any way to access this in a portable way? > ? I would like to donate this code to scipy or any other suitable > project. What can/should I do to realise it? > > Remarks: > > The code contains 3 end-user routines: > ? A pure python implementation: Easy to read and modify --- it can > be cut out into a python source code. > ? A straight forward cython implementation: About 4 times as fast > as the python implementation. > ? An optimised cython implementation earning another factor of 2 in > speed (depends on the parameters used). > I see this code as a "research grade" that would evolve in the near > future, as I am working on a related project, and hopefully > following your comments. > > Nadav > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3928 bytes Desc: not available URL: From david.huard at gmail.com Tue Aug 5 13:58:30 2008 From: david.huard at gmail.com (David Huard) Date: Tue, 5 Aug 2008 13:58:30 -0400 Subject: [Numpy-discussion] 1.2 tasks In-Reply-To: References: <91cf711d0808041141s121a4322nf54ce41684149fc4@mail.gmail.com> <91cf711d0808050848x140e3a4eq2c8dc1aacecb78b6@mail.gmail.com> Message-ID: <91cf711d0808051058m768ac1b6k4f55cde7beb09bef@mail.gmail.com> On Tue, Aug 5, 2008 at 1:18 PM, Jarrod Millman wrote: > On Tue, Aug 5, 2008 at 8:48 AM, David Huard wrote: > > Thanks for the feedback. Here is what will be printed: > > > > If new=False > > > > The original semantics of histogram is scheduled to be > > deprecated in NumPy 1.3. The new semantics fixes > > long-standing issues with outliers handling. The main > > changes concern > > 1. the definition of the bin edges, > > now including the rightmost edge, and > > 2. the handling of upper outliers, > > now ignored rather than tallied in the rightmost bin. > > > > Please read the docstring for more information. > > > > > > > > If new=None (default) > > > > The semantics of histogram has been modified in > > the current release to fix long-standing issues with > > outliers handling. The main changes concern > > 1. the definition of the bin edges, > > now including the rightmost edge, and > > 2. the handling of upper outliers, now ignored rather > > than tallied in the rightmost bin. > > The previous behaviour is still accessible using > > `new=False`, but is scheduled to be deprecated in the > > next release (1.3). > > > > *This warning will not printed in the 1.3 release.* > > > > Please read the docstring for more information. > > Thanks for taking care of this. I thought that we were going to > remove the new parameter in the 1.3 release. Is that still the plan? > If so, shouldn't the warning state "will be removed in the next minor > release (1.3)" rather than "is scheduled to be deprecated in the next > release (1.3)"? In my mind the old behavior is deprecated in this > release (1.2). > The roadmap that I propose is the following: 1.1 we warn about upcoming change, (new=False) 1.2 we make that change, (new=None) + warnings 1.3 we deprecate the old behaviour (new=True), no warnings. 1.4 remove the old behavior and remove the new keyword. It's pretty much the roadmap exposed in the related ticket that I wrote following discussions on the ML. This leaves plenty of time for people to make their changes, and my guess is that a lot of people will appreciate this, given that you were asked to delay the changes to histogram. > The 1.2 release will be longer lived (~6 months) than the 1.1 release > and I anticipate several bugfix releases (1.2.1, 1.2.2, 1.2.3, etc). > So I think it is reasonable to just remove the old behavior in the 1.3 > release. > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.huard at gmail.com Tue Aug 5 14:01:31 2008 From: david.huard at gmail.com (David Huard) Date: Tue, 5 Aug 2008 14:01:31 -0400 Subject: [Numpy-discussion] 1.2 tasks In-Reply-To: References: <91cf711d0808041141s121a4322nf54ce41684149fc4@mail.gmail.com> <91cf711d0808050848x140e3a4eq2c8dc1aacecb78b6@mail.gmail.com> <9457e7c80808051024u1a9fe26axcb2865beed869c52@mail.gmail.com> Message-ID: <91cf711d0808051101u722e505axd49cfcc231140159@mail.gmail.com> On Tue, Aug 5, 2008 at 1:36 PM, Jarrod Millman wrote: > On Tue, Aug 5, 2008 at 10:24 AM, St?fan van der Walt > wrote: > > Could you put in a check for new=True, and suppress those messages? A > > user that knows about the changes wouldn't want to see anything. > > Yes, that is all ready available. Maybe the warning message for > 'new=None' should mention this, though. > Done > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Aug 5 14:05:53 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 5 Aug 2008 13:05:53 -0500 Subject: [Numpy-discussion] 64bit issue? In-Reply-To: <48988BCA.9030203@llnl.gov> References: <48988BCA.9030203@llnl.gov> Message-ID: On Tue, Aug 5, 2008 at 12:20 PM, Charles Doutriaux wrote: > Hello I'm running into some strange error on a 64bit machine, > I tracked it down to this line returning a NULL pointer, any idea why is > this? > I tried both numpy1.1.1 and what in trunk, numpy.test() passes for both > > Ok here's the uname of the machine and the offending line: > > Linux quartic.txcorp.com 2.6.20-1.2320.fc5 #1 SMP Tue Jun 12 18:50:49 > EDT 2007 x86_64 x86_64 x86_64 GNU/Linux > > array = (PyArrayObject *)PyArray_SimpleNew(d, dims, self->type); > > where d is 3 and dims: 120,46,72 > self->type is 11 > > it seems to pass with d=1, but i'm not 100% positive. > > Any idea on why it could be failing? > What is the type of dims? Is there also a problem with a 32 bit OS? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Tue Aug 5 14:13:05 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 05 Aug 2008 11:13:05 -0700 Subject: [Numpy-discussion] "import numpy" is slow In-Reply-To: <3d375d730808041604j39c4cebfif40ca57ba803069c@mail.gmail.com> References: <3d375d730807030006v2a58860eodb6022454c8e06d6@mail.gmail.com> <20080801165351.GC24781@phare.normalesup.org> <3d375d730808011249h29460aavd39f3b0f8874c361@mail.gmail.com> <48939835.4080801@noaa.gov> <48939973.4010008@noaa.gov> <4893EB4A.6050905@ar.media.kyoto-u.ac.jp> <3d375d730808012313t66a96137l423856f9ed5f1448@mail.gmail.com> <48975765.9020301@noaa.gov> <3d375d730808041326sb6bece9q7fd0a8c4efe5cd08@mail.gmail.com> <48978A38.6010503@noaa.gov> <3d375d730808041604j39c4cebfif40ca57ba803069c@mail.gmail.com> Message-ID: <48989831.7070200@noaa.gov> Robert Kern wrote: > > It's still pretty bad, though. I do recommend running Disk Repair like Bill did. I did that, and it found and did nothing -- I suspect it ran when I re-booted -- it did take a while to reboot. However, this is pretty consistently what I'm getting now: $ time python -c "import numpy" real 0m0.728s user 0m0.327s sys 0m0.398s Which is apparently pretty slow. Robert gets: $ time python -c "import numpy" python -c "import numpy" 0.18s user 0.46s system 88% cpu 0.716 total Is that on a similar machine??? Are you running Universal binaries? Would that make any difference? I wouldn't think so, I'm just grasping at straws here. This is a Dual 1.8GHz G5 desktop, running OS-X 10.4.11, Python 2.5.2 (python.org build), numpy 1.1.1 (from binary on sourceforge) I just tried this on a colleague's machine that is identical, and got about 0.4 seconds "real" -- so faster than mine, but still slow. This still feels blazingly fast to me, as I was getting something like 7+ seconds! thanks for all the help, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From doutriaux1 at llnl.gov Tue Aug 5 14:14:29 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Tue, 05 Aug 2008 11:14:29 -0700 Subject: [Numpy-discussion] 64bit issue? In-Reply-To: References: <48988BCA.9030203@llnl.gov> Message-ID: <48989885.8070409@llnl.gov> Hi chuck, works great on 32bit int *dims; dims = (int *)malloc(self->nd*sizeof(int)); and self->nd is 3 C. Charles R Harris wrote: > > > On Tue, Aug 5, 2008 at 12:20 PM, Charles Doutriaux > > wrote: > > Hello I'm running into some strange error on a 64bit machine, > I tracked it down to this line returning a NULL pointer, any idea > why is > this? > I tried both numpy1.1.1 and what in trunk, numpy.test() passes for > both > > Ok here's the uname of the machine and the offending line: > > Linux quartic.txcorp.com > 2.6.20-1.2320.fc5 #1 SMP Tue Jun 12 18:50:49 > EDT 2007 x86_64 x86_64 x86_64 GNU/Linux > > array = (PyArrayObject *)PyArray_SimpleNew(d, dims, self->type); > > where d is 3 and dims: 120,46,72 > self->type is 11 > > it seems to pass with d=1, but i'm not 100% positive. > > Any idea on why it could be failing? > > > What is the type of dims? Is there also a problem with a 32 bit OS? > > Chuck > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http:// projects.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Tue Aug 5 14:32:02 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 5 Aug 2008 13:32:02 -0500 Subject: [Numpy-discussion] 64bit issue? In-Reply-To: <48989885.8070409@llnl.gov> References: <48988BCA.9030203@llnl.gov> <48989885.8070409@llnl.gov> Message-ID: On Tue, Aug 5, 2008 at 1:14 PM, Charles Doutriaux wrote: > Hi chuck, works great on 32bit > > int *dims; > dims = (int *)malloc(self->nd*sizeof(int)); > > and self->nd is 3 > Should be npy_intp *dims; npy_intp will be 32 bits/ 64 bits depending on the architecture, ints tend to always be 32 bits. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From doutriaux1 at llnl.gov Tue Aug 5 14:45:35 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Tue, 05 Aug 2008 11:45:35 -0700 Subject: [Numpy-discussion] 64bit issue? In-Reply-To: References: <48988BCA.9030203@llnl.gov> <48989885.8070409@llnl.gov> Message-ID: <48989FCF.5080100@llnl.gov> Wow! It did it! Is there other little tricks like this one I should track? Thanks a lot! It would have take me days to track this one! C. Charles R Harris wrote: > > > On Tue, Aug 5, 2008 at 1:14 PM, Charles Doutriaux > wrote: > > Hi chuck, works great on 32bit > > int *dims; > dims = (int *)malloc(self->nd*sizeof(int)); > > and self->nd is 3 > > > Should be > > npy_intp *dims; > > npy_intp will be 32 bits/ 64 bits depending on the architecture, ints > tend to always be 32 bits. > > Chuck > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http:// projects.scipy.org/mailman/listinfo/numpy-discussion > From millman at berkeley.edu Tue Aug 5 14:51:29 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 5 Aug 2008 11:51:29 -0700 Subject: [Numpy-discussion] 1.2 tasks In-Reply-To: <91cf711d0808051058m768ac1b6k4f55cde7beb09bef@mail.gmail.com> References: <91cf711d0808041141s121a4322nf54ce41684149fc4@mail.gmail.com> <91cf711d0808050848x140e3a4eq2c8dc1aacecb78b6@mail.gmail.com> <91cf711d0808051058m768ac1b6k4f55cde7beb09bef@mail.gmail.com> Message-ID: On Tue, Aug 5, 2008 at 10:58 AM, David Huard wrote: > The roadmap that I propose is the following: > > 1.1 we warn about upcoming change, (new=False) > 1.2 we make that change, (new=None) + warnings > 1.3 we deprecate the old behaviour (new=True), no warnings. > 1.4 remove the old behavior and remove the new keyword. > > It's pretty much the roadmap exposed in the related ticket that I wrote > following discussions on the ML. > > This leaves plenty of time for people to make their changes, and my guess > is that a lot of people will appreciate this, given that you were asked to > delay > the changes to histogram. Sounds good. Thanks for the clarification. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From aisaac at american.edu Tue Aug 5 15:00:23 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 5 Aug 2008 15:00:23 -0400 Subject: [Numpy-discussion] Bilateral filter In-Reply-To: <710F2847B0018641891D9A216027636029C204@ex3.envision.co.il> References: <1217950721.21145.26.camel@nadav.envision.co.il> <710F2847B0018641891D9A216027636029C204@ex3.envision.co.il> Message-ID: On Tue, 5 Aug 2008, Nadav Horesh apparently wrote: > As much as I know, the python's community is bound to the > BSD style licence. I was hoping that the code would be > included in scipy, after I would attach the required > declaration (I do not know which), and maybe with some > adjustment to the coding style. > In short, for the next few days I am waiting for comments > and responses that would enable me to donate the code to > an existing project (scipy? pyvision?). Meanwhile I'll > improve the co a bit. If you can't wait, and not willing > to use the code without a licence statement, I'll repost > the code with the scipy's licence declaration. I had a little trouble understanding what you meant above. So just to be clear... If you offer the code now under a BSD license, that will be acceptable to SciPy. Additionally, I cannot imagine a project (including GPL projects) who would find the BSD license a problem. That is one of the great things about the BSD and MIT licenses! Also, just to be clear, if you release as BSD and later wish to also release your code under another license, you can do so. Finally, anyone would be foolish to use the code without your permission, and once you give your permission your are licensing the code, so you may as well be clear from the start. IANAL, Alan Isaac From charlesr.harris at gmail.com Tue Aug 5 15:15:28 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 5 Aug 2008 14:15:28 -0500 Subject: [Numpy-discussion] 64bit issue? In-Reply-To: <48989FCF.5080100@llnl.gov> References: <48988BCA.9030203@llnl.gov> <48989885.8070409@llnl.gov> <48989FCF.5080100@llnl.gov> Message-ID: On Tue, Aug 5, 2008 at 1:45 PM, Charles Doutriaux wrote: > Wow! It did it! > > Is there other little tricks like this one I should track? > The main change from the deprecated compatibility functions to the new functions is the replacement of int *dims; by npy_intp *dims; Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sameerslists at gmail.com Tue Aug 5 15:26:51 2008 From: sameerslists at gmail.com (Sameer DCosta) Date: Tue, 5 Aug 2008 14:26:51 -0500 Subject: [Numpy-discussion] Error creating an array Message-ID: <8fb8cc060808051226q683bbbe4v6c453dd61ab2daba@mail.gmail.com> Im having a little trouble creating a numpy array with a specific dtype. I can create the array b with dtype=int, but not with the dtype I want. Any idea what I am doing wrong here? In [450]: import numpy as np In [451]: print np.__version__ 1.2.0.dev5243 In [452]: dtype=np.dtype([('spam', ' 1 2 3 4 5 TypeError: expected a readable buffer object In [455]: b = np.array( data, dtype=int) In [456]: print b [[1 2] [3 4] [5 6]] Thanks. Sameer From oliphant at enthought.com Tue Aug 5 15:35:28 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 05 Aug 2008 14:35:28 -0500 Subject: [Numpy-discussion] Error creating an array In-Reply-To: <8fb8cc060808051226q683bbbe4v6c453dd61ab2daba@mail.gmail.com> References: <8fb8cc060808051226q683bbbe4v6c453dd61ab2daba@mail.gmail.com> Message-ID: <4898AB80.4030709@enthought.com> Sameer DCosta wrote: > Im having a little trouble creating a numpy array with a specific > dtype. I can create the array b with dtype=int, but not with the dtype > I want. Any idea what I am doing wrong here? > You must uses tuples for the individual "records" when constructing arrays with the "array" command. Thus, data = [(1,2), (3,4), (5,6)] will work. -Travis From sameerslists at gmail.com Tue Aug 5 15:46:34 2008 From: sameerslists at gmail.com (Sameer DCosta) Date: Tue, 5 Aug 2008 14:46:34 -0500 Subject: [Numpy-discussion] Error creating an array In-Reply-To: <4898AB80.4030709@enthought.com> References: <8fb8cc060808051226q683bbbe4v6c453dd61ab2daba@mail.gmail.com> <4898AB80.4030709@enthought.com> Message-ID: <8fb8cc060808051246t625c8d92r8b72c6209c546b3@mail.gmail.com> On Tue, Aug 5, 2008 at 2:35 PM, Travis E. Oliphant wrote: > You must uses tuples for the individual "records" when constructing > arrays with the "array" command. Thanks Travis. Is there a reason why numpy insists on tuples? Anyway, moving on, this brings me to the real problem I wanted to post. I can do an attribute lookup on an element of a recarray which has been created by fromrecords, however I cannot do attribute lookups on ndarrays that have been viewed as recarrays. Any ideas how to proceed. In [469]: import numpy as np In [470]: print np.__version__ 1.2.0.dev5243 In [471]: dtype=np.dtype([('spam', '> arrays with the "array" command. >> > > Thanks Travis. Is there a reason why numpy insists on tuples? > Yeah.. complexity of the array command (it process a lot of cases). There may be a way to make it work differently for record arrays, but nobody has done that. > Anyway, moving on, this brings me to the real problem I wanted to > post. I can do an attribute lookup on an element of a recarray which > has been created by fromrecords, however I cannot do attribute lookups > on ndarrays that have been viewed as recarrays. Any ideas how to > proceed. > You can always use b[0]['spam'] to look up the field. The record is a new data-type sub-classed from void that allows attribute look up for field access. Perhaps we should automatically convert the dtype at the same time that we convert type type. But, I'm not really sure. Basically, to get the behavior you want, the dtype of the array needs to be changed also to that of record instead of the void type with fields defined. This is fundamentally different than changing just the type-object of the array itself to the record-array. You can get the behavior you want with: b = b.view(dtype=(np.record, b.dtype), type=np.recarray). This manually changes not only the type of the array to a recarray but also the data-type of the elements in the array to be "records" (with attribute lookup of fields) instead of the default "void" type with fields accesible through mapping semantics. Best regards, -Travis From oliphant at enthought.com Tue Aug 5 17:08:04 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 05 Aug 2008 16:08:04 -0500 Subject: [Numpy-discussion] Bilateral filter In-Reply-To: <1217950721.21145.26.camel@nadav.envision.co.il> References: <1217950721.21145.26.camel@nadav.envision.co.il> Message-ID: <4898C134.6060201@enthought.com> Nadav Horesh wrote: > Attached here my cython implementation of the bilateral filter, which > is my first cython program. I would ask for the following: > > 1. Is there any way to speed up the code just by "cosmetic" > modifications? > 2. In particular I use the unportable gcc function __builtin_abs: > Is there any way to access this in a portable way? > 3. I would like to donate this code to scipy or any other suitable > project. What can/should I do to realise it? > > > Remarks: > > The code contains 3 end-user routines: > > 1. A pure python implementation: Easy to read and modify --- it can > be cut out into a python source code. > 2. A straight forward cython implementation: About 4 times as fast > as the python implementation. > 3. An optimised cython implementation earning another factor of 2 > in speed (depends on the parameters used). > > I see this code as a "research grade" that would evolve in the near > future, as I am working on a related project, and hopefully following > your comments. This would be a very welcome addition to SciPy. Thank you. -Travis From fperez.net at gmail.com Tue Aug 5 19:20:06 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 5 Aug 2008 16:20:06 -0700 Subject: [Numpy-discussion] Changeset 5609 of numpy breaks scipy build Message-ID: Howdy, I just noticed that scipy wasn't building via my little script that does a numpy/mpl/scipy svn update+clean build. Chris Burn's excellent hunch pinpointed the culprit: http://projects.scipy.org/scipy/numpy/changeset/5609 Innocent looking as it may be, with numpy after that revision, building current scipy SVN gives: creating build/temp.linux-x86_64-2.5/scipy/sparse/sparsetools compile options: '-I/home/fperez/usr/opt/lib/python2.5/site-packages/numpy/core/include -I/usr/include/python2.5 -c' g++: scipy/sparse/sparsetools/csr_wrap.cxx scipy/sparse/sparsetools/csr_wrap.cxx: In function 'int require_size(PyArrayObject*, npy_intp*, int)': scipy/sparse/sparsetools/csr_wrap.cxx:2902: error: expected `)' before 'PRIdPTR' scipy/sparse/sparsetools/csr_wrap.cxx:2902: warning: spurious trailing '%' in format scipy/sparse/sparsetools/csr_wrap.cxx:2902: warning: too many arguments for format scipy/sparse/sparsetools/csr_wrap.cxx:2909: error: expected `)' before 'PRIdPTR' scipy/sparse/sparsetools/csr_wrap.cxx:2909: warning: spurious trailing '%' in format scipy/sparse/sparsetools/csr_wrap.cxx:2909: warning: too many arguments for format scipy/sparse/sparsetools/csr_wrap.cxx: In function 'int require_size(PyArrayObject*, npy_intp*, int)': scipy/sparse/sparsetools/csr_wrap.cxx:2902: error: expected `)' before 'PRIdPTR' scipy/sparse/sparsetools/csr_wrap.cxx:2902: warning: spurious trailing '%' in format scipy/sparse/sparsetools/csr_wrap.cxx:2902: warning: too many arguments for format scipy/sparse/sparsetools/csr_wrap.cxx:2909: error: expected `)' before 'PRIdPTR' scipy/sparse/sparsetools/csr_wrap.cxx:2909: warning: spurious trailing '%' in format scipy/sparse/sparsetools/csr_wrap.cxx:2909: warning: too many arguments for format error: Command "g++ -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC -I/home/fperez/usr/opt/lib/python2.5/site-packages/numpy/core/include -I/usr/include/python2.5 -c scipy/sparse/sparsetools/csr_wrap.cxx -o build/temp.linux-x86_64-2.5/scipy/sparse/sparsetools/csr_wrap.o" failed with exit status 1 ### I don't know what the right solution needs to be, but I figured we'd at least report it quickly so it can be looked into. Cheers, f From cournape at gmail.com Tue Aug 5 19:36:33 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 6 Aug 2008 08:36:33 +0900 Subject: [Numpy-discussion] Changeset 5609 of numpy breaks scipy build In-Reply-To: References: Message-ID: <5b8d13220808051636j4a4c73adg669dd9ac607837e8@mail.gmail.com> On Wed, Aug 6, 2008 at 8:20 AM, Fernando Perez wrote: > Howdy, > > I just noticed that scipy wasn't building via my little script that > does a numpy/mpl/scipy svn update+clean build. That would be me. I forgot to take into account C++ in this change. Looking at it cheers, David From cournape at gmail.com Tue Aug 5 20:14:51 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 6 Aug 2008 09:14:51 +0900 Subject: [Numpy-discussion] Changeset 5609 of numpy breaks scipy build In-Reply-To: <5b8d13220808051636j4a4c73adg669dd9ac607837e8@mail.gmail.com> References: <5b8d13220808051636j4a4c73adg669dd9ac607837e8@mail.gmail.com> Message-ID: <5b8d13220808051714j4b8bf5cucbd90d23553d4fd4@mail.gmail.com> On Wed, Aug 6, 2008 at 8:36 AM, David Cournapeau wrote: > On Wed, Aug 6, 2008 at 8:20 AM, Fernando Perez wrote: >> Howdy, >> >> I just noticed that scipy wasn't building via my little script that >> does a numpy/mpl/scipy svn update+clean build. > > That would be me. I forgot to take into account C++ in this change. > Looking at it > Ok, this should be fixed in r5614. I only tested it on mac os X, but sparsetools seems to build fine with the change, now. cheers, David From millman at berkeley.edu Tue Aug 5 21:07:15 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 5 Aug 2008 18:07:15 -0700 Subject: [Numpy-discussion] Bilteral filter In-Reply-To: <710F2847B0018641891D9A216027636029C1FC@ex3.envision.co.il> References: <710F2847B0018641891D9A216027636029C1F4@ex3.envision.co.il> <710F2847B0018641891D9A216027636029C1F8@ex3.envision.co.il> <2914DB5E-D83E-4DA3-8469-8D69C70E41DA@comcast.net> <710F2847B0018641891D9A216027636029C1FC@ex3.envision.co.il> Message-ID: 2008/8/3 Nadav Horesh : > I'll post the code in this mailing list, and hope it would find a good home. Excellent. It would be great to get this into SciPy. Perhaps there is even code in David Bolme's project that he might be interested in pushing upstream. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From mailanhilli at googlemail.com Tue Aug 5 22:26:08 2008 From: mailanhilli at googlemail.com (Matthias Hillenbrand) Date: Tue, 5 Aug 2008 21:26:08 -0500 Subject: [Numpy-discussion] Horizontal lines in diffraction image (NumPy FFT) Message-ID: <67b3a51f0808051926r1aefac3dibfbffda4100d9fb5@mail.gmail.com> Hello, I am trying to calculate the propagation of a Gaussian beam with an angular spectrum propagation algorithm. My program does the following steps: 1. Generation and multiplication of the Gaussian beam, an aperture, and a lens -> u 2. FFT(u) 3. Multiplication of FFT(u) with the angular spectrum kernel H 4. IFFT(FU*H) Steps 2-4 are repeated 1000 times to show the propagation of the beam in z-direction Unfortunately the calculation results in a lot of horizontal lines that should not be there. The program produces reasonable results besides this problem. It is especially interesting that an equivalent calculation with Matlab or Octave has the same results without these artifacts. Both calculations take approximately 90 seconds. I am not sure whether I am doing something wrong or there is a difference in the FFT codes. I assume that the problem has something to do with the propagation kernel H but I cannot see a difference between both programs. Thank you very much for your help! Matthias --------------------------------------------------------------------------------------------------- Python code with comments (version without comments below) --------------------------------------------------------------------------------------------------- from pylab import * from numpy.lib import scimath #Complex roots import os ######################################### ########## Defintion of the input variables ######### ######################################### ##General variables lam = 532.e-9 #wavelength in m k = 2*pi/lam #wave number ##Computational array arr_size = 80.e-3 #size of the computation array in m N = 2**17 #points of the 1D array ##Aperture ap = 40.e-3 #aperture size of the 1D array in m ##Gaussian beam w0 =10.e-3 #waist radius of the gaussian beam in m ##Lenses (definition of the focal length) n = 1.56 #refractive index of the glass f = .1 #focal length in m Phi = 1/f #focal power of the lens r2 = 1.e18 #radius of the second lens surface r1 = 1 / ( Phi/(n-1) + 1/r2 ) #computation of the radius of the first lens surface ##z distances zmin = 0.99*f #initial z position zmax = 1.01*f #final z position zsteps = 1000 #number of computated positions ROI = 1000 #Region of interest in the diffraction image ############################################## ############### Program execution 1D ################ ############################################### x = linspace(-arr_size/2,arr_size/2,N) A = ones(N) A[where(ap/2<=abs(x))] = 0 #Definition of the aperture G = exp(- x**2/w0**2) #Generation of the Gaussian beam delta = -r1*(1 - scimath.sqrt(1 - x**2/r1**2)) + r2*(1 - scimath.sqrt(1 - x**2/r2**2)) m = (r1**2 <= x**2 ) delta[m] = 0 #correction in case of negative roots m = (r2**2 <= x**2 ) delta[m] = 0 #correction in case of negative roots lens = exp(1j * 2 * pi / lam * (n-1) * delta) #Calculation of the lens phase function u = A*G*lens #Complex amplitude before beam propagation ############################################ ########### Start angular spectrum method ########### deltaf = 1/arr_size #spacing in frequency domain fx = r_[-N/2*deltaf:(N/2)*deltaf:deltaf] #whole frequency domain FU = fftshift(fft(u)) #1D FFT U = zeros((zsteps,ROI)) #Generation of the image array z = linspace(zmin,zmax,zsteps) #Calculation of the different z positions for i in range(zsteps): H = exp(1j * 2 * pi / lam * z[i] * scimath.sqrt(1-(lam*fx)**2)) U[i] = (ifft(fftshift(FU*H)))[N/2 - ROI/2 : N/2 + ROI/2] if i%10 == 0: t = 'z position: %4i' % i print t ############ End angular spectrum method ############ ############################################# res = abs(U) imshow(res) show() ---------------------------------------------------------- Python code without comments ---------------------------------------------------------- from pylab import * from numpy.lib import scimath #Complex roots import os lam = 532.e-9 k = 2*pi/lam arr_size = 80.e-3 N = 2**17 ap = 40.e-3 w0 =10.e-3 n = 1.56 f = .1 Phi = 1/f r2 = 1.e18 r1 = 1 / ( Phi/(n-1) + 1/r2 ) zmin = 0.99*f zmax = 1.01*f zsteps = 1000 ROI = 1000 x = linspace(-arr_size/2,arr_size/2,N) A = ones(N) A[where(ap/2<=abs(x))] = 0 G = exp(- x**2/w0**2) delta = -r1*(1 - scimath.sqrt(1 - x**2/r1**2)) + r2*(1 - scimath.sqrt(1 - x**2/r2**2)) m = (r1**2 <= x**2 ) delta[m] = 0 m = (r2**2 <= x**2 ) delta[m] = 0 lens = exp(1j * 2 * pi / lam * (n-1) * delta) u = A*G*lens deltaf = 1/arr_size fx = r_[-N/2*deltaf:(N/2)*deltaf:deltaf] FU = fftshift(fft(u)) U = zeros((zsteps,ROI)) z = linspace(zmin,zmax,zsteps) for i in range(zsteps): H = exp(1j * 2 * pi / lam * z[i] * scimath.sqrt(1-(lam*fx)**2)) U[i] = (ifft(fftshift(FU*H)))[N/2 - ROI/2 : N/2 + ROI/2] if i%10 == 0: t = 'z position: %4i' % i print t res = abs(U) imshow(res) show() ---------------------------------------------------------- Matlab, Octave code ---------------------------------------------------------- lam = 532.e-9 k = 2*pi/lam arr_size = 80.e-3 N = 2^17 ap = 40.e-3 w0 =10.e-3 n = 1.56 f = .1 Phi = 1/f r2 = 1.e18 r1 = 1 / ( Phi/(n-1) + 1/r2 ) zmin = 0.99*f zmax = 1.01*f zsteps = 1000 ROI = 1000 x=linspace(-arr_size/2,arr_size/2,N); A = ones(1,N); A(find(ap/2<=abs(x)))=0; G = exp(- x.^2/w0^2); delta = -r1*(1 - sqrt(1 - x.^2/r1^2)) + r2*(1 - sqrt(1 - x.^2/r2^2)); delta(find(r1.^2 <= x.^2 ))=0; delta(find(r2.^2 <= x.^2 ))=0; lens = exp(1j * 2 * pi / lam * (n-1) * delta); u = A.*G.*lens; deltaf = 1/arr_size; fx = [-N/2*deltaf:deltaf:(N/2-1)*deltaf]; FU = fftshift(fft(u)); U = zeros(zsteps,ROI); z = linspace(zmin,zmax,zsteps); for i=1:zsteps H = exp(1j * 2 * pi / lam * z(i) * sqrt(1-lam.^2*fx.^2)); U(i,:) = ifft(fftshift(FU.*H))(N/2 - ROI/2 : N/2 + ROI/2-1); end imagesc(abs(U)) From dagss at student.matnat.uio.no Wed Aug 6 04:04:24 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 Aug 2008 10:04:24 +0200 Subject: [Numpy-discussion] Cython/NumPy syntax Message-ID: <48995B08.6030707@student.matnat.uio.no> I'd like input from anyone interested in the following syntax questions. The current experimental Cython syntax for efficient ndarray indexing looks like this: cdef numpy.ndarray[numpy.int64, 2] arr Issue 1: One can argue that "2" above looks like a length specifier to newcomers. Issue 2: This clashes with some forms of C array notation (you can do "sizeof(int[3])" or "extern ... def myfunc(int[])" in Cython, and though such syntax is redundant it will stay in order not to break code). Solving issue #1 helps with #2 as well because without a single integer literal in it it should look a lot less like an array declaration. So, some quick proposals: - Require a "D" suffix for the number of dimensions. This should make it very clear that the 2 stands for dimensionality and not shape. cdef numpy.ndarray[numpy.int64, 2D] arr Variations: a) numpy.ndarray[numpy.int64, 2D] b) numpy.ndarray[numpy.int64 2D] c) numpy.ndarray[2D numpy.int64] (Note that I want to leave the way open for adding mode="c" or mode="fortran", so using a "," seems good for that reason). - Require an ndim keyword: cdef numpy.ndarray[numpy.int64, ndim=2] - Other type of brackets. This boils down to (for various reasons) the <> brackets: cdef numpy.ndarray (However, one should remember that Cython may want to support using C++ templates at some point). -- Dag Sverre From stefan at sun.ac.za Wed Aug 6 04:14:41 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 6 Aug 2008 10:14:41 +0200 Subject: [Numpy-discussion] Cython/NumPy syntax In-Reply-To: <48995B08.6030707@student.matnat.uio.no> References: <48995B08.6030707@student.matnat.uio.no> Message-ID: <9457e7c80808060114j73ec2a8cl930791eecfcaf690@mail.gmail.com> 2008/8/6 Dag Sverre Seljebotn : > - Require an ndim keyword: > > cdef numpy.ndarray[numpy.int64, ndim=2] I'd definitely prefer a comma between the two, and an (optional) ndim keyword argument if possible. Looking forward to hearing more! Regards St?fan From dagss at student.matnat.uio.no Wed Aug 6 04:35:06 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 Aug 2008 10:35:06 +0200 Subject: [Numpy-discussion] Cython/NumPy syntax In-Reply-To: <9457e7c80808060114j73ec2a8cl930791eecfcaf690@mail.gmail.com> References: <48995B08.6030707@student.matnat.uio.no> <9457e7c80808060114j73ec2a8cl930791eecfcaf690@mail.gmail.com> Message-ID: <4899623A.2040504@student.matnat.uio.no> St?fan van der Walt wrote: > 2008/8/6 Dag Sverre Seljebotn : >> - Require an ndim keyword: >> >> cdef numpy.ndarray[numpy.int64, ndim=2] > > I'd definitely prefer a comma between the two, and an (optional) ndim > keyword argument if possible. I'm taking this as a vote in favor of this and against "2D" and <>? The keyword is already present in optional form (you can do already do "ndarray[ndim=3, dtype=numpy.int64]" if you want to). So I meant to ask whether making it mandatory is a good solution so that the 2 doesn't look like a length specifier. (ndim=2 seems too long for me though, which is why I am pondering "2D".) > Looking forward to hearing more! It is looking bright and some experimental support will be present in the next Cython release. What might be missing first time around is support for complex numbers, records/structs and object dtypes. (As there is (at least not yet) no native complex number support in Cython you would need to handle a complex struct manually anyway. I was hoping to fix this but I won't have time within the GSoC.) -- Dag Sverre From stefan at sun.ac.za Wed Aug 6 04:51:50 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 6 Aug 2008 10:51:50 +0200 Subject: [Numpy-discussion] Cython/NumPy syntax In-Reply-To: <4899623A.2040504@student.matnat.uio.no> References: <48995B08.6030707@student.matnat.uio.no> <9457e7c80808060114j73ec2a8cl930791eecfcaf690@mail.gmail.com> <4899623A.2040504@student.matnat.uio.no> Message-ID: <9457e7c80808060151n3198d569k3490b80dc1336819@mail.gmail.com> 2008/8/6 Dag Sverre Seljebotn : >>> cdef numpy.ndarray[numpy.int64, ndim=2] >> >> I'd definitely prefer a comma between the two, and an (optional) ndim >> keyword argument if possible. > > I'm taking this as a vote in favor of this and against "2D" and <>? > > The keyword is already present in optional form (you can do already do > "ndarray[ndim=3, dtype=numpy.int64]" if you want to). So I meant to ask > whether making it mandatory is a good solution so that the 2 doesn't > look like a length specifier. I prefer having the `ndim` present -- immediately, the code becomes more transparent to a foreign eye. > (ndim=2 seems too long for me though, which is why I am pondering "2D".) The length doesn't bother me much (I prefer typing the extra 4 characters to make the code more readable). Personally, I'd prefer not to use 2D -- it's surprising to see a string like that without enclosing quotes (you'd never see it in Python, for example). > It is looking bright and some experimental support will be present in > the next Cython release. What might be missing first time around is > support for complex numbers, records/structs and object dtypes. That is great news; please keep us up to date. Thanks for all your hard work! Regards St?fan From nadavh at visionsense.com Wed Aug 6 07:59:11 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed, 06 Aug 2008 14:59:11 +0300 Subject: [Numpy-discussion] Bilateral filter In-Reply-To: <4898C134.6060201@enthought.com> References: <1217950721.21145.26.camel@nadav.envision.co.il> <4898C134.6060201@enthought.com> Message-ID: <1218023951.26827.10.camel@nadav.envision.co.il> I made the following modification to the source code, I hope it is ready to be included in scipy. 1. Added a BSD licence declaration. 2. Small optimisation. 3. The code is split into a cython back-end and a python front-end. All remarks are welcome, Nadav. On Tue, 2008-08-05 at 16:08 -0500, Travis E. Oliphant wrote: > Nadav Horesh wrote: > > Attached here my cython implementation of the bilateral filter, which > > is my first cython program. I would ask for the following: > > > > 1. Is there any way to speed up the code just by "cosmetic" > > modifications? > > 2. In particular I use the unportable gcc function __builtin_abs: > > Is there any way to access this in a portable way? > > 3. I would like to donate this code to scipy or any other suitable > > project. What can/should I do to realise it? > > > > > > Remarks: > > > > The code contains 3 end-user routines: > > > > 1. A pure python implementation: Easy to read and modify --- it can > > be cut out into a python source code. > > 2. A straight forward cython implementation: About 4 times as fast > > as the python implementation. > > 3. An optimised cython implementation earning another factor of 2 > > in speed (depends on the parameters used). > > > > I see this code as a "research grade" that would evolve in the near > > future, as I am working on a related project, and hopefully following > > your comments. > This would be a very welcome addition to SciPy. Thank you. > > -Travis > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bilateral.py Type: text/x-python Size: 1803 bytes Desc: not available URL: -------------- next part -------------- # Copyright 2008, Nadav Horesh # nadavh at visionsense dot com # # The software is licenced under BSD licence. ''' A cython implementation of the plain (and slow) algorithm for bilateral filtering. The class Bilat_fcn exposes methods to be called by nd_image.generic_filter function in orde to render the actual filter. ''' import numpy as np cdef extern from "arrayobject.h": ctypedef int intp ctypedef extern class numpy.ndarray [object PyArrayObject]: cdef char *data cdef int nd cdef intp *dimensions cdef intp *strides cdef int flags cdef extern from "math.h": double exp(double x) cdef extern: int abs(int x) cdef int GAUSS_SAMP = 32 cdef int GAUSS_IDX_MAX = GAUSS_SAMP - 1 class Bilat_fcn(object): ''' The class provides the bilaterl filter function to be called by generic_filter. initialization parameters: spat_sig: The sigma of the spatial Gaussian filter inten_sig: The sigma of the gray-levels Gaussian filter filter_size: (int) The size of the spatial convolution kernel. If not set, it is set to ~ 4*spat_sig. ''' def __init__(self, spat_sig, inten_sig, filter_size=None): if filter_size is not None and filter_size >= 2: self.xy_size = int(filter_size) else: self.xy_size = int(round(spat_sig*4)) # Make filter size odd self.xy_size += 1-self.xy_size%2 x = np.arange(self.xy_size, dtype=float) x = (x-x.mean())**2 #xy_ker: Spatial convolution kernel self.xy_ker = np.exp(-np.add.outer(x,x)/(2*spat_sig**2)).ravel() self.xy_ker /= self.xy_ker.sum() self.inten_sig = 2 * inten_sig**2 self.index = self.xy_size**2 // 2 ## An initialization for LUT instead of a Gaussian function call ## (for the fc_filter method) x = np.linspace(0,3.0, GAUSS_SAMP) self.gauss_lut = np.exp(-x**2/2) self.x_quant = 3*inten_sig / GAUSS_IDX_MAX ## ## Filtering functions ## def __call__ (self, data): 'An unoptimized (pure python) implementation' weight = np.exp(-(data-data[self.index])**2/self.inten_sig) * self.xy_ker return np.dot(data, weight) / weight.sum() def cfilter(self, ndarray data): 'An optimized implementation' cdef ndarray kernel = self.xy_ker cdef double sigma = self.inten_sig cdef double weight_i, weight, result, centre, dat_i cdef double *pdata=data.data, *pker=kernel.data cdef int i, dim = data.dimensions[0] centre = pdata[self.index] weight = 0.0 result = 0.0 for i from 0 <= i < dim: dat_i = pdata[i] weight_i = exp(-(dat_i-centre)**2 / sigma) * pker[i] weight += weight_i; result += dat_i * weight_i return result / weight def fc_filter(self, ndarray data): 'Use further optimisation by replacing exp functions calls by a LUT' cdef ndarray kernel = self.xy_ker cdef ndarray gauss_lut_arr = self.gauss_lut cdef double sigma = self.inten_sig cdef double weight_i, weight, result, centre, dat_i cdef double *pdata=data.data, *pker=kernel.data cdef int i, dim = data.dimensions[0] cdef int exp_i # Entry index for the LUT cdef double x_quant = self.x_quant cdef double *gauss_lut = gauss_lut_arr.data centre = pdata[self.index] weight = 0.0 result = 0.0 for i from 0 <= i < dim: dat_i = pdata[i] exp_i = abs(((dat_i-centre) / x_quant)) if exp_i > GAUSS_IDX_MAX: #exp_i = GAUSS_IDX_MAX continue weight_i = gauss_lut[exp_i] * pker[i] weight += weight_i; result += dat_i * weight_i return result / weight # # Filtering functions to be called from outsize # From nadavh at visionsense.com Wed Aug 6 05:09:07 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed, 6 Aug 2008 12:09:07 +0300 Subject: [Numpy-discussion] Horizontal lines in diffraction image (NumPy FFT) References: <67b3a51f0808051926r1aefac3dibfbffda4100d9fb5@mail.gmail.com> Message-ID: <710F2847B0018641891D9A216027636029C205@ex3.envision.co.il> I did about the same thing 9 year ago (in python of course). If I can recall correctly, you need to double the arrays size (with padding of 0) in order to avoid this artifact. I think that its origin is that the DFT is equivalent to periodic boundary conditions. Nadav. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Matthias Hillenbrand ????: ? 06-??????-08 05:26 ??: numpy-discussion at scipy.org ????: [Numpy-discussion] Horizontal lines in diffraction image (NumPy FFT) Hello, I am trying to calculate the propagation of a Gaussian beam with an angular spectrum propagation algorithm. My program does the following steps: 1. Generation and multiplication of the Gaussian beam, an aperture, and a lens -> u 2. FFT(u) 3. Multiplication of FFT(u) with the angular spectrum kernel H 4. IFFT(FU*H) Steps 2-4 are repeated 1000 times to show the propagation of the beam in z-direction Unfortunately the calculation results in a lot of horizontal lines that should not be there. The program produces reasonable results besides this problem. It is especially interesting that an equivalent calculation with Matlab or Octave has the same results without these artifacts. Both calculations take approximately 90 seconds. I am not sure whether I am doing something wrong or there is a difference in the FFT codes. I assume that the problem has something to do with the propagation kernel H but I cannot see a difference between both programs. Thank you very much for your help! Matthias --------------------------------------------------------------------------------------------------- Python code with comments (version without comments below) --------------------------------------------------------------------------------------------------- from pylab import * from numpy.lib import scimath #Complex roots import os ######################################### ########## Defintion of the input variables ######### ######################################### ##General variables lam = 532.e-9 #wavelength in m k = 2*pi/lam #wave number ##Computational array arr_size = 80.e-3 #size of the computation array in m N = 2**17 #points of the 1D array ##Aperture ap = 40.e-3 #aperture size of the 1D array in m ##Gaussian beam w0 =10.e-3 #waist radius of the gaussian beam in m ##Lenses (definition of the focal length) n = 1.56 #refractive index of the glass f = .1 #focal length in m Phi = 1/f #focal power of the lens r2 = 1.e18 #radius of the second lens surface r1 = 1 / ( Phi/(n-1) + 1/r2 ) #computation of the radius of the first lens surface ##z distances zmin = 0.99*f #initial z position zmax = 1.01*f #final z position zsteps = 1000 #number of computated positions ROI = 1000 #Region of interest in the diffraction image ############################################## ############### Program execution 1D ################ ############################################### x = linspace(-arr_size/2,arr_size/2,N) A = ones(N) A[where(ap/2<=abs(x))] = 0 #Definition of the aperture G = exp(- x**2/w0**2) #Generation of the Gaussian beam delta = -r1*(1 - scimath.sqrt(1 - x**2/r1**2)) + r2*(1 - scimath.sqrt(1 - x**2/r2**2)) m = (r1**2 <= x**2 ) delta[m] = 0 #correction in case of negative roots m = (r2**2 <= x**2 ) delta[m] = 0 #correction in case of negative roots lens = exp(1j * 2 * pi / lam * (n-1) * delta) #Calculation of the lens phase function u = A*G*lens #Complex amplitude before beam propagation ############################################ ########### Start angular spectrum method ########### deltaf = 1/arr_size #spacing in frequency domain fx = r_[-N/2*deltaf:(N/2)*deltaf:deltaf] #whole frequency domain FU = fftshift(fft(u)) #1D FFT U = zeros((zsteps,ROI)) #Generation of the image array z = linspace(zmin,zmax,zsteps) #Calculation of the different z positions for i in range(zsteps): H = exp(1j * 2 * pi / lam * z[i] * scimath.sqrt(1-(lam*fx)**2)) U[i] = (ifft(fftshift(FU*H)))[N/2 - ROI/2 : N/2 + ROI/2] if i%10 == 0: t = 'z position: %4i' % i print t ############ End angular spectrum method ############ ############################################# res = abs(U) imshow(res) show() ---------------------------------------------------------- Python code without comments ---------------------------------------------------------- from pylab import * from numpy.lib import scimath #Complex roots import os lam = 532.e-9 k = 2*pi/lam arr_size = 80.e-3 N = 2**17 ap = 40.e-3 w0 =10.e-3 n = 1.56 f = .1 Phi = 1/f r2 = 1.e18 r1 = 1 / ( Phi/(n-1) + 1/r2 ) zmin = 0.99*f zmax = 1.01*f zsteps = 1000 ROI = 1000 x = linspace(-arr_size/2,arr_size/2,N) A = ones(N) A[where(ap/2<=abs(x))] = 0 G = exp(- x**2/w0**2) delta = -r1*(1 - scimath.sqrt(1 - x**2/r1**2)) + r2*(1 - scimath.sqrt(1 - x**2/r2**2)) m = (r1**2 <= x**2 ) delta[m] = 0 m = (r2**2 <= x**2 ) delta[m] = 0 lens = exp(1j * 2 * pi / lam * (n-1) * delta) u = A*G*lens deltaf = 1/arr_size fx = r_[-N/2*deltaf:(N/2)*deltaf:deltaf] FU = fftshift(fft(u)) U = zeros((zsteps,ROI)) z = linspace(zmin,zmax,zsteps) for i in range(zsteps): H = exp(1j * 2 * pi / lam * z[i] * scimath.sqrt(1-(lam*fx)**2)) U[i] = (ifft(fftshift(FU*H)))[N/2 - ROI/2 : N/2 + ROI/2] if i%10 == 0: t = 'z position: %4i' % i print t res = abs(U) imshow(res) show() ---------------------------------------------------------- Matlab, Octave code ---------------------------------------------------------- lam = 532.e-9 k = 2*pi/lam arr_size = 80.e-3 N = 2^17 ap = 40.e-3 w0 =10.e-3 n = 1.56 f = .1 Phi = 1/f r2 = 1.e18 r1 = 1 / ( Phi/(n-1) + 1/r2 ) zmin = 0.99*f zmax = 1.01*f zsteps = 1000 ROI = 1000 x=linspace(-arr_size/2,arr_size/2,N); A = ones(1,N); A(find(ap/2<=abs(x)))=0; G = exp(- x.^2/w0^2); delta = -r1*(1 - sqrt(1 - x.^2/r1^2)) + r2*(1 - sqrt(1 - x.^2/r2^2)); delta(find(r1.^2 <= x.^2 ))=0; delta(find(r2.^2 <= x.^2 ))=0; lens = exp(1j * 2 * pi / lam * (n-1) * delta); u = A.*G.*lens; deltaf = 1/arr_size; fx = [-N/2*deltaf:deltaf:(N/2-1)*deltaf]; FU = fftshift(fft(u)); U = zeros(zsteps,ROI); z = linspace(zmin,zmax,zsteps); for i=1:zsteps H = exp(1j * 2 * pi / lam * z(i) * sqrt(1-lam.^2*fx.^2)); U(i,:) = ifft(fftshift(FU.*H))(N/2 - ROI/2 : N/2 + ROI/2-1); end imagesc(abs(U)) _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 5713 bytes Desc: not available URL: From wright at esrf.fr Wed Aug 6 05:03:08 2008 From: wright at esrf.fr (Jon Wright) Date: Wed, 06 Aug 2008 11:03:08 +0200 Subject: [Numpy-discussion] win32 1.1.1 testsuite issue with python -O In-Reply-To: <4899623A.2040504@student.matnat.uio.no> References: <48995B08.6030707@student.matnat.uio.no> <9457e7c80808060114j73ec2a8cl930791eecfcaf690@mail.gmail.com> <4899623A.2040504@student.matnat.uio.no> Message-ID: <489968CC.2010205@esrf.fr> Hello, If I use the "-O" switch then it seems getting some testcase failures, and finally a windows message that "python.exe has encountered a problem and needs to close. We are sorry for the inconvenience.". Running the testsuite via jepp (jepp.sourceforge.net) gives the same failures, plus 8 errors which it succeeds to report without the crash. Unfortunately the "-O" switch is defaulted to "on" for that interpreter. Am I doing something silly? Thanks, Jon D:\wright\build_software\embed_numpy>c:\python25\python -O -c "import numpy; numpy.test()" Numpy is installed in c:\python25\lib\site-packages\numpy Numpy version 1.1.1 Python version 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] Found 18/18 tests for numpy.core.tests.test_defmatrix [.snip.] Found 16/16 tests for numpy.testing.tests.test_utils Found 6/6 tests for numpy.tests.test_ctypeslib ........................................................................................................................ .............................F.......................................................................................... ........................................................................................................................ ........................................................................................................................ ..............................................................F......................................................... ........................................................................................................................ ..................................Ignoring "Python was built with Visual Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py." (one should fix me in fcompiler/compaq.py) ........................................................................................................................ ........................................................................................................................ ........................................................................................................................ ........................................................................................................................ ............................................F.F.F.F.FFFFF......... ====================================================================== FAIL: Convolve should raise an error for empty input array. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\core\tests\test_regression.py", line 629, in check_convolve_empty self.failUnlessRaises(AssertionError,np.convolve,[],[1]) AssertionError: AssertionError not raised ====================================================================== FAIL: Convolve should raise an error for empty input array. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\core\tests\test_regression.py", line 629, in check_convolve_empty self.failUnlessRaises(AssertionError,np.convolve,[],[1]) AssertionError: AssertionError not raised ====================================================================== FAIL: Test two arrays with different shapes are found not equal. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 48, in test_array_diffshape self._test_not_equal(a, b) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 20, in _test_not_equal raise AssertionError("a and b are found equal but are not") AssertionError: a and b are found equal but are not ====================================================================== FAIL: Test two different array of rank 1 are found not equal. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 34, in test_array_rank1_noteq self._test_not_equal(a, b) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 20, in _test_not_equal raise AssertionError("a and b are found equal but are not") AssertionError: a and b are found equal but are not ====================================================================== FAIL: Test two arrays with different shapes are found not equal. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 48, in test_array_diffshape self._test_not_equal(a, b) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 20, in _test_not_equal raise AssertionError("a and b are found equal but are not") AssertionError: a and b are found equal but are not ====================================================================== FAIL: Test two different array of rank 1 are found not equal. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 34, in test_array_rank1_noteq self._test_not_equal(a, b) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 20, in _test_not_equal raise AssertionError("a and b are found equal but are not") AssertionError: a and b are found equal but are not ====================================================================== FAIL: Test rank 1 array for all dtypes. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 67, in test_generic_rank1 foo(t) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 63, in foo self._test_not_equal(c, b) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 20, in _test_not_equal raise AssertionError("a and b are found equal but are not") AssertionError: a and b are found equal but are not ====================================================================== FAIL: Test rank 3 array for all dtypes. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 86, in test_generic_rank3 foo(t) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 82, in foo self._test_not_equal(c, b) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 20, in _test_not_equal raise AssertionError("a and b are found equal but are not") AssertionError: a and b are found equal but are not ====================================================================== FAIL: Test arrays with nan values in them. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 100, in test_nan_array self._test_not_equal(c, b) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 20, in _test_not_equal raise AssertionError("a and b are found equal but are not") AssertionError: a and b are found equal but are not ====================================================================== FAIL: Test record arrays. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 126, in test_recarrays self._test_not_equal(c, b) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 20, in _test_not_equal raise AssertionError("a and b are found equal but are not") AssertionError: a and b are found equal but are not ====================================================================== FAIL: Test two arrays with different shapes are found not equal. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 111, in test_string_arrays self._test_not_equal(c, b) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 20, in _test_not_equal raise AssertionError("a and b are found equal but are not") AssertionError: a and b are found equal but are not ---------------------------------------------------------------------- Ran 1300 tests in 3.078s FAILED (failures=11) From ludwigbrinckmann at gmail.com Wed Aug 6 05:30:10 2008 From: ludwigbrinckmann at gmail.com (Ludwig) Date: Wed, 6 Aug 2008 10:30:10 +0100 Subject: [Numpy-discussion] Numpy random: independent streams Message-ID: <3f7a6e1c0808060230p1f5f0df1v6573ff98b195a13c@mail.gmail.com> The Python standard random API allows me to define multiple independent random streams, which I can control with their own seed, e.g. import random generator_1 = random.Random() generator_2 = random.Random() generator_1.seed(100) generator_2.seed(100) So now generator_1 and 2 will produce the same sequence. Is there an equivalent for this in numpy? (Apart from, as it seems, the oldnumeric package http://numpy.scipy.org/numpydoc/numpy-20.html)? Ludwig -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Aug 6 05:32:04 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 6 Aug 2008 04:32:04 -0500 Subject: [Numpy-discussion] Numpy random: independent streams In-Reply-To: <3f7a6e1c0808060230p1f5f0df1v6573ff98b195a13c@mail.gmail.com> References: <3f7a6e1c0808060230p1f5f0df1v6573ff98b195a13c@mail.gmail.com> Message-ID: <3d375d730808060232h6097bc8fj214db9a1c84edc88@mail.gmail.com> On Wed, Aug 6, 2008 at 04:30, Ludwig wrote: > The Python standard random API allows me to define multiple independent > random streams, which I can control with their own seed, e.g. > > import random > generator_1 = random.Random() > generator_2 = random.Random() > > generator_1.seed(100) > generator_2.seed(100) > > So now generator_1 and 2 will produce the same sequence. > > Is there an equivalent for this in numpy? (Apart from, as it seems, the > oldnumeric package http://numpy.scipy.org/numpydoc/numpy-20.html)? from numpy import random generator_1 = random.RandomState(100) generator_2 = random.RandomState(100) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From matthieu.brucher at gmail.com Wed Aug 6 05:32:20 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 6 Aug 2008 11:32:20 +0200 Subject: [Numpy-discussion] Horizontal lines in diffraction image (NumPy FFT) In-Reply-To: <710F2847B0018641891D9A216027636029C205@ex3.envision.co.il> References: <67b3a51f0808051926r1aefac3dibfbffda4100d9fb5@mail.gmail.com> <710F2847B0018641891D9A216027636029C205@ex3.envision.co.il> Message-ID: Exactly. Using FFT to do a convolution should be done after some signal processing readings ;) (That's why I hate FFT to do signal processing as well). Matthieu 2008/8/6 Nadav Horesh : > > I did about the same thing 9 year ago (in python of course). If I can recall correctly, you need to double the arrays size (with padding of 0) in order to avoid this artifact. I think that its origin is that the DFT is equivalent to periodic boundary conditions. > > Nadav. > > -----????? ??????----- > ???: numpy-discussion-bounces at scipy.org ??? Matthias Hillenbrand > ????: ? 06-??????-08 05:26 > ??: numpy-discussion at scipy.org > ????: [Numpy-discussion] Horizontal lines in diffraction image (NumPy FFT) > > Hello, > > I am trying to calculate the propagation of a Gaussian beam with an > angular spectrum propagation algorithm. > My program does the following steps: > 1. Generation and multiplication of the Gaussian beam, an aperture, > and a lens -> u > 2. FFT(u) > 3. Multiplication of FFT(u) with the angular spectrum kernel H > 4. IFFT(FU*H) > > Steps 2-4 are repeated 1000 times to show the propagation of the beam > in z-direction > > Unfortunately the calculation results in a lot of horizontal lines > that should not be there. The program produces reasonable results > besides this problem. > > It is especially interesting that an equivalent calculation with > Matlab or Octave has the same results without these artifacts. Both > calculations take approximately 90 seconds. > > I am not sure whether I am doing something wrong or there is a > difference in the FFT codes. I assume that the problem has something > to do with the propagation kernel H but I cannot see a difference > between both programs. > > Thank you very much for your help! > > Matthias > > > --------------------------------------------------------------------------------------------------- > Python code with comments (version without comments below) > --------------------------------------------------------------------------------------------------- > > from pylab import * > from numpy.lib import scimath #Complex roots > import os > > ######################################### > ########## Defintion of the input variables ######### > ######################################### > > > ##General variables > lam = 532.e-9 #wavelength in m > k = 2*pi/lam #wave number > > ##Computational array > arr_size = 80.e-3 #size of the computation array in m > N = 2**17 #points of the 1D array > > ##Aperture > ap = 40.e-3 #aperture size of the 1D array in m > > ##Gaussian beam > w0 =10.e-3 #waist radius of the gaussian beam in m > > ##Lenses (definition of the focal length) > n = 1.56 #refractive index of the glass > f = .1 #focal length in m > Phi = 1/f #focal power of the lens > r2 = 1.e18 #radius of the second lens surface > r1 = 1 / ( Phi/(n-1) + 1/r2 ) #computation of the radius of the > first lens surface > > ##z distances > zmin = 0.99*f #initial z position > zmax = 1.01*f #final z position > zsteps = 1000 #number of computated positions > ROI = 1000 #Region of interest in the diffraction image > > ############################################## > ############### Program execution 1D ################ > ############################################### > > x = linspace(-arr_size/2,arr_size/2,N) > A = ones(N) > A[where(ap/2<=abs(x))] = 0 #Definition of the aperture > G = exp(- x**2/w0**2) #Generation of the Gaussian beam > > delta = -r1*(1 - scimath.sqrt(1 - x**2/r1**2)) > + r2*(1 - scimath.sqrt(1 - x**2/r2**2)) > m = (r1**2 <= x**2 ) > delta[m] = 0 #correction in case of negative roots > m = (r2**2 <= x**2 ) > delta[m] = 0 #correction in case of negative roots > lens = exp(1j * 2 * pi / lam * (n-1) * delta) #Calculation > of the lens phase function > > u = A*G*lens #Complex amplitude before beam propagation > > ############################################ > ########### Start angular spectrum method ########### > > deltaf = 1/arr_size #spacing in frequency > domain > fx = r_[-N/2*deltaf:(N/2)*deltaf:deltaf] #whole frequency domain > FU = fftshift(fft(u)) #1D FFT > U = zeros((zsteps,ROI)) #Generation of the image array > z = linspace(zmin,zmax,zsteps) #Calculation of the > different z positions > > for i in range(zsteps): > H = exp(1j * 2 * pi / lam * z[i] * scimath.sqrt(1-(lam*fx)**2)) > U[i] = (ifft(fftshift(FU*H)))[N/2 - ROI/2 : N/2 + ROI/2] > if i%10 == 0: > t = 'z position: %4i' % i > print t > > ############ End angular spectrum method ############ > ############################################# > > res = abs(U) > > imshow(res) > show() > > > > > > > ---------------------------------------------------------- > Python code without comments > ---------------------------------------------------------- > > from pylab import * > from numpy.lib import scimath #Complex roots > import os > > lam = 532.e-9 > k = 2*pi/lam > > arr_size = 80.e-3 > N = 2**17 > > ap = 40.e-3 > > w0 =10.e-3 > > n = 1.56 > f = .1 > Phi = 1/f > r2 = 1.e18 > r1 = 1 / ( Phi/(n-1) + 1/r2 ) > > zmin = 0.99*f > zmax = 1.01*f > zsteps = 1000 > ROI = 1000 > > x = linspace(-arr_size/2,arr_size/2,N) > A = ones(N) > A[where(ap/2<=abs(x))] = 0 > G = exp(- x**2/w0**2) > > delta = -r1*(1 - scimath.sqrt(1 - x**2/r1**2)) > + r2*(1 - scimath.sqrt(1 - x**2/r2**2)) > m = (r1**2 <= x**2 ) > delta[m] = 0 > m = (r2**2 <= x**2 ) > delta[m] = 0 > lens = exp(1j * 2 * pi / lam * (n-1) * delta) > > u = A*G*lens > > deltaf = 1/arr_size > fx = r_[-N/2*deltaf:(N/2)*deltaf:deltaf] > FU = fftshift(fft(u)) > U = zeros((zsteps,ROI)) > z = linspace(zmin,zmax,zsteps) > > for i in range(zsteps): > H = exp(1j * 2 * pi / lam * z[i] * scimath.sqrt(1-(lam*fx)**2)) > U[i] = (ifft(fftshift(FU*H)))[N/2 - ROI/2 : N/2 + ROI/2] > if i%10 == 0: > t = 'z position: %4i' % i > print t > > res = abs(U) > > imshow(res) > show() > > > > > > > ---------------------------------------------------------- > Matlab, Octave code > ---------------------------------------------------------- > > lam = 532.e-9 > k = 2*pi/lam > > arr_size = 80.e-3 > N = 2^17 > > ap = 40.e-3 > > w0 =10.e-3 > > n = 1.56 > f = .1 > Phi = 1/f > r2 = 1.e18 > r1 = 1 / ( Phi/(n-1) + 1/r2 ) > > zmin = 0.99*f > zmax = 1.01*f > zsteps = 1000 > ROI = 1000 > > x=linspace(-arr_size/2,arr_size/2,N); > A = ones(1,N); > A(find(ap/2<=abs(x)))=0; > G = exp(- x.^2/w0^2); > > delta = -r1*(1 - sqrt(1 - x.^2/r1^2)) + r2*(1 - sqrt(1 - x.^2/r2^2)); > delta(find(r1.^2 <= x.^2 ))=0; > delta(find(r2.^2 <= x.^2 ))=0; > lens = exp(1j * 2 * pi / lam * (n-1) * delta); > u = A.*G.*lens; > > deltaf = 1/arr_size; > fx = [-N/2*deltaf:deltaf:(N/2-1)*deltaf]; > FU = fftshift(fft(u)); > U = zeros(zsteps,ROI); > z = linspace(zmin,zmax,zsteps); > > for i=1:zsteps > H = exp(1j * 2 * pi / lam * z(i) * sqrt(1-lam.^2*fx.^2)); > U(i,:) = ifft(fftshift(FU.*H))(N/2 - ROI/2 : N/2 + ROI/2-1); > end > > imagesc(abs(U)) > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher From markbak at gmail.com Wed Aug 6 06:00:41 2008 From: markbak at gmail.com (mark) Date: Wed, 6 Aug 2008 03:00:41 -0700 (PDT) Subject: [Numpy-discussion] confusion on importing numpy Message-ID: <12d2498a-ee23-4881-8fa6-0b5145dc7476@59g2000hsb.googlegroups.com> Hello list. I am confused about importing numpy. When I do from numpy import * and I try the min function, I get the default Python min function. On the other hand, when I do import numpy as np and use the np.min function, I get the numpy min function (which is obviously what I want). I know, the latter is preferred, but the former is so dang easy to type. I always thought both imports work the same, but I am obviously wrong. What's the difference? Thanks, Mark From robert.kern at gmail.com Wed Aug 6 06:03:20 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 6 Aug 2008 05:03:20 -0500 Subject: [Numpy-discussion] confusion on importing numpy In-Reply-To: <12d2498a-ee23-4881-8fa6-0b5145dc7476@59g2000hsb.googlegroups.com> References: <12d2498a-ee23-4881-8fa6-0b5145dc7476@59g2000hsb.googlegroups.com> Message-ID: <3d375d730808060303n639bd9ebx8aa82ad5af78395b@mail.gmail.com> On Wed, Aug 6, 2008 at 05:00, mark wrote: > Hello list. I am confused about importing numpy. > > When I do > > from numpy import * > > and I try the min function, I get the default Python min function. > > On the other hand, when I do > > import numpy as np > > and use the np.min function, I get the numpy min function (which is > obviously what I want). > > I know, the latter is preferred, but the former is so dang easy to > type. > I always thought both imports work the same, but I am obviously wrong. > What's the difference? We specifically exclude those names which would override builtins. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stefan at sun.ac.za Wed Aug 6 06:33:21 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 6 Aug 2008 12:33:21 +0200 Subject: [Numpy-discussion] Horizontal lines in diffraction image (NumPy FFT) In-Reply-To: References: <67b3a51f0808051926r1aefac3dibfbffda4100d9fb5@mail.gmail.com> <710F2847B0018641891D9A216027636029C205@ex3.envision.co.il> Message-ID: <9457e7c80808060333y3b21dff4o84b43e5abebb7052@mail.gmail.com> 2008/8/6 Matthieu Brucher : > Exactly. Using FFT to do a convolution should be done after some > signal processing readings ;) When you convolve two signals, of lengths N and M, you need to pad the FFTs to length (N+M-1) before multiplication. You can take a look at my linear position-invariant filtering code at: http://mentat.za.net/hg/filter To construct a Gaussian filter, for example, you do: >>> def filt_func(r,c): return np.exp(-np.hypot(r,c)/1) >>> filter = LPIFilter2D(filt_func) You may then apply it on data by doing `filter(data)`. I also implemented inverse filtering, which can be accessed using `filter.inverse` and `filter.wiener`. Regards St?fan From markbak at gmail.com Wed Aug 6 06:44:32 2008 From: markbak at gmail.com (mark) Date: Wed, 6 Aug 2008 03:44:32 -0700 (PDT) Subject: [Numpy-discussion] confusion on importing numpy In-Reply-To: <3d375d730808060303n639bd9ebx8aa82ad5af78395b@mail.gmail.com> References: <12d2498a-ee23-4881-8fa6-0b5145dc7476@59g2000hsb.googlegroups.com> <3d375d730808060303n639bd9ebx8aa82ad5af78395b@mail.gmail.com> Message-ID: I guess that makes sense on a certain level, but boy it is cumbersome to explain to a class. It pretty much defeats the whole purpose of doing from numpy import *. Mark On Aug 6, 12:03?pm, "Robert Kern" wrote: > On Wed, Aug 6, 2008 at 05:00, mark wrote: > > Hello list. I am confused about importing numpy. > > > When I do > > > from numpy import * > > > and I try the min function, I get the default Python min function. > > > On the other hand, when I do > > > import numpy as np > > > and use the np.min function, I get the numpy min function (which is > > obviously what I want). > > > I know, the latter is preferred, but the former is so dang easy to > > type. > > I always thought both imports work the same, but I am obviously wrong. > > What's the difference? > > We specifically exclude those names which would override builtins. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ?-- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion From wbaxter at gmail.com Wed Aug 6 06:50:14 2008 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 6 Aug 2008 19:50:14 +0900 Subject: [Numpy-discussion] confusion on importing numpy In-Reply-To: References: <12d2498a-ee23-4881-8fa6-0b5145dc7476@59g2000hsb.googlegroups.com> <3d375d730808060303n639bd9ebx8aa82ad5af78395b@mail.gmail.com> Message-ID: Anything that defeats the purpose of doing * imports is good in my book. :-) Seriously, willy nilly import of any package into the base namespace is just asking for trouble. Tell your class to import numpy as np, then there will be no chance of confusion. Then later tell them about "from numpy import min,max,array". Or just have them do "min = np.min" etc. --bb On Wed, Aug 6, 2008 at 7:44 PM, mark wrote: > I guess that makes sense on a certain level, but boy it is cumbersome > to explain to a class. > It pretty much defeats the whole purpose of doing from numpy import *. > > Mark > > On Aug 6, 12:03 pm, "Robert Kern" wrote: >> On Wed, Aug 6, 2008 at 05:00, mark wrote: >> > Hello list. I am confused about importing numpy. >> >> > When I do >> >> > from numpy import * >> >> > and I try the min function, I get the default Python min function. >> >> > On the other hand, when I do >> >> > import numpy as np >> >> > and use the np.min function, I get the numpy min function (which is >> > obviously what I want). >> >> > I know, the latter is preferred, but the former is so dang easy to >> > type. >> > I always thought both imports work the same, but I am obviously wrong. >> > What's the difference? >> >> We specifically exclude those names which would override builtins. >> >> -- >> Robert Kern >> >> "I have come to believe that the whole world is an enigma, a harmless >> enigma that is made terrible by our own mad attempt to interpret it as >> though it had an underlying truth." >> -- Umberto Eco From cimrman3 at ntc.zcu.cz Wed Aug 6 07:45:34 2008 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Wed, 06 Aug 2008 13:45:34 +0200 Subject: [Numpy-discussion] member1d and unique elements In-Reply-To: References: <48987908.9080703@ntc.zcu.cz> Message-ID: <48998EDE.4010507@ntc.zcu.cz> Hi Greg, Greg Novak wrote: > Argh. I could swear that yesterday I typed test cases just like the > one you provide, and it behaved correctly. Nevertheless, it clearly > fails in spite of my memory, so attached is a version which I believe > gives the correct behavior. It looks ok now, although I did not check very carefully. Just make sure that it works for empty input arrays (all combinations). I would also set the default to handle_dupes to False to preserve previous functionality. I suggest you refresh your numpy repository from SVN - now np. prefix is used instead of nm. within the module. Thank you! r. From cimrman3 at ntc.zcu.cz Wed Aug 6 08:52:38 2008 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Wed, 06 Aug 2008 14:52:38 +0200 Subject: [Numpy-discussion] unique1d returning indices Message-ID: <48999E96.6070000@ntc.zcu.cz> Hi, due to popular demand, I have updated unique1d() to optionally return both kinds of indices: In [3]: b, i, j = nm.unique1d( a, return_index=True, return_inverse=True ) In [4]: a Out[4]: array([1, 1, 8, 3, 3, 5, 4]) In [6]: b Out[6]: array([1, 3, 4, 5, 8]) In [7]: a[i] Out[7]: array([1, 3, 4, 5, 8]) In [8]: b[j] Out[8]: array([1, 1, 8, 3, 3, 5, 4]) In [9]: a Out[9]: array([1, 1, 8, 3, 3, 5, 4]) Could someone needing this try the attached code? If it is ok I will update the tests, the docstring and commit it. Note also that the order of outputs has changed (previously unique1d() returned (i, b) for return_index=True). cheers, r. -------------- next part -------------- A non-text attachment was scrubbed... Name: unique1d.py Type: text/x-python Size: 2055 bytes Desc: not available URL: From matthew.czesarski at gmail.com Wed Aug 6 09:02:10 2008 From: matthew.czesarski at gmail.com (Matthew Czesarski) Date: Wed, 6 Aug 2008 15:02:10 +0200 Subject: [Numpy-discussion] Slicing without a priori knowledge of dimension Message-ID: <887b0c40808060602hebaf5b3y80b373e98573acd3@mail.gmail.com> Dear list. I've got a feeling that what I'm trying to do *should* be easy but at the moment I can't find a non-brute-force method. I'm working with quite a high-rank matrix; 7 dimensions filled with chi**2 values. It's form is something like this: chi2 = numpy.ones((3,4,5,6,7,8,9)) What I need to do is slice out certain 2 dimensional grids that I then want to plot for confidence estimation; make a nice graph. This is fine: as easy as it gets if the 2 dimensions are known. E.g. for the 2nd and 5th axes, one could hardcode something like this: subChi2 = chi2[ ia, ib, :, id, ie, :, ig ] The difficulty is that the user is to state which plane (s)he wants to slice out and we can't code the above. The function itself has to convert 2 rank numbers into an expression for a slice and I can't currently figure out how to do that. There are a huge number of options (well, there are 7*6=42). If one could just manually write a colon into a tuple like (2,2,:,2,2,:,2) or something even like 2,2,0:len(2ndDim),2,2,0:len(5thDim),2, things would be fine. But that doesn't seem to be an option. An alternative would be to set up 2 loops and incrementally read out the individual elements of the slice to a new numpy.zeros ( ( len(2ndDim), len(5thDim) ) ), but again we seem to be diverging from elegance. There must be something that I'm missing. Could somebody have a pointer as to what it is? Thanks in advance, Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From hoytak at gmail.com Wed Aug 6 09:12:35 2008 From: hoytak at gmail.com (Hoyt Koepke) Date: Wed, 6 Aug 2008 16:12:35 +0300 Subject: [Numpy-discussion] Slicing without a priori knowledge of dimension In-Reply-To: <887b0c40808060602hebaf5b3y80b373e98573acd3@mail.gmail.com> References: <887b0c40808060602hebaf5b3y80b373e98573acd3@mail.gmail.com> Message-ID: <4db580fd0808060612q51984984jce5f74d781bc8157@mail.gmail.com> Try using slice (python builtin) to create slice objects (what is created implicitly by :5, 1:20, etc.). slice takes the same arguments as range. A list of these (7 in your case) can then be passed to A[...] as a tuple. That's how I would do it, but maybe someone else has a better idea or can correct me. --Hoyt On Wed, Aug 6, 2008 at 4:02 PM, Matthew Czesarski wrote: > Dear list. > > I've got a feeling that what I'm trying to do *should* be easy but at the > moment I can't find a non-brute-force method. > > I'm working with quite a high-rank matrix; 7 dimensions filled with chi**2 > values. It's form is something like this: > > chi2 = numpy.ones((3,4,5,6,7,8,9)) > > What I need to do is slice out certain 2 dimensional grids that I then want > to plot for confidence estimation; make a nice graph. This is fine: as easy > as it gets if the 2 dimensions are known. E.g. for the 2nd and 5th axes, one > could hardcode something like this: > > subChi2 = chi2[ ia, ib, :, id, ie, :, ig ] > > The difficulty is that the user is to state which plane (s)he wants to slice > out and we can't code the above. The function itself has to convert 2 rank > numbers into an expression for a slice and I can't currently figure out how > to do that. There are a huge number of options (well, there are 7*6=42). If > one could just manually write a colon into a tuple like (2,2,:,2,2,:,2) or > something even like 2,2,0:len(2ndDim),2,2,0:len(5thDim),2, things would be > fine. But that doesn't seem to be an option. An alternative would be to set > up 2 loops and incrementally read out the individual elements of the slice > to a new numpy.zeros ( ( len(2ndDim), len(5thDim) ) ), but again we seem to > be diverging from elegance. > > There must be something that I'm missing. Could somebody have a pointer as > to what it is? > > Thanks in advance, > Matt > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- +++++++++++++++++++++++++++++++++++ Hoyt Koepke UBC Department of Computer Science http://www.cs.ubc.ca/~hoytak/ hoytak at gmail.com +++++++++++++++++++++++++++++++++++ From gael.varoquaux at normalesup.org Wed Aug 6 09:33:07 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 6 Aug 2008 15:33:07 +0200 Subject: [Numpy-discussion] Cython/NumPy syntax In-Reply-To: <4899623A.2040504@student.matnat.uio.no> References: <48995B08.6030707@student.matnat.uio.no> <9457e7c80808060114j73ec2a8cl930791eecfcaf690@mail.gmail.com> <4899623A.2040504@student.matnat.uio.no> Message-ID: <20080806133307.GA17470@phare.normalesup.org> On Wed, Aug 06, 2008 at 10:35:06AM +0200, Dag Sverre Seljebotn wrote: > St?fan van der Walt wrote: > > 2008/8/6 Dag Sverre Seljebotn : > >> - Require an ndim keyword: > >> cdef numpy.ndarray[numpy.int64, ndim=2] > > I'd definitely prefer a comma between the two, and an (optional) ndim > > keyword argument if possible. > I'm taking this as a vote in favor of this and against "2D" and <>? > The keyword is already present in optional form (you can do already do > "ndarray[ndim=3, dtype=numpy.int64]" if you want to). So I meant to ask > whether making it mandatory is a good solution so that the 2 doesn't > look like a length specifier. > (ndim=2 seems too long for me though, which is why I am pondering "2D".) I am definitely +1 on ndim keyword argument. Make it compulsory if you have to. I don't really like 2D. The nice thing about the ndim keyword argument is that is mean we still are using valid Python syntax which means that iit is easier to grok for the user, and which may come in handy one day, as we can use standard Python tools on the expression. And I don't thing that "ndim=2" is too long. Ga?l From matthew.czesarski at gmail.com Wed Aug 6 09:46:43 2008 From: matthew.czesarski at gmail.com (Matthew Czesarski) Date: Wed, 6 Aug 2008 15:46:43 +0200 Subject: [Numpy-discussion] Slicing without a priori knowledge of dimension In-Reply-To: <4db580fd0808060612q51984984jce5f74d781bc8157@mail.gmail.com> References: <887b0c40808060602hebaf5b3y80b373e98573acd3@mail.gmail.com> <4db580fd0808060612q51984984jce5f74d781bc8157@mail.gmail.com> Message-ID: <887b0c40808060646k72246d98q863ff9f1ee45296b@mail.gmail.com> Dear Hoyt, Thanks -- that hit the spot! Looks like it was a case of me looking for the answer in the wrong place. Assuming that there would be some sophisticated numpy method -- E.g. A.dim_slice( ( 2,5,... ) ) -- that slices up the array and gives you back the sub-full dimensional matrix you're looking for. But it was there all along as a builtin... Thanks for the pointer, Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Wed Aug 6 10:10:41 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Wed, 06 Aug 2008 09:10:41 -0500 Subject: [Numpy-discussion] Cython/NumPy syntax In-Reply-To: <20080806133307.GA17470@phare.normalesup.org> References: <48995B08.6030707@student.matnat.uio.no> <9457e7c80808060114j73ec2a8cl930791eecfcaf690@mail.gmail.com> <4899623A.2040504@student.matnat.uio.no> <20080806133307.GA17470@phare.normalesup.org> Message-ID: <4899B0E1.90606@enthought.com> Gael Varoquaux wrote: > On Wed, Aug 06, 2008 at 10:35:06AM +0200, Dag Sverre Seljebotn wrote: > >> St?fan van der Walt wrote: >> >>> 2008/8/6 Dag Sverre Seljebotn : >>> >>>> - Require an ndim keyword: >>>> > > >>>> cdef numpy.ndarray[numpy.int64, ndim=2] >>>> Just out of curiousity. What is the problem with using parenthesis for this purpose? cdef numpy.ndarray(dtype=numpy.int64, ndim=2) -Travis From dagss at student.matnat.uio.no Wed Aug 6 10:45:39 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 Aug 2008 16:45:39 +0200 Subject: [Numpy-discussion] Cython/NumPy syntax In-Reply-To: <4899B0E1.90606@enthought.com> References: <48995B08.6030707@student.matnat.uio.no> <9457e7c80808060114j73ec2a8cl930791eecfcaf690@mail.gmail.com> <4899623A.2040504@student.matnat.uio.no> <20080806133307.GA17470@phare.normalesup.org> <4899B0E1.90606@enthought.com> Message-ID: <4899B913.9010301@student.matnat.uio.no> Travis E. Oliphant wrote: > Gael Varoquaux wrote: >> On Wed, Aug 06, 2008 at 10:35:06AM +0200, Dag Sverre Seljebotn wrote: >> >>> St?fan van der Walt wrote: >>> >>>> 2008/8/6 Dag Sverre Seljebotn : >>>> >>>>> - Require an ndim keyword: >>>>> >> >>>>> cdef numpy.ndarray[numpy.int64, ndim=2] >>>>> > > Just out of curiousity. What is the problem with using parenthesis for > this purpose? > > cdef numpy.ndarray(dtype=numpy.int64, ndim=2) There's no technical problem, but we thought that it looked too much like constructor syntax -- it looks like an ndarray is constructed. If one is new to Cython this is what you will assume, at least the [] makes you stop up and think more. (Which, for clarity, I should mention that it is not -- you'd do cdef np.ndarray(dtype=np.int64, ndim=1) buf = \ np.array([1,2,3], dtype=np.int64) to construct a new array and get buffer access to it right away). Also, the argument list on the type is not defined by the ndarray constructor but corresponds to your buffer PEP, and I think using () obscures this fact. For NumPy this is not such a big problem as the argument lists will be rather similar, but for other libraries than NumPy supporting the buffer PEP the argument list may diverge more: cdef MyJpegImage(dtype=unsigned char, ndim=2) jpg = \ MyJpegImage("file.jpg") (and even worse if the keywords are not mandatory). Another example: One can currently do (dropping keywords) cdef object[double, 4, "strided"] buf = ... to get generic 4D strided buffer access that doesn't make assumptions about the class, but the Python object() takes no parameters... (BTW, there is a mechanism so that simpler buffer exporters like MyJpegImage can declare default buffer options, so one is unlikely to see the above in practice -- if the type always has the same dtype and ndim, it is enough to do cdef MyJpegImage jpg = ... For ndarray I use the same mechanism to let mode="strided" by default. But I should start writing documentation rather than making this email longer :-) ) -- Dag Sverre From dagss at student.matnat.uio.no Wed Aug 6 11:04:39 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 Aug 2008 17:04:39 +0200 Subject: [Numpy-discussion] Cython/NumPy syntax In-Reply-To: <4899B913.9010301@student.matnat.uio.no> References: <48995B08.6030707@student.matnat.uio.no> <9457e7c80808060114j73ec2a8cl930791eecfcaf690@mail.gmail.com> <4899623A.2040504@student.matnat.uio.no> <20080806133307.GA17470@phare.normalesup.org> <4899B0E1.90606@enthought.com> <4899B913.9010301@student.matnat.uio.no> Message-ID: <4899BD87.5010903@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Travis E. Oliphant wrote: >> Gael Varoquaux wrote: >>> On Wed, Aug 06, 2008 at 10:35:06AM +0200, Dag Sverre Seljebotn wrote: >>> >>>> St?fan van der Walt wrote: >>>> >>>>> 2008/8/6 Dag Sverre Seljebotn : >>>>> >>>>>> - Require an ndim keyword: >>>>>> >>> >>>>>> cdef numpy.ndarray[numpy.int64, ndim=2] >>>>>> >> Just out of curiousity. What is the problem with using parenthesis for >> this purpose? >> >> cdef numpy.ndarray(dtype=numpy.int64, ndim=2) > > There's no technical problem, but we thought that it looked too much > like constructor syntax -- it looks like an ndarray is constructed. If > one is new to Cython this is what you will assume, at least the [] makes > you stop up and think more. > > (Which, for clarity, I should mention that it is not -- you'd do > > cdef np.ndarray(dtype=np.int64, ndim=1) buf = \ > np.array([1,2,3], dtype=np.int64) > > to construct a new array and get buffer access to it right away). I realize that I've given too little context for this discussion. This tends to get rather longwinded, but I'll provide it for whoever is interested. What I am doing is supporting general syntax candy for the buffer PEP (and a backwards compatability layer for earlier Python versions) so that cdef object[float, 2] buf = input acquires a buffer and lets you use it using the indexing operator with 2 native int indices, as well as letting other operations (also any indexing that doesn't have exactly 2 ints) fall through to the underlying object. The most explicit syntax would be cdef ndarray arr = input cdef buffer[float, 2] buf = cython.getbuffer(arr) arr += 4.3 buf[3,2] = 2 But that is very unfriendly to use. A step down in explicitness is cdef buffer(ndarray, float, 2) arr = input arr += 4.3 # falls through to ndarray type arr[3,2] = 2 # uses buffer But overall just adding something to the end of ndarray and make it completely transparent seemed most usable. (Another option would be "cdef ndarray,buffer(float,2) arr = ...", i.e. arr "has two types".). -- Dag Sverre From mdroe at stsci.edu Wed Aug 6 11:01:48 2008 From: mdroe at stsci.edu (Michael Droettboom) Date: Wed, 06 Aug 2008 11:01:48 -0400 Subject: [Numpy-discussion] Segfault in PyArray_Item_XDECREF when using recarray object references titles In-Reply-To: <48863AFE.2040109@stsci.edu> References: <488636BF.9050205@stsci.edu> <48863AFE.2040109@stsci.edu> Message-ID: <4899BCDC.9090407@stsci.edu> I've filed a bug, with a patch to address all these issues, here: http://scipy.org/scipy/numpy/ticket/877 Cheers, Mike Michael Droettboom wrote: > I also noticed that the inverse operation, PyArray_Item_INCREF has the > potential to leak memory as it will doubly-increment each object in the > array. The solution there probably isn't quite as clean, since we can't > just mark the pointer. It will have to somehow avoid incref'ing the > objects twice when iterating through the fields dictionary. > > Cheers, > Mike > > Michael Droettboom wrote: > >> I've run into a segfault that occurs in the array destructor with >> arrays containing object references with both names and titles. >> >> When a field contains both and name and a title, the fields dictionary >> contains two entries for that field. This means that the array item >> destructor (which iterates through the fields dictionary) will decref >> the pointed-to object twice. If the first decref causes the object to >> be deleted, the second decref has the potential to segfault. >> >> It seems the simplest patch is to set the object pointer to NULL after >> decref'ing, so the second decref will do nothing. However, perhaps >> there is a way to avoid decref'ing twice in the first place. >> >> I've attached a script that exercises the segfault, a gdb backtrace, >> and a patch. You may need to adjust the number of rows until it is >> high enough to create a segfault on your system. >> >> This is on: >> RHEL4 >> Python 2.5.2 >> Numpy SVN r5497 >> >> Cheers, >> Mike >> >> >>> gdb python >>> >> GNU gdb Red Hat Linux (6.3.0.0-1.153.el4_6.2rh) >> Copyright 2004 Free Software Foundation, Inc. >> GDB is free software, covered by the GNU General Public License, and >> you are >> welcome to change it and/or distribute copies of it under certain >> conditions. >> Type "show copying" to see the conditions. >> There is absolutely no warranty for GDB. Type "show warranty" for >> details. >> This GDB was configured as "i386-redhat-linux-gnu"...Using host >> libthread_db library "/lib/tls/libthread_db.so.1". >> >> (gdb) run segfault.py >> Starting program: /wonkabar/data1/usr/bin/python segfault.py >> [Thread debugging using libthread_db enabled] >> [New Thread -1208489312 (LWP 30028)] >> len(dtype) = 1, len(dtype.fields) = 2 >> {'name': (dtype('object'), 0, 'title'), 'title': (dtype('object'), 0, >> 'title')} >> >> Program received signal SIGSEGV, Segmentation fault. >> [Switching to Thread -1208489312 (LWP 30028)] >> 0x0097285e in PyArray_Item_XDECREF ( >> data=0xb7a3e780 "\uffff_\224\uffff >> `\214\uffff(`\214\uffff0`\214\uffff8`\214\uffff@`\214\uffffH`\214\uffffP`\214\uffffX`\214\uffff``\214\uffffh`\214\uffffp`\214\uffffx`\214\uffff\200`\214\uffff\210`\214\uffff\220`\214\uffff\230`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff", >> >> descr=0x9d4680) at numpy/core/src/arrayobject.c:198 >> 198 Py_XDECREF(*temp); >> (gdb) bt >> #0 0x0097285e in PyArray_Item_XDECREF ( >> data=0xb7a3e780 "\uffff_\224\uffff >> `\214\uffff(`\214\uffff0`\214\uffff8`\214\uffff@`\214\uffffH`\214\uffffP`\214\uffffX`\214\uffff``\214\uffffh`\214\uffffp`\214\uffffx`\214\uffff\200`\214\uffff\210`\214\uffff\220`\214\uffff\230`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff", >> >> descr=0x9d4680) at numpy/core/src/arrayobject.c:198 >> #1 0x00991bc7 in PyArray_XDECREF (mp=0xb7ae4f0c) >> at numpy/core/src/arrayobject.c:211 >> #2 0x009a579b in array_dealloc (self=0xb7ae4f0c) >> at numpy/core/src/arrayobject.c:2089 >> #3 0x0809781f in subtype_dealloc (self=0xb7ae4f0c) at >> Objects/typeobject.c:709 >> #4 0x08082a02 in PyDict_SetItem (op=0xb7f56acc, key=0xb7ea7d80, >> value=0x81379c0) at Objects/dictobject.c:416 >> #5 0x08085a1e in _PyModule_Clear (m=0xb7f3e0ec) at >> Objects/moduleobject.c:136 >> #6 0x080d7138 in PyImport_Cleanup () at Python/import.c:439 >> #7 0x080e4343 in Py_Finalize () at Python/pythonrun.c:399 >> #8 0x08056633 in Py_Main (argc=1, argv=0xbff1ca24) at Modules/main.c:545 >> #9 0x08056323 in main (argc=2, argv=0xbff1ca24) at ./Modules/python.c:23 >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion >> > > -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA From Shawn.Gong at drdc-rddc.gc.ca Wed Aug 6 11:46:00 2008 From: Shawn.Gong at drdc-rddc.gc.ca (Gong, Shawn (Contractor)) Date: Wed, 6 Aug 2008 11:46:00 -0400 Subject: [Numpy-discussion] cubic spline function in numarray or numpy In-Reply-To: References: Message-ID: hi list, I am trying to find 1-D cubic spline function. But I am not able to call it. Error message can't find "ndimage" Would someone point me to 1-D cubic spline functions either in numarray or numpy? and their help pages? I use python 2.3.5 and numpy 1.0.1. Are they too old to use cubic spline? thanks, Shawn -------------- next part -------------- An HTML attachment was scrubbed... URL: From meine at informatik.uni-hamburg.de Wed Aug 6 11:47:51 2008 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Wed, 6 Aug 2008 17:47:51 +0200 Subject: [Numpy-discussion] dot/inner with kwargs? Message-ID: <200808061747.51738.meine@informatik.uni-hamburg.de> Hi, I am trying to do something like the following as efficiently as possible: result[...,0] = (dat * cos(arange(100))).sum(axis = -1) result[...,1] = (dat * sin(arange(100))).sum(axis = -1) Thus, I was looking for dot / inner with an 'output' arg. Q1: What is the difference between dot and inner (apart from the fact that dot does not seem to have docs)? In http://www.scipy.org/Numpy_Example_List_With_Doc#inner, I read that: > Like the generic NumPy equivalent the product sum is over > the last dimension of a and b. (BTW: "innerproduct" should be "inner") Q2: What is "the generic NumPy equivalent" here? -- Ciao, / / /--/ / / ANS From harry.mangalam at uci.edu Wed Aug 6 11:54:17 2008 From: harry.mangalam at uci.edu (Harry Mangalam) Date: Wed, 6 Aug 2008 08:54:17 -0700 Subject: [Numpy-discussion] f2py: undefined symbol: zgemm_ In-Reply-To: <200807301503.44503.harry.mangalam@uci.edu> References: <200807301503.44503.harry.mangalam@uci.edu> Message-ID: <200808060854.18158.harry.mangalam@uci.edu> The problem seems to have been a conflicting version of liblapack. I installed a newer version of liblapack (liblapack-dev, from the Ubuntu 8.04 tree) and recompiled and that seems to have addressed the issue. Even tho the previously offending symbols are still undefined, the wrapped code doesn't complain about them and it runs to completion satisfactorily. hjm On Wednesday 30 July 2008, Harry Mangalam wrote: > ?I had to regenerate it after some code changes (just commenting > out some debugging info). ?Altho the shared lib gets built > correctly, when the calling python is run, I now get this: > > ./1d.py > Traceback (most recent call last): > ? File "./1d.py", line 27, in > ? ? from fd_rrt1d import * > ImportError: /home/hjm/shaka/1D-Mangalam-py/fd_rrt1d.so: undefined > symbol: zgemm_ > > sure enough, nm reports it as undefined: > > ?nm fd_rrt1d.so |tail > 000065b0 t string_from_pyobj > ? ? ? ? ?U strlen@@GLIBC_2.0 > ? ? ? ? ?U strncpy@@GLIBC_2.0 > 0001df40 b u.1294 > 00013bee T umatrix1d_dms0_ > 000128c3 T umatrix1d_dms1_ > 00015458 T umatrix1d_ms0_ > ? ? ? ? ?U zgemm_ > ? ? ? ? ?U zgemv_ > ? ? ? ? ?U zgesv_ -- Harry Mangalam - Research Computing, NACS, E2148, Engineering Gateway, UC Irvine 92697 949 824-0084(o), 949 285-4487(c) -- ..Kick at the darkness til it bleeds daylight. (Lovers in a Dangerous Time) - Bruce Cockburn From zachary.pincus at yale.edu Wed Aug 6 12:09:57 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 6 Aug 2008 12:09:57 -0400 Subject: [Numpy-discussion] cubic spline function in numarray or numpy In-Reply-To: References: Message-ID: Hi Shawn, > I am trying to find 1-D cubic spline function. But I am not able to > call it. Error message can?t find ?ndimage? > Numpy provides tools for creating and dealing with multidimensional arrays and some basic FFT and linear algebra functions. You want to look at scipy, a set of packages built on numpy that provide more sophisticated domain-specific functionality. Specifically, ndimage is part of *scipy*, not numpy. Zach On Aug 6, 2008, at 11:46 AM, Gong, Shawn (Contractor) wrote: > hi list, > > I am trying to find 1-D cubic spline function. But I am not able to > call it. Error message can?t find ?ndimage? > > Would someone point me to 1-D cubic spline functions either in > numarray or numpy? and their help pages? > > > I use python 2.3.5 and numpy 1.0.1. Are they too old to use cubic > spline? > > > thanks, > > Shawn > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From nadavh at visionsense.com Wed Aug 6 12:16:01 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed, 6 Aug 2008 19:16:01 +0300 Subject: [Numpy-discussion] cubic spline function in numarray or numpy References: Message-ID: <710F2847B0018641891D9A216027636029C207@ex3.envision.co.il> You can access nd_image by: from numpy.numarray import nd_image but as Zach noted, scipy.interpolate is what you are looking for. Nadav. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Zachary Pincus ????: ? 06-??????-08 19:09 ??: Discussion of Numerical Python ????: Re: [Numpy-discussion] cubic spline function in numarray or numpy Hi Shawn, > I am trying to find 1-D cubic spline function. But I am not able to > call it. Error message can?t find ?ndimage? > Numpy provides tools for creating and dealing with multidimensional arrays and some basic FFT and linear algebra functions. You want to look at scipy, a set of packages built on numpy that provide more sophisticated domain-specific functionality. Specifically, ndimage is part of *scipy*, not numpy. Zach On Aug 6, 2008, at 11:46 AM, Gong, Shawn (Contractor) wrote: > hi list, > > I am trying to find 1-D cubic spline function. But I am not able to > call it. Error message can?t find ?ndimage? > > Would someone point me to 1-D cubic spline functions either in > numarray or numpy? and their help pages? > > > I use python 2.3.5 and numpy 1.0.1. Are they too old to use cubic > spline? > > > thanks, > > Shawn > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3754 bytes Desc: not available URL: From Shawn.Gong at drdc-rddc.gc.ca Wed Aug 6 11:35:24 2008 From: Shawn.Gong at drdc-rddc.gc.ca (Gong, Shawn (Contractor)) Date: Wed, 6 Aug 2008 11:35:24 -0400 Subject: [Numpy-discussion] cubic spline function in numarray or numpy Message-ID: hi list, I am trying to find 1-D cubic spline function. Google search yields that there is a function called spline_filter1d in numarray. But I am not able to call it. Would someone point me to 1-D cubic spline function either in numarray or numpy? I can find source codes and enter in Python. But I prefer the built-in function. thanks, Shawn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jturner at gemini.edu Wed Aug 6 12:13:50 2008 From: jturner at gemini.edu (James Turner) Date: Wed, 06 Aug 2008 12:13:50 -0400 Subject: [Numpy-discussion] cubic spline function in numarray or numpy In-Reply-To: References: Message-ID: <4899CDBE.7070708@gemini.edu> Hi Shawn, > I am trying to find 1-D cubic spline function. But I am not able to call > it. Error message can?t find ?ndimage? Ndimage is a module in SciPy, so you'd need to install that first, not just NumPy. I'm not really familar with the available 1-D cubic spline functions, but I think you will find those in SciPy too. > Would someone point me to 1-D cubic spline functions either in numarray > or numpy? and their help pages? Try looking at the cookbook: http://scipy.org/Cookbook > I use python 2.3.5 and numpy 1.0.1. Are they too old to use cubic spline? I doubt it. Cheers, James. From Shawn.Gong at drdc-rddc.gc.ca Wed Aug 6 12:25:30 2008 From: Shawn.Gong at drdc-rddc.gc.ca (Gong, Shawn (Contractor)) Date: Wed, 6 Aug 2008 12:25:30 -0400 Subject: [Numpy-discussion] cubic spline function in numarray or numpy In-Reply-To: <710F2847B0018641891D9A216027636029C207@ex3.envision.co.il> References: <710F2847B0018641891D9A216027636029C207@ex3.envision.co.il> Message-ID: Thank you both, Nadav and Zach. Shawn -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Nadav Horesh Sent: Wednesday, August 06, 2008 12:16 PM To: Discussion of Numerical Python Subject: RE: [Numpy-discussion] cubic spline function in numarray or numpy You can access nd_image by: from numpy.numarray import nd_image but as Zach noted, scipy.interpolate is what you are looking for. Nadav. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Zachary Pincus ????: ? 06-??????-08 19:09 ??: Discussion of Numerical Python ????: Re: [Numpy-discussion] cubic spline function in numarray or numpy Hi Shawn, > I am trying to find 1-D cubic spline function. But I am not able to > call it. Error message can?t find ?ndimage? > Numpy provides tools for creating and dealing with multidimensional arrays and some basic FFT and linear algebra functions. You want to look at scipy, a set of packages built on numpy that provide more sophisticated domain-specific functionality. Specifically, ndimage is part of *scipy*, not numpy. Zach On Aug 6, 2008, at 11:46 AM, Gong, Shawn (Contractor) wrote: > hi list, > > I am trying to find 1-D cubic spline function. But I am not able to > call it. Error message can?t find ?ndimage? > > Would someone point me to 1-D cubic spline functions either in > numarray or numpy? and their help pages? > > > I use python 2.3.5 and numpy 1.0.1. Are they too old to use cubic > spline? > > > thanks, > > Shawn > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion From Chris.Barker at noaa.gov Wed Aug 6 12:57:42 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 06 Aug 2008 09:57:42 -0700 Subject: [Numpy-discussion] Cython/NumPy syntax In-Reply-To: <48995B08.6030707@student.matnat.uio.no> References: <48995B08.6030707@student.matnat.uio.no> Message-ID: <4899D806.9030807@noaa.gov> Dag Sverre Seljebotn wrote: > cdef numpy.ndarray[numpy.int64, ndim=2] +1 it's very clear what this means. I think the keyword should be required. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Wed Aug 6 13:04:23 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 06 Aug 2008 10:04:23 -0700 Subject: [Numpy-discussion] confusion on importing numpy In-Reply-To: References: <12d2498a-ee23-4881-8fa6-0b5145dc7476@59g2000hsb.googlegroups.com> <3d375d730808060303n639bd9ebx8aa82ad5af78395b@mail.gmail.com> Message-ID: <4899D997.8030001@noaa.gov> mark wrote: > I guess that makes sense on a certain level, but boy it is cumbersome > to explain to a class. > It pretty much defeats the whole purpose of doing from numpy import *. which is fine by me: "Namespaces are one honking great idea -- let's do more of those!" explain namespaces to your class -- that will teach them something truly useful. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From gruben at bigpond.net.au Wed Aug 6 12:05:42 2008 From: gruben at bigpond.net.au (Gary Ruben) Date: Wed, 06 Aug 2008 17:05:42 +0100 Subject: [Numpy-discussion] cubic spline function in numarray or numpy In-Reply-To: References: Message-ID: <4899CBD6.2080501@bigpond.net.au> You're best off using scipy. Just Google search for "scipy spline": e.g. Gary R. Gong, Shawn (Contractor) wrote: > hi list, > > I am trying to find 1-D cubic spline function. But I am not able to call > it. Error message can?t find ?ndimage? > > Would someone point me to 1-D cubic spline functions either in numarray > or numpy? and their help pages? > > I use python 2.3.5 and numpy 1.0.1. Are they too old to use cubic spline? > > thanks, > > Shawn From tom.denniston at alum.dartmouth.org Wed Aug 6 13:38:07 2008 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Wed, 6 Aug 2008 12:38:07 -0500 Subject: [Numpy-discussion] Cython/NumPy syntax In-Reply-To: <4899D806.9030807@noaa.gov> References: <48995B08.6030707@student.matnat.uio.no> <4899D806.9030807@noaa.gov> Message-ID: I think the square brackets are very confusing as a numpy user not familiar with CPython. On 8/6/08, Christopher Barker wrote: > Dag Sverre Seljebotn wrote: > > cdef numpy.ndarray[numpy.int64, ndim=2] > > +1 it's very clear what this means. I think the keyword should be required. > > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From efiring at hawaii.edu Wed Aug 6 14:21:09 2008 From: efiring at hawaii.edu (Eric Firing) Date: Wed, 06 Aug 2008 08:21:09 -1000 Subject: [Numpy-discussion] confusion on importing numpy In-Reply-To: <12d2498a-ee23-4881-8fa6-0b5145dc7476@59g2000hsb.googlegroups.com> References: <12d2498a-ee23-4881-8fa6-0b5145dc7476@59g2000hsb.googlegroups.com> Message-ID: <4899EB95.4070809@hawaii.edu> mark wrote: > Hello list. I am confused about importing numpy. > > When I do > > from numpy import * > > and I try the min function, I get the default Python min function. > > On the other hand, when I do > > import numpy as np > > and use the np.min function, I get the numpy min function (which is > obviously what I want). > > I know, the latter is preferred, but the former is so dang easy to > type. > I always thought both imports work the same, but I am obviously wrong. > What's the difference? While I agree with the other posters that "import *" is not preferred, if you want to use it, the solution is to use amin and amax, which are provided precisely to avoid the conflict. Just as arange is a numpy analog of range, amin and amax are numpy analogs of min and max. Eric > > Thanks, Mark > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From chanley at stsci.edu Wed Aug 6 14:59:52 2008 From: chanley at stsci.edu (Christopher Hanley) Date: Wed, 06 Aug 2008 14:59:52 -0400 Subject: [Numpy-discussion] cubic spline function in numarray or numpy In-Reply-To: <710F2847B0018641891D9A216027636029C207@ex3.envision.co.il> References: <710F2847B0018641891D9A216027636029C207@ex3.envision.co.il> Message-ID: <4899F4A8.5030007@stsci.edu> Nadav Horesh wrote: > You can access nd_image by: > > from numpy.numarray import nd_image > > but as Zach noted, scipy.interpolate is what you are looking for. > > Nadav. > The ndimage modules is now officially supported by the scipy project. You can get it from scipy.ndimage(). Chris -- Christopher Hanley Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338 From bsouthey at gmail.com Wed Aug 6 15:15:12 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 06 Aug 2008 14:15:12 -0500 Subject: [Numpy-discussion] Functions that stores results without using the out keyword Message-ID: <4899F840.8010209@gmail.com> Hi, The following functions support a optional second argument that stores the result: isfinite, isnan, isinf, isposinf and isneginf. There may be other functions with the same behavior. Is this usage equivalent to 'out' keyword that appears in other functions like mean? If so, should these functions have the 'out' keyword? Also, I am correct that the 'out' keyword must always be an array and not a scalar? Numpy does returns the error 'TypeError: output must be an array' if a scalar is used. I would like to be explicit in the documentation by stating that a scalar in any form can not be used. Thanks Bruce import numpy as np a = np.array([[1,2, np.nan],[3,4, np.inf]]) b=np.array([[0.0],[0.0]]) c = np.array([[2,2,2],[2,2,2]]) np.mean(a, axis=1, out=b) #returns: array([[ NaN], # [ Inf]]) np.isinf(a, c) #returns: array([[0, 0, 0], # [0, 0, 1]]) np.isnan(a, c) #returns: array([[0, 0, 1], # [0, 0, 0]]) d=a.mean() np.mean(a, out=d) #Traceback (most recent call last): # File "", line 1, in # File "/usr/lib64/python2.5/site-packages/numpy/core/fromnumeric.py", line 1806, in mean # return mean(axis, dtype, out) #TypeError: output must be an array From mailanhilli at googlemail.com Wed Aug 6 15:24:59 2008 From: mailanhilli at googlemail.com (Matthias Hillenbrand) Date: Wed, 6 Aug 2008 14:24:59 -0500 Subject: [Numpy-discussion] Calculating roots with negative numbers Message-ID: <67b3a51f0808061224x79fbbd6bncdd17566651086e9@mail.gmail.com> Hello, > When you convolve two signals, of lengths N and M, you need to pad the > FFTs to length (N+M-1) before multiplication. > You can take a look at my linear position-invariant filtering code at: > http://mentat.za.net/hg/filter I understand your comments that I need zero padding when doing FFT filtering. I will need some time to understand St?fan's program as I am a NumPy/Python newbie and not used to object oriented programming. What I don't understand is that Octave/Matlab and Numpy generate different results with nearly the same code. Why are the results with Octave/Matlab OK although I have not applied additional zero padding. In my opinion I should get exactly the same results. Regards, Matthias --------------------------------------------------------------------------------------------------- Python code with comments (version without comments below) --------------------------------------------------------------------------------------------------- from pylab import * from numpy.lib import scimath #Complex roots import os ######################################### ########## Defintion of the input variables ######### ######################################### ##General variables lam = 532.e-9 #wavelength in m k = 2*pi/lam #wave number ##Computational array arr_size = 80.e-3 #size of the computation array in m N = 2**17 #points of the 1D array ##Aperture ap = 40.e-3 #aperture size of the 1D array in m ##Gaussian beam w0 =10.e-3 #waist radius of the gaussian beam in m ##Lenses (definition of the focal length) n = 1.56 #refractive index of the glass f = .1 #focal length in m Phi = 1/f #focal power of the lens r2 = 1.e18 #radius of the second lens surface r1 = 1 / ( Phi/(n-1) + 1/r2 ) #computation of the radius of the first lens surface ##z distances zmin = 0.99*f #initial z position zmax = 1.01*f #final z position zsteps = 1000 #number of computated positions ROI = 1000 #Region of interest in the diffraction image ############################################## ############### Program execution 1D ################ ############################################### x = linspace(-arr_size/2,arr_size/2,N) A = ones(N) A[where(ap/2<=abs(x))] = 0 #Definition of the aperture G = exp(- x**2/w0**2) #Generation of the Gaussian beam delta = -r1*(1 - scimath.sqrt(1 - x**2/r1**2)) + r2*(1 - scimath.sqrt(1 - x**2/r2**2)) m = (r1**2 <= x**2 ) delta[m] = 0 #correction in case of negative roots m = (r2**2 <= x**2 ) delta[m] = 0 #correction in case of negative roots lens = exp(1j * 2 * pi / lam * (n-1) * delta) #Calculation of the lens phase function u = A*G*lens #Complex amplitude before beam propagation ############################################ ########### Start angular spectrum method ########### deltaf = 1/arr_size #spacing in frequency domain fx = r_[-N/2*deltaf:(N/2)*deltaf:deltaf] #whole frequency domain FU = fftshift(fft(u)) #1D FFT U = zeros((zsteps,ROI)) #Generation of the image array z = linspace(zmin,zmax,zsteps) #Calculation of the different z positions for i in range(zsteps): H = exp(1j * 2 * pi / lam * z[i] * scimath.sqrt(1-(lam*fx)**2)) U[i] = (ifft(fftshift(FU*H)))[N/2 - ROI/2 : N/2 + ROI/2] if i%10 == 0: t = 'z position: %4i' % i print t ############ End angular spectrum method ############ ############################################# res = abs(U) imshow(res) show() ---------------------------------------------------------- Python code without comments ---------------------------------------------------------- from pylab import * from numpy.lib import scimath #Complex roots import os lam = 532.e-9 k = 2*pi/lam arr_size = 80.e-3 N = 2**17 ap = 40.e-3 w0 =10.e-3 n = 1.56 f = .1 Phi = 1/f r2 = 1.e18 r1 = 1 / ( Phi/(n-1) + 1/r2 ) zmin = 0.99*f zmax = 1.01*f zsteps = 1000 ROI = 1000 x = linspace(-arr_size/2,arr_size/2,N) A = ones(N) A[where(ap/2<=abs(x))] = 0 G = exp(- x**2/w0**2) delta = -r1*(1 - scimath.sqrt(1 - x**2/r1**2)) + r2*(1 - scimath.sqrt(1 - x**2/r2**2)) m = (r1**2 <= x**2 ) delta[m] = 0 m = (r2**2 <= x**2 ) delta[m] = 0 lens = exp(1j * 2 * pi / lam * (n-1) * delta) u = A*G*lens deltaf = 1/arr_size fx = r_[-N/2*deltaf:(N/2)*deltaf:deltaf] FU = fftshift(fft(u)) U = zeros((zsteps,ROI)) z = linspace(zmin,zmax,zsteps) for i in range(zsteps): H = exp(1j * 2 * pi / lam * z[i] * scimath.sqrt(1-(lam*fx)**2)) U[i] = (ifft(fftshift(FU*H)))[N/2 - ROI/2 : N/2 + ROI/2] if i%10 == 0: t = 'z position: %4i' % i print t res = abs(U) imshow(res) show() ---------------------------------------------------------- Matlab, Octave code ---------------------------------------------------------- lam = 532.e-9 k = 2*pi/lam arr_size = 80.e-3 N = 2^17 ap = 40.e-3 w0 =10.e-3 n = 1.56 f = .1 Phi = 1/f r2 = 1.e18 r1 = 1 / ( Phi/(n-1) + 1/r2 ) zmin = 0.99*f zmax = 1.01*f zsteps = 1000 ROI = 1000 x=linspace(-arr_size/2,arr_size/2,N); A = ones(1,N); A(find(ap/2<=abs(x)))=0; G = exp(- x.^2/w0^2); delta = -r1*(1 - sqrt(1 - x.^2/r1^2)) + r2*(1 - sqrt(1 - x.^2/r2^2)); delta(find(r1.^2 <= x.^2 ))=0; delta(find(r2.^2 <= x.^2 ))=0; lens = exp(1j * 2 * pi / lam * (n-1) * delta); u = A.*G.*lens; deltaf = 1/arr_size; fx = [-N/2*deltaf:deltaf:(N/2-1)*deltaf]; FU = fftshift(fft(u)); U = zeros(zsteps,ROI); z = linspace(zmin,zmax,zsteps); for i=1:zsteps H = exp(1j * 2 * pi / lam * z(i) * sqrt(1-lam.^2*fx.^2)); U(i,:) = ifft(fftshift(FU.*H))(N/2 - ROI/2 : N/2 + ROI/2-1); end imagesc(abs(U)) From rpyle at post.harvard.edu Wed Aug 6 15:57:58 2008 From: rpyle at post.harvard.edu (Robert Pyle) Date: Wed, 06 Aug 2008 15:57:58 -0400 Subject: [Numpy-discussion] What happened to numpy.testing.Tester ? In-Reply-To: <200807251726.16143.felix@physik3.uni-rostock.de> References: <200807251726.16143.felix@physik3.uni-rostock.de> Message-ID: <637883B2-8ABA-483C-8EA7-9119A378F3B1@post.harvard.edu> Hi all, My machine: Mac dual G5, OSX 10.5.4, Python 2.5.2 (r252:60911, Feb 22 2008, 07:57:53), numpy 1.1.1 After considerable agony, I succeeded in building and installing scipy from SVN, only to be told upon importing it: Traceback (most recent call last): File "", line 1, in File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/scipy/__init__.py", line 86, in from numpy.testing import Tester ImportError: cannot import name Tester A message here from Alan McIntyre on June 20, said, in part: "NoseTester will still be made available as numpy.testing.Tester." It appears that each of the seemingly innumerable __init__.py files throughout the scipy sources calls for numpy.testing.Tester. Should I ask over on the scipy list as well? Is there some trivial change that will fake out either numpy or scipy into thinking all is well? Thanks. Bob Pyle From alan.mcintyre at gmail.com Wed Aug 6 16:17:32 2008 From: alan.mcintyre at gmail.com (Alan McIntyre) Date: Wed, 6 Aug 2008 16:17:32 -0400 Subject: [Numpy-discussion] What happened to numpy.testing.Tester ? In-Reply-To: <637883B2-8ABA-483C-8EA7-9119A378F3B1@post.harvard.edu> References: <200807251726.16143.felix@physik3.uni-rostock.de> <637883B2-8ABA-483C-8EA7-9119A378F3B1@post.harvard.edu> Message-ID: <1d36917a0808061317q28d834eah29532e9cb786dc7b@mail.gmail.com> On Wed, Aug 6, 2008 at 3:57 PM, Robert Pyle wrote: > My machine: Mac dual G5, OSX 10.5.4, Python 2.5.2 (r252:60911, Feb 22 > 2008, 07:57:53), numpy 1.1.1 > > After considerable agony, I succeeded in building and installing scipy > from SVN, only to be told upon importing it: > > Traceback (most recent call last): > File "", line 1, in > File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ > python2.5/site-packages/scipy/__init__.py", line 86, in > from numpy.testing import Tester > ImportError: cannot import name Tester > > A message here from Alan McIntyre on June 20, said, in part: > "NoseTester will still be made available as numpy.testing.Tester." > > It appears that each of the seemingly innumerable __init__.py files > throughout the scipy sources calls for numpy.testing.Tester. > > Should I ask over on the scipy list as well? Is there some trivial > change that will fake out either numpy or scipy into thinking all is > well? You will actually need to use NumPy from svn as well, since 1.1.1 didn't have NoseTester (SciPy 0.7 will require NumPy 1.2). From mailanhilli at googlemail.com Wed Aug 6 16:55:58 2008 From: mailanhilli at googlemail.com (Matthias Hillenbrand) Date: Wed, 6 Aug 2008 15:55:58 -0500 Subject: [Numpy-discussion] Calculating roots with negative numbers In-Reply-To: <67b3a51f0808061224x79fbbd6bncdd17566651086e9@mail.gmail.com> References: <67b3a51f0808061224x79fbbd6bncdd17566651086e9@mail.gmail.com> Message-ID: <67b3a51f0808061355y2ad47e15w119a88a9bbfbd3f7@mail.gmail.com> Sorry, I have chosen the wrong title for this email. It should be a follow up to the discussion 'Horizontal lines in diffraction image (NumPy FFT)' From rpyle at post.harvard.edu Wed Aug 6 16:57:49 2008 From: rpyle at post.harvard.edu (Robert Pyle) Date: Wed, 06 Aug 2008 16:57:49 -0400 Subject: [Numpy-discussion] What happened to numpy.testing.Tester ? In-Reply-To: <1d36917a0808061317q28d834eah29532e9cb786dc7b@mail.gmail.com> References: <200807251726.16143.felix@physik3.uni-rostock.de> <637883B2-8ABA-483C-8EA7-9119A378F3B1@post.harvard.edu> <1d36917a0808061317q28d834eah29532e9cb786dc7b@mail.gmail.com> Message-ID: <8BD8E8CE-23D6-41CE-8823-B5B95704FFCA@post.harvard.edu> On Aug 6, 2008, at 4:17 PM, Alan McIntyre wrote: > You will actually need to use NumPy from svn as well, since 1.1.1 > didn't have NoseTester (SciPy 0.7 will require NumPy 1.2). Thanks. I can now import scipy, but I'm puzzled by the following: Python 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.__version__ '1.2.0.dev5616' >>> import np.testing.Tester Traceback (most recent call last): File "", line 1, in ImportError: No module named np.testing.Tester >>> import scipy >>> scipy.__version__ '0.7.0.dev4603' >>> How come 'import np.testing.Tester' fails, but scipy imports okay even though it used to fail with a complaint about the absence of np.testing.Tester? Bob From markbak at gmail.com Wed Aug 6 17:00:52 2008 From: markbak at gmail.com (mark) Date: Wed, 6 Aug 2008 14:00:52 -0700 (PDT) Subject: [Numpy-discussion] confusion on importing numpy In-Reply-To: <4899EB95.4070809@hawaii.edu> References: <12d2498a-ee23-4881-8fa6-0b5145dc7476@59g2000hsb.googlegroups.com> <4899EB95.4070809@hawaii.edu> Message-ID: <4299b9b2-f7f0-46fc-b792-9e743d877f05@56g2000hsm.googlegroups.com> Thanks for the encouragement to try to dive deeper into namespaces, but also thanks for the amin, amax suggestions. Mark On Aug 6, 8:21 pm, Eric Firing wrote: > mark wrote: > > Hello list. I am confused about importing numpy. > > > When I do > > > from numpy import * > > > and I try the min function, I get the default Python min function. > > > On the other hand, when I do > > > import numpy as np > > > and use the np.min function, I get the numpy min function (which is > > obviously what I want). > > > I know, the latter is preferred, but the former is so dang easy to > > type. > > I always thought both imports work the same, but I am obviously wrong. > > What's the difference? > > While I agree with the other posters that "import *" is not preferred, > if you want to use it, the solution is to use amin and amax, which are > provided precisely to avoid the conflict. Just as arange is a numpy > analog of range, amin and amax are numpy analogs of min and max. > > Eric > > > > > Thanks, Mark > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discuss... at scipy.org > >http://projects.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion From robert.kern at gmail.com Wed Aug 6 17:09:24 2008 From: robert.kern at gmail.com (robert.kern at gmail.com) Date: Wed, 6 Aug 2008 16:09:24 -0500 Subject: [Numpy-discussion] What happened to numpy.testing.Tester ? In-Reply-To: <8BD8E8CE-23D6-41CE-8823-B5B95704FFCA@post.harvard.edu> References: <200807251726.16143.felix@physik3.uni-rostock.de> <637883B2-8ABA-483C-8EA7-9119A378F3B1@post.harvard.edu> <1d36917a0808061317q28d834eah29532e9cb786dc7b@mail.gmail.com> <8BD8E8CE-23D6-41CE-8823-B5B95704FFCA@post.harvard.edu> Message-ID: <3d375d730808061409g56620e81i949ce657616ea46a@mail.gmail.com> Tester is a class, not a module. Try "from numpy.testing import Tester". On 2008-08-06, Robert Pyle wrote: > > On Aug 6, 2008, at 4:17 PM, Alan McIntyre wrote: > >> You will actually need to use NumPy from svn as well, since 1.1.1 >> didn't have NoseTester (SciPy 0.7 will require NumPy 1.2). > > Thanks. I can now import scipy, but I'm puzzled by the following: > > Python 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) > [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy as np > >>> np.__version__ > '1.2.0.dev5616' > >>> import np.testing.Tester > Traceback (most recent call last): > File "", line 1, in > ImportError: No module named np.testing.Tester > >>> import scipy > >>> scipy.__version__ > '0.7.0.dev4603' > >>> > > How come 'import np.testing.Tester' fails, but scipy imports okay even > though it used to fail with a complaint about the absence of > np.testing.Tester? > > Bob > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From peridot.faceted at gmail.com Wed Aug 6 17:10:34 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 6 Aug 2008 17:10:34 -0400 Subject: [Numpy-discussion] confusion on importing numpy In-Reply-To: <4899EB95.4070809@hawaii.edu> References: <12d2498a-ee23-4881-8fa6-0b5145dc7476@59g2000hsb.googlegroups.com> <4899EB95.4070809@hawaii.edu> Message-ID: 2008/8/6 Eric Firing : > While I agree with the other posters that "import *" is not preferred, > if you want to use it, the solution is to use amin and amax, which are > provided precisely to avoid the conflict. Just as arange is a numpy > analog of range, amin and amax are numpy analogs of min and max. There are actually several analogs of min and max, depending on what you want. np.minimum(a,b) takes the elementwise minimum, that is, np.minimum([1,2,3],[3,2,1]) is [1,2,1]. np.amin(a) finds the minimum value in the array a. (It's equivalent to np.minimum.reduce(a).) One reason automatically overriding min is a Bad Idea is that you will certainly shoot yourself in the foot: In [4]: numpy.min(2,1) Out[4]: 2 In [5]: min(2,1) Out[5]: 1 Note that this does not raise any kind of error, and it sometimes returns the right value. It's especially bad if you do, say, numpy.max(a,0) and expect the result to be always positive: now you get a bug in your program that only turns up with exceptional values. numpy.min is not the same as python's min, and using it as a substitute will get you in trouble. Anne From mailanhilli at googlemail.com Wed Aug 6 17:30:59 2008 From: mailanhilli at googlemail.com (Matthias Hillenbrand) Date: Wed, 6 Aug 2008 16:30:59 -0500 Subject: [Numpy-discussion] Horizontal lines in diffraction image (NumPy FFT) In-Reply-To: <67b3a51f0808051926r1aefac3dibfbffda4100d9fb5@mail.gmail.com> References: <67b3a51f0808051926r1aefac3dibfbffda4100d9fb5@mail.gmail.com> Message-ID: <67b3a51f0808061430v1f0f6e4btea1adf5d83726396@mail.gmail.com> Hello, > When you convolve two signals, of lengths N and M, you need to pad the > FFTs to length (N+M-1) before multiplication. > You can take a look at my linear position-invariant filtering code at: > http://mentat.za.net/hg/filter I understand your comments that I need zero padding when doing FFT filtering. I will need some time to understand St?fan's program as I am a NumPy/Python newbie and not used to object oriented programming. What I don't understand is that Octave/Matlab and Numpy generate different results with nearly the same code. Why are the results with Octave/Matlab OK although I have not applied additional zero padding. In my opinion I should get exactly the same results. Regards, Matthias --------------------------------------------------------------------------------------------------- Python code with comments (version without comments below) --------------------------------------------------------------------------------------------------- from pylab import * from numpy.lib import scimath #Complex roots import os ######################################### ########## Defintion of the input variables ######### ######################################### ##General variables lam = 532.e-9 #wavelength in m k = 2*pi/lam #wave number ##Computational array arr_size = 80.e-3 #size of the computation array in m N = 2**17 #points of the 1D array ##Aperture ap = 40.e-3 #aperture size of the 1D array in m ##Gaussian beam w0 =10.e-3 #waist radius of the gaussian beam in m ##Lenses (definition of the focal length) n = 1.56 #refractive index of the glass f = .1 #focal length in m Phi = 1/f #focal power of the lens r2 = 1.e18 #radius of the second lens surface r1 = 1 / ( Phi/(n-1) + 1/r2 ) #computation of the radius of the first lens surface ##z distances zmin = 0.99*f #initial z position zmax = 1.01*f #final z position zsteps = 1000 #number of computated positions ROI = 1000 #Region of interest in the diffraction image ############################################## ############### Program execution 1D ################ ############################################### x = linspace(-arr_size/2,arr_size/2,N) A = ones(N) A[where(ap/2<=abs(x))] = 0 #Definition of the aperture G = exp(- x**2/w0**2) #Generation of the Gaussian beam delta = -r1*(1 - scimath.sqrt(1 - x**2/r1**2)) + r2*(1 - scimath.sqrt(1 - x**2/r2**2)) m = (r1**2 <= x**2 ) delta[m] = 0 #correction in case of negative roots m = (r2**2 <= x**2 ) delta[m] = 0 #correction in case of negative roots lens = exp(1j * 2 * pi / lam * (n-1) * delta) #Calculation of the lens phase function u = A*G*lens #Complex amplitude before beam propagation ############################################ ########### Start angular spectrum method ########### deltaf = 1/arr_size #spacing in frequency domain fx = r_[-N/2*deltaf:(N/2)*deltaf:deltaf] #whole frequency domain FU = fftshift(fft(u)) #1D FFT U = zeros((zsteps,ROI)) #Generation of the image array z = linspace(zmin,zmax,zsteps) #Calculation of the different z positions for i in range(zsteps): H = exp(1j * 2 * pi / lam * z[i] * scimath.sqrt(1-(lam*fx)**2)) U[i] = (ifft(fftshift(FU*H)))[N/2 - ROI/2 : N/2 + ROI/2] if i%10 == 0: t = 'z position: %4i' % i print t ############ End angular spectrum method ############ ############################################# res = abs(U) imshow(res) show() ---------------------------------------------------------- Python code without comments ---------------------------------------------------------- from pylab import * from numpy.lib import scimath #Complex roots import os lam = 532.e-9 k = 2*pi/lam arr_size = 80.e-3 N = 2**17 ap = 40.e-3 w0 =10.e-3 n = 1.56 f = .1 Phi = 1/f r2 = 1.e18 r1 = 1 / ( Phi/(n-1) + 1/r2 ) zmin = 0.99*f zmax = 1.01*f zsteps = 1000 ROI = 1000 x = linspace(-arr_size/2,arr_size/2,N) A = ones(N) A[where(ap/2<=abs(x))] = 0 G = exp(- x**2/w0**2) delta = -r1*(1 - scimath.sqrt(1 - x**2/r1**2)) + r2*(1 - scimath.sqrt(1 - x**2/r2**2)) m = (r1**2 <= x**2 ) delta[m] = 0 m = (r2**2 <= x**2 ) delta[m] = 0 lens = exp(1j * 2 * pi / lam * (n-1) * delta) u = A*G*lens deltaf = 1/arr_size fx = r_[-N/2*deltaf:(N/2)*deltaf:deltaf] FU = fftshift(fft(u)) U = zeros((zsteps,ROI)) z = linspace(zmin,zmax,zsteps) for i in range(zsteps): H = exp(1j * 2 * pi / lam * z[i] * scimath.sqrt(1-(lam*fx)**2)) U[i] = (ifft(fftshift(FU*H)))[N/2 - ROI/2 : N/2 + ROI/2] if i%10 == 0: t = 'z position: %4i' % i print t res = abs(U) imshow(res) show() ---------------------------------------------------------- Matlab, Octave code ---------------------------------------------------------- lam = 532.e-9 k = 2*pi/lam arr_size = 80.e-3 N = 2^17 ap = 40.e-3 w0 =10.e-3 n = 1.56 f = .1 Phi = 1/f r2 = 1.e18 r1 = 1 / ( Phi/(n-1) + 1/r2 ) zmin = 0.99*f zmax = 1.01*f zsteps = 1000 ROI = 1000 x=linspace(-arr_size/2,arr_size/2,N); A = ones(1,N); A(find(ap/2<=abs(x)))=0; G = exp(- x.^2/w0^2); delta = -r1*(1 - sqrt(1 - x.^2/r1^2)) + r2*(1 - sqrt(1 - x.^2/r2^2)); delta(find(r1.^2 <= x.^2 ))=0; delta(find(r2.^2 <= x.^2 ))=0; lens = exp(1j * 2 * pi / lam * (n-1) * delta); u = A.*G.*lens; deltaf = 1/arr_size; fx = [-N/2*deltaf:deltaf:(N/2-1)*deltaf]; FU = fftshift(fft(u)); U = zeros(zsteps,ROI); z = linspace(zmin,zmax,zsteps); for i=1:zsteps H = exp(1j * 2 * pi / lam * z(i) * sqrt(1-lam.^2*fx.^2)); U(i,:) = ifft(fftshift(FU.*H))(N/2 - ROI/2 : N/2 + ROI/2-1); end imagesc(abs(U)) From nadavh at visionsense.com Thu Aug 7 01:34:06 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Thu, 7 Aug 2008 08:34:06 +0300 Subject: [Numpy-discussion] Horizontal lines in diffraction image (NumPyFFT) References: <67b3a51f0808051926r1aefac3dibfbffda4100d9fb5@mail.gmail.com> <67b3a51f0808061430v1f0f6e4btea1adf5d83726396@mail.gmail.com> Message-ID: <710F2847B0018641891D9A216027636029C208@ex3.envision.co.il> The 0 padding is easy in numpp/pylab as in octave/matlab, by just adding one argument. In pylab it is the "a" keyword: y = fft(x, n=2*len(x)) if n is not given then n=len(x) --- normal fft if n > len(x) then it pads x with 0. Nadav. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Matthias Hillenbrand ????: ? 07-??????-08 00:30 ??: numpy-discussion at scipy.org ????: Re: [Numpy-discussion] Horizontal lines in diffraction image (NumPyFFT) Hello, > When you convolve two signals, of lengths N and M, you need to pad the > FFTs to length (N+M-1) before multiplication. > You can take a look at my linear position-invariant filtering code at: > http://mentat.za.net/hg/filter I understand your comments that I need zero padding when doing FFT filtering. I will need some time to understand St?fan's program as I am a NumPy/Python newbie and not used to object oriented programming. What I don't understand is that Octave/Matlab and Numpy generate different results with nearly the same code. Why are the results with Octave/Matlab OK although I have not applied additional zero padding. In my opinion I should get exactly the same results. Regards, Matthias --------------------------------------------------------------------------------------------------- Python code with comments (version without comments below) --------------------------------------------------------------------------------------------------- from pylab import * from numpy.lib import scimath #Complex roots import os ######################################### ########## Defintion of the input variables ######### ######################################### ##General variables lam = 532.e-9 #wavelength in m k = 2*pi/lam #wave number ##Computational array arr_size = 80.e-3 #size of the computation array in m N = 2**17 #points of the 1D array ##Aperture ap = 40.e-3 #aperture size of the 1D array in m ##Gaussian beam w0 =10.e-3 #waist radius of the gaussian beam in m ##Lenses (definition of the focal length) n = 1.56 #refractive index of the glass f = .1 #focal length in m Phi = 1/f #focal power of the lens r2 = 1.e18 #radius of the second lens surface r1 = 1 / ( Phi/(n-1) + 1/r2 ) #computation of the radius of the first lens surface ##z distances zmin = 0.99*f #initial z position zmax = 1.01*f #final z position zsteps = 1000 #number of computated positions ROI = 1000 #Region of interest in the diffraction image ############################################## ############### Program execution 1D ################ ############################################### x = linspace(-arr_size/2,arr_size/2,N) A = ones(N) A[where(ap/2<=abs(x))] = 0 #Definition of the aperture G = exp(- x**2/w0**2) #Generation of the Gaussian beam delta = -r1*(1 - scimath.sqrt(1 - x**2/r1**2)) + r2*(1 - scimath.sqrt(1 - x**2/r2**2)) m = (r1**2 <= x**2 ) delta[m] = 0 #correction in case of negative roots m = (r2**2 <= x**2 ) delta[m] = 0 #correction in case of negative roots lens = exp(1j * 2 * pi / lam * (n-1) * delta) #Calculation of the lens phase function u = A*G*lens #Complex amplitude before beam propagation ############################################ ########### Start angular spectrum method ########### deltaf = 1/arr_size #spacing in frequency domain fx = r_[-N/2*deltaf:(N/2)*deltaf:deltaf] #whole frequency domain FU = fftshift(fft(u)) #1D FFT U = zeros((zsteps,ROI)) #Generation of the image array z = linspace(zmin,zmax,zsteps) #Calculation of the different z positions for i in range(zsteps): H = exp(1j * 2 * pi / lam * z[i] * scimath.sqrt(1-(lam*fx)**2)) U[i] = (ifft(fftshift(FU*H)))[N/2 - ROI/2 : N/2 + ROI/2] if i%10 == 0: t = 'z position: %4i' % i print t ############ End angular spectrum method ############ ############################################# res = abs(U) imshow(res) show() ---------------------------------------------------------- Python code without comments ---------------------------------------------------------- from pylab import * from numpy.lib import scimath #Complex roots import os lam = 532.e-9 k = 2*pi/lam arr_size = 80.e-3 N = 2**17 ap = 40.e-3 w0 =10.e-3 n = 1.56 f = .1 Phi = 1/f r2 = 1.e18 r1 = 1 / ( Phi/(n-1) + 1/r2 ) zmin = 0.99*f zmax = 1.01*f zsteps = 1000 ROI = 1000 x = linspace(-arr_size/2,arr_size/2,N) A = ones(N) A[where(ap/2<=abs(x))] = 0 G = exp(- x**2/w0**2) delta = -r1*(1 - scimath.sqrt(1 - x**2/r1**2)) + r2*(1 - scimath.sqrt(1 - x**2/r2**2)) m = (r1**2 <= x**2 ) delta[m] = 0 m = (r2**2 <= x**2 ) delta[m] = 0 lens = exp(1j * 2 * pi / lam * (n-1) * delta) u = A*G*lens deltaf = 1/arr_size fx = r_[-N/2*deltaf:(N/2)*deltaf:deltaf] FU = fftshift(fft(u)) U = zeros((zsteps,ROI)) z = linspace(zmin,zmax,zsteps) for i in range(zsteps): H = exp(1j * 2 * pi / lam * z[i] * scimath.sqrt(1-(lam*fx)**2)) U[i] = (ifft(fftshift(FU*H)))[N/2 - ROI/2 : N/2 + ROI/2] if i%10 == 0: t = 'z position: %4i' % i print t res = abs(U) imshow(res) show() ---------------------------------------------------------- Matlab, Octave code ---------------------------------------------------------- lam = 532.e-9 k = 2*pi/lam arr_size = 80.e-3 N = 2^17 ap = 40.e-3 w0 =10.e-3 n = 1.56 f = .1 Phi = 1/f r2 = 1.e18 r1 = 1 / ( Phi/(n-1) + 1/r2 ) zmin = 0.99*f zmax = 1.01*f zsteps = 1000 ROI = 1000 x=linspace(-arr_size/2,arr_size/2,N); A = ones(1,N); A(find(ap/2<=abs(x)))=0; G = exp(- x.^2/w0^2); delta = -r1*(1 - sqrt(1 - x.^2/r1^2)) + r2*(1 - sqrt(1 - x.^2/r2^2)); delta(find(r1.^2 <= x.^2 ))=0; delta(find(r2.^2 <= x.^2 ))=0; lens = exp(1j * 2 * pi / lam * (n-1) * delta); u = A.*G.*lens; deltaf = 1/arr_size; fx = [-N/2*deltaf:deltaf:(N/2-1)*deltaf]; FU = fftshift(fft(u)); U = zeros(zsteps,ROI); z = linspace(zmin,zmax,zsteps); for i=1:zsteps H = exp(1j * 2 * pi / lam * z(i) * sqrt(1-lam.^2*fx.^2)); U(i,:) = ifft(fftshift(FU.*H))(N/2 - ROI/2 : N/2 + ROI/2-1); end imagesc(abs(U)) _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 5644 bytes Desc: not available URL: From nwagner at iam.uni-stuttgart.de Thu Aug 7 02:30:04 2008 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 07 Aug 2008 08:30:04 +0200 Subject: [Numpy-discussion] FAIL: Test corrcoef 1 1D variable w/missing values Message-ID: Hi all, Can someone reproduce the failure with >>> numpy.__version__ '1.2.0.dev5618' ====================================================================== FAIL: Test corrcoef 1 1D variable w/missing values ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/ma/tests/test_extras.py", line 463, in test_1d_w_missing corrcoef(x, x[::-1], rowvar=False)) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/ma/testutils.py", line 120, in assert_equal return assert_array_equal(actual, desired, err_msg) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/ma/testutils.py", line 186, in assert_array_equal header='Arrays are not equal') File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/ma/testutils.py", line 180, in assert_array_compare verbose=verbose, header=header) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/testing/utils.py", line 289, in assert_array_compare assert cond, msg AssertionError: Arrays are not equal (mismatch 50.0%) x: array([[ 1. , -0.38605094], [-0.38605094, 1. ]]) y: array([[ 1. , -0.38605094], [-0.38605094, 1. ]]) Nils From jordan at math.ucsb.edu Thu Aug 7 04:33:57 2008 From: jordan at math.ucsb.edu (jordan at math.ucsb.edu) Date: Thu, 7 Aug 2008 01:33:57 -0700 (PDT) Subject: [Numpy-discussion] FFTW and config vs system_info Message-ID: <50735.71.102.224.60.1218098037.squirrel@mail.math.ucsb.edu> Hi all, I'm trying to get scipy to recognize my FFTW install. I know there is another thread on this, but I didn't understand the discussion fully. Which file do I modify to let scipy know where my FFTW install is? As a possibly related note, running scipy.config() and scipy.distutil.system_info.py give conflicting answers as to whether I have BLAS and LAPACK etc. It'd be nice to know for sure what scipy is using. Any light on this matter would be greatly appreciated. Cheers, Jordan From nadavh at visionsense.com Thu Aug 7 07:58:49 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Thu, 07 Aug 2008 14:58:49 +0300 Subject: [Numpy-discussion] FFTW and config vs system_info In-Reply-To: <50735.71.102.224.60.1218098037.squirrel@mail.math.ucsb.edu> References: <50735.71.102.224.60.1218098037.squirrel@mail.math.ucsb.edu> Message-ID: <1218110329.14570.1.camel@nadav.envision.co.il> You shoud edit site.cfg file: look for site.cfg.example and follow the instructions. Nadav On Thu, 2008-08-07 at 01:33 -0700, jordan at math.ucsb.edu wrote: > Hi all, > > I'm trying to get scipy to recognize my FFTW install. I know there is > another thread on this, but I didn't understand the discussion fully. > Which file do I modify to let scipy know where my FFTW install is? > > As a possibly related note, running scipy.config() and > scipy.distutil.system_info.py give conflicting answers as to whether I > have BLAS and LAPACK etc. It'd be nice to know for sure what scipy is > using. > > Any light on this matter would be greatly appreciated. > > Cheers, > Jordan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Thu Aug 7 10:47:49 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 7 Aug 2008 10:47:49 -0400 Subject: [Numpy-discussion] FAIL: Test corrcoef 1 1D variable w/missing values In-Reply-To: References: Message-ID: <200808071047.49908.pgmdevlist@gmail.com> Nils, > ====================================================================== > FAIL: Test corrcoef 1 1D variable w/missing values > ---------------------------------------------------------------------- ... > "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/testing/utils.p >y", line 289, in assert_array_compare > assert cond, msg > AssertionError: > Arrays are not equal > > (mismatch 50.0%) > x: array([[ 1. , -0.38605094], > [-0.38605094, 1. ]]) > y: array([[ 1. , -0.38605094], > [-0.38605094, 1. ]]) I can't, but I'm not surprised: the two arrays look the same, I should probably switch to assert_almost_equal instead of assert_equal From dpeterson at enthought.com Thu Aug 7 13:17:34 2008 From: dpeterson at enthought.com (Dave Peterson) Date: Thu, 07 Aug 2008 12:17:34 -0500 Subject: [Numpy-discussion] [ANNOUNCE] EPD 2.5.2001 for OS X released! Message-ID: <489B2E2E.2040809@enthought.com> I'm pleased to announce that Enthought has released the Enthought Python Distribution (EPD) 2.5.2001 for OS X! EPD is a Distribution of the Python Programming Language (currently version 2.5.2) that includes over 60 additional libraries, including ETS 2.7.1. Please visit the EPD website (http://www.enthought.com/epd) to get the OS X release, or just to find out more information about EPD 2.5.2001 So what?s the big deal? In addition to making everyones? life easier with installation, EPD also represents a common suite of functionality deployed across platforms such as Windows XP, RedHat Linux, and now OS X 10.4 and above. The cross-platform promise of Python is better realized because it?s trivial for everyone to get substantially the same set of libraries installed on their system with a single-click install. What?s the catch? You knew it was coming, huh? If you?d like to use EPD in a Commercial or Governmental entity, we do ask you to pay for an annual subscription to download and update EPD. For academics and non-profit, private-sector organizations, EPD is and will remain free. Let?s be clear, though. EPD is the bundle of software. People pay for the subscription to download the bundle. The included libraries are, of course, freely available separately under the terms of license for each individual package (this should sound familiar). The terms for the bundle subscription are available at http://www.enthought.com/products/epdlicense.php. BTW, anyone can try it out for free for 30 days. If you have questions, check out the FAQ at http://www.enthought.com/products/epdfaq.php or drop us a line at epd-users at enthought.com. And just one more note: Enthought is deeply grateful to all those who have contributed to these libraries over the years. We?ve built a business around the things they allow us to do and we appreciate such a nice set of tools and the privilege of being part of the community that created them. From zachary.pincus at yale.edu Thu Aug 7 15:22:26 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 7 Aug 2008 15:22:26 -0400 Subject: [Numpy-discussion] numpy / ipython heisenbug crasher help? Message-ID: Hello all, I have a bizarre bug that causes numpy to segfault, but: - only when run under ipython - only when numpy is imported *after* an other library (that does not import numpy) Here's what the code looks like. Crashes (only in ipython): import celltool.utility.pil_lite.Image as Image, numpy i = Image.open('untreated_2_logical.png') a = numpy.array(i) Does not crash: import numpy, celltool.utility.pil_lite.Image as Image i = Image.open('untreated_2_logical.png') a = numpy.array(i) The 'i' object exposes an __array_interface__, and is produced from my personal fork of the PIL. The crash is thusly reported: Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Codes: KERN_INVALID_ADDRESS at 0x000000000251f000 0 libSystem.B.dylib 0xffff0a13 __memcpy + 627 1 multiarray.so 0x022f7af8 _array_copy_into + 2840 (arrayobject.c:1113) 2 multiarray.so 0x022f88af PyArray_FromArray + 495 (arrayobject.c:8334) 3 multiarray.so 0x022ee314 PyArray_FromAny + 308 (arrayobject.c: 8814) 4 multiarray.so 0x022fb01c PyArray_CheckFromAny + 92 (arrayobject.c:8995) 5 multiarray.so 0x022fb2e6 _array_fromobject + 310 (multiarraymodule.c:5787) [And then the python stack frames] Can anyone suggest what a good next step in reducing this patently insane bug down to a more minimal test case is? I had this bug with numpy 1.1 as well as the current SVN head. Zach From zachary.pincus at yale.edu Thu Aug 7 15:41:22 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 7 Aug 2008 15:41:22 -0400 Subject: [Numpy-discussion] numpy / ipython heisenbug crasher help? In-Reply-To: References: Message-ID: Hmm, I may have identified this by other means. See my next email... Zach On Aug 7, 2008, at 3:22 PM, Zachary Pincus wrote: > Hello all, > > I have a bizarre bug that causes numpy to segfault, but: > - only when run under ipython > - only when numpy is imported *after* an other library (that does > not import numpy) > > Here's what the code looks like. > Crashes (only in ipython): > import celltool.utility.pil_lite.Image as Image, numpy > i = Image.open('untreated_2_logical.png') > a = numpy.array(i) > > Does not crash: > import numpy, celltool.utility.pil_lite.Image as Image > i = Image.open('untreated_2_logical.png') > a = numpy.array(i) > > The 'i' object exposes an __array_interface__, and is produced from my > personal fork of the PIL. > > The crash is thusly reported: > Exception Type: EXC_BAD_ACCESS (SIGSEGV) > Exception Codes: KERN_INVALID_ADDRESS at 0x000000000251f000 > > 0 libSystem.B.dylib 0xffff0a13 __memcpy + 627 > 1 multiarray.so 0x022f7af8 _array_copy_into + 2840 > (arrayobject.c:1113) > 2 multiarray.so 0x022f88af PyArray_FromArray + 495 > (arrayobject.c:8334) > 3 multiarray.so 0x022ee314 PyArray_FromAny + 308 (arrayobject.c: > 8814) > 4 multiarray.so 0x022fb01c PyArray_CheckFromAny + 92 > (arrayobject.c:8995) > 5 multiarray.so 0x022fb2e6 _array_fromobject + 310 > (multiarraymodule.c:5787) > [And then the python stack frames] > > Can anyone suggest what a good next step in reducing this patently > insane bug down to a more minimal test case is? I had this bug with > numpy 1.1 as well as the current SVN head. > > Zach > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From jordan at math.ucsb.edu Thu Aug 7 15:41:14 2008 From: jordan at math.ucsb.edu (jordan at math.ucsb.edu) Date: Thu, 7 Aug 2008 12:41:14 -0700 (PDT) Subject: [Numpy-discussion] FFTW and config vs system_info In-Reply-To: <1218110329.14570.1.camel@nadav.envision.co.il> References: <50735.71.102.224.60.1218098037.squirrel@mail.math.ucsb.edu> <1218110329.14570.1.camel@nadav.envision.co.il> Message-ID: <50360.71.102.224.60.1218138074.squirrel@mail.math.ucsb.edu> That seems to have changed something for sure. I created a modified site.cfg file. When I run system_info.py I get this message fftw_info: libraries fftw3 not found in c:\FFTW libraries fftw3 not found in C:\Python25\lib libraries fftw3 not found in C:\ libraries fftw3 not found in C:\Python25\libs Library fftw3 was not found. Ignoring FOUND: libraries = ['libfftw3-3'] library_dirs = ['c:\\FFTW'] define_macros = [('SCIPY_FFTW3_H', None)] include_dirs = ['c:\\FFTW'] Seems conflicted. Does this mean it is using FFTW? Is there any other way to definitively check what scipy is using? I've benchmarked my FFTs before and after this change and there is a difference (a slow down actually). Cheers, Jordan > You shoud edit site.cfg file: look for site.cfg.example and follow the > instructions. > > Nadav > > > On Thu, 2008-08-07 at 01:33 -0700, jordan at math.ucsb.edu wrote: > >> Hi all, >> >> I'm trying to get scipy to recognize my FFTW install. I know there is >> another thread on this, but I didn't understand the discussion fully. >> Which file do I modify to let scipy know where my FFTW install is? >> >> As a possibly related note, running scipy.config() and >> scipy.distutil.system_info.py give conflicting answers as to whether I >> have BLAS and LAPACK etc. It'd be nice to know for sure what scipy is >> using. >> >> Any light on this matter would be greatly appreciated. >> >> Cheers, >> Jordan >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From sameerslists at gmail.com Thu Aug 7 15:44:37 2008 From: sameerslists at gmail.com (Sameer DCosta) Date: Thu, 7 Aug 2008 14:44:37 -0500 Subject: [Numpy-discussion] Error using rec.fromrecords with specific dtype Message-ID: <8fb8cc060808071244k32780307w8961c3ec8cb2f590@mail.gmail.com> Hi, I am unable to specify a dtype when using numpy.rec.fromrecords. This problem only occurs when one of the dtypes is an object. (Please see below) Thanks in advance for your help. Sameer In [7]: import datetime, numpy as np In [8]: print np.__version__ 1.2.0.dev5253 In [9]: dtype=np.dtype([('spam', '|O4'),('ham', ' /home/titan/sameer/local/mins/lib/python2.4/site-packages/numpy/core/records.pyc in fromrecords(recList, dtype, shape, formats, names, titles, aligned, byteorder) 469 res = retval.view(recarray) 470 --> 471 res.dtype = sb.dtype((record, res.dtype)) 472 return res 473 /home/titan/sameer/local/mins/lib/python2.4/site-packages/numpy/core/records.pyc in __setattr__(self, attr, val) 287 if attr not in fielddict: 288 exctype, value = sys.exc_info()[:2] --> 289 raise exctype, value 290 else: 291 fielddict = ndarray.__getattribute__(self,'dtype').fields or {} TypeError: Cannot change data-type for object array. In [12]: names = "spam,ham" In [13]: n = np.rec.fromrecords(data, names=names) In [14]: print n [(datetime.date(2001, 1, 1), 2) (datetime.date(2001, 1, 2), 4) (datetime.date(2001, 1, 3), 6)] From zachary.pincus at yale.edu Thu Aug 7 15:47:28 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 7 Aug 2008 15:47:28 -0400 Subject: [Numpy-discussion] Bug? Numpy crash with bad __array_interface__ Message-ID: Hello all, As per my previous email, I encountered a strange sometimes-segfault when using 'numpy.array(thing)' to convert 'thing' (which provides an __array_interface__) to a numpy array. The offending __array_interface__ has a 'data' item that is a python string (not, as specified in the protocol, a pointer to a memory area) which is too small for the provided array size. Which, in some cases, causes a segfault. Is this a bug? That is, should numpy check the size of the 'data' (when 'data' is in fact a sized python object and not just a bare pointer) against the size of the array and provided typestr? Or is this too much of a corner case to deal with... Zach From zachary.pincus at yale.edu Thu Aug 7 16:28:21 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 7 Aug 2008 16:28:21 -0400 Subject: [Numpy-discussion] packbits / unpackbits bugs or misfeatures? Message-ID: <8DC5D083-1EDD-48DE-B78E-C823708A1575@yale.edu> Hello all, numpy.unpackbits has a docstring that states that it returns a boolean array, but the function instead returns a uint8 array. Should I enter this in trac as a documentation bug or a functionality bug? Also, numpy.packbits will not accept a bool-typed array as input (only scalar-typed arrays) -- should it have this ability, for symmetry with unpackbits? (Assuming that unpackbits should indeed unpack into bool arrays...) Zach From oc-spam66 at laposte.net Thu Aug 7 16:44:20 2008 From: oc-spam66 at laposte.net (oc-spam66) Date: Thu, 7 Aug 2008 22:44:20 +0200 (CEST) Subject: [Numpy-discussion] numpy.fromstring() : error handling ? Message-ID: <13255306.1013121218141860247.JavaMail.www@wwinf8304> Hello, the following code drives python into an endless loop : >>> import numpy >>> numpy.fromstring("abcd", dtype = float, sep = ' ') I think the function numpy.fromstring is lacking an adequate error handling for that case. Is it a bug ? Regards, -- O.C. Python 2.5.2 Debian Lenny Cr?ez votre adresse ?lectronique prenom.nom at laposte.net 1 Go d'espace de stockage, anti-spam et anti-virus int?gr?s. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.pincus at yale.edu Thu Aug 7 17:27:52 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 7 Aug 2008 17:27:52 -0400 Subject: [Numpy-discussion] numpy.fromstring() : error handling ? In-Reply-To: <13255306.1013121218141860247.JavaMail.www@wwinf8304> References: <13255306.1013121218141860247.JavaMail.www@wwinf8304> Message-ID: > the following code drives python into an endless loop : > > >>> import numpy > >>> numpy.fromstring("abcd", dtype = float, sep = ' ') It works on OS X 10.5.4 with a today's SVN head of numpy: In [1]: import numpy In [2]: numpy.fromstring("abcd", dtype = float, sep = ' ') Out[2]: array([ 0.]) In [3]: numpy.version.version Out[3]: '1.2.0.dev5619' Perhaps this was fixed recently, or the bug is platform-specific? Zach On Aug 7, 2008, at 4:44 PM, oc-spam66 wrote: > Hello, > > the following code drives python into an endless loop : > > >>> import numpy > >>> numpy.fromstring("abcd", dtype = float, sep = ' ') > > I think the function numpy.fromstring is lacking an adequate error > handling for that case. > > Is it a bug ? > > Regards, > > -- > O.C. > Python 2.5.2 > Debian Lenny > > > Cr?ez votre adresse ?lectronique prenom.nom at laposte.net > 1 Go d'espace de stockage, anti-spam et anti-virus int?gr?s. > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From pav at iki.fi Thu Aug 7 17:34:13 2008 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 7 Aug 2008 21:34:13 +0000 (UTC) Subject: [Numpy-discussion] numpy.fromstring() : error handling ? References: <13255306.1013121218141860247.JavaMail.www@wwinf8304> Message-ID: Thu, 07 Aug 2008 22:44:20 +0200, oc-spam66 wrote: > Hello, > > the following code drives python into an endless loop : > >>>> import numpy >>>> numpy.fromstring("abcd", dtype = float, sep = ' ') > > I think the function numpy.fromstring is lacking an adequate error > handling for that case. > > Is it a bug ? It was fixed in r5438 and the fix is also in Numpy 1.1.1. -- Pauli Virtanen From mailanhilli at googlemail.com Fri Aug 8 00:38:55 2008 From: mailanhilli at googlemail.com (Matthias Hillenbrand) Date: Thu, 7 Aug 2008 23:38:55 -0500 Subject: [Numpy-discussion] Horizontal lines in diffraction image (NumPy FFT) In-Reply-To: <67b3a51f0808061430v1f0f6e4btea1adf5d83726396@mail.gmail.com> References: <67b3a51f0808051926r1aefac3dibfbffda4100d9fb5@mail.gmail.com> <67b3a51f0808061430v1f0f6e4btea1adf5d83726396@mail.gmail.com> Message-ID: <67b3a51f0808072138l173f7118n910feac12310eb84@mail.gmail.com> Hello, today I found time to do concentrate on the proposed zero padding. My Gaussian beam and the lenses have a diameter of approximately 2^16 array elements while the total array has a size of 2^18. The absolute value of the Gaussian beam multiplied by the lenses converges to zero. Out of this reason I assume that zero padding is not the reason for the horizontal lines. I also inreased the amount of zero padding by the factor of 4 but the horizontal lines were still present. Could something like a difference in the calculation precision (complex numbers) be the reason for the difference between Octave and Numpy? Matthias From millman at berkeley.edu Fri Aug 8 03:12:52 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Fri, 8 Aug 2008 00:12:52 -0700 Subject: [Numpy-discussion] ATTENTION: 1.2.0b1 tagged tomorrow Message-ID: Just a reminder: I will be tagging 1.2.0b1 tomorrow evening. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From oc-spam66 at laposte.net Fri Aug 8 04:42:07 2008 From: oc-spam66 at laposte.net (oc-spam66) Date: Fri, 8 Aug 2008 10:42:07 +0200 (CEST) Subject: [Numpy-discussion] numpy.fromstring() : error handling ? Message-ID: <17490654.1031911218184927381.JavaMail.www@wwinf8403> Thank you for the answers, I am now disturbed by this result : > In [1]: import numpy > In [2]: numpy.fromstring("abcd", dtype = float, sep = ' ') > Out[2]: array([ 0.]) Shouldn't it raise an exception ValueError ? (because "abcd" is not a float) Regards, O.C. Cr?ez votre adresse ?lectronique prenom.nom at laposte.net 1 Go d'espace de stockage, anti-spam et anti-virus int?gr?s. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Fri Aug 8 12:28:37 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 8 Aug 2008 18:28:37 +0200 Subject: [Numpy-discussion] unique1d returning indices In-Reply-To: <48999E96.6070000@ntc.zcu.cz> References: <48999E96.6070000@ntc.zcu.cz> Message-ID: <9457e7c80808080928p2e6d404ak91576a44a5234e7@mail.gmail.com> Hi Robert 2008/8/6 Robert Cimrman : > Note also that the order of outputs has changed (previously unique1d() > returned (i, b) for return_index=True). Does this not constitute an API change? St?fan From oc-spam66 at laposte.net Fri Aug 8 18:33:27 2008 From: oc-spam66 at laposte.net (oc-spam66) Date: Sat, 9 Aug 2008 00:33:27 +0200 (CEST) Subject: [Numpy-discussion] Appending data to a big ndarray Message-ID: <232295.1146841218234807986.JavaMail.www@wwinf8217> Hello, I would like to build a big ndarray by adding rows progressively. I considered the following functions : append, concatenate, vstack and the like. It appears to me that they all create a new array (which requires twice the memory). Is there a method for just adding a row to a ndarray without duplicating the data ? Regards, O.C. Cr?ez votre adresse ?lectronique prenom.nom at laposte.net 1 Go d'espace de stockage, anti-spam et anti-virus int?gr?s. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Fri Aug 8 19:03:04 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 08 Aug 2008 18:03:04 -0500 Subject: [Numpy-discussion] Appending data to a big ndarray In-Reply-To: <232295.1146841218234807986.JavaMail.www@wwinf8217> References: <232295.1146841218234807986.JavaMail.www@wwinf8217> Message-ID: <489CD0A8.7090700@enthought.com> oc-spam66 wrote: > Hello, > > I would like to build a big ndarray by adding rows progressively. > > I considered the following functions : append, concatenate, vstack and > the like. > It appears to me that they all create a new array (which requires > twice the memory). > > Is there a method for just adding a row to a ndarray without > duplicating the data ? You can resize the memory for an array with the resize method. You have to be careful using this approach if you have other views looking at the same memory because the resize method may move the pointer which is very bad for all the other items looking at it: a = array([1,2,3,4]) a.resize((10,4)) -Travis From Chris.Barker at noaa.gov Fri Aug 8 19:29:56 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 08 Aug 2008 16:29:56 -0700 Subject: [Numpy-discussion] numpy.fromstring() : error handling ? In-Reply-To: <17490654.1031911218184927381.JavaMail.www@wwinf8403> References: <17490654.1031911218184927381.JavaMail.www@wwinf8403> Message-ID: <489CD6F4.80100@noaa.gov> oc-spam66 wrote: > I am now disturbed by this result : > > > In [1]: import numpy > > In [2]: numpy.fromstring("abcd", dtype = float, sep = ' ') > > Out[2]: array([ 0.]) > > Shouldn't it raise an exception ValueError ? (because "abcd" is not a float) I don't think so, but it shouldn't return a zero either. That call should mean: scan this whitespace separated string for as many floating point numbers as it has. There are none, so it should return and empty array: array([], dtype=float64) fromstring() needs some work in a few other ways -- see my message a couple days ago. Does anyone want to work on fromstring()? I'd like to , but my C skills are weak, so I could use some help. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oc-spam66 at laposte.net Fri Aug 8 21:11:41 2008 From: oc-spam66 at laposte.net (oc-spam66) Date: Sat, 9 Aug 2008 03:11:41 +0200 (CEST) Subject: [Numpy-discussion] numpy.fromstring() : error handling ? Message-ID: <26574853.1109201218244301653.JavaMail.www@wwinf8302> > > Shouldn't it raise an exception ValueError ? (because "abcd" is not a float) > > I don't think so, but it shouldn't return a zero either. > > That call should mean: scan this whitespace separated string for as many > floating point numbers as it has. There are none, so it should return > and empty array: Is it really how it should be ? I thought this method should be restrictive in what it accepts (because the goal is to fill an array with precise dimensions). When floating point numbers are mixed with something else, it should be parsed with a regexp. Cr?ez votre adresse ?lectronique prenom.nom at laposte.net 1 Go d'espace de stockage, anti-spam et anti-virus int?gr?s. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Fri Aug 8 22:22:27 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 08 Aug 2008 21:22:27 -0500 Subject: [Numpy-discussion] C-API change for 1.2 Message-ID: <489CFF63.9000703@enthought.com> Hi all, The 1.2 version of NumPy is going to be tagged. There is at least one change I'd like to add: The hasobject member of the PyArray_Descr structure should be renamed to "flags" and converted to a 32-bit integer. What does everybody think about this change? It should have minimal affect except to require a re-compile of extension modules using NumPy. The only people requiring code changes would be those making intimate use of the PyArray_Descr structure instead of using the macros. It's a simple change if there is no major opposition. -Travis From millman at berkeley.edu Sat Aug 9 01:51:26 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Fri, 8 Aug 2008 22:51:26 -0700 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: <489CFF63.9000703@enthought.com> References: <489CFF63.9000703@enthought.com> Message-ID: On Fri, Aug 8, 2008 at 7:22 PM, Travis E. Oliphant wrote: > The 1.2 version of NumPy is going to be tagged. There is at least one > change I'd like to add: The hasobject member of the PyArray_Descr > structure should be renamed to "flags" and converted to a 32-bit > integer. > > What does everybody think about this change? It should have minimal > affect except to require a re-compile of extension modules using NumPy. > The only people requiring code changes would be those making intimate > use of the PyArray_Descr structure instead of using the macros. +1 I am going to hold off on tagging 1.2.0b1 until Monday. This way if anyone has any objections to this change, they will have a few days to discuss this. If there are no major objections by Monday, go ahead and make the change and I will tag the beta release. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From Chris.Barker at noaa.gov Sat Aug 9 02:18:29 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 08 Aug 2008 23:18:29 -0700 Subject: [Numpy-discussion] numpy.fromstring() : error handling ? In-Reply-To: <26574853.1109201218244301653.JavaMail.www@wwinf8302> References: <26574853.1109201218244301653.JavaMail.www@wwinf8302> Message-ID: <489D36B5.6030200@noaa.gov> oc-spam66 wrote: > Is it really how it should be ? > > I thought this method should be restrictive in what it accepts (because > the goal is to fill an array with precise dimensions). if you know how big an array you want, then you should use the "count" keyword: np.fromstring("45.6 76.4", dtype=np.float, sep = ' ', count = 1) > When floating > point numbers are mixed with something else, it should be parsed with a > regexp. Or any number of other ways. Yes, fromstring (and from file) is for only the simplest of needs, very efficiently pulling a bunch of numbers from text. In any case, your example shows a bug, plain and simple. Parsing "kjhsdf" should not produce the number zero, period. I suppose what it should do it up for debate, it could either raise an exception or return an empty array. By the way, I've found another bug: >> np.fromstring("23 ", dtype=np.float, sep=" ", count=2) array([ 3.400000000e+01, 1.09284155e-305]) That really should raise an exception rather than returning garbage! Oh well, it looks like there is some work to be done. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception From matthieu.brucher at gmail.com Sat Aug 9 06:52:36 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Sat, 9 Aug 2008 12:52:36 +0200 Subject: [Numpy-discussion] Links for installing Numpy 1.1.1 Message-ID: Hi, There seems to be missing parameters when I try to download numpy 1.1.1 for Python 2.5 (David's superpack). And when I try to go to the numpy project on SF, the project is invalid as well. Is there something I'm missing ? Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher From nadavh at visionsense.com Sat Aug 9 11:39:33 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Sat, 9 Aug 2008 18:39:33 +0300 Subject: [Numpy-discussion] Horizontal lines in diffraction image (NumPyFFT) References: <67b3a51f0808051926r1aefac3dibfbffda4100d9fb5@mail.gmail.com><67b3a51f0808061430v1f0f6e4btea1adf5d83726396@mail.gmail.com> <67b3a51f0808072138l173f7118n910feac12310eb84@mail.gmail.com> Message-ID: <710F2847B0018641891D9A216027636029C213@ex3.envision.co.il> I run your code, and the results seems to me realistic, similar to what I got long back ago --- I do not see artefacts. You may try a more elaborate approach (vector field propagation) using camfr (sf.net/projects/camfr). Nadav. N.B. I tried to attach the image I got b y running your script, but I was too big to for this mailing list. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Matthias Hillenbrand ????: ? 08-??????-08 07:38 ??: numpy-discussion at scipy.org ????: Re: [Numpy-discussion] Horizontal lines in diffraction image (NumPyFFT) Hello, today I found time to do concentrate on the proposed zero padding. My Gaussian beam and the lenses have a diameter of approximately 2^16 array elements while the total array has a size of 2^18. The absolute value of the Gaussian beam multiplied by the lenses converges to zero. Out of this reason I assume that zero padding is not the reason for the horizontal lines. I also inreased the amount of zero padding by the factor of 4 but the horizontal lines were still present. Could something like a difference in the calculation precision (complex numbers) be the reason for the difference between Octave and Numpy? Matthias _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3732 bytes Desc: not available URL: From robert.kern at gmail.com Sat Aug 9 15:24:47 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 9 Aug 2008 14:24:47 -0500 Subject: [Numpy-discussion] Links for installing Numpy 1.1.1 In-Reply-To: References: Message-ID: <3d375d730808091224m2cab44wf1bd0e6c16f027b1@mail.gmail.com> On Sat, Aug 9, 2008 at 05:52, Matthieu Brucher wrote: > Hi, > > There seems to be missing parameters when I try to download numpy > 1.1.1 for Python 2.5 (David's superpack). And when I try to go to the > numpy project on SF, the project is invalid as well. Is there > something I'm missing ? Which links are you referring to? The one on the PyPI page works for me: http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From matthieu.brucher at gmail.com Sat Aug 9 15:40:42 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Sat, 9 Aug 2008 21:40:42 +0200 Subject: [Numpy-discussion] Links for installing Numpy 1.1.1 In-Reply-To: <3d375d730808091224m2cab44wf1bd0e6c16f027b1@mail.gmail.com> References: <3d375d730808091224m2cab44wf1bd0e6c16f027b1@mail.gmail.com> Message-ID: 2008/8/9 Robert Kern : > On Sat, Aug 9, 2008 at 05:52, Matthieu Brucher > wrote: >> Hi, >> >> There seems to be missing parameters when I try to download numpy >> 1.1.1 for Python 2.5 (David's superpack). And when I try to go to the >> numpy project on SF, the project is invalid as well. Is there >> something I'm missing ? > > Which links are you referring to? The one on the PyPI page works for me: > > http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 > > -- > Robert Kern It seems to be working now... Perhaps an issue with SF. Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher From mnandris at blueyonder.co.uk Sat Aug 9 16:19:47 2008 From: mnandris at blueyonder.co.uk (Michael) Date: Sat, 09 Aug 2008 21:19:47 +0100 Subject: [Numpy-discussion] np.bincount() won't accept list or array from np.random.beta and acts like len Message-ID: <1218313187.6554.22.camel@mik> Hi list, I have found np.bincount() not to behave as expected when provided with data from np.random.beta. It works fine with lists, but not the type returned by np.random.beta... there seems to be a block on casting too. When it does give output on np.array it just reports the number of items, like len() ?? x = np.array([1,1,1,1,2,2,4,4,5,6,6,6]) print np.bincount(x) prior = np.random.beta(2,7,1000) a = prior.tolist() print type(a) print np.bincount(a) b = [ round(i,2) for i in a ] print b; print type(b) print np.bincount(b) #c = np.array(a) #print np.bincount(c); print type(c) i know its easily solved but bincount is probably faster than making a pass over the probabilities and hashing them as strings etc have got a class that subclasses dict and accepts np.random.beta as 'seq': ... for prob in seq: self.bincount( prob ) def bincount( self, pr, increment=1 ): strPr = str(round(pr,self.dp)) self[strPr] = increment + self.get( strPr, 0 ) would prefer the one-liner: self.bincount = np.bincount(seq) any ideas? thanks. Michael Nandris (btw apologies for the cross posting) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From robert.kern at gmail.com Sat Aug 9 16:40:32 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 9 Aug 2008 15:40:32 -0500 Subject: [Numpy-discussion] np.bincount() won't accept list or array from np.random.beta and acts like len In-Reply-To: <1218313187.6554.22.camel@mik> References: <1218313187.6554.22.camel@mik> Message-ID: <3d375d730808091340j4b8ec79fx19998b98d55f1468@mail.gmail.com> On Sat, Aug 9, 2008 at 15:19, Michael wrote: > Hi list, > > I have found np.bincount() not to behave as expected when provided with > data from np.random.beta. As the docstring states, bincount() only works on sequences of non-negative integers, not floats. beta() returns floats. > It works fine with lists, but not the type returned by np.random.beta... > there seems to be a block on casting too. When it does give output on > np.array it just reports the number of items, like len() beta() yields float numbers in the range [0..1]. When you give bincount() a list of such floats, it makes an integer array out of them without warning (when you give it an array, it can easily check the type and prevent you from giving it floats). Thus you get an array of all 0s since casting to integers truncates to the next lowest integer. bincount() will give you an array of counts such that counts[0] is the number of 0s in the input; in this case, that is the length of the input. > x = np.array([1,1,1,1,2,2,4,4,5,6,6,6]) > print np.bincount(x) > > prior = np.random.beta(2,7,1000) > a = prior.tolist() > > print type(a) > print np.bincount(a) > > b = [ round(i,2) for i in a ] > print b; print type(b) > print np.bincount(b) This still doesn't work because round() gives you floats. > #c = np.array(a) > #print np.bincount(c); print type(c) > > i know its easily solved but bincount is probably faster than making a > pass over the probabilities and hashing them as strings etc You need to convert them to integers first with digitize(). Or just use histogram(), which does all this for you. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pav at iki.fi Sat Aug 9 17:00:16 2008 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 9 Aug 2008 21:00:16 +0000 (UTC) Subject: [Numpy-discussion] numpy.fromstring() : error handling ? References: <26574853.1109201218244301653.JavaMail.www@wwinf8302> <489D36B5.6030200@noaa.gov> Message-ID: Fri, 08 Aug 2008 23:18:29 -0700, Christopher Barker wrote: > oc-spam66 wrote: [clip fromfile/fromstring bugs] So that we won't forget this, a summary ticket is here: http://scipy.org/scipy/numpy/ticket/883 Another fromfile/fromstring problem, this time related to locales: http://scipy.org/scipy/numpy/ticket/884 Similar problem, but due to platform representation of inf/nan: http://scipy.org/scipy/numpy/ticket/510 I think we really ought to raise a ValueError if malformed data is encountered (before count is reached, or before EOF if no count), rather than truncating the result or returning zeros or garbage. This is of course an API change... -- Pauli Virtanen From peridot.faceted at gmail.com Sat Aug 9 17:10:41 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sat, 9 Aug 2008 17:10:41 -0400 Subject: [Numpy-discussion] Horizontal lines in diffraction image (NumPy FFT) In-Reply-To: <67b3a51f0808072138l173f7118n910feac12310eb84@mail.gmail.com> References: <67b3a51f0808051926r1aefac3dibfbffda4100d9fb5@mail.gmail.com> <67b3a51f0808061430v1f0f6e4btea1adf5d83726396@mail.gmail.com> <67b3a51f0808072138l173f7118n910feac12310eb84@mail.gmail.com> Message-ID: 2008/8/8 Matthias Hillenbrand : > My Gaussian beam and the lenses have a diameter of approximately 2^16 > array elements while the total array has a size of 2^18. The absolute > value of the Gaussian beam multiplied by the lenses converges to zero. > Out of this reason I assume that zero padding is not the reason for > the horizontal lines. I also inreased the amount of zero padding by > the factor of 4 but the horizontal lines were still present. Could > something like a difference in the calculation precision (complex > numbers) be the reason for the difference between Octave and Numpy? Your code is fairly complicated, and just looking at it I don't see a reason for there to be a difference between Octave and Numpy. But here are some things you could try: * run numpy.test() and make sure numpy passes all tests * follow each fft with an inverse fft and compare with the input * print out the input and output arrays and compare them visually * save the input array to fft from each script and compare the ffts using both tools * email the list with a minimal example: an array (preferably small) whose fft is different in Octave and numpy Good luck, Anne From peridot.faceted at gmail.com Sat Aug 9 17:28:07 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sat, 9 Aug 2008 17:28:07 -0400 Subject: [Numpy-discussion] Appending data to a big ndarray In-Reply-To: <232295.1146841218234807986.JavaMail.www@wwinf8217> References: <232295.1146841218234807986.JavaMail.www@wwinf8217> Message-ID: 2008/8/8 oc-spam66 : > Hello, > > I would like to build a big ndarray by adding rows progressively. > > I considered the following functions : append, concatenate, vstack and the > like. > It appears to me that they all create a new array (which requires twice the > memory). > > Is there a method for just adding a row to a ndarray without duplicating the > data ? Since ndarrays must be contiguous in memory, at the end of the day you must have a contiguous block big enough to contain the whole array. If you can allocate this right away, life is easy. All you have to do is np.zeros((guess_rows,columns)) and you get a big empty array. But it's probably safe to assume that if you could predict how big the array would be, you wouldn't have asked this question. If you *don't* know how big the array needs to be, then you have basically two options: * use a list of row arrays while growing and construct the final array at the end * keep enlarging the array as needed Keeping a list is actually my preferred solution; each row is contiguous, but the array as a whole is not. But python lists expand automatically, conveniently, and efficiently, and the data is not copied. At the end, though, you need to copy all those rows into one big array. While easy - just use np.array() - this does briefly require twice the virtual memory, and could be slow. Enlarging the array means, potentially, a slow copy every time it grows. If you use the resize() method Travis suggested, it may occur that these resizings can occur without copies. For small arrays, they almost certainly will require copies, since the array will tend to outgrow memory arenas. But, on Linux at least (and probably on other modern OSes), large arrays live in chunks of virtual memory requested from the system directly. If you are lucky and no other chunk of memory is allocated immediately afterward, it should be possible to enlarge this memory by just adding more pages onto the end. If not, though, you get a big copy. You also, of course, have a trade-off between using more space than you need and having (potentially) lots of copies. You can try this, but sadly, I think if your array is within a factor of two of the size of available virtual memory, you're going to be disappointed with numpy. Very many operations require a large temporary array. I recommend just going with something simple and living with a single big copy. Anne From stefan at sun.ac.za Sat Aug 9 18:30:37 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 10 Aug 2008 00:30:37 +0200 Subject: [Numpy-discussion] packbits / unpackbits bugs or misfeatures? In-Reply-To: <8DC5D083-1EDD-48DE-B78E-C823708A1575@yale.edu> References: <8DC5D083-1EDD-48DE-B78E-C823708A1575@yale.edu> Message-ID: <9457e7c80808091530t7cba466dn9c860a966a6a8d46@mail.gmail.com> Hey Zach 2008/8/7 Zachary Pincus : > Hello all, > > numpy.unpackbits has a docstring that states that it returns a boolean > array, but the function instead returns a uint8 array. Should I enter > this in trac as a documentation bug or a functionality bug? Would you mind fixing up the docs on the documentation editor? Thanks! St?fan From stefan at sun.ac.za Sat Aug 9 19:47:38 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 10 Aug 2008 01:47:38 +0200 Subject: [Numpy-discussion] Horizontal lines in diffraction image (NumPyFFT) In-Reply-To: <710F2847B0018641891D9A216027636029C208@ex3.envision.co.il> References: <67b3a51f0808051926r1aefac3dibfbffda4100d9fb5@mail.gmail.com> <67b3a51f0808061430v1f0f6e4btea1adf5d83726396@mail.gmail.com> <710F2847B0018641891D9A216027636029C208@ex3.envision.co.il> Message-ID: <9457e7c80808091647t3cb5b7c9h3e2f042af4e6ac14@mail.gmail.com> 2008/8/7 Nadav Horesh : > The 0 padding is easy in numpp/pylab as in octave/matlab, by just adding one > argument. In pylab it is the "a" keyword: > > y = fft(x, n=2*len(x)) Just be careful about using the n parameter for `ifft`; in that case it does not pad correctly. St?fan From zachary.pincus at yale.edu Sat Aug 9 21:46:45 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Sat, 9 Aug 2008 21:46:45 -0400 Subject: [Numpy-discussion] packbits / unpackbits bugs or misfeatures? In-Reply-To: <9457e7c80808091530t7cba466dn9c860a966a6a8d46@mail.gmail.com> References: <8DC5D083-1EDD-48DE-B78E-C823708A1575@yale.edu> <9457e7c80808091530t7cba466dn9c860a966a6a8d46@mail.gmail.com> Message-ID: <70CBF599-92DA-471F-BA90-DB7C8A8D6FCA@yale.edu> >> Hello all, >> >> numpy.unpackbits has a docstring that states that it returns a >> boolean >> array, but the function instead returns a uint8 array. Should I enter >> this in trac as a documentation bug or a functionality bug? > > Would you mind fixing up the docs on the documentation editor? Except that it would kind of make more sense, at least to me, for unpackbits to return a boolean array, and packbits to take boolean arrays. I could try to look into fixing the code instead of the docs, if others agree. Otherwise I'll fix the docs. Maybe that's better from an API-breakage standpoint anyway, regardless of what "makes sense to me"... Zach From cournape at gmail.com Sun Aug 10 09:45:46 2008 From: cournape at gmail.com (David Cournapeau) Date: Sun, 10 Aug 2008 08:45:46 -0500 Subject: [Numpy-discussion] numpy 1.1.rc2: win32 binaries In-Reply-To: <4896F0E8.40409@ar.media.kyoto-u.ac.jp> References: <488D58A3.6070800@ar.media.kyoto-u.ac.jp> <488D5984.202@ar.media.kyoto-u.ac.jp> <4896F0E8.40409@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220808100645p4c05c0a1q1fb9932d4d282c40@mail.gmail.com> On Mon, Aug 4, 2008 at 7:07 AM, David Cournapeau wrote: > Bruce Southey wrote: >> Hi, >> I installed the 'official binaries' on a Intel Celeron M 530 that >> supports SSE2 and SSE3 running MS Vista. All tests passed and with >> regards to the ticket: numpy.inner(F,F) resulted in 'array([[ Inf]])' >> > > Ok, it looks like this is entirely due to my own stupidity. I screwed > up, and linked the binaries to an earlier version of ATLAS, the one with > the problem... > > I will try to generate new binaries within today, The binaries on sourceforge are now the new ones, with the correct atlas linked in. If there are still problems, don't hesitate to bring it on the ML. cheers, David > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From ctw at cogsci.info Sun Aug 10 14:26:33 2008 From: ctw at cogsci.info (Christoph T. Weidemann) Date: Sun, 10 Aug 2008 14:26:33 -0400 Subject: [Numpy-discussion] ndarray subclassing bug? Message-ID: I've come across a strange behavior for classes subclassed from ndarray. Here's a minimal example that illustrates the problem: import numpy as np class TestArray(np.ndarray): def __new__(cls, data, info=None, dtype=None, copy=False): subarr = np.array(data, dtype=dtype, copy=copy) subarr = subarr.view(cls) return subarr def sort(self,*args,**kwargs): print type(self) print type(self.base) Now consider this: In [1]: tst = TestArray(np.random.rand(2,3)) In [2]: tst.sort() In [3]: np.sort(tst) Out[3]: TestArray([[ 0.90489484, 0.950291 , 0.80753772], [ 0.49020689, 0.84582283, 0.61532922]]) Why whould tst.sort() show the correct base class and np.sort show NoneType as base class for tst? I'd appreciate any insights ... From pgmdevlist at gmail.com Sun Aug 10 14:33:03 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Sun, 10 Aug 2008 14:33:03 -0400 Subject: [Numpy-discussion] ndarray subclassing bug? In-Reply-To: References: Message-ID: <200808101433.03417.pgmdevlist@gmail.com> On Sunday 10 August 2008 14:26:33 Christoph T. Weidemann wrote: > Why whould tst.sort() show the correct base class and np.sort show > NoneType as base class for tst? > I'd appreciate any insights ... numpy functions will return arrays of the type which has the largest priority, with ndarrays a priority of 1 by default. If you set a Class variable __array_priority__ to a number larger than 1, that should fix your problem. From ctw at cogsci.info Sun Aug 10 14:48:25 2008 From: ctw at cogsci.info (Christoph T. Weidemann) Date: Sun, 10 Aug 2008 14:48:25 -0400 Subject: [Numpy-discussion] ndarray subclassing bug? In-Reply-To: References: Message-ID: You wrote: > numpy functions will return arrays of the type which has the largest priority, > with ndarrays a priority of 1 by default. If you set a Class variable > __array_priority__ to a number larger than 1, that should fix your problem. The following code produces the same behavior: import numpy as np class TestArray(np.ndarray): __array_priority__=2 def __new__(cls, data, info=None, dtype=None, copy=False): subarr = np.array(data, dtype=dtype, copy=copy) subarr = subarr.view(cls) return subarr def sort(self,*args,**kwargs): print type(self) print type(self.base) Am I doing something wrong? > class TestArray(np.ndarray): > def __new__(cls, data, info=None, dtype=None, copy=False): > subarr = np.array(data, dtype=dtype, copy=copy) > subarr = subarr.view(cls) > return subarr > > def sort(self,*args,**kwargs): > print type(self) > print type(self.base) > > > Now consider this: > In [1]: tst = TestArray(np.random.rand(2,3)) > > In [2]: tst.sort() > > > > In [3]: np.sort(tst) > > > Out[3]: > TestArray([[ 0.90489484, 0.950291 , 0.80753772], > [ 0.49020689, 0.84582283, 0.61532922]]) > > > Why whould tst.sort() show the correct base class and np.sort show > NoneType as base class for tst? > I'd appreciate any insights ... > From pgmdevlist at gmail.com Sun Aug 10 15:09:21 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Sun, 10 Aug 2008 15:09:21 -0400 Subject: [Numpy-discussion] ndarray subclassing bug? In-Reply-To: References: Message-ID: <200808101509.22240.pgmdevlist@gmail.com> On Sunday 10 August 2008 14:26:33 Christoph T. Weidemann wrote: > Why whould tst.sort() show the correct base class and np.sort show > NoneType as base class for tst? > I'd appreciate any insights ... Christoph, I should take the time to read the question first before answering. So, according to the numpy book, the .base attribute is "the object the array is using for its data buffer, or None if it owns its own memory". When you create a TestArray, the object points in fact to the subarr variable you have defined in your __new__, so your TestArray doesn't own its memory, and .base points to subarr. When you call TestArray.sort, you're not doing anything at all: the .base attribute still points to the subarr temporary array you have created, a ndarray When you call np.sort, you create a copy of your TestArray that ends up owning its own memory: its .base becomes None. I hope I got it right. So no, you didn't do anything wrong. What behavior were you expecting ? From ctw at cogsci.info Sun Aug 10 15:50:15 2008 From: ctw at cogsci.info (Christoph T. Weidemann) Date: Sun, 10 Aug 2008 15:50:15 -0400 Subject: [Numpy-discussion] ndarray subclassing bug? In-Reply-To: References: Message-ID: Pierre, Thanks for the fast and informative answers! On Sun you wrote: > So no, you didn't do anything wrong. What behavior were you expecting ? I was expecting that calls to self.base would produce identical results for objects of the TestArray class, no matter how these calls are triggered. I used the base attribute to call ndarray functions on my derived class and obviously that failed when base was NoneType. I now realize that this is not a good/robust use of the base attribute and I found a different way to implement the functionality I needed. From pgmdevlist at gmail.com Sun Aug 10 15:49:29 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Sun, 10 Aug 2008 15:49:29 -0400 Subject: [Numpy-discussion] ndarray subclassing bug? In-Reply-To: References: Message-ID: <200808101549.30026.pgmdevlist@gmail.com> On Sunday 10 August 2008 15:50:15 Christoph T. Weidemann wrote: > Pierre, > Thanks for the fast and informative answers! Sorry again for the confusion, the first answer was definitely too fast... > I used the base attribute to call ndarray functions on > my derived class and obviously that failed when base was NoneType. I > now realize that this is not a good/robust use of the base attribute > and I found a different way to implement the functionality I needed. What functionality, may I ask ? I also recommend http://www.scipy.org/Subclasses on the scipy wiki. Please don't hesitate to edit it with your own observations. From rob.clewley at gmail.com Sun Aug 10 16:34:34 2008 From: rob.clewley at gmail.com (Rob Clewley) Date: Sun, 10 Aug 2008 16:34:34 -0400 Subject: [Numpy-discussion] New python module to simulate arbitrary fixed and infinite precision binary floating point Message-ID: Dear Pythonistas, How many times have we seen posts recently along the lines of "why is it that 0.1 appears as 0.10000000000000001 in python?" that lead to posters being sent to the definition of the IEEE 754 standard and the decimal.py module? I am teaching an introductory numerical analysis class this fall, and I realized that the best way to teach this stuff is to be able to play with the representations directly, in particular to be able to see it in action on a simpler system than full 64-bit precision, especially when str(f) or repr(f) won't show *all* of the significant digits stored in a float. The decimal class deliberately avoids binary representation issues, and I can't find what I want online. Consequently, I have written a module to simulate the machine representation of binary floating point numbers and their arithmetic. Values can be of arbitrary fixed precision or infinite precision, along the same lines as python's in-built decimal class. The code is here: http://www2.gsu.edu/~matrhc/binary.html The design is loosely based on that decimal module, although it doesn't get in to threads, for instance. You can play with different IEEE 754 representations with different precisions and rounding modes, and compare with infinite precision Binary numbers. For instance, it is easy to learn about machine epsilon, representation/rounding error using a much simpler format such as a 4-bit exponent and 6-bit mantissa. Such a format is easily defined in the new module and can be manipulated easily: >>> context = define_context(4, 6, ROUND_DOWN) >>> zero = context(0) Binary("0", (4, 6, ROUND_DOWN)) >>> print zero # sign, characteristic, significand bits 0 0000 000000 >>> zero.next() Binary("0.001E-9", (4, 6, ROUND_DOWN)) >>> print zero.next() 0 0000 000001 >>> largest_denormalized = context('0 0000 111111') # direct spec of the sign, characteristic, and significand bits >>> largest_denormalized Binary("0.111111E-6", (4, 6, ROUND_DOWN)) >>> largest_denormalized.as_decimal() Decimal("0.015380859375") >>> n01 = context(0.1) # nearest representable is actually stored >>> print n01, " rounded to ", n01.as_decimal() 0 0011 100110 rounded to 0.099609375 >>> Binary('-10111.0000001').as_decimal() Decimal("-23.0078125") >>> Binary('-10111.0000001', context).as_decimal() # not enough precision in this context Decimal("-23.0000") >>> diff = abs(Binary('-10111.0000001') - Binary('-10111.0000001', context)) >>> diff Binary("0.1E-6", (4, 6, ROUND_DOWN)) The usual arithmetic operations are permitted on these objects, as well as representations of their values in decimal or binary form. Default contexts for half, single, double, and quadruple IEEE 754 precision floats are provided. Binary integer classes are also provided, and some other utility functions for converting between decimal and binary string representations. The module is compatible with the numpy float classes and requires numpy to be installed. The source code is released under the BSD license, but I am amenable to other licensing ideas if there is interest in adapting the code for some other purpose. Full details of the functionality and known issues are in the module's docstring, and many examples of usage are in the accompanying file binary_tests.py (which also acts to validate the common representations against the built-in floating point types). I look forward to hearing feedback, especially in case of bugs or suggestions for improvements. -Rob -- Robert H. Clewley, Ph. D. Assistant Professor Department of Mathematics and Statistics Georgia State University 720 COE, 30 Pryor St Atlanta, GA 30303, USA tel: 404-413-6420 fax: 404-413-6403 http://www2.gsu.edu/~matrhc http://brainsbehavior.gsu.edu/ From ctw at cogsci.info Sun Aug 10 17:52:45 2008 From: ctw at cogsci.info (Christoph T. Weidemann) Date: Sun, 10 Aug 2008 17:52:45 -0400 Subject: [Numpy-discussion] ndarray subclassing bug? In-Reply-To: References: Message-ID: On Sun, Aug 10, you wrote: > What functionality, may I ask ? I am working on a dimensioned array data structure that is a subclass of ndarray and lets you keep track of the different dimensions of an array, assign labels to them (and the different levels on each dimensions) and allows for smart indexing like data['time>2000']. I'm hoping to have it at a shareable stage pretty soon. > I also recommend http://www.scipy.org/Subclasses on the scipy wiki. I know that page -- thanks so much for your work on that page. It has helped me tremendously! From ctw at cogsci.info Sun Aug 10 20:37:35 2008 From: ctw at cogsci.info (Christoph T. Weidemann) Date: Sun, 10 Aug 2008 20:37:35 -0400 Subject: [Numpy-discussion] compress() signature Message-ID: The signature for numpy.compress() seems somewhat inconsistent with other functions, because the first argument it expects is not the array but the condition. This tends to throw me off when I use this function, because the common order for other functions seems to be: 1) input array 2) function specific argument(s) if applicable 3) axis 4) out (if applicable) 5) optional function specific argument(s) if applicable I think it would be nice to change the signature of compress() to bring it in line with the other functions, but obviously such a change would break some code without added functionality. What do people here think? From robert.kern at gmail.com Sun Aug 10 21:03:47 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 10 Aug 2008 20:03:47 -0500 Subject: [Numpy-discussion] compress() signature In-Reply-To: References: Message-ID: <3d375d730808101803i4e9ee7b3l62290a4aa47bf3f@mail.gmail.com> On Sun, Aug 10, 2008 at 19:37, Christoph T. Weidemann wrote: > The signature for numpy.compress() seems somewhat inconsistent with > other functions, because the first argument it expects is not the > array but the condition. This tends to throw me off when I use this > function, because the common order for other functions seems to be: > 1) input array > 2) function specific argument(s) if applicable > 3) axis > 4) out (if applicable) > 5) optional function specific argument(s) if applicable > > I think it would be nice to change the signature of compress() to > bring it in line with the other functions, but obviously such a change > would break some code without added functionality. What do people here > think? I don't think we will break code for this. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From millman at berkeley.edu Sun Aug 10 21:48:01 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Sun, 10 Aug 2008 18:48:01 -0700 Subject: [Numpy-discussion] ATTENTION: 1.2.0b1 tagged tomorrow Message-ID: Hello, The C-API change was committed in revision 5626: http://scipy.org/scipy/numpy/changeset/5626 Please note that this change requires recompiling extension modules. A draft of the release notes can be found here: http://scipy.org/scipy/numpy/milestone/1.2.0 Please take a look and let me know if I am missing anything. I will be tagging the 1.2.0b1 tomorrow. The beta release will be followed by a release candidate at the beginning of the SciPy conference with the final release planned for the end of the conference. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From cimrman3 at ntc.zcu.cz Mon Aug 11 05:40:42 2008 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Mon, 11 Aug 2008 11:40:42 +0200 Subject: [Numpy-discussion] unique1d returning indices In-Reply-To: <9457e7c80808080928p2e6d404ak91576a44a5234e7@mail.gmail.com> References: <48999E96.6070000@ntc.zcu.cz> <9457e7c80808080928p2e6d404ak91576a44a5234e7@mail.gmail.com> Message-ID: <48A0091A.6080400@ntc.zcu.cz> Hi St?fan, St?fan van der Walt wrote: > Hi Robert > > 2008/8/6 Robert Cimrman : >> Note also that the order of outputs has changed (previously unique1d() >> returned (i, b) for return_index=True). > > Does this not constitute an API change? It does. Are there many users of unique1d( a, return_index=True ) out there? The new way is consistent with matlab, and IMHO more logical - the additional return values are appended to the returned tuple rather than prepended. It was my mistake to introduce the previous order (do not recall the motivation anymore :]). r. From Rapp at mrt.uka.de Mon Aug 11 06:17:17 2008 From: Rapp at mrt.uka.de (Holger Rapp) Date: Mon, 11 Aug 2008 12:17:17 +0200 Subject: [Numpy-discussion] SSEPlus + Framewave Message-ID: <624C2171-D06D-4CE0-8812-6C0A66BA9791@mrt.uka.de> Hi everyone, I have a question concerning performance of numpy. I'm using it for heavy image processing tasks and often need more speed than the stock numpy delivers. Especially in numeric calculations with big arrays (in my current case shape = (8,4,1200,1600), 'float'). So I often rely on self written special modules which do use the IPP (Intel Performance Primitives) to speed up simple tasks like dividing a big array by a number. I realize that it was not really feasible to support a proprietary library like the IPP in a beautifully crafted Open Source Project, but quite recently, AMD came up with two very interesting projects (SSEPlus and Framewave, links provided below) which are more or less a direct response to intels IPP. And the best: They are OpenSource (under a Apache license, afaik). My question is now: Is it intended/Is there interest to get this performance gain into numpy? Are their any political restrictions (license/project identity)? Is there already work underway? I for one would consider helping in a effort like that, because it would probably safe me time in the long run. (Sidenote: I'm aware that this optimization would only help INTEL/AMD boxes, but hardware acceleration is so common these days that it is a shame NOT to use it in a numbercrunching library. Implementing such a library on one architecture might make it easier to implement something similar on others too with other libs. Maybe we see CUDA support in numpy in the future....) Here are the links: http://sseplus.sourceforge.net/ http://framewave.sourceforge.net/ Greetings, Holger From matthieu.brucher at gmail.com Mon Aug 11 08:03:26 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 11 Aug 2008 14:03:26 +0200 Subject: [Numpy-discussion] SSEPlus + Framewave In-Reply-To: <624C2171-D06D-4CE0-8812-6C0A66BA9791@mrt.uka.de> References: <624C2171-D06D-4CE0-8812-6C0A66BA9791@mrt.uka.de> Message-ID: Hi, > Maybe we see CUDA > support in numpy in the future....) I think it always depends on people willing to code ;) For CUDA, the first step toward using it would be a complete BLAS support (as there are BLAS implementation over CUDA, and it will surely be the case for AMD as well in the future). A second step would be using IPP and such libraries when possible, but if we can have BLAS, it would be a great first step. Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher From ndbecker2 at gmail.com Mon Aug 11 08:16:37 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 11 Aug 2008 08:16:37 -0400 Subject: [Numpy-discussion] SSEPlus + Framewave References: <624C2171-D06D-4CE0-8812-6C0A66BA9791@mrt.uka.de> Message-ID: Matthieu Brucher wrote: > Hi, > >> Maybe we see CUDA >> support in numpy in the future....) > > I think it always depends on people willing to code ;) > For CUDA, the first step toward using it would be a complete BLAS > support (as there are BLAS implementation over CUDA, and it will > surely be the case for AMD as well in the future). A second step would > be using IPP and such libraries when possible, but if we can have > BLAS, it would be a great first step. > > Matthieu Doesn't seem very mature. Won't even build on Fedora F9 (gcc-4.3). From cournape at gmail.com Mon Aug 11 08:17:40 2008 From: cournape at gmail.com (David Cournapeau) Date: Mon, 11 Aug 2008 07:17:40 -0500 Subject: [Numpy-discussion] SSEPlus + Framewave In-Reply-To: <624C2171-D06D-4CE0-8812-6C0A66BA9791@mrt.uka.de> References: <624C2171-D06D-4CE0-8812-6C0A66BA9791@mrt.uka.de> Message-ID: <5b8d13220808110517t242d7ff7j91369a25c2bdf0db@mail.gmail.com> On Mon, Aug 11, 2008 at 5:17 AM, Holger Rapp wrote: > I realize that it was not really feasible to support a proprietary > library like the IPP in a beautifully crafted Open Source Project, but > quite recently, AMD came up with two very interesting projects > (SSEPlus and Framewave, links provided below) which are more or less a > direct response to intels IPP. And the best: They are OpenSource > (under a Apache license, afaik). My question is now: Is it intended/Is > there interest to get this performance gain into numpy? Are their any > political restrictions (license/project identity)? Is there already > work underway? > Hi Holger, Thank you for those information. I would say that in general, we are really interested in this kind of things in numpy/scipy (sse optimization, etc...), but as always, man power is short. So to answer your question quickly: - is there any interest ? Definitely - Are there "political" restrictions ? There is the restriction that any contribution to numpy/scipy has to be BSD. AFAIK, Apache license falls into BSD camp. Otherwise, what you are proposing is not against any policy I am aware of (implicit or explicit). > I for one would consider helping in a effort like that, because it > would probably safe me time in the long run. Great. > (Sidenote: I'm aware that this optimization would only help INTEL/AMD > boxes, but hardware acceleration is so common these days that it is a > shame NOT to use it in a numbercrunching library. This is not a problem. x86 is pervasive, and is there to stay; I think most numpy/scipy users fall into the x86 users category, so it definitely makes sense to do it. Reasonable x86-only optimization will be accepted (as long as it does not break the other architectures, of course). I will take a deeper look at those libraries, but my own opinion (which I don't claim to represent any consensus of numpy developers at large) is that for this kind of tasks to be effective and "pervasive", we need some code infrastructure which is not there yet in numpy. In particular, I would like to separate the purely computational part of numpy from the "boilerplate", so that from an architectural POV, any optimization would be done in the computational part only. There are also build/deployment issues with SSE, but using an external library (which hopefully takes care of it) would alleviate most problems in that category. But if you have code which works now, and uses those two libraries, provide a patch on numpy trac (http://projects.scipy.org/scipy/numpy) cheers, David From matthieu.brucher at gmail.com Mon Aug 11 09:44:08 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 11 Aug 2008 15:44:08 +0200 Subject: [Numpy-discussion] SSEPlus + Framewave In-Reply-To: <5b8d13220808110517t242d7ff7j91369a25c2bdf0db@mail.gmail.com> References: <624C2171-D06D-4CE0-8812-6C0A66BA9791@mrt.uka.de> <5b8d13220808110517t242d7ff7j91369a25c2bdf0db@mail.gmail.com> Message-ID: > I will take a deeper look at those libraries, but my own opinion > (which I don't claim to represent any consensus of numpy developers at > large) is that for this kind of tasks to be effective and "pervasive", > we need some code infrastructure which is not there yet in numpy. In > particular, I would like to separate the purely computational part of > numpy from the "boilerplate", so that from an architectural POV, any > optimization would be done in the computational part only. There are > also build/deployment issues with SSE, but using an external library > (which hopefully takes care of it) would alleviate most problems in > that category. Hi David, I recall you propsoed a plugin framework for numpy. Where are you on that matter ? Still waiting for numscons ? Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher From stefan at sun.ac.za Mon Aug 11 10:54:50 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 11 Aug 2008 09:54:50 -0500 Subject: [Numpy-discussion] unique1d returning indices In-Reply-To: <48A0091A.6080400@ntc.zcu.cz> References: <48999E96.6070000@ntc.zcu.cz> <9457e7c80808080928p2e6d404ak91576a44a5234e7@mail.gmail.com> <48A0091A.6080400@ntc.zcu.cz> Message-ID: <9457e7c80808110754p2238ef3o221e3a34f6fb9f7d@mail.gmail.com> 2008/8/11 Robert Cimrman : >>> Note also that the order of outputs has changed (previously unique1d() >>> returned (i, b) for return_index=True). >> >> Does this not constitute an API change? > > It does. Are there many users of unique1d( a, return_index=True ) out there? > > The new way is consistent with matlab, and IMHO more logical - the > additional return values are appended to the returned tuple rather than > prepended. It was my mistake to introduce the previous order (do not > recall the motivation anymore :]). The new way is definitely more consistent -- I think one could almost call its current behaviour a bug. Unless there are objections, let's follow the standard route: put a deprecation warning in place for one release and remove it in the next. St?fan From markbak at gmail.com Mon Aug 11 11:02:11 2008 From: markbak at gmail.com (mark) Date: Mon, 11 Aug 2008 08:02:11 -0700 (PDT) Subject: [Numpy-discussion] can I install superpack numpy 1.1.1 over an egg install of 1.0.4 egg? Message-ID: <66c0329c-aea9-4120-91a2-3c63d7e2cb8b@f63g2000hsf.googlegroups.com> Hello - I downloaded the latest Enthought distribution, where everything is an egg. I needed to upgrade to the new numpy 1.1.1, but eggs don't seem available on sourceforge. Can I use the numpy 1.1.1 super pack installer (for windows) to install over the already available egg of 1.0.4? Any reason why there are no numpy eggs? I thought eggs are the future, Thanks, Mark From markbak at gmail.com Mon Aug 11 11:11:47 2008 From: markbak at gmail.com (mark) Date: Mon, 11 Aug 2008 08:11:47 -0700 (PDT) Subject: [Numpy-discussion] are fortran extensions numpy version dependent? Message-ID: <5b2833d0-0f5e-4201-97ab-f0b78d836feb@l64g2000hse.googlegroups.com> Hello list - Easy question: do fortran extension created with numpy 1.0.4 also work with numpy 1.1.1? And vice versa? Just wondering before I start updating my version (and maybe I am just too worried, but those fortran extesions are still magic to me), Mark From cimrman3 at ntc.zcu.cz Mon Aug 11 11:40:14 2008 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Mon, 11 Aug 2008 17:40:14 +0200 Subject: [Numpy-discussion] unique1d returning indices In-Reply-To: <9457e7c80808110754p2238ef3o221e3a34f6fb9f7d@mail.gmail.com> References: <48999E96.6070000@ntc.zcu.cz> <9457e7c80808080928p2e6d404ak91576a44a5234e7@mail.gmail.com> <48A0091A.6080400@ntc.zcu.cz> <9457e7c80808110754p2238ef3o221e3a34f6fb9f7d@mail.gmail.com> Message-ID: <48A05D5E.6080903@ntc.zcu.cz> St?fan van der Walt wrote: > 2008/8/11 Robert Cimrman : >>>> Note also that the order of outputs has changed (previously unique1d() >>>> returned (i, b) for return_index=True). >>> Does this not constitute an API change? >> It does. Are there many users of unique1d( a, return_index=True ) out there? >> >> The new way is consistent with matlab, and IMHO more logical - the >> additional return values are appended to the returned tuple rather than >> prepended. It was my mistake to introduce the previous order (do not >> recall the motivation anymore :]). > > The new way is definitely more consistent -- I think one could almost > call its current behaviour a bug. Unless there are objections, let's > follow the standard route: put a deprecation warning in place for one > release and remove it in the next. Yeah, that's why I think not many people used the extra return anyway. I will do as you say unless somebody steps in. r. From oliphant at enthought.com Mon Aug 11 12:13:10 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 11 Aug 2008 11:13:10 -0500 Subject: [Numpy-discussion] are fortran extensions numpy version dependent? In-Reply-To: <5b2833d0-0f5e-4201-97ab-f0b78d836feb@l64g2000hse.googlegroups.com> References: <5b2833d0-0f5e-4201-97ab-f0b78d836feb@l64g2000hse.googlegroups.com> Message-ID: <48A06516.2050602@enthought.com> mark wrote: > Hello list - > > Easy question: do fortran extension created with numpy 1.0.4 also work > with numpy 1.1.1? And vice versa? > Yes. 1.2 will have a small change requiring a new build (and maybe a change to your code if you used the details of PyArray_Descr). -Travis From stefan at sun.ac.za Mon Aug 11 15:11:38 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 11 Aug 2008 14:11:38 -0500 Subject: [Numpy-discussion] Please volunteer a Mac OS X buildbot slave Message-ID: <9457e7c80808111211v545b3c7awa6e51ce65ccfb11c@mail.gmail.com> Hi all, Due to hardware failure, we recently lost another of our build slaves. OSX is one of our primary deployment platforms, so I'd like to replace this one. If you have a mac available with good network connectivity, please consider volunteering it as a buildslave (setup should not take more than 15 minutes of your time). Thanks, St?fan From theller at ctypes.org Mon Aug 11 15:43:34 2008 From: theller at ctypes.org (Thomas Heller) Date: Mon, 11 Aug 2008 21:43:34 +0200 Subject: [Numpy-discussion] Please volunteer a Mac OS X buildbot slave In-Reply-To: <9457e7c80808111211v545b3c7awa6e51ce65ccfb11c@mail.gmail.com> References: <9457e7c80808111211v545b3c7awa6e51ce65ccfb11c@mail.gmail.com> Message-ID: St?fan van der Walt schrieb: > Hi all, > > Due to hardware failure, we recently lost another of our build slaves. > OSX is one of our primary deployment platforms, so I'd like to > replace this one. If you have a mac available with good network > connectivity, please consider volunteering it as a buildslave (setup > should not take more than 15 minutes of your time). > > Thanks, > St?fan I can offer a x86 running 10.5, this is the machine which also runs a Python buildbot, named x86 osx.5: http://www.python.org/dev/buildbot/all/ It has the same network connection as the windows XP x86_64 machine. -- Thanks, Thomas From tom.duck at dal.ca Mon Aug 11 17:02:42 2008 From: tom.duck at dal.ca (Thomas J. Duck) Date: Mon, 11 Aug 2008 18:02:42 -0300 Subject: [Numpy-discussion] min() of array containing NaN Message-ID: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> Determining the minimum value of an array that contains NaN produces a surprising result: >>> x = numpy.array([0,1,2,numpy.nan,4,5,6]) >>> x.min() 4.0 I expected 0.0. Is this the intended behaviour or a bug? I am using numpy 1.1.1. Thanks, Tom -- Thomas J. Duck Associate Professor, Department of Physics and Atmospheric Science, Dalhousie University, Halifax, Nova Scotia, Canada, B3H 3J5. Tel: (902)494-1456 | Fax: (902)494-5191 | Lab: (902)494-3813 Web: http://aolab.phys.dal.ca/ From oliphant at enthought.com Mon Aug 11 17:39:00 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 11 Aug 2008 16:39:00 -0500 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> References: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> Message-ID: <48A0B174.9060809@enthought.com> Thomas J. Duck wrote: > Determining the minimum value of an array that contains NaN produces > a surprising result: > > >>> x = numpy.array([0,1,2,numpy.nan,4,5,6]) > >>> x.min() > 4.0 > > I expected 0.0. Is this the intended behaviour or a bug? I am using > numpy 1.1.1. > NAN's don't play well with comparisons because comparison with them is undefined. See numpy.nanmin -Travis From silva at lma.cnrs-mrs.fr Mon Aug 11 17:46:50 2008 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Mon, 11 Aug 2008 23:46:50 +0200 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> References: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> Message-ID: <1218491210.2887.2.camel@localhost.localdomain> Try nanmin function : $ python Python 2.5.2 (r252:60911, Jul 31 2008, 07:39:27) [GCC 4.3.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.__version__ '1.1.0' >>> x = numpy.array([0,1,2,numpy.nan, 4, 5, 6]) >>> x.min() 4.0 >>> numpy.nanmin(x) 0.0 There lacks some nanmin method for array instances, i.e. one can not execute >>> x.nanmin() Traceback (most recent call last): File "", line 1, in AttributeError: 'numpy.ndarray' object has no attribute 'nanmin' -- Fabrice Silva LMA UPR CNRS 7051 - ?quipe S2M From doutriaux1 at llnl.gov Mon Aug 11 18:05:01 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Mon, 11 Aug 2008 15:05:01 -0700 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: <1218491210.2887.2.camel@localhost.localdomain> References: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> <1218491210.2887.2.camel@localhost.localdomain> Message-ID: <48A0B78D.3000109@llnl.gov> Seems to me like min should automagically call nanmin if it spots any nan no ? C. Fabrice Silva wrote: > Try nanmin function : > > $ python > Python 2.5.2 (r252:60911, Jul 31 2008, 07:39:27) > [GCC 4.3.1] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > >>> numpy.__version__ > '1.1.0' > >>> x = numpy.array([0,1,2,numpy.nan, 4, 5, 6]) > >>> x.min() > 4.0 > >>> numpy.nanmin(x) > 0.0 > > There lacks some nanmin method for array instances, i.e. one can not execute > >>> x.nanmin() > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'numpy.ndarray' object has no attribute 'nanmin' > > From stefan at sun.ac.za Mon Aug 11 18:49:06 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 11 Aug 2008 17:49:06 -0500 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: <48A0B78D.3000109@llnl.gov> References: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> <1218491210.2887.2.camel@localhost.localdomain> <48A0B78D.3000109@llnl.gov> Message-ID: <9457e7c80808111549w588e515y7b17a37d1c7b2cc4@mail.gmail.com> 2008/8/11 Charles Doutriaux : > Seems to me like min should automagically call nanmin if it spots any > nan no ? Nanmin is quite a bit slower: In [2]: x = np.random.random((5000)) In [3]: timeit np.min(x) 10000 loops, best of 3: 24.8 ?s per loop In [4]: timeit np.nanmin(x) 10000 loops, best of 3: 136 ?s per loop So, I'm not sure if that will happen. One option is to use `nanmin` by default, and to provide `min` for people who need the speed. The fact that results with nan's are almost always unexpected is certainly a valid concern. Cheers St?fan From pgmdevlist at gmail.com Mon Aug 11 19:41:39 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 11 Aug 2008 19:41:39 -0400 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: <9457e7c80808111549w588e515y7b17a37d1c7b2cc4@mail.gmail.com> References: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> <48A0B78D.3000109@llnl.gov> <9457e7c80808111549w588e515y7b17a37d1c7b2cc4@mail.gmail.com> Message-ID: <200808111941.39483.pgmdevlist@gmail.com> *cough* MaskedArrays anyone ? *cough* The ideal would be for min/max to output a NaN when there's a NaN somewhere. That way, you'd know that there's a potential pb in your data, and that you should use the nanfunctions or masked arrays. is there a page on the wiki for that matter ? It seems to show up regularly... On Monday 11 August 2008 18:49:06 St?fan van der Walt wrote: > 2008/8/11 Charles Doutriaux : > > Seems to me like min should automagically call nanmin if it spots any > > nan no ? > > Nanmin is quite a bit slower: > > In [2]: x = np.random.random((5000)) > > In [3]: timeit np.min(x) > 10000 loops, best of 3: 24.8 ?s per loop > > In [4]: timeit np.nanmin(x) > 10000 loops, best of 3: 136 ?s per loop > > So, I'm not sure if that will happen. One option is to use `nanmin` > by default, and to provide `min` for people who need the speed. The > fact that results with nan's are almost always unexpected is certainly > a valid concern. > > Cheers > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From bsouthey at gmail.com Mon Aug 11 21:34:52 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 11 Aug 2008 20:34:52 -0500 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: <200808111941.39483.pgmdevlist@gmail.com> References: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> <48A0B78D.3000109@llnl.gov> <9457e7c80808111549w588e515y7b17a37d1c7b2cc4@mail.gmail.com> <200808111941.39483.pgmdevlist@gmail.com> Message-ID: I agree with using Masked arrays... Actually this could be viewed as a bug because it ignores the entries to the left of the NaN. >>> numpy.__version__ '1.1.1.dev5559' >>> x = numpy.array([0,1,2,numpy.nan, 4, 5, 6]) >>> numpy.min(x) 4.0 >>> x = numpy.array([numpy.nan,0,1,2, 4, 5, 6]) >>> x.min() 0.0 >>> x = numpy.array([0,1,2, 4, 5, 6, numpy.nan]) >>> x.min() -1.#IND As has been recently said on this list (as per Stefan's post) NaN's and infinity have a higher computational cost. I am not sure the relative cost of using say isnan first as a check or having a NaN flag stored as part of the ndarray class. As per Travis's post, technically it should return NaN. But I don't agree with Charles that it should automatically call nanmin because nanmin treats NaNs as zero, positive infinity as a really large positive number and negative infinity as a very small or negative number. This may not be want the user wants. An alternative is to change the signature to include a flag to include or exclude NaN and infinity which would also remove the need for nanmin and friends. Bruce On Mon, Aug 11, 2008 at 6:41 PM, Pierre GM wrote: > *cough* MaskedArrays anyone ? *cough* > > The ideal would be for min/max to output a NaN when there's a NaN somewhere. > That way, you'd know that there's a potential pb in your data, and that you > should use the nanfunctions or masked arrays. > > is there a page on the wiki for that matter ? It seems to show up regularly... > > On Monday 11 August 2008 18:49:06 St?fan van der Walt wrote: >> 2008/8/11 Charles Doutriaux : >> > Seems to me like min should automagically call nanmin if it spots any >> > nan no ? >> >> Nanmin is quite a bit slower: >> >> In [2]: x = np.random.random((5000)) >> >> In [3]: timeit np.min(x) >> 10000 loops, best of 3: 24.8 ?s per loop >> >> In [4]: timeit np.nanmin(x) >> 10000 loops, best of 3: 136 ?s per loop >> >> So, I'm not sure if that will happen. One option is to use `nanmin` >> by default, and to provide `min` for people who need the speed. The >> fact that results with nan's are almost always unexpected is certainly >> a valid concern. >> >> Cheers >> St?fan >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From Chris.Barker at noaa.gov Tue Aug 12 01:05:26 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 11 Aug 2008 22:05:26 -0700 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: References: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> <48A0B78D.3000109@llnl.gov> <9457e7c80808111549w588e515y7b17a37d1c7b2cc4@mail.gmail.com> <200808111941.39483.pgmdevlist@gmail.com> Message-ID: <48A11A16.8040009@noaa.gov> Bruce Southey wrote: > Actually this could be viewed as a bug because it ignores the entries > to the left of the NaN. well, it's not a bug because the result if there is a NaN is undefined. However, it sure could trip people up. If you know there is likely to be a NaN in there, then you could use nanmin() or masked arrays. The problem comes up when you have no idea there might be a NaN in there, in which case you get a bogus answer -- this is very bad. Is there an error state that will trigger an error or warning in these situations? Otherwise, I'd have to say that the default should be to test for NaN's, and either raise an error or return NaN. If that really does slow things down too much, there could be a flag that lets you turn it off. This situation now makes me very nervous. > because > nanmin treats NaNs as zero, positive infinity as a really large > positive number and negative infinity as a very small or negative > number. Actually, I think it skips over NaN -- otherwise, the min would always be zero if there where a Nan, and "a very small negative number" if there were a -inf. I have to say that one of the things I always liked about Matlab was it's handling of NaN, inf, and -inf. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception From dalke at dalkescientific.com Tue Aug 12 01:46:58 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Tue, 12 Aug 2008 07:46:58 +0200 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: <48A11A16.8040009@noaa.gov> References: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> <48A0B78D.3000109@llnl.gov> <9457e7c80808111549w588e515y7b17a37d1c7b2cc4@mail.gmail.com> <200808111941.39483.pgmdevlist@gmail.com> <48A11A16.8040009@noaa.gov> Message-ID: <5AE7D1A6-69A2-4688-AA5B-2A9519E897A8@dalkescientific.com> On Aug 12, 2008, at 7:05 AM, Christopher Barker wrote: > Actually, I think it skips over NaN -- otherwise, the min would always > be zero if there where a Nan, and "a very small negative number" if > there were a -inf. Here's the implementation, from lib/function_base.py def nanmin(a, axis=None): """Find the minimium over the given axis, ignoring NaNs. """ y = array(a,subok=True) if not issubclass(y.dtype.type, _nx.integer): y[isnan(a)] = _nx.inf return y.min(axis) This is buggy for the case of a list containing only NaNs. >>> import numpy as np >>> np.NAN nan >>> np.min([np.NAN]) nan >>> np.nanmin([np.NAN]) inf >>> Andrew dalke at dalkescientific.com From stefan at sun.ac.za Tue Aug 12 02:16:23 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 12 Aug 2008 01:16:23 -0500 Subject: [Numpy-discussion] Ticket #750 (Swig interface - Bug in FORTRAN-ordering support) Message-ID: <9457e7c80808112316x2f52be3cj5a81b5a13fae2ec3@mail.gmail.com> Hi all, I've given ticket #750 a shot. Since I was not previously familiar with SWIG, I would appreciate it if those of you who know it could take a look at the proposed patch: http://codereview.appspot.com/2890 Thanks for your time, St?fan From stefan at sun.ac.za Tue Aug 12 02:56:57 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 12 Aug 2008 01:56:57 -0500 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: <5AE7D1A6-69A2-4688-AA5B-2A9519E897A8@dalkescientific.com> References: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> <48A0B78D.3000109@llnl.gov> <9457e7c80808111549w588e515y7b17a37d1c7b2cc4@mail.gmail.com> <200808111941.39483.pgmdevlist@gmail.com> <48A11A16.8040009@noaa.gov> <5AE7D1A6-69A2-4688-AA5B-2A9519E897A8@dalkescientific.com> Message-ID: <9457e7c80808112356t7baf603bj7eca70f3daad4972@mail.gmail.com> Hi Andrew 2008/8/12 Andrew Dalke : > This is buggy for the case of a list containing only NaNs. > > >>> import numpy as np > >>> np.NAN > nan > >>> np.min([np.NAN]) > nan > >>> np.nanmin([np.NAN]) > inf > >>> Thanks for the report. This should be fixed in r5630. Regards St?fan From jh at physics.ucf.edu Tue Aug 12 03:22:36 2008 From: jh at physics.ucf.edu (Joe Harrington) Date: Tue, 12 Aug 2008 03:22:36 -0400 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: (numpy-discussion-request@scipy.org) References: Message-ID: Masked arrays are a bit clunky for something as simple and standard as NaN handling. They also have the inverse of the standard truth sense, at least as used in my field. 1 (or True) usually means the item is allowed, not denied, so that you can multiply the mask by the data to zero all bad values, add and subtract masks in sensible ways and get what's expected, etc. For example, in the "stacked, masked mean" image processing algorithm, you sum the data along an axis, sum the masks along that axis, and divide the results to get the mean image without bad pixels. This is much more accurate than taking a median, and admits to error analysis, which the median does not (easily). While the regular behavior is "just a ~ away", as Stefan pointed out to me once, that's not acceptable if the image cube is large and memory or speed are at issue, and it's also very prone to bugs if you're negating everything all the time. Further, with ma you have to convert to using an entirely different and redundant set of routines instead of having the very standard handling of NaNs found in our competitor programs, such as IDL. The issue of not having an in-place method in ma was also raised earlier. I'll add the difficulty of converting code if a standard thing like NaN handling has to be simulated in multiple calls. So, I endorse extending min() and all other statistical routines to handle NaNs, possibly with a switch to turn it on if a suitably fast algorithm cannot be found (which is competitor IDL's solution). Certainly without a switch the default behavior should be to return NaN, not to return some random value, if a NaN is present. Otherwise the user may never know a NaN is present, and therefore has to check every use for NaNs. That constand manual NaN checking is slower and more error-prone than any numerical speed advantage. So to sum, proposed for statistical routnes: if NaN is not present, return value if NaN is present, return NaN if NaN is present and nan=True, return value ignoring all NaNs OR: if NaN is not present, return value if NaN is present, return value ignoring all NaNs if NaN is present and nan=True, return NaN I'd prefer the latter. IDL does the former and it is a pain to do /nan all the time. However, the latter might trip up the unwary, whereas the former never does. This would apply at least to: min max sum prod mean median std and possibly many others. --jh-- From millman at berkeley.edu Tue Aug 12 03:40:56 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 12 Aug 2008 00:40:56 -0700 Subject: [Numpy-discussion] ATTENTION: 1.2.0b1 tagged tomorrow In-Reply-To: References: Message-ID: I am pushing back the beta release for another day. Stefan is looking into one thing I would like to get into the beta. Cheers, On Sun, Aug 10, 2008 at 6:48 PM, Jarrod Millman wrote: > Hello, > > The C-API change was committed in revision 5626: > http://scipy.org/scipy/numpy/changeset/5626 > Please note that this change requires recompiling extension modules. > > A draft of the release notes can be found here: > http://scipy.org/scipy/numpy/milestone/1.2.0 > Please take a look and let me know if I am missing anything. > > I will be tagging the 1.2.0b1 tomorrow. The beta release will be > followed by a release candidate at the beginning of the SciPy > conference with the final release planned for the end of the > conference. > > Thanks, > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From peridot.faceted at gmail.com Tue Aug 12 03:54:32 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Tue, 12 Aug 2008 03:54:32 -0400 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: <9457e7c80808112356t7baf603bj7eca70f3daad4972@mail.gmail.com> References: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> <48A0B78D.3000109@llnl.gov> <9457e7c80808111549w588e515y7b17a37d1c7b2cc4@mail.gmail.com> <200808111941.39483.pgmdevlist@gmail.com> <48A11A16.8040009@noaa.gov> <5AE7D1A6-69A2-4688-AA5B-2A9519E897A8@dalkescientific.com> <9457e7c80808112356t7baf603bj7eca70f3daad4972@mail.gmail.com> Message-ID: 2008/8/12 St?fan van der Walt : > Hi Andrew > > 2008/8/12 Andrew Dalke : >> This is buggy for the case of a list containing only NaNs. >> >> >>> import numpy as np >> >>> np.NAN >> nan >> >>> np.min([np.NAN]) >> nan >> >>> np.nanmin([np.NAN]) >> inf >> >>> > > Thanks for the report. This should be fixed in r5630. Er, is this actually a bug? I would instead consider the fact that np.min([]) raises an exception a bug of sorts - the identity of min is inf. Really nanmin of an array containing only nans should be the same as an empty array; both should be infinity. I guess this is a problem for types that don't have an infinity (returning maxint is a poor substitute), but what is the correct behaviour here? Anne From peridot.faceted at gmail.com Tue Aug 12 04:06:33 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Tue, 12 Aug 2008 04:06:33 -0400 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: References: Message-ID: 2008/8/12 Joe Harrington : > So, I endorse extending min() and all other statistical routines to > handle NaNs, possibly with a switch to turn it on if a suitably fast > algorithm cannot be found (which is competitor IDL's solution). > Certainly without a switch the default behavior should be to return > NaN, not to return some random value, if a NaN is present. Otherwise > the user may never know a NaN is present, and therefore has to check > every use for NaNs. That constand manual NaN checking is slower and > more error-prone than any numerical speed advantage. > > So to sum, proposed for statistical routnes: > if NaN is not present, return value > if NaN is present, return NaN > if NaN is present and nan=True, return value ignoring all NaNs > > OR: > if NaN is not present, return value > if NaN is present, return value ignoring all NaNs > if NaN is present and nan=True, return NaN > > I'd prefer the latter. IDL does the former and it is a pain to do > /nan all the time. However, the latter might trip up the unwary, > whereas the former never does. > > This would apply at least to: > min > max > sum > prod > mean > median > std > and possibly many others. For almost all of these the current behaviour is to propagate NaNs arithmetically. For example, the sum of anything with a NaN is NaN. I think this is perfectly sufficient, given how easy it is to strip out NaNs if that's what you want. The issue that started this thread (and the many other threads that have come up as users stub their toes on this behaviour) is that min (and other functions based on comparisons) do not propagate NaNs. If you do np.amin(A) and A contains NaNs, you can't count on getting a NaN back, unlike np.mean or np.std. the fact that you get some random value not the minimum just adds insult to injury. (It is probably also true that the value you get back depends on how the array is stored in memory.) It really isn't very hard to replace np.sum(A) with np.sum(A[~isnan(A)]) if you want to ignore NaNs instead of propagating them. So I don't feel a need for special code in sum() that treats NaN as 0. I would be content if the comparison-based functions propagated NaNs appropriately. If you did decide it was essential to make versions of the functions that removed NaNs, it would get you most of the way there to add an optional keyword argument to ufuncs' reduce method that skipped NaNs. Anne From barrywark at gmail.com Tue Aug 12 06:36:40 2008 From: barrywark at gmail.com (Barry Wark) Date: Tue, 12 Aug 2008 06:36:40 -0400 Subject: [Numpy-discussion] Please volunteer a Mac OS X buildbot slave In-Reply-To: <9457e7c80808111211v545b3c7awa6e51ce65ccfb11c@mail.gmail.com> References: <9457e7c80808111211v545b3c7awa6e51ce65ccfb11c@mail.gmail.com> Message-ID: Stefan, I'm sorry I dropped the ball on this one. I didn't have time to get things working again before I left town for a month and, obviously, there it sat. Again, sorry. Barry On Mon, Aug 11, 2008 at 3:11 PM, St?fan van der Walt wrote: > Hi all, > > Due to hardware failure, we recently lost another of our build slaves. > OSX is one of our primary deployment platforms, so I'd like to > replace this one. If you have a mac available with good network > connectivity, please consider volunteering it as a buildslave (setup > should not take more than 15 minutes of your time). > > Thanks, > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From bioinformed at gmail.com Tue Aug 12 08:42:54 2008 From: bioinformed at gmail.com (Kevin Jacobs ) Date: Tue, 12 Aug 2008 08:42:54 -0400 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: <5AE7D1A6-69A2-4688-AA5B-2A9519E897A8@dalkescientific.com> References: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> <48A0B78D.3000109@llnl.gov> <9457e7c80808111549w588e515y7b17a37d1c7b2cc4@mail.gmail.com> <200808111941.39483.pgmdevlist@gmail.com> <48A11A16.8040009@noaa.gov> <5AE7D1A6-69A2-4688-AA5B-2A9519E897A8@dalkescientific.com> Message-ID: <2e1434c10808120542g122efce2wcdad95f25c235f56@mail.gmail.com> On Tue, Aug 12, 2008 at 1:46 AM, Andrew Dalke wrote: > Here's the implementation, from lib/function_base.py > > def nanmin(a, axis=None): > """Find the minimium over the given axis, ignoring NaNs. > """ > y = array(a,subok=True) > if not issubclass(y.dtype.type, _nx.integer): > y[isnan(a)] = _nx.inf > return y.min(axis) > > No wonder nanmin is slow. A C implementation would run at virtually the same speed as min. If there is interest, I'll be happy to code C versions. A better solution would be to just support NaNs and Inf in the generic code. -Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Tue Aug 12 10:08:53 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 12 Aug 2008 09:08:53 -0500 Subject: [Numpy-discussion] Please volunteer a Mac OS X buildbot slave In-Reply-To: References: <9457e7c80808111211v545b3c7awa6e51ce65ccfb11c@mail.gmail.com> Message-ID: <9457e7c80808120708u6c618843nec72c20860a302e9@mail.gmail.com> 2008/8/12 Barry Wark : > Stefan, > > I'm sorry I dropped the ball on this one. I didn't have time to get > things working again before I left town for a month and, obviously, > there it sat. Again, sorry. No worries, Barry. That machine was troublesome, and I didn't want to bother you every week. Thomas Heller's machine should do the job, if we can get it set up. Cheers St?fan From bsouthey at gmail.com Tue Aug 12 10:18:22 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 12 Aug 2008 09:18:22 -0500 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: References: Message-ID: <48A19BAE.2010100@gmail.com> Anne Archibald wrote: > 2008/8/12 Joe Harrington : > > >> So, I endorse extending min() and all other statistical routines to >> handle NaNs, possibly with a switch to turn it on if a suitably fast >> algorithm cannot be found (which is competitor IDL's solution). >> Certainly without a switch the default behavior should be to return >> NaN, not to return some random value, if a NaN is present. Otherwise >> the user may never know a NaN is present, and therefore has to check >> every use for NaNs. That constand manual NaN checking is slower and >> more error-prone than any numerical speed advantage. >> >> So to sum, proposed for statistical routnes: >> if NaN is not present, return value >> if NaN is present, return NaN >> if NaN is present and nan=True, return value ignoring all NaNs >> >> OR: >> if NaN is not present, return value >> if NaN is present, return value ignoring all NaNs >> if NaN is present and nan=True, return NaN >> >> I'd prefer the latter. IDL does the former and it is a pain to do >> /nan all the time. However, the latter might trip up the unwary, >> whereas the former never does. >> >> This would apply at least to: >> min >> max >> sum >> prod >> mean >> median >> std >> and possibly many others. >> > > For almost all of these the current behaviour is to propagate NaNs > arithmetically. For example, the sum of anything with a NaN is NaN. I > think this is perfectly sufficient, given how easy it is to strip out > NaNs if that's what you want. The issue that started this thread (and > the many other threads that have come up as users stub their toes on > this behaviour) is that min (and other functions based on comparisons) > do not propagate NaNs. If you do np.amin(A) and A contains NaNs, you > can't count on getting a NaN back, unlike np.mean or np.std. the fact > that you get some random value not the minimum just adds insult to > injury. (It is probably also true that the value you get back depends > on how the array is stored in memory.) > > It really isn't very hard to replace > np.sum(A) > with > np.sum(A[~isnan(A)]) > if you want to ignore NaNs instead of propagating them. So I don't > feel a need for special code in sum() that treats NaN as 0. I would be > content if the comparison-based functions propagated NaNs > appropriately. > > If you did decide it was essential to make versions of the functions > that removed NaNs, it would get you most of the way there to add an > optional keyword argument to ufuncs' reduce method that skipped NaNs. > > Anne > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > Actually you probably need to use isfinite because of NumPy's support for IEEE 754 (means NaN is different from infinity). Also, doesn't this also require an additional temporary copy of A? The problem I have with this is that you must always know in advance that NaNs or infinities are present and assumes you want to ignore them. Alternatively something simple like a new function. Bruce import numpy as np def minnan(x, axis=None, out=None, hasnan=False): if hasnan: return np.nanmin(x,axis) elif np.isfinite(x).all(): return np.min(x,axis, out) else: return np.nan # actually should be something else here x = np.array([1,2,np.nan,4,5,6]) y = np.array([1,2,3,4,5,6]) print 'NumPy Min:', np.min(x) print 'NumPy NaNMin:', np.nanmin(x) print 'NumPy MinNaN:', minnan(x) print 'NumPy MinNaN T:', minnan(x, hasnan=True) print 'NumPy Min:', np.min(y) print 'NumPy NaNMin:', np.nanmin(y) print 'NumPy MinNan:', minnan(y) print 'NumPy MinNaN T:', minnan(y, hasnan=True) From Shawn.Gong at drdc-rddc.gc.ca Tue Aug 12 10:37:51 2008 From: Shawn.Gong at drdc-rddc.gc.ca (Gong, Shawn (Contractor)) Date: Tue, 12 Aug 2008 10:37:51 -0400 Subject: [Numpy-discussion] non-linear array manipulation Message-ID: hi list, The following array manipulation takes long time because I can't find ways to do in row/column, and have to do cell by cell. Would you check to see if there is a nicer/faster way for this non-linear operation? for i in range(rows): for j in range(columns): a[i][j] = math.sqrt( max(0.0, a[i][j]*a[i][j] - b[j]*c[j]) ) thanks, Shawn -------------- next part -------------- An HTML attachment was scrubbed... URL: From tom.duck at dal.ca Tue Aug 12 11:02:47 2008 From: tom.duck at dal.ca (Thomas J. Duck) Date: Tue, 12 Aug 2008 12:02:47 -0300 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: References: Message-ID: <0B43126D-D7E1-4F67-A935-C3BAD1F80F6B@dal.ca> Christopher Barker wrote: > well, it's not a bug because the result if there is a NaN is > undefined. > However, it sure could trip people up. If you know there is likely > to be > a NaN in there, then you could use nanmin() or masked arrays. The > problem comes up when you have no idea there might be a NaN in > there, in > which case you get a bogus answer -- this is very bad. This is exactly what happened to me. I was getting crazy results when contour plotting with matplotlib, although the pcolor plots looked fine. In particular, the colorscale had incorrect limits. This led me to check the min() and max() values in my array, which were clearly wrong as illustrated by the pcolor plot. Further investigation revealed unexpected NaNs in my array. > Is there an error state that will trigger an error or warning in these > situations? Otherwise, I'd have to say that the default should be to > test for NaN's, and either raise an error or return NaN. If that > really > does slow things down too much, there could be a flag that lets you > turn > it off. It is quite often the case that NaNs are unexpected, so it would be helpful to raise an Exception. Thanks for all of the helpful discussion on this issue. -- Thomas J. Duck Associate Professor, Department of Physics and Atmospheric Science, Dalhousie University, Halifax, Nova Scotia, Canada, B3H 3J5. Tel: (902)494-1456 | Fax: (902)494-5191 | Lab: (902)494-3813 Web: http://aolab.phys.dal.ca/ From nadavh at visionsense.com Tue Aug 12 11:22:05 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Tue, 12 Aug 2008 18:22:05 +0300 Subject: [Numpy-discussion] non-linear array manipulation References: Message-ID: <710F2847B0018641891D9A216027636029C222@ex3.envision.co.il> from numpy import * a = sqrt(maximum(0, a**2-repeat(b*c, columns).reshape(columns, rows))) Nadav. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Gong, Shawn (Contractor) ????: ? 12-??????-08 17:37 ??: Discussion of Numerical Python ????: [Numpy-discussion] non-linear array manipulation hi list, The following array manipulation takes long time because I can't find ways to do in row/column, and have to do cell by cell. Would you check to see if there is a nicer/faster way for this non-linear operation? for i in range(rows): for j in range(columns): a[i][j] = math.sqrt( max(0.0, a[i][j]*a[i][j] - b[j]*c[j]) ) thanks, Shawn -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3097 bytes Desc: not available URL: From nadavh at visionsense.com Tue Aug 12 11:57:32 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Tue, 12 Aug 2008 18:57:32 +0300 Subject: [Numpy-discussion] non-linear array manipulation References: Message-ID: <710F2847B0018641891D9A216027636029C223@ex3.envision.co.il> from numpy import * a = sqrt(maximum(0, a**2-repeat(b*c, columns).reshape(rows, columns))) Nadav -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Gong, Shawn (Contractor) ????: ? 12-??????-08 17:37 ??: Discussion of Numerical Python ????: [Numpy-discussion] non-linear array manipulation hi list, The following array manipulation takes long time because I can't find ways to do in row/column, and have to do cell by cell. Would you check to see if there is a nicer/faster way for this non-linear operation? for i in range(rows): for j in range(columns): a[i][j] = math.sqrt( max(0.0, a[i][j]*a[i][j] - b[j]*c[j]) ) thanks, Shawn -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3097 bytes Desc: not available URL: From wfspotz at sandia.gov Tue Aug 12 12:19:06 2008 From: wfspotz at sandia.gov (Bill Spotz) Date: Tue, 12 Aug 2008 12:19:06 -0400 Subject: [Numpy-discussion] Ticket #750 (Swig interface - Bug in FORTRAN-ordering support) In-Reply-To: <9457e7c80808112316x2f52be3cj5a81b5a13fae2ec3@mail.gmail.com> References: <9457e7c80808112316x2f52be3cj5a81b5a13fae2ec3@mail.gmail.com> Message-ID: <546CF54C-E19E-4847-B8E8-621EB338805F@sandia.gov> Stefan, The changes look reasonable, and if the old tests still pass (I'm assuming your new ones will), then I'm happy. There is one "style" issue: the (int* is_new_object) argument always appears last in those helper functions where it is used. Thanks On Aug 12, 2008, at 2:16 AM, St?fan van der Walt wrote: > Hi all, > > I've given ticket #750 a shot. Since I was not previously familiar > with SWIG, I would appreciate it if those of you who know it could > take a look at the proposed patch: > > http://codereview.appspot.com/2890 > > Thanks for your time, > St?fan ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From Shawn.Gong at drdc-rddc.gc.ca Tue Aug 12 12:41:02 2008 From: Shawn.Gong at drdc-rddc.gc.ca (Gong, Shawn (Contractor)) Date: Tue, 12 Aug 2008 12:41:02 -0400 Subject: [Numpy-discussion] non-linear array manipulation In-Reply-To: <710F2847B0018641891D9A216027636029C223@ex3.envision.co.il> References: <710F2847B0018641891D9A216027636029C223@ex3.envision.co.il> Message-ID: thank you, Nadav Shawn -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Nadav Horesh Sent: Tuesday, August 12, 2008 11:58 AM To: Discussion of Numerical Python Subject: RE: [Numpy-discussion] non-linear array manipulation from numpy import * a = sqrt(maximum(0, a**2-repeat(b*c, columns).reshape(rows, columns))) Nadav -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Gong, Shawn (Contractor) ????: ? 12-??????-08 17:37 ??: Discussion of Numerical Python ????: [Numpy-discussion] non-linear array manipulation hi list, The following array manipulation takes long time because I can't find ways to do in row/column, and have to do cell by cell. Would you check to see if there is a nicer/faster way for this non-linear operation? for i in range(rows): for j in range(columns): a[i][j] = math.sqrt( max(0.0, a[i][j]*a[i][j] - b[j]*c[j]) ) thanks, Shawn From doutriaux1 at llnl.gov Tue Aug 12 14:14:09 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Tue, 12 Aug 2008 11:14:09 -0700 Subject: [Numpy-discussion] masked_equal not commutative? Message-ID: <48A1D2F1.5000104@llnl.gov> Hi I'm using 1.1.1 and found that numpy.ma.masked_equal is not commutative! I would expect it to be in this case. Or raise an error for uncompatible shape in the first case, no ? >>> a = numpy.ma.arange(100) >>> a.shape=(10,10) >>> b=numpy.ma.masked_equal(1,a) >>> b Traceback (most recent call last): File "", line 1, in File "/lgm/cdat/latest/lib/python2.5/site-packages/numpy/ma/core.py", line 1691, in __repr__ 'data': str(self), File "/lgm/cdat/latest/lib/python2.5/site-packages/numpy/ma/core.py", line 1665, in __str__ res[m] = f IndexError: 0-d arrays can't be indexed. >>> b=numpy.ma.masked_equal(a,1) >>> b masked_array(data = [[0 -- 2 3 4 5 6 7 8 9] [10 11 12 13 14 15 16 17 18 19] [20 21 22 23 24 25 26 27 28 29] [30 31 32 33 34 35 36 37 38 39] [40 41 42 43 44 45 46 47 48 49] [50 51 52 53 54 55 56 57 58 59] [60 61 62 63 64 65 66 67 68 69] [70 71 72 73 74 75 76 77 78 79] [80 81 82 83 84 85 86 87 88 89] [90 91 92 93 94 95 96 97 98 99]], mask = [[False True False False False False False False False False] [False False False False False False False False False False] [False False False False False False False False False False] [False False False False False False False False False False] [False False False False False False False False False False] [False False False False False False False False False False] [False False False False False False False False False False] [False False False False False False False False False False] [False False False False False False False False False False] [False False False False False False False False False False]], fill_value=999999) From doutriaux1 at llnl.gov Tue Aug 12 14:18:12 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Tue, 12 Aug 2008 11:18:12 -0700 Subject: [Numpy-discussion] masked_equal not commutative Message-ID: <48A1D3E4.9060308@llnl.gov> As always as i clicked send i realized my error it is indeed not commutative and that makes sense but i'm not sure the case: numpy.ma.masked_equal(1,a) should have worked, since we don't really know how to do this comparison, the only thing that could make sense would be commutation but i thinks it's probably dangerous to assume what the user wants to do. C. From cournape at gmail.com Tue Aug 12 15:25:45 2008 From: cournape at gmail.com (David Cournapeau) Date: Tue, 12 Aug 2008 14:25:45 -0500 Subject: [Numpy-discussion] SSEPlus + Framewave In-Reply-To: References: <624C2171-D06D-4CE0-8812-6C0A66BA9791@mrt.uka.de> <5b8d13220808110517t242d7ff7j91369a25c2bdf0db@mail.gmail.com> Message-ID: <5b8d13220808121225m1ecd77dat83133f231ad8adf2@mail.gmail.com> On Mon, Aug 11, 2008 at 8:44 AM, Matthieu Brucher wrote: > > Hi David, > > I recall you propsoed a plugin framework for numpy. Where are you on > that matter ? Still waiting for numscons ? The problem is not so much the build part, but the clear separation I was talking about. My experience with ATLAS convinced me the only way to make sse work reliably is to detect the CPU arch at runtime; compiling binaries incompatible on different arch is just not scalable and confuse users. Technically, to make this work, you need a clear different in the codebase between the code which calls the C functions changed at runtime, and the ones which are called by python. Hopefully, I will have more to say within the end of the week cheers, David From peridot.faceted at gmail.com Tue Aug 12 16:28:00 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Tue, 12 Aug 2008 16:28:00 -0400 Subject: [Numpy-discussion] non-linear array manipulation In-Reply-To: <710F2847B0018641891D9A216027636029C222@ex3.envision.co.il> References: <710F2847B0018641891D9A216027636029C222@ex3.envision.co.il> Message-ID: 2008/8/12 Nadav Horesh : > from numpy import * > > a = sqrt(maximum(0, a**2-repeat(b*c, columns).reshape(columns, rows))) better: import numpy as np a = np.sqrt(np.maximum(0.0, a**2-(b*c)[np.newaxis,:])) This doesn't have to make a temporary the size of a; instead broadcasting rules are used to treat b*c as if it were the same shape as a. Also, "from numpy import *" is a good way to shoot yourself in the foot. At least "from numpy import sqrt, maximum, repeat" would have been safer. Anne From markbak at gmail.com Tue Aug 12 17:21:41 2008 From: markbak at gmail.com (mark) Date: Tue, 12 Aug 2008 14:21:41 -0700 (PDT) Subject: [Numpy-discussion] has f2py changed from 1.0.4 to 1.1.1 ???? Message-ID: <42124a41-ca5b-4f17-bd07-f7b0943e6ec9@f63g2000hsf.googlegroups.com> Hello all - I have been banging my head against the wall. f2py worked fine on windows under 1.0.4 (Python 2.5) After upgrading to numpy 1.1.1, f2py keeps complaining it cannot find msvccompiler, yet it is still in the same place (under distutils) as it was under 1.0.4. Did something change? I read something about some f2py branch that was ended in 1.1.1. Any suggestions on how to solve this? Thanks, Mark From markbak at gmail.com Tue Aug 12 17:21:19 2008 From: markbak at gmail.com (mark) Date: Tue, 12 Aug 2008 14:21:19 -0700 (PDT) Subject: [Numpy-discussion] has f2py changed from 1.0.4 to 1.1.1 ???? Message-ID: Hello all - I have been banging my head against the wall. f2py worked fine on windows under 1.0.4 (Python 2.5) After upgrading to numpy 1.1.1, f2py keeps complaining it cannot find msvccompiler, yet it is still in the same place (under distutils) as it was under 1.0.4. Did something change? I read something about some f2py branch that was ended in 1.1.1. Any suggestions on how to solve this? Thanks, Mark From markbak at gmail.com Tue Aug 12 17:21:19 2008 From: markbak at gmail.com (mark) Date: Tue, 12 Aug 2008 14:21:19 -0700 (PDT) Subject: [Numpy-discussion] has f2py changed from 1.0.4 to 1.1.1 ???? Message-ID: Hello all - I have been banging my head against the wall. f2py worked fine on windows under 1.0.4 (Python 2.5) After upgrading to numpy 1.1.1, f2py keeps complaining it cannot find msvccompiler, yet it is still in the same place (under distutils) as it was under 1.0.4. Did something change? I read something about some f2py branch that was ended in 1.1.1. Any suggestions on how to solve this? Thanks, Mark From jh at physics.ucf.edu Tue Aug 12 17:22:08 2008 From: jh at physics.ucf.edu (Joe Harrington) Date: Tue, 12 Aug 2008 17:22:08 -0400 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: (numpy-discussion-request@scipy.org) References: Message-ID: > It really isn't very hard to replace > np.sum(A) > with > np.sum(A[~isnan(A)]) > if you want to ignore NaNs instead of propagating them. So I don't > feel a need for special code in sum() that treats NaN as 0. That's all well and good, until you want to set the axis= keyword. Then you're stuck with looping. As doing stats for each pixel column in a stack of astronomical images with bad pixels and cosmic-ray hits is one of the most common actions in astronomical data analysis, this is an issue for a significant number of current and future users. >>> a=np.arange(9, dtype=float) >>> a.shape=(3,3) >>> a[1,1]=np.nan >>> a array([[ 0. , 1. , 2. ], [ 3. , nan, 5. ], [ 6. , 7. , 8. ]]) >>> np.sum(a) nan >>> np.sum(a[~np.isnan(a)]) 32.0 Good, but... >>> np.sum(a[~np.isnan(a)], axis=1) Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.5/site-packages/numpy/core/fromnumeric.py", line 634, in sum return sum(axis, dtype, out) ValueError: axis(=1) out of bounds Uh-oh... >>> np.sum(a[~np.isnan(a)], axis=0) 32.0 Worse: wrong answer but not an exception, since >>> a[~np.isnan(a)] array([ 0., 1., 2., 3., 5., 6., 7., 8.]) has the undesired side effect of irreversibly flattening the array. --jh-- From cournape at gmail.com Tue Aug 12 17:53:10 2008 From: cournape at gmail.com (David Cournapeau) Date: Tue, 12 Aug 2008 16:53:10 -0500 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: <0B43126D-D7E1-4F67-A935-C3BAD1F80F6B@dal.ca> References: <0B43126D-D7E1-4F67-A935-C3BAD1F80F6B@dal.ca> Message-ID: <5b8d13220808121453o11e8afa1w339ced50d934cea1@mail.gmail.com> On Tue, Aug 12, 2008 at 10:02 AM, Thomas J. Duck wrote: > > It is quite often the case that NaNs are unexpected, so it > would be helpful to raise an Exception. from numpy import seterr seterr(all = 'warn') Do emit a warning when encountering any kind of floating point error. You can even use raise instead of warn, in which case you will get an exception. cheers, David From matthieu.brucher at gmail.com Tue Aug 12 18:01:03 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 13 Aug 2008 00:01:03 +0200 Subject: [Numpy-discussion] SSEPlus + Framewave In-Reply-To: <5b8d13220808121225m1ecd77dat83133f231ad8adf2@mail.gmail.com> References: <624C2171-D06D-4CE0-8812-6C0A66BA9791@mrt.uka.de> <5b8d13220808110517t242d7ff7j91369a25c2bdf0db@mail.gmail.com> <5b8d13220808121225m1ecd77dat83133f231ad8adf2@mail.gmail.com> Message-ID: 2008/8/12 David Cournapeau : > On Mon, Aug 11, 2008 at 8:44 AM, Matthieu Brucher > wrote: >> >> Hi David, >> >> I recall you propsoed a plugin framework for numpy. Where are you on >> that matter ? Still waiting for numscons ? > > The problem is not so much the build part, but the clear separation I > was talking about. My experience with ATLAS convinced me the only way > to make sse work reliably is to detect the CPU arch at runtime; > compiling binaries incompatible on different arch is just not scalable > and confuse users. > > Technically, to make this work, you need a clear different in the > codebase between the code which calls the C functions changed at > runtime, and the ones which are called by python. Hopefully, I will > have more to say within the end of the week I'm looking forward to hearing from you ;) Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher From dalke at dalkescientific.com Tue Aug 12 19:13:35 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Wed, 13 Aug 2008 01:13:35 +0200 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: References: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> <48A0B78D.3000109@llnl.gov> <9457e7c80808111549w588e515y7b17a37d1c7b2cc4@mail.gmail.com> <200808111941.39483.pgmdevlist@gmail.com> <48A11A16.8040009@noaa.gov> <5AE7D1A6-69A2-4688-AA5B-2A9519E897A8@dalkescientific.com> <9457e7c80808112356t7baf603bj7eca70f3daad4972@mail.gmail.com> Message-ID: <8D667EED-619B-44F7-A214-6CC926C71FE4@dalkescientific.com> On Aug 12, 2008, at 9:54 AM, Anne Archibald wrote: > Er, is this actually a bug? I would instead consider the fact that > np.min([]) raises an exception a bug of sorts - the identity of min is > inf. That'll break consistency with the normal 'max' function in Python. > Really nanmin of an array containing only nans should be the same > as an empty array; both should be infinity. One thing I expect is that if min(x) exists then there is some i where x[i] "is" min(x) . Returning +inf for min([NaN]) breaks that. However, my expectation doesn't hold true for Python. If I use Python's object identity test 'is' then object identity is lost in numpy.min, although it is preserved under Python's min: >>> import numpy as np >>> x = [200, 300] >>> np.min(x) 200 >>> np.min(x) is x[0] False >>> min(x) is x[0] True >>> and if I use '==' for equality testing then my expectation will fail if isnan(x[i]) because then x[i] != x[i]. >>> import numpy as np >>> np.nan nan >>> np.nan == np.nan False So when I say "is" I means "acts the same as except for in some strange corner cases". Or to put it another way, it should be possible to implement a hypothetical 'argnanmin' just like there is an 'argmin' which complements 'min'. > I guess this is a problem for types that don't have an infinity > (returning maxint is a poor substitute), but what is the correct > behaviour here? "Doctor, doctor it hurts when I do this." "Well, don't do that." Raise an exception. Refuse the temptation to guess. Force the user to handle this case as appropriate. Personally, I expect that if my array 'x' has a NaN then min(x) must be a NaN. Andrew dalke at dalkescientific.com From charlesr.harris at gmail.com Tue Aug 12 20:28:22 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 12 Aug 2008 18:28:22 -0600 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: <8D667EED-619B-44F7-A214-6CC926C71FE4@dalkescientific.com> References: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> <48A0B78D.3000109@llnl.gov> <9457e7c80808111549w588e515y7b17a37d1c7b2cc4@mail.gmail.com> <200808111941.39483.pgmdevlist@gmail.com> <48A11A16.8040009@noaa.gov> <5AE7D1A6-69A2-4688-AA5B-2A9519E897A8@dalkescientific.com> <9457e7c80808112356t7baf603bj7eca70f3daad4972@mail.gmail.com> <8D667EED-619B-44F7-A214-6CC926C71FE4@dalkescientific.com> Message-ID: On Tue, Aug 12, 2008 at 5:13 PM, Andrew Dalke wrote: > On Aug 12, 2008, at 9:54 AM, Anne Archibald wrote: > > Er, is this actually a bug? I would instead consider the fact that > > np.min([]) raises an exception a bug of sorts - the identity of min is > > inf. > > > Personally, I expect that if my array 'x' has a NaN then > min(x) must be a NaN. > I suppose you could use min(a,b) = (abs(a - b) + a + b)/2 which would have that effect. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Aug 12 20:31:48 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 12 Aug 2008 18:31:48 -0600 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: References: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> <9457e7c80808111549w588e515y7b17a37d1c7b2cc4@mail.gmail.com> <200808111941.39483.pgmdevlist@gmail.com> <48A11A16.8040009@noaa.gov> <5AE7D1A6-69A2-4688-AA5B-2A9519E897A8@dalkescientific.com> <9457e7c80808112356t7baf603bj7eca70f3daad4972@mail.gmail.com> <8D667EED-619B-44F7-A214-6CC926C71FE4@dalkescientific.com> Message-ID: On Tue, Aug 12, 2008 at 6:28 PM, Charles R Harris wrote: > > > On Tue, Aug 12, 2008 at 5:13 PM, Andrew Dalke wrote: > >> On Aug 12, 2008, at 9:54 AM, Anne Archibald wrote: >> > Er, is this actually a bug? I would instead consider the fact that >> > np.min([]) raises an exception a bug of sorts - the identity of min is >> > inf. >> > > > >> >> Personally, I expect that if my array 'x' has a NaN then >> min(x) must be a NaN. >> > > I suppose you could use > > min(a,b) = (abs(a - b) + a + b)/2 > > which would have that effect. > Hmm, that is for the max, min would be (a + b - |a - b|)/2 > > Chuck > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Aug 12 21:12:06 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 12 Aug 2008 19:12:06 -0600 Subject: [Numpy-discussion] has f2py changed from 1.0.4 to 1.1.1 ???? In-Reply-To: <42124a41-ca5b-4f17-bd07-f7b0943e6ec9@f63g2000hsf.googlegroups.com> References: <42124a41-ca5b-4f17-bd07-f7b0943e6ec9@f63g2000hsf.googlegroups.com> Message-ID: On Tue, Aug 12, 2008 at 3:21 PM, mark wrote: > Hello all - > > I have been banging my head against the wall. > > f2py worked fine on windows under 1.0.4 (Python 2.5) > > After upgrading to numpy 1.1.1, f2py keeps complaining it cannot find > msvccompiler, yet it is still in the same place (under distutils) as > it was under 1.0.4. > > Did something change? I read something about some f2py branch that was > ended in 1.1.1. > > Any suggestions on how to solve this? > 1.1.1 should be the same as the trunk f2py wise, so we need to figure this out. Robert might have an idea as to what is going on. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Aug 12 21:59:11 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 12 Aug 2008 20:59:11 -0500 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: References: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> <9457e7c80808111549w588e515y7b17a37d1c7b2cc4@mail.gmail.com> <200808111941.39483.pgmdevlist@gmail.com> <48A11A16.8040009@noaa.gov> <5AE7D1A6-69A2-4688-AA5B-2A9519E897A8@dalkescientific.com> <9457e7c80808112356t7baf603bj7eca70f3daad4972@mail.gmail.com> <8D667EED-619B-44F7-A214-6CC926C71FE4@dalkescientific.com> Message-ID: <3d375d730808121859g5934bd3s83ac3556c93b8681@mail.gmail.com> On Tue, Aug 12, 2008 at 19:28, Charles R Harris wrote: > > > On Tue, Aug 12, 2008 at 5:13 PM, Andrew Dalke > wrote: >> >> On Aug 12, 2008, at 9:54 AM, Anne Archibald wrote: >> > Er, is this actually a bug? I would instead consider the fact that >> > np.min([]) raises an exception a bug of sorts - the identity of min is >> > inf. > > > >> >> Personally, I expect that if my array 'x' has a NaN then >> min(x) must be a NaN. > > I suppose you could use > > min(a,b) = (abs(a - b) + a + b)/2 > > which would have that effect. Or we could implement the inner loop of the minimum ufunc to return NaN if there is a NaN. Currently it just compares the two values (which causes the unpredictable results since having a NaN on either side of the < is always False). I would be amenable to that provided that the C isnan() call does not cause too much slowdown in the normal case. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Tue Aug 12 22:25:45 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 12 Aug 2008 21:25:45 -0500 Subject: [Numpy-discussion] has f2py changed from 1.0.4 to 1.1.1 ???? In-Reply-To: References: <42124a41-ca5b-4f17-bd07-f7b0943e6ec9@f63g2000hsf.googlegroups.com> Message-ID: <3d375d730808121925i20665a13sf3a89bd42b2d1935@mail.gmail.com> On Tue, Aug 12, 2008 at 20:12, Charles R Harris wrote: > > > On Tue, Aug 12, 2008 at 3:21 PM, mark wrote: >> >> Hello all - >> >> I have been banging my head against the wall. >> >> f2py worked fine on windows under 1.0.4 (Python 2.5) >> >> After upgrading to numpy 1.1.1, f2py keeps complaining it cannot find >> msvccompiler, yet it is still in the same place (under distutils) as >> it was under 1.0.4. >> >> Did something change? I read something about some f2py branch that was >> ended in 1.1.1. >> >> Any suggestions on how to solve this? > > 1.1.1 should be the same as the trunk f2py wise, so we need to figure this > out. Robert might have an idea as to what is going on. Without more information (like a traceback, the Fortran compiler that the OP is trying to use, and the --fcompiler, etc. options used), not really. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ctw at cogsci.info Tue Aug 12 23:00:01 2008 From: ctw at cogsci.info (Christoph T. Weidemann) Date: Tue, 12 Aug 2008 23:00:01 -0400 Subject: [Numpy-discussion] ravel() in ma/core.py Message-ID: Hi! I'm working with a subclass of ndarray and ran into an issue that seems to be caused by a line in numpy/ma/core.py The offending line is no. 1837 in version 1.1.0 or 2053 in the latest SVN version (revision 5635): r = ndarray.ravel(self._data).view(type(self)) The problem is that my subclass of ndarray has its own implementation of the ravel method that takes care of some important things before calling the ndarray.ravel class. Calling ndarray.ravel directly on this data structure doesn't allow for the necessary house-keeping to take place and my data structure complains by throwing an exception.I believe this issue would be solved if the above line instead read: r = self.ravel() or alternatively r = np.ravel(self) or something along those lines -- can anybody here please take a look and see if these (or something else) would be a viable alternative that would cause the subclass's ravel function to be called instead of ndarray.ravel()? Thanks, CTW PS: I ran into this issue while trying to plot my data with the imshow function in matplotlib. If vmin and/or vmax are not specified, a masked array version of the input array is generated in matplotlib/colors.py in this line: val = ma.asarray(value).astype(np.float) Then ma.minimum() and ma.maximum() are called and eventually the line highlighted above gets executed. I don't know if/why this conversion to masked arrays is necessary in the matplotlib code, but it seems pretty clear that the source of the problem is the application of the ndarray.ravel() method rather than that for the subclass in the line highlighted above. From Christopher.E.Kees at usace.army.mil Wed Aug 13 00:27:17 2008 From: Christopher.E.Kees at usace.army.mil (Kees, Christopher E) Date: Tue, 12 Aug 2008 23:27:17 -0500 Subject: [Numpy-discussion] Mac OSX 4-way universal Message-ID: Hi, I'm trying to install numpy 1.1.1 into a 4-way universal build of python 2.6. It builds fine, but I get an error in install saying I can't install when cross-compiling. Does anybody know how to get around this? Chris From robert.kern at gmail.com Wed Aug 13 01:40:44 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 13 Aug 2008 00:40:44 -0500 Subject: [Numpy-discussion] Mac OSX 4-way universal In-Reply-To: References: Message-ID: <3d375d730808122240l7cb5847fpa557ab587c99bfab@mail.gmail.com> On Tue, Aug 12, 2008 at 23:27, Kees, Christopher E wrote: > Hi, I'm trying to install numpy 1.1.1 into a 4-way universal build of python > 2.6. It builds fine, but I get an error in install saying I can't install > when cross-compiling. Does anybody know how to get around this? When reporting error messages, please state exactly what you did (e.g. copy-and-pasting the command lines you executed) and the exact error messages (by copy-and-pasting the traceback). Thanks. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From markbak at gmail.com Wed Aug 13 04:15:40 2008 From: markbak at gmail.com (mark) Date: Wed, 13 Aug 2008 01:15:40 -0700 (PDT) Subject: [Numpy-discussion] has f2py changed from 1.0.4 to 1.1.1 ???? In-Reply-To: <3d375d730808121925i20665a13sf3a89bd42b2d1935@mail.gmail.com> References: <42124a41-ca5b-4f17-bd07-f7b0943e6ec9@f63g2000hsf.googlegroups.com> <3d375d730808121925i20665a13sf3a89bd42b2d1935@mail.gmail.com> Message-ID: <1662a6e0-53b6-48a5-8042-1783161fc914@59g2000hsb.googlegroups.com> Charles and Robert - I am running Python 2.5.2 on Windows XP. I use the Enthought distribution, which comes with numpy 1.0.4. f2py works fine using mingw32 and gfortran. My command line is: python c:\python25\scripts\f2py.py -c -m besselaes --compiler=mingw32 --fcompiler=gnu95 besselaes.f90 Next I upgraded to numpy 1.1.1 (most importantly, because matplotlib 0.98 requires numpy 1.1.x). I run exactly the same command line and get the error: Ignoring "Python was built with Visual Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py." (one should fix me in fcompiler/compaq.py) running build running scons No module named msvccompiler in numpy.distutils; trying from distutils I don't understand, because distutils still contains the files msvccompiler.py and .pyc. I removed numpy 1.1.1 and reinstalled the numpy1.0.4 egg, but that didn't help. I had to reinstall the entire Enthought distribution to get f2py to work again. Any help is greatly appreciated. Mark On Aug 13, 4:25?am, "Robert Kern" wrote: > On Tue, Aug 12, 2008 at 20:12, Charles R Harris > > > > wrote: > > > On Tue, Aug 12, 2008 at 3:21 PM, mark wrote: > > >> Hello all - > > >> I have been banging my head against the wall. > > >> f2py worked fine on windows under 1.0.4 (Python 2.5) > > >> After upgrading to numpy 1.1.1, f2py keeps complaining it cannot find > >> msvccompiler, yet it is still in the same place (under distutils) as > >> it was under 1.0.4. > > >> Did something change? I read something about some f2py branch that was > >> ended in 1.1.1. > > >> Any suggestions on how to solve this? > > > 1.1.1 should be the ?same as the trunk f2py wise, so we need to figure this > > out. Robert might have an idea as to what is going on. > > Without more information (like a traceback, the Fortran compiler that > the OP is trying to use, and the --fcompiler, etc. options used), not > really. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ?-- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion From markbak at gmail.com Wed Aug 13 04:25:45 2008 From: markbak at gmail.com (mark) Date: Wed, 13 Aug 2008 01:25:45 -0700 (PDT) Subject: [Numpy-discussion] has f2py changed from 1.0.4 to 1.1.1 ???? In-Reply-To: <3d375d730808121925i20665a13sf3a89bd42b2d1935@mail.gmail.com> References: <42124a41-ca5b-4f17-bd07-f7b0943e6ec9@f63g2000hsf.googlegroups.com> <3d375d730808121925i20665a13sf3a89bd42b2d1935@mail.gmail.com> Message-ID: <9245111a-aebc-4bce-ada9-6f2615230af6@i76g2000hsf.googlegroups.com> Robert, Charles - I am running Python 2.5.2 on Windows XP. I use the Enthought distribution, which comes with numpy 1.0.4. f2py works fine using mingw32 and gfortran. My command line is: python c:\python25\scripts\f2py.py -c -m besselaes --compiler=mingw32 --fcompiler=gnu95 besselaes.f90 Next I upgraded to numpy 1.1.1 (most importantly, because matplotlib 0.98 requires numpy 1.1.x). I run exactly the same command line and get the error: Ignoring "Python was built with Visual Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py." (one should fix me in fcompiler/compaq.py) running build running scons No module named msvccompiler in numpy.distutils; trying from distutils I don't understand, because distutils still contains the files msvccompiler.py and .pyc. I removed numpy 1.1.1 and reinstalled the numpy1.0.4 egg, but that didn't help. I had to reinstall the entire Enthought distribution to get f2py to work again. Any help is greatly appreciated. Mark From Rapp at mrt.uka.de Wed Aug 13 04:28:54 2008 From: Rapp at mrt.uka.de (Holger Rapp) Date: Wed, 13 Aug 2008 10:28:54 +0200 Subject: [Numpy-discussion] SSEPlus + Framewave In-Reply-To: <5b8d13220808121225m1ecd77dat83133f231ad8adf2@mail.gmail.com> References: <624C2171-D06D-4CE0-8812-6C0A66BA9791@mrt.uka.de> <5b8d13220808110517t242d7ff7j91369a25c2bdf0db@mail.gmail.com> <5b8d13220808121225m1ecd77dat83133f231ad8adf2@mail.gmail.com> Message-ID: <495759AE-1DE6-48A6-9968-477C2EC885DA@mrt.uka.de> Hello David, > The problem is not so much the build part, but the clear separation I > was talking about. My experience with ATLAS convinced me the only way > to make sse work reliably is to detect the CPU arch at runtime; > compiling binaries incompatible on different arch is just not scalable > and confuse users. What do you mean by compiling incompatible? It is my understanding that (for example) Framewave (but also IPP) come in different flavors (32bit, 64bit) which of course must be compiled in at compile time. But which CPU is available and which features it delivers is indeed done at runtime (framewave: fwStaticInit()), the choice of how to implement things with which assembler code is then up to the framewave library. I do not consider it a good idea to write a own dispatcher library into numpy to choose which opcode to use. Or do it get you completly wrong? Is your intention to make a plugin architecture in the sense of: copy some directory with libs and config in your site-packages and then your multiplications are much faster? I would consider such a framework a bit overengineered, since speedy calculations are a nice feature for every numpy user. Greetings, Holger From matthieu.brucher at gmail.com Wed Aug 13 06:03:23 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 13 Aug 2008 12:03:23 +0200 Subject: [Numpy-discussion] SSEPlus + Framewave In-Reply-To: <495759AE-1DE6-48A6-9968-477C2EC885DA@mrt.uka.de> References: <624C2171-D06D-4CE0-8812-6C0A66BA9791@mrt.uka.de> <5b8d13220808110517t242d7ff7j91369a25c2bdf0db@mail.gmail.com> <5b8d13220808121225m1ecd77dat83133f231ad8adf2@mail.gmail.com> <495759AE-1DE6-48A6-9968-477C2EC885DA@mrt.uka.de> Message-ID: > What do you mean by compiling incompatible? It is my understanding > that (for example) Framewave (but also IPP) come in different flavors > (32bit, 64bit) which of course must be compiled in at compile time. > But which CPU is available and which features it delivers is indeed > done at runtime (framewave: fwStaticInit()), the choice of how to > implement things with which assembler code is then up to the framewave > library. > I do not consider it a good idea to write a own dispatcher library > into numpy to choose which opcode to use. This would enable the detection at runtime of the fast libraries and the choice of the best one. It will indeed be an additional dispatcher, but not every numerical library has such a dispatcher, and the dispatcher only works for one library and not for the whole Python module. > Or do it get you completly wrong? Is your intention to make a plugin > architecture in the sense of: copy some directory with libs and config > in your site-packages and then your multiplications are much faster? I > would consider such a framework a bit overengineered, since speedy > calculations are a nice feature for every numpy user. You have to detect the presence of the library. If there are no such framework, you have to compile the module again (and for scipy under Windows, it is not simple). So developing a good plugin framework will help people code fast libraries plugins if it is possible, there will be only one module and not one for Intel/AMD/Atlas/CUDA/... thus less bugs, ... Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher From Christopher.E.Kees at usace.army.mil Wed Aug 13 09:00:45 2008 From: Christopher.E.Kees at usace.army.mil (Chris Kees) Date: Wed, 13 Aug 2008 08:00:45 -0500 Subject: [Numpy-discussion] Mac OSX 4-way universal In-Reply-To: <3d375d730808122240l7cb5847fpa557ab587c99bfab@mail.gmail.com> Message-ID: %python setup.py build %sudo python setup.py install ...snipped adding 'build/scripts.macosx-10.5-universal-2.6/f2py64-32' to scripts error: Can't install when cross-compiling On 8/13/08 12:40 AM, "Robert Kern" wrote: > On Tue, Aug 12, 2008 at 23:27, Kees, Christopher E > wrote: >> Hi, I'm trying to install numpy 1.1.1 into a 4-way universal build of python >> 2.6. It builds fine, but I get an error in install saying I can't install >> when cross-compiling. Does anybody know how to get around this? > > When reporting error messages, please state exactly what you did (e.g. > copy-and-pasting the command lines you executed) and the exact error > messages (by copy-and-pasting the traceback). Thanks. From Rapp at mrt.uka.de Wed Aug 13 09:27:43 2008 From: Rapp at mrt.uka.de (Holger Rapp) Date: Wed, 13 Aug 2008 15:27:43 +0200 Subject: [Numpy-discussion] SSEPlus + Framewave In-Reply-To: References: <624C2171-D06D-4CE0-8812-6C0A66BA9791@mrt.uka.de> <5b8d13220808110517t242d7ff7j91369a25c2bdf0db@mail.gmail.com> <5b8d13220808121225m1ecd77dat83133f231ad8adf2@mail.gmail.com> <495759AE-1DE6-48A6-9968-477C2EC885DA@mrt.uka.de> Message-ID: <18A8863F-3E22-4F1B-ADB5-D4C3934C931A@mrt.uka.de> > >> Or do it get you completly wrong? Is your intention to make a plugin >> architecture in the sense of: copy some directory with libs and >> config >> in your site-packages and then your multiplications are much >> faster? I >> would consider such a framework a bit overengineered, since speedy >> calculations are a nice feature for every numpy user. > > You have to detect the presence of the library. If there are no such > framework, you have to compile the module again (and for scipy under > Windows, it is not simple). So developing a good plugin framework will > help people code fast libraries plugins if it is possible, there will > be only one module and not one for Intel/AMD/Atlas/CUDA/... thus less > bugs, ... Well Cuda might not be available everywhere, so a plugin architecture might provide usefull in this case (though i doubt that it would be easier than to just have some packagers responsible for offering python with this enabled to download). But Waveframe for example works readily on all I386 architecture out of the box and without any configuration. Given that there would be support for it in python, why would someone choose to disable it since it has no downsides? Why would you provide it as a plugin if you would ship it always? Same thought, different argumentation: If i have a cuda card, i probably do not want to use it for every possible calculation. For example array((2))**2 would be faster in software (or even faster with framewave support) then cuda (because cudas bottleneck is the datatransfer to the graphicscard, which only makes it usefull for huge arrays). The plugin would therefore be micromanaged or must offer some on/off toggle functionality; it would therefore not replace existing calls. Conclusion: I would vote for libraries which only speed things up to be included in the release and I for one would prefer a numpy aware cude extension module for python instead of a plugin for numpy. Greetings, Holger From cournape at gmail.com Wed Aug 13 09:51:36 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 13 Aug 2008 08:51:36 -0500 Subject: [Numpy-discussion] SSEPlus + Framewave In-Reply-To: <495759AE-1DE6-48A6-9968-477C2EC885DA@mrt.uka.de> References: <624C2171-D06D-4CE0-8812-6C0A66BA9791@mrt.uka.de> <5b8d13220808110517t242d7ff7j91369a25c2bdf0db@mail.gmail.com> <5b8d13220808121225m1ecd77dat83133f231ad8adf2@mail.gmail.com> <495759AE-1DE6-48A6-9968-477C2EC885DA@mrt.uka.de> Message-ID: <5b8d13220808130651v54d7cdcej6c2f37a9fa7b1769@mail.gmail.com> On Wed, Aug 13, 2008 at 3:28 AM, Holger Rapp wrote: > What do you mean by compiling incompatible? It is my understanding > that (for example) Framewave (but also IPP) come in different flavors > (32bit, 64bit) which of course must be compiled in at compile time. > But which CPU is available and which features it delivers is indeed > done at runtime (framewave: fwStaticInit()), the choice of how to > implement things with which assembler code is then up to the framewave > library. Yes, but not all libraries do that. Typically, Atlas doesn't. > I do not consider it a good idea to write a own dispatcher library > into numpy to choose which opcode to use. Having a clear separation between computational part and "boilerplate" has another big advantage: to make it easier to provide numpy facilities other C extensions (typically, I would like to be able to use special functions, fft, blas/lapack, etc... at the C level in some scipy extensions). It also makes testing, benchmarking easier. This is cleaner, too. And once you have this separation, having a plugin system is not that difficult. > Or do it get you completly wrong? Is your intention to make a plugin > architecture in the sense of: copy some directory with libs and config > in your site-packages and then your multiplications are much faster? Not exactly: for binaries, we would compile all architectures, and it will be transparant to the user. The plugin-system would be an internal thing, the user would not be aware of it. Like matlab does it for example. > would consider such a framework a bit overengineered, since speedy > calculations are a nice feature for every numpy user. But that's the point: having optimization for every user, without the user having to care about which numpy to install. cheers, David From pgmdevlist at gmail.com Wed Aug 13 10:19:26 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 13 Aug 2008 10:19:26 -0400 Subject: [Numpy-discussion] ravel() in ma/core.py In-Reply-To: References: Message-ID: <200808131019.26492.pgmdevlist@gmail.com> Christoph, Sorry to hear you're running into pbs w/ numpy.ma. Could you send me a stripped down version of your class so that I could perform tests and check whether calling np.ravel would work ? You could also write your own subclass of MaskedArray + your subclass and implement your own ravel. In any case, the MO would roughly stay the same: call ravel on the ._data part (viz, on a pure ndarray or your subclass), take a view and ravel the mask if any. Let me know, and sorry for the inconvenience again. P. From cimrman3 at ntc.zcu.cz Wed Aug 13 10:32:28 2008 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Wed, 13 Aug 2008 16:32:28 +0200 Subject: [Numpy-discussion] unique1d returning indices In-Reply-To: <48A05D5E.6080903@ntc.zcu.cz> References: <48999E96.6070000@ntc.zcu.cz> <9457e7c80808080928p2e6d404ak91576a44a5234e7@mail.gmail.com> <48A0091A.6080400@ntc.zcu.cz> <9457e7c80808110754p2238ef3o221e3a34f6fb9f7d@mail.gmail.com> <48A05D5E.6080903@ntc.zcu.cz> Message-ID: <48A2F07C.1000207@ntc.zcu.cz> Robert Cimrman wrote: > St?fan van der Walt wrote: >> 2008/8/11 Robert Cimrman : >>>>> Note also that the order of outputs has changed (previously unique1d() >>>>> returned (i, b) for return_index=True). >>>> Does this not constitute an API change? >>> It does. Are there many users of unique1d( a, return_index=True ) out there? >>> >>> The new way is consistent with matlab, and IMHO more logical - the >>> additional return values are appended to the returned tuple rather than >>> prepended. It was my mistake to introduce the previous order (do not >>> recall the motivation anymore :]). >> The new way is definitely more consistent -- I think one could almost >> call its current behaviour a bug. Unless there are objections, let's >> follow the standard route: put a deprecation warning in place for one >> release and remove it in the next. > > Yeah, that's why I think not many people used the extra return anyway. > I will do as you say unless somebody steps in. ... but not before August 25, as I am about to leave on holidays and have not managed to do it yet. I do not want to mess with the SVN now as I would not be able to follow it. If you think the patch is ok, and have time, then go ahead :) r. From cournape at gmail.com Wed Aug 13 11:34:29 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 13 Aug 2008 10:34:29 -0500 Subject: [Numpy-discussion] SSEPlus + Framewave In-Reply-To: <18A8863F-3E22-4F1B-ADB5-D4C3934C931A@mrt.uka.de> References: <624C2171-D06D-4CE0-8812-6C0A66BA9791@mrt.uka.de> <5b8d13220808110517t242d7ff7j91369a25c2bdf0db@mail.gmail.com> <5b8d13220808121225m1ecd77dat83133f231ad8adf2@mail.gmail.com> <495759AE-1DE6-48A6-9968-477C2EC885DA@mrt.uka.de> <18A8863F-3E22-4F1B-ADB5-D4C3934C931A@mrt.uka.de> Message-ID: <5b8d13220808130834p4ab75d9di3f4563dbd0b2e4c0@mail.gmail.com> On Wed, Aug 13, 2008 at 8:27 AM, Holger Rapp wrote: >> >> You have to detect the presence of the library. If there are no such >> framework, you have to compile the module again (and for scipy under >> Windows, it is not simple). So developing a good plugin framework will >> help people code fast libraries plugins if it is possible, there will >> be only one module and not one for Intel/AMD/Atlas/CUDA/... thus less >> bugs, ... > Well Cuda might not be available everywhere, so a plugin architecture > might provide usefull > in this case (though i doubt that it would be easier than to just have > some packagers responsible > for offering python with this enabled to download). But Waveframe for > example works readily on all > I386 architecture out of the box and without any configuration. Given > that there would be support for it in > python, why would someone choose to disable it since it has no > downsides? Why would you provide > it as a plugin if you would ship it always? The way I see things is that we would not ship plugins, we would ship numpy, but internally it uses plugins so that it can select things at runtime. Again, matlab does it this way: it uses (used to ?) ATLAS, but it runs on every architecture: it ships with several atlas, and loads the right one at runtime. We do depend, today, on libraries which are arch dependent. On windows, it caused constant problems because ATLAS would crash on machines without say SSE. We solved this by encompassing different full numpy in the installer, each built with different atlas, and install the right one at install time. For numpy, it is still ok because it is small, but for scipy already, this is not so nice (I build the binaries for 3 architectures right now: nothing, SSE2 and SSE3, which means windows installer is potentially 3 times bigger than a single binary). > Same thought, different argumentation: If i have a cuda card, i > probably do not want to use it for every possible calculation. For > example array((2))**2 would be faster in software (or even faster with > framewave support) then cuda (because cudas bottleneck is the > datatransfer to the graphicscard, which only makes it usefull for huge > arrays). The plugin would therefore be micromanaged or must offer some > on/off toggle functionality; it would therefore not replace existing > calls. This is a mostly orthogonal issue: whether it is in a dynamically loaded library or not internally, you will have to take care of this. cheers, David From stefan at sun.ac.za Wed Aug 13 13:23:55 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 13 Aug 2008 12:23:55 -0500 Subject: [Numpy-discussion] unique1d returning indices In-Reply-To: <48A2F07C.1000207@ntc.zcu.cz> References: <48999E96.6070000@ntc.zcu.cz> <9457e7c80808080928p2e6d404ak91576a44a5234e7@mail.gmail.com> <48A0091A.6080400@ntc.zcu.cz> <9457e7c80808110754p2238ef3o221e3a34f6fb9f7d@mail.gmail.com> <48A05D5E.6080903@ntc.zcu.cz> <48A2F07C.1000207@ntc.zcu.cz> Message-ID: <9457e7c80808131023i1dc346c1i225ebfaacf30254b@mail.gmail.com> 2008/8/13 Robert Cimrman : >> Yeah, that's why I think not many people used the extra return anyway. >> I will do as you say unless somebody steps in. > > ... but not before August 25, as I am about to leave on holidays and > have not managed to do it yet. I do not want to mess with the SVN now as > I would not be able to follow it. > > If you think the patch is ok, and have time, then go ahead :) Thanks, Robert. This has been merged in r5639. St?fan From as8ca at virginia.edu Wed Aug 13 12:52:19 2008 From: as8ca at virginia.edu (Alok Singhal) Date: Wed, 13 Aug 2008 12:52:19 -0400 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: References: <9457e7c80808111549w588e515y7b17a37d1c7b2cc4@mail.gmail.com> <200808111941.39483.pgmdevlist@gmail.com> <48A11A16.8040009@noaa.gov> <5AE7D1A6-69A2-4688-AA5B-2A9519E897A8@dalkescientific.com> <9457e7c80808112356t7baf603bj7eca70f3daad4972@mail.gmail.com> <8D667EED-619B-44F7-A214-6CC926C71FE4@dalkescientific.com> Message-ID: <20080813165219.GA14623@virginia.edu> On 12/08/08: 18:31, Charles R Harris wrote: > On Tue, Aug 12, 2008 at 6:28 PM, Charles R Harris > <[1]charlesr.harris at gmail.com> wrote: > I suppose you could use > min(a,b) = (abs(a - b) + a + b)/2 > which would have that effect. > > Hmm, that is for the max, min would be > (a + b - |a - b|)/2 This would break when there is an overflow because of addition/subtraction: def new_min(a, b): return (a + b - abs(a-b))/2 a = 1e308 b = -1e308 new_min(a, b) # returns -inf min(a, b) # returns -1e308 -- * * Alok Singhal * * * http://www.astro.virginia.edu/~as8ca/ * * From Chris.Barker at noaa.gov Wed Aug 13 13:33:12 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 13 Aug 2008 10:33:12 -0700 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: <3d375d730808121859g5934bd3s83ac3556c93b8681@mail.gmail.com> References: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> <9457e7c80808111549w588e515y7b17a37d1c7b2cc4@mail.gmail.com> <200808111941.39483.pgmdevlist@gmail.com> <48A11A16.8040009@noaa.gov> <5AE7D1A6-69A2-4688-AA5B-2A9519E897A8@dalkescientific.com> <9457e7c80808112356t7baf603bj7eca70f3daad4972@mail.gmail.com> <8D667EED-619B-44F7-A214-6CC926C71FE4@dalkescientific.com> <3d375d730808121859g5934bd3s83ac3556c93b8681@mail.gmail.com> Message-ID: <48A31AD8.9030201@noaa.gov> Robert Kern wrote: > Or we could implement the inner loop of the minimum ufunc to return > NaN if there is a NaN. Currently it just compares the two values > (which causes the unpredictable results since having a NaN on either > side of the < is always False). I would be amenable to that provided > that the C isnan() call does not cause too much slowdown in the normal > case. +1 -- this seems to be the only reasonable option. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From efiring at hawaii.edu Wed Aug 13 13:54:47 2008 From: efiring at hawaii.edu (Eric Firing) Date: Wed, 13 Aug 2008 07:54:47 -1000 Subject: [Numpy-discussion] ravel() in ma/core.py In-Reply-To: References: Message-ID: <48A31FE7.9050802@hawaii.edu> Christoph T. Weidemann wrote: > Hi! > > I'm working with a subclass of ndarray and ran into an issue that > seems to be caused by a line in numpy/ma/core.py > The offending line is no. 1837 in version 1.1.0 or 2053 in the latest > SVN version (revision 5635): > r = ndarray.ravel(self._data).view(type(self)) > > The problem is that my subclass of ndarray has its own implementation > of the ravel method that takes care of some important things before > calling the ndarray.ravel class. Calling ndarray.ravel directly on > this data structure doesn't allow for the necessary house-keeping to > take place and my data structure complains by throwing an exception.I > believe this issue would be solved if the above line instead read: > r = self.ravel() > or alternatively > r = np.ravel(self) > or something along those lines -- can anybody here please take a look > and see if these (or something else) would be a viable alternative > that would cause the subclass's ravel function to be called instead of > ndarray.ravel()? > > Thanks, > CTW > > PS: I ran into this issue while trying to plot my data with the imshow > function in matplotlib. If vmin and/or vmax are not specified, a > masked array version of the input array is generated in > matplotlib/colors.py in this line: > val = ma.asarray(value).astype(np.float) > Then ma.minimum() and ma.maximum() are called and eventually the line > highlighted above gets executed. I don't know if/why this conversion > to masked arrays is necessary in the matplotlib code, but it seems > pretty clear that the source of the problem is the application of the > ndarray.ravel() method rather than that for the subclass in the line > highlighted above. We are using masked arrays because we need to allow for missing values; we need to be able to find the max and min of the valid values. Using ma.asarray etc. is the simplest way to do this, assuming the user will be using masked arrays when input data have invalid values. With more code we could avoid the conversion when an input array is not masked. The strategy for supporting masked arrays, nans, and infs in mpl is not completely settled yet. Eric From cournape at gmail.com Wed Aug 13 14:06:47 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 13 Aug 2008 13:06:47 -0500 Subject: [Numpy-discussion] C99 math funcs and windows Message-ID: <5b8d13220808131106j5424c023n29f2b9e8677aac3c@mail.gmail.com> Hi, Many tests (14) are failing on windows because of the recent changes in some maths functions (C99 conformance). I believe the problem is caused by mingw compiler (which is stuck at 3.4.5), but I still have to check that. To solve this, I see three solutions: - disabling the C99 tests on windows. Easy - using a more recent version of mingw: mingw does not officially support gcc 4, but we can built it ourselves. Not too difficult (I have some makefiles to build a toolchain from scratch on linux/mac os x), but I don't like the idea of using our own compiler. - we don't care, tests fail on windows. Do some people have a strong opinion on that ? Disabling sounds like the way to go for 1.2.0, but at some point, we will have to deal with this C99 problem on windows if people add more C99 code in it. cheers, David From cournape at gmail.com Wed Aug 13 15:35:39 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 13 Aug 2008 14:35:39 -0500 Subject: [Numpy-discussion] C99 on windows Message-ID: <5b8d13220808131235p3e9f127s2a78c96813934138@mail.gmail.com> Hi, The current trunk has 14 failures on windows (with mingw). 12 of them are related to C99 (see ticket 869). Can the people involved in recent changes to complex functions take a look at it ? I think this is high priority for 1.2.0 thanks, David From jh at physics.ucf.edu Wed Aug 13 15:37:09 2008 From: jh at physics.ucf.edu (Joe Harrington) Date: Wed, 13 Aug 2008 15:37:09 -0400 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: (numpy-discussion-request@scipy.org) References: Message-ID: >On Tue, Aug 12, 2008 at 19:28, Charles R Harris > wrote: >> >> >> On Tue, Aug 12, 2008 at 5:13 PM, Andrew Dalke >> wrote: >>> >>> On Aug 12, 2008, at 9:54 AM, Anne Archibald wrote: >>> > Er, is this actually a bug? I would instead consider the fact that >>> > np.min([]) raises an exception a bug of sorts - the identity of min is >>> > inf. >> >> >> >>> >>> Personally, I expect that if my array 'x' has a NaN then >>> min(x) must be a NaN. >> >> I suppose you could use >> >> min(a,b) = (abs(a - b) + a + b)/2 >> >> which would have that effect. >Or we could implement the inner loop of the minimum ufunc to return >NaN if there is a NaN. Currently it just compares the two values >(which causes the unpredictable results since having a NaN on either >side of the < is always False). I would be amenable to that provided >that the C isnan() call does not cause too much slowdown in the normal >case. While you're doing that, can you do it so that if keyword nan=False it returns NaN if NaNs exist, and if keyword nan=True it ignores NaNs? We can argue which should be the default (see my prior post). Both are compatible with the current undefined behavior. I assume that the fastest way to do it is two separate loops for the separate cases, but it might be fast enough straight (with a conditional in the inner loop), or with some other trick (macro magic, function pointer, whatever). Thanks, --jh-- From robert.kern at gmail.com Wed Aug 13 16:01:46 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 13 Aug 2008 15:01:46 -0500 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: References: Message-ID: <3d375d730808131301h73044a9co8efab7a4e446a649@mail.gmail.com> On Wed, Aug 13, 2008 at 14:37, Joe Harrington wrote: >>On Tue, Aug 12, 2008 at 19:28, Charles R Harris >> wrote: >>> >>> >>> On Tue, Aug 12, 2008 at 5:13 PM, Andrew Dalke >>> wrote: >>>> >>>> On Aug 12, 2008, at 9:54 AM, Anne Archibald wrote: >>>> > Er, is this actually a bug? I would instead consider the fact that >>>> > np.min([]) raises an exception a bug of sorts - the identity of min is >>>> > inf. >>> >>> >>> >>>> >>>> Personally, I expect that if my array 'x' has a NaN then >>>> min(x) must be a NaN. >>> >>> I suppose you could use >>> >>> min(a,b) = (abs(a - b) + a + b)/2 >>> >>> which would have that effect. > >>Or we could implement the inner loop of the minimum ufunc to return >>NaN if there is a NaN. Currently it just compares the two values >>(which causes the unpredictable results since having a NaN on either >>side of the < is always False). I would be amenable to that provided >>that the C isnan() call does not cause too much slowdown in the normal >>case. > > While you're doing that, can you do it so that if keyword nan=False it > returns NaN if NaNs exist, and if keyword nan=True it ignores NaNs? I'm doing nothing. Someone else must volunteer. But I'm not in favor of using a keyword argument. There is a reasonable design rule that if you have a boolean argument which you expect to only be passed literal Trues and Falses, you should instead just have two different functions. Since we already have names staked out for this alternate version (nanmin() and nanmax()), we might as well use them. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Daniel.Lenski at seagate.com Wed Aug 13 15:56:48 2008 From: Daniel.Lenski at seagate.com (Dan Lenski) Date: Wed, 13 Aug 2008 19:56:48 +0000 (UTC) Subject: [Numpy-discussion] reading *big* inhomogenous text matrices *fast*? Message-ID: Hi all, I'm using NumPy to read and process data from ASCII UCD files. This is a file format for describing unstructured finite-element meshes. Most of the file consists of rectangular, numerical text matrices, easily and efficiently read with loadtxt(). But there is one particularly nasty section that consists of matrices with variable numbers of columns, like this: # index property type nodes 1 1 tet 620 583 1578 1792 2 1 tet 656 551 553 566 3 1 tet 1565 766 1600 1646 4 1 tet 1545 631 1566 1665 5 1 hex 1531 1512 1559 1647 1648 1732 6 1 hex 777 1536 1556 1599 1601 1701 7 1 quad 296 1568 1535 1604 8 1 quad 54 711 285 666 As you might guess, the "type" label in the third column does indicate the number of following columns. Some of my files contain sections like this of *more than 1 million lines*, so I need to be able to read them fast. I have not yet come up with a good way to do this. What I do right now is I split them up into separate arrays based on the "type" label: lines = [f.next() for i in range(n)] lines = [l.split(None, 3) for l in lines] id, prop, types, nodes = apply(zip, lines) # THIS TAKES /FOREVER/ id = array(id, dtype=uint) prop = array(id, dtype=uint) types = array(types) cells = {} for t in N.unique(types): these = N.nonzero(types==t) # THIS NEXT LINE TAKES FOREVER these_nodes = array([nodes[ii].split() for ii in these], dtype=uint).T cells[t] = N.row_stack(( id[these], prop[these], these_nodes )) This is really pretty slow and sub-optimal. Has anyone developed a more efficient way to read arrays with variable numbers of columns??? Dan From robert.kern at gmail.com Wed Aug 13 16:15:10 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 13 Aug 2008 15:15:10 -0500 Subject: [Numpy-discussion] Mac OSX 4-way universal In-Reply-To: References: <3d375d730808122240l7cb5847fpa557ab587c99bfab@mail.gmail.com> Message-ID: <3d375d730808131315o316c5608u490c40f6efda8013@mail.gmail.com> On Wed, Aug 13, 2008 at 08:00, Chris Kees wrote: > %python setup.py build > %sudo python setup.py install > ...snipped > adding 'build/scripts.macosx-10.5-universal-2.6/f2py64-32' to scripts > error: Can't install when cross-compiling Please don't snip. I need more context. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dalke at dalkescientific.com Wed Aug 13 16:18:05 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Wed, 13 Aug 2008 22:18:05 +0200 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: <48A31AD8.9030201@noaa.gov> References: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> <9457e7c80808111549w588e515y7b17a37d1c7b2cc4@mail.gmail.com> <200808111941.39483.pgmdevlist@gmail.com> <48A11A16.8040009@noaa.gov> <5AE7D1A6-69A2-4688-AA5B-2A9519E897A8@dalkescientific.com> <9457e7c80808112356t7baf603bj7eca70f3daad4972@mail.gmail.com> <8D667EED-619B-44F7-A214-6CC926C71FE4@dalkescientific.com> <3d375d730808121859g5934bd3s83ac3556c93b8681@mail.gmail.com> <48A31AD8.9030201@noaa.gov> Message-ID: <5BFA2FEE-7EE5-4484-AFC4-AD5707F16769@dalkescientific.com> Robert Kern wrote: > Or we could implement the inner loop of the minimum ufunc to return > NaN if there is a NaN. Currently it just compares the two values > (which causes the unpredictable results since having a NaN on either > side of the < is always False). I would be amenable to that provided > that the C isnan() call does not cause too much slowdown in the normal > case. Reading this again, I realize that I don't know how ufuncs work so this suggestion might not be feasible. .... It doesn't need to be unpredictable. Make sure the first value is not a NaN (if it is, quit). The test against NaN always returns false, so by inverting the comparison then inverting the result you end up with a test for "is a new minimum OR is NaN". (I checked the assembly output. There's no effective different in code length between the normal and the inverted forms. I didn't test performance.) For random values in the array the test should pass less and less often, so sticking the isnan test in there has something like O(log(N)) cost instead of O(N) cost. That's handwaving, btw, but it's probably a log because the effect is scale invariant. Here's example code #include #include double nan_min(int n, double *data) { int i; double best = data[0]; if (isnan(best)) { return best; } for (i=1; i References: <5b8d13220808131106j5424c023n29f2b9e8677aac3c@mail.gmail.com> Message-ID: > Do some people have a strong opinion on that ? Disabling sounds like > the way to go for 1.2.0, but at some point, we will have to deal with > this C99 problem on windows if people add more C99 code in it. Besides, not every one has a C99 compliant compiler on Windows... Is mingw compatible with VS 9.0 now ? Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher From christopher.e.kees at erdc.usace.army.mil Wed Aug 13 16:26:54 2008 From: christopher.e.kees at erdc.usace.army.mil (Chris Kees) Date: Wed, 13 Aug 2008 15:26:54 -0500 Subject: [Numpy-discussion] Mac OSX 4-way universal In-Reply-To: <3d375d730808131315o316c5608u490c40f6efda8013@mail.gmail.com> Message-ID: Here's a bit more. I tried sending the redirected output as attachments but it got held up by the list server. building extension "numpy.lib._compiled_base" sources building extension "numpy.numarray._capi" sources building extension "numpy.fft.fftpack_lite" sources building extension "numpy.linalg.lapack_lite" sources adding 'numpy/linalg/lapack_litemodule.c' to sources. adding 'numpy/linalg/python_xerbla.c' to sources. building extension "numpy.random.mtrand" sources customize NAGFCompiler customize AbsoftFCompiler customize IBMFCompiler customize IntelFCompiler customize GnuFCompiler customize Gnu95FCompiler customize Gnu95FCompiler customize Gnu95FCompiler using config C compiler: gcc -arch i386 -arch ppc -arch ppc64 -arch x86_64 -isysroot / -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -f wrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/Library/Frameworks/Python64.framework/Versions/2.6/include/python2.6 -I /Library/Frameworks/Python64.framework/Versions/2.6/include/python2.6 -c' gcc: _configtest.c gcc _configtest.o -o _configtest _configtest failure. removing: _configtest.c _configtest.o _configtest building data_files sources running build_py copying build/src.macosx-10.3-i386-2.6/numpy/__config__.py -> build/lib.macosx-10.5-universal-2.6/numpy copying build/src.macosx-10.3-i386-2.6/numpy/distutils/__config__.py -> build/lib.macosx-10.5-universal-2.6/numpy/distutils running build_ext customize UnixCCompiler customize UnixCCompiler using build_ext running build_scripts adding 'build/scripts.macosx-10.5-universal-2.6/f2py64-32' to scripts error: Can't install when cross-compiling On 8/13/08 3:15 PM, "Robert Kern" wrote: > On Wed, Aug 13, 2008 at 08:00, Chris Kees > wrote: >> %python setup.py build >> %sudo python setup.py install >> ...snipped >> adding 'build/scripts.macosx-10.5-universal-2.6/f2py64-32' to scripts >> error: Can't install when cross-compiling > > Please don't snip. I need more context. From pav at iki.fi Wed Aug 13 16:52:16 2008 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 13 Aug 2008 20:52:16 +0000 (UTC) Subject: [Numpy-discussion] C99 math funcs and windows References: <5b8d13220808131106j5424c023n29f2b9e8677aac3c@mail.gmail.com> Message-ID: Hi, Wed, 13 Aug 2008 13:06:47 -0500, David Cournapeau wrote: > Many tests (14) are failing on windows because of the recent changes in > some maths functions (C99 conformance). I believe the problem is caused > by mingw compiler (which is stuck at 3.4.5), but I still have to check > that. To solve this, I see three solutions: > - disabling the C99 tests on windows. Easy > - using a more recent > version of mingw: mingw does not officially > support gcc 4, but we can built it ourselves. Not too difficult (I have > some makefiles to build a toolchain from scratch on linux/mac os x), but > I don't like the idea of using our own compiler. > - we don't care, tests fail on windows. > > Do some people have a strong opinion on that ? Disabling sounds like the > way to go for 1.2.0, but at some point, we will have to deal with this > C99 problem on windows if people add more C99 code in it. I went ahead and backed out the TestC99 testcases in r5644. They don't currently really serve any purpose as they are not guaranteed to succeed on all platforms, and we'll definitely want all tests to pass. I suggest we put them back in only after we actually have some guarantees about inf/nan handling. -- Pauli Virtanen From zachary.pincus at yale.edu Wed Aug 13 16:57:32 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 13 Aug 2008 16:57:32 -0400 Subject: [Numpy-discussion] reading *big* inhomogenous text matrices *fast*? In-Reply-To: References: Message-ID: <43BE1FA8-3639-4BF6-AD02-ACDA845CBBEE@yale.edu> Hi Dan, Your approach generates numerous large temporary arrays and lists. If the files are large, the slowdown could be because all that memory allocation is causing some VM thrashing. I've run into that at times parsing large text files. Perhaps better would be to iterate through the file, building up your cells dictionary incrementally. Finally, once the file is read in fully, you could convert what you can to arrays... f = open('big_file') header = f.readline() cells = {'tet':[], 'hex':[], 'quad':[]} for line in f: vals = line.split() index_property = vals[:2] type=vals[3] nodes = vals[3:] cells[type].append(index_property+nodes) for type, vals in cells: cells[type] = numpy.array(vals, dtype=int) I'm not sure if this is exactly what you want, but you get the idea... Anyhow, the above only uses about twice as much RAM as the size of the file. Your approach looks like it uses several times more than that. Also you could see if: cells[type].append(numpy.array([index, property]+nodes, dtype=int)) is faster than what's above... it's worth testing. If even that's too slow, maybe you'll need to do this in C? That shouldn't be too hard, really. Zach On Aug 13, 2008, at 3:56 PM, Dan Lenski wrote: > Hi all, > I'm using NumPy to read and process data from ASCII UCD files. This > is a > file format for describing unstructured finite-element meshes. > > Most of the file consists of rectangular, numerical text matrices, > easily > and efficiently read with loadtxt(). But there is one particularly > nasty > section that consists of matrices with variable numbers of columns, > like > this: > > # index property type nodes > 1 1 tet 620 583 1578 1792 > 2 1 tet 656 551 553 566 > 3 1 tet 1565 766 1600 1646 > 4 1 tet 1545 631 1566 1665 > 5 1 hex 1531 1512 1559 1647 1648 1732 > 6 1 hex 777 1536 1556 1599 1601 1701 > 7 1 quad 296 1568 1535 1604 > 8 1 quad 54 711 285 666 > > As you might guess, the "type" label in the third column does indicate > the number of following columns. > > Some of my files contain sections like this of *more than 1 million > lines*, so I need to be able to read them fast. I have not yet come > up > with a good way to do this. What I do right now is I split them up > into > separate arrays based on the "type" label: > > lines = [f.next() for i in range(n)] > lines = [l.split(None, 3) for l in lines] > id, prop, types, nodes = apply(zip, lines) # THIS TAKES /FOREVER/ > > id = array(id, dtype=uint) > prop = array(id, dtype=uint) > types = array(types) > > cells = {} > for t in N.unique(types): > these = N.nonzero(types==t) > # THIS NEXT LINE TAKES FOREVER > these_nodes = array([nodes[ii].split() for ii in these], dtype=uint).T > cells[t] = N.row_stack(( id[these], prop[these], these_nodes )) > > This is really pretty slow and sub-optimal. Has anyone developed a > more > efficient way to read arrays with variable numbers of columns??? > > Dan > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From cournape at gmail.com Wed Aug 13 17:11:16 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 13 Aug 2008 16:11:16 -0500 Subject: [Numpy-discussion] C99 math funcs and windows In-Reply-To: References: <5b8d13220808131106j5424c023n29f2b9e8677aac3c@mail.gmail.com> Message-ID: <5b8d13220808131411p783604c6x5ac567911ae50084@mail.gmail.com> On Wed, Aug 13, 2008 at 3:20 PM, Matthieu Brucher wrote: >> Do some people have a strong opinion on that ? Disabling sounds like >> the way to go for 1.2.0, but at some point, we will have to deal with >> this C99 problem on windows if people add more C99 code in it. > > Besides, not every one has a C99 compliant compiler on Windows... Is > mingw compatible with VS 9.0 now ? Actually, the problem may well be different from what I first thought. Still the compiler on windows situation is a bit worrying, because we will need this compatibility with VS 2008 (for 2.6, typically). There are some alpha somewhere for 4.3.0 (alpha quality), and some binaries here: http://www.develer.com/oss/GccWinBinaries But building extensions with 4.3.0 does not work IIRC. I am working on it right now, actually. cheers, David From robert.kern at gmail.com Wed Aug 13 17:20:17 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 13 Aug 2008 16:20:17 -0500 Subject: [Numpy-discussion] Mac OSX 4-way universal In-Reply-To: References: <3d375d730808131315o316c5608u490c40f6efda8013@mail.gmail.com> Message-ID: <3d375d730808131420m1451d553ue8a46d2649aa961b@mail.gmail.com> On Wed, Aug 13, 2008 at 15:26, Chris Kees wrote: > Here's a bit more. I tried sending the redirected output as attachments but > it got held up by the list server. I beg your pardon, then. You can send me the full logs in private email. > building extension "numpy.lib._compiled_base" sources > building extension "numpy.numarray._capi" sources > building extension "numpy.fft.fftpack_lite" sources > building extension "numpy.linalg.lapack_lite" sources > adding 'numpy/linalg/lapack_litemodule.c' to sources. > adding 'numpy/linalg/python_xerbla.c' to sources. > building extension "numpy.random.mtrand" sources > customize NAGFCompiler > customize AbsoftFCompiler > customize IBMFCompiler > customize IntelFCompiler > customize GnuFCompiler > customize Gnu95FCompiler > customize Gnu95FCompiler > customize Gnu95FCompiler using config > C compiler: gcc -arch i386 -arch ppc -arch ppc64 -arch x86_64 -isysroot / > -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -f > wrapv -O3 -Wall -Wstrict-prototypes > > compile options: '-Inumpy/core/src -Inumpy/core/include > -I/Library/Frameworks/Python64.framework/Versions/2.6/include/python2.6 -I > /Library/Frameworks/Python64.framework/Versions/2.6/include/python2.6 -c' > gcc: _configtest.c > gcc _configtest.o -o _configtest > _configtest > failure. > removing: _configtest.c _configtest.o _configtest > building data_files sources > running build_py > copying build/src.macosx-10.3-i386-2.6/numpy/__config__.py -> > build/lib.macosx-10.5-universal-2.6/numpy > copying build/src.macosx-10.3-i386-2.6/numpy/distutils/__config__.py -> > build/lib.macosx-10.5-universal-2.6/numpy/distutils > running build_ext > customize UnixCCompiler > customize UnixCCompiler using build_ext > running build_scripts > adding 'build/scripts.macosx-10.5-universal-2.6/f2py64-32' to scripts > error: Can't install when cross-compiling Hmm. Odd. I can't find the string "Can't install when cross-compiling" anywhere in the numpy or Python sources. Can you try again with the environment variable DISTUTILS_DEBUG=1 set? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From bioinformed at gmail.com Wed Aug 13 17:36:20 2008 From: bioinformed at gmail.com (Kevin Jacobs ) Date: Wed, 13 Aug 2008 17:36:20 -0400 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: <3d375d730808131301h73044a9co8efab7a4e446a649@mail.gmail.com> References: <3d375d730808131301h73044a9co8efab7a4e446a649@mail.gmail.com> Message-ID: <2e1434c10808131436s7bb8fc8ah36ddf5791c6e8787@mail.gmail.com> On Wed, Aug 13, 2008 at 4:01 PM, Robert Kern wrote: > On Wed, Aug 13, 2008 at 14:37, Joe Harrington wrote: > >>On Tue, Aug 12, 2008 at 19:28, Charles R Harris > >> wrote: > >>> > >>> > >>> On Tue, Aug 12, 2008 at 5:13 PM, Andrew Dalke < > dalke at dalkescientific.com> > >>> wrote: > >>>> > >>>> On Aug 12, 2008, at 9:54 AM, Anne Archibald wrote: > >>>> > Er, is this actually a bug? I would instead consider the fact that > >>>> > np.min([]) raises an exception a bug of sorts - the identity of min > is > >>>> > inf. > >>> > >>> > >>> > >>>> > >>>> Personally, I expect that if my array 'x' has a NaN then > >>>> min(x) must be a NaN. > >>> > >>> I suppose you could use > >>> > >>> min(a,b) = (abs(a - b) + a + b)/2 > >>> > >>> which would have that effect. > > > >>Or we could implement the inner loop of the minimum ufunc to return > >>NaN if there is a NaN. Currently it just compares the two values > >>(which causes the unpredictable results since having a NaN on either > >>side of the < is always False). I would be amenable to that provided > >>that the C isnan() call does not cause too much slowdown in the normal > >>case. > > > > While you're doing that, can you do it so that if keyword nan=False it > > returns NaN if NaNs exist, and if keyword nan=True it ignores NaNs? > > I'm doing nothing. Someone else must volunteer. > I've volunteered to implement this functionality and will have some time over the weekend to prepare and post a patch for further discussion. -Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Wed Aug 13 17:37:28 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 13 Aug 2008 16:37:28 -0500 Subject: [Numpy-discussion] Mac OSX 4-way universal In-Reply-To: <3d375d730808131420m1451d553ue8a46d2649aa961b@mail.gmail.com> References: <3d375d730808131315o316c5608u490c40f6efda8013@mail.gmail.com> <3d375d730808131420m1451d553ue8a46d2649aa961b@mail.gmail.com> Message-ID: <5b8d13220808131437k3aae9a6cg327b21fe5b1de98f@mail.gmail.com> On Wed, Aug 13, 2008 at 4:20 PM, Robert Kern wrote: > > Hmm. Odd. I can't find the string "Can't install when cross-compiling" > anywhere in the numpy or Python sources. Can you try again with the > environment variable DISTUTILS_DEBUG=1 set? You can find it in python svn: the message seems python 2.6 specific. cheers, David From christopher.e.kees at erdc.usace.army.mil Wed Aug 13 18:41:01 2008 From: christopher.e.kees at erdc.usace.army.mil (Chris Kees) Date: Wed, 13 Aug 2008 17:41:01 -0500 Subject: [Numpy-discussion] Mac OSX 4-way universal In-Reply-To: <5b8d13220808131437k3aae9a6cg327b21fe5b1de98f@mail.gmail.com> Message-ID: You're right. I think it's coming from here: Lib/distutils/command/install.py:573 def run (self): # Obviously have to build before we can install if not self.skip_build: self.run_command('build') # If we built for any other platform, we can't install. build_plat = self.distribution.get_command_obj('build').plat_name # check warn_dir - it is a clue that the 'install' is happening # internally, and not to sys.path, so we don't check the platform # matches what we are running. if self.warn_dir and build_plat != get_platform(): raise DistutilsPlatformError("Can't install when " "cross-compiling") On 8/13/08 4:37 PM, "David Cournapeau" wrote: > On Wed, Aug 13, 2008 at 4:20 PM, Robert Kern wrote: >> >> Hmm. Odd. I can't find the string "Can't install when cross-compiling" >> anywhere in the numpy or Python sources. Can you try again with the >> environment variable DISTUTILS_DEBUG=1 set? > > You can find it in python svn: the message seems python 2.6 specific. > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From robert.kern at gmail.com Wed Aug 13 19:08:14 2008 From: robert.kern at gmail.com (robert.kern at gmail.com) Date: Wed, 13 Aug 2008 18:08:14 -0500 Subject: [Numpy-discussion] Mac OSX 4-way universal In-Reply-To: <5b8d13220808131437k3aae9a6cg327b21fe5b1de98f@mail.gmail.com> References: <3d375d730808131315o316c5608u490c40f6efda8013@mail.gmail.com> <3d375d730808131420m1451d553ue8a46d2649aa961b@mail.gmail.com> <5b8d13220808131437k3aae9a6cg327b21fe5b1de98f@mail.gmail.com> Message-ID: <3d375d730808131608p2e3233feq40aac81906e88543@mail.gmail.com> On 2008-08-13, David Cournapeau wrote: > On Wed, Aug 13, 2008 at 4:20 PM, Robert Kern wrote: >> >> Hmm. Odd. I can't find the string "Can't install when cross-compiling" >> anywhere in the numpy or Python sources. Can you try again with the >> environment variable DISTUTILS_DEBUG=1 set? > > You can find it in python svn: the message seems python 2.6 specific. Okay, it looks like this happens when distutils.util.get_platform() and the build command's plat_name are different. Chris, can you do the following and show me the output? $ python setup.py build --help ... $ python -c "from distutils import util;print util.get_platform()" ... Probably a workaround is to do $ python setup.py build --plat-name=... install where ... is whatever the output of the second command above gives. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From myeates at jpl.nasa.gov Wed Aug 13 20:11:10 2008 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Wed, 13 Aug 2008 17:11:10 -0700 Subject: [Numpy-discussion] test errors with numpy-1.1.0 Message-ID: <48A3781E.4030306@jpl.nasa.gov> On an AMD x86_64 with ATLAS installed I am getting errors like ValueError: On entry to DLASD0 parameter number 9 had an illegal value ValueError: On entry to ILAENV parameter number 2 had an illegal value Anybody seen this before? Mathew From cournape at gmail.com Wed Aug 13 21:27:06 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 13 Aug 2008 20:27:06 -0500 Subject: [Numpy-discussion] test errors with numpy-1.1.0 In-Reply-To: <48A3781E.4030306@jpl.nasa.gov> References: <48A3781E.4030306@jpl.nasa.gov> Message-ID: <5b8d13220808131827w37ba729cy8a37601f04915e78@mail.gmail.com> On Wed, Aug 13, 2008 at 7:11 PM, Mathew Yeates wrote: > On an AMD x86_64 with ATLAS installed I am getting errors like > ValueError: On entry to DLASD0 parameter number 9 had an illegal value > ValueError: On entry to ILAENV parameter number 2 had an illegal value Which platform are you on ? Which version of atlas ? Do you have code to reproduce the bug ? David From dlenski at gmail.com Wed Aug 13 21:44:13 2008 From: dlenski at gmail.com (Daniel Lenski) Date: Thu, 14 Aug 2008 01:44:13 +0000 (UTC) Subject: [Numpy-discussion] reading *big* inhomogenous text matrices *fast*? References: <43BE1FA8-3639-4BF6-AD02-ACDA845CBBEE@yale.edu> Message-ID: On Wed, 13 Aug 2008 16:57:32 -0400, Zachary Pincus wrote: > Your approach generates numerous large temporary arrays and lists. If > the files are large, the slowdown could be because all that memory > allocation is causing some VM thrashing. I've run into that at times > parsing large text files. Thanks, Zach. I do think you have the right explanation for what was wrong with my code. I thought the slowdown was due to the overhead of interpreted code. So I tried to do everything in list comprehensions and array statements rather than explicit Python loops. But your were definitely right, the slowdown was due to memory use, not interpreted code. > Perhaps better would be to iterate through the file, building up your > cells dictionary incrementally. Finally, once the file is read in > fully, you could convert what you can to arrays... > > f = open('big_file') > header = f.readline() > cells = {'tet':[], 'hex':[], 'quad':[]} for line in f: > vals = line.split() > index_property = vals[:2] > type=vals[3] > nodes = vals[3:] > cells[type].append(index_property+nodes) > for type, vals in cells: > cells[type] = numpy.array(vals, dtype=int) This is similar to what I tried originally! Unfortunately, repeatedly appending to a list seems to be very slow... I guess Python keeps reallocating and copying the list as it grows. (It would be nice to be able to tune the increments by which the list size increases.) > I'm not sure if this is exactly what you want, but you get the idea... > Anyhow, the above only uses about twice as much RAM as the size of the > file. Your approach looks like it uses several times more than that. > > Also you could see if: > cells[type].append(numpy.array([index, property]+nodes, dtype=int)) > > is faster than what's above... it's worth testing. Repeatedly concatenating arrays with numpy.append or numpy.concatenate is also quite slow, unfortunately. :-( > If even that's too slow, maybe you'll need to do this in C? That > shouldn't be too hard, really. Yeah, I eventually came up with a decent solution Python solution: preallocate the arrays to the maximum size that might be needed. Trim them down afterwards. This is very wasteful of memory when there may be many cell types (less so if the OS does lazy allocation), but in the typical case of only a few cell types it works great: def _read_cells(self, f, n, debug=False): cells = dict() count = dict() curtype = None for i in xrange(n): cell = f.readline().split() celltype = cell[2] if celltype!=curtype: curtype = celltype if curtype not in cells: # allocate as big an array as might possibly be needed cells[curtype] = N.empty((n-i, len(cell)-1), dtype=int) count[curtype] = 0 block = cells[curtype] # put the line just read into the preallocated array block[count[curtype]] = cell[:2]+cell[3:] count[curtype] += 1 # trim the arrays down to size actually used for k in cells: cells[k] = cells[k][:count[k]].T return cells I hope this recipe may prove useful to others. It would be nice if NumPy had a built-in facility for arrays that intelligently expend their allocation as they grow. But I suppose that reading from badly-designed file formats would be one of the only applications for it :-( Dan From robert.kern at gmail.com Wed Aug 13 21:55:02 2008 From: robert.kern at gmail.com (robert.kern at gmail.com) Date: Wed, 13 Aug 2008 20:55:02 -0500 Subject: [Numpy-discussion] reading *big* inhomogenous text matrices *fast*? In-Reply-To: References: <43BE1FA8-3639-4BF6-AD02-ACDA845CBBEE@yale.edu> Message-ID: <3d375d730808131855s7f9b9d40l6b369c88754a5d4a@mail.gmail.com> On 2008-08-13, Daniel Lenski wrote: > On Wed, 13 Aug 2008 16:57:32 -0400, Zachary Pincus wrote: >> Your approach generates numerous large temporary arrays and lists. If >> the files are large, the slowdown could be because all that memory >> allocation is causing some VM thrashing. I've run into that at times >> parsing large text files. > > Thanks, Zach. I do think you have the right explanation for what was > wrong with my code. > > I thought the slowdown was due to the overhead of interpreted code. So I > tried to do everything in list comprehensions and array statements rather > than explicit Python loops. But your were definitely right, the slowdown > was due to memory use, not interpreted code. > >> Perhaps better would be to iterate through the file, building up your >> cells dictionary incrementally. Finally, once the file is read in >> fully, you could convert what you can to arrays... >> >> f = open('big_file') >> header = f.readline() >> cells = {'tet':[], 'hex':[], 'quad':[]} for line in f: >> vals = line.split() >> index_property = vals[:2] >> type=vals[3] >> nodes = vals[3:] >> cells[type].append(index_property+nodes) >> for type, vals in cells: >> cells[type] = numpy.array(vals, dtype=int) > > This is similar to what I tried originally! Unfortunately, repeatedly > appending to a list seems to be very slow... I guess Python keeps > reallocating and copying the list as it grows. (It would be nice to be > able to tune the increments by which the list size increases.) The list reallocation schedule is actually fairly well-tuned as it is. Appending to a list object should be amortized O(1) time. >> I'm not sure if this is exactly what you want, but you get the idea... >> Anyhow, the above only uses about twice as much RAM as the size of the >> file. Your approach looks like it uses several times more than that. >> >> Also you could see if: >> cells[type].append(numpy.array([index, property]+nodes, dtype=int)) >> >> is faster than what's above... it's worth testing. > > Repeatedly concatenating arrays with numpy.append or numpy.concatenate is > also quite slow, unfortunately. :-( Yes. There is no preallocation here. >> If even that's too slow, maybe you'll need to do this in C? That >> shouldn't be too hard, really. > > Yeah, I eventually came up with a decent solution Python solution: > preallocate the arrays to the maximum size that might be needed. Trim > them down afterwards. This is very wasteful of memory when there may be > many cell types (less so if the OS does lazy allocation), but in the > typical case of only a few cell types it works great: Another approach would be to preallocate a substantial chunk at a time, then concatenate all of the chunks. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dlenski at gmail.com Wed Aug 13 21:55:19 2008 From: dlenski at gmail.com (Daniel Lenski) Date: Thu, 14 Aug 2008 01:55:19 +0000 (UTC) Subject: [Numpy-discussion] non-linear array manipulation References: Message-ID: On Tue, 12 Aug 2008 10:37:51 -0400, Gong, Shawn (Contractor) wrote: > The following array manipulation takes long time because I can't find > ways to do in row/column, and have to do cell by cell. Would you check > to see if there is a nicer/faster way for this non-linear operation? > > for i in range(rows): > for j in range(columns): > a[i][j] = math.sqrt( max(0.0, a[i][j]*a[i][j] - b[j]*c[j]) ) In order to figure out how to do things like this efficiently, I like to write out the mathematical formula in subscript-summation notation first: a_ij = sqrt( max(0.0, a_ij - b_j*c_j) ) Now this could be done on an element-wise basis if you redefine b and c as matrices: B_ij = b_j and C_ij = c_j => a_ij = sqrt( max(0.0, a_ij - B_ij*C_ij) ) Fortunately, with NumPy this is easy and doesn't require any data copying or extra memory use: B = b[newaxis, :] C = c[newaxis, :] a = sqrt(maximum(0.0, a-B*C)) That's your solution. It's a standard application of the broadcasting technique, which is crucial for time- and memory- efficient array-based algorithms. It is explained in detail in the NumPy tutorial. Hope that helps, Dan From dlenski at gmail.com Wed Aug 13 22:07:29 2008 From: dlenski at gmail.com (Daniel Lenski) Date: Thu, 14 Aug 2008 02:07:29 +0000 (UTC) Subject: [Numpy-discussion] reading *big* inhomogenous text matrices *fast*? References: <43BE1FA8-3639-4BF6-AD02-ACDA845CBBEE@yale.edu> <3d375d730808131855s7f9b9d40l6b369c88754a5d4a@mail.gmail.com> Message-ID: On Wed, 13 Aug 2008 20:55:02 -0500, robert.kern wrote: >> This is similar to what I tried originally! Unfortunately, repeatedly >> appending to a list seems to be very slow... I guess Python keeps >> reallocating and copying the list as it grows. (It would be nice to be >> able to tune the increments by which the list size increases.) > > The list reallocation schedule is actually fairly well-tuned as it is. > Appending to a list object should be amortized O(1) time. Hi Robert, This is very interesting! Do you know what exactly is the allocation strategy of list.append, or have a reference? According to this page, http://effbot.org/zone/python-list.htm: "The time needed to append an item to the list is ?amortized constant?; whenever the list needs to allocate more memory, it allocates room for a few items more than it actually needs, to avoid having to reallocate on each call (this assumes that the memory allocator is fast; /for huge lists, the allocation overhead may push the behaviour towards O(n*n))/." If I'm not running into O(n^2), there must have been some other inefficiency in my first attempt at a solution! In that case, I guess I've come up with an unnecessarily complex solution to a problem where Python just Does The Right Thing. Reminds me of my old Perl days ;-) >> Repeatedly concatenating arrays with numpy.append or numpy.concatenate >> is also quite slow, unfortunately. :-( > > Yes. There is no preallocation here. Makes sense. I'm sure this cuts down on the amount of metadata that has to be carried along with each array instance, and there aren't too many applications for it. Thanks! Dan From zachary.pincus at yale.edu Wed Aug 13 22:11:07 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 13 Aug 2008 22:11:07 -0400 Subject: [Numpy-discussion] reading *big* inhomogenous text matrices *fast*? In-Reply-To: References: <43BE1FA8-3639-4BF6-AD02-ACDA845CBBEE@yale.edu> Message-ID: > This is similar to what I tried originally! Unfortunately, repeatedly > appending to a list seems to be very slow... I guess Python keeps > reallocating and copying the list as it grows. (It would be nice to > be > able to tune the increments by which the list size increases.) Robert's right, as ever -- repeated appending to a list is an *extremely* common operation, which you see often in idiomatic python. The implementation of list.append should be very fast, and smart about pre-allocating as needed. Try profiling the code just to make sure that it is the list append that's slow, and not something else happening on that line, e.g.. > I hope this recipe may prove useful to others. It would be nice if > NumPy > had a built-in facility for arrays that intelligently expend their > allocation as they grow. It appears to be the general consensus on this mailing list that the best solution when an expandable array is required is to append to a python list, and then once you've built it up completely, convert it to an array. So I'm at least surprised that this is turning out to be so slow for you... But if the profiler says that's where the trouble is, then so it is... >> Also you could see if: >> cells[type].append(numpy.array([index, property]+nodes, dtype=int)) >> >> is faster than what's above... it's worth testing. > > Repeatedly concatenating arrays with numpy.append or > numpy.concatenate is > also quite slow, unfortunately. :-( Actually, my suggestion was to compare building up a list-of-lists and then converting that to a 2d array versus building up a list-of- arrays, and then converting that to a 2d array... one might wind up being faster or more memory-efficient than the other... Zach From robert.kern at gmail.com Wed Aug 13 22:42:51 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 13 Aug 2008 21:42:51 -0500 Subject: [Numpy-discussion] reading *big* inhomogenous text matrices *fast*? In-Reply-To: References: <43BE1FA8-3639-4BF6-AD02-ACDA845CBBEE@yale.edu> <3d375d730808131855s7f9b9d40l6b369c88754a5d4a@mail.gmail.com> Message-ID: <3d375d730808131942x72116400o54831d0f5223b067@mail.gmail.com> On Wed, Aug 13, 2008 at 21:07, Daniel Lenski wrote: > On Wed, 13 Aug 2008 20:55:02 -0500, robert.kern wrote: >>> This is similar to what I tried originally! Unfortunately, repeatedly >>> appending to a list seems to be very slow... I guess Python keeps >>> reallocating and copying the list as it grows. (It would be nice to be >>> able to tune the increments by which the list size increases.) >> >> The list reallocation schedule is actually fairly well-tuned as it is. >> Appending to a list object should be amortized O(1) time. > > Hi Robert, > > This is very interesting! Do you know what exactly is the allocation > strategy of list.append, or have a reference? Here is the appropriate snippet in Objects/listobject.c: /* This over-allocates proportional to the list size, making room * for additional growth. The over-allocation is mild, but is * enough to give linear-time amortized behavior over a long * sequence of appends() in the presence of a poorly-performing * system realloc(). * The growth pattern is: 0, 4, 8, 16, 25, 35, 46, 58, 72, 88, ... */ new_allocated = (newsize >> 3) + (newsize < 9 ? 3 : 6) + newsize; Raymond Hettinger had a good talk at PyCon this year about the details of the Python containers. Here are the slides from the EuroPython version (I assume). http://www.pycon.it/static/pycon2/slides/containers.ppt >>> Repeatedly concatenating arrays with numpy.append or numpy.concatenate >>> is also quite slow, unfortunately. :-( >> >> Yes. There is no preallocation here. > > Makes sense. I'm sure this cuts down on the amount of metadata that has > to be carried along with each array instance, and there aren't too many > applications for it. Primarily, it's the fact that we have views of arrays that might be floating around that prevents us from reallocating as a matter of course. Now, we do have a .resize() method which will explicitly reallocate the array, but it will only work if you don't have any views on the array floating around. During your file reading, this is probably valid, so you may want to give it a try using a similar reallocation strategy as lists. I'd be interested in seeing some benchmarks comparing this strategy with the others. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dlenski at gmail.com Thu Aug 14 00:32:45 2008 From: dlenski at gmail.com (Daniel Lenski) Date: Thu, 14 Aug 2008 04:32:45 +0000 (UTC) Subject: [Numpy-discussion] reading *big* inhomogenous text matrices *fast*? References: <43BE1FA8-3639-4BF6-AD02-ACDA845CBBEE@yale.edu> <3d375d730808131855s7f9b9d40l6b369c88754a5d4a@mail.gmail.com> <3d375d730808131942x72116400o54831d0f5223b067@mail.gmail.com> Message-ID: On Wed, 13 Aug 2008 21:42:51 -0500, Robert Kern wrote: > Here is the appropriate snippet in Objects/listobject.c: > > /* This over-allocates proportional to the list size, making > room > * for additional growth. The over-allocation is mild, but is * > enough to give linear-time amortized behavior over a long * > sequence of appends() in the presence of a poorly-performing * > system realloc(). > * The growth pattern is: 0, 4, 8, 16, 25, 35, 46, 58, 72, 88, > ... */ > new_allocated = (newsize >> 3) + (newsize < 9 ? 3 : 6) + > newsize; > > Raymond Hettinger had a good talk at PyCon this year about the details > of the Python containers. Here are the slides from the EuroPython > version (I assume). > > http://www.pycon.it/static/pycon2/slides/containers.ppt > Thanks! Looks like the only caveat is that the whole thing may slow down if the reallocation operation itself is very inefficient. Which probably isn't the case with a modern Linux distro and recent libc. I'm thinking whatever went wrong had to be my fault :-) > Primarily, it's the fact that we have views of arrays that might be > floating around that prevents us from reallocating as a matter of > course. Now, we do have a .resize() method which will explicitly > reallocate the array, but it will only work if you don't have any views > on the array floating around. During your file reading, this is probably > valid, so you may want to give it a try using a similar reallocation > strategy as lists. I'd be interested in seeing some benchmarks comparing > this strategy with the others. That will be the next thing for me to try if my current approach becomes too memory-inefficient. Good idea! Dan From dlenski at gmail.com Thu Aug 14 00:40:16 2008 From: dlenski at gmail.com (Daniel Lenski) Date: Thu, 14 Aug 2008 04:40:16 +0000 (UTC) Subject: [Numpy-discussion] reading *big* inhomogenous text matrices *fast*? References: <43BE1FA8-3639-4BF6-AD02-ACDA845CBBEE@yale.edu> Message-ID: On Wed, 13 Aug 2008 22:11:07 -0400, Zachary Pincus wrote: > Try profiling the code just to make sure that it is the list append > that's slow, and not something else happening on that line, e.g.. >From what you and others have pointed out, I'm pretty sure I must have been doing something else wrong although my code wasn't in SVN yet so I'm not sure exactly what. > It appears to be the general consensus on this mailing list that the > best solution when an expandable array is required is to append to a > python list, and then once you've built it up completely, convert it to > an array. So I'm at least surprised that this is turning out to be so > slow for you... But if the profiler says that's where the trouble is, > then so it is... That does seem to be the standard idiom used by NumPy, such as in loadtxt. And loadtxt is usually fast enough for me. > Actually, my suggestion was to compare building up a list-of-lists and > then converting that to a 2d array versus building up a list-of- arrays, > and then converting that to a 2d array... one might wind up being faster > or more memory-efficient than the other... I assume that list-of-arrays is more memory-efficient since array elements don't have the overhead of full-blown Python objects. But list- of-lists is probably more time-efficient since I think it's faster to convert the whole array at once than do it row-by-row. Dan From hep.sebastien.binet at gmail.com Thu Aug 14 00:41:14 2008 From: hep.sebastien.binet at gmail.com (Sebastien Binet) Date: Wed, 13 Aug 2008 21:41:14 -0700 Subject: [Numpy-discussion] =?iso-8859-1?q?reading_*big*_inhomogenous_text?= =?iso-8859-1?q?_matrices=09*fast*=3F?= In-Reply-To: References: <3d375d730808131942x72116400o54831d0f5223b067@mail.gmail.com> Message-ID: <200808132141.14248.binet@cern.ch> Hi, > > Raymond Hettinger had a good talk at PyCon this year about the details > > of the Python containers. Here are the slides from the EuroPython > > version (I assume). > > > > http://www.pycon.it/static/pycon2/slides/containers.ppt > > Thanks! Looks like the only caveat is that the whole thing may slow down > if the reallocation operation itself is very inefficient. Which probably > isn't the case with a modern Linux distro and recent libc. I'm thinking > whatever went wrong had to be my fault :-) just thinking out loud... could the slow down be induced by the memory allocation, triggering many GC collections ? I recall a post on c.l.p where disabling the gc during pickling/unpickling a complicated data structure tremendously improved performances... cheers, sebastien. -- ################################### # Dr. Sebastien Binet # Lawrence Berkeley National Lab. # 1 Cyclotron Road # Berkeley, CA 94720 ################################### From jh at physics.ucf.edu Thu Aug 14 02:29:12 2008 From: jh at physics.ucf.edu (Joe Harrington) Date: Thu, 14 Aug 2008 02:29:12 -0400 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: (numpy-discussion-request@scipy.org) References: Message-ID: > I'm doing nothing. Someone else must volunteer. Fair enough. Would the code be accepted if contributed? > There is a > reasonable design rule that if you have a boolean argument which you > expect to only be passed literal Trues and Falses, you should instead > just have two different functions. Robert, can you list some reasons to favor this design rule? Here are some reasons to favor richly functional routines: User's code is more readable because subtle differences affect args, not functions Easier learning for new users Much briefer and more readable docs Similar behavior across languages Smaller number of functions in the core package (a recent list topic) Many fewer routines to maintain, particularly if multiple switches exist Availability of the NaN functionality in a method of ndarray The last point is key. The NaN behavior is central to analyzing real data containing unavoidable bad values, which is the bread and butter of a substantial fraction of the user base. In the languages they're switching from, handling NaNs is just part of doing business, and is an option of every relevant routine; there's no need for redundant sets of routines. In contrast, numpy appears to consider data analysis to be secondary, somehow, to pure math, and takes the NaN functionality out of routines like min() and std(). This means it's not possible to use many ndarray methods. If we're ready to handle a NaN by returning it, why not enable the more useful behavior of ignoring it, at user discretion? --jh-- From Norbert.Nemec.list at gmx.de Thu Aug 14 03:15:43 2008 From: Norbert.Nemec.list at gmx.de (Norbert Nemec) Date: Thu, 14 Aug 2008 08:15:43 +0100 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: <48A0B174.9060809@enthought.com> References: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> <48A0B174.9060809@enthought.com> Message-ID: <48A3DB9F.9060603@gmx.de> Travis E. Oliphant wrote: > Thomas J. Duck wrote: > >> Determining the minimum value of an array that contains NaN produces >> a surprising result: >> >> >>> x = numpy.array([0,1,2,numpy.nan,4,5,6]) >> >>> x.min() >> 4.0 >> >> I expected 0.0. Is this the intended behaviour or a bug? I am using >> numpy 1.1.1. >> >> > NAN's don't play well with comparisons because comparison with them is > undefined. See numpy.nanmin > This is not true! Each single comparison with a NaN has a well defined outcome. The difficulty is only that certain logical assumptions do not hold any more when NaNs are involved (e.g. [A=B)]). Assuming an IEEE compliant processor and C compiler, it should be possible to code a NaN safe min routine without additional overhead. From cimrman3 at ntc.zcu.cz Thu Aug 14 03:56:42 2008 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Thu, 14 Aug 2008 09:56:42 +0200 Subject: [Numpy-discussion] unique1d returning indices In-Reply-To: <9457e7c80808131023i1dc346c1i225ebfaacf30254b@mail.gmail.com> References: <48999E96.6070000@ntc.zcu.cz> <9457e7c80808080928p2e6d404ak91576a44a5234e7@mail.gmail.com> <48A0091A.6080400@ntc.zcu.cz> <9457e7c80808110754p2238ef3o221e3a34f6fb9f7d@mail.gmail.com> <48A05D5E.6080903@ntc.zcu.cz> <48A2F07C.1000207@ntc.zcu.cz> <9457e7c80808131023i1dc346c1i225ebfaacf30254b@mail.gmail.com> Message-ID: <48A3E53A.3010705@ntc.zcu.cz> St?fan van der Walt wrote: > 2008/8/13 Robert Cimrman : >>> Yeah, that's why I think not many people used the extra return anyway. >>> I will do as you say unless somebody steps in. >> ... but not before August 25, as I am about to leave on holidays and >> have not managed to do it yet. I do not want to mess with the SVN now as >> I would not be able to follow it. >> >> If you think the patch is ok, and have time, then go ahead :) > > Thanks, Robert. This has been merged in r5639. Nice, thank you. After I return I will go through the other arraysetops functions and add return_index-like flags when appropriate. r. From peridot.faceted at gmail.com Thu Aug 14 05:46:27 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 14 Aug 2008 05:46:27 -0400 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: <48A3DB9F.9060603@gmx.de> References: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> <48A0B174.9060809@enthought.com> <48A3DB9F.9060603@gmx.de> Message-ID: 2008/8/14 Norbert Nemec : > Travis E. Oliphant wrote: >> NAN's don't play well with comparisons because comparison with them is >> undefined. See numpy.nanmin >> > This is not true! Each single comparison with a NaN has a well defined > outcome. The difficulty is only that certain logical assumptions do not > hold any more when NaNs are involved (e.g. [A [not(A>=B)]). Assuming an IEEE compliant processor and C compiler, it > should be possible to code a NaN safe min routine without additional > overhead. Sadly, it's not possible without extra overhead. Specifically: the NaN-ignorant implementation does a single comparison between each array element and a placeholder, and decides based on the result which to keep. If you try to rewrite the comparison to do the right thing when a NaN is involved, you get stuck: any comparison with a NaN on either side always returns False, so you cannot distinguish between the temporary being a NaN and the new element being a non-NaN (keep the temporary) and the temporary being a non-NaN and the new element being a NaN (replace the temporary). If you're willing to do two tests, sure, but that's overhead (and probably comparable to an isnan). If you're willing to do arithmetic you might even be able to pull it off, since NaNs tend to propagate: if (new References: <43BE1FA8-3639-4BF6-AD02-ACDA845CBBEE@yale.edu> <3d375d730808131855s7f9b9d40l6b369c88754a5d4a@mail.gmail.com> <3d375d730808131942x72116400o54831d0f5223b067@mail.gmail.com> Message-ID: <48A46288.20807@noaa.gov> One other potential downside of using python lists to accumulate numbers is that you are storing python objects (python ints or floats, or...) rather than raw numbers, which has got to incur some memory overhead. How does array.array perform in this context? It has an append() method, and one would hope it uses a similar memory allocation scheme. Also, does numpy convert array.array objects to numpy arrays more efficiently? It could, of course, but someone would have to have written the special case code. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From kwgoodman at gmail.com Thu Aug 14 13:20:42 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 14 Aug 2008 10:20:42 -0700 Subject: [Numpy-discussion] Different results from repeated calculation, part 2 Message-ID: I get slightly different results when I repeat a calculation. I've seen this problem before (it went away but has returned): http://projects.scipy.org/pipermail/numpy-discussion/2007-January/025724.html A unit test is attached. It contains three tests: In test1, I construct matrices x and y and then repeatedly calculate z = calc(x,y). The result z is the same every time. So this test passes. In test2, I construct matrices x and y each time before calculating z = calc(x,y). Sometimes z is slightly different. But the x's test to be equal and so do the y's. This test fails (on Debian Lenny, Core 2 Duo, with libatlas3gf-sse2 but not with libatlas3gf-sse). test3 is the same as test2 but I calculate z like this: z = calc(100*x,y) / (100 * 100). This test passes. I get: ====================================================================== FAIL: repeatability #2 ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/[snip]/test/repeat_test.py", line 73, in test_repeat_2 self.assert_(result, msg) AssertionError: Max difference = 2.04946e-16 ---------------------------------------------------------------------- Should a unit test like this be added to numpy? -------------- next part -------------- A non-text attachment was scrubbed... Name: repeat_test.py Type: text/x-python Size: 2695 bytes Desc: not available URL: From as8ca at virginia.edu Thu Aug 14 13:48:14 2008 From: as8ca at virginia.edu (Alok Singhal) Date: Thu, 14 Aug 2008 13:48:14 -0400 Subject: [Numpy-discussion] Different results from repeated calculation, part 2 In-Reply-To: References: Message-ID: <20080814174814.GA30436@virginia.edu> On 14/08/08: 10:20, Keith Goodman wrote: > A unit test is attached. It contains three tests: > > In test1, I construct matrices x and y and then repeatedly calculate z > = calc(x,y). The result z is the same every time. So this test passes. > > In test2, I construct matrices x and y each time before calculating z > = calc(x,y). Sometimes z is slightly different. But the x's test to be > equal and so do the y's. This test fails (on Debian Lenny, Core 2 Duo, > with libatlas3gf-sse2 but not with libatlas3gf-sse). > > test3 is the same as test2 but I calculate z like this: z = > calc(100*x,y) / (100 * 100). This test passes. > > I get: > > ====================================================================== > FAIL: repeatability #2 > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/[snip]/test/repeat_test.py", line 73, in test_repeat_2 > self.assert_(result, msg) > AssertionError: Max difference = 2.04946e-16 Could this be because of how the calculations are done? If the floating point numbers are stored in the cpu registers, in this case (intel core duo), they are 80-bit values, whereas 'double' precision is 64-bits. Depending upon gcc's optimization settings, the amount of automatic variables, etc., it is entirely possible that the numbers are stored in registers only in some cases, and are in the RAM in other cases. Thus, in your tests, sometimes some numbers get stored in the cpu registers, making the calculations with those values different from the case if they were not stored in the registers. See "The pitfalls of verifying floating-point computations" at http://portal.acm.org/citation.cfm?doid=1353445.1353446 (or if that needs subscription, you can download the PDF from http://arxiv.org/abs/cs/0701192). The paper has a lot of examples of surprises like this. Quote: We shall discuss the following myths, among others: ... - "Arithmetic operations are deterministic; that is, if I do z=x+y in two places in the same program and my program never touches x and y in the meantime, then the results should be the same." - A variant: "If x < 1 tests true at one point, then x < 1 stays true later if I never modify x." ... -Alok -- * * Alok Singhal * * * http://www.astro.virginia.edu/~as8ca/ * * From robert.kern at gmail.com Thu Aug 14 14:10:09 2008 From: robert.kern at gmail.com (robert.kern at gmail.com) Date: Thu, 14 Aug 2008 13:10:09 -0500 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: References: Message-ID: <3d375d730808141110r3f03e1d2ud83a52411997c42b@mail.gmail.com> On 2008-08-14, Joe Harrington wrote: >> I'm doing nothing. Someone else must volunteer. > > Fair enough. Would the code be accepted if contributed? Like I said, I would be amenable to such a change. The other developers haven't weighed in on this particular proposal, but I suspect they will agree with me. >> There is a >> reasonable design rule that if you have a boolean argument which you >> expect to only be passed literal Trues and Falses, you should instead >> just have two different functions. > > Robert, can you list some reasons to favor this design rule? nanmin(x) vs. min(x, nan=True) A boolean argument that will almost always take literal Trues and Falses basically is just a switch between different functionality. The usual mechanism for the programmer to pick between different functionality is to use the appropriate function. The =True is extraneous, and puts important semantic information last rather than at the front. > Here are some reasons to favor richly functional routines: > > User's code is more readable because subtle differences affect args, > not functions This isn't subtle. > Easier learning for new users You have no evidence of this. > Much briefer and more readable docs Briefer is possible. More readable is debatable. "Much" is overstating the case. > Similar behavior across languages This is not, has never been, and never will be a goal. Similar behavior happens because of convergent design constraints and occasionally laziness, never for it's own sake. > Smaller number of functions in the core package (a recent list topic) In general, this is a reasonable concern that must be traded off with the other concerns. In this particular case, it has no weight. nanmin() and nanmax() already exist. > Many fewer routines to maintain, particularly if multiple switches exist Again, in this case, neither of these are relevant. Yes, if there are multiple boolean switches, it might make sense to keep them all into the same function. Typically, these switches will also be affecting the semantics only in minor details, too. > Availability of the NaN functionality in a method of ndarray Point, but see below. > The last point is key. The NaN behavior is central to analyzing real > data containing unavoidable bad values, which is the bread and butter > of a substantial fraction of the user base. In the languages they're > switching from, handling NaNs is just part of doing business, and is > an option of every relevant routine; there's no need for redundant > sets of routines. In contrast, numpy appears to consider data > analysis to be secondary, somehow, to pure math, and takes the NaN > functionality out of routines like min() and std(). This means it's > not possible to use many ndarray methods. If we're ready to handle a > NaN by returning it, why not enable the more useful behavior of > ignoring it, at user discretion? Let's get something straight. numpy has no opinion on the primacy of data analysis tasks versus "pure math", however you want to define those. Now, the numpy developers *do* tend to have an opinion on how NaNs are used. NaNs were invented to handle invalid results of *computations*. They were not invented as place markers for missing data. They can frequently be used as such because the IEEE-754 semantics of NaNs sometimes works for missing data (e.g. in z=x+y, z will have a NaN wherever either x or y have NaNs). But at least as frequently, they don't, and other semantics need to be specifically placed on top of it (e.g. nanmin()). numpy is a general purpose computational tool that needs to apply to many different fields and use cases. Consequently, when presented with a choice like this, we tend to go for the path that makes the minimum of assumptions and overlaid semantics. Now to address the idea that all of the relevant ndarray methods should take nan=True arguments. I am sympathetic to the idea that we should have the functionality somewhere. I do doubt that the users you are thinking about will be happy adding nan=True to a substantial fraction of their calls. My experience with such APIs is that it gets tedious real fast. Instead, I would suggest that if you want a wide range of nan-skipping versions of functions that we have, let's put them all as functions into a module. This gives the programmer the possibility of using relatively clean calls. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Rapp at mrt.uka.de Thu Aug 14 14:29:00 2008 From: Rapp at mrt.uka.de (Holger Rapp) Date: Thu, 14 Aug 2008 20:29:00 +0200 Subject: [Numpy-discussion] Different results from repeated calculation, part 2 In-Reply-To: <20080814174814.GA30436@virginia.edu> References: <20080814174814.GA30436@virginia.edu> Message-ID: <212F30A4-9E26-412B-B883-B4BA3BEE55CE@mrt.uka.de> Hi, Am 14.08.2008 um 19:48 schrieb Alok Singhal: > On 14/08/08: 10:20, Keith Goodman wrote: >> A unit test is attached. It contains three tests: >> >> In test1, I construct matrices x and y and then repeatedly >> calculate z >> = calc(x,y). The result z is the same every time. So this test >> passes. >> >> In test2, I construct matrices x and y each time before calculating z >> = calc(x,y). Sometimes z is slightly different. But the x's test to >> be >> equal and so do the y's. This test fails (on Debian Lenny, Core 2 >> Duo, >> with libatlas3gf-sse2 but not with libatlas3gf-sse). >> >> test3 is the same as test2 but I calculate z like this: z = >> calc(100*x,y) / (100 * 100). This test passes. >> >> I get: >> >> = >> ===================================================================== >> FAIL: repeatability #2 >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "/home/[snip]/test/repeat_test.py", line 73, in test_repeat_2 >> self.assert_(result, msg) >> AssertionError: Max difference = 2.04946e-16 > > Could this be because of how the calculations are done? If the > floating point numbers are stored in the cpu registers, in this case > (intel core duo), they are 80-bit values, whereas 'double' precision > is 64-bits. Depending upon gcc's optimization settings, the amount of > automatic variables, etc., it is entirely possible that the numbers > are stored in registers only in some cases, and are in the RAM in > other cases. Thus, in your tests, sometimes some numbers get stored > in the cpu registers, making the calculations with those values > different from the case if they were not stored in the registers. The tests never fail on my CoreDuo 2 on MacOS X, just for the records ;) Holger From bsouthey at gmail.com Thu Aug 14 14:29:29 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 14 Aug 2008 13:29:29 -0500 Subject: [Numpy-discussion] Different results from repeated calculation, part 2 In-Reply-To: References: Message-ID: <48A47989.8090904@gmail.com> Keith Goodman wrote: > I get slightly different results when I repeat a calculation. > > I've seen this problem before (it went away but has returned): > > http://projects.scipy.org/pipermail/numpy-discussion/2007-January/025724.html > > A unit test is attached. It contains three tests: > > In test1, I construct matrices x and y and then repeatedly calculate z > = calc(x,y). The result z is the same every time. So this test passes. > > In test2, I construct matrices x and y each time before calculating z > = calc(x,y). Sometimes z is slightly different. But the x's test to be > equal and so do the y's. This test fails (on Debian Lenny, Core 2 Duo, > with libatlas3gf-sse2 but not with libatlas3gf-sse). > > test3 is the same as test2 but I calculate z like this: z = > calc(100*x,y) / (100 * 100). This test passes. > > I get: > > ====================================================================== > FAIL: repeatability #2 > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/[snip]/test/repeat_test.py", line 73, in test_repeat_2 > self.assert_(result, msg) > AssertionError: Max difference = 2.04946e-16 > > ---------------------------------------------------------------------- > > Should a unit test like this be added to numpy? > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion Hi, In the function 'test_repeat_2' you are redefining variables 'x and y' that were first defined using the setup function. (Also, you are not using the __init__ function.) I vaguely recall there are some quirks to Python classes with this, so does the problem go away with if you use 'a,b' instead of 'x, y'? (I suspect the answer is yes given test_repeat_3). Note that you should also test that 'x' and 'y' are same here as well (but these have been redefined...). Otherwise, can you please provide your OS (version), computer processor, Python version, numpy version, version of atlas (or similar) and compiler used? I went back and reread the thread but I could not see this information. Bruce From christopher.e.kees at usace.army.mil Thu Aug 14 14:38:54 2008 From: christopher.e.kees at usace.army.mil (Chris Kees) Date: Thu, 14 Aug 2008 13:38:54 -0500 Subject: [Numpy-discussion] Mac OSX 4-way universal Re: [Pythonmac-SIG] python 2.6 trunk In-Reply-To: <3d375d730808131608p2e3233feq40aac81906e88543@mail.gmail.com> Message-ID: The 4-way universal install of numpy-1.1.1 is working now with the Python 2.6b2+ (trunk:65678), and all the tests pass (running as i386 and x86_64 at least). Unfortunately, I didn't find exactly what was causing it. I just erased /Library/Frameworks/Python64.framework and rebuilt the 4-way universal python and numpy module again because I had noticed that the numpy build directory kept coming up with this structure: %ls build lib.macosx-10.5-universal-2.6 scripts.macosx-10.5-universal-2.6 src.macosx-10.3-i386-2.6 temp.macosx-10.5-universal-2.6 Apparently something in my python framework was leftover from a previous install and causing the src.macosx-10.3-i386-2.6 to get built, which seems to be related to distutils deciding that I was cross-compiling. Anyway, thanks for the help and sorry for the trouble. I guess one should always erase the python framework before re-installing it? Should I be running an uninstall script instead of just erasing it? Chris On 8/13/08 6:08 PM, "robert.kern at gmail.com" wrote: > On 2008-08-13, David Cournapeau wrote: >> On Wed, Aug 13, 2008 at 4:20 PM, Robert Kern wrote: >>> >>> Hmm. Odd. I can't find the string "Can't install when cross-compiling" >>> anywhere in the numpy or Python sources. Can you try again with the >>> environment variable DISTUTILS_DEBUG=1 set? >> >> You can find it in python svn: the message seems python 2.6 specific. > > Okay, it looks like this happens when distutils.util.get_platform() > and the build command's plat_name are different. Chris, can you do the > following and show me the output? > > $ python setup.py build --help > ... > $ python -c "from distutils import util;print util.get_platform()" > ... > > Probably a workaround is to do > > $ python setup.py build --plat-name=... install > > where ... is whatever the output of the second command above gives. From kwgoodman at gmail.com Thu Aug 14 14:57:33 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 14 Aug 2008 11:57:33 -0700 Subject: [Numpy-discussion] Different results from repeated calculation, part 2 In-Reply-To: <48A47989.8090904@gmail.com> References: <48A47989.8090904@gmail.com> Message-ID: On Thu, Aug 14, 2008 at 11:29 AM, Bruce Southey wrote: > Keith Goodman wrote: >> I get slightly different results when I repeat a calculation. >> >> I've seen this problem before (it went away but has returned): >> >> http://projects.scipy.org/pipermail/numpy-discussion/2007-January/025724.html >> >> A unit test is attached. It contains three tests: >> >> In test1, I construct matrices x and y and then repeatedly calculate z >> = calc(x,y). The result z is the same every time. So this test passes. >> >> In test2, I construct matrices x and y each time before calculating z >> = calc(x,y). Sometimes z is slightly different. But the x's test to be >> equal and so do the y's. This test fails (on Debian Lenny, Core 2 Duo, >> with libatlas3gf-sse2 but not with libatlas3gf-sse). >> >> test3 is the same as test2 but I calculate z like this: z = >> calc(100*x,y) / (100 * 100). This test passes. >> >> I get: >> >> ====================================================================== >> FAIL: repeatability #2 >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "/home/[snip]/test/repeat_test.py", line 73, in test_repeat_2 >> self.assert_(result, msg) >> AssertionError: Max difference = 2.04946e-16 >> >> ---------------------------------------------------------------------- >> >> Should a unit test like this be added to numpy? >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion > Hi, > In the function 'test_repeat_2' you are redefining variables 'x and y' > that were first defined using the setup function. (Also, you are not > using the __init__ function.) I vaguely recall there are some quirks to > Python classes with this, so does the problem go away with if you use > 'a,b' instead of 'x, y'? (I suspect the answer is yes given test_repeat_3). > > Note that you should also test that 'x' and 'y' are same here as well > (but these have been redefined...). > > Otherwise, can you please provide your OS (version), computer processor, > Python version, numpy version, version of atlas (or similar) and > compiler used? > > I went back and reread the thread but I could not see this information. Here's a test that doesn't use classes and checks that x and y do not change: http://projects.scipy.org/pipermail/numpy-discussion/attachments/20070127/52b3a51c/attachment.py I'm using binaries from Debian Lenny: $ uname -a Linux jan 2.6.25-2-686 #1 SMP Fri Jul 18 17:46:56 UTC 2008 i686 GNU/Linux $ python -V Python 2.5.2 >> numpy.__version__ '1.1.0' $ cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz stepping : 6 cpu MHz : 2402.004 cache size : 4096 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 2 fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr lahf_lm bogomips : 4807.45 clflush size : 64 processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz stepping : 6 cpu MHz : 2402.004 cache size : 4096 KB physical id : 0 siblings : 2 core id : 1 cpu cores : 2 fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr lahf_lm bogomips : 4750.69 clflush size : 64 From dalke at dalkescientific.com Thu Aug 14 15:07:20 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Thu, 14 Aug 2008 21:07:20 +0200 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: References: <6B69DA41-6095-4D6A-96A5-588FE6A41610@dal.ca> <48A0B174.9060809@enthought.com> <48A3DB9F.9060603@gmx.de> Message-ID: <6595E426-9CFB-4471-907B-49DCF051B25E@dalkescientific.com> Anne Archibald: > Sadly, it's not possible without extra overhead. Specifically: the > NaN-ignorant implementation does a single comparison between each > array element and a placeholder, and decides based on the result which > to keep. Did my example code go through? The test for NaN only needs to be done when a new min value is found, which will occur something like O(log(n)) in a randomly distributed array. (Here's the hand-waving. The first requires a NaN check. The second has a 1/2 chance of being the new minimum. The third has a 1/3 chance, etc. The sum of the harmonic series goes as O(ln(n)).) This depends on a double inverting so the test for a new min value and a test for NaN occur at the same time. Here's pseudocode: best = array[0] if isnan(best): return best for item in array[1:]: if !(best <= item): best = item if isnan(best): return best return item > If you're willing to do two tests, sure, but that's overhead (and > probably comparable to an isnan). In Python the extra inversion costs an extra PVM instruction. In C by comparison the resulting assembly code for "best > item" and "!(best <= item)" have identical lengths, with no real performance difference. There's no extra cost for doing the extra inversion in the common case, and for large arrays the ratio of (NaN check) / (no check) -> 1.0. > What do compilers' min builtins do with NaNs? This might well be > faster than an if statement even in the absence of NaNs... This comes from a g++ implementation of min: /** * @brief This does what you think it does. * @param a A thing of arbitrary type. * @param b Another thing of arbitrary type. * @return The lesser of the parameters. * * This is the simple classic generic implementation. It will work on * temporary expressions, since they are only evaluated once, unlike a * preprocessor macro. */ template inline const _Tp& min(const _Tp& __a, const _Tp& __b) { // concept requirements __glibcxx_function_requires(_LessThanComparableConcept<_Tp>) //return __b < __a ? __b : __a; if (__b < __a) return __b; return __a; } The isnan function another version of gcc uses a bunch of #defs, leading to static __inline__ int __inline_isnanf( float __x ) { return __x != __x; } static __inline__ int __inline_isnand( double __x ) { return __x != __x; } static __inline__ int __inline_isnan( long double __x ) { return __x != __x; } Andrew dalke at dalkescientific.com From robert.kern at gmail.com Thu Aug 14 16:12:12 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 14 Aug 2008 15:12:12 -0500 Subject: [Numpy-discussion] reading *big* inhomogenous text matrices *fast*? In-Reply-To: <48A46288.20807@noaa.gov> References: <43BE1FA8-3639-4BF6-AD02-ACDA845CBBEE@yale.edu> <3d375d730808131855s7f9b9d40l6b369c88754a5d4a@mail.gmail.com> <3d375d730808131942x72116400o54831d0f5223b067@mail.gmail.com> <48A46288.20807@noaa.gov> Message-ID: <3d375d730808141312t760c86b5ke9eed80dcef6bfb9@mail.gmail.com> On Thu, Aug 14, 2008 at 11:51, Christopher Barker wrote: > > One other potential downside of using python lists to accumulate numbers > is that you are storing python objects (python ints or floats, or...) > rather than raw numbers, which has got to incur some memory overhead. > > How does array.array perform in this context? Pretty well for 1D arrays, at least. > It has an append() method, > and one would hope it uses a similar memory allocation scheme. It does. > Also, does numpy convert array.array objects to numpy arrays more > efficiently? It could, of course, but someone would have to have written > the special case code. It does. array.array() exposes the Python buffer interface. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From millman at berkeley.edu Thu Aug 14 16:14:13 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Thu, 14 Aug 2008 13:14:13 -0700 Subject: [Numpy-discussion] NumPy 1.2.0b2 released Message-ID: Hey, NumPy 1.2.0b2 is now available. Please test this so that we can uncover any problems ASAP. SVN tag: http://svn.scipy.org/svn/numpy/tags/1.2.0b2 Mac binary: https://cirl.berkeley.edu/numpy/numpy-1.2.0b2-py2.5-macosx10.5.dmg Windows binary: http://www.enthought.com/~gvaroquaux/numpy-1.2.0b2-win32.zip Source tarball: https://cirl.berkeley.edu/numpy/numpy-1.2.0b2.tar.gz Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From aisaac at american.edu Thu Aug 14 16:58:49 2008 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 14 Aug 2008 16:58:49 -0400 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: References: Message-ID: <48A49C89.6000800@american.edu> Two odd failures in test_print.py. Platform: Win XP SP3 on Intel T2600. Alan Isaac >>> np.test() Running unit tests for numpy NumPy version 1.2.0b2 NumPy is installed in C:\Python25\lib\site-packages\numpy Python version 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] nose version 0.11.0 .......................................................................................... .......................................................................................... ..............................................................FF.......................... .......................................................................................... ...........................................................................S.............. ..................................................................Ignoring "Python was bui lt with Visual Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py." (one should fix me in fcompiler/compaq.py) .......................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... ............................................................. ====================================================================== FAIL: Check formatting. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\core\tests\test_print.py", line 28, in test_complex_types assert_equal(str(t(x)), str(complex(x))) File "C:\Python25\Lib\site-packages\numpy\testing\utils.py", line 180, in assert_equal assert desired == actual, msg AssertionError: Items are not equal: ACTUAL: '(0+5.9287877500949585e-323j)' DESIRED: '(1+0j)' ====================================================================== FAIL: Check formatting. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\core\tests\test_print.py", line 16, in test_fl oat_types assert_equal(str(t(x)), str(float(x))) File "C:\Python25\Lib\site-packages\numpy\testing\utils.py", line 180, in assert_equal assert desired == actual, msg AssertionError: Items are not equal: ACTUAL: '0.0' DESIRED: '1.0' ---------------------------------------------------------------------- Ran 1567 tests in 8.234s FAILED (SKIP=1, failures=2) >>> From aisaac at american.edu Thu Aug 14 17:07:44 2008 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 14 Aug 2008 17:07:44 -0400 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <48A49C89.6000800@american.edu> References: <48A49C89.6000800@american.edu> Message-ID: <48A49EA0.7020805@american.edu> Btw, numpy loads noticeably faster. Alan From wright at esrf.fr Thu Aug 14 17:32:59 2008 From: wright at esrf.fr (Jon Wright) Date: Thu, 14 Aug 2008 23:32:59 +0200 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: References: Message-ID: <48A4A48B.1050005@esrf.fr> Jarrod Millman wrote: > Hey, > > NumPy 1.2.0b2 is now available. Please test this so that we can > uncover any problems ASAP. > > Windows binary: > http://www.enthought.com/~gvaroquaux/numpy-1.2.0b2-win32.zip > As well as the ones from Alan, if you add the "-O" for optimise flag to your python, there is still the interpreter crash as well as seeing some extra failures. -Jon ---- c:\python25\python -O -c "import numpy; numpy.test()" Running unit tests for numpy NumPy version 1.2.0b2 NumPy is installed in c:\python25\lib\site-packages\numpy Python version 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] nose version 0.10.3 ........................................................................................................................ ........................................................................................................................ ........................................F............................................................................... ...........................................................................S............................................ ....................................Ignoring "Python was built with Visual Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py." (one should fix me in fcompiler/compaq.py) ........................................................................................................................ ........................................................................................................................ ........................................................................................................................ ........................................................................................................................ ........................................................................................................................ ........................................................................................................................ ........................................................................................................................ ........................................................................................................................ .....................................................................F.F.F.F.FFFFF......... ====================================================================== FAIL: Convolve should raise an error for empty input array. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\core\tests\test_regression.py", line 626, in test_convolve_empty self.failUnlessRaises(AssertionError,np.convolve,[],[1]) AssertionError: AssertionError not raised ====================================================================== FAIL: Test two arrays with different shapes are found not equal. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 46, in test_array_diffshape self._test_not_equal(a, b) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 18, in _test_not_equal raise AssertionError("a and b are found equal but are not") AssertionError: a and b are found equal but are not ====================================================================== FAIL: Test two different array of rank 1 are found not equal. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 32, in test_array_rank1_noteq self._test_not_equal(a, b) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 18, in _test_not_equal raise AssertionError("a and b are found equal but are not") AssertionError: a and b are found equal but are not ====================================================================== FAIL: Test two arrays with different shapes are found not equal. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 46, in test_array_diffshape self._test_not_equal(a, b) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 18, in _test_not_equal raise AssertionError("a and b are found equal but are not") AssertionError: a and b are found equal but are not ====================================================================== FAIL: Test two different array of rank 1 are found not equal. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 32, in test_array_rank1_noteq self._test_not_equal(a, b) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 18, in _test_not_equal raise AssertionError("a and b are found equal but are not") AssertionError: a and b are found equal but are not ====================================================================== FAIL: Test rank 1 array for all dtypes. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 65, in test_generic_rank1 foo(t) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 61, in foo self._test_not_equal(c, b) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 18, in _test_not_equal raise AssertionError("a and b are found equal but are not") AssertionError: a and b are found equal but are not ====================================================================== FAIL: Test rank 3 array for all dtypes. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 84, in test_generic_rank3 foo(t) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 80, in foo self._test_not_equal(c, b) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 18, in _test_not_equal raise AssertionError("a and b are found equal but are not") AssertionError: a and b are found equal but are not ====================================================================== FAIL: Test arrays with nan values in them. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 98, in test_nan_array self._test_not_equal(c, b) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 18, in _test_not_equal raise AssertionError("a and b are found equal but are not") AssertionError: a and b are found equal but are not ====================================================================== FAIL: Test record arrays. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 124, in test_recarrays self._test_not_equal(c, b) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 18, in _test_not_equal raise AssertionError("a and b are found equal but are not") AssertionError: a and b are found equal but are not ====================================================================== FAIL: Test two arrays with different shapes are found not equal. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 109, in test_string_arrays self._test_not_equal(c, b) File "C:\Python25\Lib\site-packages\numpy\testing\tests\test_utils.py", line 18, in _test_not_equal raise AssertionError("a and b are found equal but are not") AssertionError: a and b are found equal but are not ---------------------------------------------------------------------- Ran 1567 tests in 8.503s FAILED (SKIP=1, failures=10) C:\> From Daniel.Lenski at seagate.com Thu Aug 14 17:38:28 2008 From: Daniel.Lenski at seagate.com (Dan Lenski) Date: Thu, 14 Aug 2008 21:38:28 +0000 (UTC) Subject: [Numpy-discussion] reading *big* inhomogenous text matrices *fast*? References: <43BE1FA8-3639-4BF6-AD02-ACDA845CBBEE@yale.edu> Message-ID: On Thu, 14 Aug 2008 04:40:16 +0000, Daniel Lenski wrote: > I assume that list-of-arrays is more memory-efficient since array > elements don't have the overhead of full-blown Python objects. But > list- of-lists is probably more time-efficient since I think it's faster > to convert the whole array at once than do it row-by-row. > > Dan Just a follow-up... Well, I tried the simple, straightforward list-of-lists approach and it's the fastest. About 20 seconds for 1.5 million cells on my machine: def _read_cells(self, f, n, debug=False): cells = dict() for i in xrange(n): cell = f.readline().split() celltype = cell.pop(2) if celltype not in cells: cells[celltype]=[] cells[celltype].append(cell) for k in cells: cells[k] = N.array(cells[k], dtype=int).T return cells List-of-arrays uses about 20% less memory, but is about 4-5 times slower (presumably due to the overhead of array creation?). And my preallocation approach is 2-3 times slower than list-of-lists. Again, I *think* this is due to array creation/conversion overhead, when assigning to a slice of the array: def _read_cells2(self, f, n, debug=False): cells = dict() count = dict() curtype = None for i in xrange(n): cell = f.readline().split() celltype = cell[2] if celltype!=curtype: curtype = celltype if curtype not in cells: cells[curtype] = N.empty((n-i, len(cell)-1), type=int) count[curtype] = 0 block = cells[curtype] block[count[curtype]] = cell[:2]+cell[3:] ### THIS LINE HERE count[curtype] += 1 for k in cells: cells[k] = cells[k][:count[k]].T return cells So my conclusion is... you guys are right. List-of-lists is the fastest way to build up an array. Then do the string-to-numeric and list-to- array conversion ALL AT ONCE with numpy.array(list_of_lists, dtype=int). Thanks for all the insight! Dan From schaffer at optonline.net Thu Aug 14 21:45:37 2008 From: schaffer at optonline.net (Les Schaffer) Date: Thu, 14 Aug 2008 21:45:37 -0400 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: References: Message-ID: <48A4DFC1.7080506@optonline.net> Jarrod Millman wrote: > Mac binary: > https://cirl.berkeley.edu/numpy/numpy-1.2.0b2-py2.5-macosx10.5.dmg > is it really necessary to label these dmg's for 10.5 only? i assume more than myself run 10.4 but have python 2.5.X installed on their machine. will this dmg install on 10.4 if py2.5 is available? thanks Les From cburns at berkeley.edu Thu Aug 14 23:41:43 2008 From: cburns at berkeley.edu (Christopher Burns) Date: Thu, 14 Aug 2008 20:41:43 -0700 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <48A4DFC1.7080506@optonline.net> References: <48A4DFC1.7080506@optonline.net> Message-ID: <764e38540808142041p56338409vf323e67ecbaee563@mail.gmail.com> On Thu, Aug 14, 2008 at 6:45 PM, Les Schaffer wrote: > is it really necessary to label these dmg's for 10.5 only? No. This is done automatically by the tool used to build the mpkg. I'll look at changing this to 10.4, thanks for the reminder. > will this dmg install on 10.4 if py2.5 is available? It should. Let us know otherwise. -- Christopher Burns Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From charlesr.harris at gmail.com Thu Aug 14 23:45:35 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 14 Aug 2008 21:45:35 -0600 Subject: [Numpy-discussion] Generalized ufuncs? Message-ID: Stefan, I notice that you have merged some new ufunc infrastructure. I think these sort of things should be discussed and reviewed on the list before being committed. Could you explain what the purpose of these patches is? The commit messages are rather skimpy. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Aug 14 23:55:59 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 14 Aug 2008 22:55:59 -0500 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: References: Message-ID: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> On Thu, Aug 14, 2008 at 22:45, Charles R Harris wrote: > Stefan, > > I notice that you have merged some new ufunc infrastructure. I think these > sort of things should be discussed and reviewed on the list before being > committed. Could you explain what the purpose of these patches is? The > commit messages are rather skimpy. St?fan happens to be in our offices this week, so he did discuss it with Travis, at least. This was actually contributed to us with extensive details from Wenjie Fu and Hans-Andreas Engel here: http://projects.scipy.org/scipy/numpy/ticket/887 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From zachary.pincus at yale.edu Fri Aug 15 00:00:09 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Fri, 15 Aug 2008 00:00:09 -0400 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <764e38540808142041p56338409vf323e67ecbaee563@mail.gmail.com> References: <48A4DFC1.7080506@optonline.net> <764e38540808142041p56338409vf323e67ecbaee563@mail.gmail.com> Message-ID: >> is it really necessary to label these dmg's for 10.5 only? > No. This is done automatically by the tool used to build the mpkg. > I'll look at changing this to 10.4, thanks for the reminder. If the dmg name is generated from the distribution name that the python distutils makes (e.g. macosx-10.5-i386-2.5), then the following may be of note: It appears that the MACOSX_DEPLOYMENT_TARGET environment variable controls (among other things) the distutils name. I generally set mine to 10.4, or even 10.3, depending on whether anything that I'm building requires later features (I'm pretty sure that numpy builds don't.) Zach On Aug 14, 2008, at 11:41 PM, Christopher Burns wrote: > On Thu, Aug 14, 2008 at 6:45 PM, Les Schaffer > wrote: >> is it really necessary to label these dmg's for 10.5 only? > > No. This is done automatically by the tool used to build the mpkg. > I'll look at changing this to 10.4, thanks for the reminder. > >> will this dmg install on 10.4 if py2.5 is available? > > It should. Let us know otherwise. > > -- > Christopher Burns > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Fri Aug 15 00:05:58 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 14 Aug 2008 22:05:58 -0600 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> References: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> Message-ID: On Thu, Aug 14, 2008 at 9:55 PM, Robert Kern wrote: > On Thu, Aug 14, 2008 at 22:45, Charles R Harris > wrote: > > Stefan, > > > > I notice that you have merged some new ufunc infrastructure. I think > these > > sort of things should be discussed and reviewed on the list before being > > committed. Could you explain what the purpose of these patches is? The > > commit messages are rather skimpy. > > St?fan happens to be in our offices this week, so he did discuss it > with Travis, at least. This was actually contributed to us with > extensive details from Wenjie Fu and Hans-Andreas Engel here: > > http://projects.scipy.org/scipy/numpy/ticket/887 > Can we fix the ticket notification mailings some day? It's been almost four months now. Re: the patch. I noticed the replacement of the signed type int by an unsigned size_t. This is a risky sort of thing and needs to be checked. Nor is it clear we should use size_t instead of one of the python or numpy types. The use of inline and the local declaration of variables would also have been caught early in a code review. So I think in this case the patch should have been discussed and reviewed on the list. An internal discussion at Enthought doesn't serve the same purposel. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Fri Aug 15 01:45:56 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 15 Aug 2008 00:45:56 -0500 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: References: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> Message-ID: <9457e7c80808142245s47b980far98c5c257d6bbd8e6@mail.gmail.com> Hi Charles 2008/8/14 Charles R Harris : > Re: the patch. I noticed the replacement of the signed type int by an > unsigned size_t. This is a risky sort of thing and needs to be checked. Nor > is it clear we should use size_t instead of one of the python or numpy > types. The use of inline and the local declaration of variables would also > have been caught early in a code review. So I think in this case the patch > should have been discussed and reviewed on the list. An internal discussion > at Enthought doesn't serve the same purposel. I apologise for not keeping the list up to date with the progress on this front. The patch is such a great contribution that I wanted it to become part of NumPy for 1.2b3. The idea was to merge it and, once done, report on the list. As is, I am still busy fixing some bugs on the Windows platform and integrating unit tests. I did, however, get Travis to review the patch beforehand, and we will keep reviewing the changes made until 1.2b3 goes out. The patch does not influence current NumPy behaviour in any way -- it simply provides hooks for general ufuncs, which can be implemented in the future. Thanks for your concern, Regards St?fan From charlesr.harris at gmail.com Fri Aug 15 01:54:55 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 14 Aug 2008 23:54:55 -0600 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: <9457e7c80808142245s47b980far98c5c257d6bbd8e6@mail.gmail.com> References: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> <9457e7c80808142245s47b980far98c5c257d6bbd8e6@mail.gmail.com> Message-ID: On Thu, Aug 14, 2008 at 11:45 PM, St?fan van der Walt wrote: > Hi Charles > > 2008/8/14 Charles R Harris : > > Re: the patch. I noticed the replacement of the signed type int by an > > unsigned size_t. This is a risky sort of thing and needs to be checked. > Nor > > is it clear we should use size_t instead of one of the python or numpy > > types. The use of inline and the local declaration of variables would > also > > have been caught early in a code review. So I think in this case the > patch > > should have been discussed and reviewed on the list. An internal > discussion > > at Enthought doesn't serve the same purposel. > > I apologise for not keeping the list up to date with the progress on > this front. The patch is such a great contribution that I wanted it > to become part of NumPy for 1.2b3. The idea was to merge it and, once Numpy 1.2 is for documentation, bug fixes, and getting the new testing framework in place. Discipline is called for if we are going to have timely releases. > > done, report on the list. Wrong way around. > As is, I am still busy fixing some bugs on > the Windows platform and integrating unit tests. I did, however, get > Travis to review the patch beforehand, and we will keep reviewing the > changes made until 1.2b3 goes out. We is the numpy community, not you and Travis. > The patch does not influence > current NumPy behaviour in any way -- it simply provides hooks for > general ufuncs, which can be implemented in the future. > Why not wait until after the release then? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Fri Aug 15 02:16:01 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 15 Aug 2008 01:16:01 -0500 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: References: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> <9457e7c80808142245s47b980far98c5c257d6bbd8e6@mail.gmail.com> Message-ID: <48A51F21.6040208@enthought.com> > > Numpy 1.2 is for documentation, bug fixes, and getting the new testing > framework in place. Discipline is called for if we are going to have > timely releases. We also agreed to a change in the C-API (or at least did not object too loudly). I'm in favor of minimizing that sort of change. > > > Why not wait until after the release then? The biggest reason is that the patch requires changing the C-API and we are already doing that for 1.2. I would rather not do it again for another 6 months at least. I don't think we should make the patch wait that long. Your code review is very much appreciated. -Travis From millman at berkeley.edu Fri Aug 15 02:19:33 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Thu, 14 Aug 2008 23:19:33 -0700 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: References: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> <9457e7c80808142245s47b980far98c5c257d6bbd8e6@mail.gmail.com> Message-ID: On Thu, Aug 14, 2008 at 10:54 PM, Charles R Harris wrote: > Numpy 1.2 is for documentation, bug fixes, and getting the new testing > framework in place. Discipline is called for if we are going to have timely > releases. First, all your points are very valid. And I apologize for the role I played in this. Thanks for calling us on it. That said while you are correct that this release is mainly about documentation, bug fixes, and getting the new testing framework in place, there are several other things that have gone in. Their have been a few planned API changes and even a C-API change. Travis emailed me asking where we were on the beta release and whether we should discuss including this change on the list. I contacted Stefan and asked him if he could do me a huge favor and see if we could quickly apply the patch before making the beta release. My reasoning was that this looked very good and useful and just offered something new. Stefan was hesitant, but I persisted. He didn't like that it didn't have any tests, but I said if he put it in time for the beta he could add tests afterward. I wanted to make sure no new features got in after a beta. Also we are all ready requiring recompiling with this release, so I thought now would be a good time to add it. > We is the numpy community, not you and Travis. Absolutely. There were several of us involved, not just Travis and Stefan. But that is no excuse. Stefan, David, Chris, and I have been trying very hard to get the beta out over the last few days and had started talking among ourselves since we were mostly just coordinating. Taking that over to feature adding was a mistake. > Why not wait until after the release then? The motivation is that we are not allowing features in bugfix releases anymore. So it can't go in in 1.2.x if it isn't in 1.2.0. I also want to get several 1.2.x releases out. That means the earliest we could get it in is 1.3.0. But I would prefer not having to require recompiling extension code with every minor release. Sorry. This was handled poorly. But I think this would still be very useful and I would like to see it get in. We were planning on releasing a 1.2.0b3 early next week. But this is it, I promise. How about we work on it and see where we are early next week. If it doesn't look good, we can pull it. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From oliphant at enthought.com Fri Aug 15 02:28:56 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 15 Aug 2008 01:28:56 -0500 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: References: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> Message-ID: <48A52228.5070106@enthought.com> > > Can we fix the ticket notification mailings some day? It's been almost > four months now. That would be fabulous. So far nobody has figured out how... Jarrod?? > > Re: the patch. I noticed the replacement of the signed type int by an > unsigned size_t. Where did you notice this? I didn't see it. > python or numpy types. The use of inline and the local declaration of > variables would also have been caught early in a code review. What do you mean by the local declaration of variables? -Travis From dalke at dalkescientific.com Fri Aug 15 02:34:54 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Fri, 15 Aug 2008 08:34:54 +0200 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <48A49EA0.7020805@american.edu> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> Message-ID: <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> On Aug 14, 2008, at 11:07 PM, Alan G Isaac wrote: > Btw, numpy loads noticeably faster. Any chance of someone reviewing my suggestions for making the import somewhat faster still? http://scipy.org/scipy/numpy/ticket/874 Andrew dalke at dalkescientific.com From oliphant at enthought.com Fri Aug 15 02:35:37 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 15 Aug 2008 01:35:37 -0500 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: <48A52228.5070106@enthought.com> References: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> <48A52228.5070106@enthought.com> Message-ID: <48A523B9.104@enthought.com> Travis E. Oliphant wrote: >> Can we fix the ticket notification mailings some day? It's been almost >> four months now. >> > That would be fabulous. So far nobody has figured out how... Jarrod?? > >> Re: the patch. I noticed the replacement of the signed type int by an >> unsigned size_t. >> > Where did you notice this? I didn't see it. > Are you referring to Stefan's patch to the Fu's _parse_signature code in r5654. This is a local function, I'm not sure why there is a concern. >> python or numpy types. The use of inline and the local declaration of >> variables would also have been caught early in a code review. >> > What do you mean by the local declaration of variables? > > Never mind, I understand it's the mid-code declaration of variables (without a separate block defined) that Stefan fixed. -Travis From charlesr.harris at gmail.com Fri Aug 15 02:36:14 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 15 Aug 2008 00:36:14 -0600 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: <48A52228.5070106@enthought.com> References: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> <48A52228.5070106@enthought.com> Message-ID: On Fri, Aug 15, 2008 at 12:28 AM, Travis E. Oliphant wrote: > > > > > Can we fix the ticket notification mailings some day? It's been almost > > four months now. > That would be fabulous. So far nobody has figured out how... Jarrod?? > > > > Re: the patch. I noticed the replacement of the signed type int by an > > unsigned size_t. > Where did you notice this? I didn't see it. r5654. > > > python or numpy types. The use of inline and the local declaration of > > variables would also have been caught early in a code review. > What do you mean by the local declaration of variables? > r5653, gcc allows variables to be declared where used rather than at the beginning of a block. I believe this is part of a recent (proposed?) standard, but will fail for other compilers. The inline keyword also tends to be gcc/icc specific, although it is part of the C99 standard. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Aug 15 02:42:26 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 15 Aug 2008 00:42:26 -0600 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: <48A523B9.104@enthought.com> References: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> <48A52228.5070106@enthought.com> <48A523B9.104@enthought.com> Message-ID: On Fri, Aug 15, 2008 at 12:35 AM, Travis E. Oliphant wrote: > Travis E. Oliphant wrote: > >> Can we fix the ticket notification mailings some day? It's been almost > >> four months now. > >> > > That would be fabulous. So far nobody has figured out how... Jarrod?? > > > >> Re: the patch. I noticed the replacement of the signed type int by an > >> unsigned size_t. > >> > > Where did you notice this? I didn't see it. > > > Are you referring to Stefan's patch to the Fu's _parse_signature code in > r5654. This is a local function, I'm not sure why there is a concern. > There probably isn't a problem, but the use of unsigned types in loop counters and such can lead to subtle errors, so when a signed type is changed to an unsigned type the code has to be audited to make sure there won't be any unintended consequences. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalke at dalkescientific.com Fri Aug 15 02:45:59 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Fri, 15 Aug 2008 08:45:59 +0200 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: References: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> <48A52228.5070106@enthought.com> Message-ID: <5AC544D3-0161-47C3-AA64-4BE758E68625@dalkescientific.com> On Aug 15, 2008, at 8:36 AM, Charles R Harris wrote: > The inline keyword also tends to be gcc/icc specific, although it > is part of the C99 standard. For reference, a page on using inline and doing so portably: http://www.greenend.org.uk/rjk/2003/03/inline.html Andrew dalke at dalkescientific.com From oliphant at enthought.com Fri Aug 15 03:00:31 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 15 Aug 2008 02:00:31 -0500 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> Message-ID: <48A5298F.80701@enthought.com> Andrew Dalke wrote: > On Aug 14, 2008, at 11:07 PM, Alan G Isaac wrote: > >> Btw, numpy loads noticeably faster. >> > > > Any chance of someone reviewing my suggestions for > making the import somewhat faster still? > > http://scipy.org/scipy/numpy/ticket/874 > > So, what is the attitude of people here? Here's my take: 1) Removing ctypeslib import * Can break code if somebody has been doing import numpy and then using numpy.ctypeslib * I'm fine with people needing to import numpy.ctypeslib to use the capability as long as we clearly indicate these breakages. 2&3) I think defering imports for _datasource.py is a great idea. 4) Removing "import doc" * This was added recently I think. I'm not sure why it's there, but it might be there as part of the documentation effort. It should be imported with ipython, probably, but not by default. 5) The testing code seems like a lot of duplication to save .01 seconds 6) Remove unused glob --- fine. 7 - 9) These seem fine. In sum: I think 2, 3, 6, 7, 8, and 9 can be done immediately. 1) and 4) could be O.K. but 1) does break code and 4 I'm not sure about. 5 seems like it's too much code duplication for too little savings for my taste. We need to push of the release of 1.2 I think. We are rushing to get it out by SciPy and it is causing some rushing of collaboration so that people who would like to comment are feeling that they can't or that their comments are not desired or valued. I'm sorry for what I've done that might have left that impression. -Travis O. From charlesr.harris at gmail.com Fri Aug 15 03:02:04 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 15 Aug 2008 01:02:04 -0600 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: <5AC544D3-0161-47C3-AA64-4BE758E68625@dalkescientific.com> References: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> <48A52228.5070106@enthought.com> <5AC544D3-0161-47C3-AA64-4BE758E68625@dalkescientific.com> Message-ID: On Fri, Aug 15, 2008 at 12:45 AM, Andrew Dalke wrote: > On Aug 15, 2008, at 8:36 AM, Charles R Harris wrote: > > The inline keyword also tends to be gcc/icc specific, although it > > is part of the C99 standard. > > > For reference, a page on using inline and doing so portably: > > http://www.greenend.org.uk/rjk/2003/03/inline.html Doesn't do the trick for compilers that aren't C99 compliant. And there are many of them. For gcc there are other options. -finline-functionsIntegrate all simple functions into their callers. The compiler heuristically decides which functions are simple enough to be worth integrating in this way. If all calls to a given function are integrated, and the function is declared static, then the function is normally not output as assembler code in its own right. Enabled at level -O3. -finline-functions-called-onceConsider all static functions called once for inlining into their caller even if they are not marked inline. If a call to a given function is integrated, then the function is not output as assembler code in its own right. Enabled if -funit-at-a-time is enabled. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Fri Aug 15 03:02:19 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 15 Aug 2008 02:02:19 -0500 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: References: Message-ID: <9457e7c80808150002p68043553l83cd06b3bcb3c77a@mail.gmail.com> 2008/8/14 Charles R Harris : > I notice that you have merged some new ufunc infrastructure. I think these For those of you who were wondering, this is the ticket that this conversation refers to: http://projects.scipy.org/scipy/numpy/ticket/887 St?fan From millman at berkeley.edu Fri Aug 15 03:04:02 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Fri, 15 Aug 2008 00:04:02 -0700 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: References: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> Message-ID: On Thu, Aug 14, 2008 at 9:05 PM, Charles R Harris wrote: > Can we fix the ticket notification mailings some day? It's been almost four > months now. It should work now. Let me know if you aren't getting them now. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From charlesr.harris at gmail.com Fri Aug 15 03:09:39 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 15 Aug 2008 01:09:39 -0600 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: References: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> Message-ID: On Fri, Aug 15, 2008 at 1:04 AM, Jarrod Millman wrote: > On Thu, Aug 14, 2008 at 9:05 PM, Charles R Harris > wrote: > > Can we fix the ticket notification mailings some day? It's been almost > four > > months now. > > It should work now. Let me know if you aren't getting them now. > Thanks, it seems to be working now. What did you do? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalke at dalkescientific.com Fri Aug 15 03:11:20 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Fri, 15 Aug 2008 09:11:20 +0200 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <48A5298F.80701@enthought.com> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> Message-ID: On Aug 15, 2008, at 9:00 AM, Travis E. Oliphant wrote: > 5) The testing code seems like a lot of duplication to save .01 > seconds Personally I want to get rid of all in-body test code and use nosetests or something similar. I know that's not going to happen at the very least because I was told that people have been told for years to test numpy using: import numpy; numpy.test() Therefore my second choice is to only implement that top-level function (using deferred imports) and to get rid of all of the other test() and bench() functions. This patch is actually my third choice - full compatibility - but it's easier to trim code from a patch than it is to add code to a patch, so I submitted it that way. Andrew dalke at dalkescientific.com From stefan at sun.ac.za Fri Aug 15 03:15:28 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 15 Aug 2008 02:15:28 -0500 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: References: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> Message-ID: <9457e7c80808150015i2b6f62c3p3034c23448dfc698@mail.gmail.com> 2008/8/15 Jarrod Millman : > On Thu, Aug 14, 2008 at 9:05 PM, Charles R Harris > wrote: >> Can we fix the ticket notification mailings some day? It's been almost four >> months now. > > It should work now. Let me know if you aren't getting them now. Thanks, Jarrod. That is a great improvement! Cheers St?fan From charlesr.harris at gmail.com Fri Aug 15 03:15:32 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 15 Aug 2008 01:15:32 -0600 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: <48A51F21.6040208@enthought.com> References: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> <9457e7c80808142245s47b980far98c5c257d6bbd8e6@mail.gmail.com> <48A51F21.6040208@enthought.com> Message-ID: On Fri, Aug 15, 2008 at 12:16 AM, Travis E. Oliphant wrote: > > > > > Numpy 1.2 is for documentation, bug fixes, and getting the new testing > > framework in place. Discipline is called for if we are going to have > > timely releases. > We also agreed to a change in the C-API (or at least did not object too > loudly). I'm in favor of minimizing that sort of change. I thought it a bit iffy, but didn't think it worth objecting to a single change. But it seems to have been the top of the 'ol slippery slope. > > > > > > > Why not wait until after the release then? > The biggest reason is that the patch requires changing the C-API and we > are already doing that for 1.2. I would rather not do it again for > another 6 months at least. I don't think we should make the patch wait > that long. > Rushing to a deadline can have the unfortunate side effect of hasty decisions and regret savored at leisure. But the most important thing is the principal of least surprise. That is why it is important to publicise a change *even if no one objects to it*. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalke at dalkescientific.com Fri Aug 15 03:18:52 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Fri, 15 Aug 2008 09:18:52 +0200 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: References: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> <48A52228.5070106@enthought.com> <5AC544D3-0161-47C3-AA64-4BE758E68625@dalkescientific.com> Message-ID: <427D3C8A-EC36-491E-9A5E-32EDD1D3E7E8@dalkescientific.com> Andrew Dalke: > For reference, a page on using inline and doing so portably: > > http://www.greenend.org.uk/rjk/2003/03/inline.html On Aug 15, 2008, at 9:02 AM, Charles R Harris wrote: > Doesn't do the trick for compilers that aren't C99 compliant. And > there are many of them. For gcc there are other options. Pardon? It seems to list several ways to handle compilers that don't implement C99. Mostly by having a single INLINE define in one form or another to trigger the correct support. For example: You can support legacy compilers (i.e. anything without "inline") via -Dinline="", although this wastes space. The gcc options like -finline-functions and -finline-functions-called- once don't affect the grammar and don't seem relevant here. Andrew dalke at dalkescientific.com From millman at berkeley.edu Fri Aug 15 03:19:21 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Fri, 15 Aug 2008 00:19:21 -0700 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: References: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> Message-ID: On Fri, Aug 15, 2008 at 12:09 AM, Charles R Harris wrote: > Thanks, it seems to be working now. What did you do? It was an incorrect spam filter in mailman. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From charlesr.harris at gmail.com Fri Aug 15 03:28:52 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 15 Aug 2008 01:28:52 -0600 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: <427D3C8A-EC36-491E-9A5E-32EDD1D3E7E8@dalkescientific.com> References: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> <48A52228.5070106@enthought.com> <5AC544D3-0161-47C3-AA64-4BE758E68625@dalkescientific.com> <427D3C8A-EC36-491E-9A5E-32EDD1D3E7E8@dalkescientific.com> Message-ID: On Fri, Aug 15, 2008 at 1:18 AM, Andrew Dalke wrote: > Andrew Dalke: > > For reference, a page on using inline and doing so portably: > > > > http://www.greenend.org.uk/rjk/2003/03/inline.html > > On Aug 15, 2008, at 9:02 AM, Charles R Harris wrote: > > Doesn't do the trick for compilers that aren't C99 compliant. And > > there are many of them. For gcc there are other options. > > Pardon? It seems to list several ways to handle compilers that don't > implement C99. Mostly by having a single INLINE define in one form > or another to trigger the correct support. > > For example: > > You can support legacy compilers (i.e. anything without "inline") > via -Dinline="", although this wastes space. > I missed that on a quick scan. ISTR reading the article before... > > The gcc options like -finline-functions and -finline-functions-called- > once don't affect the grammar and don't seem relevant here. > > They are flags, so can be made part of your local CFLAGS or part of the compiler specific options. I use the first option to get inlining of the string comparisons in the sort functions. It would be nice if MSVC, the main offender against C99, could be relied on, and maybe the newest versions are an improvement. But until then it is best to be conservative. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Fri Aug 15 03:30:49 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 15 Aug 2008 02:30:49 -0500 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <48A5298F.80701@enthought.com> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> Message-ID: <9457e7c80808150030q614d74c2j1be4f93b425e711d@mail.gmail.com> 2008/8/15 Travis E. Oliphant : > So, what is the attitude of people here? Here's my take: I wonder if we are going about this process the right way. We will forever be adjusting imports to improve load times. Why don't we provide an alternate API, something like numpy.api from which you can import exactly what you want? We can disable populating the full numpy namespace by exporting an environment variable or setting some flag. This way, importing numpy is instantaneous, whilst our interactive users still get the full benefit of having everything available? St?fan From robert.kern at gmail.com Fri Aug 15 03:39:58 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 15 Aug 2008 02:39:58 -0500 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <9457e7c80808150030q614d74c2j1be4f93b425e711d@mail.gmail.com> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <9457e7c80808150030q614d74c2j1be4f93b425e711d@mail.gmail.com> Message-ID: <3d375d730808150039u34f5b28sa401f58af1161e6@mail.gmail.com> On Fri, Aug 15, 2008 at 02:30, St?fan van der Walt wrote: > 2008/8/15 Travis E. Oliphant : >> So, what is the attitude of people here? Here's my take: > > I wonder if we are going about this process the right way. We will > forever be adjusting imports to improve load times. Why don't we > provide an alternate API, something like numpy.api from which you can > import exactly what you want? We can disable populating the full > numpy namespace by exporting an environment variable or setting some > flag. This way, importing numpy is instantaneous, whilst our > interactive users still get the full benefit of having everything > available? The devil is in the details. What exactly do you propose? When we discussed this last time, the participants more or less agreed that environment variables could cause more fragility than they're worth. It also breaks the first time you try to import a numpy-using library that was not written with this in mind. Basically, you're stuck with only code that you've written. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stefan at sun.ac.za Fri Aug 15 03:59:43 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 15 Aug 2008 02:59:43 -0500 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <3d375d730808150039u34f5b28sa401f58af1161e6@mail.gmail.com> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <9457e7c80808150030q614d74c2j1be4f93b425e711d@mail.gmail.com> <3d375d730808150039u34f5b28sa401f58af1161e6@mail.gmail.com> Message-ID: <9457e7c80808150059t22bc7c74t62f319e29de4f5d2@mail.gmail.com> 2008/8/15 Robert Kern : > The devil is in the details. What exactly do you propose? When we > discussed this last time, the participants more or less agreed that > environment variables could cause more fragility than they're worth. > It also breaks the first time you try to import a numpy-using library > that was not written with this in mind. Basically, you're stuck with > only code that you've written. First, I propose that I write some code. Second, I do not suggest the behaviour above, but: 1) Expose a new interface to numpy, called numpy.api 2) If a certain environment variable is set, the numpy namespace is not populated, and numpy.api becomes instantaneous to load. Even if the user forgets to set the variable, everything works as planned. If the user is aware of the variable, he won't be using numpy the normal way, so the fact that numpy.* is not available won't matter. Cheers St?fan From robert.kern at gmail.com Fri Aug 15 04:24:15 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 15 Aug 2008 03:24:15 -0500 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <9457e7c80808150059t22bc7c74t62f319e29de4f5d2@mail.gmail.com> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <9457e7c80808150030q614d74c2j1be4f93b425e711d@mail.gmail.com> <3d375d730808150039u34f5b28sa401f58af1161e6@mail.gmail.com> <9457e7c80808150059t22bc7c74t62f319e29de4f5d2@mail.gmail.com> Message-ID: <3d375d730808150124v7f61ff4fn35e0e392816db2d9@mail.gmail.com> On Fri, Aug 15, 2008 at 02:59, St?fan van der Walt wrote: > 2008/8/15 Robert Kern : >> The devil is in the details. What exactly do you propose? When we >> discussed this last time, the participants more or less agreed that >> environment variables could cause more fragility than they're worth. >> It also breaks the first time you try to import a numpy-using library >> that was not written with this in mind. Basically, you're stuck with >> only code that you've written. > > First, I propose that I write some code. Second, I do not suggest the > behaviour above, but: > > 1) Expose a new interface to numpy, called numpy.api > 2) If a certain environment variable is set, the numpy namespace is > not populated, and numpy.api becomes instantaneous to load. > > Even if the user forgets to set the variable, everything works as > planned. If the user is aware of the variable, he won't be using > numpy the normal way, so the fact that numpy.* is not available won't > matter. I'm afraid that I still don't understand. Please expand on the following four cases (let's call the environment variable NUMPY_FAST_IMPORT): 1) NUMPY_FAST_IMPORT=0 (or simply absent) import numpy print dir(numpy) 2) NUMPY_FAST_IMPORT=0 import numpy.api print dir(numpy.api) 3) NUMPY_FAST_IMPORT=1 import numpy print dir(numpy) 4) NUMPY_FAST_IMPORT=1 import numpy.api print dir(numpy.api) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stefan at sun.ac.za Fri Aug 15 04:44:39 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 15 Aug 2008 03:44:39 -0500 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <3d375d730808150124v7f61ff4fn35e0e392816db2d9@mail.gmail.com> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <9457e7c80808150030q614d74c2j1be4f93b425e711d@mail.gmail.com> <3d375d730808150039u34f5b28sa401f58af1161e6@mail.gmail.com> <9457e7c80808150059t22bc7c74t62f319e29de4f5d2@mail.gmail.com> <3d375d730808150124v7f61ff4fn35e0e392816db2d9@mail.gmail.com> Message-ID: <9457e7c80808150144h2a8dd4e8ta007639a561c44d6@mail.gmail.com> 2008/8/15 Robert Kern : > I'm afraid that I still don't understand. Please expand on the Sorry, it's late. My explanation is probably not too lucid. The variable should rather read something like NUMPY_VIA_API, but here goes. > 1) NUMPY_FAST_IMPORT=0 (or simply absent) > import numpy > print dir(numpy) Full numpy import, exactly as it is now. > 2) NUMPY_FAST_IMPORT=0 > import numpy.api > print dir(numpy.api) Numpy.*, exactly as it is now. numpy.api provides a more nested API to NumPy. Import time is the same as current NumPy import. > 3) NUMPY_FAST_IMPORT=1 > import numpy > print dir(numpy) > 4) NUMPY_FAST_IMPORT=1 > import numpy.api > print dir(numpy.api) numpy.* is now probably close to empty. numpy.api is accessible as before. Import time for numpy.api is now super snappy since numpy.* is not populated. If this is not clear, then I need to sleep and implement a proof of concept before I try to explain further. Cheers St?fan From wright at esrf.fr Fri Aug 15 07:21:03 2008 From: wright at esrf.fr (Jon Wright) Date: Fri, 15 Aug 2008 13:21:03 +0200 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: References: Message-ID: <48A5669F.3080007@esrf.fr> Jarrod Millman wrote: > Hey, > > NumPy 1.2.0b2 is now available. Please test this so that we can > uncover any problems ASAP. > > Windows binary: > http://www.enthought.com/~gvaroquaux/numpy-1.2.0b2-win32.zip > Hello Again, It seems the new release breaks matplotlib, for those pauvres who are using pre-compiled at least. If this means all C-modules compiled against numpy have to be recompiled, then this will make me very unhappy. -Jon ---- H:\>c:\python25\python -c "import matplotlib; print matplotlib.__version__; import matplotlib.pylab" 0.98.3 RuntimeError: module compiled against version 1000009 of C-API but this version of numpy is 100000a Traceback (most recent call last): File "", line 1, in File "C:\Python25\Lib\site-packages\matplotlib\pylab.py", line 206, in from matplotlib import mpl # pulls in most modules File "C:\Python25\Lib\site-packages\matplotlib\mpl.py", line 1, in from matplotlib import artist File "C:\Python25\Lib\site-packages\matplotlib\artist.py", line 4, in from transforms import Bbox, IdentityTransform, TransformedBbox, TransformedPath File "C:\Python25\Lib\site-packages\matplotlib\transforms.py", line 34, in from matplotlib._path import affine_transform ImportError: numpy.core.multiarray failed to import From jh at physics.ucf.edu Fri Aug 15 08:12:09 2008 From: jh at physics.ucf.edu (Joe Harrington) Date: Fri, 15 Aug 2008 08:12:09 -0400 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: (numpy-discussion-request@scipy.org) References: Message-ID: > If you're willing to do arithmetic you might even be able to > pull it off, since NaNs tend to propagate: > if (new Whether the speed of this is worth its impenetrability I couldn't say. Code comments cure impenetrability, and have no cost in speed. One could write a paragraph explaining it (if it really needed that much). The comments could even reference the current discussion. --jh-- From cournape at gmail.com Fri Aug 15 08:29:12 2008 From: cournape at gmail.com (David Cournapeau) Date: Fri, 15 Aug 2008 07:29:12 -0500 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: <48A51F21.6040208@enthought.com> References: <3d375d730808142055q7ae4e4cevcc3dfca6212bcfcb@mail.gmail.com> <9457e7c80808142245s47b980far98c5c257d6bbd8e6@mail.gmail.com> <48A51F21.6040208@enthought.com> Message-ID: <5b8d13220808150529i16e1e746qfdb7489cca931894@mail.gmail.com> On Fri, Aug 15, 2008 at 1:16 AM, Travis E. Oliphant wrote: > > The biggest reason is that the patch requires changing the C-API and we > are already doing that for 1.2. I would rather not do it again for > another 6 months at least. I don't think we should make the patch wait > that long. I understand the concern, but that should have been discussed I think. Changing C code affects most process wrt releases, not just people concerned with API stability. From my POV, recent C code caused me a lot of trouble wrt binaries building. If we keep changing C code during the beta, I won't be able to follow. The problem I see with any C (not necessarily C API) change is that they can break a lot of things. For example, I did not notice, but several generated code (umath stuff, mtrand) break Visual Studio compilation because of too long strings. If we accept changes in the C code during the beta phase, it just does not mean much to have beta. The point to have a time-based release is to enforce this kind of things; if we don't, then not only time-based release do not make sense, but they make those problem even worse (no benefit, and we rush things out which do not work). I see a lot of bugs in scipy/numpy, matplotlib problems in the last few days report on the ML and trac. Putting non bug fix-related C code will make this an ever-ending battle. cheers, David From bsouthey at gmail.com Fri Aug 15 09:59:50 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 15 Aug 2008 08:59:50 -0500 Subject: [Numpy-discussion] Different results from repeated calculation, part 2 In-Reply-To: References: <48A47989.8090904@gmail.com> Message-ID: <48A58BD6.4090903@gmail.com> Keith Goodman wrote: > On Thu, Aug 14, 2008 at 11:29 AM, Bruce Southey wrote: > >> Keith Goodman wrote: >> >>> I get slightly different results when I repeat a calculation. >>> >>> I've seen this problem before (it went away but has returned): >>> >>> http://projects.scipy.org/pipermail/numpy-discussion/2007-January/025724.html >>> >>> A unit test is attached. It contains three tests: >>> >>> In test1, I construct matrices x and y and then repeatedly calculate z >>> = calc(x,y). The result z is the same every time. So this test passes. >>> >>> In test2, I construct matrices x and y each time before calculating z >>> = calc(x,y). Sometimes z is slightly different. But the x's test to be >>> equal and so do the y's. This test fails (on Debian Lenny, Core 2 Duo, >>> with libatlas3gf-sse2 but not with libatlas3gf-sse). >>> >>> test3 is the same as test2 but I calculate z like this: z = >>> calc(100*x,y) / (100 * 100). This test passes. >>> >>> I get: >>> >>> ====================================================================== >>> FAIL: repeatability #2 >>> ---------------------------------------------------------------------- >>> Traceback (most recent call last): >>> File "/home/[snip]/test/repeat_test.py", line 73, in test_repeat_2 >>> self.assert_(result, msg) >>> AssertionError: Max difference = 2.04946e-16 >>> >>> ---------------------------------------------------------------------- >>> >>> Should a unit test like this be added to numpy? >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> Numpy-discussion mailing list >>> Numpy-discussion at scipy.org >>> http://projects.scipy.org/mailman/listinfo/numpy-discussion >>> >> Hi, >> In the function 'test_repeat_2' you are redefining variables 'x and y' >> that were first defined using the setup function. (Also, you are not >> using the __init__ function.) I vaguely recall there are some quirks to >> Python classes with this, so does the problem go away with if you use >> 'a,b' instead of 'x, y'? (I suspect the answer is yes given test_repeat_3). >> >> Note that you should also test that 'x' and 'y' are same here as well >> (but these have been redefined...). >> >> Otherwise, can you please provide your OS (version), computer processor, >> Python version, numpy version, version of atlas (or similar) and >> compiler used? >> >> I went back and reread the thread but I could not see this information. >> > > Here's a test that doesn't use classes and checks that x and y do not change: > > http://projects.scipy.org/pipermail/numpy-discussion/attachments/20070127/52b3a51c/attachment.py > > I'm using binaries from Debian Lenny: > > $ uname -a > Linux jan 2.6.25-2-686 #1 SMP Fri Jul 18 17:46:56 UTC 2008 i686 GNU/Linux > > $ python -V > Python 2.5.2 > > >>> numpy.__version__ >>> > '1.1.0' > > $ cat /proc/cpuinfo > processor : 0 > vendor_id : GenuineIntel > cpu family : 6 > model : 15 > model name : Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz > stepping : 6 > cpu MHz : 2402.004 > cache size : 4096 KB > physical id : 0 > siblings : 2 > core id : 0 > cpu cores : 2 > fdiv_bug : no > hlt_bug : no > f00f_bug : no > coma_bug : no > fpu : yes > fpu_exception : yes > cpuid level : 10 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov > pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm > constant_tsc arch_perfmon pebs bts pni monitor ds_cpl vmx est tm2 > ssse3 cx16 xtpr lahf_lm > bogomips : 4807.45 > clflush size : 64 > > processor : 1 > vendor_id : GenuineIntel > cpu family : 6 > model : 15 > model name : Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz > stepping : 6 > cpu MHz : 2402.004 > cache size : 4096 KB > physical id : 0 > siblings : 2 > core id : 1 > cpu cores : 2 > fdiv_bug : no > hlt_bug : no > f00f_bug : no > coma_bug : no > fpu : yes > fpu_exception : yes > cpuid level : 10 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov > pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm > constant_tsc arch_perfmon pebs bts pni monitor ds_cpl vmx est tm2 > ssse3 cx16 xtpr lahf_lm > bogomips : 4750.69 > clflush size : 64 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > I do not get this on my Intel Quad core2 Linux x64 system with a x86_64 running Fedora 10 supplied Python. I do compile my own versions of NumPy and currently don't use or really plan to use altas. But I know that you previously indicated that this was atlas related (http://projects.scipy.org/pipermail/numpy-discussion/2007-January/025750.html) . From Intel's website, the Intel Core2 Duo E6600 (http://processorfinder.intel.com/details.aspx?sSpec=SL9S8) supports EM64T so it is x86 64-bit processor. I do not know Debian but i686 generally refers to 32-bit kernel as x86_64 refers to 64-bit. If so, then you are running a 32-bits on 64-bit processor. So I would suggest you start by compiling your own NumPy without any extras and see if it goes away. If not, then it is NumPy otherwise add the extras until you get the same system back. Bruce From gael.varoquaux at normalesup.org Fri Aug 15 09:30:20 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 15 Aug 2008 15:30:20 +0200 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <9457e7c80808150059t22bc7c74t62f319e29de4f5d2@mail.gmail.com> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <9457e7c80808150030q614d74c2j1be4f93b425e711d@mail.gmail.com> <3d375d730808150039u34f5b28sa401f58af1161e6@mail.gmail.com> <9457e7c80808150059t22bc7c74t62f319e29de4f5d2@mail.gmail.com> Message-ID: <20080815133020.GD18337@phare.normalesup.org> On Fri, Aug 15, 2008 at 02:59:43AM -0500, St?fan van der Walt wrote: > 2008/8/15 Robert Kern : > > The devil is in the details. What exactly do you propose? When we > > discussed this last time, the participants more or less agreed that > > environment variables could cause more fragility than they're worth. > > It also breaks the first time you try to import a numpy-using library > > that was not written with this in mind. Basically, you're stuck with > > only code that you've written. > First, I propose that I write some code. Second, I do not suggest the > behaviour above, but: > 1) Expose a new interface to numpy, called numpy.api > 2) If a certain environment variable is set, the numpy namespace is > not populated, and numpy.api becomes instantaneous to load. That doesn't work because of a "feature" in Python's import: when loading foo.bar, Python loads foo.__init__ first. This is why we have "api" modules all over ETS. Ga?l From pav at iki.fi Fri Aug 15 10:38:10 2008 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 15 Aug 2008 14:38:10 +0000 (UTC) Subject: [Numpy-discussion] NumPy 1.2.0b2 released References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <9457e7c80808150030q614d74c2j1be4f93b425e711d@mail.gmail.com> <3d375d730808150039u34f5b28sa401f58af1161e6@mail.gmail.com> <9457e7c80808150059t22bc7c74t62f319e29de4f5d2@mail.gmail.com> <20080815133020.GD18337@phare.normalesup.org> Message-ID: Fri, 15 Aug 2008 15:30:20 +0200, Gael Varoquaux wrote: > On Fri, Aug 15, 2008 at 02:59:43AM -0500, St?fan van der Walt wrote: [clip] >> 1) Expose a new interface to numpy, called numpy.api 2) If a certain >> environment variable is set, the numpy namespace is not populated, and >> numpy.api becomes instantaneous to load. > > That doesn't work because of a "feature" in Python's import: when > loading foo.bar, Python loads foo.__init__ first. This is why we have > "api" modules all over ETS. I think you can still do something evil, like this: import os if os.environ.get('NUMPY_VIA_API', '0') != '0': from numpy.lib.fromnumeric import * ... But I'm not sure how many milliseconds must be gained to justify this... -- Pauli Virtanen From David.Kaplan at ird.fr Fri Aug 15 10:54:24 2008 From: David.Kaplan at ird.fr (David M. Kaplan) Date: Fri, 15 Aug 2008 16:54:24 +0200 Subject: [Numpy-discussion] patch for new mgrid / ogrid functionality Message-ID: <1218812064.7194.17.camel@localhost> Hi, A while back, I sent some changes to index_tricks.py that would allow mgrid and ogrid to mesh things other than slices. For example: >>> mgrid[['a','b'],[float,int],:3] [array([[['a', 'a', 'a'], ['a', 'a', 'a']], [['b', 'b', 'b'], ['b', 'b', 'b']]], dtype='|S1'), array([[[, , ], [, , ]], [[, , ], [, , ]]], dtype=object), array([[[0, 1, 2], [0, 1, 2]], [[0, 1, 2], [0, 1, 2]]])] At the time, there wasn't much follow-up, but I am hoping that there is still interest in this functionality, as I have gone ahead and finished the patch including documentation changes and updates to test_index_tricks.py. Attached is a patch set to the latest subversion of the numpy trunk. I don't think I am allowed to commit the changes myself - correct me if I am wrong. This functionality seems like a nice addition to me as it allows one to mesh things that are not uniformly spaced and potentially not even numbers. The changes don't affect functionality that existed previously except for one minor change - instead of returning a numpy array of arrays, mgrid/ogrid now return a list of arrays. However, this is unlikely to be a problem as the majority of users generally unpack the results of mgrid/ogrid so that each matrix can be used individually. Comments welcome. Cheers, David -- ********************************** David M. Kaplan Charge de Recherche 1 Institut de Recherche pour le Developpement Centre de Recherche Halieutique Mediterraneenne et Tropicale av. Jean Monnet B.P. 171 34203 Sete cedex France Phone: +33 (0)4 99 57 32 27 Fax: +33 (0)4 99 57 32 95 http://www.ur097.ird.fr/team/dkaplan/index.html ********************************** -------------- next part -------------- A non-text attachment was scrubbed... Name: index_tricks.patch Type: text/x-patch Size: 8077 bytes Desc: not available URL: From cournape at gmail.com Fri Aug 15 11:24:52 2008 From: cournape at gmail.com (David Cournapeau) Date: Fri, 15 Aug 2008 10:24:52 -0500 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <48A49C89.6000800@american.edu> References: <48A49C89.6000800@american.edu> Message-ID: <5b8d13220808150824s3cc45965o22107eca5256c500@mail.gmail.com> On Thu, Aug 14, 2008 at 3:58 PM, Alan G Isaac wrote: > Two odd failures in test_print.py. > Platform: Win XP SP3 on Intel T2600. > Alan Isaac > I got the fixes to make numpy buildable again with VS 2003, and the errors are mingw specific. Either a compiler bug or more likely a configuration problem (mingw and vs not using the same codepath somewhere). At least, now, I can compare the two and it should not take too much time to sorting that out. cheers, David From millman at berkeley.edu Fri Aug 15 12:32:27 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Fri, 15 Aug 2008 09:32:27 -0700 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <48A5669F.3080007@esrf.fr> References: <48A5669F.3080007@esrf.fr> Message-ID: On Fri, Aug 15, 2008 at 4:21 AM, Jon Wright wrote: > It seems the new release breaks matplotlib, for those pauvres who are > using pre-compiled at least. If this means all C-modules compiled > against numpy have to be recompiled, then this will make me very unhappy. Yes, the new release requires a recompile. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From dalke at dalkescientific.com Fri Aug 15 12:41:26 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Fri, 15 Aug 2008 18:41:26 +0200 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <9457e7c80808150030q614d74c2j1be4f93b425e711d@mail.gmail.com> <3d375d730808150039u34f5b28sa401f58af1161e6@mail.gmail.com> <9457e7c80808150059t22bc7c74t62f319e29de4f5d2@mail.gmail.com> <20080815133020.GD18337@phare.normalesup.org> Message-ID: On Aug 15, 2008, at 4:38 PM, Pauli Virtanen wrote: > I think you can still do something evil, like this: > > import os > if os.environ.get('NUMPY_VIA_API', '0') != '0': > from numpy.lib.fromnumeric import * > ... > > But I'm not sure how many milliseconds must be gained to justify > this... I don't think it's enough. I don't like environmental variable tricks like that. My tests suggest: current SVN: 0.12 seconds my patch: 0.10 seconds removing some top-level imports: 0.09 seconds my patch and removing some additional top-level imports: 0.08 seconds (this is a guess) First, I reverted my patch, so my import times went from 0.10 second to 0.12 seconds. Second, I commented out the pure module imports from numpy/__init__.py import linalg import fft import random import ctypeslib import ma import doc The import time went to 0.089. Note that my patch also gets rid of "import doc" and "import ctypeslib", which take up a good chunk of time. The fft, linalg, and random libraries take 0.002 seconds each, and ma takes 0.007. Not doing these imports makes code about 0.01 second faster than my patches, which shaved off 0.02 seconds. That 0.01 second comes from not importing the fft, linalg, and ma modules. My patch does improve things in a few other places, so perhaps those other places adds another 0.01 seconds of performance. Why can't things be better? Take a look at the slowest imports. (Note, times are inclusive of the children) == Slowest (including children) == 0.089 numpy (None) 0.085 add_newdocs (numpy) 0.079 lib (add_newdocs) 0.041 type_check (lib) 0.040 numpy.core.numeric (type_check) 0.015 _internal (numpy.core.numeric) 0.014 numpy.testing (lib) 0.014 re (_internal) 0.010 unittest (numpy.testing) 0.010 numeric (numpy.core.numeric) 0.009 io (lib) Most of the time is spent importing 'lib'. Can that be made quicker? Not easily. "lib" is first imported in "add_newdocs". Personally, I want to get rid of add_newdocs and move the docstrings into the correct locations. Stubbing the function out by adding def add_newdoc(*args): pass to the tops of add_newdocs.py saves 0.005 seconds, but if you try it out and remove the "import lib" from add_newdocs.py then you'll have to fix a cyclical dependency. numpy/__init__.py: import core numpy/core/__init__.py: from defmatrix import * numpy/core/defmatrix.py: from numpy.lib.utils import issubdtype numpy/lib/__init__.py: from type_check import * numpy/lib/type_check.py: import numpy.core.numeric as _nx AttributeError: 'module' object has no attribute 'core' The only way out of the loop is to have numpy/__init__.py import lib before importing core. It's possible to clean up the code so this loop doesn't exist, and fix things so that fewer things are imported when some environment variable is set, but it doesn't look easy. Modules depend on other modules a bit too much to make me happy. Andrew dalke at dalkescientific.com From cournape at gmail.com Fri Aug 15 12:49:28 2008 From: cournape at gmail.com (David Cournapeau) Date: Fri, 15 Aug 2008 11:49:28 -0500 Subject: [Numpy-discussion] VS 2003 problems with cython-generated code Message-ID: <5b8d13220808150949v53ae76e1n106f607adf8f07@mail.gmail.com> Hi, I noticed this morning that numpy 1.2 is not buildable with VS 2003 (the one you have to use for official python releases for at least python 2.5, and maybe 2.4). When we generate C code, both with internal code (numpy/core/code_generator) and with external tools (cython/pyrex for mtrand), the string literals generated for docstrings are too long for visual studio. We have to break them (e.g. "foo bar" becomes "foo""bar"), but doing so with cython-generated code is only doable by changing cython itself. So I did patch cython to break those, and regenerate the mtrand.c. This is done in the vs_longstring branch. Is it ok to put this for 1.2 ? Without it, I don't see a way to have numpy 1.2 buildable with VS. cheers, David P.S: I attached the necessary patches to cython bug tracker too, so that hopefully, the problem can be solved for a future version of cython. From Chris.Barker at noaa.gov Fri Aug 15 12:50:34 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 15 Aug 2008 09:50:34 -0700 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: References: Message-ID: <48A5B3DA.2000104@noaa.gov> Jarrod Millman wrote: > NumPy 1.2.0b2 is now available. Please test this so that we can > uncover any problems ASAP. > Mac binary: > https://cirl.berkeley.edu/numpy/numpy-1.2.0b2-py2.5-macosx10.5.dmg Ran 1715 tests in 12.671s OK (SKIP=1) OS-X 10.4.11 Dual G5 PPC Python version 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) [GCC 4.0.1 It also seems to work so far with my code... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Fri Aug 15 13:12:45 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 15 Aug 2008 11:12:45 -0600 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: References: <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <9457e7c80808150030q614d74c2j1be4f93b425e711d@mail.gmail.com> <3d375d730808150039u34f5b28sa401f58af1161e6@mail.gmail.com> <9457e7c80808150059t22bc7c74t62f319e29de4f5d2@mail.gmail.com> <20080815133020.GD18337@phare.normalesup.org> Message-ID: On Fri, Aug 15, 2008 at 10:41 AM, Andrew Dalke wrote: > On Aug 15, 2008, at 4:38 PM, Pauli Virtanen wrote: > > I think you can still do something evil, like this: > > > > import os > > if os.environ.get('NUMPY_VIA_API', '0') != '0': > > from numpy.lib.fromnumeric import * > > ... > > > > But I'm not sure how many milliseconds must be gained to justify > > this... > > I don't think it's enough. I don't like environmental > variable tricks like that. My tests suggest: > current SVN: 0.12 seconds > my patch: 0.10 seconds > removing some top-level imports: 0.09 seconds > my patch and removing some > additional top-level imports: 0.08 seconds (this is a guess) > > > First, I reverted my patch, so my import times went from > 0.10 second to 0.12 seconds. > > Second, I commented out the pure module imports from numpy/__init__.py > > import linalg > import fft > import random > import ctypeslib > import ma > import doc > > The import time went to 0.089. Note that my patch also > gets rid of "import doc" and "import ctypeslib", which > take up a good chunk of time. The fft, linalg, and > random libraries take 0.002 seconds each, and ma takes 0.007. > > > Not doing these imports makes code about 0.01 second > faster than my patches, which shaved off 0.02 seconds. > That 0.01 second comes from not importing the > fft, linalg, and ma modules. > > My patch does improve things in a few other places, so > perhaps those other places adds another 0.01 seconds > of performance. > > > Why can't things be better? Take a look at the slowest > imports. (Note, times are inclusive of the children) > > == Slowest (including children) == > 0.089 numpy (None) > 0.085 add_newdocs (numpy) > 0.079 lib (add_newdocs) > 0.041 type_check (lib) > 0.040 numpy.core.numeric (type_check) > 0.015 _internal (numpy.core.numeric) > 0.014 numpy.testing (lib) > 0.014 re (_internal) > 0.010 unittest (numpy.testing) > 0.010 numeric (numpy.core.numeric) > 0.009 io (lib) > > Most of the time is spent importing 'lib'. > > Can that be made quicker? Not easily. "lib" is > first imported in "add_newdocs". Personally, I > want to get rid of add_newdocs and move the > docstrings into the correct locations. > And those would be? I hope you aren't thinking of moving them into the C code. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalke at dalkescientific.com Fri Aug 15 13:17:04 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Fri, 15 Aug 2008 19:17:04 +0200 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <48A5298F.80701@enthought.com> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> Message-ID: <6CA4A2EF-0E86-4DC1-8B8E-158B7D249458@dalkescientific.com> I forgot to mention.. On Aug 15, 2008, at 9:00 AM, Travis E. Oliphant wrote: > 1) Removing ctypeslib import > > * Can break code if somebody has been doing import numpy and then > using > numpy.ctypeslib > * I'm fine with people needing to import numpy.ctypeslib to use the > capability as long as we clearly indicate these breakages. You were the one who had numpy/__init__.py always import ctypeslib r3027 | oliphant | 2006-08-15 11:53:49 +0200 (Tue, 15 Aug 2006) | 1 line import ctypeslib on numpy load and change name from ctypes_load_library to load_library Was there a driving reason for that other than decreased user burden? There will be breakage in the wild. I found: http://mail.python.org/pipermail/python-list/2007-December/469132.html http://www.scipy.org/Cookbook/Ctypes and a Google Code search found a couple hits too: http://www.google.com/codesearch?q=numpy +ctypeslib&hl=en&btnG=Search+Code It doesn't looks like there will be a big impact. This is not a widely used package (in public code), and many examples seem to prefer this form: from numpy.ctypeslib import ndpointer, load_library Andrew dalke at dalkescientific.com From charlesr.harris at gmail.com Fri Aug 15 13:18:15 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 15 Aug 2008 11:18:15 -0600 Subject: [Numpy-discussion] VS 2003 problems with cython-generated code In-Reply-To: <5b8d13220808150949v53ae76e1n106f607adf8f07@mail.gmail.com> References: <5b8d13220808150949v53ae76e1n106f607adf8f07@mail.gmail.com> Message-ID: On Fri, Aug 15, 2008 at 10:49 AM, David Cournapeau wrote: > Hi, > > I noticed this morning that numpy 1.2 is not buildable with VS 2003 > (the one you have to use for official python releases for at least > python 2.5, and maybe 2.4). When we generate C code, both with > internal code (numpy/core/code_generator) and with external tools > (cython/pyrex for mtrand), the string literals generated for > docstrings are too long for visual studio. We have to break them (e.g. > "foo bar" becomes "foo""bar"), but doing so with cython-generated code > is only doable by changing cython itself. > > So I did patch cython to break those, and regenerate the mtrand.c. > This is done in the vs_longstring branch. Is it ok to put this for 1.2 > ? Without it, I don't see a way to have numpy 1.2 buildable with VS. > Be careful if you break across lines. The gnu compilers will accept "foo" "bar" But for some others you need to use a line continuation. "foo"\ "bar" Is this mostly for the ufuncs? I'm not sure why we can't make that operate like add_newdocs. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Fri Aug 15 14:16:22 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 15 Aug 2008 13:16:22 -0500 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <6CA4A2EF-0E86-4DC1-8B8E-158B7D249458@dalkescientific.com> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <6CA4A2EF-0E86-4DC1-8B8E-158B7D249458@dalkescientific.com> Message-ID: <48A5C7F6.7050800@enthought.com> Andrew Dalke wrote: > I forgot to mention.. > > > On Aug 15, 2008, at 9:00 AM, Travis E. Oliphant wrote: > >> 1) Removing ctypeslib import >> >> * Can break code if somebody has been doing import numpy and then >> using >> numpy.ctypeslib >> * I'm fine with people needing to import numpy.ctypeslib to use the >> capability as long as we clearly indicate these breakages. >> > > > You were the one who had numpy/__init__.py always import ctypeslib > > > r3027 | oliphant | 2006-08-15 11:53:49 +0200 (Tue, 15 Aug 2006) | 1 line > > import ctypeslib on numpy load and change name from > ctypes_load_library to load_library > > Was there a driving reason for that other than decreased user burden? > > > Not that I can recall. -Travis From stefan at sun.ac.za Fri Aug 15 14:19:10 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 15 Aug 2008 13:19:10 -0500 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: References: <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <9457e7c80808150030q614d74c2j1be4f93b425e711d@mail.gmail.com> <3d375d730808150039u34f5b28sa401f58af1161e6@mail.gmail.com> <9457e7c80808150059t22bc7c74t62f319e29de4f5d2@mail.gmail.com> <20080815133020.GD18337@phare.normalesup.org> Message-ID: <9457e7c80808151119q62e3dd98q862fddce30c75f02@mail.gmail.com> 2008/8/15 Andrew Dalke : > I don't think it's enough. I don't like environmental > variable tricks like that. My tests suggest: > current SVN: 0.12 seconds > my patch: 0.10 seconds > removing some top-level imports: 0.09 seconds > my patch and removing some > additional top-level imports: 0.08 seconds (this is a guess) There are two different concerns being addressed here: ease of use and load time. I am not sure the two can be optimised simultaneously. On the other hand, if we had two public APIs for importing (similar to matplotlib's pylab vs. pyplot), we could satisfy both parties, without placing too much of a burden upon developers. St?fan From cournape at gmail.com Fri Aug 15 15:22:07 2008 From: cournape at gmail.com (David Cournapeau) Date: Fri, 15 Aug 2008 14:22:07 -0500 Subject: [Numpy-discussion] VS 2003 problems with cython-generated code In-Reply-To: References: <5b8d13220808150949v53ae76e1n106f607adf8f07@mail.gmail.com> Message-ID: <5b8d13220808151222u7fa2f08cwb9c8a4292013c7b5@mail.gmail.com> On Fri, Aug 15, 2008 at 12:18 PM, Charles R Harris wrote: > > Be careful if you break across lines. The gnu compilers will accept > > "foo" > "bar" > > But for some others you need to use a line continuation. > > "foo"\ > "bar" > I don't put newlines: I really do "foo""bar", to avoid this exact problem. I tested the changes with gcc and visual studio (that's really the minimal set we want to support at any release). The changes are done in the code generator, because that's where the problem was, but maybe this could have been changed somewhere else. For mtrand, that's different, though: I am the only one who can generate the file right now: that's the object of my email, and the reason why I created a new branch for this. cheers, David From cournape at gmail.com Fri Aug 15 15:25:44 2008 From: cournape at gmail.com (David Cournapeau) Date: Fri, 15 Aug 2008 14:25:44 -0500 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: References: <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <9457e7c80808150030q614d74c2j1be4f93b425e711d@mail.gmail.com> <3d375d730808150039u34f5b28sa401f58af1161e6@mail.gmail.com> <9457e7c80808150059t22bc7c74t62f319e29de4f5d2@mail.gmail.com> <20080815133020.GD18337@phare.normalesup.org> Message-ID: <5b8d13220808151225o3fad5108v49671d0c4e3199bd@mail.gmail.com> On Fri, Aug 15, 2008 at 11:41 AM, Andrew Dalke wrote: > > It's possible to clean up the code so this loop > doesn't exist, and fix things so that fewer things > are imported when some environment variable is set, > but it doesn't look easy. Modules depend on other > modules a bit too much to make me happy. Yes. numpy.core should not depend on anything else. That would be the easy thing to do: there is only one function used IIRC from numpy.lib. As you said, the hairy stuff (from import dependency POV) is in numpy.lib. cheers, David From oliphant at enthought.com Fri Aug 15 15:58:42 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 15 Aug 2008 14:58:42 -0500 Subject: [Numpy-discussion] Addition of a dict object to all NumPy objects Message-ID: <48A5DFF2.6040809@enthought.com> Hello all, While we are on the subject of C-API changes, I've noticed that quite a few of the sub-classes of ndarray are constructed to basically add meta-information to the array. What if the base-class ndarray grew a dict object at it's end to hold meta information. Naturally, several questions arise: 1) How would you access the dictionary? (i.e. __dict__?) 2) Would attribute setting and getting retrieve from this dictionary (how are conflicts managed). * I think I would prefer a dict attribute on the numpy array that gets and sets into the dictionary. 3) Are the additional 4-8 bytes too expensive 4) Should there instead be a C-level array sub-class with the dict attached. 5) How should dict's be propagated through views? My preference is to not propagate them at all. Thoughts? -Travis From charlesr.harris at gmail.com Fri Aug 15 16:10:03 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 15 Aug 2008 14:10:03 -0600 Subject: [Numpy-discussion] Addition of a dict object to all NumPy objects In-Reply-To: <48A5DFF2.6040809@enthought.com> References: <48A5DFF2.6040809@enthought.com> Message-ID: On Fri, Aug 15, 2008 at 1:58 PM, Travis E. Oliphant wrote: > > Hello all, > > While we are on the subject of C-API changes, I've noticed that quite a > few of the sub-classes of ndarray are constructed to basically add > meta-information to the array. > > What if the base-class ndarray grew a dict object at it's end to hold > meta information. > > Naturally, several questions arise: > > 1) How would you access the dictionary? (i.e. __dict__?) > > 2) Would attribute setting and getting retrieve from this dictionary > (how are conflicts managed). > * I think I would prefer a dict attribute on the numpy array that > gets and sets into the dictionary. > > 3) Are the additional 4-8 bytes too expensive > One of the problems with numarray was the time taken to allocate small arrays. Would adding a dictionary slow down the allocation of numpy arrays? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Aug 15 16:15:03 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 15 Aug 2008 14:15:03 -0600 Subject: [Numpy-discussion] Addition of a dict object to all NumPy objects In-Reply-To: References: <48A5DFF2.6040809@enthought.com> Message-ID: On Fri, Aug 15, 2008 at 2:10 PM, Charles R Harris wrote: > > > On Fri, Aug 15, 2008 at 1:58 PM, Travis E. Oliphant < > oliphant at enthought.com> wrote: > >> >> Hello all, >> >> While we are on the subject of C-API changes, I've noticed that quite a >> few of the sub-classes of ndarray are constructed to basically add >> meta-information to the array. >> >> What if the base-class ndarray grew a dict object at it's end to hold >> meta information. >> >> Naturally, several questions arise: >> >> 1) How would you access the dictionary? (i.e. __dict__?) >> >> 2) Would attribute setting and getting retrieve from this dictionary >> (how are conflicts managed). >> * I think I would prefer a dict attribute on the numpy array that >> gets and sets into the dictionary. >> >> 3) Are the additional 4-8 bytes too expensive >> > > One of the problems with numarray was the time taken to allocate small > arrays. Would adding a dictionary slow down the allocation of numpy arrays? > That said, I think we should keep things as simple and orthogonal as possible. If we go this way, I think a subclass with a dictionary would be the best approach to avoid the heartbreak of creeping featuritis. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Aug 15 17:06:40 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 15 Aug 2008 16:06:40 -0500 Subject: [Numpy-discussion] Addition of a dict object to all NumPy objects In-Reply-To: References: <48A5DFF2.6040809@enthought.com> Message-ID: <3d375d730808151406g449d27aas3e5b5da89124db0a@mail.gmail.com> On Fri, Aug 15, 2008 at 15:15, Charles R Harris wrote: > > On Fri, Aug 15, 2008 at 2:10 PM, Charles R Harris > wrote: >> >> On Fri, Aug 15, 2008 at 1:58 PM, Travis E. Oliphant >> wrote: >>> >>> Hello all, >>> >>> While we are on the subject of C-API changes, I've noticed that quite a >>> few of the sub-classes of ndarray are constructed to basically add >>> meta-information to the array. >>> >>> What if the base-class ndarray grew a dict object at it's end to hold >>> meta information. >>> >>> Naturally, several questions arise: >>> >>> 1) How would you access the dictionary? (i.e. __dict__?) >>> >>> 2) Would attribute setting and getting retrieve from this dictionary >>> (how are conflicts managed). >>> * I think I would prefer a dict attribute on the numpy array that >>> gets and sets into the dictionary. >>> >>> 3) Are the additional 4-8 bytes too expensive >> >> One of the problems with numarray was the time taken to allocate small >> arrays. Would adding a dictionary slow down the allocation of numpy arrays? > > That said, I think we should keep things as simple and orthogonal as > possible. If we go this way, I think a subclass with a dictionary would be > the best approach to avoid the heartbreak of creeping featuritis. The point of the feature is to avoid subclasses. There are a number of use cases for annotating arrays with metadata. Currently, they are forced to use subclasses. Every time you use ndarray subclasses, you are essentially forcing yourself into your subclass's ghetto of functions that only work on your subclass. I think you could make the dictionary created lazily on the first getattr(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at enthought.com Fri Aug 15 17:12:43 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 15 Aug 2008 16:12:43 -0500 Subject: [Numpy-discussion] Addition of a dict object to all NumPy objects In-Reply-To: <3d375d730808151406g449d27aas3e5b5da89124db0a@mail.gmail.com> References: <48A5DFF2.6040809@enthought.com> <3d375d730808151406g449d27aas3e5b5da89124db0a@mail.gmail.com> Message-ID: <48A5F14B.4030802@enthought.com> Robert Kern wrote: > >>>> 3) Are the additional 4-8 bytes too expensive >>>> >>> One of the problems with numarray was the time taken to allocate small >>> arrays. Would adding a dictionary slow down the allocation of numpy arrays? >>> No, I don't think so, not if we did nothing by default but set the dict to NULL (i.e. no propagation of meta-information to new arrays). >> That said, I think we should keep things as simple and orthogonal as >> possible. If we go this way, I think a subclass with a dictionary would be >> the best approach to avoid the heartbreak of creeping featuritis. >> > > The point of the feature is to avoid subclasses. There are a number of > use cases for annotating arrays with metadata. Currently, they are > forced to use subclasses. Every time you use ndarray subclasses, you > are essentially forcing yourself into your subclass's ghetto of > functions that only work on your subclass. > This would be one-step better in the sense that there would be a single sub-class to handle all cases of just needing meta information. But, I tend to agree that adding the dictionary to all arrays is preferable. > I think you could make the dictionary created lazily on the first getattr(). > Yes, that could easily be done. It would just be set to NULL on creation and the penalty/overhead would be the extra pointer in the array structure. -Travis From oliphant at enthought.com Fri Aug 15 17:18:47 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 15 Aug 2008 16:18:47 -0500 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <9457e7c80808150030q614d74c2j1be4f93b425e711d@mail.gmail.com> <3d375d730808150039u34f5b28sa401f58af1161e6@mail.gmail.com> <9457e7c80808150059t22bc7c74t62f319e29de4f5d2@mail.gmail.com> <20080815133020.GD18337@phare.normalesup.org> Message-ID: <48A5F2B7.3070301@enthought.com> Andrew Dalke wrote: > > Can that be made quicker? Not easily. "lib" is > first imported in "add_newdocs". Personally, I > want to get rid of add_newdocs and move the > docstrings into the correct locations. > Where would that be, in the C-code? The reason for add_newdocs is to avoid writing docstrings in C-code which is a pain. > It's possible to clean up the code so this loop > doesn't exist, and fix things so that fewer things > are imported when some environment variable is set, > but it doesn't look easy. Modules depend on other > modules a bit too much to make me happy. > I've removed this loop. Are there other places in numpy.core that depend on numpy.lib? Thanks for the very helpful analysis. -Travis From charlesr.harris at gmail.com Fri Aug 15 17:19:16 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 15 Aug 2008 15:19:16 -0600 Subject: [Numpy-discussion] Addition of a dict object to all NumPy objects In-Reply-To: <48A5F14B.4030802@enthought.com> References: <48A5DFF2.6040809@enthought.com> <3d375d730808151406g449d27aas3e5b5da89124db0a@mail.gmail.com> <48A5F14B.4030802@enthought.com> Message-ID: On Fri, Aug 15, 2008 at 3:12 PM, Travis E. Oliphant wrote: > Robert Kern wrote: > > > >>>> 3) Are the additional 4-8 bytes too expensive > >>>> > >>> One of the problems with numarray was the time taken to allocate small > >>> arrays. Would adding a dictionary slow down the allocation of numpy > arrays? > >>> > No, I don't think so, not if we did nothing by default but set the dict > to NULL (i.e. no propagation of meta-information to new arrays). > >> That said, I think we should keep things as simple and orthogonal as > >> possible. If we go this way, I think a subclass with a dictionary would > be > >> the best approach to avoid the heartbreak of creeping featuritis. > >> > > > > The point of the feature is to avoid subclasses. There are a number of > > use cases for annotating arrays with metadata. Currently, they are > > forced to use subclasses. Every time you use ndarray subclasses, you > > are essentially forcing yourself into your subclass's ghetto of > > functions that only work on your subclass. > > > This would be one-step better in the sense that there would be a single > sub-class to handle all cases of just needing meta information. But, > I tend to agree that adding the dictionary to all arrays is preferable. > > I think you could make the dictionary created lazily on the first > getattr(). > > > Yes, that could easily be done. It would just be set to NULL on > creation and the penalty/overhead would be the extra pointer in the > array structure. > That doesn't sound bad, and the convenience is probably worth it. Can this be done in a way that doesn't require a new compile? That is, can it look like a subclass in C? I'm opposed to adding anything until 1.2 is out. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmay31 at gmail.com Fri Aug 15 18:04:49 2008 From: rmay31 at gmail.com (Ryan May) Date: Fri, 15 Aug 2008 17:04:49 -0500 Subject: [Numpy-discussion] min() of array containing NaN In-Reply-To: References: Message-ID: > > Availability of the NaN functionality in a method of ndarray > > The last point is key. The NaN behavior is central to analyzing real > data containing unavoidable bad values, which is the bread and butter > of a substantial fraction of the user base. In the languages they're > switching from, handling NaNs is just part of doing business, and is > an option of every relevant routine; there's no need for redundant > sets of routines. In contrast, numpy appears to consider data > analysis to be secondary, somehow, to pure math, and takes the NaN > functionality out of routines like min() and std(). This means it's > not possible to use many ndarray methods. If we're ready to handle a > NaN by returning it, why not enable the more useful behavior of > ignoring it, at user discretion? > Maybe I missed this somewhere, but this seems like a better use for masked arrays, not NaN's. Masked arrays were specifically designed to add functions that work well with masked/invalid data points. Why reinvent the wheel here? Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at cheimes.de Fri Aug 15 18:31:20 2008 From: lists at cheimes.de (Christian Heimes) Date: Sat, 16 Aug 2008 00:31:20 +0200 Subject: [Numpy-discussion] Addition of a dict object to all NumPy objects In-Reply-To: <3d375d730808151406g449d27aas3e5b5da89124db0a@mail.gmail.com> References: <48A5DFF2.6040809@enthought.com> <3d375d730808151406g449d27aas3e5b5da89124db0a@mail.gmail.com> Message-ID: Robert Kern wrote: > I think you could make the dictionary created lazily on the first getattr(). In order to make it work you have to reserve space for a PyObject* pointer for the instance dict somewhere in your type definition. It's going to increase the size of every object by 4 bytes on a 32bit OS or 8 bytes on a 64bit OS, aka sizeof(uintptr_t). An empty dict increases the size of every object by ~30 byte. Christian From alan at ajackson.org Fri Aug 15 18:26:46 2008 From: alan at ajackson.org (Alan Jackson) Date: Fri, 15 Aug 2008 17:26:46 -0500 Subject: [Numpy-discussion] Questions about some of the random functions Message-ID: <20080815172646.18680b53@ajackson.org> Just a question - I'm gradually working through the distributions for the documentation marathon and I realised that there is a whole nest of them named "standard-xxxx". For several (e.g., normal) they are just the regular distribution with all the parameters except size set to "standard" values. So my first question is - why? They seem very redundant. Second question - why is there is a "standard-t" for Student's T-test (or the distribution associated with it) but no corresponding "t" distribution. I probably have too much time to think about it this weekend. 8-) -- ----------------------------------------------------------------------- | Alan K. Jackson | To see a World in a Grain of Sand | | alan at ajackson.org | And a Heaven in a Wild Flower, | | www.ajackson.org | Hold Infinity in the palm of your hand | | Houston, Texas | And Eternity in an hour. - Blake | ----------------------------------------------------------------------- From stefan at sun.ac.za Fri Aug 15 19:10:52 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 15 Aug 2008 18:10:52 -0500 Subject: [Numpy-discussion] [REVIEW] Update NumPy API format to support updates that don't break binary compatibility In-Reply-To: <001636283760d7bea9045487b140@google.com> References: <001636283760d7bea9045487b140@google.com> Message-ID: <9457e7c80808151610l78c07b32y5f203435d368a5b8@mail.gmail.com> The current NumPy API number, stored as NPY_VERSION in the header files, needs to be incremented every time the NumPy C-API changes. The counter tells developers with exactly which revision of the API they are dealing. NumPy does some checking to make sure that it does not run against an old version of the API. Currently, we have no way of distinguishing between changes that break binary compatibility and those that don't. The proposed fix breaks the version number up into two counters -- one that gets increased when binary compatibility is broken, and another when the API is changed without breaking compatibility. Backward compatibility with packages such as Matplotlib is maintained by renaming NPY_VERSION to NPY_BINARY_VERSION. Please review the proposed change at http://codereview.appspot.com/2946 Regards St?fan From robert.kern at gmail.com Fri Aug 15 19:13:48 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 15 Aug 2008 18:13:48 -0500 Subject: [Numpy-discussion] Questions about some of the random functions In-Reply-To: <20080815172646.18680b53@ajackson.org> References: <20080815172646.18680b53@ajackson.org> Message-ID: <3d375d730808151613w3e1cf582u152fb86e89a589fa@mail.gmail.com> On Fri, Aug 15, 2008 at 17:26, Alan Jackson wrote: > Just a question - I'm gradually working through the distributions for > the documentation marathon and I realised that there is a whole nest > of them named "standard-xxxx". > > For several (e.g., normal) they are just the regular distribution with all the > parameters except size set to "standard" values. Not quite. Rather the regular distributions are built up from the standard version by transformation. > So my first question is - why? They seem very redundant. At the C level, they sometimes exist because they are components of other distributions that don't need the "x*1.0 + 0.0" waste. At the Python level, they usually exist for backwards compatibility with the libraries I was replacing or because I thought they would be useful for Python-level implementations of some distributions in scipy.stats. > Second question - why is there is a "standard-t" for Student's T-test > (or the distribution associated with it) but no corresponding "t" > distribution. Apathy, to be honest. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Fri Aug 15 19:22:34 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 15 Aug 2008 18:22:34 -0500 Subject: [Numpy-discussion] Addition of a dict object to all NumPy objects In-Reply-To: References: <48A5DFF2.6040809@enthought.com> <3d375d730808151406g449d27aas3e5b5da89124db0a@mail.gmail.com> Message-ID: <3d375d730808151622l66a95f27g52ad85d4c51dbd64@mail.gmail.com> On Fri, Aug 15, 2008 at 17:31, Christian Heimes wrote: > Robert Kern wrote: >> I think you could make the dictionary created lazily on the first getattr(). > > In order to make it work you have to reserve space for a PyObject* > pointer for the instance dict somewhere in your type definition. It's > going to increase the size of every object by 4 bytes on a 32bit OS or 8 > bytes on a 64bit OS, aka sizeof(uintptr_t). An empty dict increases the > size of every object by ~30 byte. Yes, we know that. The concern I was addressing was the time overhead for creating the new dict object every time an ndarray gets instantiated. Most of these dict objects would be unused, so we would be wasting a substantial amount of time. If you push off the creation of the dict to the first time the user accesses it, then we're not wasting any time. We do realize that space for the pointer must still be reserved. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cournape at gmail.com Fri Aug 15 19:43:03 2008 From: cournape at gmail.com (David Cournapeau) Date: Fri, 15 Aug 2008 18:43:03 -0500 Subject: [Numpy-discussion] long double woes on win32 Message-ID: <5b8d13220808151643v4834397frd64b31707afbaa78@mail.gmail.com> Hi, The test failures on windows with 1.2b2 are due to buggy long double behavior of mingw. My understanding is that on windows, long double is 8 bytes (that's the sizeof value returned by VS 2003), but mingw says 12 bytes. One solution would be forcing numpy to configure itself to handle long double as double. But doing this breaks the configuration in many ways, and this would require relatively important changes which is quite fragile in numpy ATM (the whole math function things in umathmodule.c.src). I don't see this happening for 1.2. Another one is to use VS 2003 for the binaries: but then we are back to the problem of non buildable numpy with VS 2003. The last one is to ignore this for 1.2 (long double was already broken before in numpy win32 binaries). What do people think ? cheers, David From stefan at sun.ac.za Fri Aug 15 19:45:05 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 15 Aug 2008 18:45:05 -0500 Subject: [Numpy-discussion] Generalised ufuncs branch Message-ID: <9457e7c80808151645v457e155apa192bdbcf3f91d47@mail.gmail.com> Hi all, I have moved the generalised ufuncs functionality off to a branch: http://svn.scipy.org/svn/numpy/branches/gen_ufuncs Please try it out and give us your feedback. We shall also pound on it at the sprint during SciPy'08, and thereafter decide how and when to best integrate it into NumPy. Thanks to Wenjie Fu and Hans-Andreas Engel for taking the time to think this issue through and to submit such a high-quality patch. Regards St?fan From charlesr.harris at gmail.com Fri Aug 15 20:11:48 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 15 Aug 2008 18:11:48 -0600 Subject: [Numpy-discussion] long double woes on win32 In-Reply-To: <5b8d13220808151643v4834397frd64b31707afbaa78@mail.gmail.com> References: <5b8d13220808151643v4834397frd64b31707afbaa78@mail.gmail.com> Message-ID: On Fri, Aug 15, 2008 at 5:43 PM, David Cournapeau wrote: > Hi, > > The test failures on windows with 1.2b2 are due to buggy long double > behavior of mingw. My understanding is that on windows, long double is > 8 bytes (that's the sizeof value returned by VS 2003), but mingw says > 12 bytes. > Doesn't mingw use the MSVC library? IIRC, in MSVC long doubles and doubles are the same type, so this seems to be a bug in mingw. I suppose numpy could detect this situation and define the types to be identical, but if this is impractical, then perhaps the best thing to do is issue an error message. > > One solution would be forcing numpy to configure itself to handle long > double as double. But doing this breaks the configuration in many > ways, and this would require relatively important changes which is > quite fragile in numpy ATM (the whole math function things in > umathmodule.c.src). I don't see this happening for 1.2. > > Another one is to use VS 2003 for the binaries: but then we are back > to the problem of non buildable numpy with VS 2003. > > The last one is to ignore this for 1.2 (long double was already broken > before in numpy win32 binaries). > > What do people think ? > There's isn't much you can do about long doubles while maintaining MSVC compatibility. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at cheimes.de Fri Aug 15 20:30:22 2008 From: lists at cheimes.de (Christian Heimes) Date: Sat, 16 Aug 2008 02:30:22 +0200 Subject: [Numpy-discussion] Addition of a dict object to all NumPy objects In-Reply-To: <3d375d730808151622l66a95f27g52ad85d4c51dbd64@mail.gmail.com> References: <48A5DFF2.6040809@enthought.com> <3d375d730808151406g449d27aas3e5b5da89124db0a@mail.gmail.com> <3d375d730808151622l66a95f27g52ad85d4c51dbd64@mail.gmail.com> Message-ID: Robert Kern wrote: > Yes, we know that. The concern I was addressing was the time overhead > for creating the new dict object every time an ndarray gets > instantiated. Most of these dict objects would be unused, so we would > be wasting a substantial amount of time. If you push off the creation > of the dict to the first time the user accesses it, then we're not > wasting any time. We do realize that space for the pointer must still > be reserved. I'm sorry for pointing to the obvious. I *guess* it's possible to delay the creation even further. You don't have to create a dict until somebody assigns a new attribute to an instance. It'd require some more code and you'd have to trade memory efficiency for slightly slower access to the additional attributes. Please also note that CPython uses a freelist of unused dict instances. The default size of the dict free list is 80 elements. The allocation and deallocation of dicts is cheap if you can stay below the threshold. Christian From xavier.gnata at gmail.com Fri Aug 15 21:02:52 2008 From: xavier.gnata at gmail.com (Xavier Gnata) Date: Sat, 16 Aug 2008 03:02:52 +0200 Subject: [Numpy-discussion] Different results from repeated calculation, part 2 In-Reply-To: <20080814174814.GA30436@virginia.edu> References: <20080814174814.GA30436@virginia.edu> Message-ID: <48A6273C.5080109@gmail.com> Alok Singhal wrote: > On 14/08/08: 10:20, Keith Goodman wrote: > >> A unit test is attached. It contains three tests: >> >> In test1, I construct matrices x and y and then repeatedly calculate z >> = calc(x,y). The result z is the same every time. So this test passes. >> >> In test2, I construct matrices x and y each time before calculating z >> = calc(x,y). Sometimes z is slightly different. But the x's test to be >> equal and so do the y's. This test fails (on Debian Lenny, Core 2 Duo, >> with libatlas3gf-sse2 but not with libatlas3gf-sse). >> >> test3 is the same as test2 but I calculate z like this: z = >> calc(100*x,y) / (100 * 100). This test passes. >> >> I get: >> >> ====================================================================== >> FAIL: repeatability #2 >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "/home/[snip]/test/repeat_test.py", line 73, in test_repeat_2 >> self.assert_(result, msg) >> AssertionError: Max difference = 2.04946e-16 >> > > Could this be because of how the calculations are done? If the > floating point numbers are stored in the cpu registers, in this case > (intel core duo), they are 80-bit values, whereas 'double' precision > is 64-bits. Depending upon gcc's optimization settings, the amount of > automatic variables, etc., it is entirely possible that the numbers > are stored in registers only in some cases, and are in the RAM in > other cases. Thus, in your tests, sometimes some numbers get stored > in the cpu registers, making the calculations with those values > different from the case if they were not stored in the registers. > > See "The pitfalls of verifying floating-point computations" at > http://portal.acm.org/citation.cfm?doid=1353445.1353446 (or if that > needs subscription, you can download the PDF from > http://arxiv.org/abs/cs/0701192). The paper has a lot of examples of > surprises like this. Quote: > > We shall discuss the following myths, among others: > > ... > > - "Arithmetic operations are deterministic; that is, if I do z=x+y in > two places in the same program and my program never touches x and y > in the meantime, then the results should be the same." > > - A variant: "If x < 1 tests true at one point, then x < 1 stays true > later if I never modify x." > > ... > > -Alok > > yep! The code is buggy. Please **nerver** use == to test if two floating point numbers are equal. **nerver**. As explained by Alok, floating point computations are *not* the same as computations over the field of reals numbers. Intel 80bits registers versus 64bits representation, IEE754, sse or not sse or mmx : Fun with floating point arithmetic :) The same code with and without SSE could provide you with "no equal" results. Never use == on flaots?!? Well, you would have to use it in some corner cases but please remember that computers are only working on a small subset (finite) of natural numbers. http://docs.python.org/tut/node16.html is a simple part of the story. 80 bits register are the ugly part (but the fun one :)) abs(a-b) References: <5b8d13220808131235p3e9f127s2a78c96813934138@mail.gmail.com> Message-ID: David Cournapeau wrote: > The current trunk has 14 failures on windows (with mingw). 12 of them > are related to C99 (see ticket 869). Can the people involved in recent > changes to complex functions take a look at it ? I think this is high > priority for 1.2.0 I'm asking just out of curiosity. Why is NumPy using C99 and what features of C99 are used? The Microsoft compilers aren't supporting C99 and they'll probably never will. I don't know if the Intel CC supports C99. Even GCC doesn't implement C99 to its full extend. Are you planing to restrict yourself to MinGW32? I'm not a NumPy developer but I'm a Python core developer. I've laid the foundation for the VS 2008 build system for 2.6 / 3.0. Marc Dickinson and I have put lots of work into mathematical, numerical and IEEE 754 fixes. The work was mostly based on the C99 specs but we used C89 code. That should explain my interest in the matter. :] Christian From dalke at dalkescientific.com Fri Aug 15 21:43:34 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Sat, 16 Aug 2008 03:43:34 +0200 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <9457e7c80808150030q614d74c2j1be4f93b425e711d@mail.gmail.com> <3d375d730808150039u34f5b28sa401f58af1161e6@mail.gmail.com> <9457e7c80808150059t22bc7c74t62f319e29de4f5d2@mail.gmail.com> <20080815133020.GD18337@phare.normalesup.org> Message-ID: On Aug 15, 2008, at 6:41 PM, Andrew Dalke wrote: > I don't think it's enough. I don't like environmental > variable tricks like that. My tests suggest: > current SVN: 0.12 seconds > my patch: 0.10 seconds > removing some top-level imports: 0.09 seconds > my patch and removing some > additional top-level imports: 0.08 seconds (this is a guess) > > > First, I reverted my patch, so my import times went from > 0.10 second to 0.12 seconds. Turns out I didn't revert everything. As of the SVN version from 10 minutes ago, "import numpy" on my machine takes 0.18 seconds, not 0.12 seconds. My patch should cut the import time by about 30-40% more from what it is. On some machines. Your milage may vary :) In my issue report I said the import time was 0.15 seconds. Adding up the times I saved doesn't match up with my final value. So take my numbers as guidelines. For those curious, top cumulative import times from SVN are: 0.184 numpy (None) 0.103 add_newdocs (numpy) 0.097 lib (add_newdocs) 0.049 type_check (lib) 0.048 numpy.core.numeric (type_check) 0.028 io (lib) 0.022 ctypeslib (numpy) 0.022 ctypes (ctypeslib) 0.021 random (numpy) 0.021 mtrand (random) 0.019 _import_tools (numpy) 0.019 glob (_import_tools) 0.018 _datasource (io) 0.016 fnmatch (glob) 0.015 numpy.testing (numpy.core.numeric) 0.014 re (fnmatch) Andrew dalke at dalkescientific.com From charlesr.harris at gmail.com Fri Aug 15 21:48:05 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 15 Aug 2008 19:48:05 -0600 Subject: [Numpy-discussion] C99 on windows In-Reply-To: References: <5b8d13220808131235p3e9f127s2a78c96813934138@mail.gmail.com> Message-ID: On Fri, Aug 15, 2008 at 7:25 PM, Christian Heimes wrote: > David Cournapeau wrote: > > The current trunk has 14 failures on windows (with mingw). 12 of them > > are related to C99 (see ticket 869). Can the people involved in recent > > changes to complex functions take a look at it ? I think this is high > > priority for 1.2.0 > > I'm asking just out of curiosity. Why is NumPy using C99 and what > features of C99 are used? The Microsoft compilers aren't supporting C99 > and they'll probably never will. I don't know if the Intel CC supports > C99. Even GCC doesn't implement C99 to its full extend. Are you planing > to restrict yourself to MinGW32? > I believe C99 was used as a guide to how complex corner cases involving +/-0, +/-inf, etc. should behave. However, it doesn't look possible to make that behaviour portable without a lot of work and it probably isn't worth the trouble. At the moment the failing tests have been removed. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Fri Aug 15 21:55:27 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 15 Aug 2008 21:55:27 -0400 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: References: <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <9457e7c80808150030q614d74c2j1be4f93b425e711d@mail.gmail.com> <3d375d730808150039u34f5b28sa401f58af1161e6@mail.gmail.com> <9457e7c80808150059t22bc7c74t62f319e29de4f5d2@mail.gmail.com> <20080815133020.GD18337@phare.normalesup.org> Message-ID: 2008/8/15 Andrew Dalke : > On Aug 15, 2008, at 6:41 PM, Andrew Dalke wrote: >> I don't think it's enough. I don't like environmental >> variable tricks like that. My tests suggest: >> current SVN: 0.12 seconds >> my patch: 0.10 seconds >> removing some top-level imports: 0.09 seconds >> my patch and removing some >> additional top-level imports: 0.08 seconds (this is a guess) >> >> >> First, I reverted my patch, so my import times went from >> 0.10 second to 0.12 seconds. > > Turns out I didn't revert everything. > > As of the SVN version from 10 minutes ago, "import numpy" on my > machine takes 0.18 seconds, not 0.12 seconds. My patch should > cut the import time by about 30-40% more from what it is. > On some machines. Your milage may vary :) I realize this is already a very complicated issue, but it's worth pointing out that the times you measure are not necessarily the times users care about. These numbers are once everything is loaded into disk cache. They don't reflect, say, interactive startup time, or time it takes in a script that uses substantial disk access (i.e. which fills the cache with something else). I realize this is the only available basis for comparison, but do keep in mind that improvements of a few milliseconds here may make a much larger difference in practice - or a much smaller difference. Anne From dalke at dalkescientific.com Fri Aug 15 22:03:55 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Sat, 16 Aug 2008 04:03:55 +0200 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <48A5F2B7.3070301@enthought.com> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <9457e7c80808150030q614d74c2j1be4f93b425e711d@mail.gmail.com> <3d375d730808150039u34f5b28sa401f58af1161e6@mail.gmail.com> <9457e7c80808150059t22bc7c74t62f319e29de4f5d2@mail.gmail.com> <20080815133020.GD18337@phare.normalesup.org> <48A5F2B7.3070301@enthought.com> Message-ID: <9CEFB730-64D1-499E-A497-CEF0FA37FC8A@dalkescientific.com> On Aug 15, 2008, at 11:18 PM, Travis E. Oliphant wrote: > I've removed this loop. Are there other places in numpy.core that > depend on numpy.lib? That fixed the loop I identified. I removed the "import lib" in add_newdocs.py and things imported fine. I then commented out the following lines #import lib #from lib import * in numpy/__init__.py . This identified a loop in fft. [josiah:~/src] dalke% python time_import.py Traceback (most recent call last): File "time_import.py", line 31, in import numpy File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/__init__.py", line 146, in import fft File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/fft/__init__.py", line 38, in from fftpack import * File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/fft/fftpack.py", line 541, in from numpy import deprecate ImportError: cannot import name deprecate Removing the "import fft" gives another loop for deprecate: import numpy File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/__init__.py", line 148, in import ctypeslib File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/ctypeslib.py", line 56, in from numpy import integer, ndarray, dtype as _dtype, deprecate, array ImportError: cannot import name deprecate Removing the "import ctypeslib" gives the following loop: Traceback (most recent call last): File "time_import.py", line 31, in import numpy File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/__init__.py", line 149, in import ma File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/ma/__init__.py", line 44, in import core File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/ma/core.py", line 66, in from numpy import ndarray, typecodes, amax, amin, iscomplexobj,\ ImportError: cannot import name iscomplexobj Removing the "import ma" and I ended up with no ImportErrors. The code still end up importing numpy.lib because of File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/linalg/linalg.py", line 28, in from numpy.lib import triu Take that out and "import numpy" does not imply "import nupy.lib" > Andrew dalke at dalkescientific.com From dalke at dalkescientific.com Fri Aug 15 22:06:54 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Sat, 16 Aug 2008 04:06:54 +0200 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <48A5F2B7.3070301@enthought.com> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <9457e7c80808150030q614d74c2j1be4f93b425e711d@mail.gmail.com> <3d375d730808150039u34f5b28sa401f58af1161e6@mail.gmail.com> <9457e7c80808150059t22bc7c74t62f319e29de4f5d2@mail.gmail.com> <20080815133020.GD18337@phare.normalesup.org> <48A5F2B7.3070301@enthought.com> Message-ID: On Aug 15, 2008, at 11:18 PM, Travis E. Oliphant wrote: > Where would that be, in the C-code? The reason for add_newdocs is to > avoid writing docstrings in C-code which is a pain. That was my thought. I could see that the code might useful during module development, where you don't want text changes to incur a recompile hit. But come release time, if someone volunteers to migrate the docstrings to C, in order to get a small bit of performance increase, then I don't see why not. Andrew dalke at dalkescientific.com From lists at cheimes.de Fri Aug 15 22:09:33 2008 From: lists at cheimes.de (Christian Heimes) Date: Sat, 16 Aug 2008 04:09:33 +0200 Subject: [Numpy-discussion] C99 on windows In-Reply-To: References: <5b8d13220808131235p3e9f127s2a78c96813934138@mail.gmail.com> Message-ID: Charles R Harris wrote: > I believe C99 was used as a guide to how complex corner cases involving > +/-0, +/-inf, etc. should behave. However, it doesn't look possible to make > that behaviour portable without a lot of work and it probably isn't worth > the trouble. At the moment the failing tests have been removed. We used the C99 specs as guideline, too. If I recall correctly mostly Annex F and G. We got it all sorted out but it took us a tremendous amount of time and work. We had to reimplement a bunch of math functions like log1p and the reversed hyperbolic functions. Mark introduced a system of lookup tables for complex corner cases. You can find them at the end of the cmath module: http://svn.python.org/projects/python/trunk/Modules/cmathmodule.c The new pymath files contain a series of macros and re-implemenation of C99 features and cross platform workarounds: http://svn.python.org/projects/python/trunk/Include/pymath.h http://svn.python.org/projects/python/trunk/Python/pymath.c HTH Christian From dalke at dalkescientific.com Fri Aug 15 22:18:57 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Sat, 16 Aug 2008 04:18:57 +0200 Subject: [Numpy-discussion] Different results from repeated calculation, part 2 In-Reply-To: <48A6273C.5080109@gmail.com> References: <20080814174814.GA30436@virginia.edu> <48A6273C.5080109@gmail.com> Message-ID: <6CE4D31A-0018-4983-B710-122CEDD7DE37@dalkescientific.com> On Aug 16, 2008, at 3:02 AM, Xavier Gnata wrote: > abs(a-b) References: <48A5DFF2.6040809@enthought.com> <3d375d730808151406g449d27aas3e5b5da89124db0a@mail.gmail.com> <3d375d730808151622l66a95f27g52ad85d4c51dbd64@mail.gmail.com> Message-ID: <3d375d730808152302x74d57cafu1f218c9aaeb4936a@mail.gmail.com> On Fri, Aug 15, 2008 at 19:30, Christian Heimes wrote: > Please also note that CPython uses a freelist of unused dict instances. > The default size of the dict free list is 80 elements. The allocation > and deallocation of dicts is cheap if you can stay below the threshold. That's very useful information. Thanks! -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cournape at gmail.com Sat Aug 16 02:47:53 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 16 Aug 2008 01:47:53 -0500 Subject: [Numpy-discussion] long double woes on win32 In-Reply-To: References: <5b8d13220808151643v4834397frd64b31707afbaa78@mail.gmail.com> Message-ID: <5b8d13220808152347s695a6b01mddc66c407a7ce147@mail.gmail.com> On Fri, Aug 15, 2008 at 7:11 PM, Charles R Harris wrote: > > Doesn't mingw use the MSVC library? Yes, it does. But long double is both a compiler and library issue. sizeof(long double) is defined by the compiler, and it is different with mingw and visual studio ATM. I don't know what the correct behavior should be, but the fact is that they are different. > are the same type, so this seems to be a bug in mingw. I suppose numpy could > detect this situation and define the types to be identical, but if this is > impractical, then perhaps the best thing to do is issue an error message. It is impractical in the short term because defining SIZE_LONG_DOUBLE to 8 breaks the mingw build, and fixing it would require fixing the whole math function configuration (the whole HAVE_LONGDOUBLE_FUNCS and co mess). That's highly non trivial code because of the tens of different possible configurations, and is fundamental to the whole numpy, since it affects the math functions effectively used. > There's isn't much you can do about long doubles while maintaining MSVC > compatibility. Does that mean you are in favor of letting things the way they are now for 1.2, and fix this on 1.3/1.2.1 ? cheers, David From strawman at astraw.com Sat Aug 16 03:21:35 2008 From: strawman at astraw.com (Andrew Straw) Date: Sat, 16 Aug 2008 09:21:35 +0200 Subject: [Numpy-discussion] [REVIEW] Update NumPy API format to support updates that don't break binary compatibility In-Reply-To: <9457e7c80808151610l78c07b32y5f203435d368a5b8@mail.gmail.com> References: <001636283760d7bea9045487b140@google.com> <9457e7c80808151610l78c07b32y5f203435d368a5b8@mail.gmail.com> Message-ID: <48A67FFF.904@astraw.com> Looking at the code, but not testing it -- this looks fine to me. (I wrote the original NPY_VERSION stuff and sent it to Travis, who modified and included it.) I have added a couple of extremely minor points to the code review tool -- as much as a chance to play with the tool as to comment on the code. -Andrew St?fan van der Walt wrote: > The current NumPy API number, stored as NPY_VERSION in the header files, needs > to be incremented every time the NumPy C-API changes. The counter tells > developers with exactly which revision of the API they are dealing. NumPy does > some checking to make sure that it does not run against an old version of the > API. Currently, we have no way of distinguishing between changes that break > binary compatibility and those that don't. > > The proposed fix breaks the version number up into two counters -- one that gets > increased when binary compatibility is broken, and another when the API is > changed without breaking compatibility. > > Backward compatibility with packages such as Matplotlib is maintained by > renaming NPY_VERSION to NPY_BINARY_VERSION. > > Please review the proposed change at http://codereview.appspot.com/2946 > > Regards > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From stefan at sun.ac.za Sat Aug 16 04:07:25 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sat, 16 Aug 2008 03:07:25 -0500 Subject: [Numpy-discussion] [REVIEW] Update NumPy API format to support updates that don't break binary compatibility In-Reply-To: <48A67FFF.904@astraw.com> References: <001636283760d7bea9045487b140@google.com> <9457e7c80808151610l78c07b32y5f203435d368a5b8@mail.gmail.com> <48A67FFF.904@astraw.com> Message-ID: <9457e7c80808160107w6c0570f7kb49c23f8a39997ec@mail.gmail.com> 2008/8/16 Andrew Straw : > Looking at the code, but not testing it -- this looks fine to me. (I > wrote the original NPY_VERSION stuff and sent it to Travis, who modified > and included it.) > > I have added a couple of extremely minor points to the code review tool > -- as much as a chance to play with the tool as to comment on the code. Excellent, thank you! I have updated the patch accordingly. If anybody else has anything to add, please do so. Unless there are objections, I'd like to merge the patch tomorrow. Regards St?fan From wright at esrf.fr Sat Aug 16 05:34:51 2008 From: wright at esrf.fr (Jon Wright) Date: Sat, 16 Aug 2008 11:34:51 +0200 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: <489CFF63.9000703@enthought.com> References: <489CFF63.9000703@enthought.com> Message-ID: <48A69F3B.9040506@esrf.fr> Travis, St?fan, I missed Travis mail previously. Are you *really* sure you want force all C code which uses numpy arrays to be recompiled? If you mean that all your matplotlib/PIL/pyopengl/etc users are going to have to make a co-ordinated upgrade, then this seems to be a grave mistake. Does St?fan's patch fix this problem to avoid all C code being recompiled? A co-ordinated recompile of *all* C code using numpy is certainly not a minimal effect! I really hope you can find a way around the recompiling, it is really a problem for anyone trying to distribute code as a python module for windows. Jon Travis E. Oliphant wrote: > Hi all, > > The 1.2 version of NumPy is going to be tagged. There is at least one > change I'd like to add: The hasobject member of the PyArray_Descr > structure should be renamed to "flags" and converted to a 32-bit > integer. > > What does everybody think about this change? It should have minimal > affect except to require a re-compile of extension modules using NumPy. > The only people requiring code changes would be those making intimate > use of the PyArray_Descr structure instead of using the macros. > > It's a simple change if there is no major opposition. > > -Travis > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From robert.kern at gmail.com Sat Aug 16 05:43:04 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 16 Aug 2008 04:43:04 -0500 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: <48A69F3B.9040506@esrf.fr> References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> Message-ID: <3d375d730808160243i1bed66a8se9b17214f9e05bcb@mail.gmail.com> On Sat, Aug 16, 2008 at 04:34, Jon Wright wrote: > Travis, St?fan, > > I missed Travis mail previously. Are you *really* sure you want force > all C code which uses numpy arrays to be recompiled? If you mean that > all your matplotlib/PIL/pyopengl/etc users are going to have to make a > co-ordinated upgrade, then this seems to be a grave mistake. FWIW, neither PIL nor PyOpenGL have C code which uses numpy arrays, so they are entirely unaffected. And this does not require an *upgrade* of the actually affected packages, just a rebuild of the binary. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From strawman at astraw.com Sat Aug 16 08:41:40 2008 From: strawman at astraw.com (Andrew Straw) Date: Sat, 16 Aug 2008 14:41:40 +0200 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: <3d375d730808160243i1bed66a8se9b17214f9e05bcb@mail.gmail.com> References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <3d375d730808160243i1bed66a8se9b17214f9e05bcb@mail.gmail.com> Message-ID: <48A6CB04.5070706@astraw.com> Robert Kern wrote: > On Sat, Aug 16, 2008 at 04:34, Jon Wright wrote: > >> Travis, St?fan, >> >> I missed Travis mail previously. Are you *really* sure you want force >> all C code which uses numpy arrays to be recompiled? If you mean that >> all your matplotlib/PIL/pyopengl/etc users are going to have to make a >> co-ordinated upgrade, then this seems to be a grave mistake. >> > > FWIW, neither PIL nor PyOpenGL have C code which uses numpy arrays, so > they are entirely unaffected. And this does not require an *upgrade* > of the actually affected packages, just a rebuild of the binary. > > I'll also point out that PEP 3118 will make this unnecessary in the future for many applications. http://www.python.org/dev/peps/pep-3118/ From what I can tell ( http://svn.python.org/view?rev=61491&view=rev ), this is already in Python 2.6. -Andrew From charlesr.harris at gmail.com Sat Aug 16 09:29:06 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 16 Aug 2008 07:29:06 -0600 Subject: [Numpy-discussion] long double woes on win32 In-Reply-To: <5b8d13220808152347s695a6b01mddc66c407a7ce147@mail.gmail.com> References: <5b8d13220808151643v4834397frd64b31707afbaa78@mail.gmail.com> <5b8d13220808152347s695a6b01mddc66c407a7ce147@mail.gmail.com> Message-ID: On Sat, Aug 16, 2008 at 12:47 AM, David Cournapeau wrote: > On Fri, Aug 15, 2008 at 7:11 PM, Charles R Harris > wrote: > > > > Doesn't mingw use the MSVC library? > > Yes, it does. But long double is both a compiler and library issue. > sizeof(long double) is defined by the compiler, and it is different > with mingw and visual studio ATM. I don't know what the correct > behavior should be, but the fact is that they are different. > > > are the same type, so this seems to be a bug in mingw. I suppose numpy > could > > detect this situation and define the types to be identical, but if this > is > > impractical, then perhaps the best thing to do is issue an error message. > > It is impractical in the short term because defining SIZE_LONG_DOUBLE > to 8 breaks the mingw build, and fixing it would require fixing the > whole math function configuration (the whole HAVE_LONGDOUBLE_FUNCS and > co mess). That's highly non trivial code because of the tens of > different possible configurations, and is fundamental to the whole > numpy, since it affects the math functions effectively used. > > > There's isn't much you can do about long doubles while maintaining MSVC > > compatibility. > > Does that mean you are in favor of letting things the way they are now > for 1.2, and fix this on 1.3/1.2.1 ? > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sat Aug 16 09:46:17 2008 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 16 Aug 2008 13:46:17 +0000 (UTC) Subject: [Numpy-discussion] C99 on windows References: <5b8d13220808131235p3e9f127s2a78c96813934138@mail.gmail.com> Message-ID: Hi, Sat, 16 Aug 2008 03:25:11 +0200, Christian Heimes wrote: > David Cournapeau wrote: >> The current trunk has 14 failures on windows (with mingw). 12 of them >> are related to C99 (see ticket 869). Can the people involved in recent >> changes to complex functions take a look at it ? I think this is high >> priority for 1.2.0 > > I'm asking just out of curiosity. Why is NumPy using C99 and what > features of C99 are used? The Microsoft compilers aren't supporting C99 > and they'll probably never will. I don't know if the Intel CC supports > C99. Even GCC doesn't implement C99 to its full extend. Are you planing > to restrict yourself to MinGW32? To clarify this again: *no* features of C99 were used. The C99 specs were only used as a guideline to what behavior we want of complex math functions, and I wrote tests for this, and marked failing ones as skipped. However, it turned out that different tests fail on different platforms, which means that the inf/nan handling of our complex-valued functions is effectively undefined. Eventually, most of the tests had to be marked as skipped, and so it made more sense to remove them altogether. -- Pauli Virtanen From charlesr.harris at gmail.com Sat Aug 16 10:11:01 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 16 Aug 2008 08:11:01 -0600 Subject: [Numpy-discussion] long double woes on win32 In-Reply-To: <5b8d13220808152347s695a6b01mddc66c407a7ce147@mail.gmail.com> References: <5b8d13220808151643v4834397frd64b31707afbaa78@mail.gmail.com> <5b8d13220808152347s695a6b01mddc66c407a7ce147@mail.gmail.com> Message-ID: On Sat, Aug 16, 2008 at 12:47 AM, David Cournapeau wrote: > On Fri, Aug 15, 2008 at 7:11 PM, Charles R Harris > wrote: > > > > Doesn't mingw use the MSVC library? > > Yes, it does. But long double is both a compiler and library issue. > sizeof(long double) is defined by the compiler, and it is different > with mingw and visual studio ATM. I don't know what the correct > behavior should be, but the fact is that they are different. > > > are the same type, so this seems to be a bug in mingw. I suppose numpy > could > > detect this situation and define the types to be identical, but if this > is > > impractical, then perhaps the best thing to do is issue an error message. > > It is impractical in the short term because defining SIZE_LONG_DOUBLE > to 8 breaks the mingw build, and fixing it would require fixing the > whole math function configuration (the whole HAVE_LONGDOUBLE_FUNCS and > co mess). That's highly non trivial code because of the tens of > different possible configurations, and is fundamental to the whole > numpy, since it affects the math functions effectively used. > > > There's isn't much you can do about long doubles while maintaining MSVC > > compatibility. > > Does that mean you are in favor of letting things the way they are now > for 1.2, and fix this on 1.3/1.2.1 ? > Yes. I don't think MS will support "true" long doubles any time soon and this affects printing and the math functions. I'm not sure what the best solution is, there are various possibilities. 1) We could define the numpy longdouble type to be double, which makes us compatible with MS and is effectively what numpy compiled with MSVC does since MSVC long doubles are doubles. Perhaps this could be done by adding a -DNOLONGDOUBLE flag to the compiler flags and then defining longdouble to double. But this means auditing the code to make sure the long double type is never explicitly used so that the mingw compiler does the right thing. I don't think this should be a problem otherwise except for the loss of "true" long doubles and their extra bit of precision. 2) We can keep the mingw "true" long doubles and avoid any reliance on the MS library. This may already be done for the math functions by effectively computing the long double functions as doubles, but I will have to check on that. In any case, some of the usefulness of long doubles will still be lost. 3) We can write our own library functions. That seems like a lot of work but perhaps there is something in the BSD library we could use. I think BSD uses the gnu compiler but has it's own libraries. In any case, these solutions all look to be more than two weeks down the road, so in the short term we probably have to go with the current behaviour. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at cheimes.de Sat Aug 16 10:21:12 2008 From: lists at cheimes.de (Christian Heimes) Date: Sat, 16 Aug 2008 16:21:12 +0200 Subject: [Numpy-discussion] C99 on windows In-Reply-To: References: <5b8d13220808131235p3e9f127s2a78c96813934138@mail.gmail.com> Message-ID: Pauli Virtanen wrote: > To clarify this again: *no* features of C99 were used. The C99 specs were > only used as a guideline to what behavior we want of complex math > functions, and I wrote tests for this, and marked failing ones as skipped. Got it. > However, it turned out that different tests fail on different platforms, > which means that the inf/nan handling of our complex-valued functions is > effectively undefined. Eventually, most of the tests had to be marked as > skipped, and so it made more sense to remove them altogether. We hit the same issue during our work for Python 2.6 and 3.0. We came to the conclusion that we can't rely on the platform's math functions (especially trigonometric and hyperbolic functions) for special values. So Mark came up with the idea of lookup tables for special values. Read my other mail for more informations. Christian From charlesr.harris at gmail.com Sat Aug 16 10:44:35 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 16 Aug 2008 08:44:35 -0600 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: <3d375d730808160243i1bed66a8se9b17214f9e05bcb@mail.gmail.com> References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <3d375d730808160243i1bed66a8se9b17214f9e05bcb@mail.gmail.com> Message-ID: On Sat, Aug 16, 2008 at 3:43 AM, Robert Kern wrote: > On Sat, Aug 16, 2008 at 04:34, Jon Wright wrote: > > Travis, St?fan, > > > > I missed Travis mail previously. Are you *really* sure you want force > > all C code which uses numpy arrays to be recompiled? If you mean that > > all your matplotlib/PIL/pyopengl/etc users are going to have to make a > > co-ordinated upgrade, then this seems to be a grave mistake. > > FWIW, neither PIL nor PyOpenGL have C code which uses numpy arrays, so > they are entirely unaffected. And this does not require an *upgrade* > of the actually affected packages, just a rebuild of the binary. > What about SAGE, MPL, PyTables, and friends? I don't really know what all depends on Numpy at this point, but I think Ron's point about Windows packages is a good one. On Linux I just compile all these from svn anyway, but I suspect windows users mostly depend on precompiled packages. And this probably also effects packagers for Linux binary distros like ubuntu and fedora also, as they have to recompile all those packages and keep the dependencies straight. True, most of the packages trail numpy development by some time, but that is also an arguement against the urge to push things in early. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sat Aug 16 11:06:56 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 16 Aug 2008 10:06:56 -0500 Subject: [Numpy-discussion] long double woes on win32 In-Reply-To: References: <5b8d13220808151643v4834397frd64b31707afbaa78@mail.gmail.com> <5b8d13220808152347s695a6b01mddc66c407a7ce147@mail.gmail.com> Message-ID: <5b8d13220808160806rb6e7842x5cf7d3801c5d7e62@mail.gmail.com> On Sat, Aug 16, 2008 at 9:11 AM, Charles R Harris wrote: > > 1) We could define the numpy longdouble type to be double, which makes us > compatible with MS and is effectively what numpy compiled with MSVC does > since MSVC long doubles are doubles. Perhaps this could be done by adding a > -DNOLONGDOUBLE flag to the compiler flags and then defining longdouble to > double. But this means auditing the code to make sure the long double type > is never explicitly used so that the mingw compiler does the right thing. I > don't think this should be a problem otherwise except for the loss of "true" > long doubles and their extra bit of precision. According to Travis, we never use long double, always npy_long_double. But as I mentioned above, forcing a mode with long double being effectively double breaks the configuration. This is the problem, because fixing it (which I will do once the trunk is open for 1.3) requires some non trivial changes in the configuration, namely: we look for some math functions (says acoshf) and assumes that finding one guarantees all of the same kind (float versions of hyperbolic float functions) exist, which only worked on some platforms, but not on others. The solution is not complicated (testing for every function + bisecting to avoid a too long configuration stage), but will need thorough testing. > 2) We can keep the mingw "true" long doubles and avoid any reliance on the > MS library. This may already be done for the math functions by effectively > computing the long double functions as doubles, but I will have to check on > that. In any case, some of the usefulness of long doubles will still be > lost. The problem is not only the computation, but all the support around: printf, etc... (the current test failures are in those functions) And anyway, since python is linked against the MS C runtime, I think it is kind of pointless to do so. > 3) We can write our own library functions. That seems like a lot of work but > perhaps there is something in the BSD library we could use. I think BSD uses > the gnu compiler but has it's own libraries. Yes, the C library in BSD* is certainly not the glibc. But see above for the limited usefulness of this. We have to use long double as double in the configuration; the question is not if, but how (using VS for binaries, updating mingw to fix its broken configuration when sizeof(npy_double) == sizeof(npy_longdouble)) and when. cheers, David From oliphant at enthought.com Sat Aug 16 11:47:25 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sat, 16 Aug 2008 10:47:25 -0500 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: <48A69F3B.9040506@esrf.fr> References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> Message-ID: <48A6F68D.3020006@enthought.com> Jon Wright wrote: > Travis, St?fan, > > I missed Travis mail previously. Are you *really* sure you want force > all C code which uses numpy arrays to be recompiled? Re-compilation is necessary at some point. We have not required recompilation for a long time now. Yes, it is a pain for distribution, but those who don't want to re-compile can point people to 1.1.1 which will still work until they make a new release compiled against the newer NumPy. I would encourage people to make use of things like Python(x,y), EPD, and SAGE for their distribution needs. -Travis From oliphant at enthought.com Sat Aug 16 11:50:53 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sat, 16 Aug 2008 10:50:53 -0500 Subject: [Numpy-discussion] long double woes on win32 In-Reply-To: References: <5b8d13220808151643v4834397frd64b31707afbaa78@mail.gmail.com> <5b8d13220808152347s695a6b01mddc66c407a7ce147@mail.gmail.com> Message-ID: <48A6F75D.8020004@enthought.com> Charles R Harris wrote: > > > Yes. I don't think MS will support "true" long doubles any time soon > and this affects printing and the math functions. I'm not sure what > the best solution is, there are various possibilities. > > 1) We could define the numpy longdouble type to be double, which makes > us compatible with MS and is effectively what numpy compiled with MSVC > does since MSVC long doubles are doubles. Perhaps this could be done > by adding a -DNOLONGDOUBLE flag to the compiler flags and then > defining longdouble to double. But this means auditing the code to > make sure the long double type is never explicitly used so that the > mingw compiler does the right thing. I don't think this should be a > problem otherwise except for the loss of "true" long doubles and their > extra bit of precision. All code in numpy uses npy_longdouble which is typedef'd to double if SIZEOF_LONGDOUBLE is the same as SIZEOF_DOUBLE. I don't understand why it's a problem to just define SIZEOF_LONGDOUBLE = 8 for mingw which is what it is for MSVC. This makes longdouble 8 bytes. -Travis From charlesr.harris at gmail.com Sat Aug 16 12:15:37 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 16 Aug 2008 10:15:37 -0600 Subject: [Numpy-discussion] long double woes on win32 In-Reply-To: <48A6F75D.8020004@enthought.com> References: <5b8d13220808151643v4834397frd64b31707afbaa78@mail.gmail.com> <5b8d13220808152347s695a6b01mddc66c407a7ce147@mail.gmail.com> <48A6F75D.8020004@enthought.com> Message-ID: On Sat, Aug 16, 2008 at 9:50 AM, Travis E. Oliphant wrote: > Charles R Harris wrote: > > > > > > Yes. I don't think MS will support "true" long doubles any time soon > > and this affects printing and the math functions. I'm not sure what > > the best solution is, there are various possibilities. > > > > 1) We could define the numpy longdouble type to be double, which makes > > us compatible with MS and is effectively what numpy compiled with MSVC > > does since MSVC long doubles are doubles. Perhaps this could be done > > by adding a -DNOLONGDOUBLE flag to the compiler flags and then > > defining longdouble to double. But this means auditing the code to > > make sure the long double type is never explicitly used so that the > > mingw compiler does the right thing. I don't think this should be a > > problem otherwise except for the loss of "true" long doubles and their > > extra bit of precision. > All code in numpy uses npy_longdouble which is typedef'd to double if > SIZEOF_LONGDOUBLE is the same as SIZEOF_DOUBLE. I don't understand why > it's a problem to just define SIZEOF_LONGDOUBLE = 8 for mingw which is > what it is for MSVC. This makes longdouble 8 bytes. > That's my opinion also, I just thought that -DNOLONGDOUBLE was an easy way to force that choice. David thinks that the function detection in the ufunc module will be a problem. I need to take a look, perhaps there is a dependency on include files that differ between the two compilers. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sat Aug 16 13:10:58 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 16 Aug 2008 12:10:58 -0500 Subject: [Numpy-discussion] long double woes on win32 In-Reply-To: References: <5b8d13220808151643v4834397frd64b31707afbaa78@mail.gmail.com> <5b8d13220808152347s695a6b01mddc66c407a7ce147@mail.gmail.com> <48A6F75D.8020004@enthought.com> Message-ID: <5b8d13220808161010q4b8232b1lfe8856159e77f757@mail.gmail.com> On Sat, Aug 16, 2008 at 11:15 AM, Charles R Harris wrote: > > That's my opinion also, I just thought that -DNOLONGDOUBLE was an easy way > to force that choice. David thinks that the function detection in the ufunc > module will be a problem. Forget what I said, I think I used a broken mingw toolchain, which caused the problems I got. In any case, I cannot reproduce the problem anymore. But forcing SIZEOF_LONG_DOUBLE to 8 at configuration stage caused test failures too (at different points, though): http://scipy.org/scipy/numpy/ticket/890 The first one looks strange. From charlesr.harris at gmail.com Sat Aug 16 13:15:02 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 16 Aug 2008 11:15:02 -0600 Subject: [Numpy-discussion] long double woes on win32 In-Reply-To: <5b8d13220808161010q4b8232b1lfe8856159e77f757@mail.gmail.com> References: <5b8d13220808151643v4834397frd64b31707afbaa78@mail.gmail.com> <5b8d13220808152347s695a6b01mddc66c407a7ce147@mail.gmail.com> <48A6F75D.8020004@enthought.com> <5b8d13220808161010q4b8232b1lfe8856159e77f757@mail.gmail.com> Message-ID: On Sat, Aug 16, 2008 at 11:10 AM, David Cournapeau wrote: > On Sat, Aug 16, 2008 at 11:15 AM, Charles R Harris > wrote: > > > > > That's my opinion also, I just thought that -DNOLONGDOUBLE was an easy > way > > to force that choice. David thinks that the function detection in the > ufunc > > module will be a problem. > > Forget what I said, I think I used a broken mingw toolchain, which > caused the problems I got. In any case, I cannot reproduce the problem > anymore. > > But forcing SIZEOF_LONG_DOUBLE to 8 at configuration stage caused test > failures too (at different points, though): > > http://scipy.org/scipy/numpy/ticket/890 > > The first one looks strange. I was just going to look at that; it's nice to have the ticket mailing list working again. Is there an easy way to force the SIZEOF_LONG_DOUBLE to 8 so I can test this on linux? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sat Aug 16 13:24:59 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 16 Aug 2008 12:24:59 -0500 Subject: [Numpy-discussion] long double woes on win32 In-Reply-To: References: <5b8d13220808151643v4834397frd64b31707afbaa78@mail.gmail.com> <5b8d13220808152347s695a6b01mddc66c407a7ce147@mail.gmail.com> <48A6F75D.8020004@enthought.com> <5b8d13220808161010q4b8232b1lfe8856159e77f757@mail.gmail.com> Message-ID: <5b8d13220808161024q67d1484fi34390e00ccde6675@mail.gmail.com> On Sat, Aug 16, 2008 at 12:15 PM, Charles R Harris wrote: > > I was just going to look at that; it's nice to have the ticket mailing list > working again. Is there an easy way to force the SIZEOF_LONG_DOUBLE to 8 so > I can test this on linux? Changing this line in numpy?core?setup.py: - ('SIZEOF_LONG_DOUBLE', 'long double'), + ('SIZEOF_LONG_DOUBLE', 'double'), is what I did to get the result on windows. But it only "works" because I know the C runtime really has long double of 8 bytes. On platforms where it is not true, it is likely to break things. cheers, David From cournape at gmail.com Sat Aug 16 13:26:57 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 16 Aug 2008 12:26:57 -0500 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: <48A6F68D.3020006@enthought.com> References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <48A6F68D.3020006@enthought.com> Message-ID: <5b8d13220808161026t31368596ydf1b5bbf7b7e86a7@mail.gmail.com> On Sat, Aug 16, 2008 at 10:47 AM, Travis E. Oliphant wrote: > > Re-compilation is necessary at some point. We have not required > recompilation for a long time now. Yes, it is a pain for > distribution, but those who don't want to re-compile can point people to > 1.1.1 which will still work until they make a new release compiled > against the newer NumPy. Does that mean we will continue breaking the ABI from time to time during the 1.* cycle ? cheers, David From charlesr.harris at gmail.com Sat Aug 16 13:39:11 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 16 Aug 2008 11:39:11 -0600 Subject: [Numpy-discussion] long double woes on win32 In-Reply-To: <5b8d13220808161024q67d1484fi34390e00ccde6675@mail.gmail.com> References: <5b8d13220808151643v4834397frd64b31707afbaa78@mail.gmail.com> <5b8d13220808152347s695a6b01mddc66c407a7ce147@mail.gmail.com> <48A6F75D.8020004@enthought.com> <5b8d13220808161010q4b8232b1lfe8856159e77f757@mail.gmail.com> <5b8d13220808161024q67d1484fi34390e00ccde6675@mail.gmail.com> Message-ID: On Sat, Aug 16, 2008 at 11:24 AM, David Cournapeau wrote: > On Sat, Aug 16, 2008 at 12:15 PM, Charles R Harris > wrote: > > > > I was just going to look at that; it's nice to have the ticket mailing > list > > working again. Is there an easy way to force the SIZEOF_LONG_DOUBLE to 8 > so > > I can test this on linux? > > Changing this line in numpy?core?setup.py: > > - ('SIZEOF_LONG_DOUBLE', 'long double'), > + ('SIZEOF_LONG_DOUBLE', 'double'), > > is what I did to get the result on windows. But it only "works" > because I know the C runtime really has long double of 8 bytes. On > platforms where it is not true, it is likely to break things. > Hmm. ISTM that numpy should be set up so that the change works on all platforms. However, making it so might be something else. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wright at esrf.fr Sat Aug 16 13:44:32 2008 From: wright at esrf.fr (Jon Wright) Date: Sat, 16 Aug 2008 19:44:32 +0200 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: <5b8d13220808161026t31368596ydf1b5bbf7b7e86a7@mail.gmail.com> References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <48A6F68D.3020006@enthought.com> <5b8d13220808161026t31368596ydf1b5bbf7b7e86a7@mail.gmail.com> Message-ID: <48A71200.6050000@esrf.fr> David Cournapeau wrote: > Does that mean we will continue breaking the ABI from time to time > during the 1.* cycle ? Can someone help me to understand me what is the compelling reason for this change? If it only means everyone recompiles, it is hard to see what we, as users, are gaining by doing that. Thanks, Jon From charlesr.harris at gmail.com Sat Aug 16 13:56:06 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 16 Aug 2008 11:56:06 -0600 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: <48A71200.6050000@esrf.fr> References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <48A6F68D.3020006@enthought.com> <5b8d13220808161026t31368596ydf1b5bbf7b7e86a7@mail.gmail.com> <48A71200.6050000@esrf.fr> Message-ID: On Sat, Aug 16, 2008 at 11:44 AM, Jon Wright wrote: > David Cournapeau wrote: > > Does that mean we will continue breaking the ABI from time to time > > during the 1.* cycle ? > > > Can someone help me to understand me what is the compelling reason for > this change? If it only means everyone recompiles, it is hard to see > what we, as users, are gaining by doing that. > Turns out that ipython needs to be recompiled also because of the newly added version checking. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Aug 16 14:05:01 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 16 Aug 2008 12:05:01 -0600 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <48A6F68D.3020006@enthought.com> <5b8d13220808161026t31368596ydf1b5bbf7b7e86a7@mail.gmail.com> <48A71200.6050000@esrf.fr> Message-ID: On Sat, Aug 16, 2008 at 11:56 AM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Sat, Aug 16, 2008 at 11:44 AM, Jon Wright wrote: > >> David Cournapeau wrote: >> > Does that mean we will continue breaking the ABI from time to time >> > during the 1.* cycle ? >> >> >> Can someone help me to understand me what is the compelling reason for >> this change? If it only means everyone recompiles, it is hard to see >> what we, as users, are gaining by doing that. >> > > Turns out that ipython needs to be recompiled also because of the newly > added version checking. > Looks like there is a bug in the new API version tracking: >>> import numpy as np RuntimeError: module compiled against version 1000009 of C-API but this version of numpy is 100000a Traceback (most recent call last): File "", line 1, in File "/usr/lib/python2.5/site-packages/numpy/__init__.py", line 132, in import add_newdocs File "/usr/lib/python2.5/site-packages/numpy/add_newdocs.py", line 9, in from lib import add_newdoc File "/usr/lib/python2.5/site-packages/numpy/lib/__init__.py", line 4, in from type_check import * File "/usr/lib/python2.5/site-packages/numpy/lib/type_check.py", line 8, in import numpy.core.numeric as _nx File "/usr/lib/python2.5/site-packages/numpy/core/__init__.py", line 19, in import scalarmath ImportError: numpy.core.multiarray failed to import Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Aug 16 14:17:47 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 16 Aug 2008 12:17:47 -0600 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <48A6F68D.3020006@enthought.com> <5b8d13220808161026t31368596ydf1b5bbf7b7e86a7@mail.gmail.com> <48A71200.6050000@esrf.fr> Message-ID: On Sat, Aug 16, 2008 at 12:05 PM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Sat, Aug 16, 2008 at 11:56 AM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Sat, Aug 16, 2008 at 11:44 AM, Jon Wright wrote: >> >>> David Cournapeau wrote: >>> > Does that mean we will continue breaking the ABI from time to time >>> > during the 1.* cycle ? >>> >>> >>> Can someone help me to understand me what is the compelling reason for >>> this change? If it only means everyone recompiles, it is hard to see >>> what we, as users, are gaining by doing that. >>> >> >> Turns out that ipython needs to be recompiled also because of the newly >> added version checking. >> > > Looks like there is a bug in the new API version tracking: > > >>> import numpy as np > RuntimeError: module compiled against version 1000009 of C-API but this > version of numpy is 100000a > Traceback (most recent call last): > File "", line 1, in > File "/usr/lib/python2.5/site-packages/numpy/__init__.py", line 132, in > > import add_newdocs > File "/usr/lib/python2.5/site-packages/numpy/add_newdocs.py", line 9, in > > from lib import add_newdoc > File "/usr/lib/python2.5/site-packages/numpy/lib/__init__.py", line 4, in > > from type_check import * > File "/usr/lib/python2.5/site-packages/numpy/lib/type_check.py", line 8, > in > import numpy.core.numeric as _nx > File "/usr/lib/python2.5/site-packages/numpy/core/__init__.py", line 19, > in > import scalarmath > ImportError: numpy.core.multiarray failed to import > False alarm. I needed to remove the build directory first. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Aug 16 15:07:44 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 16 Aug 2008 13:07:44 -0600 Subject: [Numpy-discussion] long double woes on win32 In-Reply-To: References: <5b8d13220808151643v4834397frd64b31707afbaa78@mail.gmail.com> <5b8d13220808152347s695a6b01mddc66c407a7ce147@mail.gmail.com> <48A6F75D.8020004@enthought.com> <5b8d13220808161010q4b8232b1lfe8856159e77f757@mail.gmail.com> <5b8d13220808161024q67d1484fi34390e00ccde6675@mail.gmail.com> Message-ID: On Sat, Aug 16, 2008 at 11:39 AM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Sat, Aug 16, 2008 at 11:24 AM, David Cournapeau wrote: > >> On Sat, Aug 16, 2008 at 12:15 PM, Charles R Harris >> wrote: >> > >> > I was just going to look at that; it's nice to have the ticket mailing >> list >> > working again. Is there an easy way to force the SIZEOF_LONG_DOUBLE to 8 >> so >> > I can test this on linux? >> >> Changing this line in numpy?core?setup.py: >> >> - ('SIZEOF_LONG_DOUBLE', 'long double'), >> + ('SIZEOF_LONG_DOUBLE', 'double'), >> >> is what I did to get the result on windows. But it only "works" >> because I know the C runtime really has long double of 8 bytes. On >> platforms where it is not true, it is likely to break things. >> > > Hmm. ISTM that numpy should be set up so that the change works on all > platforms. However, making it so might be something else. > Almost works, I get the same two failures as you plus a failure in test_precisions_consistent. ERROR: Test generic loops. ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/numpy/core/tests/test_ufunc.py", line 79, in test_generic_loops assert_almost_equal(fone(x), fone_val, err_msg=msg) File "/usr/lib/python2.5/site-packages/numpy/testing/utils.py", line 205, in assert_almost_equal return assert_array_almost_equal(actual, desired, decimal, err_msg) File "/usr/lib/python2.5/site-packages/numpy/testing/utils.py", line 304, in assert_array_almost_equal header='Arrays are not almost equal') File "/usr/lib/python2.5/site-packages/numpy/testing/utils.py", line 272, in assert_array_compare val = comparison(x[~xnanid], y[~ynanid]) IndexError: 0-d arrays can't be indexed ====================================================================== FAIL: test_large_types (test_scalarmath.TestPower) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/numpy/core/tests/test_scalarmath.py", line 54, in test_large_types assert_almost_equal(b, 6765201, err_msg=msg) File "/usr/lib/python2.5/site-packages/numpy/testing/utils.py", line 207, in assert_almost_equal assert round(abs(desired - actual),decimal) == 0, msg AssertionError: Items are not equal: error with : got inf ACTUAL: inf DESIRED: 6765201 ====================================================================== FAIL: test_umath.TestComplexFunctions.test_precisions_consistent ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/nose/case.py", line 203, in runTest self.test(*self.arg) File "/usr/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", line 206, in test_precisions_consistent assert_almost_equal(fcl, fcd, decimal=15, err_msg='fch-fcl %s'%f) File "/usr/lib/python2.5/site-packages/numpy/testing/utils.py", line 207, in assert_almost_equal assert round(abs(desired - actual),decimal) == 0, msg AssertionError: Items are not equal: fch-fcl ACTUAL: (0.66623943249251527+1.0612750619050355j) DESIRED: (0.66623943249251527+1.0612750619050355j) ====================================================================== SKIP: test_umath.TestComplexFunctions.test_branch_cuts_failing ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/nose/case.py", line 203, in runTest self.test(*self.arg) File "/usr/lib/python2.5/site-packages/numpy/testing/decorators.py", line 93, in skipper raise nose.SkipTest, 'This test is known to fail' SkipTest: This test is known to fail There is seems to be a problem in defining the functions called for the different types. In [3]: x = np.zeros(10, dtype=np.longdouble)[0::2] In [4]: x.dtype.itemsize Out[4]: 8 In [5]: x Out[5]: array([0.0, 0.0, 0.0, 0.0, 0.0], dtype=float64) In [6]: np.exp(x) Out[6]: array([NaN, NaN, NaN, NaN, NaN], dtype=float64) In [7]: np.sin(x) Out[7]: array([NaN, NaN, NaN, NaN, NaN], dtype=float64) In [8]: x.dtype.char Out[8]: 'g' If I force the function to the double version this bit works fine. The odd error message is a bug in numpy.testing where assert_array_compare fails for arrays that contain only nan's. They all get masked out. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Sat Aug 16 15:26:49 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 16 Aug 2008 12:26:49 -0700 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <48A6F68D.3020006@enthought.com> <5b8d13220808161026t31368596ydf1b5bbf7b7e86a7@mail.gmail.com> <48A71200.6050000@esrf.fr> Message-ID: On Sat, Aug 16, 2008 at 10:56 AM, Charles R Harris wrote: > Turns out that ipython needs to be recompiled also because of the newly > added version checking. I'm sorry, can you clarify this? ipython has no C code at all, so I'm not sure what you mean here. Cheers, f From charlesr.harris at gmail.com Sat Aug 16 15:35:45 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 16 Aug 2008 13:35:45 -0600 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <48A6F68D.3020006@enthought.com> <5b8d13220808161026t31368596ydf1b5bbf7b7e86a7@mail.gmail.com> <48A71200.6050000@esrf.fr> Message-ID: On Sat, Aug 16, 2008 at 1:26 PM, Fernando Perez wrote: > On Sat, Aug 16, 2008 at 10:56 AM, Charles R Harris > wrote: > > > Turns out that ipython needs to be recompiled also because of the newly > > added version checking. > > I'm sorry, can you clarify this? ipython has no C code at all, so I'm > not sure what you mean here. > It was a false alarm. I was calling ipython with pylab and MPL needs to be recompiled. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sat Aug 16 16:41:10 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 16 Aug 2008 15:41:10 -0500 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: <48A71200.6050000@esrf.fr> References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <48A6F68D.3020006@enthought.com> <5b8d13220808161026t31368596ydf1b5bbf7b7e86a7@mail.gmail.com> <48A71200.6050000@esrf.fr> Message-ID: <5b8d13220808161341k25fbc75fxbe9b820d0995c8b0@mail.gmail.com> On Sat, Aug 16, 2008 at 12:44 PM, Jon Wright wrote: > David Cournapeau wrote: >> Does that mean we will continue breaking the ABI from time to time >> during the 1.* cycle ? > > > Can someone help me to understand me what is the compelling reason for > this change? If it only means everyone recompiles, it is hard to see > what we, as users, are gaining by doing that. Breaking the ABI forces users to recompile, that's pretty much the definition of ABI breaks. Unfortunately, up to now, we did not have a mechanism to make the difference between extending the API (which can be done without breaking the ABI if done carefully) and breaking the API (which induces ABI breakage). Hopefully, once this is in place, we won't have to do it anymore. cheers, David From cournape at gmail.com Sat Aug 16 16:46:01 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 16 Aug 2008 15:46:01 -0500 Subject: [Numpy-discussion] long double woes on win32 In-Reply-To: References: <5b8d13220808151643v4834397frd64b31707afbaa78@mail.gmail.com> <5b8d13220808152347s695a6b01mddc66c407a7ce147@mail.gmail.com> <48A6F75D.8020004@enthought.com> <5b8d13220808161010q4b8232b1lfe8856159e77f757@mail.gmail.com> <5b8d13220808161024q67d1484fi34390e00ccde6675@mail.gmail.com> Message-ID: <5b8d13220808161346m39d7302coa010dfc5c89ca8f@mail.gmail.com> On Sat, Aug 16, 2008 at 2:07 PM, Charles R Harris wrote: > > > There is seems to be a problem in defining the functions called for the > different types. I don't know enough about this part of the code to be sure about the whole function calls stack, but I would guess this is not surprising and even expected: you force long double to be double, but you still call the long double function if you do np.exp, so this cannot work. On windows, it should, because the runtime (e.g. the C function expl) does actually expects a double (expl expects an 8 bytes value). But on mac os X, it is not necessarily the case (although it could, I don't know the status of long double on that platform). cheers, David From charlesr.harris at gmail.com Sat Aug 16 16:47:49 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 16 Aug 2008 14:47:49 -0600 Subject: [Numpy-discussion] long double woes on win32 In-Reply-To: References: <5b8d13220808151643v4834397frd64b31707afbaa78@mail.gmail.com> <5b8d13220808152347s695a6b01mddc66c407a7ce147@mail.gmail.com> <48A6F75D.8020004@enthought.com> <5b8d13220808161010q4b8232b1lfe8856159e77f757@mail.gmail.com> <5b8d13220808161024q67d1484fi34390e00ccde6675@mail.gmail.com> Message-ID: On Sat, Aug 16, 2008 at 1:07 PM, Charles R Harris wrote: > > > On Sat, Aug 16, 2008 at 11:39 AM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Sat, Aug 16, 2008 at 11:24 AM, David Cournapeau wrote: >> >>> On Sat, Aug 16, 2008 at 12:15 PM, Charles R Harris >>> wrote: >>> > >>> > I was just going to look at that; it's nice to have the ticket mailing >>> list >>> > working again. Is there an easy way to force the SIZEOF_LONG_DOUBLE to >>> 8 so >>> > I can test this on linux? >>> >>> Changing this line in numpy?core?setup.py: >>> >>> - ('SIZEOF_LONG_DOUBLE', 'long double'), >>> + ('SIZEOF_LONG_DOUBLE', 'double'), >>> >>> is what I did to get the result on windows. But it only "works" >>> because I know the C runtime really has long double of 8 bytes. On >>> platforms where it is not true, it is likely to break things. >>> >> >> Hmm. ISTM that numpy should be set up so that the change works on all >> platforms. However, making it so might be something else. >> > > Almost works, I get the same two failures as you plus a failure in > test_precisions_consistent. > Simply undefining HAVE_LONGDOUBLE_FUNCS doesn't work because the ufuncs are defined the usual way with the l/f suffixes and there is a conflict with the math.h include file. We could fix that by defining our own functions, i.e., #define npy_sinl sin etc. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Aug 16 17:02:16 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 16 Aug 2008 15:02:16 -0600 Subject: [Numpy-discussion] long double woes on win32 In-Reply-To: <5b8d13220808161346m39d7302coa010dfc5c89ca8f@mail.gmail.com> References: <5b8d13220808151643v4834397frd64b31707afbaa78@mail.gmail.com> <48A6F75D.8020004@enthought.com> <5b8d13220808161010q4b8232b1lfe8856159e77f757@mail.gmail.com> <5b8d13220808161024q67d1484fi34390e00ccde6675@mail.gmail.com> <5b8d13220808161346m39d7302coa010dfc5c89ca8f@mail.gmail.com> Message-ID: On Sat, Aug 16, 2008 at 2:46 PM, David Cournapeau wrote: > On Sat, Aug 16, 2008 at 2:07 PM, Charles R Harris > wrote: > > > > > > There is seems to be a problem in defining the functions called for the > > different types. > > I don't know enough about this part of the code to be sure about the > whole function calls stack, but I would guess this is not surprising > and even expected: you force long double to be double, but you still > call the long double function if you do np.exp, so this cannot work. Part of the problem is that the loops/functions called depend on the typecode, i.e., dtype.char, which stays as 'g' even when the underlying type is float64. We can't really fix that because the C-API still allows folks to create arrays using the typecode and we would have to fudge things so the right "wrong" descr was returned. Ugly, and a left over from numeric which defined types to match the c types instead of the precision. Maybe we should rewrite in FORTRAN ;) Anyway, I think the easiest solution might be to use npy_func internally and add a -DNOLONGDOUBLE flag to override the values returned by the distutils test code. Whether we should try such a big modification before 1.2 is another question. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wright at esrf.fr Sat Aug 16 17:02:56 2008 From: wright at esrf.fr (Jon Wright) Date: Sat, 16 Aug 2008 23:02:56 +0200 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: <3d375d730808160243i1bed66a8se9b17214f9e05bcb@mail.gmail.com> References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <3d375d730808160243i1bed66a8se9b17214f9e05bcb@mail.gmail.com> Message-ID: <48A74080.2010002@esrf.fr> Robert Kern wrote: > > FWIW, neither PIL nor PyOpenGL have C code which uses numpy arrays, so > they are entirely unaffected. OK, so here are some projects which might notice a 1.2 installation, in as much as they turn up on a google code search for: #include "numpy/arrayobject.h" -scipy -enthought ... so one might imagine their programs will suddenly stop working. Based on the first 40 of >200 hits, these projects seem to offer binary windows downloads and have a C source which includes numpy/arrayobject.h : PyQwt3d ScientificPython RPy PyTables pygsl VPython bayes-blocks http://forge.pascal-network.org/projects/bblocks/ fdopython http://fdo.osgeo.org For these ones I only found source, so they'd be daft to complain about a program that previously worked just fine has stopped working: neuron (www.neuron.yale.edu) pysl - bridge between python and S-lang BGL - Biggles Graphics Library www.eos.ubc.ca/research/clouds/software.html astronomyworks.googlecode.com pyrap.google.com pyroms.googlecode.com pyamg.googlecode.com mdanalysis.googlecode.com pythoncall - python to matlab interface code.astraw.com/projects/motmot (It is impressive that there are so many users out there, and it turns out that this is a great way to find interesting code.) Even if there turn out to be a lot of duplicates in the >200 hits, already most of those projects have notices saying things like "be sure to get numpy, not Numeric or numarray". Do you want all of them to be delivering a matrix of binaries for different python versions multiplied by numpy <1.1.x versus >1.2.x ? What happens when someone wants to use *two* modules at the time, but one is distributing 1.1 binaries and the other has 1.2? The main reason I changed to numpy was that you stopped doing the Numeric binaries for python2.5. There was no way to distribute my own code for 2.5 without shipping Numeric too, which I didn't want to do, given that you were trying to get everyone to switch. What is the cool new feature that 1.2 has gained which is making all this worthwhile? Are you really 100% certain you need that new feature enough to make all those strangers need to do all that work? Can you give a concrete example of something which is gained by: > "The hasobject member of the PyArray_Descr structure should be renamed to "flags" and converted to a 32-bit integer." Try to look 12 months into the future and ask yourselves if it was really a good idea to break the ABI. Jon From dpeterson at enthought.com Sat Aug 16 17:53:32 2008 From: dpeterson at enthought.com (Dave Peterson) Date: Sat, 16 Aug 2008 16:53:32 -0500 Subject: [Numpy-discussion] ANNOUNCE: ETS 3.0.0 released! Message-ID: <48A74C5C.4050404@enthought.com> Hello, I'm pleased to announce that ETS 3.0.0 has just been tagged and released! Source distributions have been pushed to PyPi and over the next couple hours, Win32 and OSX binaries will also be uploaded to PyPi. This means you can install ETS, assuming you have the prereq software installed, via the simple command: easy_install ETS[nonets] Please see the Install page on our wiki for more detailed installation instructions: https://svn.enthought.com/enthought/wiki/Install Developers of ETS will find that the projects' trunks have already been bumped up to the next version numbers so a simple "ets up" (or svn up) should bring you up to date. Others may wish to grab a complete new checkout via a "ets co ETS". The release branches that had been created are now removed. The next release is currently expected to be ETS 3.0.1 -- Dave Enthought Tool Suite --------------------------- The Enthought Tool Suite (ETS) is a collection of components developed by Enthought and open source participants, which we use every day to construct custom scientific applications. It includes a wide variety of components, including: * an extensible application framework * application building blocks * 2-D and 3-D graphics libraries * scientific and math libraries * developer tools The cornerstone on which these tools rest is the Traits package, which provides explicit type declarations in Python; its features include initialization, validation, delegation, notification, and visualization of typed attributes. More information is available for all the packages within ETS from the Enthought Tool Suite development home page at http://code.enthought.com/projects/tool-suite.php. Testimonials ---------------- "I set out to rebuild an application in one week that had been developed over the last seven years (in C by generations of post-docs). Pyface and Traits were my cornerstones and I knew nothing about Pyface or Wx. It has been a hectic week. But here ... sits in front of me a nice application that does most of what it should. I think this has been a huge success. ... Thanks to the tools Enthought built, and thanks to the friendly support from people on the [enthought-dev] list, I have been able to build what I think is the best application so far. I have built similar applications (controlling cameras for imaging Bose-Einstein condensate) in C+MFC, Matlab, and C+labWindows, each time it has taken me at least four times longer to get to a result I regard as inferior. So I just wanted to say a big "thank you". Thank you to Enthought for providing this great software open-source. Thank you for everybody on the list for your replies." ? Ga?l Varoquaux, Laboratoire Charles Fabry, Institut d?Optique, Palaiseau, France "I'm currently writing a realtime data acquisition/display application ? I'm using Enthought Tool Suite and Traits, and Chaco for display. IMHO, I think that in five years ETS/Traits will be the most comonly used framework for scientific applications." ? Gary Pajer, Department of Chemistry, Biochemistry and Physics, Rider University, Lawrenceville NJ From charlesr.harris at gmail.com Sat Aug 16 18:18:09 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 16 Aug 2008 16:18:09 -0600 Subject: [Numpy-discussion] long double woes on win32 In-Reply-To: References: <5b8d13220808151643v4834397frd64b31707afbaa78@mail.gmail.com> <48A6F75D.8020004@enthought.com> <5b8d13220808161010q4b8232b1lfe8856159e77f757@mail.gmail.com> <5b8d13220808161024q67d1484fi34390e00ccde6675@mail.gmail.com> <5b8d13220808161346m39d7302coa010dfc5c89ca8f@mail.gmail.com> Message-ID: On Sat, Aug 16, 2008 at 3:02 PM, Charles R Harris wrote: > > > On Sat, Aug 16, 2008 at 2:46 PM, David Cournapeau wrote: > >> On Sat, Aug 16, 2008 at 2:07 PM, Charles R Harris >> wrote: >> > >> > >> > There is seems to be a problem in defining the functions called for the >> > different types. >> >> I don't know enough about this part of the code to be sure about the >> whole function calls stack, but I would guess this is not surprising >> and even expected: you force long double to be double, but you still >> call the long double function if you do np.exp, so this cannot work. > > > Part of the problem is that the loops/functions called depend on the > typecode, i.e., dtype.char, which stays as 'g' even when the underlying type > is float64. We can't really fix that because the C-API still allows folks to > create arrays using the typecode and we would have to fudge things so the > right "wrong" descr was returned. Ugly, and a left over from numeric which > defined types to match the c types instead of the precision. Maybe we should > rewrite in FORTRAN ;) Anyway, I think the easiest solution might be to use > npy_func internally and add a -DNOLONGDOUBLE flag to override the values > returned by the distutils test code. Whether we should try such a big > modification before 1.2 is another question. > Another place this could be fixed for the ufuncs is in the ufunc code generator, i.e., in __umath_generated.c lines like exp_functions[2] = PyUFunc_g_g; exp_data[2] = (void *) expl; could have expl replaced by exp. But there are likely other problems that will need fixing. Too bad there isn't a flag in gcc to automatically compile long doubles as doubles. There *is* one to compile floats as doubles. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sat Aug 16 19:46:52 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 16 Aug 2008 18:46:52 -0500 Subject: [Numpy-discussion] long double woes on win32 In-Reply-To: References: <5b8d13220808151643v4834397frd64b31707afbaa78@mail.gmail.com> <5b8d13220808161010q4b8232b1lfe8856159e77f757@mail.gmail.com> <5b8d13220808161024q67d1484fi34390e00ccde6675@mail.gmail.com> <5b8d13220808161346m39d7302coa010dfc5c89ca8f@mail.gmail.com> Message-ID: <5b8d13220808161646i1ac3b7f2r5d18ae2aedea3820@mail.gmail.com> On Sat, Aug 16, 2008 at 5:18 PM, Charles R Harris wrote: > > could have expl replaced by exp. But there are likely other problems that > will need fixing. I think this is red-herring. Does it really make sense to force configuring long double as double if the C runtime and compiler do support long double as a different type than double ? The problem is only a concern on windows because of compilers bugs (or more exactly compiler/runtime mismatch): on windows, long double are double, and so long double functions are given 8 bytes double, which is why it should work. My only worry is related to ticket 891 (http://projects.scipy.org/scipy/numpy/ticket/891), which confused me around this long double is double thing. Anyway, that will be wait for 1.3. cheers, David From charlesr.harris at gmail.com Sat Aug 16 19:55:57 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 16 Aug 2008 17:55:57 -0600 Subject: [Numpy-discussion] long double woes on win32 In-Reply-To: <5b8d13220808161646i1ac3b7f2r5d18ae2aedea3820@mail.gmail.com> References: <5b8d13220808151643v4834397frd64b31707afbaa78@mail.gmail.com> <5b8d13220808161010q4b8232b1lfe8856159e77f757@mail.gmail.com> <5b8d13220808161024q67d1484fi34390e00ccde6675@mail.gmail.com> <5b8d13220808161346m39d7302coa010dfc5c89ca8f@mail.gmail.com> <5b8d13220808161646i1ac3b7f2r5d18ae2aedea3820@mail.gmail.com> Message-ID: On Sat, Aug 16, 2008 at 5:46 PM, David Cournapeau wrote: > On Sat, Aug 16, 2008 at 5:18 PM, Charles R Harris > wrote: > > > > > could have expl replaced by exp. But there are likely other problems that > > will need fixing. > > I think this is red-herring. Does it really make sense to force > configuring long double as double if the C runtime and compiler do > support long double as a different type than double ? The problem is > only a concern on windows because of compilers bugs (or more exactly > compiler/runtime mismatch): on windows, long double are double, and so > long double functions are given 8 bytes double, which is why it should > work. > Yeah, I think that is right. Although it would be nice if things were set up so a few simple defines would make everything work. There is really no reason it couldn't work, just that numpy wasn't designed that way. Anyway, where does mingw get its math.h include file? I suspect it might be a problem if it doesn't match the windows version because the testcode might not correctly detect the existence, or lack thereof, of various functions. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Aug 17 00:16:35 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 16 Aug 2008 22:16:35 -0600 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: <48A74080.2010002@esrf.fr> References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <3d375d730808160243i1bed66a8se9b17214f9e05bcb@mail.gmail.com> <48A74080.2010002@esrf.fr> Message-ID: On Sat, Aug 16, 2008 at 3:02 PM, Jon Wright wrote: > > > Try to look 12 months into the future and ask yourselves if it was > really a good idea to break the ABI. > I'm slowly coming to the conviction that there should be no C-ABI changes in 1.2. And maybe not in 1.3 either, instead we should work on fixing bugs and beating the code into shape. The selling point of 1.1.x is python 2.3 compatibility, the selling point of 1.2.x should be bug fixes and cleanups, including testing and documentation. ABI changes should be scheduled for 2.0. At some point down the road we can start thinking of Python 3.0, but I don't expect Python 3.0 adoption to be quick, it is a big step and a lot of projects will have to make the transition before it becomes compelling. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sun Aug 17 00:59:22 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 16 Aug 2008 23:59:22 -0500 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <3d375d730808160243i1bed66a8se9b17214f9e05bcb@mail.gmail.com> <48A74080.2010002@esrf.fr> Message-ID: <5b8d13220808162159h2bd417ffg61de6995a3a27f8d@mail.gmail.com> On Sat, Aug 16, 2008 at 11:16 PM, Charles R Harris wrote: > > I'm slowly coming to the conviction that there should be no C-ABI changes in > 1.2. It does not make sense to revert those changes anymore, but we keep having those discussions, and I still don't understand whether there is a consensus here. Breaking a bit or breaking a lot is the same: we broke API in 1.1, we are breaking in 1.2, will we break in 1.3 ? If we we will always break when we "need" to, at least I would like to be clear on that. I personally think it is wrong to break API and ABI. I don't care about the version where we break; I think the focus on the version is a bit misplaced, because in python (and numpy), the version is different from what is the norm in other big open source projects anyway (in any open source project I know, breaking between N.x and N.x+1 is a bug). The focus should be the period of time you can rely on a stable API (and ABI). Changing the API every three months sounds really wrong to me, whatever the version is. That's something that most if not all successful open source projects used by other ones do (gtk, qt, kde, etc...). I don't see what's so different in numpy so that we could afford doing something those other projects can't. cheers, David From fperez.net at gmail.com Sun Aug 17 01:03:57 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 16 Aug 2008 22:03:57 -0700 Subject: [Numpy-discussion] Possible new multiplication operators for Python Message-ID: Hi all, [ please keep all replies to this only on the numpy list. I'm cc'ing the scipy ones to make others aware of the topic, but do NOT reply on those lists so we can have an organized thread for future reference] In the Python-dev mailing lists, there were recently two threads regarding the possibility of adding to the language new multiplication operators (amongst others). This would allow one to define things like an element-wise and a matrix product for numpy arrays, for example: http://mail.python.org/pipermail/python-dev/2008-July/081508.html http://mail.python.org/pipermail/python-dev/2008-July/081551.html It turns out that there's an old pep on this issue: http://www.python.org/dev/peps/pep-0225/ which hasn't been ruled out, simply postponed. At this point it seems that there is room for some discussion, and obviously the input of the numpy/scipy crowd would be very welcome. I volunteered to host a BOF next week at scipy so we could collect feedback from those present, but it's important that those NOT present at the conference can equally voice their ideas/opinions. So I wanted to open this thread here to collect feedback. We'll then try to have the bof next week at the conference, and I'll summarize everything for python-dev. Obviously this doesn't mean that we'll get any changes in, but at least there's interest in discussing a topic that has been dear to everyone here. Cheers, f From cournape at gmail.com Sun Aug 17 01:21:19 2008 From: cournape at gmail.com (David Cournapeau) Date: Sun, 17 Aug 2008 00:21:19 -0500 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: <5b8d13220808162159h2bd417ffg61de6995a3a27f8d@mail.gmail.com> References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <3d375d730808160243i1bed66a8se9b17214f9e05bcb@mail.gmail.com> <48A74080.2010002@esrf.fr> <5b8d13220808162159h2bd417ffg61de6995a3a27f8d@mail.gmail.com> Message-ID: <5b8d13220808162221u2892ab7ah90d4aa9b14e0eb2d@mail.gmail.com> On Sat, Aug 16, 2008 at 11:59 PM, David Cournapeau wrote: > On Sat, Aug 16, 2008 at 11:16 PM, Charles R Harris > wrote: >> >> I'm slowly coming to the conviction that there should be no C-ABI changes in >> 1.2. > > It does not make sense to revert those changes anymore, Actually, I did not follow the discussion when this change happened, but it does not look difficult to change the code such as we do not break the ABI. Instead of replacing the flag, we can put it at the end, and deprecate (but not remove) the old one. Would anyone be strongly against that ? From charlesr.harris at gmail.com Sun Aug 17 01:22:35 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 16 Aug 2008 23:22:35 -0600 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: References: Message-ID: On Sat, Aug 16, 2008 at 11:03 PM, Fernando Perez wrote: > Hi all, > > [ please keep all replies to this only on the numpy list. I'm cc'ing > the scipy ones to make others aware of the topic, but do NOT reply on > those lists so we can have an organized thread for future reference] > > In the Python-dev mailing lists, there were recently two threads > regarding the possibility of adding to the language new multiplication > operators (amongst others). This would allow one to define things > like an element-wise and a matrix product for numpy arrays, for > example: > > http://mail.python.org/pipermail/python-dev/2008-July/081508.html > http://mail.python.org/pipermail/python-dev/2008-July/081551.html > > It turns out that there's an old pep on this issue: > > http://www.python.org/dev/peps/pep-0225/ > > which hasn't been ruled out, simply postponed. At this point it seems > that there is room for some discussion, and obviously the input of the > numpy/scipy crowd would be very welcome. I volunteered to host a BOF > next week at scipy so we could collect feedback from those present, > but it's important that those NOT present at the conference can > equally voice their ideas/opinions. > > So I wanted to open this thread here to collect feedback. We'll then > try to have the bof next week at the conference, and I'll summarize > everything for python-dev. Obviously this doesn't mean that we'll get > any changes in, but at least there's interest in discussing a topic > that has been dear to everyone here. > Geez, some of those folks in those threads are downright rude. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Sun Aug 17 01:43:51 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 16 Aug 2008 22:43:51 -0700 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: References: Message-ID: On Sat, Aug 16, 2008 at 10:22 PM, Charles R Harris wrote: > Geez, some of those folks in those threads are downright rude. Python-dev is nowhere nearly as civil as these lists, which I consider to be an asset of ours which we should always strive to protect. In this list even strong disagreement is mostly very well handled, and hopefully we'll keep this tradition as we grow. Python-dev may not be as bad as the infamous LKML, but it can certainly be unpleasant at times (this is one more reason to praise Travis O for all the work he's pushed over there, he certainly had to take quite a few gang-like beatings in the process). No worries though: I'll collect all the feedback and report back up there. Cheers, f From charlesr.harris at gmail.com Sun Aug 17 01:51:38 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 16 Aug 2008 23:51:38 -0600 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: <5b8d13220808162221u2892ab7ah90d4aa9b14e0eb2d@mail.gmail.com> References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <3d375d730808160243i1bed66a8se9b17214f9e05bcb@mail.gmail.com> <48A74080.2010002@esrf.fr> <5b8d13220808162159h2bd417ffg61de6995a3a27f8d@mail.gmail.com> <5b8d13220808162221u2892ab7ah90d4aa9b14e0eb2d@mail.gmail.com> Message-ID: On Sat, Aug 16, 2008 at 11:21 PM, David Cournapeau wrote: > On Sat, Aug 16, 2008 at 11:59 PM, David Cournapeau > wrote: > > On Sat, Aug 16, 2008 at 11:16 PM, Charles R Harris > > wrote: > >> > >> I'm slowly coming to the conviction that there should be no C-ABI > changes in > >> 1.2. > > > > It does not make sense to revert those changes anymore, > > Actually, I did not follow the discussion when this change happened, > but it does not look difficult to change the code such as we do not > break the ABI. Instead of replacing the flag, we can put it at the > end, and deprecate (but not remove) the old one. > > Would anyone be strongly against that ? I have nothing against extensions when they can be made to serve. If a dictionary gets added to ndarrays I hope it is done that way, likewise for generalized ufuncs. In the present case I think Travis wants to preserve the functionality while changing the name and type, and that doesn't really fit the extension model. But I might be wrong about that. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Sun Aug 17 04:09:35 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 17 Aug 2008 03:09:35 -0500 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <3d375d730808160243i1bed66a8se9b17214f9e05bcb@mail.gmail.com> <48A74080.2010002@esrf.fr> <5b8d13220808162159h2bd417ffg61de6995a3a27f8d@mail.gmail.com> <5b8d13220808162221u2892ab7ah90d4aa9b14e0eb2d@mail.gmail.com> Message-ID: <9457e7c80808170109s5c5e8131h4408901fa5c35404@mail.gmail.com> 2008/8/17 Charles R Harris : > I have nothing against extensions when they can be made to serve. If a > dictionary gets added to ndarrays I hope it is done that way, likewise for > generalized ufuncs. In the present case I think Travis wants to preserve the > functionality while changing the name and type, and that doesn't really fit > the extension model. But I might be wrong about that. I believe that is more or less accurate; Travis can confirm. With the new API version counter in place (it's not, btw -- still up for review), we should be able to keep the breakage to a minimum from here onwards. Unfortunately, I don't see an easy way to avoid it for the 1.2 release. Regards St?fan From cournape at gmail.com Sun Aug 17 07:25:07 2008 From: cournape at gmail.com (David Cournapeau) Date: Sun, 17 Aug 2008 06:25:07 -0500 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <3d375d730808160243i1bed66a8se9b17214f9e05bcb@mail.gmail.com> <48A74080.2010002@esrf.fr> <5b8d13220808162159h2bd417ffg61de6995a3a27f8d@mail.gmail.com> <5b8d13220808162221u2892ab7ah90d4aa9b14e0eb2d@mail.gmail.com> Message-ID: <5b8d13220808170425x413c6665p34bfe94fd8e47ced@mail.gmail.com> On Sun, Aug 17, 2008 at 12:51 AM, Charles R Harris wrote: > > > > I have nothing against extensions when they can be made to serve. If a > dictionary gets added to ndarrays I hope it is done that way, likewise for > generalized ufuncs. But that's a totally different matter. You can add functionalities and not breaking API or ABI. I mean gtk 2 exists for maybe 7 or 8 years now, and a program linked against gtk 2.0.0 can run with gtk 2.12.0, but gtk 2.12.0 certainly is different than 2.0.0. Having a policy about API/ABI stability does not mean we can't change anything. cheers, David From numpy-discussion at maubp.freeserve.co.uk Sun Aug 17 08:33:42 2008 From: numpy-discussion at maubp.freeserve.co.uk (Peter) Date: Sun, 17 Aug 2008 13:33:42 +0100 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: <48A71200.6050000@esrf.fr> References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <48A6F68D.3020006@enthought.com> <5b8d13220808161026t31368596ydf1b5bbf7b7e86a7@mail.gmail.com> <48A71200.6050000@esrf.fr> Message-ID: <320fb6e00808170533k572a1d49s630e6efabe4d9923@mail.gmail.com> > David Cournapeau wrote: >> Does that mean we will continue breaking the ABI from time to time >> during the 1.* cycle ? > Jon Wright wrote: > Can someone help me to understand me what is the compelling reason for > this change? If it only means everyone recompiles, it is hard to see > what we, as users, are gaining by doing that. As another NumPy user, and developer on another python project using Numeric/numpy at the C level, I had much the same thought. For our users who compile from source or use their Linux distribution's packages this isn't that big problem (the Linux packagers however may disagree!). However, to support Windows users, this means that in addition to providing one installation exe per version of python, we'd also need to provide separate versions for numpy 1.1 and 1.2. I am also not looking forward to the additional up front effort to reorganise our build system to cope with two versions of numpy. I imagine most 3rd party projects which use the NumPy at the C level will have similar views, and quite likely many of them are currently blissfully unaware of this impending API breakage, as I would have expect more comments. (I would have replied earlier but I had to track down which email address I was subscribed to this list with.) I really can't see why NumPy 1.2 is going to break the C-API compatibility for something that sounds so trivial as renaming the "hasobject" member of the PyArray_Descr structure to "flags". This doesn't even add any new functionality! I do understand that the API will have to evolve at some point, and St?fan van der Walt's version numbering scheme sounds like a sensible change to make when you do change the C-API. I'm just questioning the rational for breaking the C-API in NumPy 1.2 for what seems to me a trivial change. I don't know if this constitutes "major opposition", but is keeping the same C-API for NumPy 1.2 unchanged still a possibility? Please? Thank you, Peter From aisaac at american.edu Sun Aug 17 09:01:40 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 17 Aug 2008 09:01:40 -0400 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: References: Message-ID: <48A82134.6090501@american.edu> Aside from "more operators needed", is there a consensus view among the developers? Taking a user's perspective, I see a short run and a long run. SR: I am very comfortable with adding dot versions of operators. I am not worried about reversing the Matlab/GAUSS meanings, but if others are very worried, we could append the dot instead of prepending it. LR: It would be great to use unicode math operators. On this issue, Fortress is being foresightful. Accepting the "times" symbol would be a fairly small move for most users, since it is in the Latin 1 extension of ASCII. Cheers, Alan Isaac From fperez.net at gmail.com Sun Aug 17 14:10:11 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Sun, 17 Aug 2008 11:10:11 -0700 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: <48A82134.6090501@american.edu> References: <48A82134.6090501@american.edu> Message-ID: On Sun, Aug 17, 2008 at 6:01 AM, Alan G Isaac wrote: > Aside from "more operators needed", is there a consensus > view among the developers? I don't think so, but given that pep 225 exists and is fully fleshed out, I guess it should be considered the starting point of the discussion for reference. This doesn't mean that modifications to it can't be suggested, but that I'm assuming python-dev will want that as the reference point. For something as big as this, they would definitely want to work off a real pep. Having said that, I think all ideas are fair game at this point. I personally would like to see it happen, but if not I'd like to see a final pronouncement on the matter rather than seeing pep 225 deferred forever. > Taking a user's perspective, I see a short run and a long > run. > > SR: I am very comfortable with adding dot versions of operators. > I am not worried about reversing the Matlab/GAUSS meanings, > but if others are very worried, we could append the dot > instead of prepending it. > > LR: It would be great to use unicode math operators. > On this issue, Fortress is being foresightful. > Accepting the "times" symbol would be a fairly small move > for most users, since it is in the Latin 1 extension of > ASCII. I'll be sure to list this as part of the received feedback. I'm personally not too crazy about unicode operators (at least not to the extent that Fortress went, where basically a special IDE would be required to write the code in any reasonable scenario). But I'm willing to change my mind, and I'm *definitely* acting as scribe here, so everything that is presented will be reported back. As we get more info I'll start a summary document, which will then be completed with 'live' feedback from the session at scipy next week. Thanks! Cheers, f From nadavh at visionsense.com Sun Aug 17 14:12:24 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Sun, 17 Aug 2008 21:12:24 +0300 Subject: [Numpy-discussion] Possible new multiplication operators for Python References: <48A82134.6090501@american.edu> Message-ID: <710F2847B0018641891D9A216027636029C236@ex3.envision.co.il> But would it be not-trivial to enter times ans alike unicode symbols within "normal" text editors? Otherwise it is a compelling proposition at first glance. Nadav. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Alan G Isaac ????: ? 17-??????-08 16:01 ??: Discussion of Numerical Python ????: Re: [Numpy-discussion] Possible new multiplication operators for Python Aside from "more operators needed", is there a consensus view among the developers? Taking a user's perspective, I see a short run and a long run. SR: I am very comfortable with adding dot versions of operators. I am not worried about reversing the Matlab/GAUSS meanings, but if others are very worried, we could append the dot instead of prepending it. LR: It would be great to use unicode math operators. On this issue, Fortress is being foresightful. Accepting the "times" symbol would be a fairly small move for most users, since it is in the Latin 1 extension of ASCII. Cheers, Alan Isaac _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion From aisaac at american.edu Sun Aug 17 14:59:20 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 17 Aug 2008 14:59:20 -0400 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: <710F2847B0018641891D9A216027636029C236@ex3.envision.co.il> References: <48A82134.6090501@american.edu> <710F2847B0018641891D9A216027636029C236@ex3.envision.co.il> Message-ID: <48A87508.9060107@american.edu> Nadav Horesh wrote: > But would it be not-trivial to enter times ans alike > unicode symbols within "normal" text editors? Otherwise it > is a compelling proposition at first glance. First, what is a "normal" text editor? Handling utf-8 seems pretty common these days. http://en.wikipedia.org/wiki/Unicode_input There are two separable issues: entry and display. Entry of unicode operators would presumably involve digraphs. http://en.wikipedia.org/wiki/Digraph_(computing) Display seems largely addressed by the STIX fonts: http://www.stixfonts.org/project.html fwiw, Alan Isaac From dalke at dalkescientific.com Sun Aug 17 15:00:01 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Sun, 17 Aug 2008 21:00:01 +0200 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: References: <48A82134.6090501@american.edu> Message-ID: <7A5393FE-E9DE-44E1-BE31-22A22BBA07AF@dalkescientific.com> Fernando Perez wrote: > For something as big as this, they would > definitely want to work off a real pep. What might be interesting, for those who want to experiment with this syntax, is to take my Python parser for Python (python4ply - http://www.dalkescientific.com/Python/python4ply.html ) and add support for the proposed syntax. It's only a grammar change and it shouldn't be that hard to change the syntax tree so that a proposed "~+" or whatever gets converted to the right method call. Python4ply generates PVM byte code, so the end result is code that works with the existing Python. You could even be adventurous and map things like a \power b into binary operators, perhaps named "__op_power__" It's loads of fun to tweak the grammar. My tutorial even walks through how to add Perl-like syntax for regex creation and match operator, to allow for line in open("python_yacc.py"): if line =~ m/def (?P\w+) *(?P\(.*\)) *:/: print repr($1), repr($args) :) >> LR: It would be great to use unicode math operators. Though if you want to experiment with this .. I make neither guarantees nor warrantees about Unicode support in what I did. I don't even support the encoding hint that Python code can have. Andrew dalke at dalkescientific.com From gael.varoquaux at normalesup.org Sun Aug 17 15:14:35 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 17 Aug 2008 21:14:35 +0200 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: References: <48A82134.6090501@american.edu> Message-ID: <20080817191435.GB692@phare.normalesup.org> On Sun, Aug 17, 2008 at 11:10:11AM -0700, Fernando Perez wrote: > > LR: It would be great to use unicode math operators. > > On this issue, Fortress is being foresightful. > > Accepting the "times" symbol would be a fairly small move > > for most users, since it is in the Latin 1 extension of > > ASCII. > I'll be sure to list this as part of the received feedback. I'm > personally not too crazy about unicode operators (at least not to the > extent that Fortress went, where basically a special IDE would be > required to write the code in any reasonable scenario). But I'm > willing to change my mind, and I'm *definitely* acting as scribe here, > so everything that is presented will be reported back. I am very much against unicode operators. I can see a huge amount of problems this will generate, for little gain. There are still some possibilities to use a plain ascii character (I can think of '^', '/', '$', '!', althought each one of these might lead to confusion, and they have a feeling of perl). Could we also go for multiple cahracter operators? Anybody care for '.*'? Ga?l From dalke at dalkescientific.com Sun Aug 17 15:33:33 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Sun, 17 Aug 2008 21:33:33 +0200 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: <20080817191435.GB692@phare.normalesup.org> References: <48A82134.6090501@american.edu> <20080817191435.GB692@phare.normalesup.org> Message-ID: Ga?l Varoquaux wrote: > Anybody care for '.*'? That's border-line case, and probably on the bad idea side because 1.*2 already means something in normal Python. If added there would be a difference between 1.*2 and 1 .*2 This problem already exists. Consider >>> 1 .__str__() '1' >>> 1.__str__() File "", line 1 1.__str__() ^ SyntaxError: invalid syntax >>> 1..__str__() '1.0' >>> but it doesn't come up much because it's extremely rare for anyone to call numerical methods directly. Whereas people already write things like "2*x" so writing "2.*x" to mean piecewise but actually get object-wise seems likely. This is why I think being able to write experimental support for new syntax, and gain experience about it's advantages and disadvantages, could be useful and could provide additional sway for python-dev folks. Andrew dalke at dalkescientific.com From gael.varoquaux at normalesup.org Sun Aug 17 15:35:21 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 17 Aug 2008 21:35:21 +0200 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: References: <48A82134.6090501@american.edu> <20080817191435.GB692@phare.normalesup.org> Message-ID: <20080817193521.GD692@phare.normalesup.org> On Sun, Aug 17, 2008 at 09:33:33PM +0200, Andrew Dalke wrote: > Ga?l Varoquaux wrote: > > Anybody care for '.*'? > That's border-line case, and probably on the bad > idea side because 1.*2 already means something in > normal Python. If added there would be a difference > between > 1.*2 > and > 1 .*2 Good point. This was more of a joke than a serious proposition. The joke is that in Matlab, '.*' is a different operator than '*'. Ga?l From fperez.net at gmail.com Sun Aug 17 15:35:23 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Sun, 17 Aug 2008 12:35:23 -0700 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: <20080817191435.GB692@phare.normalesup.org> References: <48A82134.6090501@american.edu> <20080817191435.GB692@phare.normalesup.org> Message-ID: On Sun, Aug 17, 2008 at 12:14 PM, Gael Varoquaux wrote: > On Sun, Aug 17, 2008 at 11:10:11AM -0700, Fernando Perez wrote: >> > LR: It would be great to use unicode math operators. >> > On this issue, Fortress is being foresightful. >> > Accepting the "times" symbol would be a fairly small move >> > for most users, since it is in the Latin 1 extension of >> > ASCII. > >> I'll be sure to list this as part of the received feedback. I'm >> personally not too crazy about unicode operators (at least not to the >> extent that Fortress went, where basically a special IDE would be >> required to write the code in any reasonable scenario). But I'm >> willing to change my mind, and I'm *definitely* acting as scribe here, >> so everything that is presented will be reported back. > > I am very much against unicode operators. I can see a huge amount of > problems this will generate, for little gain. There are still some > possibilities to use a plain ascii character (I can think of '^', '/', > '$', '!', althought each one of these might lead to confusion, and they > have a feeling of perl). Could we also go for multiple cahracter > operators? Anybody care for '.*'? Please read the pep first: the proposal is already for multichar operators, and the various alternatives are discussed in detail there. The .OP form is specifically addressed in the pep, so if you want to discuss that one, you need to cover the objections to it already raised in the pep. As I said before, this discussion shouldn't start from a blank slate: consider the pep 225 http://www.python.org/dev/peps/pep-0225/ as the starting point that python-dev will want for any useful discussion. Cheers, f From charlesr.harris at gmail.com Sun Aug 17 15:38:13 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 17 Aug 2008 13:38:13 -0600 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: <48A82134.6090501@american.edu> References: <48A82134.6090501@american.edu> Message-ID: On Sun, Aug 17, 2008 at 7:01 AM, Alan G Isaac wrote: > Aside from "more operators needed", is there a consensus > view among the developers? > > Taking a user's perspective, I see a short run and a long > run. > > SR: I am very comfortable with adding dot versions of operators. > I am not worried about reversing the Matlab/GAUSS meanings, > but if others are very worried, we could append the dot > instead of prepending it. > > LR: It would be great to use unicode math operators. > On this issue, Fortress is being foresightful. > Accepting the "times" symbol would be a fairly small move > for most users, since it is in the Latin 1 extension of > ASCII. > I kinda like the unicode idea because of the current dearth of usable symbols. In fact, my system is already using utf-8. $[charris at f9 ~]$ locale LANG=en_US.utf8 LC_CTYPE="en_US.utf8" LC_NUMERIC="en_US.utf8" LC_TIME="en_US.utf8" LC_COLLATE="en_US.utf8" LC_MONETARY="en_US.utf8" LC_MESSAGES="en_US.utf8" LC_PAPER="en_US.utf8" LC_NAME="en_US.utf8" LC_ADDRESS="en_US.utf8" LC_TELEPHONE="en_US.utf8" LC_MEASUREMENT="en_US.utf8" LC_IDENTIFICATION="en_US.utf8" LC_ALL= And keymaps are not that hard to do in most editors. However... there are various unicode encodings and locales, often indicated by the first few bytes in a file. It's not clear to me that these encodings are universally implemented at this time and I would be loath to depend on them without some testing. Even so, it might be good to reserve a few symbols for future use as operators. The resistance to adding these operators might be less among Python developers because they are less visible, not multicharater, and won't conflict with current python usage. So at the least I think we should try to get some unicode symbols set aside and maybe several years from now we can start using them. Here is an interesting little article on unicode for unix. Here are some links to various symbols. And here is a bit of unicode just so we can see how it looks for various folks. A = B?C Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalke at dalkescientific.com Sun Aug 17 15:58:29 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Sun, 17 Aug 2008 21:58:29 +0200 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: References: <48A82134.6090501@american.edu> Message-ID: On Aug 17, 2008, at 9:38 PM, Charles R Harris wrote: > And here is a bit of unicode just so we can see how it looks for > various folks. > > A = B?C Or write B \circledast C ? (Or \oast?) Try using Google to search for that character. Andrew dalke at dalkescientific.com From charlesr.harris at gmail.com Sun Aug 17 16:28:36 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 17 Aug 2008 14:28:36 -0600 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: References: <48A82134.6090501@american.edu> Message-ID: On Sun, Aug 17, 2008 at 1:58 PM, Andrew Dalke wrote: > On Aug 17, 2008, at 9:38 PM, Charles R Harris wrote: > > > And here is a bit of unicode just so we can see how it looks for > > various folks. > > > > A = B?C > > > Or write B \circledast C ? (Or \oast?) Try using Google to search > for that character. > That's what I sent, it's in utf-8. I'm thinking the circleddot looks better, but here a few others A = B?C #circled asterisk A = B?C #circled dot A = B?C #circled star A = B?C #circled times (tensor) A = B?C #dot A = B?C #star Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Sun Aug 17 16:28:55 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 17 Aug 2008 16:28:55 -0400 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: <20080817191435.GB692@phare.normalesup.org> References: <48A82134.6090501@american.edu> <20080817191435.GB692@phare.normalesup.org> Message-ID: <48A88A07.4050507@american.edu> Gael Varoquaux wrote: > I am very much against unicode operators. I can see a huge > amount of problems this will generate, for little gain. I actually basically like PEP 225, although I find @*, @+, etc more readable, and to provide the right visual emphasis. (Rather than ~*, ~+, etc.) Additionally, I do not think Unicode addresses the problem central to the PEP, as there is as far as I know no standard symbolic distinction between, say, elementwise multiplication and matrix multiplication. So I think the PEP is basically needed (perhaps with @ instead of ~). That said, what kind of problems do you have in mind? Cheers, Alan Isaac PS Here are the core unicode math operators: http://www.alanwood.net/unicode/mathematical_operators.html or if you do not have appropriate fonts you can try http://www.unicode.org/charts/PDF/U2200.pdf From lists at cheimes.de Sun Aug 17 16:35:26 2008 From: lists at cheimes.de (Christian Heimes) Date: Sun, 17 Aug 2008 22:35:26 +0200 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: References: <48A82134.6090501@american.edu> Message-ID: Andrew Dalke wrote: > Or write B \circledast C ? (Or \oast?) Try using Google to search > for that character. >>> unicodedata.lookup('CIRCLED ASTERISK OPERATOR') '?' From ondrej at certik.cz Sun Aug 17 18:00:51 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 18 Aug 2008 00:00:51 +0200 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: References: Message-ID: <85b5c3130808171500i731c8a01v6b11a67e4c3e413f@mail.gmail.com> On Sun, Aug 17, 2008 at 7:03 AM, Fernando Perez wrote: > Hi all, > > [ please keep all replies to this only on the numpy list. I'm cc'ing > the scipy ones to make others aware of the topic, but do NOT reply on > those lists so we can have an organized thread for future reference] > > In the Python-dev mailing lists, there were recently two threads > regarding the possibility of adding to the language new multiplication > operators (amongst others). This would allow one to define things > like an element-wise and a matrix product for numpy arrays, for > example: > > http://mail.python.org/pipermail/python-dev/2008-July/081508.html > http://mail.python.org/pipermail/python-dev/2008-July/081551.html > > It turns out that there's an old pep on this issue: > > http://www.python.org/dev/peps/pep-0225/ > > which hasn't been ruled out, simply postponed. At this point it seems > that there is room for some discussion, and obviously the input of the > numpy/scipy crowd would be very welcome. I volunteered to host a BOF > next week at scipy so we could collect feedback from those present, > but it's important that those NOT present at the conference can > equally voice their ideas/opinions. > > So I wanted to open this thread here to collect feedback. We'll then > try to have the bof next week at the conference, and I'll summarize > everything for python-dev. Obviously this doesn't mean that we'll get > any changes in, but at least there's interest in discussing a topic > that has been dear to everyone here. I like that Python is so easy to learn, so I hope no more operators and definitely not unicode operators will be added. And if, then only if they are really needed, which I think they are not. What I like on Python is that I can remember all it's syntax in my memory. The more operators, the more difficult this will be. There is some inconsistency though, for example one can override A() + A(), but one cannot override 1 + 1. This could (should) be fixed somehow. But apart from that, I very much like Python as it is now and I hope it will not become a bloated language. Ondrej From robert.kern at gmail.com Sun Aug 17 18:59:15 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 17 Aug 2008 17:59:15 -0500 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: <85b5c3130808171500i731c8a01v6b11a67e4c3e413f@mail.gmail.com> References: <85b5c3130808171500i731c8a01v6b11a67e4c3e413f@mail.gmail.com> Message-ID: <3d375d730808171559y5747dd44ped72f8ca42c739b7@mail.gmail.com> On Sun, Aug 17, 2008 at 17:00, Ondrej Certik wrote: > There is some inconsistency though, for example one can override A() + > A(), but one cannot override 1 + 1. This could (should) be fixed > somehow. This is getting off-topic, but I really hope that never changes. The difference between A.__add__ and int.__add__ is that you are defining A. Python defines int. If you mess with fundamental types like that, you will break other libraries and must confine yourself to just code that you've written. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dalke at dalkescientific.com Sun Aug 17 19:19:31 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Mon, 18 Aug 2008 01:19:31 +0200 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: References: <48A82134.6090501@american.edu> Message-ID: > On Aug 17, 2008, at 10:35 PM, Christian Heimes wrote: >> Andrew Dalke wrote: >>> Or write B \circledast C ? (Or \oast?) Try using Google to search >>> for that character. >> >>>>> unicodedata.lookup('CIRCLED ASTERISK OPERATOR') >> '?' >> I mean, go to Google and search for "?". It finds no hits. I didn't even know the name for that character. I had to copy and paste it into a terminal window that didn't understand the encoding, so it gave me "\342\212\233". I then used Google and found a LaTeX page listing octal Unicode escape characters for various LaTeX definitions, those being "\circledast" which has "\oast" as an alias. Those two terms I can search for. Not some sort of unknown-to-me unicode character. Andrew dalke at dalkescientific.com From dalke at dalkescientific.com Sun Aug 17 19:50:36 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Mon, 18 Aug 2008 01:50:36 +0200 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: <85b5c3130808171500i731c8a01v6b11a67e4c3e413f@mail.gmail.com> References: <85b5c3130808171500i731c8a01v6b11a67e4c3e413f@mail.gmail.com> Message-ID: On Aug 18, 2008, at 12:00 AM, Ondrej Certik wrote: > There is some inconsistency though, for example one can override A() + > A(), but one cannot override 1 + 1. This could (should) be fixed > somehow. That will never, ever change in Python. There's no benefit to being able to redefine int.__add__ and doing so will break entirely too many assumptions. Here's one assumption - the C implementation does some simple constant folding: >>> def f(): ... print 1+1 ... >>> import dis >>> dis.dis(f) 2 0 LOAD_CONST 2 (2) 3 PRINT_ITEM 4 PRINT_NEWLINE 5 LOAD_CONST 0 (None) 8 RETURN_VALUE >>> With what you want that's not possible. Just think of the implementation difficulty. Are changes on the per-module or per-scope or per-thread level? And performance would be lousy (or require deep inferencing analysis) because every operation in C would need to go through the Python API just in case some fundamental definition like this was changed. Such a language is possible. I wouldn't be surprised if you could do it in Smalltalk and mostly do it in Ruby. But the general belief is that good design follows the "open/closed principle": "software entities (classes, modules, functions, etc.) should be open for extension, but closed for modification" - Bertrand Meyer, quoted by http://en.wikipedia.org/wiki/Open/closed_principle In Python, all types are closed for modification, and while classes are open for modification it's highly frowned upon to do so. The name "monkey-patch" sounds somewhat derisive for a reason. Andrew dalke at dalkescientific.com From charlesr.harris at gmail.com Sun Aug 17 19:54:11 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 17 Aug 2008 17:54:11 -0600 Subject: [Numpy-discussion] Problem with the mailing list? Message-ID: Hi All, I received an email from Hans Andreas -- the gen-ufuncs guy -- and he is unable to post to the list even though subscribed. Anyone know what might be the problem? Please cc Hans-Andreas.Engel at deshaw.com if you reply to this post. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Hans-Andreas.Engel at deshaw.com Sun Aug 17 20:13:51 2008 From: Hans-Andreas.Engel at deshaw.com (Engel, Hans-Andreas) Date: Sun, 17 Aug 2008 20:13:51 -0400 Subject: [Numpy-discussion] Generalized ufuncs? Message-ID: I am sorry that our submission http://projects.scipy.org/scipy/numpy/ticket/887 has created some annoyance; presumably we have taken the "Make contributions (e.g. code patches), (...) by submitting a 'ticket' on the Trac pages linked below" on http://scipy.org/Developer_Zone somewhat too literally. It was great to receive very positive responses to our patch and to receive a very timely review; this is encouraging for submitting code to the numpy repository. I would like to add that the proposed change is not that arbitrary; it is a well-established concept -- it is outlined on the GeneralLoopingFunctions wiki page, and it is a prominent concept of perl's PDL vector library. Of course, there is still room for arguing about details. The fact that no explicit "generalized" ufuncs are provided, should in my opinion not be an argument why not to include the change in the 1.2.0 release. Writing extension libraries that implement such generalized ufuncs, while being able to use the standard numpy distribution, would certainly be very valuable. Furthermore, the risk for including the proposed patch in 1.2.0 is very low: existing functionality is not touched. (Except the glitch we had by declaring variables in a gcc-way.) For standard ufuncs, it should be straightforward to see that the patch does not modify the behavior at all. I wish everyone a great SciPy'08 conference and very much hope that you can include the proposed functionality in the forthcoming NumPy release. Best regards, Hansres From Hans-Andreas.Engel at deshaw.com Sun Aug 17 20:16:45 2008 From: Hans-Andreas.Engel at deshaw.com (Engel, Hans-Andreas) Date: Sun, 17 Aug 2008 20:16:45 -0400 Subject: [Numpy-discussion] Problem with the mailing list? Message-ID: "Charles R Harris" wrote in message news:... > Hi All, > > I received an email from Hans Andreas -- the gen-ufuncs guy -- and he is > unable to post to the list even though subscribed. Anyone know what might be > the problem? Please cc Hans-Andreas.Engel at deshaw.com if you reply to this > post. > > Chuck > Thanks Chuck for following up on this -- it works now! (Initially I signed up a different address than the one I used for posting.) From Hans-Andreas.Engel at deshaw.com Sun Aug 17 20:38:37 2008 From: Hans-Andreas.Engel at deshaw.com (Engel, Hans-Andreas) Date: Sun, 17 Aug 2008 20:38:37 -0400 Subject: [Numpy-discussion] Generalised ufuncs branch Message-ID: "St?fan van der Walt" wrote in message news:<9457e7c80808151645v457e155apa192bdbcf3f91d47 at mail.gmail.com>... Hi all, I have moved the generalised ufuncs functionality off to a branch: http://svn.scipy.org/svn/numpy/branches/gen_ufuncs Please try it out and give us your feedback. We shall also pound on it at the sprint during SciPy'08, and thereafter decide how and when to best integrate it into NumPy. Thanks to Wenjie Fu and Hans-Andreas Engel for taking the time to think this issue through and to submit such a high-quality patch. Regards St?fan ---------- Motivated by Stefan's and Chuck's off-line comments, I have attached an example implementation of a generalized ufunc: /* * This implements the function out = inner1d(in1, in2) with * out[K] = sum_i { in1[K, i] * in2[K, i] } * and multi-index K, as described on * http://scipy.org/scipy/numpy/wiki/GeneralLoopingFunctions * and on http://projects.scipy.org/scipy/numpy/ticket/887. */ static void DOUBLE_inner1d(char **args, intp *dimensions, intp *steps, void *func) { /* Standard ufunc loop length and strides. */ intp dn = dimensions[0]; intp s0 = steps[0]; intp s1 = steps[1]; intp s2 = steps[2]; intp n; /* Additional loop length and strides for dimension "i" in elementary function. */ intp di = dimensions[1]; intp i_s1 = steps[3]; intp i_s2 = steps[4]; intp i; /* Outer loop: equivalent to standard ufuncs */ for (n = 0; n < dn; n++, args[0] += s0, args[1] += s1, args[2] += s2) { char *ip1 = args[0], *ip2 = args[1], *op = args[2]; /* Implement elementary function: out = sum_i { in1[i] * in2[i] } */ npy_double sum = 0; for (i = 0; i < di; i++) { sum += (*(npy_double *)ip1) * (*(npy_double *)ip2); ip1 += i_s1; ip2 += i_s2; } *(npy_double *)op = sum; } } /* Actually create the ufunc object */ static PyUFuncGenericFunction inner1d_functions[] = { DOUBLE_inner1d }; static void *inner1d_data[] = { (void *)NULL }; static char inner1d_sigs[] = { PyArray_DOUBLE, PyArray_DOUBLE, PyArray_DOUBLE }; static void addInner1d(PyObject *dictionary) { PyObject *f = PyUFunc_FromFuncAndDataAndSignature(inner1d_functions, inner1d_data, inner1d_sigs, 1, 2, 1, PyUFunc_None, "inner1d", "inner on the last dimension and broadcast on the other dimensions", 0, "(i),(i)->()"); PyDict_SetItemString(dictionary, "inner1d", f); } From robert.kern at gmail.com Sun Aug 17 20:43:58 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 17 Aug 2008 19:43:58 -0500 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: References: Message-ID: <3d375d730808171743j166e4d30vb01417e8d901c2ab@mail.gmail.com> On Sun, Aug 17, 2008 at 19:13, Engel, Hans-Andreas wrote: > I am sorry that our submission > http://projects.scipy.org/scipy/numpy/ticket/887 has created some > annoyance; presumably we have taken the "Make contributions (e.g. code > patches), (...) by submitting a 'ticket' on the Trac pages linked below" > on http://scipy.org/Developer_Zone somewhat too literally. Not at all. You did everything right. Unfortunately, it comes at a slightly awkward time. We are just about to make a release, and this is a substantial feature that some of us developers think needs a little more consideration than we can give it if we were to put it into this release. It's only the timing that triggered the conflict, not anything you really had control over. > It was great to receive very positive responses to our patch and to > receive a very timely review; this is encouraging for submitting code to > the numpy repository. > > I would like to add that the proposed change is not that arbitrary; it > is a well-established concept -- it is outlined on the > GeneralLoopingFunctions wiki page, and it is a prominent concept of > perl's PDL vector library. Of course, there is still room for arguing > about details. > > The fact that no explicit "generalized" ufuncs are provided, should in > my opinion not be an argument why not to include the change in the 1.2.0 > release. Writing extension libraries that implement such generalized > ufuncs, while being able to use the standard numpy distribution, would > certainly be very valuable. > > Furthermore, the risk for including the proposed patch in 1.2.0 is very > low: existing functionality is not touched. (Except the glitch we had > by declaring variables in a gcc-way.) For standard ufuncs, it should be > straightforward to see that the patch does not modify the behavior at > all. My concern is not that the patch might be buggy or anything like that. Rather, this is a substantial feature, and one that affects the C API. It will be very difficult to remove it or modify it later if we find that it needs even a slight tweak to its design. Despite its use in other array languages, it's new to us, and I think we need to explore its use cases a little. numpy isn't PDL, and the way this feature integrates with the rest of numpy's semantics and idioms is important to figure out. It's possible that there are problems that it could solve with just a slight modification to the design. I suggested that we move it to a branch for the time being so we can play with it and come up with examples of its use. If you have examples that you have already written, I would love to see them. I, for one, am amenable to seeing this in 1.2.0, but only if we push back the release by at least a few weeks. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dpeterson at enthought.com Sun Aug 17 21:05:08 2008 From: dpeterson at enthought.com (Dave Peterson) Date: Sun, 17 Aug 2008 20:05:08 -0500 Subject: [Numpy-discussion] [ANNOUNCE] EPD with Py2.5 v4.0.3001 Beta1 now available Message-ID: <48A8CAC4.7070409@enthought.com> Hello, Thanks to heroic efforts by Chris Galvan this weekend, and significant efforts by the team that finalized ETS 3.0.0 this week, we've been able to publish public beta releases of EPD with Py2.5 v4.0.30001 Beta1 for Windows and Mac OS X today. I've uploaded them to the downloads website and updated the EPD product pages to provide download links for the public. You can find the link to the betas here: http://www.enthought.com/products/epddownload.php Please give them a try and report any bugs to the EPD Trac site at https://svn.enthought.com/epd. In this release, EPD has been updated to include ETS 3.0.0, NumPy 1.1.1, IPython 0.9.beta, Matplotlib 0.98.1, Sphinx 0.4.2, pyhdf 0.8, VTK 5.0.4, wxPython 2.8.8.1, and many more updated projects. There are a few issues known at this time, but remember these are our first beta release of this version: * The included documentation hasn't been updated to the current versions of the third-party libraries. * Some of the product branding is not up-to-date with regard to the product name change to "EPD with Py2.5", nor with the version number of 4.0.30001 Beta 1. -- Dave From charlesr.harris at gmail.com Sun Aug 17 21:15:36 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 17 Aug 2008 19:15:36 -0600 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: References: Message-ID: On Sun, Aug 17, 2008 at 6:13 PM, Engel, Hans-Andreas < Hans-Andreas.Engel at deshaw.com> wrote: > I am sorry that our submission > http://projects.scipy.org/scipy/numpy/ticket/887 has created some > annoyance; presumably we have taken the "Make contributions (e.g. code > patches), (...) by submitting a 'ticket' on the Trac pages linked below" > on http://scipy.org/Developer_Zone somewhat too literally. > > It was great to receive very positive responses to our patch and to > receive a very timely review; this is encouraging for submitting code to > the numpy repository. > > I would like to add that the proposed change is not that arbitrary; it > is a well-established concept -- it is outlined on the > GeneralLoopingFunctions wiki page, and it is a prominent concept of > perl's PDL vector library. Of course, there is still room for arguing > about details. > > The fact that no explicit "generalized" ufuncs are provided, should in > my opinion not be an argument why not to include the change in the 1.2.0 > release. Writing extension libraries that implement such generalized > ufuncs, while being able to use the standard numpy distribution, would > certainly be very valuable. > > Furthermore, the risk for including the proposed patch in 1.2.0 is very > low: existing functionality is not touched. (Except the glitch we had > by declaring variables in a gcc-way.) For standard ufuncs, it should be > straightforward to see that the patch does not modify the behavior at > all. > > I wish everyone a great SciPy'08 conference and very much hope that you > can include the proposed functionality in the forthcoming NumPy release. > For those interested in the description posted with ticket #887, here it is. The patch can be found with the ticket. I note that PyUFunc_FromFuncAndDataAndSignature is not put at the end of the ufunc_api_order.txt list and might break the current API, which has entries like #define PyUFunc_Type (*(PyTypeObject *)PyUFunc_API[0]) So we need to keep the PyUFunc_API entries in order. Chuck ============================================================================================ Generalized Universal Functions ? There is a general need for looping over not only functions on scalars but also over functions on vectors (or arrays), as explained on http://scipy.org/scipy/numpy/wiki/GeneralLoopingFunctions. We propose to realize this concept by generalizing the universal functions (ufuncs), and provide a C implementation that adds ~500 lines to the numpy code base. In current (specialized) ufuncs, the elementary function is limited to element-by-element operations, whereas the generalized version supports "sub-array" by "sub-array" operations. The Perl vector library PDL provides a similar functionality and its terms are re-used in the following. Each generalized ufunc has information associated with it that states what the "core" dimensionality of the inputs is, as well as the corresponding dimensionality of the outputs (the element-wise ufuncs have zero core dimensions). The list of the core dimensions for all arguments is called the "signature" of a ufunc. For example, the ufunc numpy.add has signature "(),()->()" defining two scalar inputs and one scalar output. Another example is (see the GeneralLoopingFunctionspage) the function inner1d(a,b) with a signature of "(i),(i)->()". This applies the inner product along the last axis of each input, but keeps the remaining indices intact. For example, where a is of shape (3,5,N) and b is of shape (5,N), this will return an output of shape (3,5). The underlying elementary function is called 3*5 times. In the signature, we specify one core dimension "(i)" for each input and zero core dimensions "()" for the output, since it takes two 1-d arrays and returns a scalar. By using the same name "i", we specify that the two corresponding dimensions should be of the same size (or one of them is of size 1 and will be broadcasted). The dimensions beyond the core dimensions are called "loop" dimensions. In the above example, this corresponds to (3,5). The usual numpy "broadcasting" rules apply, where the signature determines how the dimensions of each input/output object are split into core and loop dimensions: 1. While an input array has a smaller dimensionality than the corresponding number of core dimensions, 1's are pre-pended to its shape. 2. The core dimensions are removed from all inputs and the remaining dimensions are broadcasted; defining the loop dimensions. 3. The output is given by the loop dimensions plus the output core dimensions. Definitions ? *Elementary Function:* Each ufunc consists of an elementary function that performs the most basic operation on the smallest portion of array arguments (e.g. adding two numbers is the most basic operation in adding two arrays). The ufunc applies the elementary function multiple times on different parts of the arrays. The input/output of elementary functions can be vectors; e.g., the elementary function of inner1d takes two vectors as input. *Signature:* A signature is a string describing the input/output dimensions of the elementary function of a ufunc. See section below for more details. *Core Dimension:* The dimensionality of each input/output of an elementary function is defined by its core dimensions (zero core dimensions correspond to a scalar input/output). The core dimensions are mapped to the last dimensions of the input/output arrays. *Dimension Name:* A dimension name represents a core dimension in the signature. Different dimensions may share a name, indicating that they are of the same size (or are broadcastable). *Dimension Index:* A dimension index is an integer representing a dimension name. It enumerates the dimension names according to the order of the first occurrence of each name in the signature. Details of Signature ? The signature defines "core" dimensionality of input and output variables, and thereby also defines the contraction of the dimensions. The signature is represented by a string of the following format: - Core dimensions of each input or output array are represented by a list of dimension names in parentheses, "(i_1,...,i_N)"; a scalar input/output is denoted by "()". Instead of "i_1", "i_2", etc, one can use any valid Python variable name. - Dimension lists for different arguments are separated by ",". Input/output arguments are separated by "->". - If one uses the same dimension name in multiple locations, this enforces the same size (or broadcastable size) of the corresponding dimensions. The formal syntax of signatures is as follows. ::= "->" ::= ::= ::= nil | | "," ::= "(" ")" ::= nil | | "," ::= valid Python variable name Notes: 1. All quotes are for clarity. 2. Core dimensions that share the same name must be broadcastable, as the two "i"s in our example above. Each dimension name typically corresponding to one level of looping in the elementary function's implementation. 3. White spaces are ignored. Here are some examples of signatures. add "(),()->()" inner1d "(i),(i)->()" sum1d "(i)->()" dot2d "(m,n),(n,p)->(m,p)" (matrix multiplication) outer_inner "(i,t),(j,t)->(i,j)" (inner over the last dimension, outer over the second to last, and loop/broadcast over the rest.) C-API for implementing Elementary Functions ? The current interface remains unchanged, and PyUFunc_FromFuncAndData can still be used to implement (specialized) ufuncs, consisting of scalar elementary functions. One can use PyUFunc_FromFuncAndDataAndSignature to declare a more general ufunc. The argument list is the same as PyUFunc_FromFuncAndData, with an additional argument specifying the signature as C string. Furthermore, the callback function is of the same type as before, void (*foo)(char **args, intp *dimensions, intp *steps, void *func). When invoked, args is a list of length nargs containing the data of all input/output arguments. For a scalar elementary function, steps is also of length nargs, denoting the strides used for the arguments. dimensions is a pointer to a single integer defining the size of the axis to be looped over. For a non-trivial signature, dimensions will also contain the sizes of the core dimensions as well, starting at the second entry. Only one size is provided for each unique dimension name and the sizes are given according to the first occurrence of a dimension name in the signature. The first nargs elements of steps remain the same as for scalar ufuncs. The following elements contain the strides of all core dimensions for all arguments in order. For example, consider a ufunc with signature "(i,j),(i)->()". In this case, args will contain three pointers to the data of the input/output arrays a, b, c. Furthermore, dimensions will be [N, I, J] to define the size of N of the loop and the sizes I and J for the core dimensions i and j. Finally, stepswill be [a_N, b_N, c_N, a_i, a_j, b_i], containing all necessary strides. License ? The attached code and the above documentation was written by Wenjie Fu < fuw at deshaw.com> and Hans-Andreas Engel , and is provided under the numpy license: -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Sun Aug 17 21:56:41 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 17 Aug 2008 20:56:41 -0500 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: <3d375d730808171743j166e4d30vb01417e8d901c2ab@mail.gmail.com> References: <3d375d730808171743j166e4d30vb01417e8d901c2ab@mail.gmail.com> Message-ID: <9457e7c80808171856m50a71aabtb6c7421d57b7334@mail.gmail.com> 2008/8/17 Robert Kern : > I suggested that we move it to a branch for the time being so we can > play with it and come up with examples of its use. That branch is here: http://stefan at svn.scipy.org/svn/numpy/branches/gen_ufuncs St?fan From charlesr.harris at gmail.com Sun Aug 17 22:29:22 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 17 Aug 2008 20:29:22 -0600 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: <9457e7c80808171856m50a71aabtb6c7421d57b7334@mail.gmail.com> References: <3d375d730808171743j166e4d30vb01417e8d901c2ab@mail.gmail.com> <9457e7c80808171856m50a71aabtb6c7421d57b7334@mail.gmail.com> Message-ID: On Sun, Aug 17, 2008 at 7:56 PM, St?fan van der Walt wrote: > 2008/8/17 Robert Kern : > > I suggested that we move it to a branch for the time being so we can > > play with it and come up with examples of its use. > > That branch is here: > > http://stefan at svn.scipy.org/svn/numpy/branches/gen_ufuncs > > For an earlier thread about using vector valued ufuncs for sorts and such -- and negative reactions to the suggestion -- go here. One of the major objections was how to call such functions with the ufunc machinery and the needed machinery for type promotions, sub classes, and all that nonsense. Are these dealt with in the patch? The current numpy code for all that is a bit of a mess anyway, and it would be nice to figure out some unified interface to call through and clean up the current code in the process. In fact, I've been making some preliminary efforts in that direction by formatting the current code and working through it. Also, do we also want reduce methods and such? I suspect there is still a lot of work to do to get this whole thing up and running. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Sun Aug 17 22:55:22 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sun, 17 Aug 2008 22:55:22 -0400 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: <3d375d730808171743j166e4d30vb01417e8d901c2ab@mail.gmail.com> References: <3d375d730808171743j166e4d30vb01417e8d901c2ab@mail.gmail.com> Message-ID: 2008/8/17 Robert Kern : > > I suggested that we move it to a branch for the time being so we can > play with it and come up with examples of its use. If you have > examples that you have already written, I would love to see them. I, > for one, am amenable to seeing this in 1.2.0, but only if we push back > the release by at least a few weeks. This is not a worked example, but this is exactly what is needed to make possible the "arrays of matrices" functions that were discussed some time ago. For example, users frequently want to do something like multiply an array of matrices by an array of matrices; that is not currently feasible without a very large temporary array (or a loop). But I think you were looking for examples of code using the interface, to see whether it worked well. Anne From robert.kern at gmail.com Sun Aug 17 23:15:54 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 17 Aug 2008 22:15:54 -0500 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: References: <3d375d730808171743j166e4d30vb01417e8d901c2ab@mail.gmail.com> Message-ID: <3d375d730808172015t62739266hbee205b97bdc86c3@mail.gmail.com> On Sun, Aug 17, 2008 at 21:55, Anne Archibald wrote: > 2008/8/17 Robert Kern : >> >> I suggested that we move it to a branch for the time being so we can >> play with it and come up with examples of its use. If you have >> examples that you have already written, I would love to see them. I, >> for one, am amenable to seeing this in 1.2.0, but only if we push back >> the release by at least a few weeks. > > This is not a worked example, but this is exactly what is needed to > make possible the "arrays of matrices" functions that were discussed > some time ago. For example, users frequently want to do something like > multiply an array of matrices by an array of matrices; that is not > currently feasible without a very large temporary array (or a loop). > > But I think you were looking for examples of code using the interface, > to see whether it worked well. I'll take what I can get. In order of increasing utility: 1. Descriptions of use cases (as above). 2. Mockups of the Python code demonstrating the use case (e.g. nonfunctional Python code showing a potential generalized ufunc with inputs and outputs). 3. The C code implementing the actual functionality for the use case. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Mon Aug 18 01:43:07 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 17 Aug 2008 23:43:07 -0600 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: <3d375d730808172015t62739266hbee205b97bdc86c3@mail.gmail.com> References: <3d375d730808171743j166e4d30vb01417e8d901c2ab@mail.gmail.com> <3d375d730808172015t62739266hbee205b97bdc86c3@mail.gmail.com> Message-ID: On Sun, Aug 17, 2008 at 9:15 PM, Robert Kern wrote: > On Sun, Aug 17, 2008 at 21:55, Anne Archibald > wrote: > > 2008/8/17 Robert Kern : > >> > >> I suggested that we move it to a branch for the time being so we can > >> play with it and come up with examples of its use. If you have > >> examples that you have already written, I would love to see them. I, > >> for one, am amenable to seeing this in 1.2.0, but only if we push back > >> the release by at least a few weeks. > > > > This is not a worked example, but this is exactly what is needed to > > make possible the "arrays of matrices" functions that were discussed > > some time ago. For example, users frequently want to do something like > > multiply an array of matrices by an array of matrices; that is not > > currently feasible without a very large temporary array (or a loop). > > > > But I think you were looking for examples of code using the interface, > > to see whether it worked well. > > I'll take what I can get. In order of increasing utility: > > 1. Descriptions of use cases (as above). It is also possible to do sums, cumulative sums, and other such things. A generalization of the proposed ufuncs useful for these sorts of things would be mixed type arguments, which would also help in implementing such things as argsort and casting. This requires a different kind of infrastructure for defining and looking up the ufuncs, but I don't think of this as a complication, but rather a simplification as it would allow us to make good use of the code generator and introduce a certain uniformity to the implementation of numpy. Uniformity is good. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Mon Aug 18 03:09:52 2008 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 18 Aug 2008 09:09:52 +0200 Subject: [Numpy-discussion] Interrupting long running calculations .... Message-ID: Hi, Could someone remind me of the current state of numpy with regards to honoring KeyboardInterrupts !? I think KeyboardInterrupt is the exception that would have to be used to do this kind of thing - right !? E.g. by pressing Ctrl-C.... Another question, and not quite numpy specific, is how to generate this exception from another thread -- effectively "injecting" it into the calling stack !? I ask this, because I'm using wxPython, which makes things even more complicated .... :-( Thanks for any help, Sebastian Haase From fperez.net at gmail.com Mon Aug 18 03:21:43 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 18 Aug 2008 00:21:43 -0700 Subject: [Numpy-discussion] Interrupting long running calculations .... In-Reply-To: References: Message-ID: On Mon, Aug 18, 2008 at 12:09 AM, Sebastian Haase wrote: > Another question, and not quite numpy specific, is how to generate > this exception from another thread -- effectively "injecting" it into > the calling stack !? I ask this, because I'm using wxPython, which > makes things even more complicated .... :-( > An undocumented, unsupported, private hack is needed. Have a look at shell.py in ipython, around line 291: http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/1107?file_id=shell.py-20080216095032-xb0is4a97lmosv2z-38 for how to do it. Cheers, f From ondrej at certik.cz Mon Aug 18 04:40:24 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 18 Aug 2008 10:40:24 +0200 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: References: <85b5c3130808171500i731c8a01v6b11a67e4c3e413f@mail.gmail.com> Message-ID: <85b5c3130808180140v4caa0ddbw7abe1a999c8418db@mail.gmail.com> On Mon, Aug 18, 2008 at 1:50 AM, Andrew Dalke wrote: > On Aug 18, 2008, at 12:00 AM, Ondrej Certik wrote: >> There is some inconsistency though, for example one can override A() + >> A(), but one cannot override 1 + 1. This could (should) be fixed >> somehow. > > That will never, ever change in Python. There's no benefit > to being able to redefine int.__add__ and doing so will break > entirely too many assumptions. > > Here's one assumption - the C implementation does some > simple constant folding: > > >>> def f(): > ... print 1+1 > ... > >>> import dis > >>> dis.dis(f) > 2 0 LOAD_CONST 2 (2) > 3 PRINT_ITEM > 4 PRINT_NEWLINE > 5 LOAD_CONST 0 (None) > 8 RETURN_VALUE > >>> > > With what you want that's not possible. > > > Just think of the implementation difficulty. Are changes on the > per-module or per-scope or per-thread level? And performance > would be lousy (or require deep inferencing analysis) because > every operation in C would need to go through the Python API > just in case some fundamental definition like this was changed. > > Such a language is possible. I wouldn't be surprised if > you could do it in Smalltalk and mostly do it in Ruby. But > the general belief is that good design follows the > "open/closed principle": > > "software entities (classes, modules, functions, etc.) > should be open for extension, but closed for modification" > - Bertrand Meyer, quoted by > http://en.wikipedia.org/wiki/Open/closed_principle > > In Python, all types are closed for modification, and > while classes are open for modification it's highly frowned > upon to do so. The name "monkey-patch" sounds somewhat > derisive for a reason. Yeah, I understand the reasoning. My reason is that sometimes you want to do something else on 1/2 rather than to get 0, or 0.5000000. I would like to get my own class called, but if it's again Python, then I am perfectly happy with Python as it is now. No changes needed. Anyway, this is off topic. Ondrej From barrywark at gmail.com Mon Aug 18 10:16:36 2008 From: barrywark at gmail.com (Barry Wark) Date: Mon, 18 Aug 2008 10:16:36 -0400 Subject: [Numpy-discussion] Please volunteer a Mac OS X buildbot slave In-Reply-To: <9457e7c80808120708u6c618843nec72c20860a302e9@mail.gmail.com> References: <9457e7c80808111211v545b3c7awa6e51ce65ccfb11c@mail.gmail.com> <9457e7c80808120708u6c618843nec72c20860a302e9@mail.gmail.com> Message-ID: St?fan, Again, thanks to you and Thomas. cheers, Barry On Tue, Aug 12, 2008 at 10:08 AM, St?fan van der Walt wrote: > 2008/8/12 Barry Wark : >> Stefan, >> >> I'm sorry I dropped the ball on this one. I didn't have time to get >> things working again before I left town for a month and, obviously, >> there it sat. Again, sorry. > > No worries, Barry. That machine was troublesome, and I didn't want to > bother you every week. Thomas Heller's machine should do the job, if > we can get it set up. > > Cheers > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From Hans-Andreas.Engel at deshaw.com Mon Aug 18 10:50:16 2008 From: Hans-Andreas.Engel at deshaw.com (Engel, Hans-Andreas) Date: Mon, 18 Aug 2008 10:50:16 -0400 Subject: [Numpy-discussion] Generalized ufuncs? Message-ID: "Robert Kern" wrote in message news:<3d375d730808172015t62739266hbee205b97bdc86c3 at mail.gmail.com>... > On Sun, Aug 17, 2008 at 21:55, Anne Archibald wrote: > > 2008/8/17 Robert Kern : > >> > >> I suggested that we move it to a branch for the time being so we can > >> play with it and come up with examples of its use. If you have > >> examples that you have already written, I would love to see them. I, > >> for one, am amenable to seeing this in 1.2.0, but only if we push back > >> the release by at least a few weeks. > > > > This is not a worked example, but this is exactly what is needed to > > make possible the "arrays of matrices" functions that were discussed > > some time ago. For example, users frequently want to do something like > > multiply an array of matrices by an array of matrices; that is not > > currently feasible without a very large temporary array (or a loop). > > > > But I think you were looking for examples of code using the interface, > > to see whether it worked well. > > I'll take what I can get. In order of increasing utility: > > 1. Descriptions of use cases (as above). > 2. Mockups of the Python code demonstrating the use case (e.g. > nonfunctional Python code showing a potential generalized ufunc > with inputs and outputs). > 3. The C code implementing the actual functionality for the use case. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco Please find an example on "inner1d" in the following. 1. One use case for an inner function is described on http://scipy.org/scipy/numpy/wiki/GeneralLoopingFunctions and http://thread.gmane.org/gmane.comp.python.numeric.general/17694. (Another one would be the "array of matrices" usage mentioned above; we have called this "dot2d" with signature "(m,n),(n,p)->(m,p)" on http://scipy.org/scipy/numpy/ticket/887: here the matrix multiplication would occur with respect to the last two dimensions.) 2. The mockup python code would be: >>> from numpy import * >>> N = 10 >>> a = random.randn(3, 5, N) >>> b = random.randn(5, N) >>> # standard inner function ... inner(a, b).shape (3, 5, 5) >>> # new ufunc "inner1d" with signature "(i),(i)->()", satisfying GeneralLoopingFunctions use case ... inner1d(a, b).shape (3, 5) 3. Concrete implementation of inner1d in C: /* * This implements the function out = inner1d(in1, in2) with * out[K] = sum_i { in1[K, i] * in2[K, i] } * and multi-index K, as described on * http://scipy.org/scipy/numpy/wiki/GeneralLoopingFunctions * and on http://projects.scipy.org/scipy/numpy/ticket/887. */ static void DOUBLE_inner1d(char **args, intp *dimensions, intp *steps, void *func) { /* Standard ufunc loop length and strides. */ intp dn = dimensions[0]; intp s0 = steps[0]; intp s1 = steps[1]; intp s2 = steps[2]; intp n; /* Additional loop length and strides for dimension "i" in elementary function. */ intp di = dimensions[1]; intp i_s1 = steps[3]; intp i_s2 = steps[4]; intp i; /* Outer loop: equivalent to standard ufuncs */ for (n = 0; n < dn; n++, args[0] += s0, args[1] += s1, args[2] += s2) { char *ip1 = args[0], *ip2 = args[1], *op = args[2]; /* Implement elementary function: out = sum_i { in1[i] * in2[i] } */ npy_double sum = 0; for (i = 0; i < di; i++) { sum += (*(npy_double *)ip1) * (*(npy_double *)ip2); ip1 += i_s1; ip2 += i_s2; } *(npy_double *)op = sum; } } /* Actually create the ufunc object */ static PyUFuncGenericFunction inner1d_functions[] = { DOUBLE_inner1d }; static void *inner1d_data[] = { (void *)NULL }; static char inner1d_sigs[] = { PyArray_DOUBLE, PyArray_DOUBLE, PyArray_DOUBLE }; static void addInner1d(PyObject *dictionary) { PyObject *f = PyUFunc_FromFuncAndDataAndSignature(inner1d_functions, inner1d_data, inner1d_sigs, 1, 2, 1, PyUFunc_None, "inner1d", "inner on the last dimension and broadcast on the other dimensions", 0, "(i),(i)->()"); PyDict_SetItemString(dictionary, "inner1d", f); } From Hans-Andreas.Engel at deshaw.com Mon Aug 18 11:15:59 2008 From: Hans-Andreas.Engel at deshaw.com (Engel, Hans-Andreas) Date: Mon, 18 Aug 2008 11:15:59 -0400 Subject: [Numpy-discussion] Generalized ufuncs? Message-ID: "Charles R Harris" wrote in message news:... On Sun, Aug 17, 2008 at 7:56 PM, St?fan van der Walt wrote: > 2008/8/17 Robert Kern : > > I suggested that we move it to a branch for the time being so we can > > play with it and come up with examples of its use. > > That branch is here: > > http://stefan at svn.scipy.org/svn/numpy/branches/gen_ufuncs > > For an earlier thread about using vector valued ufuncs for sorts and such -- and negative reactions to the suggestion -- go here. One of the major objections was how to call such functions with the ufunc machinery and the needed machinery for type promotions, sub classes, and all that nonsense. Are these dealt with in the patch? The current numpy code for all that is a bit of a mess anyway, and it would be nice to figure out some unified interface to call through and clean up the current code in the process. In fact, I've been making some preliminary efforts in that direction by formatting the current code and working through it. Also, do we also want reduce methods and such? I suspect there is still a lot of work to do to get this whole thing up and running. Chuck ---------- The good news is that the patch just uses of the existing code to deal with all the tricky issues (this is why the patch is so short). By the way, sort could be implemented with the proposed specifications, its signature would be "(i)->(i)". I agree that it would be nice if that code could be made somewhat clearer; however, I think that this task is orthogonal to the generalized ufuncs patch, because there is no code overlap. The way the suggested implementation basically works is to remove the "core dimensions" from the input/output arrays, and then have the existing code handle all the intricacies over the "loop" dimensions. Reduce methods are currently not supported (an error is raised). Therefore, the current patch does not forestall anything and the desired functionality can be added whenever it is clear what would be best. I do not think that it would makes sense to specify/implement all possible extensions, optimizations, concrete ufuncs, morphing of existing numpy functions to ufuncs, etc. at once; presumably it is much better to start with a small but extremely flexible specification of generalized ufuncs first. Best, Hansres From aisaac at american.edu Mon Aug 18 12:04:54 2008 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 18 Aug 2008 12:04:54 -0400 Subject: [Numpy-discussion] indexing (compared to matlab) In-Reply-To: <4FE21854-E95C-42A9-9A10-2EA545B50897@bryant.edu> References: <4FE21854-E95C-42A9-9A10-2EA545B50897@bryant.edu> Message-ID: <48A99DA6.1010005@american.edu> > this should definitely be in the Numpy for Matlab users > page, http://www.scipy.org/NumPy_for_Matlab_Users, right > after the line: > Matlab Numpy Notes Good form is to make that change yourself when you get useful advice. But I did it this time. Cheers, Alan Isaac From oliphant at enthought.com Mon Aug 18 12:13:51 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 18 Aug 2008 11:13:51 -0500 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: References: Message-ID: <48A99FBF.7010406@enthought.com> > The good news is that the patch just uses of the existing code to deal with all the tricky issues (this is why the patch is so short). By the way, sort could be implemented with the proposed specifications, its signature would be "(i)->(i)". I agree that it would be nice if that code could be made somewhat clearer; however, I think that this task is orthogonal to the generalized ufuncs patch, because there is no code overlap. > I agree with this. > The way the suggested implementation basically works is to remove the "core dimensions" from the input/output arrays, and then have the existing code handle all the intricacies over the "loop" dimensions. > > Reduce methods are currently not supported (an error is raised). Therefore, the current patch does not forestall anything and the desired functionality can be added whenever it is clear what would be best. > > I do not think that it would makes sense to specify/implement all possible extensions, optimizations, concrete ufuncs, morphing of existing numpy functions to ufuncs, etc. at once; presumably it is much better to start with a small but extremely flexible specification of generalized ufuncs first. > One of the key reasons I'm enthused about the patch is because it's so small. By enhancing the ufunc object and without changing the signature of the underlying function, the patch is able to implement the general description of a generalized ufunc. I think it is useful to evaluate whether or not a few more changes will allow more functionality with little cost, but I don't think it is worth holding up the patch hoping that the code will get "cleaned-up" (which all code needs according to somebody's definition of cleaning). -Travis From oliphant at enthought.com Mon Aug 18 12:21:53 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 18 Aug 2008 11:21:53 -0500 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: <5b8d13220808162221u2892ab7ah90d4aa9b14e0eb2d@mail.gmail.com> References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <3d375d730808160243i1bed66a8se9b17214f9e05bcb@mail.gmail.com> <48A74080.2010002@esrf.fr> <5b8d13220808162159h2bd417ffg61de6995a3a27f8d@mail.gmail.com> <5b8d13220808162221u2892ab7ah90d4aa9b14e0eb2d@mail.gmail.com> Message-ID: <48A9A1A1.4070207@enthought.com> David Cournapeau wrote: > On Sat, Aug 16, 2008 at 11:59 PM, David Cournapeau wrote: > >> On Sat, Aug 16, 2008 at 11:16 PM, Charles R Harris >> wrote: >> >>> I'm slowly coming to the conviction that there should be no C-ABI changes in >>> 1.2. >>> >> It does not make sense to revert those changes anymore, >> > > Actually, I did not follow the discussion when this change happened, > but it does not look difficult to change the code such as we do not > break the ABI. Instead of replacing the flag, we can put it at the > end, and deprecate (but not remove) the old one. > > Would anyone be strongly against that ? > No, we could do that. -Travis From oliphant at enthought.com Mon Aug 18 12:26:15 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 18 Aug 2008 11:26:15 -0500 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <3d375d730808160243i1bed66a8se9b17214f9e05bcb@mail.gmail.com> <48A74080.2010002@esrf.fr> <5b8d13220808162159h2bd417ffg61de6995a3a27f8d@mail.gmail.com> <5b8d13220808162221u2892ab7ah90d4aa9b14e0eb2d@mail.gmail.com> Message-ID: <48A9A2A7.1080505@enthought.com> Charles R Harris wrote: > > > On Sat, Aug 16, 2008 at 11:21 PM, David Cournapeau > wrote: > > On Sat, Aug 16, 2008 at 11:59 PM, David Cournapeau > > wrote: > > On Sat, Aug 16, 2008 at 11:16 PM, Charles R Harris > > > > wrote: > >> > >> I'm slowly coming to the conviction that there should be no > C-ABI changes in > >> 1.2. > > > > It does not make sense to revert those changes anymore, > > Actually, I did not follow the discussion when this change happened, > but it does not look difficult to change the code such as we do not > break the ABI. Instead of replacing the flag, we can put it at the > end, and deprecate (but not remove) the old one. > > Would anyone be strongly against that ? > > > I have nothing against extensions when they can be made to serve. If a > dictionary gets added to ndarrays I hope it is done that way, likewise > for generalized ufuncs. In the present case I think Travis wants to > preserve the functionality while changing the name and type, and that > doesn't really fit the extension model. But I might be wrong about that. The problem was that I didn't break ABI compatibility at 1.0. I new the char was too small to hold what the field had really become (a flag field). I had intended for 1.1 to be a potential ABI breakage, but this was changed when the release strategy changed. But, there is no real functionality added by changing the ABI at this point. I'm just looking to the future, but I can be convinced that it's too late. What about the version number changes. -Travis From charlesr.harris at gmail.com Mon Aug 18 12:32:08 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 18 Aug 2008 10:32:08 -0600 Subject: [Numpy-discussion] Generalized ufuncs? In-Reply-To: <48A99FBF.7010406@enthought.com> References: <48A99FBF.7010406@enthought.com> Message-ID: On Mon, Aug 18, 2008 at 10:13 AM, Travis E. Oliphant wrote: > > > The good news is that the patch just uses of the existing code to deal > with all the tricky issues (this is why the patch is so short). By the way, > sort could be implemented with the proposed specifications, its signature > would be "(i)->(i)". I agree that it would be nice if that code could be > made somewhat clearer; however, I think that this task is orthogonal to the > generalized ufuncs patch, because there is no code overlap. > > > I agree with this. > > The way the suggested implementation basically works is to remove the > "core dimensions" from the input/output arrays, and then have the existing > code handle all the intricacies over the "loop" dimensions. > > > > Reduce methods are currently not supported (an error is raised). > Therefore, the current patch does not forestall anything and the desired > functionality can be added whenever it is clear what would be best. > > > > I do not think that it would makes sense to specify/implement all > possible extensions, optimizations, concrete ufuncs, morphing of existing > numpy functions to ufuncs, etc. at once; presumably it is much better to > start with a small but extremely flexible specification of generalized > ufuncs first. > > > One of the key reasons I'm enthused about the patch is because it's so > small. By enhancing the ufunc object and without changing the > signature of the underlying function, the patch is able to implement the > general description of a generalized ufunc. > > I think it is useful to evaluate whether or not a few more changes will > allow more functionality with little cost, but I don't think it is worth > holding up the patch hoping that the code will get "cleaned-up" (which > all code needs according to somebody's definition of cleaning). > I think the plan is that 1.2.1 will come out before the end of the year and it would be reasonable to put the patch in there. As gen_ufuncs are currently unused there is no practical effect to waiting until after the 1.2 release. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Aug 18 13:04:36 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 18 Aug 2008 11:04:36 -0600 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: <48A9A2A7.1080505@enthought.com> References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <3d375d730808160243i1bed66a8se9b17214f9e05bcb@mail.gmail.com> <48A74080.2010002@esrf.fr> <5b8d13220808162159h2bd417ffg61de6995a3a27f8d@mail.gmail.com> <5b8d13220808162221u2892ab7ah90d4aa9b14e0eb2d@mail.gmail.com> <48A9A2A7.1080505@enthought.com> Message-ID: On Mon, Aug 18, 2008 at 10:26 AM, Travis E. Oliphant wrote: > Charles R Harris wrote: > > > > > > On Sat, Aug 16, 2008 at 11:21 PM, David Cournapeau > > wrote: > > > > On Sat, Aug 16, 2008 at 11:59 PM, David Cournapeau > > > wrote: > > > On Sat, Aug 16, 2008 at 11:16 PM, Charles R Harris > > > > > > wrote: > > >> > > >> I'm slowly coming to the conviction that there should be no > > C-ABI changes in > > >> 1.2. > > > > > > It does not make sense to revert those changes anymore, > > > > Actually, I did not follow the discussion when this change happened, > > but it does not look difficult to change the code such as we do not > > break the ABI. Instead of replacing the flag, we can put it at the > > end, and deprecate (but not remove) the old one. > > > > Would anyone be strongly against that ? > > > > > > I have nothing against extensions when they can be made to serve. If a > > dictionary gets added to ndarrays I hope it is done that way, likewise > > for generalized ufuncs. In the present case I think Travis wants to > > preserve the functionality while changing the name and type, and that > > doesn't really fit the extension model. But I might be wrong about that. > The problem was that I didn't break ABI compatibility at 1.0. I new the > char was too small to hold what the field had really become (a flag > field). I had intended for 1.1 to be a potential ABI breakage, but > this was changed when the release strategy changed. > > But, there is no real functionality added by changing the ABI at this > point. I'm just looking to the future, but I can be convinced that it's > too late. > > What about the version number changes. > You mean the version number tracking ABI changes? I think it will be useful if/when we do break the ABI so that people will be informed, until then, we could leave it out. If you could figure out how to add a new flags field without affecting the old one or requiring existion applications to be recompiled, that would be good. We also need to distinquish between internal and external ABI's. One way to avoid the problems like this is to put some "spares" in the original structure to reserve space for future enhancements. It can also be useful to use getters and setters, ala C++, in the interface. These would be actual functions instead of macros and hide changes to the structures, just accepting pointers that can be downcast. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From gregory.lielens at fft.be Mon Aug 18 12:55:27 2008 From: gregory.lielens at fft.be (=?ISO-8859-1?Q?Gr=E9gory?= Lielens) Date: Mon, 18 Aug 2008 18:55:27 +0200 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: References: Message-ID: <1219078527.16529.42.camel@gauss.fft> On Sat, 2008-08-16 at 22:03 -0700, Fernando Perez wrote: > Hi all, > > [ please keep all replies to this only on the numpy list. I'm cc'ing > the scipy ones to make others aware of the topic, but do NOT reply on > those lists so we can have an organized thread for future reference] > > In the Python-dev mailing lists, there were recently two threads > regarding the possibility of adding to the language new multiplication > operators (amongst others). This would allow one to define things > like an element-wise and a matrix product for numpy arrays, for > example: > > http://mail.python.org/pipermail/python-dev/2008-July/081508.html > http://mail.python.org/pipermail/python-dev/2008-July/081551.html > > It turns out that there's an old pep on this issue: > > http://www.python.org/dev/peps/pep-0225/ > > which hasn't been ruled out, simply postponed. At this point it seems > that there is room for some discussion, and obviously the input of the > numpy/scipy crowd would be very welcome. I volunteered to host a BOF > next week at scipy so we could collect feedback from those present, > but it's important that those NOT present at the conference can > equally voice their ideas/opinions. > > So I wanted to open this thread here to collect feedback. We'll then > try to have the bof next week at the conference, and I'll summarize > everything for python-dev. Obviously this doesn't mean that we'll get > any changes in, but at least there's interest in discussing a topic > that has been dear to everyone here. > > Cheers, > > f As one of the original author behind the PEP225, I think this is an excellent idea. (BTW, thanks for resurrecting this old PEP :-) and considering it useful :-) :-) ). I think I do not need to speak to much for the PEP, telling that I did not change my mind should be enough ;-)...but still, I can not resist adding a few considerations: Demands for Elementwise operators and/or matrix product operator is likely to resurface from time to time on Python-dev or Python-idea, given that this is a central feature of Matlab and Matlab is a de-facto standard when it comes to numeric-oriented interpreted languages (well, at least in the engineering world, it is in my experience the biggest player by far). At the time of the original discussion on python-dev and of PEP225 redaction , I was new to Python and fresh from Matlab, and the default behavior of elementwise-product annoyed me a lot. Since then, I have learned to appreciate the power of numpy broadcast (Use it extensively in my code :-) ), so the default dehavior do not annoy me anymore... But I still feel that 2 sets of operator would be very useful ( especially in some code which implement directly heavy linear algebra formula), the only thing where my point of view has changed is that I now think that the Matlab way ( defining * as matrix product and .* as elementwise product) is not necessarily the best choice, the reverse choice is as valid... Given the increasing success of Python as a viable alternative, I think that settling the Elementwise operator issue is probably a good idea. Especially as the Python 3000 transition is maybe a good time to investigate syntax changes/extension. > > I don't think so, but given that pep 225 exists and is fully fleshed > out, I guess it should be considered the starting point of the > discussion for reference. This doesn't mean that modifications to it > can't be suggested, but that I'm assuming python-dev will want that as > the reference point. For something as big as this, they would > definitely want to work off a real pep. > > Having said that, I think all ideas are fair game at this point. I > personally would like to see it happen, but if not I'd like to see a > final pronouncement on the matter rather than seeing pep 225 deferred > forever. > I agree 100%. Keeping PEP 225 in limbo is the worst situation imho, given that the discussion about elementwise operator (or matrix product) keep showing again and again, having a final decision (even if negative) would be better. And as I said above, I feel the timing is right for this final decision... Best regards, Greg. From charlesr.harris at gmail.com Mon Aug 18 13:21:09 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 18 Aug 2008 11:21:09 -0600 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: <1219078527.16529.42.camel@gauss.fft> References: <1219078527.16529.42.camel@gauss.fft> Message-ID: On Mon, Aug 18, 2008 at 10:55 AM, Gr?gory Lielens wrote: > On Sat, 2008-08-16 at 22:03 -0700, Fernando Perez wrote: > > Hi all, > > > > [ please keep all replies to this only on the numpy list. I'm cc'ing > > the scipy ones to make others aware of the topic, but do NOT reply on > > those lists so we can have an organized thread for future reference] > > > > In the Python-dev mailing lists, there were recently two threads > > regarding the possibility of adding to the language new multiplication > > operators (amongst others). This would allow one to define things > > like an element-wise and a matrix product for numpy arrays, for > > example: > > > > http://mail.python.org/pipermail/python-dev/2008-July/081508.html > > http://mail.python.org/pipermail/python-dev/2008-July/081551.html > > > > It turns out that there's an old pep on this issue: > > > > http://www.python.org/dev/peps/pep-0225/ > > > > which hasn't been ruled out, simply postponed. At this point it seems > > that there is room for some discussion, and obviously the input of the > > numpy/scipy crowd would be very welcome. I volunteered to host a BOF > > next week at scipy so we could collect feedback from those present, > > but it's important that those NOT present at the conference can > > equally voice their ideas/opinions. > > > > So I wanted to open this thread here to collect feedback. We'll then > > try to have the bof next week at the conference, and I'll summarize > > everything for python-dev. Obviously this doesn't mean that we'll get > > any changes in, but at least there's interest in discussing a topic > > that has been dear to everyone here. > > > > Cheers, > > > > f > > > As one of the original author behind the PEP225, I think this is an > excellent idea. > (BTW, thanks for resurrecting this old PEP :-) and considering it > useful :-) :-) ). > > I think I do not need to speak to much for the PEP, telling that I did > not change my mind should be enough ;-)...but still, I can not resist > adding a few considerations: > > Demands for Elementwise operators and/or matrix product operator is > likely to resurface from time to time on Python-dev or Python-idea, > given that this is a central feature of Matlab and Matlab is a de-facto > standard when it comes to numeric-oriented interpreted languages (well, > at least in the engineering world, it is in my experience the biggest > player by far). > > At the time of the original discussion on python-dev and of PEP225 > redaction , I was new to Python and fresh from Matlab, and the default > behavior of elementwise-product annoyed me a lot. Since then, I have > learned to appreciate the power of numpy broadcast (Use it extensively > in my code :-) ), so the default dehavior do not annoy me anymore... > But I still feel that 2 sets of operator would be very useful > ( especially in some code which implement directly heavy linear algebra > formula), the only thing where my point of view has changed is that I > now think that the Matlab way ( defining * as matrix product and .* as > elementwise product) is not necessarily the best choice, the reverse > choice is as valid... > > Given the increasing success of Python as a viable alternative, I think > that settling the Elementwise operator issue is probably a good idea. > Especially as the Python 3000 transition is maybe a good time to > investigate syntax changes/extension. > > > > > > I don't think so, but given that pep 225 exists and is fully fleshed > > out, I guess it should be considered the starting point of the > > discussion for reference. This doesn't mean that modifications to it > > can't be suggested, but that I'm assuming python-dev will want that as > > the reference point. For something as big as this, they would > > definitely want to work off a real pep. > > > > Having said that, I think all ideas are fair game at this point. I > > personally would like to see it happen, but if not I'd like to see a > > final pronouncement on the matter rather than seeing pep 225 deferred > > forever. > > > > I agree 100%. Keeping PEP 225 in limbo is the worst situation imho, > given that the discussion about elementwise operator (or matrix product) > keep showing again and again, having a final decision (even if negative) > would be better. And as I said above, I feel the timing is right for > this final decision... > Tim Hochberg proposed using the call operator for matrix multiplication, i.e., A(B(C)) Which has the advantage of using an existing operator. It looks like function composition, which isn't that far off the mark if matrices are looked at as mappings, but might take a bit of getting used to. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalke at dalkescientific.com Mon Aug 18 16:00:35 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Mon, 18 Aug 2008 22:00:35 +0200 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <48A5298F.80701@enthought.com> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> Message-ID: <74DB7743-9CF5-4FCE-AD7D-9A52F89AF188@dalkescientific.com> Andrew Dalke: > Any chance of someone reviewing my suggestions for > making the import somewhat faster still? > > http://scipy.org/scipy/numpy/ticket/874 > Travis E. Oliphant: > In sum: I think 2, 3, 6, 7, 8, and 9 can be done immediately. 1) and > 4) could be O.K. but 1) does break code and 4 I'm not sure > about. 5 > seems like it's too much code duplication for too little savings > for my > taste. Since no one else has said yea or nay, and 2.1 release draws nigh[*], the simplest solution is to do 2, 3, 6, 7, 8, and 9. I showed that 1 will break existing code. As for #4 - as far as I can tell the code in 'doc' is recent, so no user code depends on it. Plus, the documentation that's there is effectively unusable, with files like: """ ====== Jargon ====== Placeholder for computer science, engineering and other jargon. """ so I still want to remove the "import doc" in numpy/__init__.py . As for #5, that should probably be tied to the nosetests migration, so it will be done "soon"ish, but not for this release. Is there a lack of disagreement on this? Should I construct a patch accordingly? Or wait longer? Andrew dalke at dalkescientific.com [*] nigh: a word I don't use often except after "draw". http://en.wiktionary.org/wiki/nigh Interesting. English once used "nigh, near, next" instead of "near", "nearer", "nearest". From ondrej at certik.cz Mon Aug 18 16:03:02 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 18 Aug 2008 22:03:02 +0200 Subject: [Numpy-discussion] global overloading of 1+1 -> MyClass(1, 1) In-Reply-To: <85b5c3130808181301s4a96d2b9l6ca26fc553296e4c@mail.gmail.com> References: <85b5c3130808181301s4a96d2b9l6ca26fc553296e4c@mail.gmail.com> Message-ID: <85b5c3130808181303n3b59c3b7hf966105b7469025c@mail.gmail.com> Hi, with Andrew's permission, I am starting a new thread, where our discussion is ontopic. :) My original question was, that I would like to override 1+1 to return MyClass(1, 1) or something. Robert said it would break other libraries and Andrew said this: On Mon, Aug 18, 2008 at 9:23 PM, Andrew Dalke wrote: >> There are basically two possible options: >> >> 1) global overloading of + >> 2) allow to tell Python that "1" is not its int, but ours Integer. >> >> BTW, Sage just preparses Python for exactly this reason and >> substitutes all numbers like 1 with Integer(1). This really sucks >> imho. > > How would you like to do it? Any solution I can think of > would cause huge problems. For example, being able to change > int.__add__ would cause systemic problems throughout the > code base and would prevent, or very much hinder, migration > of Python code to C. Agree. > > Changes to support a different constructor for basic types, > on a per-module basis has its own problems. When are the > new types specified? In the module itself or by the code > which imports the module? Can the definitions be changed > on a scope-by-scope level? > > Now, I can imagine a Python with builtin multimethod > dispatch defined via static scoping, so that > > def __add__(left:int, right:int): > return __builtin__.__add__(__builtin__.__add__(left, right), 2) > > def __int__(s): > return Decimal(s) > > might work, but even here there's a problem because the "2" in > __add__ gets replaced by a Decimal, when I really want it to > be an integer. Not getting why is "2" replaced by Decimal if you don't want to? If you don't want it, you can just reimplement __int__(s). This would of course be only active in the module where it was defined. > > So I don't see how you can do it in the context of Python. > In the context of a very different programming language, > sure, it's possible. > > >> But that's what Sage is doing, they just preparse the input. But in >> SymPy we'd like to use it as a regular Python library in users >> programs, so imho preparsing, or custom compiling using your package >> is not really an option, is it? > > Use a different extension, like ".sympy" and an import hook > which does the .sympy -> .pyc conversion. > > But it's not one I recommend for real code, at least not > without a lot of personal experience to see if it's worthwhile > in the face of the downsides. Yes, this is a mess, this is just like preparsing. Well, not like -- this is preparsing. Ok, in the current state, you don't know either what's going to happen. If you write In [1]: x/2*3/4 you have no idea what the result is going to be, you need to analyze x.__div__() and start from there. But if you write In [2]: 1/2*3/4 currently you know it will be 0. But imho you could as well analyze the global __mul__ (or global __int__, depending on how this would be technically implemented) to see what's going to happen. I mean what is the difference between [1] and [2]? Ondrej From oliphant at enthought.com Mon Aug 18 16:04:05 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 18 Aug 2008 15:04:05 -0500 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <74DB7743-9CF5-4FCE-AD7D-9A52F89AF188@dalkescientific.com> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <74DB7743-9CF5-4FCE-AD7D-9A52F89AF188@dalkescientific.com> Message-ID: <48A9D5B5.80808@enthought.com> Andrew Dalke wrote: > Andrew Dalke: > >> Any chance of someone reviewing my suggestions for >> making the import somewhat faster still? >> >> http://scipy.org/scipy/numpy/ticket/874 >> >> > > > Travis E. Oliphant: > >> In sum: I think 2, 3, 6, 7, 8, and 9 can be done immediately. 1) and >> 4) could be O.K. but 1) does break code and 4 I'm not sure >> about. 5 >> seems like it's too much code duplication for too little savings >> for my >> taste. >> > > Since no one else has said yea or nay, and 2.1 release draws nigh[*], > the simplest solution is to do 2, 3, 6, 7, 8, and 9. I showed that > 1 will break existing code. As for #4 - as far as I can tell the code > in 'doc' is recent, so no user code depends on it. Plus, the > documentation that's there is effectively unusable, with files like: > > I say go ahead including changing #1 and #4. Let's leave 5 for the moment. -Travis From dalke at dalkescientific.com Mon Aug 18 16:45:32 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Mon, 18 Aug 2008 22:45:32 +0200 Subject: [Numpy-discussion] global overloading of 1+1 -> MyClass(1, 1) In-Reply-To: <85b5c3130808181303n3b59c3b7hf966105b7469025c@mail.gmail.com> References: <85b5c3130808181301s4a96d2b9l6ca26fc553296e4c@mail.gmail.com> <85b5c3130808181303n3b59c3b7hf966105b7469025c@mail.gmail.com> Message-ID: <20366E66-798E-4624-B9EA-B03E3896A04A@dalkescientific.com> On Aug 18, 2008, at 10:01 PM, Ondrej Certik wrote: > with Andrew permission, I am starting a new thread, where our > discussion is ontopic. :) Though I want to point out that without specific proposals of how the implementation might look, this thread will not go anywhere as it will be too distant from usable code. I sent examples to show how such a system might look, as the basis for getting a feel if it was practical. I do not think my examples are practical, but they were meant as an example of how such a proposal might look. Since I know that the Python implementation will not change to support per-module or per-scope redefinitions for "1+2" and builtin object constructors, the only feasible mechanism is through some sort of alternate grammar that compiles to either Python or directly to the Python virtual machine. One such was is through import hooks. > Yes, this is a mess, this is just like preparsing. Well, not like -- > this is preparsing. > It's not preparsing. It's parsing. There's no pre about it. It's not a macro language. My ply4python tutorial compiles various Python-like languages to the Python virtual machine bytecode. > I mean what is the difference between [1] and [2]? I want to see how you would extend Python to support such a mechanism before I worried about how to interpret it. Or in other words, the difference between [1] and [2] is that [2] can be fully evaluated through simple static analysis, while [1] cannot. BTW, this is unexpected. Python does constant folding of that expression, but only with specific settings. >>> import dis >>> def f(): ... print 1/2*3/4 ... >>> dis.dis(f) 2 0 LOAD_CONST 1 (1) 3 LOAD_CONST 2 (2) 6 BINARY_DIVIDE 7 LOAD_CONST 3 (3) 10 BINARY_MULTIPLY 11 LOAD_CONST 4 (4) 14 BINARY_DIVIDE 15 PRINT_ITEM 16 PRINT_NEWLINE 17 LOAD_CONST 0 (None) 20 RETURN_VALUE >>> >>> from __future__ import division >>> def f(): ... print 1/2*3/4 ... >>> dis.dis(f) 2 0 LOAD_CONST 7 (0.375) 3 PRINT_ITEM 4 PRINT_NEWLINE 5 LOAD_CONST 0 (None) 8 RETURN_VALUE >>> The only way I can see to do what you want requires multimethods, which don't currently exist in Python except as third-party extensions. The one I know about, from Philip J. Eby, works on a global-level, not module level, because of how registration happens, so it does not support what you would like. Andrew dalke at dalkescientific.com From ondrej at certik.cz Mon Aug 18 17:17:26 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 18 Aug 2008 23:17:26 +0200 Subject: [Numpy-discussion] global overloading of 1+1 -> MyClass(1, 1) In-Reply-To: <20366E66-798E-4624-B9EA-B03E3896A04A@dalkescientific.com> References: <85b5c3130808181301s4a96d2b9l6ca26fc553296e4c@mail.gmail.com> <85b5c3130808181303n3b59c3b7hf966105b7469025c@mail.gmail.com> <20366E66-798E-4624-B9EA-B03E3896A04A@dalkescientific.com> Message-ID: <85b5c3130808181417h5d45c0b1s654b81c596ca3211@mail.gmail.com> On Mon, Aug 18, 2008 at 10:45 PM, Andrew Dalke wrote: > On Aug 18, 2008, at 10:01 PM, Ondrej Certik wrote: > >> with Andrew permission, I am starting a new thread, where our >> discussion is ontopic. :) > > Though I want to point out that without specific proposals > of how the implementation might look, this thread will > not go anywhere as it will be too distant from usable code. > > I sent examples to show how such a system might look, as > the basis for getting a feel if it was practical. I do > not think my examples are practical, but they were meant > as an example of how such a proposal might look. > > Since I know that the Python implementation will not change > to support per-module or per-scope redefinitions for "1+2" > and builtin object constructors, the only feasible mechanism > is through some sort of alternate grammar that compiles to > either Python or directly to the Python virtual machine. > > One such was is through import hooks. > >> Yes, this is a mess, this is just like preparsing. Well, not like -- >> this is preparsing. >> > > It's not preparsing. It's parsing. There's no pre about it. > It's not a macro language. My ply4python tutorial compiles > various Python-like languages to the Python virtual machine > bytecode. > > >> I mean what is the difference between [1] and [2]? > > I want to see how you would extend Python to support such > a mechanism before I worried about how to interpret it. > > Or in other words, the difference between [1] and [2] > is that [2] can be fully evaluated through simple static > analysis, while [1] cannot. > > BTW, this is unexpected. Python does constant folding > of that expression, but only with specific settings. > > >>> import dis > >>> def f(): > ... print 1/2*3/4 > ... > >>> dis.dis(f) > 2 0 LOAD_CONST 1 (1) > 3 LOAD_CONST 2 (2) > 6 BINARY_DIVIDE > 7 LOAD_CONST 3 (3) > 10 BINARY_MULTIPLY > 11 LOAD_CONST 4 (4) > 14 BINARY_DIVIDE > 15 PRINT_ITEM > 16 PRINT_NEWLINE > 17 LOAD_CONST 0 (None) > 20 RETURN_VALUE > >>> > >>> from __future__ import division > >>> def f(): > ... print 1/2*3/4 > ... > >>> dis.dis(f) > 2 0 LOAD_CONST 7 (0.375) > 3 PRINT_ITEM > 4 PRINT_NEWLINE > 5 LOAD_CONST 0 (None) > 8 RETURN_VALUE > >>> > > > > The only way I can see to do what you want requires > multimethods, which don't currently exist in Python > except as third-party extensions. The one I know about, > from Philip J. Eby, works on a global-level, not module > level, because of how registration happens, so it does > not support what you would like. One way to fix that would be to use the trick that Travis Oliphant told me at EuroSciPy -- hack the while (or if) statement and do the preparsing in there. So clearly the language seems to support that, so imho it could be made more official. E.g. you would write: while sympy: e = 1/2 and "e" would be Integer(1)/Integer(2). But anyway, it's kind of hackish and I don't know what to say more about it, besides what I already said. The problem is that I don't have time to dig more into Python internals and without it it seems I cannot provide more constructive answer besides I want some way to avoid Python reducing 1/2 to 0. Generally I believe that if there is will, it can always be done. :) Ondrej From lists at cheimes.de Mon Aug 18 17:22:11 2008 From: lists at cheimes.de (Christian Heimes) Date: Mon, 18 Aug 2008 23:22:11 +0200 Subject: [Numpy-discussion] global overloading of 1+1 -> MyClass(1, 1) In-Reply-To: <85b5c3130808181303n3b59c3b7hf966105b7469025c@mail.gmail.com> References: <85b5c3130808181301s4a96d2b9l6ca26fc553296e4c@mail.gmail.com> <85b5c3130808181303n3b59c3b7hf966105b7469025c@mail.gmail.com> Message-ID: Ondrej Certik wrote: > Ok, in the current state, you don't know either what's going to > happen. If you write > > In [1]: x/2*3/4 > > you have no idea what the result is going to be, you need to analyze > x.__div__() and start from there. But if you write > > In [2]: 1/2*3/4 > > currently you know it will be 0. But imho you could as well analyze > the global __mul__ (or global __int__, depending on how this would be > technically implemented) to see what's going to happen. > > I mean what is the difference between [1] and [2]? Andrew has already pointed it out very well. I like to comment on your proposal from the perspective of a Python core developer as well as the perspective of somebody who has worked with Guido for more than a year. I'd bet my life that Guido is never every going to allow it. The core types are fundemental to the Python interpreter. Even the possibility of pluggable type methods would make the implementation slower, more fragile and much more complicated. We'd have to remove several speed tricks and special cases for e.g. ints and replace them with slower implementations. But don't give up hope yet! During the alpha phase of Python 3.0 and the revamping of the decimal module, some core developers had an even better idea. We were discussing the possibility of making decimals the default for float literals. The idea was rejected eventually but it gave birth to yet another idea. What about making the *result* of a literal pluggable? The Python creates a float for the literal 1.0. Some header in a module could replace the default target 'float' with e.g. decimal.Decimal. Example syntax (rough idea): >>> type(1.0) >>> with float as from decimal import Decimal >>> type(1.0) Wouldn't that solve your general problem more elegant without breaking other modules? Christian From ondrej at certik.cz Mon Aug 18 17:37:41 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 18 Aug 2008 23:37:41 +0200 Subject: [Numpy-discussion] global overloading of 1+1 -> MyClass(1, 1) In-Reply-To: References: <85b5c3130808181301s4a96d2b9l6ca26fc553296e4c@mail.gmail.com> <85b5c3130808181303n3b59c3b7hf966105b7469025c@mail.gmail.com> Message-ID: <85b5c3130808181437y5501109ctaaab89730ca85226@mail.gmail.com> Hi Christian, On Mon, Aug 18, 2008 at 11:22 PM, Christian Heimes wrote: > Ondrej Certik wrote: > > Ok, in the current state, you don't know either what's going to >> happen. If you write >> >> In [1]: x/2*3/4 >> >> you have no idea what the result is going to be, you need to analyze >> x.__div__() and start from there. But if you write >> >> In [2]: 1/2*3/4 >> >> currently you know it will be 0. But imho you could as well analyze >> the global __mul__ (or global __int__, depending on how this would be >> technically implemented) to see what's going to happen. >> >> I mean what is the difference between [1] and [2]? > > Andrew has already pointed it out very well. I like to comment on your > proposal from the perspective of a Python core developer as well as the > perspective of somebody who has worked with Guido for more than a year. > > I'd bet my life that Guido is never every going to allow it. The core > types are fundemental to the Python interpreter. Even the possibility of > pluggable type methods would make the implementation slower, more > fragile and much more complicated. We'd have to remove several speed > tricks and special cases for e.g. ints and replace them with slower > implementations. > > But don't give up hope yet! During the alpha phase of Python 3.0 and the > revamping of the decimal module, some core developers had an even better > idea. We were discussing the possibility of making decimals the default > for float literals. The idea was rejected eventually but it gave birth > to yet another idea. What about making the *result* of a literal > pluggable? The Python creates a float for the literal 1.0. Some header > in a module could replace the default target 'float' with e.g. > decimal.Decimal. > > Example syntax (rough idea): > > >>> type(1.0) > > >>> with float as from decimal import Decimal > >>> type(1.0) > > > Wouldn't that solve your general problem more elegant without breaking > other modules? It absolutely would. Thanks very much for the email. How is your proposal (redefine literals) different to just saying to Python -- hey, just call my class when someone writes "1", e.g. proposition 2) from my first email? Or am I missing something. I agree with the technical reasonings, why some particular solution is not good. I.e. I didn't do any proposal, I am just trying to find a way, so that we don't have to always type In [3]: Integer(1)/2 * x sometimes, but In [4]: x/2 some other times. If you know what I mean. Both do the same thing, but [1] is very annoying to write and a source of common mistakes that people do with SymPy, it simply returns 0. Ondrej From robert.kern at gmail.com Mon Aug 18 18:29:01 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 18 Aug 2008 17:29:01 -0500 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <48A9D5B5.80808@enthought.com> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <74DB7743-9CF5-4FCE-AD7D-9A52F89AF188@dalkescientific.com> <48A9D5B5.80808@enthought.com> Message-ID: <3d375d730808181529u5938f20dod714ce9021ad7dbf@mail.gmail.com> On Mon, Aug 18, 2008 at 15:04, Travis E. Oliphant wrote: > I say go ahead including changing #1 and #4. Let's leave 5 for the moment. I think we can just delete all of the test() and bench() functions except for numpy.{bench,test}(). That way, there is no code duplication. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dalke at dalkescientific.com Mon Aug 18 18:32:53 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Tue, 19 Aug 2008 00:32:53 +0200 Subject: [Numpy-discussion] global overloading of 1+1 -> MyClass(1, 1) In-Reply-To: References: <85b5c3130808181301s4a96d2b9l6ca26fc553296e4c@mail.gmail.com> <85b5c3130808181303n3b59c3b7hf966105b7469025c@mail.gmail.com> Message-ID: On Aug 18, 2008, at 11:22 PM, Christian Heimes wrote: > Example syntax (rough idea): > >>>> type(1.0) > >>>> with float as from decimal import Decimal >>>> type(1.0) > When would this "with float ... " considered valid? For example, could I define things before asking for a redefinition? def f(a=1.0): print a+2.0 with float as from decimal import Decimal In this case would the 1.0 be a float and the 2.0 be a decimal? Or would they both be floats? Would the following be allowed? def f(x): with float as from decimal import Decimal return 1.0 and would that affect things in the function scope or the entire module scope? What if I stuck in a "global int" in that function? What about the following, which uses gmpy if it's available, otherwise uses decimal. try: import gmpy # Pretend there's a bug in some versions # of the library, and don't use gmpy # if that bug is present. if gmpy.mpf("3.0") == 3.0: with float as from gmpy import gmpy.mpf else: # work around the hypothetical bug raise ImportError except ImportError: with float as from decimal import Decimal Hmm.. though this could be handled with some support library, making the result be with float as support_library.the_float_to_use The simplest is that if that statement type appears in the module at all then int/float/complex/string/ dict/list creation goes through a module function, and the statement simply redefines that function. This would slow down the entire module, but if it's what the developer wanted... But I really don't like the unexpected slow down and I think that will be a constant gotcha, with people rewriting code from sum(1 for x in data if x > 100) into something like one = 1 one_hundred = 100 # or possibly: one = __builtin__.int("1") sum(one for x in data if x > one_hundred) in order to eek out performance by not calling new_int("1") so many times. Also, in the use-case for SymPy it means that all modules would start with: with float as from SymPy import float with int as from SymPy import int with complex as from SymPy import complex which makes for tedious boilerplate. Better might be with (float, int, complex) as from SymPy import (float, int, complex) If there's support for with str as .... then how does one create an actual Python string in a module where string is redefined? For example, with str as from MyModule import string open(__builtins__.str("/path/to/nowhere")) wouldn't necessarily work because "/path/to" gets converted to something else, and that something else might not be convertible back to a Python string. Blowing my own horn again, I wrote python4ply precisely to support experimentation like this. We can discuss pros and cons but with a change this extensive it's hard to judge the usefulness with without some real-world experience. And I didn't want to wait for PyPy. ;) Andrew dalke at dalkescientific.com From robert.kern at gmail.com Mon Aug 18 18:37:29 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 18 Aug 2008 17:37:29 -0500 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: References: <1219078527.16529.42.camel@gauss.fft> Message-ID: <3d375d730808181537t6bf2dfc5jb23dde9f0e1e59cd@mail.gmail.com> On Mon, Aug 18, 2008 at 12:21, Charles R Harris wrote: > Tim Hochberg proposed using the call operator for matrix multiplication, > i.e., > > A(B(C)) > > Which has the advantage of using an existing operator. It looks like > function composition, which isn't that far off the mark if matrices are > looked at as mappings, but might take a bit of getting used to. It's certainly worth exploring. My personal opinion is that I could just use a single operator for doing matrix multiplication. I don't want to see two variants of every single operator. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dagss at student.matnat.uio.no Mon Aug 18 18:41:33 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 19 Aug 2008 00:41:33 +0200 (CEST) Subject: [Numpy-discussion] global overloading of 1+1 -> MyClass(1, 1) In-Reply-To: <85b5c3130808181437y5501109ctaaab89730ca85226@mail.gmail.com> References: <85b5c3130808181301s4a96d2b9l6ca26fc553296e4c@mail.gmail.com> <85b5c3130808181303n3b59c3b7hf966105b7469025c@mail.gmail.com> <85b5c3130808181437y5501109ctaaab89730ca85226@mail.gmail.com> Message-ID: <53618.193.157.229.67.1219099293.squirrel@webmail.uio.no> Ondrej Certik wrote: > Hi Christian, > > On Mon, Aug 18, 2008 at 11:22 PM, Christian Heimes > wrote: >> Ondrej Certik wrote: >> > Ok, in the current state, you don't know either what's going to >>> happen. If you write >>> >>> In [1]: x/2*3/4 >>> >>> you have no idea what the result is going to be, you need to analyze >>> x.__div__() and start from there. But if you write >>> >>> In [2]: 1/2*3/4 >>> >>> currently you know it will be 0. But imho you could as well analyze >>> the global __mul__ (or global __int__, depending on how this would be >>> technically implemented) to see what's going to happen. >>> >>> I mean what is the difference between [1] and [2]? >> >> Andrew has already pointed it out very well. I like to comment on your >> proposal from the perspective of a Python core developer as well as the >> perspective of somebody who has worked with Guido for more than a year. >> >> I'd bet my life that Guido is never every going to allow it. The core >> types are fundemental to the Python interpreter. Even the possibility of >> pluggable type methods would make the implementation slower, more >> fragile and much more complicated. We'd have to remove several speed >> tricks and special cases for e.g. ints and replace them with slower >> implementations. >> >> But don't give up hope yet! During the alpha phase of Python 3.0 and the >> revamping of the decimal module, some core developers had an even better >> idea. We were discussing the possibility of making decimals the default >> for float literals. The idea was rejected eventually but it gave birth >> to yet another idea. What about making the *result* of a literal >> pluggable? The Python creates a float for the literal 1.0. Some header >> in a module could replace the default target 'float' with e.g. >> decimal.Decimal. >> >> Example syntax (rough idea): >> >> >>> type(1.0) >> >> >>> with float as from decimal import Decimal >> >>> type(1.0) >> >> >> Wouldn't that solve your general problem more elegant without breaking >> other modules? > > It absolutely would. Thanks very much for the email. How is your > proposal (redefine literals) different to just saying to Python -- > hey, just call my class when someone writes "1", e.g. proposition 2) > from my first email? Or am I missing something. > > > I agree with the technical reasonings, why some particular solution is > not good. I.e. I didn't do any proposal, I am just trying to find a > way, so that we don't have to always type > > In [3]: Integer(1)/2 * x > > sometimes, but > > In [4]: x/2 > > some other times. If you know what I mean. Both do the same thing, but > [1] is very annoying to write and a source of common mistakes that > people do with SymPy, it simply returns 0. It should be mentioned here that this is exactly what the SAGE preparser does -- all "2" turns into "Integer(2)", and that idea definitely seems to work for them. (And so if you want Python behaviour in SAGE, you simply do "Integer = int", while normally "1/2" becomes a rational object in QQ because one object in ZZ is divided by another one). Pluggable literals seems like a much better idea to me, not from pragmatic but from design reasons -- I would prefer them even in the (very unlikely) case that they were slower and harder to implement than overriding the operators for the builtin types. (But I'm just a bystander here.) Dag Sverre From lists at cheimes.de Mon Aug 18 19:06:42 2008 From: lists at cheimes.de (Christian Heimes) Date: Tue, 19 Aug 2008 01:06:42 +0200 Subject: [Numpy-discussion] global overloading of 1+1 -> MyClass(1, 1) In-Reply-To: References: <85b5c3130808181301s4a96d2b9l6ca26fc553296e4c@mail.gmail.com> <85b5c3130808181303n3b59c3b7hf966105b7469025c@mail.gmail.com> Message-ID: Andrew Dalke wrote: > When would this "with float ... " considered valid? [long posting] Oh h... what have I done ... *g* Slow down, please. For now there are no concrete plans what-so-ever to implement the feature in the near future. Some developers have expressed their interest in a way to alter the resulting type of a literal. It was my attention to show you, that we have discussed the idea, too. Now for the "with type as from import" syntax. I came up with the syntax idea about an hour ago. I tried to come up with some nice syntax that reuses existing keywords. IMHO it has a nice ring. Other possibilities I came up with: def float as factory def float as from module import factory with float yield factory with float yield from module import factory After some careful thinking I'm in favor of "with ... yield ...". It's less ambiguous and can't be mistaken for "with open(filename) as fh". The ideas needs a good PEP. You are definitely up to something. You also came up with a list of possible issues and corner cases. Are you interested in pursuing the proposal? *wink* Christian From aisaac at american.edu Mon Aug 18 19:31:45 2008 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 18 Aug 2008 19:31:45 -0400 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: <3d375d730808181537t6bf2dfc5jb23dde9f0e1e59cd@mail.gmail.com> References: <1219078527.16529.42.camel@gauss.fft> <3d375d730808181537t6bf2dfc5jb23dde9f0e1e59cd@mail.gmail.com> Message-ID: <48AA0661.1030104@american.edu> > On Mon, Aug 18, 2008 at 12:21, Charles R Harris wrote: >> Tim Hochberg proposed using the call operator for matrix multiplication, >> i.e., >> A(B(C)) Robert Kern wrote: > It's certainly worth exploring. My personal opinion is that I could > just use a single operator for doing matrix multiplication. I don't > want to see two variants of every single operator. User opinion on Tim's proposal: it is not bad, especially if you write A(B)(C) instead of A(B(C)). But in the end I think it looks acceptable only if you do not plan to use matrices much. For teaching purposes: I think it fails for not being explicit enough. I would even prefer A.dot(B).dot(C) for this reason. But I prefer a normal binary multiplication operator! User opinion on Robert's comment: I agree. I use matrix powers and so would feel some loss there, but I could accept say M.pow(n) instead of M**n. So it seems to me, as a user, that the SciPy community should definitely ask for a way to have a distinct matrix multiplication operator. Proposal 1: PEP 225, but *just* for multiplication. Either ~* as in the PEP or @* (which I prefer). (This looks simplest.) Proposal 2: PEP 225 (but maybe using @ instead of ~). Proposal 3: use of a unicode character, perhaps ? since it is in the Latin1 extension of ASCII. Proposal 4: user defined operators, perhaps as described in PEP 225. In this case, @dot would be a natural. (This looks most complex but also most powerful.) But is A @dot B really better than say A.dot(B)? Maybe. Propsal 5: use the existing language, either using __call__ as in Tim's proposal or adding a ``dot`` method. Did I miss any? As PEP 225 notes, implementing proposals 1 or 2 does not rule out future interest in proposal 4. Cheers, Alan Isaac From ondrej at certik.cz Mon Aug 18 19:44:21 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Tue, 19 Aug 2008 01:44:21 +0200 Subject: [Numpy-discussion] global overloading of 1+1 -> MyClass(1, 1) In-Reply-To: References: <85b5c3130808181301s4a96d2b9l6ca26fc553296e4c@mail.gmail.com> <85b5c3130808181303n3b59c3b7hf966105b7469025c@mail.gmail.com> Message-ID: <85b5c3130808181644q62df9e00u7b2932a14d28f838@mail.gmail.com> On Tue, Aug 19, 2008 at 1:06 AM, Christian Heimes wrote: > Andrew Dalke wrote: >> When would this "with float ... " considered valid? > > [long posting] > > Oh h... what have I done ... *g* > > Slow down, please. For now there are no concrete plans what-so-ever to > implement the feature in the near future. Some developers have expressed > their interest in a way to alter the resulting type of a literal. It was > my attention to show you, that we have discussed the idea, too. > > Now for the "with type as from import" syntax. I came up with the syntax > idea about an hour ago. I tried to come up with some nice syntax that > reuses existing keywords. IMHO it has a nice ring. Other possibilities I > came up with: > > def float as factory > def float as from module import factory > with float yield factory > with float yield from module import factory > > After some careful thinking I'm in favor of "with ... yield ...". It's > less ambiguous and can't be mistaken for "with open(filename) as fh". > > The ideas needs a good PEP. You are definitely up to something. You also > came up with a list of possible issues and corner cases. Are you > interested in pursuing the proposal? *wink* Are we able to provide an actual patch to Python that implements this? If so, then I am. Imho the proposal should come with an actual patch, otherwise it's difficult to judge it. Ondrej From stefan at sun.ac.za Mon Aug 18 19:48:34 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 18 Aug 2008 18:48:34 -0500 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <48A9D5B5.80808@enthought.com> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <74DB7743-9CF5-4FCE-AD7D-9A52F89AF188@dalkescientific.com> <48A9D5B5.80808@enthought.com> Message-ID: <9457e7c80808181648i1ba902d9g2b95c94e435e5e3e@mail.gmail.com> 2008/8/18 Travis E. Oliphant : > I say go ahead including changing #1 and #4. Let's leave 5 for the moment. I ran several benchmarks and made sure that these imports take a minimal amount of time. Wouldn't we want users to have access with the doc framework without doing anything special? And, yes, some of the documents are empty, but a number of them have already been written. I still think we are going about this the wrong way. We have two different sets of expectations, and we can't satisfy both by ripping everything apart. I'd much prefer two entry points into NumPy: one for people who need speed, and one for those who need the convenience of everything being at hand. St?fan From dalke at dalkescientific.com Mon Aug 18 19:57:20 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Tue, 19 Aug 2008 01:57:20 +0200 Subject: [Numpy-discussion] global overloading of 1+1 -> MyClass(1, 1) In-Reply-To: References: <85b5c3130808181301s4a96d2b9l6ca26fc553296e4c@mail.gmail.com> <85b5c3130808181303n3b59c3b7hf966105b7469025c@mail.gmail.com> Message-ID: <84696766-122F-47E4-8446-680754C965AC@dalkescientific.com> On Aug 19, 2008, at 1:06 AM, Christian Heimes wrote: > [long posting] > > Oh h... what have I done ... *g* *shrug* I write long emails. I've been told that by several people. It's probably a bad thing. > The ideas needs a good PEP. You are definitely up to something. You > also > came up with a list of possible issues and corner cases. Are you > interested in pursuing the proposal? *wink* No. I'm doing this to head off endless discussion about some proposed hypothetical future Python. If someone wants to experiment with new operators, new ways of handling int/float/whatever constructions, new lexical forms, or whatever then IMNSHO the best thing is to implement it and see if it's actually useful. In most cases there are seriously hard questions that are hand-waved away, because it's more fun to talk about language design than to implement one and because it's hard to tweak CPython to try out new things. Hence I wrote python4ply and an extensive tutorial specifically so people could easily jump into the implementation details, find the sticky points, then either quickly reject a bad idea or iterate to get a good solution. Many of these changes can be done with less than a couple of days of work. Then to show that there are sticky points, I listed a few. BTW, it's *fun* to modify an existing language and afterwards it you know a secret - that programming languages are just flimsy facades held together by a shared hallucination. Like in a dream, change things too much or leave gaps and people notice the illogic, wake up, and look elsewhere for refuge from harsh digital machine reality. Hmmm, it really is too late for me. Andrew dalke at dalkescientific.com From oliphant at enthought.com Mon Aug 18 20:11:26 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 18 Aug 2008 19:11:26 -0500 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <9457e7c80808181648i1ba902d9g2b95c94e435e5e3e@mail.gmail.com> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <74DB7743-9CF5-4FCE-AD7D-9A52F89AF188@dalkescientific.com> <48A9D5B5.80808@enthought.com> <9457e7c80808181648i1ba902d9g2b95c94e435e5e3e@mail.gmail.com> Message-ID: <48AA0FAE.8000105@enthought.com> St?fan van der Walt wrote: > 2008/8/18 Travis E. Oliphant : > >> I say go ahead including changing #1 and #4. Let's leave 5 for the moment. >> > > I ran several benchmarks and made sure that these imports take a > minimal amount of time. Wouldn't we want users to have access with > the doc framework without doing anything special? And, yes, some of > the documents are empty, but a number of them have already been > written. > > I still think we are going about this the wrong way. We have two > different sets of expectations, and we can't satisfy both by ripping > everything apart. I'd much prefer two entry points into NumPy: one > for people who need speed, and one for those who need the convenience > of everything being at hand. > > I think you are right Stefan. It would be great to have another name-space that is lighter from which numpy imports. But there is no reason to hold up these useful speed increases waiting for a better solution. -Travis From sransom at nrao.edu Mon Aug 18 20:26:48 2008 From: sransom at nrao.edu (Scott Ransom) Date: Mon, 18 Aug 2008 20:26:48 -0400 Subject: [Numpy-discussion] global overloading of 1+1 -> MyClass(1, 1) In-Reply-To: <84696766-122F-47E4-8446-680754C965AC@dalkescientific.com> References: <85b5c3130808181301s4a96d2b9l6ca26fc553296e4c@mail.gmail.com> <85b5c3130808181303n3b59c3b7hf966105b7469025c@mail.gmail.com> <84696766-122F-47E4-8446-680754C965AC@dalkescientific.com> Message-ID: <20080819002648.GA20027@ssh.cv.nrao.edu> On Tue, Aug 19, 2008 at 01:57:20AM +0200, Andrew Dalke wrote: > BTW, it's *fun* to modify an existing language and > afterwards it you know a secret - that programming > languages are just flimsy facades held together by > a shared hallucination. Like in a dream, change > things too much or leave gaps and people notice the > illogic, wake up, and look elsewhere for refuge from > harsh digital machine reality. That is the quote of the month for sure. Scott -- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: sransom at nrao.edu Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From dalke at dalkescientific.com Mon Aug 18 20:46:53 2008 From: dalke at dalkescientific.com (Andrew Dalke) Date: Tue, 19 Aug 2008 02:46:53 +0200 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <9457e7c80808181648i1ba902d9g2b95c94e435e5e3e@mail.gmail.com> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <74DB7743-9CF5-4FCE-AD7D-9A52F89AF188@dalkescientific.com> <48A9D5B5.80808@enthought.com> <9457e7c80808181648i1ba902d9g2b95c94e435e5e3e@mail.gmail.com> Message-ID: <86FB73FF-8A37-472B-8EF3-5459EBC754D0@dalkescientific.com> On Aug 19, 2008, at 1:48 AM, St?fan van der Walt wrote: > Wouldn't we want users to have access with > the doc framework without doing anything special? And, yes, some of > the documents are empty, but a number of them have already been > written. How do users know that those are present? How do users view those docs? You're the one who added that directory, yes?, so you've probably got the most experience with it. I couldn't figure out it, and the README in the doc/ directory wasn't helpful. [josiah:numpy/numpy/doc] dalke% svn log __init__.py ------------------------------------------------------------------------ r5371 | stefan | 2008-07-09 10:13:18 +0200 (Wed, 09 Jul 2008) | 2 lines Add numpy.doc topical documentation framework. The files with 1K bytes or less are undocumented -rw-r--r-- 1 dalke staff 307 Aug 3 00:59 __init__.py -rw-r--r-- 1 dalke staff 5203 Aug 15 17:44 basics.py -rw-r--r-- 1 dalke staff 5413 Aug 15 17:44 broadcasting.py -rw-r--r-- 1 dalke staff 5078 Aug 15 17:44 creation.py -rw-r--r-- 1 dalke staff 9854 Aug 15 17:44 glossary.py -rw-r--r-- 1 dalke staff 94 Aug 3 00:59 howtofind.py -rw-r--r-- 1 dalke staff 14286 Aug 15 17:44 indexing.py -rw-r--r-- 1 dalke staff 9608 Aug 15 17:44 internals.py -rw-r--r-- 1 dalke staff 82 Aug 3 00:59 io.py -rw-r--r-- 1 dalke staff 96 Aug 3 00:59 jargon.py -rw-r--r-- 1 dalke staff 130 Aug 3 00:59 methods_vs_functions.py -rw-r--r-- 1 dalke staff 81 Aug 3 00:59 misc.py -rw-r--r-- 1 dalke staff 100 Aug 3 00:59 performance.py -rw-r--r-- 1 dalke staff 7256 Aug 15 17:44 structured_arrays.py -rw-r--r-- 1 dalke staff 5520 Aug 15 17:44 ufuncs.py 8 documentation files, 6 placeholder files. I agree, the load time is very small. But with all my patches in place the import time goes down from about 0.18 second to about 0.10 seconds. Times add up. > I still think we are going about this the wrong way. We have two > different sets of expectations, and we can't satisfy both by ripping > everything apart. I'd much prefer two entry points into NumPy: one > for people who need speed, and one for those who need the convenience > of everything being at hand. I thought I was very careful to not rip things apart. :( Everything I did was API compatible except for the proposed removals of numpy.ctypeslib and numpy.doc. I chose ctypeslib because importing ctypes takes 10% of the total load time on my box. I chose numpy.doc because I couldn't figure out how it's used. Now with Robert's go-ahead I'll also remove the "test" and "bench" entry points from everywhere except numpy.test and numpy.bench, so I will break some more compatibility. But not that "bench" doesn't currently work. I agree about two entry points but that's not going to happen by the next release. Actually, here's my quote from elsewhere in this discussion. I happen to think it's a mistake and there are other ways to have addressed the underlying requirement, but I know that's not going to change. (For example, follow matplotlib approach where there's a special library designed to be imported in interactive use. But I am *not* proposing this change.) I stressed the *not* because so far I've gone through: import Numeric import numarray import numpy and there's probably also a "from matplotlib import numerix" somewhere in there. It seems like every time I use num* (which isn't often) I need to learn a new library. I don't want to switch again for a few years. Andrew dalke at dalkescientific.com From gael.varoquaux at normalesup.org Mon Aug 18 21:23:23 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 19 Aug 2008 03:23:23 +0200 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: <48A88A07.4050507@american.edu> References: <48A82134.6090501@american.edu> <20080817191435.GB692@phare.normalesup.org> <48A88A07.4050507@american.edu> Message-ID: <20080819012323.GA855@phare.normalesup.org> On Sun, Aug 17, 2008 at 04:28:55PM -0400, Alan G Isaac wrote: > That said, what kind of problems do you have in mind? A lot of software still don't deal well with unicode (wxPython's unicode situation under windows, for instance, in "interesting"). But wht I am most worried about is not being able to enter the symbol, because I am in an editor I don't know, and the symbol is not on my keyboard. This has happened to me more than once with unicode. So I see the losss, and not the gain. I actually think PEP 225 is pretty good. I have not big opinion about "~" vs "@". Ga?l From stefan at sun.ac.za Mon Aug 18 21:36:05 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 18 Aug 2008 20:36:05 -0500 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <48AA0FAE.8000105@enthought.com> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <74DB7743-9CF5-4FCE-AD7D-9A52F89AF188@dalkescientific.com> <48A9D5B5.80808@enthought.com> <9457e7c80808181648i1ba902d9g2b95c94e435e5e3e@mail.gmail.com> <48AA0FAE.8000105@enthought.com> Message-ID: <9457e7c80808181836n5b2720bw1820a22a26234860@mail.gmail.com> 2008/8/18 Travis E. Oliphant : >> I still think we are going about this the wrong way. We have two >> different sets of expectations, and we can't satisfy both by ripping >> everything apart. I'd much prefer two entry points into NumPy: one >> for people who need speed, and one for those who need the convenience >> of everything being at hand. >> >> > I think you are right Stefan. It would be great to have another > name-space that is lighter from which numpy imports. But there is no > reason to hold up these useful speed increases waiting for a better > solution. Just to be clear: I am very happy for the speed improvements. I'm just urging for the same caution in changing the user-visible API that has been shown for the C-level API. While the C-level changes require a recompile, the user-visible changes require a rewrite. This does not pertain to numpy.doc, which has never been exposed to the world before. There is a bigger issue that needs to be considered here, though, and that is whether NumPy will move from its historic idiom of exposing everything to the user upon import, or whether we'll require more imports to be made manually. Regards St?fan From travis at enthought.com Mon Aug 18 21:38:57 2008 From: travis at enthought.com (Travis Vaught) Date: Mon, 18 Aug 2008 20:38:57 -0500 Subject: [Numpy-discussion] global overloading of 1+1 -> MyClass(1, 1) In-Reply-To: <84696766-122F-47E4-8446-680754C965AC@dalkescientific.com> References: <85b5c3130808181301s4a96d2b9l6ca26fc553296e4c@mail.gmail.com> <85b5c3130808181303n3b59c3b7hf966105b7469025c@mail.gmail.com> <84696766-122F-47E4-8446-680754C965AC@dalkescientific.com> Message-ID: <7FE2FF17-84D6-46EC-AFA7-4E5C48212CED@enthought.com> On Aug 18, 2008, at 6:57 PM, Andrew Dalke wrote: > ... > > BTW, it's *fun* to modify an existing language and > afterwards it you know a secret - that programming > languages are just flimsy facades held together by > a shared hallucination. Like in a dream, change > things too much or leave gaps and people notice the > illogic, wake up, and look elsewhere for refuge from > harsh digital machine reality. > I think you meant to say "shared abstraction," right Andrew ... (call the guards!) Travis From stefan at sun.ac.za Mon Aug 18 21:49:51 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 18 Aug 2008 18:49:51 -0700 Subject: [Numpy-discussion] NumPy 1.2.0b2 released In-Reply-To: <86FB73FF-8A37-472B-8EF3-5459EBC754D0@dalkescientific.com> References: <48A49C89.6000800@american.edu> <48A49EA0.7020805@american.edu> <51648AB4-BC9E-4834-B387-397CDED06024@dalkescientific.com> <48A5298F.80701@enthought.com> <74DB7743-9CF5-4FCE-AD7D-9A52F89AF188@dalkescientific.com> <48A9D5B5.80808@enthought.com> <9457e7c80808181648i1ba902d9g2b95c94e435e5e3e@mail.gmail.com> <86FB73FF-8A37-472B-8EF3-5459EBC754D0@dalkescientific.com> Message-ID: <9457e7c80808181849x1cf9c540q4ca993bff83de42e@mail.gmail.com> 2008/8/18 Andrew Dalke : > How do users know that those are present? How do users > view those docs? You're the one who added that directory, yes?, > so you've probably got the most experience with it. I > couldn't figure out it, and the README in the doc/ directory > wasn't helpful. The numpy/doc directory existed before I implemented this, which may explain some "odd" design decisions. Usage is meant to happen via "help" or IPython's "?": In [2]: np.doc? Type: module Base Class: String Form: Namespace: Interactive File: /Users/stefan/lib/python2.5/site-packages/numpy/doc/__init__.py Docstring: The following topics are available: - basics - broadcasting - creation - glossary - howtofind - indexing - internals - io - jargon - methods_vs_functions - misc - performance - structured_arrays - ufuncs In [3]: np.doc.broadcasting? Type: module Base Class: String Form: Namespace: Interactive File: /Users/stefan/lib/python2.5/site-packages/numpy/doc/reference/broadcasting.py Docstring: ======================== Broadcasting over arrays ======================== [...] > I agree, the load time is very small. But with all my patches > in place the import time goes down from about 0.18 second to > about 0.10 seconds. Times add up. Here are some of the timings I did, for interest's sake: For each trial, I included N copies of the NumPy documentation guide as topics under "numpy.do", and took the best of 3 trials. The topic number is currently 14. Without numpy.doc: real 0m0.259s user 0m0.082s sys 0m0.169s -------------------- 200 files real 0m0.341s user 0m0.095s sys 0m0.232s --------------------- 100 real 0m0.282s user 0m0.087s sys 0m0.190s --------------------- 50 real 0m0.273s user 0m0.085s sys 0m0.179s stefan at appel:/tmp$ time python -c 'import numpy' --------------------- 20 real 0m0.262s user 0m0.083s sys 0m0.173s ------------------------ >> I still think we are going about this the wrong way. We have two >> different sets of expectations, and we can't satisfy both by ripping >> everything apart. I'd much prefer two entry points into NumPy: one >> for people who need speed, and one for those who need the convenience >> of everything being at hand. > > I thought I was very careful to not rip things apart. :( > > Everything I did was API compatible except for the > proposed removals of numpy.ctypeslib and numpy.doc. I > chose ctypeslib because importing ctypes takes 10% of > the total load time on my box. I chose numpy.doc because > I couldn't figure out how it's used. Sorry, I did not mean to make you sound like a back-yard surgeon! Maybe hyperbole is best avoided. I am quite happy with the non-API changing modifications you propose, and probably with the others too: I just want us to get our heads together and decide on a policy before we proceed (see my reply to Travis). > It seems like every time I use num* (which isn't often) I > need to learn a new library. I don't want to switch again > for a few years. Sure, we all need to get work done. But in "we" I include those who already wrote apps using numpy.ctypeslib. Cheers St?fan From stefan at sun.ac.za Mon Aug 18 21:55:46 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 18 Aug 2008 18:55:46 -0700 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: <20080819012323.GA855@phare.normalesup.org> References: <48A82134.6090501@american.edu> <20080817191435.GB692@phare.normalesup.org> <48A88A07.4050507@american.edu> <20080819012323.GA855@phare.normalesup.org> Message-ID: <9457e7c80808181855m186154e9md3eeae5282d1db6@mail.gmail.com> 2008/8/18 Gael Varoquaux : > I actually think PEP 225 is pretty good. I have not big opinion about "~" > vs "@". Both of these already have meanings ("boolean not" and "decorator"), so it's pretty much a toss-up for me. In a way, the concept of a decorator could still apply: @* takes the function `mul` and returns a new function, array_mul. Cheers St?fan From cournape at gmail.com Mon Aug 18 22:49:10 2008 From: cournape at gmail.com (David Cournapeau) Date: Mon, 18 Aug 2008 19:49:10 -0700 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <3d375d730808160243i1bed66a8se9b17214f9e05bcb@mail.gmail.com> <48A74080.2010002@esrf.fr> <5b8d13220808162159h2bd417ffg61de6995a3a27f8d@mail.gmail.com> <5b8d13220808162221u2892ab7ah90d4aa9b14e0eb2d@mail.gmail.com> <48A9A2A7.1080505@enthought.com> Message-ID: <5b8d13220808181949q4bb9fd43g3b36d21ccd7cb39e@mail.gmail.com> On Mon, Aug 18, 2008 at 10:04 AM, Charles R Harris wrote: > >If you could figure out how to add a new flags field > without affecting the old one or requiring existion applications to be > recompiled, that would be good. Adding a member to a struct does not break ABI as long as the struct is not used in another struct (or you could use something like C++ pimpl: using struct inside struct is how inheritence is implemented by most compilers I believe in C++, and pimpl is often used to add members to inherited classes). > We also need to distinquish between internal > and external ABI's. What is internal and external ABI ? > One way to avoid the problems like this is to put some > "spares" in the original structure to reserve space for future enhancements. If pimpl can be used in C python, I think it is better to use this than reserving space. http://en.wikipedia.org/wiki/Opaque_pointer#C.2B.2B I still don't know enough about C internals of numpy, but I believe both would break a lot of code at the API level. > It can also be useful to use getters and setters, ala C++, in the interface. Yes, that's basically what the pimpl principle is about. But it breaks ABI and API (see gtk 3.0 discussion for the same kind of problems). http://tirania.org/blog/archive/2008/Jul-14.html http://techbase.kde.org/Policies/Binary_Compatibility_Issues_With_C%2B%2B cheers, David From gregory.lielens at fft.be Tue Aug 19 03:49:14 2008 From: gregory.lielens at fft.be (=?ISO-8859-1?Q?Gr=E9gory?= Lielens) Date: Tue, 19 Aug 2008 09:49:14 +0200 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: References: <1219078527.16529.42.camel@gauss.fft> Message-ID: <1219132154.16529.73.camel@gauss.fft> On Mon, 2008-08-18 at 11:21 -0600, Charles R Harris wrote: > > > Tim Hochberg proposed using the call operator for matrix > multiplication, i.e., > > A(B(C)) > > Which has the advantage of using an existing operator. It looks like > function composition, which isn't that far off the mark if matrices > are looked at as mappings, but might take a bit of getting used to. > > Chuck > This was proposed at the time (It is mentioned in the PEP, but not in details). Clever, but it leads to problems that, imho, makes the new operator superior: -It does not have the classic priority likes normal multiplication, but the presence of parenthesis somewhat mitigates that. Associativity is also not obvious, and mixing with scalar product does not always play nice... -It bares nothing in common with elementwise multiplication, which is bothering if you enjoy orthogonality in language syntax (I do, it help me memorize syntax tremendously), and looks like a (not pretty) trick, especially when teaching the language to beginners -It is not so easy to read: A simple linear algebra formula like system solution could be written x = (A.I)(b), which looks very nice.... [ A.I for lazy inversion of A, evaluated at assignation or as late as possible, to allow for efficient implementation of A * x = b -> x = A.I*b. This lazy operands were considered preferable to left and right division ? la Matlab at the time....] Now if you go back to an "heavy" formula from the PEP, it gives, using classic * as matrix multiply and / as right-divide (x/Y = x * Y.I) b = a.I - a.I * u / (c.I + v/a*u) * v / a or with the tilde-operators b = a.I - a.I ~* u / (c.I + v~/a~*u) ~* v ~/ a Using __call__ as matmul: b = a.I - ( (a.I)(u) / (c.I + (v/a)(u)) )(v) / a Still quite nice, but more difficult to parse visually imho... However, as most editors have parenthesis highlighting, it is worth considering, notation can be abused but I agree that if you use spaces and parenthesis cleverly, you can have some nice expressions...Pity the original discussions are not available anymore, it would have been nice to find the original drawbacks mentioned about __call__.... From gregory.lielens at fft.be Tue Aug 19 04:10:40 2008 From: gregory.lielens at fft.be (=?ISO-8859-1?Q?Gr=E9gory?= Lielens) Date: Tue, 19 Aug 2008 10:10:40 +0200 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: <1219132154.16529.73.camel@gauss.fft> References: <1219078527.16529.42.camel@gauss.fft> <1219132154.16529.73.camel@gauss.fft> Message-ID: <1219133440.16529.83.camel@gauss.fft> On Tue, 2008-08-19 at 09:49 +0200, Gr?gory Lielens wrote: > Using __call__ as matmul: > b = a.I - ( (a.I)(u) / (c.I + (v/a)(u)) )(v) / a oups, of course you do not have right-divide in this case, it would thus read b = a.I - (a.I) (u) ( ( c.I + (v)(a.I)(u) ).I ) (v) (a.I) hum, given the amount of re-read I had to do, even with parenthesis-highlighting, I think that visual parsing for complex formula was the biggest drawback ;-) From strawman at astraw.com Tue Aug 19 06:48:13 2008 From: strawman at astraw.com (Andrew Straw) Date: Tue, 19 Aug 2008 12:48:13 +0200 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: <48A6CB04.5070706@astraw.com> References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <3d375d730808160243i1bed66a8se9b17214f9e05bcb@mail.gmail.com> <48A6CB04.5070706@astraw.com> Message-ID: <48AAA4ED.3010808@astraw.com> Andrew Straw wrote: > Robert Kern wrote: >> On Sat, Aug 16, 2008 at 04:34, Jon Wright wrote: >> >>> Travis, St?fan, >>> >>> I missed Travis mail previously. Are you *really* sure you want force >>> all C code which uses numpy arrays to be recompiled? If you mean that >>> all your matplotlib/PIL/pyopengl/etc users are going to have to make a >>> co-ordinated upgrade, then this seems to be a grave mistake. >>> >> FWIW, neither PIL nor PyOpenGL have C code which uses numpy arrays, so >> they are entirely unaffected. And this does not require an *upgrade* >> of the actually affected packages, just a rebuild of the binary. >> >> > I'll also point out that PEP 3118 will make this unnecessary in the > future for many applications. http://www.python.org/dev/peps/pep-3118/ > > From what I can tell ( http://svn.python.org/view?rev=61491&view=rev ), > this is already in Python 2.6. I just tried to make use of this functionality with the Python 2.6 beta 2 release and numpy svn trunk. Everything is present in Python itself, but numpy doesn't yet support the PEP. -Andrew From bblais at bryant.edu Tue Aug 19 07:36:11 2008 From: bblais at bryant.edu (Brian Blais) Date: Tue, 19 Aug 2008 07:36:11 -0400 Subject: [Numpy-discussion] indexing (compared to matlab) In-Reply-To: <48A99DA6.1010005@american.edu> References: <4FE21854-E95C-42A9-9A10-2EA545B50897@bryant.edu> <48A99DA6.1010005@american.edu> Message-ID: On Aug 18, 2008, at 12:04 , Alan G Isaac wrote: >> this should definitely be in the Numpy for Matlab users >> page, http://www.scipy.org/NumPy_for_Matlab_Users, right >> after the line: >> Matlab Numpy Notes > I did put it in! (did it disappear or something?) I also modified the previous line to say that the syntax a[0:3][:,4:9] gave read-only access. bb -- Brian Blais bblais at bryant.edu http://web.bryant.edu/~bblais -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Tue Aug 19 07:57:45 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 19 Aug 2008 07:57:45 -0400 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: <20080819012323.GA855@phare.normalesup.org> References: <48A82134.6090501@american.edu> <20080817191435.GB692@phare.normalesup.org> <48A88A07.4050507@american.edu> <20080819012323.GA855@phare.normalesup.org> Message-ID: <48AAB539.7070906@american.edu> > On Sun, Aug 17, 2008 at 04:28:55PM -0400, Alan G Isaac wrote: >> That said, what kind of problems do you have in mind? Gael Varoquaux wrote: > wht I am most worried about is not being able to enter the > symbol, because I am in an editor I don't know, and the > symbol is not on my keyboard. This has happened to me more > than once with unicode. Well anyone who travels ends up dealing with strange keyboards, I suppose, but I would hate for the possibility that someone is in the odd situation of this happening while not having access to the net (and thus to Vim or emacs) dictate the decision on this. That would just not be very forward looking, especially when one can *in this case* always use ``dot`` instead. But we apparently agree that PEP 225 meets the need for a multiplication operator (with either ~ or @). Do you agree with Robert that *only* the multiplication operator is needed? (See my previous post.) Alan From aisaac at american.edu Tue Aug 19 08:11:32 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 19 Aug 2008 08:11:32 -0400 Subject: [Numpy-discussion] indexing (compared to matlab) In-Reply-To: References: <4FE21854-E95C-42A9-9A10-2EA545B50897@bryant.edu> <48A99DA6.1010005@american.edu> Message-ID: <48AAB874.50301@american.edu> Brian Blais wrote: >>> this should definitely be in the Numpy for Matlab users >>> page, http://www.scipy.org/NumPy_for_Matlab_Users, right >>> after the line: >>> Matlab Numpy Notes Brian Blais wrote: > I did put it in! (did it disappear or something?) I also modified the > previous line to say that the syntax a[0:3][:,4:9] gave read-only access. I see; I misunderstood your intended location. Hmmm. I think the partial duplication ends up being a good thing, so I'm not going to change the Notes. Thanks, Alan From ondrej at certik.cz Tue Aug 19 08:47:06 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Tue, 19 Aug 2008 14:47:06 +0200 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: <48AAB539.7070906@american.edu> References: <48A82134.6090501@american.edu> <20080817191435.GB692@phare.normalesup.org> <48A88A07.4050507@american.edu> <20080819012323.GA855@phare.normalesup.org> <48AAB539.7070906@american.edu> Message-ID: <85b5c3130808190547s5521087n656b60226b810f9b@mail.gmail.com> On Tue, Aug 19, 2008 at 1:57 PM, Alan G Isaac wrote: >> On Sun, Aug 17, 2008 at 04:28:55PM -0400, Alan G Isaac wrote: >>> That said, what kind of problems do you have in mind? > > Gael Varoquaux wrote: >> wht I am most worried about is not being able to enter the >> symbol, because I am in an editor I don't know, and the >> symbol is not on my keyboard. This has happened to me more >> than once with unicode. > > Well anyone who travels ends up dealing with strange > keyboards, I suppose, but I would hate for the possibility > that someone is in the odd situation of this happening > while not having access to the net (and thus to Vim or emacs) > dictate the decision on this. That would just not be > very forward looking, especially when one can *in this > case* always use ``dot`` instead. I absolutely agree with Gael here. Using anything besides lower (<128) ascii is imho not wise. Ondrej From markbak at gmail.com Tue Aug 19 11:31:03 2008 From: markbak at gmail.com (mark) Date: Tue, 19 Aug 2008 08:31:03 -0700 (PDT) Subject: [Numpy-discussion] a good polygon class? Message-ID: <48728505-c51a-4aaf-af22-6c1547d3f5a0@x35g2000hsb.googlegroups.com> Hello List - I am looking for a good polygon class. My main interest it to be able to figure out if a point is inside or outside the polygon, which can have any shape (well, as long as it is a polygon). Any suggestions? Thanks, Mark From bpederse at gmail.com Tue Aug 19 11:37:05 2008 From: bpederse at gmail.com (Brent Pedersen) Date: Tue, 19 Aug 2008 08:37:05 -0700 Subject: [Numpy-discussion] a good polygon class? In-Reply-To: <48728505-c51a-4aaf-af22-6c1547d3f5a0@x35g2000hsb.googlegroups.com> References: <48728505-c51a-4aaf-af22-6c1547d3f5a0@x35g2000hsb.googlegroups.com> Message-ID: hi, http://pypi.python.org/pypi/Shapely supports the array interface, and has all the geos geometry operations: http://gispython.org/shapely/manual.html#contains On Tue, Aug 19, 2008 at 8:31 AM, mark wrote: > Hello List - > > I am looking for a good polygon class. > > My main interest it to be able to figure out if a point is inside or > outside the polygon, which can have any shape (well, as long as it is > a polygon). > > Any suggestions? > > Thanks, Mark > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From pgmdevlist at gmail.com Tue Aug 19 11:36:32 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 19 Aug 2008 11:36:32 -0400 Subject: [Numpy-discussion] a good polygon class? In-Reply-To: <48728505-c51a-4aaf-af22-6c1547d3f5a0@x35g2000hsb.googlegroups.com> References: <48728505-c51a-4aaf-af22-6c1547d3f5a0@x35g2000hsb.googlegroups.com> Message-ID: <200808191136.32279.pgmdevlist@gmail.com> Mark, I find GDAL/OGR quite useful for manipulating geometries. OGR in particular allows you to manipulate vector data, and perform simple operations such as adding/subtracting. It's also easy to plug in matplotlib. http://www.gdal.org/ogr/ From stefan at sun.ac.za Tue Aug 19 12:49:27 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 19 Aug 2008 09:49:27 -0700 Subject: [Numpy-discussion] a good polygon class? In-Reply-To: <48728505-c51a-4aaf-af22-6c1547d3f5a0@x35g2000hsb.googlegroups.com> References: <48728505-c51a-4aaf-af22-6c1547d3f5a0@x35g2000hsb.googlegroups.com> Message-ID: <9457e7c80808190949t37a89cb6pb81920a1c205aa92@mail.gmail.com> Hi Mark 2008/8/19 mark : > Hello List - > > I am looking for a good polygon class. > > My main interest it to be able to figure out if a point is inside or > outside the polygon, which can have any shape (well, as long as it is > a polygon). > > Any suggestions? I have optimised Point-in-Polygon code here: http://mentat.za.net/source/pnpoly.tar.bz2 There are three versions: pure Python, ctypes and weave. A complete library (a NumPy port of Jorg Raedler's polygon lib) is available as part of my super-resolution library at http://mentat.za.net/supreme or more specifically http://bazaar.launchpad.net/~stefanv/supreme/main/files/180?file_id=Polygon1.16-20060502184912-b1a90b5a4870e7ba Regards St?fan From gael.varoquaux at normalesup.org Tue Aug 19 13:06:45 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 19 Aug 2008 19:06:45 +0200 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: <48AAB539.7070906@american.edu> References: <48A82134.6090501@american.edu> <20080817191435.GB692@phare.normalesup.org> <48A88A07.4050507@american.edu> <20080819012323.GA855@phare.normalesup.org> <48AAB539.7070906@american.edu> Message-ID: <20080819170645.GA26288@phare.normalesup.org> On Tue, Aug 19, 2008 at 07:57:45AM -0400, Alan G Isaac wrote: > But we apparently agree that PEP 225 meets the need for > a multiplication operator (with either ~ or @). Do you > agree with Robert that *only* the multiplication operator is > needed? (See my previous post.) No big opinion on that. Ga?l From loisel at temple.edu Tue Aug 19 18:19:40 2008 From: loisel at temple.edu (Sebastien Loisel) Date: Tue, 19 Aug 2008 18:19:40 -0400 Subject: [Numpy-discussion] Possible new multiplication operators for Python Message-ID: Hi, Just a quick note to let you guys know that I'm the one who most recently revived the discussion on python-dev. My needs are research and teaching in mathematics. So ideally, the programming notation should require only minimal explanations, and (equally importantly), it should be easy to set up your programming environment so that you can type your programs in. I like the PEP225 very much. I don't like @ operators because the @ is too large and I find it hard to read. ~ operators are fine (the dotted operators are fine too). I don't like unicode operators because they are hard to set up (I have no idea how), and also I'm wary of APL. Most of the unicode operators that were proposed on this list got rendered as indistinct blobs in my firefox. (Screenshot attached, I don't know if the mailing list will allow it -- if not, the picture is here: http://www.math.temple.edu/~loisel/unicode-operators.png) Sincerely, -- S?bastien Loisel -------------- next part -------------- A non-text attachment was scrubbed... Name: unicode-operators.png Type: image/png Size: 2881 bytes Desc: not available URL: From millman at berkeley.edu Tue Aug 19 21:36:41 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 19 Aug 2008 18:36:41 -0700 Subject: [Numpy-discussion] C-API change for 1.2 In-Reply-To: <320fb6e00808170533k572a1d49s630e6efabe4d9923@mail.gmail.com> References: <489CFF63.9000703@enthought.com> <48A69F3B.9040506@esrf.fr> <48A6F68D.3020006@enthought.com> <5b8d13220808161026t31368596ydf1b5bbf7b7e86a7@mail.gmail.com> <48A71200.6050000@esrf.fr> <320fb6e00808170533k572a1d49s630e6efabe4d9923@mail.gmail.com> Message-ID: On Sun, Aug 17, 2008 at 5:33 AM, Peter wrote: > I don't know if this constitutes "major opposition", but is keeping > the same C-API for NumPy 1.2 unchanged still a possibility? Please? Sorry I haven't commented on this yet; I have been busy and am still thinking about the issue. This note is just to say that we are taking this concern *very* seriously and will be discussing it in detail over the next few days at the SciPy conference. We will be looking more closely at the proposed code changes, discussing whether we really need to change the C-API, and if we do need to change the C-API (either now or in the future) how we should handle these sort of changes. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From klemm at phys.ethz.ch Wed Aug 20 08:13:21 2008 From: klemm at phys.ethz.ch (Hanno Klemm) Date: Wed, 20 Aug 2008 14:13:21 +0200 Subject: [Numpy-discussion] Problem with correlate? Message-ID: Hi All, after the discussion on numpy.correlate some time ago, regarding complex conjugation, etc. I today was pointed to yet another oddity, which I hope somebody could explain to me, as to why that's a feature, rather than a bug. I'm thoroughly confused by the following behaviour: In [29]: x = array([0.,0.,1, 0, 0]) In [30]: y = array([0, 0, 0, 1., 0, 0, 0]) In [31]: correlate(x,y) Out[31]: array([ 0., 1., 0.]) In [32]: correlate(x,y, mode='full') Out[32]: array([ 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.]) In [33]: y = array([0,0,1.,0,0]) In [34]: correlate(x,y,mode='full') Out[34]: array([ 0., 0., 0., 0., 1., 0., 0., 0., 0.]) So far, everything is allright and as expected. Now: In [35]: y1 = array([1,0,0,0,0]) In [36]: correlate(x,y1,mode='full') Out[36]: array([ 0., 0., 0., 0., 0., 0., 1., 0., 0.]) In [37]: y2 = array([1,0,0,0,0,0,0]) In [38]: correlate(x,y2,mode='full') Out[38]: array([ 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.]) Could somebody please help me understanding, why the one switches places? Furthermore, this behaviour does not seem to be very consistent, if we switch the x and y argument: In [39]: correlate(y2,x,mode='full') Out[39]: array([ 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.]) In [40]: correlate(y1,x,mode='full') Out[40]: array([ 0., 0., 1., 0., 0., 0., 0., 0., 0.]) The answer remains the same. Any help would be appreciated, the numpy version is 1.0.4. Best regards, Hanno -- Hanno Klemm klemm at phys.ethz.ch From lists at cheimes.de Wed Aug 20 09:01:00 2008 From: lists at cheimes.de (Christian Heimes) Date: Wed, 20 Aug 2008 15:01:00 +0200 Subject: [Numpy-discussion] global overloading of 1+1 -> MyClass(1, 1) In-Reply-To: <85b5c3130808181644q62df9e00u7b2932a14d28f838@mail.gmail.com> References: <85b5c3130808181301s4a96d2b9l6ca26fc553296e4c@mail.gmail.com> <85b5c3130808181303n3b59c3b7hf966105b7469025c@mail.gmail.com> <85b5c3130808181644q62df9e00u7b2932a14d28f838@mail.gmail.com> Message-ID: Ondrej Certik wrote: > Are we able to provide an actual patch to Python that implements this? > If so, then I am. > Imho the proposal should come with an actual patch, otherwise it's > difficult to judge it. Your better off with writing a PEP first. In order to implement the proposal you've to make changes to the parser and tokenizer - perhaps to the grammar, too. The code is rather complex and tricky - and not very well documented. Christian From eike.welk at gmx.net Wed Aug 20 10:31:36 2008 From: eike.welk at gmx.net (Eike Welk) Date: Wed, 20 Aug 2008 16:31:36 +0200 Subject: [Numpy-discussion] Possible new multiplication operators for Python In-Reply-To: <48AA0661.1030104@american.edu> References: <3d375d730808181537t6bf2dfc5jb23dde9f0e1e59cd@mail.gmail.com> <48AA0661.1030104@american.edu> Message-ID: <200808201631.36655.eike.welk@gmx.net> On Tuesday 19 August 2008, Alan G Isaac wrote: > > Proposal 1: PEP 225, but *just* for multiplication. > Either ~* as in the PEP or @* (which I prefer). > (This looks simplest.) > > Proposal 2: PEP 225 > (but maybe using @ instead of ~). > > Proposal 3: use of a unicode character, > perhaps ? since it is in the Latin1 > extension of ASCII. > > Proposal 4: user defined operators, perhaps > as described in PEP 225. > In this case, @dot would be a natural. > (This looks most complex but also most powerful.) > But is A @dot B really better than say A.dot(B)? > Maybe. > > Propsal 5: use the existing language, either > using __call__ as in Tim's proposal or adding > a ``dot`` method. > > Did I miss any? You could abuse Python's current capabilities to create something that looks like proposal 4: C = A *dot* B You would need a helper object named "dot". This has been proposed on the Python-Dev list. I find this kinds of pseudo operator a bit clunky. It might be quite confusing to beginners because really the the "*" operator is invoked twice. The pseudo operator has one nice aspect: It can take additional arguments, which could be nice for the array-of-matrices application. mmul = dot((1,0),(0,1)) C = A *mmul* B -- I'd like PEP 225 to be implemented. The new operators are kind of elegant, and they don't introduce any new concepts. They just introduce a bunch of new special functions. Regards, Eike. From doutriaux1 at llnl.gov Wed Aug 20 11:04:55 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Wed, 20 Aug 2008 08:04:55 -0700 Subject: [Numpy-discussion] numpy build fail under cygwin (Vista) Message-ID: <48AC3297.8080800@llnl.gov> Hi both trunk and 1.1.1 fail to build under cygwin vista (latest version) I'm copy/pasting the end of the log C. copying numpy/doc/reference/__init__.py -> build/lib.cygwin-1.5.25-i686-2.5/numpy/doc/reference running build_ext customize UnixCCompiler customize UnixCCompiler using build_ext customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using build_ext building 'numpy.core.multiarray' extension compiling C sources C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes creating build/temp.cygwin-1.5.25-i686-2.5 creating build/temp.cygwin-1.5.25-i686-2.5/numpy creating build/temp.cygwin-1.5.25-i686-2.5/numpy/core creating build/temp.cygwin-1.5.25-i686-2.5/numpy/core/src compile options: '-g -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -I/usr/include/python2.5 -c' gcc: numpy/core/src/multiarraymodule.c gcc -shared -Wl,--enable-auto-image-base -g build/temp.cygwin-1.5.25-i686-2.5/numpy/core/src/multiarraymodule.o -L/usr/lib/python2.5/config -lpython2.5 -o build/lib.cygwin-1.5.25-i686-2.5/numpy/core/multiarray.dll building 'numpy.core.umath' extension compiling C sources C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes creating build/temp.cygwin-1.5.25-i686-2.5/build creating build/temp.cygwin-1.5.25-i686-2.5/build/src.cygwin-1.5.25-i686-2.5 creating build/temp.cygwin-1.5.25-i686-2.5/build/src.cygwin-1.5.25-i686-2.5/numpy creating build/temp.cygwin-1.5.25-i686-2.5/build/src.cygwin-1.5.25-i686-2.5/numpy/core creating build/temp.cygwin-1.5.25-i686-2.5/build/src.cygwin-1.5.25-i686-2.5/numpy/core/src compile options: '-g -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -I/usr/include/python2.5 -c' gcc: build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/umathmodule.c /cygdrive/c/Users/CHARLE~1/AppData/Local/Temp/ccib5Stl.s: Assembler messages: /cygdrive/c/Users/CHARLE~1/AppData/Local/Temp/ccib5Stl.s:72843: Error: suffix or operands invalid for `fnstsw' /cygdrive/c/Users/CHARLE~1/AppData/Local/Temp/ccib5Stl.s:73098: Error: suffix or operands invalid for `fnstsw' /cygdrive/c/Users/CHARLE~1/AppData/Local/Temp/ccib5Stl.s: Assembler messages: /cygdrive/c/Users/CHARLE~1/AppData/Local/Temp/ccib5Stl.s:72843: Error: suffix or operands invalid for `fnstsw' /cygdrive/c/Users/CHARLE~1/AppData/Local/Temp/ccib5Stl.s:73098: Error: suffix or operands invalid for `fnstsw' error: Command "gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -g -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -I/usr/include/python2.5 -c build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/umathmodule.c -o build/temp.cygwin-1.5.25-i686-2.5/build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/umathmodule.o" failed with exit status 1 From cournape at gmail.com Wed Aug 20 11:25:06 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 20 Aug 2008 08:25:06 -0700 Subject: [Numpy-discussion] numpy build fail under cygwin (Vista) In-Reply-To: <48AC3297.8080800@llnl.gov> References: <48AC3297.8080800@llnl.gov> Message-ID: <5b8d13220808200825s5dcf9dd8qf16fcfb094673004@mail.gmail.com> On Wed, Aug 20, 2008 at 8:04 AM, Charles Doutriaux wrote: > Hi both trunk and 1.1.1 fail to build under cygwin vista (latest version) > > I'm copy/pasting the end of the log > It is a cygwin bug: http://www.mail-archive.com/numpy-discussion at scipy.org/msg10051.html cheers, David From doutriaux1 at llnl.gov Wed Aug 20 12:37:18 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Wed, 20 Aug 2008 09:37:18 -0700 Subject: [Numpy-discussion] numpy build fail under cygwin (Vista) In-Reply-To: <5b8d13220808200825s5dcf9dd8qf16fcfb094673004@mail.gmail.com> References: <48AC3297.8080800@llnl.gov> <5b8d13220808200825s5dcf9dd8qf16fcfb094673004@mail.gmail.com> Message-ID: <48AC483E.4010005@llnl.gov> Thx David, Is there any plans on applying the suggested fix into numpy ? C. David Cournapeau wrote: > On Wed, Aug 20, 2008 at 8:04 AM, Charles Doutriaux wrote: > >> Hi both trunk and 1.1.1 fail to build under cygwin vista (latest version) >> >> I'm copy/pasting the end of the log >> >> > > It is a cygwin bug: > > http:// www. mail-archive.com/numpy-discussion at scipy.org/msg10051.html > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http:// projects.scipy.org/mailman/listinfo/numpy-discussion > > > From dagss at student.matnat.uio.no Wed Aug 20 16:18:04 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 20 Aug 2008 22:18:04 +0200 Subject: [Numpy-discussion] The "NumPy" Cython release + tutorial Message-ID: <48AC7BFC.8060309@student.matnat.uio.no> Cython just had a release, and amongst the new features are efficient NumPy array indexing for integers, real floats and Python objects. You can get it at http://cython.org For those new to Cython, I've written a tutorial specifically targeted for NumPy users here: http://wiki.cython.org/tutorials/numpy ...documenting the process of getting from 1.8 seconds to 6 milliseconds for the same data by adding typing in a sample for-loop-intensive code snippet. The last section of that tutorial (Efficient indexing) should be useful for those who know Cython as well. -- Dag Sverre From cournape at gmail.com Wed Aug 20 16:21:32 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 20 Aug 2008 13:21:32 -0700 Subject: [Numpy-discussion] numpy build fail under cygwin (Vista) In-Reply-To: <48AC483E.4010005@llnl.gov> References: <48AC3297.8080800@llnl.gov> <5b8d13220808200825s5dcf9dd8qf16fcfb094673004@mail.gmail.com> <48AC483E.4010005@llnl.gov> Message-ID: <5b8d13220808201321k49e5c7abgc149b285445f36fc@mail.gmail.com> On Wed, Aug 20, 2008 at 9:37 AM, Charles Doutriaux wrote: > Thx David, > > Is there any plans on applying the suggested fix into numpy ? Ha, I was not aware we used any asm in numpy, which is why I did not think it could be a numpy problem. I commented on the ticket, and there is definitely something wrong in the code, but I don't get why it worked before (since the asm instruction take a 2 bytes, and we feed them with 4 bytes values). David From stefan at sun.ac.za Wed Aug 20 17:49:14 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 20 Aug 2008 14:49:14 -0700 Subject: [Numpy-discussion] Problem with correlate? In-Reply-To: References: Message-ID: <9457e7c80808201449g2f356c0bg4fb7a93e04050331@mail.gmail.com> 2008/8/20 Hanno Klemm : > In [29]: x = array([0.,0.,1, 0, 0]) > In [35]: y1 = array([1,0,0,0,0]) > > In [36]: correlate(x,y1,mode='full') > Out[36]: array([ 0., 0., 0., 0., 0., 0., 1., 0., 0.]) That doesn't look right. Under r5661: In [60]: np.convolve([0, 0, 1, 0, 0], [1, 0, 0, 0, 0], mode='f') Out[60]: array([0, 0, 1, 0, 0, 0, 0, 0, 0]) Regards St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Wed Aug 20 18:17:18 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 20 Aug 2008 15:17:18 -0700 Subject: [Numpy-discussion] numpy build fail under cygwin (Vista) In-Reply-To: <5b8d13220808201321k49e5c7abgc149b285445f36fc@mail.gmail.com> References: <48AC3297.8080800@llnl.gov> <5b8d13220808200825s5dcf9dd8qf16fcfb094673004@mail.gmail.com> <48AC483E.4010005@llnl.gov> <5b8d13220808201321k49e5c7abgc149b285445f36fc@mail.gmail.com> Message-ID: <5b8d13220808201517w4dceb751w4e6ba5c5cc489ecb@mail.gmail.com> On Wed, Aug 20, 2008 at 1:21 PM, David Cournapeau wrote: > On Wed, Aug 20, 2008 at 9:37 AM, Charles Doutriaux wrote: >> Thx David, >> >> Is there any plans on applying the suggested fix into numpy ? > > Ha, I was not aware we used any asm in numpy, which is why I did not > think it could be a numpy problem. I commented on the ticket, and > there is definitely something wrong in the code, but I don't get why > it worked before (since the asm instruction take a 2 bytes, and we > feed them with 4 bytes values). Ok, I applied one change to fenv.c/fenv.h which seems to solve the problem, but I have no idea why: I still don't understand why using an int with an instruction expecting a word can work, but I don't know much about gcc inline asm syntax anyway. The change is taken from FreeBSD, from which the original file is coming from, and it looks like we should do the change anyway. Another problem is that I can't make nose test find any test under cygwin (I applied the change to both trunk and 1.1.x branches, and on 1.1.x, all but one tests pass, the failure not being related to this). cheers, David From william at resolversystems.com Thu Aug 21 06:16:33 2008 From: william at resolversystems.com (William Reade) Date: Thu, 21 Aug 2008 11:16:33 +0100 Subject: [Numpy-discussion] numpy.core.numerictypes questions Message-ID: <48AD4081.1050105@resolversystems.com> Hi I hope this is the right place for this question... if not, I apologise, and ask only that you please direct me somewhere more appropriate. Line 532 of numerictypes.py reads "_unicodesize = array('u','U1').itemsize", but _unicodesize itself does not appear to be referenced anywhere else. My questions are: * Does this line have any use? * If not, would it be possible to remove it from a future release? I ask because I'm working on a project to get CPython extensions (specifically numpy) working with IronPython, and this line is blocking me. If I comment it out, I can actually import numpy and use some features, but I'm not sure how important unicode arrays are: if they're not widely used, I'd be keen to ignore them for a while and work on, say, floating-point arrays. Best William From animator333 at yahoo.com Thu Aug 21 13:40:47 2008 From: animator333 at yahoo.com (Prashant Saxena) Date: Thu, 21 Aug 2008 10:40:47 -0700 (PDT) Subject: [Numpy-discussion] first post, simple questions Message-ID: <149208.18754.qm@web33401.mail.mud.yahoo.com> Hi, numpy rocks!!! import numpy linsp = numpy.linspace red = linsp(0, 255, 50) green = linsp(125, 150, 50) blue = linsp(175, 255, 50) array's elements are float. How do I convert them into integer? I need to build a new array from red, green, blue. like this: [[ red[0], green[0], blue[0] ], [ red[1], green[1], blue[1] ], [ red[2], green[3], blue[3] ], .... .... [ red[49], green[49], blue[49] ], ] how do I create such an array? Are there any issues regarding compilation of simple python scripts using numpy functions and cython? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Thu Aug 21 13:55:57 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 21 Aug 2008 10:55:57 -0700 Subject: [Numpy-discussion] first post, simple questions In-Reply-To: <149208.18754.qm@web33401.mail.mud.yahoo.com> References: <149208.18754.qm@web33401.mail.mud.yahoo.com> Message-ID: On Thu, Aug 21, 2008 at 10:40 AM, Prashant Saxena wrote: > Hi, > > numpy rocks!!! > > import numpy > linsp = numpy.linspace > red = linsp(0, 255, 50) > green = linsp(125, 150, 50) > blue = linsp(175, 255, 50) > > array's elements are float. How do I convert them into integer? > > I need to build a new array from red, green, blue. like this: > > [[ red[0], green[0], blue[0] ], > [ red[1], green[1], blue[1] ], > [ red[2], green[3], blue[3] ], > .... > .... > [ red[49], green[49], blue[49] ], > ] > > how do I create such an array? Here's one way: np.c_[red.reshape(-1,1), green.reshape(-1,1), blue.reshape(-1,1)] From pgmdevlist at gmail.com Thu Aug 21 13:58:12 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 21 Aug 2008 13:58:12 -0400 Subject: [Numpy-discussion] first post, simple questions In-Reply-To: <149208.18754.qm@web33401.mail.mud.yahoo.com> References: <149208.18754.qm@web33401.mail.mud.yahoo.com> Message-ID: <200808211358.12979.pgmdevlist@gmail.com> On Thursday 21 August 2008 13:40:47 Prashant Saxena wrote: > Hi, > > numpy rocks!!! > > import numpy > linsp = numpy.linspace > red = linsp(0, 255, 50) > green = linsp(125, 150, 50) > blue = linsp(175, 255, 50) > > array's elements are float. How do I convert them into integer? Use .astype(int) red = linsp(0, 255, 50).astype(int) > I need to build a new array from red, green, blue. like this: > > [[ red[0], green[0], blue[0] ], numpy.column_stack((red,green,blue)) From zachary.pincus at yale.edu Thu Aug 21 14:12:24 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 21 Aug 2008 14:12:24 -0400 Subject: [Numpy-discussion] first post, simple questions In-Reply-To: <149208.18754.qm@web33401.mail.mud.yahoo.com> References: <149208.18754.qm@web33401.mail.mud.yahoo.com> Message-ID: <69A36E45-03E7-4E5F-B4DF-43167F3CAD67@yale.edu> > import numpy > linsp = numpy.linspace > red = linsp(0, 255, 50) > green = linsp(125, 150, 50) > blue = linsp(175, 255, 50) > > array's elements are float. How do I convert them into integer? > > I need to build a new array from red, green, blue. like this: > > [[ red[0], green[0], blue[0] ], > [ red[1], green[1], blue[1] ], > [ red[2], green[3], blue[3] ], > .... > .... > [ red[49], green[49], blue[49] ], > ] > > how do I create such an array? We can convert them to integer and build the array at the same time: colors = numpy.empty((50, 3), dtype=numpy.uint8) colors[:,0] = numpy.linspace(0, 255, 50) colors[:,1] = numpy.linspace(125, 150, 50) colors[:,2] = numpy.linspace(175, 255, 50) Or we could use a fancy-dtype record array: colors = numpy.empty(50, dtype=[('r', numpy.uint8), ('g', numpy.uint8), ('b', numpy.uint8)]) colors['r'] = numpy.linspace(0, 255, 50) colors['g'] = numpy.linspace(125, 150, 50) colors['b'] = numpy.linspace(175, 255, 50) To see the array look like the old non-fancy type, use: colors.view((numpy.uint8, 3)) Zach From rbastian at free.fr Thu Aug 21 13:53:38 2008 From: rbastian at free.fr (R. Bastian) Date: Thu, 21 Aug 2008 19:53:38 +0200 Subject: [Numpy-discussion] first post, simple questions In-Reply-To: <149208.18754.qm@web33401.mail.mud.yahoo.com> References: <149208.18754.qm@web33401.mail.mud.yahoo.com> Message-ID: <20080821195338.fe139add.rbastian@free.fr> On Thu, 21 Aug 2008 10:40:47 -0700 (PDT) Prashant Saxena wrote: > Hi, > > numpy rocks!!! > > import numpy > linsp = numpy.linspace > red = linsp(0, 255, 50) > green = linsp(125, 150, 50) > blue = linsp(175, 255, 50) > > array's elements are float. How do I convert them into integer? > > I need to build a new array from red, green, blue. like this: > > [[ red[0], green[0], blue[0] ], > [ red[1], green[1], blue[1] ], > [ red[2], green[3], blue[3] ], > .... > .... > [ red[49], green[49], blue[49] ], > ] > > how do I create such an array? > > Are there any issues regarding compilation of simple python scripts using numpy functions and cython? > > Thanks 1. read the doc 2. use 'astype' and 'array([.... tra la la ... ]) -- R. Bastian www.musiques-rb.org http://www.centrotemporeale.it/index.php?lang=it&sez=news&id=20080926 From michael.abshoff at googlemail.com Thu Aug 21 17:15:06 2008 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Thu, 21 Aug 2008 14:15:06 -0700 Subject: [Numpy-discussion] Revisiting numpy/scipy on 64 bit OSX Message-ID: <48ADDADA.1040109@gmail.com> Hi, I have been playing with 64 bit numpy/scipy on OSX 10.5 Intel again. I thought everything worked after the last time we discussed this, but I noticed that I had Scipy import failures in for example optimize since the gfortran I used creates 32 bit code per default. So I build a gcc 4.2.4 on OSX, then build python in 64 bit mode (fixing configure.in since Python assumes flags only available with the Apple gcc on Darwin), added a wrapper around gfortran that injects a "-m64" compile flag. Then I checkout out and build numpy (r5671), scipy (r4661) and the current nose release and ran the tests. I had some more failures than I expected (details below). Any thoughts on the failures? I am about to build ATLAS and a couple other dependencies on that box. William Stein is at Scipy 08 and there seems to be at least some interest in 64 bit OSX binaries. If anyone wants one let me know and I can put one together once Scipy 0.7 and Numpy 1.1.2 are out. Cheers, Michael numpy failed tests: **************************************************************** File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests/test_polynomial.py", line 29, in test_polynomial Failed example: p / q Expected: (poly1d([ 0.33333333]), poly1d([ 1.33333333, 2.66666667])) Got: (poly1d([ 0.333]), poly1d([ 1.333, 2.667])) ********************************************************************** File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests/test_polynomial.py", line 51, in test_polynomial Failed example: p.integ() Expected: poly1d([ 0.33333333, 1. , 3. , 0. ]) Got: poly1d([ 0.333, 1. , 3. , 0. ]) **********************************************************************File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests/test_polynomial.py", line 53, in test_polynomialFailed example: p.integ(1) Expected: poly1d([ 0.33333333, 1. , 3. , 0. ]) Got: poly1d([ 0.333, 1. , 3. , 0. ]) ********************************************************************** File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests/test_polynomial.py", line 55, in test_polynomial Failed example: p.integ(5) Expected: poly1d([ 0.00039683, 0.00277778, 0.025 , 0. , 0. , 0. , 0. , 0. ]) Got: poly1d([ 0. , 0.003, 0.025, 0. , 0. , 0. , 0. , 0. ]) ******************************** File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests/test_ufunclike.py", line 48, in test_ufunclike Failed example: U.log2(a) Expected: array([ 2.169925 , 1.20163386, 2.70043972]) Got: array([ 2.17 , 1.202, 2.7 ]) ********************************************************************** File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests/test_ufunclike.py", line 53, in test_ufunclike Failed example: U.log2(a, y) Expected: array([ 2.169925 , 1.20163386, 2.70043972]) Got: array([ 2.17 , 1.202, 2.7 ]) ********************************************************************** File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests/test_ufunclike.py", line 55, in test_ufunclike Failed example: y Expected: array([ 2.169925 , 1.20163386, 2.70043972]) Got: array([ 2.17 , 1.202, 2.7 ]) ====================================================================== FAIL: test_umath.TestComplexFunctions.test_it ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/nose/case.py", line 182, in runTest self.test(*self.arg) File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", line 196, in test_it assert_almost_equal(fz.real, fr, err_msg='real part %s'%f) File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/testing/utils.py", line 207, in assert_almost_equal assert round(abs(desired - actual),decimal) == 0, msg AssertionError: Items are not equal: real part ACTUAL: -0.3010299956639812 DESIRED: 0.0 scipy: ====================================================================== ERROR: test_nonsymmetric_modes (test_arpack.TestEigenNonSymmetric) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 204, in test_nonsymmetric_modes self.eval_evec(m,typ,k,which) File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 186, in eval_evec eval,evec=eigen(a,k,which=which,**kwds) File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/linalg/eigen/arpack/arpack.py", line 220, in eigen raise RuntimeError("Error info=%d in arpack"%info) RuntimeError: Error info=-8 in arpack ====================================================================== ERROR: test_starting_vector (test_arpack.TestEigenNonSymmetric) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 214, in test_starting_vector self.eval_evec(self.symmetric[0],typ,k,which='LM',v0=v0) File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 186, in eval_evec eval,evec=eigen(a,k,which=which,**kwds) File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/linalg/eigen/arpack/arpack.py", line 220, in eigen raise RuntimeError("Error info=%d in arpack"%info) RuntimeError: Error info=-8 in arpack ====================================================================== FAIL: test_implicit (test_odr.TestODR) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/odr/tests/test_odr.py", line 118, in test_implicit 2.0778813389755596e-03]]), File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/testing/utils.py", line 304, in assert_array_almost_equal header='Arrays are not almost equal') File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/testing/utils.py", line 289, in assert_array_compare assert cond, msg AssertionError: Arrays are not almost equal (mismatch 4.0%) x: array([[ 2.10892628e+00, -1.94376722e+00, 7.02635228e-02, -4.71752493e-02, 5.25155371e-02], [ -1.94376722e+00, 2.04814914e+00, -6.16004809e-02,... y: array([[ 2.10892746e+00, -1.94376864e+00, 7.02635509e-02, -4.71752674e-02, 5.25155759e-02], [ -1.94376864e+00, 2.04815092e+00, -6.16005159e-02,... From Joris.DeRidder at ster.kuleuven.be Thu Aug 21 19:34:53 2008 From: Joris.DeRidder at ster.kuleuven.be (Joris De Ridder) Date: Fri, 22 Aug 2008 01:34:53 +0200 Subject: [Numpy-discussion] The "NumPy" Cython release + tutorial In-Reply-To: <48AC7BFC.8060309@student.matnat.uio.no> References: <48AC7BFC.8060309@student.matnat.uio.no> Message-ID: <824F2905-5AFB-40D6-B0BD-54DF24CCBE26@ster.kuleuven.be> On 20 Aug 2008, at 22:18 , Dag Sverre Seljebotn wrote: > Cython just had a release, and amongst the new features are efficient > NumPy array indexing for integers, real floats and Python objects. > > You can get it at http://cython.org > > For those new to Cython, I've written a tutorial specifically targeted > for NumPy users here: > > http://wiki.cython.org/tutorials/numpy > > ...documenting the process of getting from 1.8 seconds to 6 > milliseconds > for the same data by adding typing in a sample for-loop-intensive code > snippet. The last section of that tutorial (Efficient indexing) should > be useful for those who know Cython as well. This looks really great! Thanks! Joris Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From stefan at sun.ac.za Thu Aug 21 20:13:35 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 21 Aug 2008 17:13:35 -0700 Subject: [Numpy-discussion] Revisiting numpy/scipy on 64 bit OSX In-Reply-To: <48ADDADA.1040109@gmail.com> References: <48ADDADA.1040109@gmail.com> Message-ID: <9457e7c80808211713y2a19921al1bd73c3946676986@mail.gmail.com> 2008/8/21 Michael Abshoff : > William Stein is at Scipy 08 and there seems to be at least some > interest in 64 bit OSX binaries. If anyone wants one let me know and I > can put one together once Scipy 0.7 and Numpy 1.1.2 are out. There is still interest in having a 64-bit OSX buildbot, which would allow us to support this platform. Would you still be able to host it, as per our previous discussion? Regards St?fan From michael.abshoff at googlemail.com Thu Aug 21 20:20:20 2008 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Thu, 21 Aug 2008 17:20:20 -0700 Subject: [Numpy-discussion] Revisiting numpy/scipy on 64 bit OSX In-Reply-To: <9457e7c80808211713y2a19921al1bd73c3946676986@mail.gmail.com> References: <48ADDADA.1040109@gmail.com> <9457e7c80808211713y2a19921al1bd73c3946676986@mail.gmail.com> Message-ID: <48AE0644.4040705@gmail.com> St?fan van der Walt wrote: > 2008/8/21 Michael Abshoff : >> William Stein is at Scipy 08 and there seems to be at least some >> interest in 64 bit OSX binaries. If anyone wants one let me know and I >> can put one together once Scipy 0.7 and Numpy 1.1.2 are out. > > There is still interest in having a 64-bit OSX buildbot, which would > allow us to support this platform. Would you still be able to host > it, as per our previous discussion? Yep, I really got distracted the last months doing other things, but now the numpy and scipy upgrades in Sage are high priority since it fixes a number of build issues and segfaults on Solaris and OSX 64 bit :) > Regards > St?fan Cheers, Michael > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From millman at berkeley.edu Thu Aug 21 20:29:41 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Thu, 21 Aug 2008 17:29:41 -0700 Subject: [Numpy-discussion] The "NumPy" Cython release + tutorial In-Reply-To: <48AC7BFC.8060309@student.matnat.uio.no> References: <48AC7BFC.8060309@student.matnat.uio.no> Message-ID: On Wed, Aug 20, 2008 at 1:18 PM, Dag Sverre Seljebotn wrote: > Cython just had a release, and amongst the new features are efficient > NumPy array indexing for integers, real floats and Python objects. Excellent. Thanks for working on this. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From stefan at sun.ac.za Thu Aug 21 20:33:24 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 21 Aug 2008 17:33:24 -0700 Subject: [Numpy-discussion] Revisiting numpy/scipy on 64 bit OSX In-Reply-To: <48AE0644.4040705@gmail.com> References: <48ADDADA.1040109@gmail.com> <9457e7c80808211713y2a19921al1bd73c3946676986@mail.gmail.com> <48AE0644.4040705@gmail.com> Message-ID: <9457e7c80808211733v41771765l3c1889e9d2e2865f@mail.gmail.com> 2008/8/21 Michael Abshoff : > St?fan van der Walt wrote: >> 2008/8/21 Michael Abshoff : >>> William Stein is at Scipy 08 and there seems to be at least some >>> interest in 64 bit OSX binaries. If anyone wants one let me know and I >>> can put one together once Scipy 0.7 and Numpy 1.1.2 are out. >> >> There is still interest in having a 64-bit OSX buildbot, which would >> allow us to support this platform. Would you still be able to host >> it, as per our previous discussion? > > Yep, I really got distracted the last months doing other things, but now > the numpy and scipy upgrades in Sage are high priority since it fixes a > number of build issues and segfaults on Solaris and OSX 64 bit :) Would you like to host one or two buildbots? I currently have "Solaris 10 SPARC" reserved for you. Would you like another one for MacOSX 64? Cheers St?fan From stefan at sun.ac.za Thu Aug 21 20:35:39 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 21 Aug 2008 17:35:39 -0700 Subject: [Numpy-discussion] Revisiting numpy/scipy on 64 bit OSX In-Reply-To: <9457e7c80808211733v41771765l3c1889e9d2e2865f@mail.gmail.com> References: <48ADDADA.1040109@gmail.com> <9457e7c80808211713y2a19921al1bd73c3946676986@mail.gmail.com> <48AE0644.4040705@gmail.com> <9457e7c80808211733v41771765l3c1889e9d2e2865f@mail.gmail.com> Message-ID: <9457e7c80808211735i2ffac3eat3098965f6b1030ba@mail.gmail.com> 2008/8/21 St?fan van der Walt : > I currently have "Solaris 10 SPARC" reserved for you. Would you like > another one for MacOSX 64? I have re-activated the configuration for Solaris 10 SPARC. Regards St?fan From fperez.net at gmail.com Fri Aug 22 02:52:45 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 21 Aug 2008 23:52:45 -0700 Subject: [Numpy-discussion] The "NumPy" Cython release + tutorial In-Reply-To: <48AC7BFC.8060309@student.matnat.uio.no> References: <48AC7BFC.8060309@student.matnat.uio.no> Message-ID: Hey Dag, On Wed, Aug 20, 2008 at 1:18 PM, Dag Sverre Seljebotn wrote: > Cython just had a release, and amongst the new features are efficient > NumPy array indexing for integers, real floats and Python objects. > > You can get it at http://cython.org > > For those new to Cython, I've written a tutorial specifically targeted > for NumPy users here: > > http://wiki.cython.org/tutorials/numpy > > ...documenting the process of getting from 1.8 seconds to 6 milliseconds > for the same data by adding typing in a sample for-loop-intensive code > snippet. The last section of that tutorial (Efficient indexing) should > be useful for those who know Cython as well. It's a bummer you couldn't be at SciPy'08, but there's a lot of interest and excitement about this. Thanks not only for the work, but also for documenting it well. Hopefully you'll be able to continue improving it. Best, f From klemm at phys.ethz.ch Fri Aug 22 03:44:05 2008 From: klemm at phys.ethz.ch (Hanno Klemm) Date: Fri, 22 Aug 2008 09:44:05 +0200 Subject: [Numpy-discussion] Problem with correlate? In-Reply-To: <9457e7c80808201449g2f356c0bg4fb7a93e04050331@mail.gmail.com> References: , Message-ID: Hi Stefan, yes, indeed, that's what I thought. This result is odd. Has correlate been changed since version 1.0.4, or should I submit this as a bug? Best regards, Hanno St?fan van der Walt said: > ------=_Part_25307_10322093.1219268954678 > Content-Type: text/plain; charset=ISO-8859-1 > Content-Transfer-Encoding: quoted-printable > Content-Disposition: inline > > 2008/8/20 Hanno Klemm : > > In [29]: x =3D array([0.,0.,1, 0, 0]) > > In [35]: y1 =3D array([1,0,0,0,0]) > > > > In [36]: correlate(x,y1,mode=3D'full') > > Out[36]: array([ 0., 0., 0., 0., 0., 0., 1., 0., 0.]) > > That doesn't look right. Under r5661: > > In [60]: np.convolve([0, 0, 1, 0, 0], [1, 0, 0, 0, 0], mode=3D'f') > Out[60]: array([0, 0, 1, 0, 0, 0, 0, 0, 0]) > > Regards > St=E9fan > > ------=_Part_25307_10322093.1219268954678 > Content-Type: text/html; charset=ISO-8859-1 > Content-Transfer-Encoding: quoted-printable > Content-Disposition: inline > >
2008/8/20 Hanno Klemm < z.ch">klemm at phys.ethz.ch>:
> In [29]: x =3D array([0.,0.,1, 0,= > 0])
> In [35]: y1 =3D array([1,0,0,0,0])
>
> In [36]: co= > rrelate(x,y1,mode=3D'full')
> > Out[36]: array([ 0.,  0.,  0.,  0.,  0.,  0.,= >  1.,  0.,  0.])

That doesn't look right.  U= > nder r5661:

In [60]: np.convolve([0, 0, 1, 0, 0], [1, 0, 0, 0, 0], m= > ode=3D'f')
Out[60]: array([0, 0, 1, 0, 0, 0, 0, 0, 0])
>
Regards
St=E9fan

> > ------=_Part_25307_10322093.1219268954678-- > -- Hanno Klemm klemm at phys.ethz.ch From stefan at sun.ac.za Fri Aug 22 04:03:46 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 22 Aug 2008 01:03:46 -0700 Subject: [Numpy-discussion] Problem with correlate? In-Reply-To: References: <9457e7c80808201449g2f356c0bg4fb7a93e04050331@mail.gmail.com> Message-ID: <9457e7c80808220103x52cb4c8fgf50bfb10912ab11e@mail.gmail.com> Hi Hanno 2008/8/22 Hanno Klemm : > yes, indeed, that's what I thought. This result is odd. Has correlate > been changed since version 1.0.4, or should I submit this as a bug? Is there any way that you could try out the latest release on your machine and see if it solves your problem? We probably won't be releasing bug-fixes on 1.0.4, but if it exists in 1.1 we'll still address it. I'm not aware of any changes, but I may simply have missed it. Regards St?fan From stefan at sun.ac.za Fri Aug 22 04:26:13 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 22 Aug 2008 01:26:13 -0700 Subject: [Numpy-discussion] [SciPy08] Documentation BoF Message-ID: <9457e7c80808220126u765e5b7vc1a0a15ee4a28c11@mail.gmail.com> Hi all, This is my personal recollection of the documentation BoF. Feel free to comment or correct the text below. Regards St?fan Summary of the Documentation Birds-of-a-Feather Session ======================================================= Topics proposed for discussion ------------------------------ - Path towards getting SciPy documented using the online documentation editor - Intersphinx: linking Sphinx-using projects - Culture / methodology: what do we expect from code accepted into NumPy/SciPy? - Tools for reading documentation (Help 2.0) - The complete SciPy book -- how to get there - NumPy Documentation Standard: is it useful beyond NumPy? Conclusions reached ------------------- Documentation format ```````````````````` - Question: could we standardise the descriptions of data-types between docs of different projects? They should overlap to some extent. - If we could agree on some common standard (or at least a common subset of a format) across projects, IPython could parse and display docstrings with special markup. - Matplotlib should be involved in this conversation. They have already written a vast amount of documentation, and their needs may be different to those of the SciPy projects. Formal requirements ``````````````````` For functions going into NumPy and SciPy, we have a certain minimum documentation requirements. However, we'd rather take a somewhat liberal, open approach, and accept code with somwhat inferior docstrings, working with the author to improve them. This way, we remain accessible as a community, while striving to improve the quality of the code base. Marketing SciPy ``````````````` The current entry point for SciPy on the web (http://www.scipy.org) may be confusing to beginners. The web page should be redesigned, in a similar fashion as the new Sage and code.enthought.com pages, linking to the messy wiki behind it in a consistent fashion. Joe Harrington may be able to hire someone to work on this important aspect of SciPy marketing. Cross-project documentation searching ````````````````````````````````````` Ideally, one should be able to perform a documentation search across several projects. In order to achieve this, we should negotiate a common convention for installing Sphinx-generated documentation into a standard location. A javascript-backed webpage can then be provided to do a search, using the json indices, or some other means. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Fri Aug 22 06:19:05 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 22 Aug 2008 05:19:05 -0500 Subject: [Numpy-discussion] numpy.core.numerictypes questions In-Reply-To: <48AD4081.1050105@resolversystems.com> References: <48AD4081.1050105@resolversystems.com> Message-ID: <48AE9299.9030206@enthought.com> William Reade wrote: > Hi > > I hope this is the right place for this question... if not, I apologise, > and ask only that you please direct me somewhere more appropriate. > > Line 532 of numerictypes.py reads "_unicodesize = > array('u','U1').itemsize", but _unicodesize itself does not appear to be > referenced anywhere else. My questions are: > > * Does this line have any use? > > * If not, would it be possible to remove it from a future release? > I don't know the answer to these questions. > I ask because I'm working on a project to get CPython extensions > (specifically numpy) working with IronPython, and this line is blocking > me. If I comment it out, I can actually import numpy and use some > features, but I'm not sure how important unicode arrays are: if they're > not widely used, I'd be keen to ignore them for a while and work on, > say, floating-point arrays. > I would say that unicode arrays are rare and I would definitely work on floating-point first. Any chance we can see some of your work, by the way. -Travis From william at resolversystems.com Fri Aug 22 06:55:33 2008 From: william at resolversystems.com (William Reade) Date: Fri, 22 Aug 2008 11:55:33 +0100 Subject: [Numpy-discussion] numpy.core.numerictypes questions In-Reply-To: <48AE9299.9030206@enthought.com> References: <48AD4081.1050105@resolversystems.com> <48AE9299.9030206@enthought.com> Message-ID: <48AE9B25.6030308@resolversystems.com> Travis E. Oliphant wrote: > I don't know the answer to these questions. > Thanks for your response -- can anyone suggest a better forum for these sorts of issues? > I would say that unicode arrays are rare and I would definitely work on > floating-point first. Any chance we can see some of your work, by the > way. > > -Travis > Jolly good, that's what I was hoping :). I've recently released v0.5 of Ironclad, which you can get from http://code.google.com/p/ironclad/ -- although I recommend grabbing the svn head rather than v0.5, as I've made some significant improvements over the last couple of days (specifically, finfo now gives sensible epsilons, while v0.5 would claim that finfo(single).eps == finfo(double).eps == 0.125). Be aware that it's still very much a work in progress, and it'll be some time before it's actually useful ;). The (very low-traffic) list for the project is at http://groups.google.com/group/c-extensions-for-ironpython -- and I'd be very keen to hear from anyone who would like to use numpy with IronPython, so I can make sure the most important parts work first. Cheers William > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From klemm at phys.ethz.ch Fri Aug 22 07:23:19 2008 From: klemm at phys.ethz.ch (Hanno Klemm) Date: Fri, 22 Aug 2008 13:23:19 +0200 Subject: [Numpy-discussion] Problem with correlate? In-Reply-To: <9457e7c80808220103x52cb4c8fgf50bfb10912ab11e@mail.gmail.com> References: <9457e7c80808201449g2f356c0bg4fb7a93e04050331@mail.gmail.com> , Message-ID: Hi Stefan, I checked it with numpy version 1.1.1 just now and the result is the same: >>> x = N.array([0.,0,1,0,0]) >>> y1 = N.array([1.,0,0,0,0]) >>> N.correlate(x,y1, mode='full') array([ 0., 0., 0., 0., 0., 0., 1., 0., 0.]) >>> y2 = N.array([1.,0,0,0,0,0,0]) >>> N.correlate(x,y2, mode='full') array([ 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.]) >>> N.__version__ '1.1.1' >>> Best regards, Hanno St?fan van der Walt said: > Hi Hanno > > 2008/8/22 Hanno Klemm : > > yes, indeed, that's what I thought. This result is odd. Has correlate > > been changed since version 1.0.4, or should I submit this as a bug? > > Is there any way that you could try out the latest release on your > machine and see if it solves your problem? We probably won't be > releasing bug-fixes on 1.0.4, but if it exists in 1.1 we'll still > address it. I'm not aware of any changes, but I may simply have > missed it. > > Regards > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- Hanno Klemm klemm at phys.ethz.ch From mmetz at astro.uni-bonn.de Fri Aug 22 07:05:01 2008 From: mmetz at astro.uni-bonn.de (Manuel Metz) Date: Fri, 22 Aug 2008 13:05:01 +0200 Subject: [Numpy-discussion] generalized eigenvector problem Message-ID: <48AE9D5D.7030300@astro.uni-bonn.de> Hi list, are there any plans to implement a routine to solve the "generalized eigenvector problem" as is done in matlab ? see http://www.mathworks.com/access/helpdesk/help/techdoc/ref/eig.html manuel From wright at esrf.fr Fri Aug 22 08:49:55 2008 From: wright at esrf.fr (Jonathan Wright) Date: Fri, 22 Aug 2008 14:49:55 +0200 Subject: [Numpy-discussion] generalized eigenvector problem In-Reply-To: <48AE9D5D.7030300@astro.uni-bonn.de> References: <48AE9D5D.7030300@astro.uni-bonn.de> Message-ID: <48AEB5F3.2030503@esrf.fr> Manuel Metz wrote: > are there any plans to implement a routine to solve the "generalized > eigenvector problem" as is done in matlab ? > see http://www.mathworks.com/access/helpdesk/help/techdoc/ref/eig.html >>> import numpy >>> help(numpy.linalg.eig) Is it what you wanted? HTH, Jon From mmetz at astro.uni-bonn.de Fri Aug 22 09:04:36 2008 From: mmetz at astro.uni-bonn.de (Manuel Metz) Date: Fri, 22 Aug 2008 15:04:36 +0200 Subject: [Numpy-discussion] generalized eigenvector problem In-Reply-To: <48AEB5F3.2030503@esrf.fr> References: <48AE9D5D.7030300@astro.uni-bonn.de> <48AEB5F3.2030503@esrf.fr> Message-ID: <48AEB964.1080501@astro.uni-bonn.de> Jonathan Wright wrote: > Manuel Metz wrote: >> are there any plans to implement a routine to solve the "generalized >> eigenvector problem" as is done in matlab ? >> see http://www.mathworks.com/access/helpdesk/help/techdoc/ref/eig.html > > >>> import numpy > >>> help(numpy.linalg.eig) > > Is it what you wanted? Not, but I found a solution just a few minutes ago... I was looking for a routine to solve A x = l * B x A,B are NxN matrices, l are the eigenvalues and x the eigenvectors. But I've just found out that this is possible with scipy.linalg.eig, which supports the "generalized eigenvector problem" ... mm From christopher.e.kees at usace.army.mil Fri Aug 22 10:00:19 2008 From: christopher.e.kees at usace.army.mil (Chris Kees) Date: Fri, 22 Aug 2008 09:00:19 -0500 Subject: [Numpy-discussion] Revisiting numpy/scipy on 64 bit OSX In-Reply-To: <48ADDADA.1040109@gmail.com> Message-ID: I've been experimenting with both a non-framework, non-universal 64-bit build and a 4-way universal build of the python (2.6) trunk with numpy 1.1.1. The non-framework 64 build appears to give me exactly the same results from numpy.test() as the standard 32-bit version (as well as allowing large arrays like numpy.zeros((1000,1000,1000),'d') ), which is Our numerical models also seem to run fine with it using 8-16G. The 4-way universal python gives the same results in 32-bit but when running in 64-bit I get an error in the tests below, which I haven't had time to look at. It also gives the error >>> a = numpy.zeros((1000,1000,1000),'d') Traceback (most recent call last): File "", line 1, in ValueError: dimensions too large. Anyway, this all goes to say I'm interested in making sure numpy is dependable in 64-bit on OSX 10.5 and preferably for 4-way universal builds, but I haven't seen test failures like yours, at least not recently. In my experience so far on the Mac it's still important to make sure Xcode and the OS are up to date (or to use a completely segregated set of consistently build gnu compilers like you did) and to make sure the fink and mac ports stuff is not in your path before building python and numpy. Otherwise configure and distutils end up making bad choices that I don't have the skill to root out. I've had various versions of non-framework 64-bit builds of python running with numpy and Numeric for a while now (a year maybe). The framework and universal builds of python have only worked recently, and I'm still not confident in them. I don't know what's involved in the buildbots. If it's simple, doesn't take much time to set up, and works behind a tight firewall I might be able to set some up. Chris Numpy is installed in /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-package s/numpy Numpy version 1.1.1 Python version 2.6b2+ (trunk:65908, Aug 20 2008, 08:47:16) [GCC 4.0.1 (Apple Inc. build 5484)] Found 18/18 tests for numpy.core.tests.test_defmatrix Found 3/3 tests for numpy.core.tests.test_errstate Found 3/3 tests for numpy.core.tests.test_memmap Found 290/290 tests for numpy.core.tests.test_multiarray Found 72/72 tests for numpy.core.tests.test_numeric Found 36/36 tests for numpy.core.tests.test_numerictypes Found 12/12 tests for numpy.core.tests.test_records Found 145/145 tests for numpy.core.tests.test_regression Found 7/7 tests for numpy.core.tests.test_scalarmath Found 2/2 tests for numpy.core.tests.test_ufunc Found 16/16 tests for numpy.core.tests.test_umath Found 63/63 tests for numpy.core.tests.test_unicode Found 4/4 tests for numpy.distutils.tests.test_fcompiler_gnu Found 5/5 tests for numpy.distutils.tests.test_misc_util Found 2/2 tests for numpy.fft.tests.test_fftpack Found 3/3 tests for numpy.fft.tests.test_helper Found 25/25 tests for numpy.lib.tests.test__datasource Found 10/10 tests for numpy.lib.tests.test_arraysetops Found 1/1 tests for numpy.lib.tests.test_financial Found 55/55 tests for numpy.lib.tests.test_function_base Found 5/5 tests for numpy.lib.tests.test_getlimits Found 6/6 tests for numpy.lib.tests.test_index_tricks /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-package s/numpy/lib/tests/test_io.py:11: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(c.readlines(), Found 15/15 tests for numpy.lib.tests.test_io Found 1/1 tests for numpy.lib.tests.test_machar Found 4/4 tests for numpy.lib.tests.test_polynomial Found 1/1 tests for numpy.lib.tests.test_regression Found 49/49 tests for numpy.lib.tests.test_shape_base Found 15/15 tests for numpy.lib.tests.test_twodim_base Found 43/43 tests for numpy.lib.tests.test_type_check Found 1/1 tests for numpy.lib.tests.test_ufunclike Found 89/89 tests for numpy.linalg.tests.test_linalg Found 3/3 tests for numpy.linalg.tests.test_regression Traceback (most recent call last): File "", line 1, in File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packag es/numpy/__init__.py", line 126, in test return NumpyTest().test(*args, **kw) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packag es/numpy/testing/numpytest.py", line 579, in test level, verbosity) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packag es/numpy/testing/numpytest.py", line 524, in _test_suite_from_all_tests verbosity=verbosity) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packag es/numpy/testing/numpytest.py", line 417, in _get_suite_list suite = obj(mthname) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packag es/numpy/ma/tests/test_core.py", line 36, in __init__ self.setUp() File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packag es/numpy/ma/tests/test_core.py", line 49, in setUp xf = numpy.where(m1, 1.e+20, x) ValueError: invalid entry in choice array On 8/21/08 4:15 PM, "Michael Abshoff" wrote: > Hi, > > I have been playing with 64 bit numpy/scipy on OSX 10.5 Intel again. I > thought everything worked after the last time we discussed this, but I > noticed that I had Scipy import failures in for example optimize since > the gfortran I used creates 32 bit code per default. > > So I build a gcc 4.2.4 on OSX, then build python in 64 bit mode (fixing > configure.in since Python assumes flags only available with the Apple > gcc on Darwin), added a wrapper around gfortran that injects a "-m64" > compile flag. Then I checkout out and build numpy (r5671), scipy (r4661) > and the current nose release and ran the tests. I had some more failures > than I expected (details below). > > Any thoughts on the failures? I am about to build ATLAS and a couple > other dependencies on that box. > > William Stein is at Scipy 08 and there seems to be at least some > interest in 64 bit OSX binaries. If anyone wants one let me know and I > can put one together once Scipy 0.7 and Numpy 1.1.2 are out. > > Cheers, > > Michael > > numpy failed tests: > > **************************************************************** > File > "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests > /test_polynomial.py", > line 29, in test_polynomial > Failed example: > p / q > Expected: > (poly1d([ 0.33333333]), poly1d([ 1.33333333, 2.66666667])) > Got: > (poly1d([ 0.333]), poly1d([ 1.333, 2.667])) > ********************************************************************** > File > "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests > /test_polynomial.py", > line 51, in test_polynomial > Failed example: > p.integ() > Expected: > poly1d([ 0.33333333, 1. , 3. , 0. ]) > Got: > poly1d([ 0.333, 1. , 3. , 0. ]) > **********************************************************************File > "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests > /test_polynomial.py", > line 53, in test_polynomialFailed example: > p.integ(1) > Expected: > poly1d([ 0.33333333, 1. , 3. , 0. ]) > Got: > poly1d([ 0.333, 1. , 3. , 0. ]) > ********************************************************************** > File > "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests > /test_polynomial.py", > line 55, in test_polynomial > Failed example: > p.integ(5) > Expected: > poly1d([ 0.00039683, 0.00277778, 0.025 , 0. , 0. > , > 0. , 0. , 0. ]) > Got: > poly1d([ 0. , 0.003, 0.025, 0. , 0. , 0. , 0. , 0. > ]) > > ******************************** > File > "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests > /test_ufunclike.py", > line 48, in test_ufunclike > Failed example: > U.log2(a) > Expected: > array([ 2.169925 , 1.20163386, 2.70043972]) > Got: > array([ 2.17 , 1.202, 2.7 ]) > ********************************************************************** > File > "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests > /test_ufunclike.py", > line 53, in test_ufunclike > Failed example: > U.log2(a, y) > Expected: > array([ 2.169925 , 1.20163386, 2.70043972]) > Got: > array([ 2.17 , 1.202, 2.7 ]) > ********************************************************************** > File > "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests > /test_ufunclike.py", > line 55, in test_ufunclike > Failed example: > y > Expected: > array([ 2.169925 , 1.20163386, 2.70043972]) > Got: > array([ 2.17 , 1.202, 2.7 ]) > ====================================================================== > FAIL: test_umath.TestComplexFunctions.test_it > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/nose/case.py", > line 182, in runTest > self.test(*self.arg) > File > "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/core/test > s/test_umath.py", > line 196, in test_it > assert_almost_equal(fz.real, fr, err_msg='real part %s'%f) > File > "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/testing/u > tils.py", > line 207, in assert_almost_equal > assert round(abs(desired - actual),decimal) == 0, msg > AssertionError: > Items are not equal: real part > ACTUAL: -0.3010299956639812 > DESIRED: 0.0 > > > scipy: > > ====================================================================== > ERROR: test_nonsymmetric_modes (test_arpack.TestEigenNonSymmetric) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/li > nalg/eigen/arpack/tests/test_arpack.py", > line 204, in test_nonsymmetric_modes > self.eval_evec(m,typ,k,which) > File > "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/li > nalg/eigen/arpack/tests/test_arpack.py", > line 186, in eval_evec > eval,evec=eigen(a,k,which=which,**kwds) > File > "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/li > nalg/eigen/arpack/arpack.py", > line 220, in eigen > raise RuntimeError("Error info=%d in arpack"%info) > RuntimeError: Error info=-8 in arpack > > ====================================================================== > ERROR: test_starting_vector (test_arpack.TestEigenNonSymmetric) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/li > nalg/eigen/arpack/tests/test_arpack.py", > line 214, in test_starting_vector > self.eval_evec(self.symmetric[0],typ,k,which='LM',v0=v0) > File > "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/li > nalg/eigen/arpack/tests/test_arpack.py", > line 186, in eval_evec > eval,evec=eigen(a,k,which=which,**kwds) > File > "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/li > nalg/eigen/arpack/arpack.py", > line 220, in eigen > raise RuntimeError("Error info=%d in arpack"%info) > RuntimeError: Error info=-8 in arpack > ====================================================================== > FAIL: test_implicit (test_odr.TestODR) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/odr/tests > /test_odr.py", > line 118, in test_implicit > 2.0778813389755596e-03]]), > File > "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/testing/u > tils.py", > line 304, in assert_array_almost_equal > header='Arrays are not almost equal') > File > "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/testing/u > tils.py", > line 289, in assert_array_compare > assert cond, msg > AssertionError: > Arrays are not almost equal > > (mismatch 4.0%) > x: array([[ 2.10892628e+00, -1.94376722e+00, 7.02635228e-02, > -4.71752493e-02, 5.25155371e-02], > [ -1.94376722e+00, 2.04814914e+00, -6.16004809e-02,... > y: array([[ 2.10892746e+00, -1.94376864e+00, 7.02635509e-02, > -4.71752674e-02, 5.25155759e-02], > [ -1.94376864e+00, 2.04815092e+00, -6.16005159e-02,... > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From robert.kern at gmail.com Fri Aug 22 10:58:23 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 22 Aug 2008 07:58:23 -0700 Subject: [Numpy-discussion] Revisiting numpy/scipy on 64 bit OSX In-Reply-To: References: <48ADDADA.1040109@gmail.com> Message-ID: <3d375d730808220758j5537ed8aqe27421b7e3bfaef5@mail.gmail.com> On Fri, Aug 22, 2008 at 07:00, Chris Kees wrote: > I've been experimenting with both a non-framework, non-universal 64-bit > build and a 4-way universal build of the python (2.6) trunk with numpy > 1.1.1. The non-framework 64 build appears to give me exactly the same > results from numpy.test() as the standard 32-bit version (as well as > allowing large arrays like numpy.zeros((1000,1000,1000),'d') ), which is > > > > Our numerical models also seem to run fine with it using 8-16G. The 4-way > universal python gives the same results in 32-bit but when running in > 64-bit I get an error in the tests below, which I haven't had time to look > at. It also gives the error > >>>> a = numpy.zeros((1000,1000,1000),'d') > Traceback (most recent call last): > File "", line 1, in > ValueError: dimensions too large. Much of our configuration occurs by compiling small C programs and executing them. Probably, one of these got run in 32-bit mode, and that fooled the numpy build into thinking that it was for 32-bit only. Unfortunately, what you are trying to do is tantamount to cross-compiling, and neither distutils nor the additions we have built on top of it work very well with cross-compiling. It's possible that we could special case the configuration on OS X, though. Instead of trusting the results of the executables, we can probably recognize each of the 4 OS X variants through #ifdefs and reset the discovered results. This isn't easily extended to all platforms (which is why we went with the executable approach in the first place), but OS X on both 32-bit and 64-bit will be increasingly common but still manageable. I would welcome contributions in this area. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From michael.abshoff at googlemail.com Fri Aug 22 12:34:33 2008 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Fri, 22 Aug 2008 09:34:33 -0700 Subject: [Numpy-discussion] Revisiting numpy/scipy on 64 bit OSX In-Reply-To: <3d375d730808220758j5537ed8aqe27421b7e3bfaef5@mail.gmail.com> References: <48ADDADA.1040109@gmail.com> <3d375d730808220758j5537ed8aqe27421b7e3bfaef5@mail.gmail.com> Message-ID: <48AEEA99.2070505@gmail.com> Robert Kern wrote: > On Fri, Aug 22, 2008 at 07:00, Chris Kees > wrote: Hi, >> I've been experimenting with both a non-framework, non-universal 64-bit >> build and a 4-way universal build of the python (2.6) trunk with numpy >> 1.1.1. The non-framework 64 build appears to give me exactly the same >> results from numpy.test() as the standard 32-bit version (as well as >> allowing large arrays like numpy.zeros((1000,1000,1000),'d') ), which is >> >> I used a gcc 4.2.4 build from sources and last time I used XCode (and in addition a gfortran build from 4.2.3 sources) most things went fine. Note that I am using Python 2.5.2 and that I had to make configure.in not add some default BASEFLAGS since those added flags only exist for the Apple version of gcc. This was also numpy and scipy svn tip :) >> Our numerical models also seem to run fine with it using 8-16G. The 4-way >> universal python gives the same results in 32-bit but when running in >> 64-bit I get an error in the tests below, which I haven't had time to look >> at. It also gives the error >> >>>>> a = numpy.zeros((1000,1000,1000),'d') >> Traceback (most recent call last): >> File "", line 1, in >> ValueError: dimensions too large. > > Much of our configuration occurs by compiling small C programs and > executing them. Probably, one of these got run in 32-bit mode, and > that fooled the numpy build into thinking that it was for 32-bit only. Yeah, building universal binaries is still fraught with issues since much code out there uses values derived at configure time for endianess and other values. IIRC Apple patches Python to make some of those constants functions, but that recollection might be wrong in case of Python. I usually use lipo to make universal binaries theses days to get around that limitation, but that means four compilations instead of two. My main goal in all of this is a 64 bit Sage on OSX (which I am reasonable close to fully working), but due to above mentioned problems for example with gmp it seems unlikely that I can produce a universal version directly and lipo is a way out of this. > Unfortunately, what you are trying to do is tantamount to > cross-compiling, and neither distutils nor the additions we have built > on top of it work very well with cross-compiling. It's possible that > we could special case the configuration on OS X, though. Instead of > trusting the results of the executables, we can probably recognize > each of the 4 OS X variants through #ifdefs and reset the discovered > results. This isn't easily extended to all platforms (which is why we > went with the executable approach in the first place), but OS X on > both 32-bit and 64-bit will be increasingly common but still > manageable. I would welcome contributions in this area. I am actually fairly confident that 64 bit Intel will dominate the Apple userbase in the short term. Every laptop and workstation on sale by Apple now and in the last two years or so has been 64 bit capable and for console applications OSX 10.4 or higher will do. So I see little benefit from doing 32 bit on OSX except for legacy support :). I know that Apple hardware tends to stick around longer, so even Sage will support 32 bit OSX for a while. Cheers, Michael From Catherine.M.Moroney at jpl.nasa.gov Fri Aug 22 13:38:48 2008 From: Catherine.M.Moroney at jpl.nasa.gov (Catherine Moroney) Date: Fri, 22 Aug 2008 10:38:48 -0700 Subject: [Numpy-discussion] subsampling array without loops Message-ID: <93455E9E-3297-40CF-8796-ADEACB11A182@jpl.nasa.gov> I'm looking for a way to acccomplish the following task without lots of loops involved, which are really slowing down my code. I have a 128x512 array which I want to break down into 2x2 squares. Then, for each 2x2 square I want to do some simple calculations such as finding the maximum value, which I then store in a 64x256 array. Here is the actual code involved. It's only slightly more complex than what I described above, since it also involves doing a bit of masking on the 2x2 sub-arrays. Any hints are much appreciated! An inordinate amount of time is being spent in this function and another one like it. Catherine def calc_sdcm_at_rlra(self,iblock): npixl = self.nlineht/self.nlinerl npixs = self.nsmpht/self.nsmprl sdcm_out = numpy.array([Constants.CLOUDMASK_NR] *self.nlinerl*self.nsmprl,'int8') \ .reshape(self.nlinerl,self.nsmprl) for ilinerl in range(0,self.nlinerl): for ismprl in range(0,self.nsmprl): height = self.data[iblock].height[ilinerl*2:ilinerl*2 +2, ismprl*2:ismprl*2+2] sdcm = self.data[iblock].sdcm[ilinerl*2:ilinerl*2 +2, ismprl*2:ismprl*2+2] source = self.data[iblock].heightsrc [ilinerl*2:ilinerl*2+2, ismprl*2:ismprl*2+2] mask1 = (source == Constants.HEIGHT_STEREO) mask2 = ( (source == Constants.HEIGHT_SURFACE) | \ (source == Constants.HEIGHT_DEFAULT) ) if (mask1.any()): loc = height[mask1].argmax() sdcm_out[ilinerl,ismprl] = sdcm[mask1].ravel()[loc] elif (mask2.any()): loc = height[mask2].argmax() sdcm_out[ilinerl,ismprl] = sdcm[mask2].ravel()[loc] return sdcm_out From zachary.pincus at yale.edu Fri Aug 22 14:27:40 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Fri, 22 Aug 2008 14:27:40 -0400 Subject: [Numpy-discussion] subsampling array without loops In-Reply-To: <93455E9E-3297-40CF-8796-ADEACB11A182@jpl.nasa.gov> References: <93455E9E-3297-40CF-8796-ADEACB11A182@jpl.nasa.gov> Message-ID: <31BD018B-A9E3-4F09-8F8F-A8D2ABBEF245@yale.edu> > I'm looking for a way to acccomplish the following task without lots > of loops involved, which are really slowing down my code. > > I have a 128x512 array which I want to break down into 2x2 squares. > Then, for each 2x2 square I want to do some simple calculations > such as finding the maximum value, which I then store in a 64x256 > array. Perhaps you could take the relevant slices of the arrays with appropriate striding: a = numpy.arange(128*512).reshape((128, 512)) top_left = a[::2, ::2] top_right = a[::2, 1::2] bottom_left = a[1::2, ::2] bottom_right = a[1::2, 1::2] tiles = numpy.array([top_left, top_right, bottom_left, bottom_right]) maxes = tiles.max(axis=0) Similarly if you want overlapping tiles, you could leave out the final :2 in the slice specifications above. Zach > Here is the actual code involved. It's only slightly more complex > than what I described above, since it also involves doing a bit of > masking on the 2x2 sub-arrays. > > Any hints are much appreciated! An inordinate amount of time is > being spent in this function and another one like it. > > Catherine > > def calc_sdcm_at_rlra(self,iblock): > > npixl = self.nlineht/self.nlinerl > npixs = self.nsmpht/self.nsmprl > > sdcm_out = numpy.array([Constants.CLOUDMASK_NR] > *self.nlinerl*self.nsmprl,'int8') \ > .reshape(self.nlinerl,self.nsmprl) > > for ilinerl in range(0,self.nlinerl): > for ismprl in range(0,self.nsmprl): > > height = self.data[iblock].height[ilinerl*2:ilinerl*2 > +2, ismprl*2:ismprl*2+2] > sdcm = self.data[iblock].sdcm[ilinerl*2:ilinerl*2 > +2, ismprl*2:ismprl*2+2] > source = self.data[iblock].heightsrc > [ilinerl*2:ilinerl*2+2, ismprl*2:ismprl*2+2] > > mask1 = (source == Constants.HEIGHT_STEREO) > mask2 = ( (source == Constants.HEIGHT_SURFACE) | \ > (source == Constants.HEIGHT_DEFAULT) ) > > if (mask1.any()): > loc = height[mask1].argmax() > sdcm_out[ilinerl,ismprl] = sdcm[mask1].ravel() > [loc] > elif (mask2.any()): > loc = height[mask2].argmax() > sdcm_out[ilinerl,ismprl] = sdcm[mask2].ravel() > [loc] > > return sdcm_out > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From stefan at sun.ac.za Fri Aug 22 15:59:54 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 22 Aug 2008 12:59:54 -0700 Subject: [Numpy-discussion] numpy.core.numerictypes questions In-Reply-To: <48AD4081.1050105@resolversystems.com> References: <48AD4081.1050105@resolversystems.com> Message-ID: <9457e7c80808221259r7cdf8640kb7dc7bc943fe8900@mail.gmail.com> 2008/8/21 William Reade : > Line 532 of numerictypes.py reads "_unicodesize = > array('u','U1').itemsize", but _unicodesize itself does not appear to be > referenced anywhere else. My questions are: I removed this in r5674. Regards St?fan From christopher.e.kees at usace.army.mil Fri Aug 22 17:08:39 2008 From: christopher.e.kees at usace.army.mil (Chris Kees) Date: Fri, 22 Aug 2008 16:08:39 -0500 Subject: [Numpy-discussion] Revisiting numpy/scipy on 64 bit OSX In-Reply-To: <48AEEA99.2070505@gmail.com> Message-ID: On 8/22/08 11:34 AM, "Michael Abshoff" wrote: > Robert Kern wrote: >> On Fri, Aug 22, 2008 at 07:00, Chris Kees >> wrote: > > Hi, > >>> I've been experimenting with both a non-framework, non-universal 64-bit >>> build and a 4-way universal build of the python (2.6) trunk with numpy >>> 1.1.1. The non-framework 64 build appears to give me exactly the same >>> results from numpy.test() as the standard 32-bit version (as well as >>> allowing large arrays like numpy.zeros((1000,1000,1000),'d') ), which is >>> >>> > > I used a gcc 4.2.4 build from sources and last time I used XCode (and in > addition a gfortran build from 4.2.3 sources) most things went fine. > Note that I am using Python 2.5.2 and that I had to make configure.in > not add some default BASEFLAGS since those added flags only exist for > the Apple version of gcc. This was also numpy and scipy svn tip :) > >>> Our numerical models also seem to run fine with it using 8-16G. The 4-way >>> universal python gives the same results in 32-bit but when running in >>> 64-bit I get an error in the tests below, which I haven't had time to look >>> at. It also gives the error >>> >>>>>> a = numpy.zeros((1000,1000,1000),'d') >>> Traceback (most recent call last): >>> File "", line 1, in >>> ValueError: dimensions too large. >> >> Much of our configuration occurs by compiling small C programs and >> executing them. Probably, one of these got run in 32-bit mode, and >> that fooled the numpy build into thinking that it was for 32-bit only. > > Yeah, building universal binaries is still fraught with issues since > much code out there uses values derived at configure time for endianess > and other values. IIRC Apple patches Python to make some of those > constants functions, but that recollection might be wrong in case of > Python. I usually use lipo to make universal binaries theses days to get > around that limitation, but that means four compilations instead of two. > > My main goal in all of this is a 64 bit Sage on OSX (which I am > reasonable close to fully working), but due to above mentioned problems > for example with gmp it seems unlikely that I can produce a universal > version directly and lipo is a way out of this. > >> Unfortunately, what you are trying to do is tantamount to >> cross-compiling, and neither distutils nor the additions we have built >> on top of it work very well with cross-compiling. It's possible that >> we could special case the configuration on OS X, though. Instead of >> trusting the results of the executables, we can probably recognize >> each of the 4 OS X variants through #ifdefs and reset the discovered >> results. This isn't easily extended to all platforms (which is why we >> went with the executable approach in the first place), but OS X on >> both 32-bit and 64-bit will be increasingly common but still >> manageable. I would welcome contributions in this area. > This sounds like a reasonable approach to me. I wouldn't be disappointed if everything but x86_64 went away, but I'm afraid it's not going to happen soon enough to avoid dealing with i386 and ppc64 in some way. It would be convenient for me if I could just distributed a 4-way universal python/numpy along with my modules. Unfortunately I don't know enough about distutils and the numpy additions to be confident that I can make the fixes. Can we can get all the information we need by comparing the 4-way universal build with a pure x86_64 build? -Chris > I am actually fairly confident that 64 bit Intel will dominate the Apple > userbase in the short term. Every laptop and workstation on sale by > Apple now and in the last two years or so has been 64 bit capable and > for console applications OSX 10.4 or higher will do. So I see little > benefit from doing 32 bit on OSX except for legacy support :). I know > that Apple hardware tends to stick around longer, so even Sage will > support 32 bit OSX for a while. > > Cheers, > > Michael > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From listservs at mac.com Fri Aug 22 21:35:49 2008 From: listservs at mac.com (Chris Fonnesbeck) Date: Sat, 23 Aug 2008 01:35:49 +0000 (UTC) Subject: [Numpy-discussion] generating float sequences Message-ID: Is there any prospect for easily getting ranges of floats in numpy, rather than just integers? In R, for example, you can specify a decimal value for the step size. It would be great to be able to do the following: arange(0, 100, 0.1) for a sequence from 0 to 100 in steps of 0.1. It seems rather unpythonic to have to do this: arange(0, 1000)/10. From wnbell at gmail.com Fri Aug 22 22:08:43 2008 From: wnbell at gmail.com (Nathan Bell) Date: Fri, 22 Aug 2008 22:08:43 -0400 Subject: [Numpy-discussion] generating float sequences In-Reply-To: References: Message-ID: On Fri, Aug 22, 2008 at 9:35 PM, Chris Fonnesbeck wrote: > Is there any prospect for easily getting ranges of floats in numpy, rather than > just integers? In R, for example, you can specify a decimal value for the step > size. It would be great to be able to do the following: > > arange(0, 100, 0.1) > It appears to be great already :) In other words, arange(0, 100, 0.1) does exactly what you want. -- Nathan Bell wnbell at gmail.com http://graphics.cs.uiuc.edu/~wnbell/ From aisaac at american.edu Fri Aug 22 22:09:26 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 22 Aug 2008 22:09:26 -0400 Subject: [Numpy-discussion] generating float sequences In-Reply-To: References: Message-ID: <48AF7156.9050402@american.edu> Chris Fonnesbeck wrote: > Is there any prospect for easily getting ranges of floats > in numpy, rather than just integers? >>> help(np.linspace) Help on function linspace in module numpy.lib.function_base: linspace(start, stop, num=50, endpoint=True, retstep=False) Return evenly spaced numbers. Return num evenly spaced samples from start to stop. If endpoint is True, the last sample is stop. If retstep is True then return (seq, step_value), where step_value used. > for a sequence from 0 to 100 in steps of 0.1. It seems rather unpythonic to > have to do this: > arange(0, 1000)/10. I find it plenty Pythonic, although it is not the answer to your question. ;-) Cheers, Alan Isaac From aisaac at american.edu Fri Aug 22 22:44:10 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 22 Aug 2008 22:44:10 -0400 Subject: [Numpy-discussion] generating float sequences In-Reply-To: References: Message-ID: <48AF797A.6070508@american.edu> Nathan Bell wrote: > arange(0, 100, 0.1) does exactly what you want. Hmmm; maybe. I thought Chris wanted the endpoint... Cheers, Alan Isaac From listservs at mac.com Fri Aug 22 23:18:43 2008 From: listservs at mac.com (Chris Fonnesbeck) Date: Sat, 23 Aug 2008 03:18:43 +0000 (UTC) Subject: [Numpy-discussion] generating float sequences References: Message-ID: Chris Fonnesbeck mac.com> writes: > It would be great to be able to do the following: > > arange(0, 100, 0.1) As it turns out, I am an idiot. Apologies. From listservs at mac.com Sat Aug 23 00:35:03 2008 From: listservs at mac.com (Chris Fonnesbeck) Date: Sat, 23 Aug 2008 04:35:03 +0000 (UTC) Subject: [Numpy-discussion] problems generating negative binomials Message-ID: I'm trying to generate negative binomial random numbers where the n parameter is non integer (<1 in fact). This should be possible, as n only needs to be real. However, I get the following message: ValueError: n <= 0 I assume this is because some rounding is going on behind the scenes. The reason I am doing this is because I'm trying to use the alternative parameterisation of the NB, with parameters k,m for which p=k/(k+m). I noticed that this parameterisation does not exist over in the scipy distributions either. Has anyone else run into this problem? The latter parameterisation is far more common in ecology, because it is an overdispersed Poisson. Thanks, cf From eads at soe.ucsc.edu Sat Aug 23 11:42:45 2008 From: eads at soe.ucsc.edu (Damian Eads) Date: Sat, 23 Aug 2008 09:42:45 -0600 Subject: [Numpy-discussion] sprint: IRC chat? Message-ID: <48B02FF5.5020109@soe.ucsc.edu> Friends, Are we meeting over IRC chat? I'd like to help with the sprint but remotely. I have to leave LA today, unfortunately. Thanks! Damian From gael.varoquaux at normalesup.org Sat Aug 23 12:04:42 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 23 Aug 2008 18:04:42 +0200 Subject: [Numpy-discussion] sprint: IRC chat? In-Reply-To: <48B02FF5.5020109@soe.ucsc.edu> References: <48B02FF5.5020109@soe.ucsc.edu> Message-ID: <20080823160442.GA21354@phare.normalesup.org> On Sat, Aug 23, 2008 at 09:42:45AM -0600, Damian Eads wrote: > Are we meeting over IRC chat? I'd like to help with the sprint but > remotely. I have to leave LA today, unfortunately. I am on #scipy on freenode. Ga?l From oliphant at enthought.com Sat Aug 23 13:58:24 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sat, 23 Aug 2008 12:58:24 -0500 Subject: [Numpy-discussion] How to replace a loop for a ufunc (to test optimizations) Message-ID: <48B04FC0.1020409@enthought.com> Yesterday at the SciPy conference I suggested it would be a good idea to add a function to replace one of the inner loops in a ufunc so that an extension module could test an optimized inner loop more easily. It turns out that this is such a good idea that it's actually already in the code. The UFUNC_API call that does this is called: PyUFunc_ReplaceLoopBySignature You would use it something like this static void _new_add_loopfunc(char **args, intp *dimensions, intp *steps, void *func) { /* my new faster code */ } Then in the extension module (either in the init or in a method call) something like this: PyUFuncObject *ufunc; PyUFuncGenericFunction oldloopfunc, dummy; int signature[3] /* This should be the number of arguments to the ufunc you are replacing */ PyObject *module, *dict; module = PyImport_ImportModule("umath"); dict = PyModule_GetDict(module); ufunc = PyDict_GetItemString(dict, "add") /* get the ufunc where you want to replace one of the loops */ signature[0] = NPY_DOUBLE; signature[1] = NPY_DOUBLE; signature[2] = NPY_DOUBLE; if (PyUFunc_ReplaceLoopBySignature(ufunc, _new_add_loopfunc, signature, &old1dloop) < 0) { PyErr_SetString(PyExc_RuntimeError, "problem replacing loop"); /* error return i.e. return NULL or return or return -1*/ } Then to restore the original function: if (PyUFunc_ReplaceLoopBySignature(ufunc, old1dloop, signature, &dummy) < 0) { /* error */ } Remember to use import_umath() in the module you are using the UFUNC_API in. Now, lot's of folks should be able to test optimizations. -Travis From oliphant at enthought.com Sat Aug 23 19:39:30 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sat, 23 Aug 2008 18:39:30 -0500 Subject: [Numpy-discussion] Report from SciPy Message-ID: <48B09FB2.1050009@enthought.com> Hi everybody, Robert K, Chuck H, Stefan VdW, Jarrod M, David C, and I had a nice discussion about the future directions of NumPy. We resolved some things and would like community feedback on them if there are opinions. * we will be moving to time-based releases (at least 2 times a year -- November / May) with major changes not accepted about 4 weeks before the release. * The releases will be numbered major.minor.bugfix * There will be no ABI changes in minor releases * There will be no API changes in bugfix releases * Any API changes in minor releases will be done in a backward compatible fashion (possibly with deprecations). * Thus 1.2 will not break ABI compatibility but will add new API features. * We will push the generalized ufuncs off to 1.3 * NumPy 2.0 will be a library and will not automagically import numpy.fft * We will suggest that other libraries use from numpy import fft instead of import numpy as np; np.fft * We will apply most of Andrew Dalke's speed up patches but will keep ctypeslib import * We will remove automatic import of numpy.doc from trunk * NumPy 1.2 will come out shortly after the conference (rc1 on Sunday). * SciPy 0.7b1 will come out after the sprints. If there is anything else I missed from those who were there, please let us all know. By the way, as promised, the NumPy book is now available for download and the source to the book is checked in to the numpy SVN tree: http://svn.scipy.org/svn/numpy/trunk/numpy/doc/numpybook Best regards, -Travis From stefan at sun.ac.za Sat Aug 23 19:49:52 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sat, 23 Aug 2008 16:49:52 -0700 Subject: [Numpy-discussion] Remove ParametricTestCase from numpy.testing Message-ID: <9457e7c80808231649y36a158ecj975bd2f88896e4c1@mail.gmail.com> Hey all, Now that we have switched to Nose as a test framework, we should look into deprecating ParametricTestCase. Alan, have you looked into this before? Cheers St?fan From stefan at sun.ac.za Sat Aug 23 19:57:09 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sat, 23 Aug 2008 16:57:09 -0700 Subject: [Numpy-discussion] Report from SciPy In-Reply-To: <48B09FB2.1050009@enthought.com> References: <48B09FB2.1050009@enthought.com> Message-ID: <9457e7c80808231657u75dde588x316aac30bf117eb8@mail.gmail.com> 2008/8/23 Travis E. Oliphant : > By the way, as promised, the NumPy book is now available for download > and the source to the book is checked in to the numpy SVN tree: > > http://svn.scipy.org/svn/numpy/trunk/numpy/doc/numpybook We just reorganised the docs somewhat, so this URL is now http://svn.scipy.org/svn/numpy/trunk/doc/numpybook/ A great big thanks to Travis for releasing his book to the community! Regards St?fan From charlesr.harris at gmail.com Sat Aug 23 20:03:27 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 23 Aug 2008 18:03:27 -0600 Subject: [Numpy-discussion] Report from SciPy In-Reply-To: <48B09FB2.1050009@enthought.com> References: <48B09FB2.1050009@enthought.com> Message-ID: On Sat, Aug 23, 2008 at 5:39 PM, Travis E. Oliphant wrote: > Hi everybody, > > Robert K, Chuck H, Stefan VdW, Jarrod M, David C, and I had a nice > discussion about the future directions of NumPy. We resolved some > things and would like community feedback on them if there are opinions. > > * we will be moving to time-based releases (at least 2 times a year -- > November / May) with major changes not accepted about 4 weeks before the > release. > * The releases will be numbered major.minor.bugfix > * There will be no ABI changes in minor releases > * There will be no API changes in bugfix releases > * Any API changes in minor releases will be done in a backward > compatible fashion (possibly with deprecations). > * Thus 1.2 will not break ABI compatibility but will add new API features. > * We will push the generalized ufuncs off to 1.3 > * NumPy 2.0 will be a library and will not automagically import numpy.fft > * We will suggest that other libraries use from numpy import fft > instead of import numpy as np; np.fft > * We will apply most of Andrew Dalke's speed up patches but will keep > ctypeslib import > * We will remove automatic import of numpy.doc from trunk > * NumPy 1.2 will come out shortly after the conference (rc1 on Sunday). > * SciPy 0.7b1 will come out after the sprints. > > If there is anything else I missed from those who were there, please let > us all know. > There is an excellent discussion of ABI/API issues here. I think we have pretty much come around to the authors point of view, but it is useful to have things clearly written up. > > By the way, as promised, the NumPy book is now available for download > and the source to the book is checked in to the numpy SVN tree: > > http://svn.scipy.org/svn/numpy/trunk/numpy/doc/numpybook > Thanks for that. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jbenson at sextans.lowell.edu Sat Aug 23 22:04:09 2008 From: jbenson at sextans.lowell.edu (James A. Benson) Date: Sat, 23 Aug 2008 19:04:09 -0700 (MST) Subject: [Numpy-discussion] Report from SciPy In-Reply-To: <9457e7c80808231657u75dde588x316aac30bf117eb8@mail.gmail.com> References: <48B09FB2.1050009@enthought.com> <9457e7c80808231657u75dde588x316aac30bf117eb8@mail.gmail.com> Message-ID: On Sat, 23 Aug 2008, St?fan van der Walt wrote: > 2008/8/23 Travis E. Oliphant : >> By the way, as promised, the NumPy book is now available for download >> and the source to the book is checked in to the numpy SVN tree: >> >> http://svn.scipy.org/svn/numpy/trunk/numpy/doc/numpybook > > We just reorganised the docs somewhat, so this URL is now > > http://svn.scipy.org/svn/numpy/trunk/doc/numpybook/ > > A great big thanks to Travis for releasing his book to the community! > Agreed!!! I do not see any pdf file etc. of his book at the above link. Cheers, Jim (purchaser of his book before release). From kwgoodman at gmail.com Sat Aug 23 23:50:57 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Sat, 23 Aug 2008 20:50:57 -0700 Subject: [Numpy-discussion] Report from SciPy In-Reply-To: References: <48B09FB2.1050009@enthought.com> <9457e7c80808231657u75dde588x316aac30bf117eb8@mail.gmail.com> Message-ID: On Sat, Aug 23, 2008 at 7:04 PM, James A. Benson wrote: > On Sat, 23 Aug 2008, St?fan van der Walt wrote: > >> 2008/8/23 Travis E. Oliphant : >>> >>> By the way, as promised, the NumPy book is now available for download >>> and the source to the book is checked in to the numpy SVN tree: >>> >>> http://svn.scipy.org/svn/numpy/trunk/numpy/doc/numpybook >> >> We just reorganised the docs somewhat, so this URL is now >> >> http://svn.scipy.org/svn/numpy/trunk/doc/numpybook/ >> >> A great big thanks to Travis for releasing his book to the community! >> > > Agreed!!! > > I do not see any pdf file etc. of his book at the above > link. http://www.tramy.us/numpybook.pdf From alan.mcintyre at gmail.com Sun Aug 24 01:23:34 2008 From: alan.mcintyre at gmail.com (Alan McIntyre) Date: Sat, 23 Aug 2008 22:23:34 -0700 Subject: [Numpy-discussion] Remove ParametricTestCase from numpy.testing In-Reply-To: <9457e7c80808231649y36a158ecj975bd2f88896e4c1@mail.gmail.com> References: <9457e7c80808231649y36a158ecj975bd2f88896e4c1@mail.gmail.com> Message-ID: <1d36917a0808232223x589f9ea7yac72e9e3fd51aab5@mail.gmail.com> On Sat, Aug 23, 2008 at 4:49 PM, St?fan van der Walt wrote: > Now that we have switched to Nose as a test framework, we should look > into deprecating ParametricTestCase. Alan, have you looked into this > before? Actually, it was removed right after the nose framework was working, but I think a decision was made to keep it around until 1.3 in case somebody was using it, so I put it back. :) After the 1.2 release it can just be deleted without any issues, I think. From wright at esrf.fr Sun Aug 24 12:12:09 2008 From: wright at esrf.fr (Jon Wright) Date: Sun, 24 Aug 2008 18:12:09 +0200 Subject: [Numpy-discussion] Report from SciPy In-Reply-To: <48B09FB2.1050009@enthought.com> References: <48B09FB2.1050009@enthought.com> Message-ID: <48B18859.6080304@esrf.fr> Travis E. Oliphant wrote: > ... > * Thus 1.2 will not break ABI compatibility but will add new API features. > This is really great news (amongst the other good things). Many thanks for keeping the ABI compatible! All the best, Jon . From bsouthey at gmail.com Sun Aug 24 13:58:43 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Sun, 24 Aug 2008 12:58:43 -0500 Subject: [Numpy-discussion] Report from SciPy In-Reply-To: <48B09FB2.1050009@enthought.com> References: <48B09FB2.1050009@enthought.com> Message-ID: Hi, I think this is a great idea but I am curious about what NumPy will be doing with Python 3. The Python 3 final is scheduled for 1st October release so is there a policy on handling the migration to Python 3 or dual support of the 2 and 3 series? Thanks Bruce On Sat, Aug 23, 2008 at 6:39 PM, Travis E. Oliphant wrote: > Hi everybody, > > Robert K, Chuck H, Stefan VdW, Jarrod M, David C, and I had a nice > discussion about the future directions of NumPy. We resolved some > things and would like community feedback on them if there are opinions. > > * we will be moving to time-based releases (at least 2 times a year -- > November / May) with major changes not accepted about 4 weeks before the > release. > * The releases will be numbered major.minor.bugfix > * There will be no ABI changes in minor releases > * There will be no API changes in bugfix releases > * Any API changes in minor releases will be done in a backward > compatible fashion (possibly with deprecations). > * Thus 1.2 will not break ABI compatibility but will add new API features. > * We will push the generalized ufuncs off to 1.3 > * NumPy 2.0 will be a library and will not automagically import numpy.fft > * We will suggest that other libraries use from numpy import fft > instead of import numpy as np; np.fft > * We will apply most of Andrew Dalke's speed up patches but will keep > ctypeslib import > * We will remove automatic import of numpy.doc from trunk > * NumPy 1.2 will come out shortly after the conference (rc1 on Sunday). > * SciPy 0.7b1 will come out after the sprints. > > If there is anything else I missed from those who were there, please let > us all know. > > By the way, as promised, the NumPy book is now available for download > and the source to the book is checked in to the numpy SVN tree: > > http://svn.scipy.org/svn/numpy/trunk/numpy/doc/numpybook > > Best regards, > > -Travis > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From aisaac at american.edu Sun Aug 24 14:27:07 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 24 Aug 2008 14:27:07 -0400 Subject: [Numpy-discussion] Report from SciPy In-Reply-To: References: <48B09FB2.1050009@enthought.com> Message-ID: <48B1A7FB.8040009@american.edu> Bruce Southey wrote: > I think this is a great idea but I am curious about what NumPy will be > doing with Python 3. The Python 3 final is scheduled for 1st October > release so is there a policy on handling the migration to Python 3 or > dual support of the 2 and 3 series? As a footnote to this query, has the Scons community addressed the problem Scons had working with VS 2008 (=VS9)? Cheers, Alan Isaac From robert.kern at gmail.com Sun Aug 24 14:57:28 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 24 Aug 2008 11:57:28 -0700 Subject: [Numpy-discussion] Report from SciPy In-Reply-To: References: <48B09FB2.1050009@enthought.com> Message-ID: <3d375d730808241157s426f947anbb89f9500df7f4a9@mail.gmail.com> On Sun, Aug 24, 2008 at 10:58, Bruce Southey wrote: > Hi, > I think this is a great idea but I am curious about what NumPy will be > doing with Python 3. The Python 3 final is scheduled for 1st October > release so is there a policy on handling the migration to Python 3 or > dual support of the 2 and 3 series? We're still evaluating the situation. Rahul Garg has spent some time yesterday at the sprint looking at it. http://www.scipy.org/Python3k The situation is complicated for us because we have C code that uses some of the removed parts of the C API. The recommended approach (single 2.x source tree, and convert using 2to3 and manual patches for 3.x) isn't quite germane. If the necessary changes are small enough, then it is possible that manual patches or some #ifdefs will be sufficient. Given Rahul's initial findings (thank you, Rahul!), I think this will probably be feasible. If it is not, then we have a significant problem. If we have to have two different code bases, none of the alternatives are very appealing. We do want to support Python 3.0 as soon as possible, but we need more hands and eyes on the code. If Python 3.0 support is important to you, please take a look at Rahul's findings and think about how to address them. Thanks. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ondrej at certik.cz Sun Aug 24 18:05:06 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 25 Aug 2008 00:05:06 +0200 Subject: [Numpy-discussion] [SciPy08] Documentation BoF In-Reply-To: <9457e7c80808220126u765e5b7vc1a0a15ee4a28c11@mail.gmail.com> References: <9457e7c80808220126u765e5b7vc1a0a15ee4a28c11@mail.gmail.com> Message-ID: <85b5c3130808241505p25d8a8b9k35a10a7b63b36e96@mail.gmail.com> On Fri, Aug 22, 2008 at 10:26 AM, St?fan van der Walt wrote: > Hi all, > > This is my personal recollection of the documentation BoF. Feel free to > comment or correct the text below. > > Regards > St?fan > > > Summary of the Documentation Birds-of-a-Feather Session > ======================================================= > > Topics proposed for discussion > ------------------------------ > > - Path towards getting SciPy documented using the online documentation > editor > - Intersphinx: linking Sphinx-using projects > - Culture / methodology: what do we expect from code accepted into > NumPy/SciPy? > - Tools for reading documentation (Help 2.0) > - The complete SciPy book -- how to get there > - NumPy Documentation Standard: is it useful beyond NumPy? > > Conclusions reached > ------------------- > > Documentation format > ```````````````````` > - Question: could we standardise the descriptions of data-types between docs > of different projects? They should overlap to some extent. > > - If we could agree on some common standard (or at least a common subset of > a format) across projects, IPython could parse and display docstrings with > special markup. > > - Matplotlib should be involved in this conversation. They have already > written a vast amount of documentation, and their needs may be different to > those of the SciPy projects. > > Formal requirements > ``````````````````` > For functions going into NumPy and SciPy, we have a certain minimum > documentation requirements. However, we'd rather take a somewhat liberal, > open approach, and accept code with somwhat inferior docstrings, working > with the author to improve them. This way, we remain accessible as a > community, while striving to improve the quality of the code base. > > Marketing SciPy > ``````````````` > The current entry point for SciPy on the web (http://www.scipy.org) may be > confusing to beginners. The web page should be redesigned, in a similar > fashion as the new Sage and code.enthought.com pages, linking to the messy > wiki behind it in a consistent fashion. Joe Harrington may be able to hire > someone to work on this important aspect of SciPy marketing. > > Cross-project documentation searching > ````````````````````````````````````` > Ideally, one should be able to perform a documentation search across several > projects. In order to achieve this, we should negotiate a common convention > for installing Sphinx-generated documentation into a standard location. A > javascript-backed webpage can then be provided to do a search, using the > json indices, or some other means. +1 Currently sphinx can't handle scipy docstrings, can it? It didn't for me at least. It'd be nice if whatever format you agre upon, could work with sphinx's autodoc. Also I am very interested in your web based editing of docstrings, so that people can just fix and improve the docstrings easily. I can imagine something like In [1]: e.series??? in ipython and it would fireup a browser pointing to the docstring and one would just fix it. Since you did most of the work already, maybe the above won't be that hard to do either. Ondrej From millman at berkeley.edu Sun Aug 24 18:20:01 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Sun, 24 Aug 2008 15:20:01 -0700 Subject: [Numpy-discussion] Remove ParametricTestCase from numpy.testing In-Reply-To: <1d36917a0808232223x589f9ea7yac72e9e3fd51aab5@mail.gmail.com> References: <9457e7c80808231649y36a158ecj975bd2f88896e4c1@mail.gmail.com> <1d36917a0808232223x589f9ea7yac72e9e3fd51aab5@mail.gmail.com> Message-ID: On Sat, Aug 23, 2008 at 10:23 PM, Alan McIntyre wrote: > Actually, it was removed right after the nose framework was working, > but I think a decision was made to keep it around until 1.3 in case > somebody was using it, so I put it back. :) After the 1.2 release it > can just be deleted without any issues, I think. Could you through a deprecation warning in? Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From millman at berkeley.edu Sun Aug 24 18:20:47 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Sun, 24 Aug 2008 15:20:47 -0700 Subject: [Numpy-discussion] Remove ParametricTestCase from numpy.testing In-Reply-To: References: <9457e7c80808231649y36a158ecj975bd2f88896e4c1@mail.gmail.com> <1d36917a0808232223x589f9ea7yac72e9e3fd51aab5@mail.gmail.com> Message-ID: On Sun, Aug 24, 2008 at 3:20 PM, Jarrod Millman wrote: > On Sat, Aug 23, 2008 at 10:23 PM, Alan McIntyre wrote: >> Actually, it was removed right after the nose framework was working, >> but I think a decision was made to keep it around until 1.3 in case >> somebody was using it, so I put it back. :) After the 1.2 release it >> can just be deleted without any issues, I think. > > Could you through a deprecation warning in? I suppose throw makes more sense. Sorry, J From robert.kern at gmail.com Sun Aug 24 18:21:05 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 24 Aug 2008 15:21:05 -0700 Subject: [Numpy-discussion] [SciPy08] Documentation BoF In-Reply-To: <85b5c3130808241505p25d8a8b9k35a10a7b63b36e96@mail.gmail.com> References: <9457e7c80808220126u765e5b7vc1a0a15ee4a28c11@mail.gmail.com> <85b5c3130808241505p25d8a8b9k35a10a7b63b36e96@mail.gmail.com> Message-ID: <3d375d730808241521s77cac6a5m126bd4484fc5e777@mail.gmail.com> On Sun, Aug 24, 2008 at 15:05, Ondrej Certik wrote: > Currently sphinx can't handle scipy docstrings, can it? It didn't for > me at least. It'd be nice if whatever format you agre upon, could work > with sphinx's autodoc. We do some preprocessing, I believe. > Also I am very interested in your web based editing of docstrings, so > that people can just fix and improve the docstrings easily. I can > imagine something like > > In [1]: e.series??? > > in ipython and it would fireup a browser pointing to the docstring and > one would just fix it. Since you did most of the work already, maybe > the above won't be that hard to do either. The idea of having a this_docstring_sucks() function was certainly bounced around at the BoF. :-) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dlenski at gmail.com Sun Aug 24 22:03:42 2008 From: dlenski at gmail.com (Dan Lenski) Date: Mon, 25 Aug 2008 02:03:42 +0000 (UTC) Subject: [Numpy-discussion] weights parameter of np.average() doesn't work (BUG?) Message-ID: Hi all, Is there a good reason why the weights parameter of np.average() doesn't broadcast properly? This is with the Ubuntu Hardy x86_64 numpy package, version 1.0.4. In [293]: a=arange(100).reshape(10,10) # Things work fine when weights have the exact same shape as a In [297]: average(a, axis=1, weights=ones((10,10))) Out[297]: array([ 4.5, 14.5, 24.5, 34.5, 44.5, 54.5, 64.5, 74.5, 84.5, 94.5]) # Bizarre and incorrect result with length-10 weight array In [298]: average(a, axis=1, weights=ones(10)) Out[298]: array([[[[[[[[[ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19.], [ 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 30., 31., 32., 33., 34., 35., 36., 37., 38., 39.], [ 40., 41., 42., 43., 44., 45., 46., 47., 48., 49.], [ 50., 51., 52., 53., 54., 55., 56., 57., 58., 59.], [ 60., 61., 62., 63., 64., 65., 66., 67., 68., 69.], [ 70., 71., 72., 73., 74., 75., 76., 77., 78., 79.], [ 80., 81., 82., 83., 84., 85., 86., 87., 88., 89.], [ 90., 91., 92., 93., 94., 95., 96., 97., 98., 99.] ]]]]]]]]) Doing the weighted-sum explicitly works fine for me: In [311]: sum(a*ones(10), axis=-1)/sum(ones(10)) Out[311]: array([ 4.5, 14.5, 24.5, 34.5, 44.5, 54.5, 64.5, 74.5, 84.5, 94.5]) This seems like a bug, especially since average.__doc__ states that: If weights are given, result is: sum(a * weights,axis) / sum(weights,axis), where the weights must have a's shape or be 1D with length the size of a in the given axis. Integer weights are converted to Float. Not specifying weights is equivalent to specifying weights that are all 1. Frankly, I don't even see why weights is constrained to be 1D or the same shape as a... why not anything that's broadcastable to the same shape as a? Dan From charlesr.harris at gmail.com Sun Aug 24 22:57:43 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 24 Aug 2008 20:57:43 -0600 Subject: [Numpy-discussion] weights parameter of np.average() doesn't work (BUG?) In-Reply-To: References: Message-ID: On Sun, Aug 24, 2008 at 8:03 PM, Dan Lenski wrote: > Hi all, > Is there a good reason why the weights parameter of np.average() doesn't > broadcast properly? This is with the Ubuntu Hardy x86_64 numpy package, > version 1.0.4. > > > In [293]: a=arange(100).reshape(10,10) > > # Things work fine when weights have the exact same shape as a > > In [297]: average(a, axis=1, weights=ones((10,10))) > Out[297]: array([ 4.5, 14.5, 24.5, 34.5, 44.5, 54.5, 64.5, 74.5, > 84.5, 94.5]) > > # Bizarre and incorrect result with length-10 weight array > > In [298]: average(a, axis=1, weights=ones(10)) > Out[298]: > array([[[[[[[[[ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.], > [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19.], > [ 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], > [ 30., 31., 32., 33., 34., 35., 36., 37., 38., 39.], > [ 40., 41., 42., 43., 44., 45., 46., 47., 48., 49.], > [ 50., 51., 52., 53., 54., 55., 56., 57., 58., 59.], > [ 60., 61., 62., 63., 64., 65., 66., 67., 68., 69.], > [ 70., 71., 72., 73., 74., 75., 76., 77., 78., 79.], > [ 80., 81., 82., 83., 84., 85., 86., 87., 88., 89.], > [ 90., 91., 92., 93., 94., 95., 96., 97., 98., 99.] > ]]]]]]]]) > This has been fixed in later versions: In [2]: a=arange(100).reshape(10,10) In [3]: average(a, axis=1, weights=ones(10)) Out[3]: array([ 4.5, 14.5, 24.5, 34.5, 44.5, 54.5, 64.5, 74.5, 84.5, 94.5]) However, broadcasting in the axis=None case is not allowed. The thinking is that the weights must be fully specified for the axis (or None) that is being averaged over. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dlenski at gmail.com Sun Aug 24 23:48:54 2008 From: dlenski at gmail.com (Daniel Lenski) Date: Mon, 25 Aug 2008 03:48:54 +0000 (UTC) Subject: [Numpy-discussion] doing zillions of 3x3 determinants fast Message-ID: Hi all, I need to take the determinants of a large number of 3x3 matrices, in order to determine for each of N points, in which of M tetrahedral cells they lie. I arrange the matrices in an ndarray of shape (N,M,5,3,3). As far as I can tell, Numpy doesn't have a function to do determinants over a specific set of axes... so I can't say: dets = np.linalg.det(a, axis1=-1, axis2=-1) Using an explicit Python loop is very slow... doing the determinants of 100,000 random 3x3 matrices in a loop and inserting the results into a preallocated results array takes about 5.1 seconds. So instead I coded up my own routine to do the determinants over the last 2 axes, based on the naive textbook formula for a 3x3 determinant: def det3(ar): a=ar[...,0,0]; b=ar[...,0,1]; c=ar[...,0,2] d=ar[...,1,0]; e=ar[...,1,1]; f=ar[...,1,2] g=ar[...,2,0]; h=ar[...,2,1]; i=ar[...,2,2] return (a*e*i+b*f*g+c*d*h)-(g*e*c+h*f*a+i*d*b) This is *much much* faster... it does the same 100,000 determinants in 0.07 seconds. And the results differ from linalg.det's results by less than 1 part in 10^15 on average (w/ a std dev of about 1 part in 10^13). Does anyone know of any well-written routines to do lots and lots of determinants of a fixed size? My routine has a couple problems I can see: * it's fast enough for 100,000 determinants, but it bogs due to all the temporary arrays when I try to do 1,000,000 determinants (=72 MB array) * I've heard the numerical stability of the naive determinant algorithms is fairly poor, so I'm reluctant to use this function on real data. Any advice will be appreciated! Dan From dlenski at gmail.com Sun Aug 24 23:56:08 2008 From: dlenski at gmail.com (Daniel Lenski) Date: Mon, 25 Aug 2008 03:56:08 +0000 (UTC) Subject: [Numpy-discussion] weights parameter of np.average() doesn't work (BUG?) References: Message-ID: On Sun, 24 Aug 2008 20:57:43 -0600, Charles R Harris wrote: On Sun, Aug 24, 2008 at 8:03 PM, Dan Lenski wrote: > This has been fixed in later versions: > > In [2]: a=arange(100).reshape(10,10) > > In [3]: average(a, axis=1, weights=ones(10)) Out[3]: array([ 4.5, > 14.5, 24.5, 34.5, 44.5, 54.5, 64.5, 74.5, 84.5, 94.5]) Ah, good to know! I guess it's time to upgrade. > However, broadcasting in the axis=None case is not allowed. The thinking > is that the weights must be fully specified for the axis (or None) that > is being averaged over. Hmm, that's a pity :-( My real-world use case has a 2D array of weights, shape (a,b), that I'd like to broadcast to a 3D array, shape (n,a,b), which I average over the first dimension. Currently, I just use repeat() to make it the right size... but this is less-than-ideal for huge arrays. Dan From dlenski at gmail.com Mon Aug 25 00:01:18 2008 From: dlenski at gmail.com (Daniel Lenski) Date: Mon, 25 Aug 2008 04:01:18 +0000 (UTC) Subject: [Numpy-discussion] doing zillions of 3x3 determinants fast References: Message-ID: On Mon, 25 Aug 2008 03:48:54 +0000, Daniel Lenski wrote: > * it's fast enough for 100,000 determinants, but it bogs due to > all the temporary arrays when I try to do 1,000,000 determinants > (=72 MB array) I've managed to reduce the memory usage significantly by getting the number of temporary arrays down to exactly two: def det3(ar): a=ar[...,0,0]; b=ar[...,0,1]; c=ar[...,0,2] d=ar[...,1,0]; e=ar[...,1,1]; f=ar[...,1,2] g=ar[...,2,0]; h=ar[...,2,1]; i=ar[...,2,2] t=a.copy(); t*=e; t*=i; tot =t t=b.copy(); t*=f; t*=g; tot+=t t=c.copy(); t*=d; t*=h; tot+=t t=g.copy(); t*=e; t*=c; tot-=t t=h.copy(); t*=f; t*=a; tot-=t t=i.copy(); t*=d; t*=b; tot-=t return tot Now it runs very fast with 1,000,000 determinants to do (<10X the time required to do 100,000) but I'm still worried about the stability with real-world data. Dan From peridot.faceted at gmail.com Mon Aug 25 00:53:33 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Mon, 25 Aug 2008 00:53:33 -0400 Subject: [Numpy-discussion] doing zillions of 3x3 determinants fast In-Reply-To: References: Message-ID: 2008/8/25 Daniel Lenski : > On Mon, 25 Aug 2008 03:48:54 +0000, Daniel Lenski wrote: >> * it's fast enough for 100,000 determinants, but it bogs due to >> all the temporary arrays when I try to do 1,000,000 determinants >> (=72 MB array) > > I've managed to reduce the memory usage significantly by getting the > number of temporary arrays down to exactly two: > > def det3(ar): > a=ar[...,0,0]; b=ar[...,0,1]; c=ar[...,0,2] > d=ar[...,1,0]; e=ar[...,1,1]; f=ar[...,1,2] > g=ar[...,2,0]; h=ar[...,2,1]; i=ar[...,2,2] > > t=a.copy(); t*=e; t*=i; tot =t > t=b.copy(); t*=f; t*=g; tot+=t > t=c.copy(); t*=d; t*=h; tot+=t > t=g.copy(); t*=e; t*=c; tot-=t > t=h.copy(); t*=f; t*=a; tot-=t > t=i.copy(); t*=d; t*=b; tot-=t > > return tot > > Now it runs very fast with 1,000,000 determinants to do (<10X the time > required to do 100,000) but I'm still worried about the stability with > real-world data. As far as I know, that's the best you can do (though come to think of it, a 3D determinant is a cross-product followed by a dot product, so you might be able to cheat and use some built-in routines). It's a current limitation of numpy that there is not much support for doing many linear algebra problems. We do have perennial discussions about improving support for array-of-matrices calculations, and in fact currently in numpy SVN is code to allow C code doing a 3x3 determinant to be easily made into a "generalized ufunc" that does ufunc-like broadcasting and iteration over all but the last two axes. Anne From charlesr.harris at gmail.com Mon Aug 25 01:44:28 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 24 Aug 2008 23:44:28 -0600 Subject: [Numpy-discussion] weights parameter of np.average() doesn't work (BUG?) In-Reply-To: References: Message-ID: On Sun, Aug 24, 2008 at 9:56 PM, Daniel Lenski wrote: > On Sun, 24 Aug 2008 20:57:43 -0600, Charles R Harris wrote: > On Sun, Aug 24, 2008 at 8:03 PM, Dan Lenski wrote: > > This has been fixed in later versions: > > > > In [2]: a=arange(100).reshape(10,10) > > > > In [3]: average(a, axis=1, weights=ones(10)) Out[3]: array([ 4.5, > > 14.5, 24.5, 34.5, 44.5, 54.5, 64.5, 74.5, 84.5, 94.5]) > > Ah, good to know! I guess it's time to upgrade. > > > However, broadcasting in the axis=None case is not allowed. The thinking > > is that the weights must be fully specified for the axis (or None) that > > is being averaged over. > > Hmm, that's a pity :-( > > My real-world use case has a 2D array of weights, shape (a,b), that I'd > like to broadcast to a 3D array, shape (n,a,b), which I average over the > first dimension. Currently, I just use repeat() to make it the right > size... but this is less-than-ideal for huge arrays. > I expect you can do this explicitly, which is pretty much what average does anyway. Average is pure python, so you won't be losing any efficiency. If you were a bit clearer with your use case I could be more explicit. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Aug 25 01:49:45 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 24 Aug 2008 23:49:45 -0600 Subject: [Numpy-discussion] doing zillions of 3x3 determinants fast In-Reply-To: References: Message-ID: On Sun, Aug 24, 2008 at 9:48 PM, Daniel Lenski wrote: > Hi all, > I need to take the determinants of a large number of 3x3 matrices, in > order to determine for each of N points, in which of M tetrahedral cells > they lie. I arrange the matrices in an ndarray of shape (N,M,5,3,3). > If you step back a bit and describe the actual problem you have, rather than your current solution, it might be that there are algorithms in scipy to solve it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Aug 25 07:18:52 2008 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 25 Aug 2008 11:18:52 +0000 (UTC) Subject: [Numpy-discussion] [SciPy08] Documentation BoF References: <9457e7c80808220126u765e5b7vc1a0a15ee4a28c11@mail.gmail.com> <85b5c3130808241505p25d8a8b9k35a10a7b63b36e96@mail.gmail.com> Message-ID: Mon, 25 Aug 2008 00:05:06 +0200, Ondrej Certik wrote: > On Fri, Aug 22, 2008 at 10:26 AM, St?fan van der Walt > wrote: [clip] >> - Path towards getting SciPy documented using the online documentation >> editor >> - Intersphinx: linking Sphinx-using projects - Culture / methodology: >> what do we expect from code accepted into NumPy/SciPy? >> - Tools for reading documentation (Help 2.0) - The complete SciPy book >> -- how to get there - NumPy Documentation Standard: is it useful beyond >> NumPy? [clip] > > +1 > > Currently sphinx can't handle scipy docstrings, can it? It didn't for me > at least. It'd be nice if whatever format you agre upon, could work with > sphinx's autodoc. Sphinx offers enough hooks in autodoc for proper mangling of the docstrings, so the numpy format we have already agreed on can be used without problems. What we have now can be found here: https://code.launchpad.net/~pauli-virtanen/scipy/numpy-refguide https://code.launchpad.net/~stefanv/scipy/numpy-refguide Splitting the docstrings so that there's one docstring per page proved more difficult to do, especially if we want informative listings of the functions available. Currently our autosummary:: directive + automatically generated RST files can manage it, but it's not very elegant. Another question that pops up is how to integrate Travis's newly free Guide to Numpy to numpy documentation, as there is already some overlap with the docstrings and the reference part of the Guide. The main questions I see is that - What were the conclusions about this in the BoF and other discussions in Scipy'08? - Should we have a separate User manual and a Reference manual, or only a single manual? - What to do with the numpy.doc.* docstrings and Travis's book? There is obviously some overlap here. How to combine the material? - I vaguely remember someone mentioned having done numpybook -> RST conversion. If so, is the result somewhere available? Was something done towards this in the Scipy'08 sprints? I already wrote a rough LyX -> RST converter myself, but would like to avoid further duplication of work... > Also I am very interested in your web based editing of docstrings, so > that people can just fix and improve the docstrings easily. I can > imagine something like > > In [1]: e.series??? > > in ipython and it would fireup a browser pointing to the docstring and > one would just fix it. Since you did most of the work already, maybe the > above won't be that hard to do either. Pointing the browser at %(BASEURL)s/doc/%(obj.__name__)s/ should do it. -- Pauli Virtanen From alan.mcintyre at gmail.com Mon Aug 25 09:36:33 2008 From: alan.mcintyre at gmail.com (Alan McIntyre) Date: Mon, 25 Aug 2008 06:36:33 -0700 Subject: [Numpy-discussion] Remove ParametricTestCase from numpy.testing In-Reply-To: References: <9457e7c80808231649y36a158ecj975bd2f88896e4c1@mail.gmail.com> <1d36917a0808232223x589f9ea7yac72e9e3fd51aab5@mail.gmail.com> Message-ID: <1d36917a0808250636j74f697d5t656188c0a1ac1a5f@mail.gmail.com> On Sun, Aug 24, 2008 at 3:20 PM, Jarrod Millman wrote: > On Sun, Aug 24, 2008 at 3:20 PM, Jarrod Millman wrote: >> On Sat, Aug 23, 2008 at 10:23 PM, Alan McIntyre wrote: >>> Actually, it was removed right after the nose framework was working, >>> but I think a decision was made to keep it around until 1.3 in case >>> somebody was using it, so I put it back. :) After the 1.2 release it >>> can just be deleted without any issues, I think. >> >> Could you through a deprecation warning in? > > I suppose throw makes more sense. > Sorry, > J Done. :) From bsouthey at gmail.com Mon Aug 25 10:31:01 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 25 Aug 2008 09:31:01 -0500 Subject: [Numpy-discussion] Report from SciPy In-Reply-To: <3d375d730808241157s426f947anbb89f9500df7f4a9@mail.gmail.com> References: <48B09FB2.1050009@enthought.com> <3d375d730808241157s426f947anbb89f9500df7f4a9@mail.gmail.com> Message-ID: <48B2C225.4040000@gmail.com> Robert Kern wrote: > On Sun, Aug 24, 2008 at 10:58, Bruce Southey wrote: > >> Hi, >> I think this is a great idea but I am curious about what NumPy will be >> doing with Python 3. The Python 3 final is scheduled for 1st October >> release so is there a policy on handling the migration to Python 3 or >> dual support of the 2 and 3 series? >> > > We're still evaluating the situation. Rahul Garg has spent some time > yesterday at the sprint looking at it. > > http://www.scipy.org/Python3k > > The situation is complicated for us because we have C code that uses > some of the removed parts of the C API. The recommended approach > (single 2.x source tree, and convert using 2to3 and manual patches for > 3.x) isn't quite germane. If the necessary changes are small enough, > then it is possible that manual patches or some #ifdefs will be > sufficient. Given Rahul's initial findings (thank you, Rahul!), I > think this will probably be feasible. If it is not, then we have a > significant problem. If we have to have two different code bases, none > of the alternatives are very appealing. > > We do want to support Python 3.0 as soon as possible, but we need more > hands and eyes on the code. If Python 3.0 support is important to you, > please take a look at Rahul's findings and think about how to address > them. > > Thanks. > > Thanks Rahul for that work as it is very interesting! At present my concern for the Python 3k is only one of inevitability. Bruce From matthieu.brucher at gmail.com Mon Aug 25 10:38:48 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 25 Aug 2008 16:38:48 +0200 Subject: [Numpy-discussion] Report from SciPy In-Reply-To: <48B1A7FB.8040009@american.edu> References: <48B09FB2.1050009@enthought.com> <48B1A7FB.8040009@american.edu> Message-ID: > As a footnote to this query, has the Scons community > addressed the problem Scons had working with VS 2008 (=VS9)? No. I'm still facing the same issues, but I'll try to help David Cournapeau with his work on that (he is trying to simplify all Visual Studio supports). Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher From Daniel.Lenski at seagate.com Mon Aug 25 10:59:01 2008 From: Daniel.Lenski at seagate.com (Dan Lenski) Date: Mon, 25 Aug 2008 14:59:01 +0000 (UTC) Subject: [Numpy-discussion] doing zillions of 3x3 determinants fast References: Message-ID: On Sun, 24 Aug 2008 23:49:45 -0600, Charles R Harris wrote: > On Sun, Aug 24, 2008 at 9:48 PM, Daniel Lenski > wrote: > >> Hi all, >> I need to take the determinants of a large number of 3x3 matrices, in >> order to determine for each of N points, in which of M tetrahedral >> cells they lie. I arrange the matrices in an ndarray of shape >> (N,M,5,3,3). >> >> > If you step back a bit and describe the actual problem you have, rather > than your current solution, it might be that there are algorithms in > scipy to solve it. > Hi Charles, I have an irregular/unstructured mesh of tetrahedral cells, and I need to interpolate some data over this mesh, at arbitrary points within the complete volume. So this is basically 3D interpolation. I've done some looking into this already, and it seems that the only facility for 3D interpolation in Scipy is map_coordinates, which only works on a regular grid. So I have to roll my own interpolation! I start by trying to figure out in which of the tetrahedral cells each interpolation point lies. The standard way to do this is to check that for every three vertices of each tetrahedron (v1,v2,v3), the point in question lies on the same side as the fourth point (v4). This can be checked with: | v1x-x v1y-y v1z-z | | v1x-v4x v1y-v4y v1z-v4z | | v2x-x v2y-y v2z-z | = | v2x-v4x v2y-v4y v2z-v4z | | v3x-x v3y-y v3z-z | | v3x-v4x v3y-v4y v3z-v4z | (Here's a description of a nearly identical algorithm: http:// steve.hollasch.net/cgindex/geometry/ptintet.html) Dan From millman at berkeley.edu Mon Aug 25 11:27:39 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 25 Aug 2008 08:27:39 -0700 Subject: [Numpy-discussion] [SciPy08] Documentation BoF In-Reply-To: References: <9457e7c80808220126u765e5b7vc1a0a15ee4a28c11@mail.gmail.com> <85b5c3130808241505p25d8a8b9k35a10a7b63b36e96@mail.gmail.com> Message-ID: On Mon, Aug 25, 2008 at 4:18 AM, Pauli Virtanen wrote: > - I vaguely remember someone mentioned having done numpybook -> RST > conversion. If so, is the result somewhere available? > Was something done towards this in the Scipy'08 sprints? Yes. Gabriel Gellner is working on this and is pretty far along. He is traveling at the moment, but should be checking something in later this week. Cheers, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From oliphant at enthought.com Mon Aug 25 13:36:03 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 25 Aug 2008 12:36:03 -0500 Subject: [Numpy-discussion] [SciPy08] Documentation BoF In-Reply-To: References: <9457e7c80808220126u765e5b7vc1a0a15ee4a28c11@mail.gmail.com> <85b5c3130808241505p25d8a8b9k35a10a7b63b36e96@mail.gmail.com> Message-ID: <48B2ED83.2010304@enthought.com> Pauli Virtanen wrote: > > - I vaguely remember someone mentioned having done numpybook -> RST > conversion. If so, is the result somewhere available? > Was something done towards this in the Scipy'08 sprints? > > Yes, Gabriel Gellner made progress on this during the sprints, and I asked him to post his scripts for doing it. I don't have his email address currently, but perhaps he will respond. -Travis From dg.numpy at thesamovar.net Mon Aug 25 13:51:31 2008 From: dg.numpy at thesamovar.net (Dan Goodman) Date: Mon, 25 Aug 2008 17:51:31 +0000 (UTC) Subject: [Numpy-discussion] Inplace dot and blas access Message-ID: Hi all, I'm sorry for what is probably a very simple question but despite looking I didn't manage to find anything. I have some code (solving linear differential equation) which is doing the following operation lots of times: S[:] = dot(A,S)+C here S is a (M,N) matrix, A is (M,M) and C is (M,1). M is usually smallish and N is largish. My question is, can this be turned into an inplace operation to save memory and time? As far as I can see, although some array functions allow you to specify the destination of the output, dot always generates a new array. So I would like to have something like this: dot(A,S, out=S) S+=C Ideally, I could do this in one line. I believe the BLAS DGEMM routine does what I want (if I converted C to a matrix rather than a vector). Is there any way to access this using Python and NumPy (e.g. something like numpy.blas.DGEMM) or would I need to use C code to do this? Many thanks in advance, Dan Goodman From Catherine.M.Moroney at jpl.nasa.gov Mon Aug 25 14:29:49 2008 From: Catherine.M.Moroney at jpl.nasa.gov (Catherine Moroney) Date: Mon, 25 Aug 2008 11:29:49 -0700 Subject: [Numpy-discussion] subsampling array without loops Message-ID: <42842135-3C5C-4BD3-97FD-6E6E7CED48DC@jpl.nasa.gov> This almost works. Is there a way to do some masking on tiles, for instance taking the maximum height of each 2x2 square that is an odd number? I've tried playing around with masking and where, but they don't return an array of the original size and shape of "tiles" below. Catherine > Perhaps you could take the relevant slices of the arrays with > appropriate striding: > > a = numpy.arange(128*512).reshape((128, 512)) > top_left = a[::2, ::2] > top_right = a[::2, 1::2] > bottom_left = a[1::2, ::2] > bottom_right = a[1::2, 1::2] > > tiles = numpy.array([top_left, top_right, bottom_left, bottom_right]) > maxes = tiles.max(axis=0) > > Similarly if you want overlapping tiles, you could leave out the > final :2 in the slice specifications above. > > Zach > > > >> I'm looking for a way to acccomplish the following task without lots >> of loops involved, which are really slowing down my code. >> >> I have a 128x512 array which I want to break down into 2x2 squares. >> Then, for each 2x2 square I want to do some simple calculations >> such as finding the maximum value, which I then store in a 64x256 >> array. >> >> Here is the actual code involved. It's only slightly more complex >> than what I described above, since it also involves doing a bit of >> masking on the 2x2 sub-arrays. >> >> Any hints are much appreciated! An inordinate amount of time is >> being spent in this function and another one like it. >> >> Catherine >> >> def calc_sdcm_at_rlra(self,iblock): >> >> npixl = self.nlineht/self.nlinerl >> npixs = self.nsmpht/self.nsmprl >> >> sdcm_out = numpy.array([Constants.CLOUDMASK_NR] \ >> *self.nlinerl*self.nsmprl,'int8') \ >> .reshape(self.nlinerl,self.nsmprl) >> >> for ilinerl in range(0,self.nlinerl): >> for ismprl in range(0,self.nsmprl): >> >> height = self.data[iblock].height >> [ilinerl*2:ilinerl*2+2, \ >> ismprl*2:ismprl*2+2] >> sdcm = self.data[iblock].sdcm[ilinerl*2:ilinerl*2 >> +2, \ >> ismprl*2:ismprl*2+2] >> source = self.data[iblock].heightsrc >> [ilinerl*2:ilinerl*2+2, \ >> >> ismprl*2:ismprl*2+2] >> >> mask1 = (source == Constants.HEIGHT_STEREO) >> mask2 = ( (source == Constants.HEIGHT_SURFACE) | \ >> (source == Constants.HEIGHT_DEFAULT) ) >> >> if (mask1.any()): >> loc = height[mask1].argmax() >> sdcm_out[ilinerl,ismprl] = sdcm[mask1].ravel() >> [loc] >> elif (mask2.any()): >> loc = height[mask2].argmax() >> sdcm_out[ilinerl,ismprl] = sdcm[mask2].ravel() >> [loc] >> >> return sdcm_out > From pgmdevlist at gmail.com Mon Aug 25 14:32:01 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 25 Aug 2008 14:32:01 -0400 Subject: [Numpy-discussion] subsampling array without loops In-Reply-To: <42842135-3C5C-4BD3-97FD-6E6E7CED48DC@jpl.nasa.gov> References: <42842135-3C5C-4BD3-97FD-6E6E7CED48DC@jpl.nasa.gov> Message-ID: <200808251432.02504.pgmdevlist@gmail.com> On Monday 25 August 2008 14:29:49 Catherine Moroney wrote: > This almost works. Is there a way to do some masking on tiles, for > instance taking the maximum height of each 2x2 square that is an > odd number? I've tried playing around with masking and where, but > they don't return an array of the original size and shape of "tiles" > below. Catherine, Sorry, I'm not following you: what were you *actually* doing with masking and where ? A short, self-contained example would be great. From zachary.pincus at yale.edu Mon Aug 25 15:04:03 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Mon, 25 Aug 2008 15:04:03 -0400 Subject: [Numpy-discussion] subsampling array without loops In-Reply-To: <42842135-3C5C-4BD3-97FD-6E6E7CED48DC@jpl.nasa.gov> References: <42842135-3C5C-4BD3-97FD-6E6E7CED48DC@jpl.nasa.gov> Message-ID: <5BA9F37B-1F2F-43C3-BFF1-59C1D4ECCF6A@yale.edu> > This almost works. Is there a way to do some masking on tiles, for > instance taking the maximum height of each 2x2 square that is an > odd number? I've tried playing around with masking and where, but > they don't return an array of the original size and shape of "tiles" > below. Could you provide an example (maybe not code, just descriptive) of what you want to do? I'm not sure what specifically you mean by "height" and "odd" above... Zach > Catherine > >> Perhaps you could take the relevant slices of the arrays with >> appropriate striding: >> >> a = numpy.arange(128*512).reshape((128, 512)) >> top_left = a[::2, ::2] >> top_right = a[::2, 1::2] >> bottom_left = a[1::2, ::2] >> bottom_right = a[1::2, 1::2] >> >> tiles = numpy.array([top_left, top_right, bottom_left, bottom_right]) >> maxes = tiles.max(axis=0) >> >> Similarly if you want overlapping tiles, you could leave out the >> final :2 in the slice specifications above. >> >> Zach >> >> >> >>> I'm looking for a way to acccomplish the following task without lots >>> of loops involved, which are really slowing down my code. >>> >>> I have a 128x512 array which I want to break down into 2x2 squares. >>> Then, for each 2x2 square I want to do some simple calculations >>> such as finding the maximum value, which I then store in a 64x256 >>> array. >>> >>> Here is the actual code involved. It's only slightly more complex >>> than what I described above, since it also involves doing a bit of >>> masking on the 2x2 sub-arrays. >>> >>> Any hints are much appreciated! An inordinate amount of time is >>> being spent in this function and another one like it. >>> >>> Catherine >>> >>> def calc_sdcm_at_rlra(self,iblock): >>> >>> npixl = self.nlineht/self.nlinerl >>> npixs = self.nsmpht/self.nsmprl >>> >>> sdcm_out = numpy.array([Constants.CLOUDMASK_NR] \ >>> *self.nlinerl*self.nsmprl,'int8') \ >>> .reshape(self.nlinerl,self.nsmprl) >>> >>> for ilinerl in range(0,self.nlinerl): >>> for ismprl in range(0,self.nsmprl): >>> >>> height = self.data[iblock].height >>> [ilinerl*2:ilinerl*2+2, \ >>> ismprl*2:ismprl*2+2] >>> sdcm = self.data[iblock].sdcm[ilinerl*2:ilinerl*2 >>> +2, \ >>> ismprl*2:ismprl*2+2] >>> source = self.data[iblock].heightsrc >>> [ilinerl*2:ilinerl*2+2, \ >>> >>> ismprl*2:ismprl*2+2] >>> >>> mask1 = (source == Constants.HEIGHT_STEREO) >>> mask2 = ( (source == Constants.HEIGHT_SURFACE) | \ >>> (source == Constants.HEIGHT_DEFAULT) ) >>> >>> if (mask1.any()): >>> loc = height[mask1].argmax() >>> sdcm_out[ilinerl,ismprl] = sdcm[mask1].ravel() >>> [loc] >>> elif (mask2.any()): >>> loc = height[mask2].argmax() >>> sdcm_out[ilinerl,ismprl] = sdcm[mask2].ravel() >>> [loc] >>> >>> return sdcm_out >> > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From hep.sebastien.binet at gmail.com Mon Aug 25 15:50:13 2008 From: hep.sebastien.binet at gmail.com (Sebastien Binet) Date: Mon, 25 Aug 2008 12:50:13 -0700 Subject: [Numpy-discussion] [SciPy08] Documentation BoF Message-ID: <200808251250.13239.binet@cern.ch> On Monday 25 August 2008 10:36:03 Travis E. Oliphant wrote: > Pauli Virtanen wrote: > > - I vaguely remember someone mentioned having done numpybook -> RST > > conversion. If so, is the result somewhere available? > > Was something done towards this in the Scipy'08 sprints? > > Yes, Gabriel Gellner made progress on this during the sprints, and I > asked him to post his scripts for doing it. I don't have his email > address currently, but perhaps he will respond. Note that a set of latex->reST converters seem to already exist there: http://svn.python.org/view/doctools/converter/ (haven't tried myself, so, YMMV) cheers, sebastien. -- ################################### # Dr. Sebastien Binet # Lawrence Berkeley National Lab. # 1 Cyclotron Road # Berkeley, CA 94720 ################################### From ColesworthyD at Otologics.com Mon Aug 25 19:25:08 2008 From: ColesworthyD at Otologics.com (Dan Colesworthy) Date: Mon, 25 Aug 2008 17:25:08 -0600 Subject: [Numpy-discussion] Converting audio data to array for further processing Message-ID: <86989B801589354BBDE224488335AAF4012576@xchange2.otologics.com> Hello all, I need an efficient way to convert 24 bit signed audio data to a numpy array for further processing. The data will be in a .wav file, and can be recovered via the python wave module. At that point it is a byte string - likely in little endian order. Somewhere in that data will be a 1 second length tone of prescribed but varying amplitude and frequency which must be measured for amplitude, frequency, duration, thd% etc. Also need to make the conversion quickly - over relatively large amounts of data. Something like 5 seconds sampled at 32000 samples/second. This same job will have to be repeated > 300 times as part of a test module. Thanks in advance for any and all ideas. Dan Colesworthy Otologics LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Mon Aug 25 23:03:53 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Mon, 25 Aug 2008 23:03:53 -0400 Subject: [Numpy-discussion] subsampling array without loops In-Reply-To: <93455E9E-3297-40CF-8796-ADEACB11A182@jpl.nasa.gov> References: <93455E9E-3297-40CF-8796-ADEACB11A182@jpl.nasa.gov> Message-ID: 2008/8/22 Catherine Moroney : > I'm looking for a way to acccomplish the following task without lots > of loops involved, which are really slowing down my code. > > I have a 128x512 array which I want to break down into 2x2 squares. > Then, for each 2x2 square I want to do some simple calculations > such as finding the maximum value, which I then store in a 64x256 > array. You should be able to do some of this with reshape and transpose: In [1]: import numpy as np In [3]: A = np.zeros((128,512)) In [4]: B = np.reshape(A,(64,2,256,2)) In [5]: C = np.transpose(B,(0,2,1,3)) In [6]: C.shape Out[6]: (64, 256, 2, 2) Now C[i,j] is the 2 by 2 array [ [A[2*i,2*j], A[2*i,2*j+1]], [A[2*i+1,2*j], A[2*i+1, 2*j+1]] ] (or maybe I got those indices the wrong way around.) Now most operations you want to do on the two-by-two matrices can be done, using ufuncs, on the whole array at once. For example if you wanted the max of each two-by-two matrix, you'd write: In [7]: D = np.amax(np.amax(C,axis=-1),axis=-1) In [8]: D.shape Out[8]: (64, 256) Anne From alan.mcintyre at gmail.com Mon Aug 25 23:16:26 2008 From: alan.mcintyre at gmail.com (Alan McIntyre) Date: Mon, 25 Aug 2008 20:16:26 -0700 Subject: [Numpy-discussion] Buildbot failure Message-ID: <1d36917a0808252016l5965625bgc1eed5f47e88e7ce@mail.gmail.com> In case it hasn't been noted yet, three of the buildbots (Windows_XP_x86_64_MSVC, Linux_SPARC_64_Debian, Linux_SPARC_64_Debian_gcc4) are failing to build. The Linux builds have the error: gcc: build/src.linux-sparc64-2.4/numpy/core/src/umathmodule.c numpy/core/src/umathmodule.c.src:332: error: static declaration of 'trunc' follows non-static declaration The MSVC build appears to fail with build\src.win32-2.5\numpy\core\include\numpy\__umath_generated.c(580) : error C2065: 'truncf' : undeclared identifier From charlesr.harris at gmail.com Mon Aug 25 23:20:25 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 25 Aug 2008 21:20:25 -0600 Subject: [Numpy-discussion] Buildbot failure In-Reply-To: <1d36917a0808252016l5965625bgc1eed5f47e88e7ce@mail.gmail.com> References: <1d36917a0808252016l5965625bgc1eed5f47e88e7ce@mail.gmail.com> Message-ID: On Mon, Aug 25, 2008 at 9:16 PM, Alan McIntyre wrote: > In case it hasn't been noted yet, three of the buildbots > (Windows_XP_x86_64_MSVC, Linux_SPARC_64_Debian, > Linux_SPARC_64_Debian_gcc4) are failing to build. The Linux builds > have the error: > > gcc: build/src.linux-sparc64-2.4/numpy/core/src/umathmodule.c > numpy/core/src/umathmodule.c.src:332: error: static declaration of > 'trunc' follows non-static declaration > > The MSVC build appears to fail with > > build\src.win32-2.5\numpy\core\include\numpy\__umath_generated.c(580) > : error C2065: 'truncf' : undeclared identifier > _ Blame David ;) Anyway, I expect he will fix it up soon. Maybe we should shove some of these trickier fixes off to the next release? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Tue Aug 26 00:45:12 2008 From: cournape at gmail.com (David Cournapeau) Date: Mon, 25 Aug 2008 21:45:12 -0700 Subject: [Numpy-discussion] Converting audio data to array for further processing In-Reply-To: <86989B801589354BBDE224488335AAF4012576@xchange2.otologics.com> References: <86989B801589354BBDE224488335AAF4012576@xchange2.otologics.com> Message-ID: <5b8d13220808252145i5fc928b2y69024f8c1dc86d97@mail.gmail.com> On Mon, Aug 25, 2008 at 4:25 PM, Dan Colesworthy wrote: > Hello all, > > > > I need an efficient way to convert 24 bit signed audio data to a numpy array > for further processing. The data will be in a .wav file, and can be > recovered via the python wave module. At that point it is a byte string ? > likely in little endian order. There are some wav io facilities in scipy.io. If you need more advanced facilities and ndo not ming using LGPL-licensed code, there is scikits.audiolab which is a wrapper around sndfile and supports a large range of audio file formats: http://www.mega-nerd.com/libsndfile/ http://projects.scipy.org/scipy/scikits david From beckers at orn.mpg.de Tue Aug 26 04:27:42 2008 From: beckers at orn.mpg.de (Gabriel J.L. Beckers) Date: Tue, 26 Aug 2008 10:27:42 +0200 Subject: [Numpy-discussion] Converting audio data to array for further processing In-Reply-To: <86989B801589354BBDE224488335AAF4012576@xchange2.otologics.com> References: <86989B801589354BBDE224488335AAF4012576@xchange2.otologics.com> Message-ID: <1219739262.6257.19.camel@gabriel-desktop> I would go with scikits.audiolab to get you wav data into numpy arrays, as David mentioned. I use it all the time for bioacoustic research and it is great (unfortunately still in beta, but works well in practice). Reading sound data of 5 seconds @ 32000 Hz and finding/measuring your tones fast should not be a problem with numpy(.fft) and audiolab. If you need very accurate tone frequency estimation (i.e. more accurate than a simple fft can give you): I implemented in numpy an algorithm (from the speech analysis program Praat) to do this, but as yet it is not optimized and very slow. Best, Gabriel On Mon, 2008-08-25 at 17:25 -0600, Dan Colesworthy wrote: > Hello all, > > > > I need an efficient way to convert 24 bit signed audio data to a numpy > array for further processing. The data will be in a .wav file, and > can be recovered via the python wave module. At that point it is a > byte string ? likely in little endian order. Somewhere in that data > will be a 1 second length tone of prescribed but varying amplitude and > frequency which must be measured for amplitude, frequency, duration, > thd% etc. > > > > Also need to make the conversion quickly ? over relatively large > amounts of data. Something like 5 seconds sampled at 32000 > samples/second. This same job will have to be repeated > 300 times as > part of a test module. > > > > Thanks in advance for any and all ideas. > > > > Dan Colesworthy > > Otologics LLC > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From neilcrighton at gmail.com Tue Aug 26 06:58:03 2008 From: neilcrighton at gmail.com (Neil Crighton) Date: Tue, 26 Aug 2008 11:58:03 +0100 Subject: [Numpy-discussion] [SciPy08] Documentation BoF Message-ID: <63751c30808260358y2010b7a0s6c5ebffbe40d181b@mail.gmail.com> > > - Should we have a separate User manual and a Reference manual, or only > a single manual? > Are there still plans to write a 10 page 'Getting started with NumPy' document? I think this would be very useful. Ideally a 'getting started' document, the docstrings, and a reference manual is all the documentation you'd need. I try to avoid reading long user manuals unless I'm forced to, but I don't know what other people do. Neil -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Tue Aug 26 07:24:24 2008 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 26 Aug 2008 11:24:24 +0000 (UTC) Subject: [Numpy-discussion] Reusing Guide to Numpy (in the doc marathon) References: <9457e7c80808220126u765e5b7vc1a0a15ee4a28c11@mail.gmail.com> <85b5c3130808241505p25d8a8b9k35a10a7b63b36e96@mail.gmail.com> Message-ID: Mon, 25 Aug 2008 11:18:52 +0000, Pauli Virtanen wrote: [clip] > https://code.launchpad.net/~pauli-virtanen/scipy/numpy-refguide > > https://code.launchpad.net/~stefanv/scipy/numpy-refguide For coordination: I'm starting to add and Sphinxify relevant C-API reference documentation parts from Travis's book [1] (numpy/doc/numpybook/ capi.lyx) to the above "reference manual". Big thanks to Travis for making the book free! Good ideas what to do with the overlap in numpy.doc.* docstrings and the material in the book is also appreciated; I wasn't present in Scipy'08, so I'd be interested in hearing what kind of conclusions were reached there. I think it would also be great if some of the documentation in the book eventually hit the Numpy docstrings: I'd encourage everyone currently writing stuff in the Documentation Marathon to take a look at the book: the material describing functions etc. looks excellent for making good docstrings. .. [1] http://www.tramy.us/numpybook.pdf (Note that this PDF is apparently slightly old and still says the book is released only in 2010 and not in 2008 like the version in numpy SVN says). -- Pauli Virtanen From simon.palmer at gmail.com Tue Aug 26 09:52:14 2008 From: simon.palmer at gmail.com (SimonPalmer) Date: Tue, 26 Aug 2008 06:52:14 -0700 (PDT) Subject: [Numpy-discussion] Can't get to numpy.org or scipy.org today... Message-ID: <11717cb6-c97c-413f-8262-76629ffe9121@w39g2000prb.googlegroups.com> ...are they down? From william at resolversystems.com Tue Aug 26 10:22:46 2008 From: william at resolversystems.com (William Reade) Date: Tue, 26 Aug 2008 15:22:46 +0100 Subject: [Numpy-discussion] numpy.core.numerictypes questions In-Reply-To: <9457e7c80808221259r7cdf8640kb7dc7bc943fe8900@mail.gmail.com> References: <48AD4081.1050105@resolversystems.com> <9457e7c80808221259r7cdf8640kb7dc7bc943fe8900@mail.gmail.com> Message-ID: <48B411B6.70207@resolversystems.com> That's great news -- thank you very much William St?fan van der Walt wrote: > 2008/8/21 William Reade : > >> Line 532 of numerictypes.py reads "_unicodesize = >> array('u','U1').itemsize", but _unicodesize itself does not appear to be >> referenced anywhere else. My questions are: >> > > I removed this in r5674. > > Regards > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From millman at berkeley.edu Tue Aug 26 10:38:53 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 26 Aug 2008 07:38:53 -0700 Subject: [Numpy-discussion] Can't get to numpy.org or scipy.org today... In-Reply-To: <11717cb6-c97c-413f-8262-76629ffe9121@w39g2000prb.googlegroups.com> References: <11717cb6-c97c-413f-8262-76629ffe9121@w39g2000prb.googlegroups.com> Message-ID: On Tue, Aug 26, 2008 at 6:52 AM, SimonPalmer wrote: > ...are they down? They are back up now. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From pav at iki.fi Tue Aug 26 13:43:21 2008 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 26 Aug 2008 17:43:21 +0000 (UTC) Subject: [Numpy-discussion] Reusing Guide to Numpy (in the doc marathon) References: <9457e7c80808220126u765e5b7vc1a0a15ee4a28c11@mail.gmail.com> <85b5c3130808241505p25d8a8b9k35a10a7b63b36e96@mail.gmail.com> Message-ID: Tue, 26 Aug 2008 11:24:24 +0000, Pauli Virtanen wrote: [clip] > For coordination: I'm starting to add and Sphinxify relevant C-API > reference documentation parts from Travis's book [1] > (numpy/doc/numpybook/ capi.lyx) to the above "reference manual". Ok, one Sphinxified capi.lyx is here: [1], using this converter [2]. But it required largish amounts of manual editing on top, so I'll leave numpybook.lyx for later or for other people to do -- there should be a way to do much of the cross-referencing & function annotation drudgework automatically. .. [1] http://www.elisanet.fi/ptvirtan/tmp/numpy-refguide/_sources/c-api/index.txt .. [2] http://www.elisanet.fi/ptvirtan/tmp/lyx2rst -- Pauli Virtanen From ryan.neve at gmail.com Tue Aug 26 14:23:46 2008 From: ryan.neve at gmail.com (Ryan Neve) Date: Tue, 26 Aug 2008 14:23:46 -0400 Subject: [Numpy-discussion] converting array of timedelta to array of integers. Message-ID: <4b9c95910808261123x3d8103bfud4d23eb2d45e637a@mail.gmail.com> Apologies in advance if this is an obvious answer, but I'm new to most of this. My overall goal is to produce a contour plot of some irregular time series data. I've imported the data from mySQL into three arrays x,y,and z where x is an array of datetime.timedelta objects. I need an array of the values of x converted into an array of integers representing the number of minutes in the timedelta. I've tried a few things, and spent hours searching for examples, but to no avail. Once I get the x array as integers, I can use griddata() -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Tue Aug 26 14:28:10 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 26 Aug 2008 14:28:10 -0400 Subject: [Numpy-discussion] Compile error w/ rev5703 Message-ID: <200808261428.10699.pgmdevlist@gmail.com> Here's the interesting part: -------------------- creating build/temp.linux-x86_64-2.4/build/src.linux-x86_64-2.4/numpy/core/src compile options: '-Ibuild/src.linux-x86_64-2.4/numpy/core/src -Inumpy/core/include -Ibuild/src.linux-x86_64-2.4/numpy/core/include/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' x86_64-pc-linux-gnu-gcc: build/src.linux-x86_64-2.4/numpy/core/src/umathmodule.c numpy/core/src/umathmodule.c.src:332: error: static declaration of ?trunc? follows non-static declaration ---------------------- Here's some generic info about the system. gcc (GCC) 4.1.2 (Gentoo 4.1.2 p1.0.1) Copyright (C) 2006 Free Software Foundation, Inc. C compiler: x86_64-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC compile options: '-c' x86_64-pc-linux-gnu-gcc: _configtest.c x86_64-pc-linux-gnu-gcc -pthread _configtest.o -L/usr/lib64 -lf77blas -lcblas -latlas -o _configtest ATLAS version 3.8.2 built by root on Thu Jun 19 14:40:27 EDT 2008: UNAME : Linux back2tux 2.6.22-suspend2-r1 #8 Fri Feb 15 17:00:17 EST 2008 x86_64 Mobile AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux INSTFLG : -1 0 -a 1 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_HAMMER -DATL_CPUMHZ=2205 -DATL_SSE2 -DATL_SSE1 -DATL_3DNow -DATL_USE64BITS -DATL_GAS_x8664 F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 262144 F77 : gfortran, version GNU Fortran 95 (GCC) 4.1.2 (Gentoo 4.1.2 p1.0.1) F77FLAGS : -O2 -m64 SMC : x86_64-pc-linux-gnu-gcc, version x86_64-pc-linux-gnu-gcc (GCC) 4.1.2 (Gentoo 4.1.2 p1.0.1) SMCFLAGS : -march=k8 -O2 -pipe -m64 SKC : x86_64-pc-linux-gnu-gcc, version x86_64-pc-linux-gnu-gcc (GCC) 4.1.2 (Gentoo 4.1.2 p1.0.1) SKCFLAGS : -march=k8 -O2 -pipe -m64 From kwgoodman at gmail.com Tue Aug 26 14:47:33 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 26 Aug 2008 11:47:33 -0700 Subject: [Numpy-discussion] converting array of timedelta to array of integers. In-Reply-To: <4b9c95910808261123x3d8103bfud4d23eb2d45e637a@mail.gmail.com> References: <4b9c95910808261123x3d8103bfud4d23eb2d45e637a@mail.gmail.com> Message-ID: On Tue, Aug 26, 2008 at 11:23 AM, Ryan Neve wrote: > Apologies in advance if this is an obvious answer, but I'm new to most of > this. > > My overall goal is to produce a contour plot of some irregular time series > data. > > I've imported the data from mySQL into three arrays x,y,and z where x is an > array of datetime.timedelta objects. > > I need an array of the values of x converted into an array of integers > representing the number of minutes in the timedelta. > > I've tried a few things, and spent hours searching for examples, but to no > avail. > > Once I get the x array as integers, I can use griddata() I got stuck: >> import numpy as np >> import datetime >> >> x = np.array([datetime.timedelta(1), datetime.timedelta(2)]) >> def convert(x): ...: return x.days * 24.0 * 60 + x.seconds / 60.0 + x.microseconds / 6000.0 ...: >> convert(x[0]) 1440.0 >> convert(x[1]) 2880.0 >> np.apply_along_axis(convert, 0, x) --------------------------------------------------------------------------- AttributeError: 'numpy.ndarray' object has no attribute 'days' From kwgoodman at gmail.com Tue Aug 26 15:03:31 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 26 Aug 2008 12:03:31 -0700 Subject: [Numpy-discussion] converting array of timedelta to array of integers. In-Reply-To: References: <4b9c95910808261123x3d8103bfud4d23eb2d45e637a@mail.gmail.com> Message-ID: On Tue, Aug 26, 2008 at 11:47 AM, Keith Goodman wrote: > On Tue, Aug 26, 2008 at 11:23 AM, Ryan Neve wrote: >> Apologies in advance if this is an obvious answer, but I'm new to most of >> this. >> >> My overall goal is to produce a contour plot of some irregular time series >> data. >> >> I've imported the data from mySQL into three arrays x,y,and z where x is an >> array of datetime.timedelta objects. >> >> I need an array of the values of x converted into an array of integers >> representing the number of minutes in the timedelta. >> >> I've tried a few things, and spent hours searching for examples, but to no >> avail. >> >> Once I get the x array as integers, I can use griddata() > > I got stuck: > >>> import numpy as np >>> import datetime >>> >>> x = np.array([datetime.timedelta(1), datetime.timedelta(2)]) >>> def convert(x): > ...: return x.days * 24.0 * 60 + x.seconds / 60.0 + > x.microseconds / 6000.0 > ...: >>> convert(x[0]) > 1440.0 >>> convert(x[1]) > 2880.0 >>> np.apply_along_axis(convert, 0, x) > --------------------------------------------------------------------------- > AttributeError: 'numpy.ndarray' object has no attribute 'days' This is the first time I've used vectorize. Seems to work: >> vconvert = np.vectorize(convert) >> vconvert(x) array([ 1440., 2880.]) From fperez.net at gmail.com Tue Aug 26 20:23:54 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 26 Aug 2008 17:23:54 -0700 Subject: [Numpy-discussion] numpy trunk build broken on 32-bit ubuntu Message-ID: Howdy, building numpy from trunk right now (r5708) on a 32-bit ubuntu box gives: compiling C sources C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC creating build/temp.linux-i686-2.5/build creating build/temp.linux-i686-2.5/build/src.linux-i686-2.5 creating build/temp.linux-i686-2.5/build/src.linux-i686-2.5/numpy creating build/temp.linux-i686-2.5/build/src.linux-i686-2.5/numpy/core creating build/temp.linux-i686-2.5/build/src.linux-i686-2.5/numpy/core/src compile options: '-Ibuild/src.linux-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.linux-i686-2.5/numpy/core/include/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: build/src.linux-i686-2.5/numpy/core/src/umathmodule.c numpy/core/src/umathmodule.c.src:332: error: static declaration of 'trunc' follows non-static declaration numpy/core/src/umathmodule.c.src: In function 'trunc': numpy/core/src/umathmodule.c.src:333: warning: unused variable 'r' numpy/core/src/umathmodule.c.src:333: warning: unused variable 'y' numpy/core/src/umathmodule.c.src:332: error: static declaration of 'trunc' follows non-static declaration numpy/core/src/umathmodule.c.src: In function 'trunc': numpy/core/src/umathmodule.c.src:333: warning: unused variable 'r' numpy/core/src/umathmodule.c.src:333: warning: unused variable 'y' error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -Ibuild/src.linux-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.linux-i686-2.5/numpy/core/include/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c build/src.linux-i686-2.5/numpy/core/src/umathmodule.c -o build/temp.linux-i686-2.5/build/src.linux-i686-2.5/numpy/core/src/umathmodule.o" failed with exit status 1 Notes (Matthew Brett found all of this): - on 64-bit fedora it builds fine - calling a similar test on 32-bit ubuntu with "gcc -std=c99" works. Perhaps the build is using something similar... - Not that I want to blame anyone, but I suspect this has something to do with David C's recent checkins: ------------------------------------------------------------------------ r5699 | cdavid | 2008-08-25 15:05:29 -0700 (Mon, 25 Aug 2008) | 2 lines Add a trunc function in umath module. ------------------------------------------------------------------------ r5698 | cdavid | 2008-08-25 15:05:21 -0700 (Mon, 25 Aug 2008) | 2 lines Add pure C trunc function implementations for platform which do not have it. as that is where the errror is coming from. Cheers, f From charlesr.harris at gmail.com Tue Aug 26 20:41:44 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 26 Aug 2008 18:41:44 -0600 Subject: [Numpy-discussion] numpy trunk build broken on 32-bit ubuntu In-Reply-To: References: Message-ID: On Tue, Aug 26, 2008 at 6:23 PM, Fernando Perez wrote: > Howdy, > > building numpy from trunk right now (r5708) on a 32-bit ubuntu box gives: > > compiling C sources > C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 > -Wall -Wstrict-prototypes -fPIC > > creating build/temp.linux-i686-2.5/build > creating build/temp.linux-i686-2.5/build/src.linux-i686-2.5 > creating build/temp.linux-i686-2.5/build/src.linux-i686-2.5/numpy > creating build/temp.linux-i686-2.5/build/src.linux-i686-2.5/numpy/core > creating build/temp.linux-i686-2.5/build/src.linux-i686-2.5/numpy/core/src > compile options: '-Ibuild/src.linux-i686-2.5/numpy/core/src > -Inumpy/core/include > -Ibuild/src.linux-i686-2.5/numpy/core/include/numpy -Inumpy/core/src > -Inumpy/core/include -I/usr/include/python2.5 -c' > gcc: build/src.linux-i686-2.5/numpy/core/src/umathmodule.c > numpy/core/src/umathmodule.c.src:332: error: static declaration of > 'trunc' follows non-static declaration > numpy/core/src/umathmodule.c.src: In function 'trunc': > numpy/core/src/umathmodule.c.src:333: warning: unused variable 'r' > numpy/core/src/umathmodule.c.src:333: warning: unused variable 'y' > numpy/core/src/umathmodule.c.src:332: error: static declaration of > 'trunc' follows non-static declaration > numpy/core/src/umathmodule.c.src: In function 'trunc': > numpy/core/src/umathmodule.c.src:333: warning: unused variable 'r' > numpy/core/src/umathmodule.c.src:333: warning: unused variable 'y' > error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv > -O2 -Wall -Wstrict-prototypes -fPIC > -Ibuild/src.linux-i686-2.5/numpy/core/src -Inumpy/core/include > -Ibuild/src.linux-i686-2.5/numpy/core/include/numpy -Inumpy/core/src > -Inumpy/core/include -I/usr/include/python2.5 -c > build/src.linux-i686-2.5/numpy/core/src/umathmodule.c -o > > build/temp.linux-i686-2.5/build/src.linux-i686-2.5/numpy/core/src/umathmodule.o" > failed with exit status 1 > > > > Notes (Matthew Brett found all of this): > > - on 64-bit fedora it builds fine > - calling a similar test on 32-bit ubuntu with "gcc -std=c99" works. > Perhaps the build is using something similar... > > - Not that I want to blame anyone, but I suspect this has something to > do with David C's recent checkins: > > ------------------------------------------------------------------------ > r5699 | cdavid | 2008-08-25 15:05:29 -0700 (Mon, 25 Aug 2008) | 2 lines > > Add a trunc function in umath module. > > ------------------------------------------------------------------------ > r5698 | cdavid | 2008-08-25 15:05:21 -0700 (Mon, 25 Aug 2008) | 2 lines > > Add pure C trunc function implementations for platform which do not have > it. > > > as that is where the errror is coming from. > I've been thinking of reverting the changes rather than waiting for David to get back. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Aug 26 20:49:38 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 26 Aug 2008 19:49:38 -0500 Subject: [Numpy-discussion] numpy trunk build broken on 32-bit ubuntu In-Reply-To: References: Message-ID: <3d375d730808261749r32bebf1co953cb1e0f7a93040@mail.gmail.com> On Tue, Aug 26, 2008 at 19:41, Charles R Harris wrote: > I've been thinking of reverting the changes rather than waiting for David to > get back. Go ahead. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pgmdevlist at gmail.com Tue Aug 26 20:44:16 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 26 Aug 2008 20:44:16 -0400 Subject: [Numpy-discussion] numpy trunk build broken on 32-bit ubuntu In-Reply-To: References: Message-ID: <200808262044.16628.pgmdevlist@gmail.com> On Tuesday 26 August 2008 20:41:44 Charles R Harris wrote: > I've been thinking of reverting the changes rather than waiting for David > to get back. As mentioned in a previous email, that fails on x86_64 AMD as well, same place. Commenting out `trunc` in umathmodule.c.src does the trick. From matthew.brett at gmail.com Wed Aug 27 06:03:14 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 27 Aug 2008 03:03:14 -0700 Subject: [Numpy-discussion] Array subclassing - question for Travis Message-ID: <1e2af89e0808270303s632067b2nc0951290785f18cd@mail.gmail.com> Hi Travis and team, I am just writing some docs for subclassing, and ran into some behavior I didn't understand: In [143]: class A(np.ndarray): pass In [144]: arr = np.arange(5) In [145]: obj = arr.copy().view(A) In [146]: type(obj) Out[146]: In [147]: obj.__array_priority__ # shouldn't this be 1.0 by default (from the numpy book)? Out[147]: 0.0 In [148]: type(np.multiply(arr, obj)) # this is what I expected Out[148]: In [149]: type(np.multiply.outer(arr, obj)) # this is not - I expected class A again Out[149]: In [151]: class A(np.ndarray): __array_priority__ = 20.0 # setting array priority does not affect behavior In [152]: obj = arr.copy().view(A) In [153]: type(np.multiply.outer(arr, obj)) Out[153]: Is this what you would expect? Thanks for clarification, and apologies if I misunderstood... Matthew From cgoued at gmail.com Wed Aug 27 10:48:48 2008 From: cgoued at gmail.com (Claude Gouedard) Date: Wed, 27 Aug 2008 16:48:48 +0200 Subject: [Numpy-discussion] About asarray (list) Message-ID: <3f5fb21a0808270748h2541f09bo7f51869873ba5c7e@mail.gmail.com> Hi , I'm just surprised by the behaviour of numpy.asarray on lists. Can someone comment this : ===================== a=(1) aa=asarray(a) print aa.size , aa.shape >> 1 ( ) ===================== The shape doesnot reflect the actual size. If a=(1,2) there is no problem . ===================== aa=asarray((1,2)) print aa.size , aa.shape >> 2 (2, ) ===================== ... same for sequences: ===================== aa=asarray( [1] ) print aa.size , aa.shape >> 1 (1, ) ===================== Thanks Claude Gouedard -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmetz at astro.uni-bonn.de Wed Aug 27 10:52:49 2008 From: mmetz at astro.uni-bonn.de (Manuel Metz) Date: Wed, 27 Aug 2008 16:52:49 +0200 Subject: [Numpy-discussion] About asarray (list) In-Reply-To: <3f5fb21a0808270748h2541f09bo7f51869873ba5c7e@mail.gmail.com> References: <3f5fb21a0808270748h2541f09bo7f51869873ba5c7e@mail.gmail.com> Message-ID: <48B56A41.3080405@astro.uni-bonn.de> Claude Gouedard wrote: > Hi , > > I'm just surprised by the behaviour of numpy.asarray on lists. > > Can someone comment this : > ===================== > a=(1) > aa=asarray(a) > print aa.size , aa.shape >>> 1 ( ) > ===================== > > The shape doesnot reflect the actual size. Because a is not a tuple, but an int !!! Try: a=(1) print a, type(a) Manuel > If a=(1,2) there is no problem . > ===================== > aa=asarray((1,2)) > print aa.size , aa.shape >>> 2 (2, ) > ===================== > ... same for sequences: > ===================== > aa=asarray( [1] ) > print aa.size , aa.shape >>> 1 (1, ) > ===================== > > Thanks > Claude Gouedard > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From dagss at student.matnat.uio.no Wed Aug 27 11:00:14 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 27 Aug 2008 17:00:14 +0200 Subject: [Numpy-discussion] About asarray (list) In-Reply-To: <48B56A41.3080405@astro.uni-bonn.de> References: <3f5fb21a0808270748h2541f09bo7f51869873ba5c7e@mail.gmail.com> <48B56A41.3080405@astro.uni-bonn.de> Message-ID: <48B56BFE.3030302@student.matnat.uio.no> Manuel Metz wrote: > Claude Gouedard wrote: >> Hi , >> >> I'm just surprised by the behaviour of numpy.asarray on lists. >> >> Can someone comment this : >> ===================== >> a=(1) >> aa=asarray(a) >> print aa.size , aa.shape >>>> 1 ( ) >> ===================== >> >> The shape doesnot reflect the actual size. > > Because a is not a tuple, but an int !!! Try: > a=(1) > print a, type(a) > If you want a tuple with one element, try a = (1,) -- Dag Sverre From pav at iki.fi Wed Aug 27 10:55:18 2008 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 27 Aug 2008 14:55:18 +0000 (UTC) Subject: [Numpy-discussion] About asarray (list) References: <3f5fb21a0808270748h2541f09bo7f51869873ba5c7e@mail.gmail.com> Message-ID: Wed, 27 Aug 2008 16:48:48 +0200, Claude Gouedard wrote: > Hi , > > I'm just surprised by the behaviour of numpy.asarray on lists. > > Can someone comment this : > ===================== > a=(1) > aa=asarray(a) > print aa.size , aa.shape >>> 1 ( ) > ===================== () are not list delimiters in Python; [] are, see [1,2]: >>> a = (1) >>> type(a) >>> a = [1] >>> type(a) .. [1] http://docs.python.org/tut/node5.html#SECTION005140000000000000000 .. [2] http://docs.python.org/tut/node7.html#SECTION007300000000000000000 -- Pauli Virtanen From cgoued at gmail.com Wed Aug 27 11:01:15 2008 From: cgoued at gmail.com (Claude Gouedard) Date: Wed, 27 Aug 2008 17:01:15 +0200 Subject: [Numpy-discussion] About asarray (list) Message-ID: <3f5fb21a0808270801y78f74857p97116bac3b4add3e@mail.gmail.com> Ok , The same for asarray(1) .. The problem is that aa=asarray(1) is an numpy.array (right ? ) with a size 1 and a shape ( ) ! No surprising ? Claude -------------- next part -------------- An HTML attachment was scrubbed... URL: From dagss at student.matnat.uio.no Wed Aug 27 11:20:12 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 27 Aug 2008 17:20:12 +0200 Subject: [Numpy-discussion] About asarray (list) In-Reply-To: <3f5fb21a0808270801y78f74857p97116bac3b4add3e@mail.gmail.com> References: <3f5fb21a0808270801y78f74857p97116bac3b4add3e@mail.gmail.com> Message-ID: <48B570AC.4080005@student.matnat.uio.no> Claude Gouedard wrote: > Ok , > The same for asarray(1) .. > The problem is that > aa=asarray(1) is an numpy.array (right ? ) with a size 1 and a shape ( ) ! > No surprising ? I think it is considered a 0-dimensional array (= a scalar). -- Dag Sverre From dalcinl at gmail.com Wed Aug 27 11:44:44 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 27 Aug 2008 12:44:44 -0300 Subject: [Numpy-discussion] About asarray (list) In-Reply-To: <3f5fb21a0808270801y78f74857p97116bac3b4add3e@mail.gmail.com> References: <3f5fb21a0808270801y78f74857p97116bac3b4add3e@mail.gmail.com> Message-ID: On Wed, Aug 27, 2008 at 12:01 PM, Claude Gouedard wrote: > Ok , > The same for asarray(1) .. > The problem is that > aa=asarray(1) is an numpy.array (right ? ) with a size 1 and a shape ( ) ! > No surprising ? For me, this is not surprising at all :-) . Furthermore, if you try In [1]: import numpy as np In [2]: a = np.asarray(1) In [3]: a.flags Out[3]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False You see that this 0-dim array is WRITEABLE, this is a nice way to get something similar to an scalar with can be safely modified in-place and (pointer) passed to C/C++/Fortran. BTW, I'm using numpy 1.1.0, and the it seems 0-dim arrays are NOT Fortran contiguous... Is this OK? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From simon.palmer at gmail.com Wed Aug 27 17:00:01 2008 From: simon.palmer at gmail.com (SimonPalmer) Date: Wed, 27 Aug 2008 14:00:01 -0700 (PDT) Subject: [Numpy-discussion] normalizing a vector so it has magnitude 1 Message-ID: bit of a newb question, is there a method for normalising a 1D vector so it ends up with magnitude 1? I can do it manually but I was hoping there was a neat numpy - or scipy - trick. I've been web surfing but nothing really leaps out From gruben at bigpond.net.au Wed Aug 27 17:18:33 2008 From: gruben at bigpond.net.au (Gary Ruben) Date: Wed, 27 Aug 2008 22:18:33 +0100 Subject: [Numpy-discussion] normalizing a vector so it has magnitude 1 In-Reply-To: References: Message-ID: <48B5C4A9.6020700@bigpond.net.au> I don't know what you mean by a 1D vector, but for a 3-vector, you can do this (also works for N-dimensions) In [1]: a=r_[1.,2.,3.] In [2]: a Out[2]: array([ 1., 2., 3.]) In [3]: b=a/norm(a) In [4]: b Out[4]: array([ 0.26726124, 0.53452248, 0.80178373]) Gary R > bit of a newb question, is there a method for normalising a 1D vector > so it ends up with magnitude 1? > > I can do it manually but I was hoping there was a neat numpy - or > scipy - trick. I've been web surfing but nothing really leaps out From simon.palmer at gmail.com Wed Aug 27 18:00:06 2008 From: simon.palmer at gmail.com (SimonPalmer) Date: Wed, 27 Aug 2008 15:00:06 -0700 (PDT) Subject: [Numpy-discussion] normalizing a vector so it has magnitude 1 In-Reply-To: <48B5C4A9.6020700@bigpond.net.au> References: <48B5C4A9.6020700@bigpond.net.au> Message-ID: <9540563e-84b5-4d10-bdaa-2e46c8cb34f4@25g2000hsx.googlegroups.com> sorry, 1D array this is perfect, thanks. On Aug 27, 10:18 pm, Gary Ruben wrote: > I don't know what you mean by a 1D vector, but for a 3-vector, you can > do this (also works for N-dimensions) > > In [1]: a=r_[1.,2.,3.] > In [2]: a > Out[2]: array([ 1., 2., 3.]) > In [3]: b=a/norm(a) > In [4]: b > Out[4]: array([ 0.26726124, 0.53452248, 0.80178373]) > > Gary R > > > bit of a newb question, is there a method for normalising a 1D vector > > so it ends up with magnitude 1? > > > I can do it manually but I was hoping there was a neat numpy - or > > scipy - trick. I've been web surfing but nothing really leaps out > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion From stefan at sun.ac.za Thu Aug 28 01:15:07 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 27 Aug 2008 22:15:07 -0700 Subject: [Numpy-discussion] Array subclassing - question for Travis In-Reply-To: <1e2af89e0808270303s632067b2nc0951290785f18cd@mail.gmail.com> References: <1e2af89e0808270303s632067b2nc0951290785f18cd@mail.gmail.com> Message-ID: <9457e7c80808272215t770ab998l21734ec5e43f2339@mail.gmail.com> Hey Matthew 2008/8/27 Matthew Brett : > In [148]: type(np.multiply(arr, obj)) # this is what I expected > Out[148]: > > In [149]: type(np.multiply.outer(arr, obj)) # this is not - I expected > class A again > Out[149]: Since both those objects have an __array_priority__ of 0.0, I guess it just takes whichever class comes first. > In [151]: class A(np.ndarray): __array_priority__ = 20.0 # setting > array priority does not affect behavior > > In [152]: obj = arr.copy().view(A) > > In [153]: type(np.multiply.outer(arr, obj)) > Out[153]: You should set the number lower: In [15]: class A(np.ndarray): ....: __array_priority__ = -1.0 ....: ....: In [16]: obj = arr.copy().view(A) In [17]: np.multiply(obj, arr) Out[17]: A([ 0, 1, 4, 9, 16]) In [18]: np.multiply(arr, obj) Out[18]: A([ 0, 1, 4, 9, 16]) Cheers St?fan From stefan at sun.ac.za Thu Aug 28 01:42:20 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 27 Aug 2008 22:42:20 -0700 Subject: [Numpy-discussion] [SciPy08] Documentation BoF In-Reply-To: <63751c30808260358y2010b7a0s6c5ebffbe40d181b@mail.gmail.com> References: <63751c30808260358y2010b7a0s6c5ebffbe40d181b@mail.gmail.com> Message-ID: <9457e7c80808272242y2b31572cr822eb9ac28c1bbe2@mail.gmail.com> Hi Neil 2008/8/26 Neil Crighton : >> >> - Should we have a separate User manual and a Reference manual, or only >> a single manual? >> > > Are there still plans to write a 10 page 'Getting started with NumPy' > document? I think this would be very useful. Ideally a 'getting started' > document, the docstrings, and a reference manual is all the documentation > you'd need. I try to avoid reading long user manuals unless I'm forced to, > but I don't know what other people do. I did some work towards that: http://mentat.za.net/numpy/intro/intro.html But in the end we decided that we needed something more marketing and less tutorial oriented. Cheers St?fan From stefan at sun.ac.za Thu Aug 28 02:02:27 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 27 Aug 2008 23:02:27 -0700 Subject: [Numpy-discussion] subsampling array without loops In-Reply-To: References: <93455E9E-3297-40CF-8796-ADEACB11A182@jpl.nasa.gov> Message-ID: <9457e7c80808272302ta4be1c1xb31562c49cbf4f19@mail.gmail.com> 2008/8/26 Anne Archibald : > 2008/8/22 Catherine Moroney : >> I'm looking for a way to acccomplish the following task without lots >> of loops involved, which are really slowing down my code. >> >> I have a 128x512 array which I want to break down into 2x2 squares. >> Then, for each 2x2 square I want to do some simple calculations >> such as finding the maximum value, which I then store in a 64x256 >> array. > > You should be able to do some of this with reshape and transpose: > In [1]: import numpy as np > > In [3]: A = np.zeros((128,512)) > > In [4]: B = np.reshape(A,(64,2,256,2)) > > In [5]: C = np.transpose(B,(0,2,1,3)) > > In [6]: C.shape > Out[6]: (64, 256, 2, 2) Or you can obtain a similar effect using my new favorite hammer: from numpy.lib import stride_tricks rows, cols = x.shape el = x.dtype.itemsize y = stride_tricks.as_strided(x, shape=(rows/2, cols/2, 2, 2), strides=(el*2*cols, el*2, el*cols, el)) Cheers St?fan From matthew.brett at gmail.com Thu Aug 28 02:50:52 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 27 Aug 2008 23:50:52 -0700 Subject: [Numpy-discussion] Array subclassing - question for Travis In-Reply-To: <9457e7c80808272215t770ab998l21734ec5e43f2339@mail.gmail.com> References: <1e2af89e0808270303s632067b2nc0951290785f18cd@mail.gmail.com> <9457e7c80808272215t770ab998l21734ec5e43f2339@mail.gmail.com> Message-ID: <1e2af89e0808272350v4d19f447lffe48fb31635f9ca@mail.gmail.com> Hi Stefan, Thanks for the kindly reply... > Since both those objects have an __array_priority__ of 0.0, I guess it > just takes whichever class comes first. > In [15]: class A(np.ndarray): > ....: __array_priority__ = -1.0 I think, playing more... For np.multiply, it does not seem possible to demote the subclass priority below that of the base ndarray type, but differences in array priority do affect the priority of subtypes. np.multiply.outer, always returns an ndarray type, regardless of the subtypes it is passed. This: import numpy as np class SubClass1(np.ndarray): pass class SubClass2(np.ndarray): __array_priority__ = -10.0 class SubClass3(np.ndarray): __array_priority__ = 1.0 arr = np.arange(5) obj1 = arr.copy().view(SubClass1) obj2 = arr.copy().view(SubClass2) obj3 = arr.copy().view(SubClass3) # Returns sub-type regardless of order or array priority print type(np.multiply(obj1, arr)) print type(np.multiply(arr, obj1)) print type(np.multiply(obj2, arr)) # As expected print type(np.multiply(obj1, obj2)) print type(np.multiply(obj1, obj3)) print type(np.multiply(obj2, obj3)) # multiply outer always returns ndarray type print type(np.multiply.outer(obj1, obj2)) print type(np.multiply.outer(obj1, obj3)) print type(np.multiply.outer(obj2, obj3)) gives: Do you agree? See you... Matthew From cournape at gmail.com Thu Aug 28 03:28:47 2008 From: cournape at gmail.com (David Cournapeau) Date: Thu, 28 Aug 2008 16:28:47 +0900 Subject: [Numpy-discussion] numpy trunk build broken on 32-bit ubuntu In-Reply-To: <200808262044.16628.pgmdevlist@gmail.com> References: <200808262044.16628.pgmdevlist@gmail.com> Message-ID: <5b8d13220808280028k8711868naecb7531586cc777@mail.gmail.com> On Wed, Aug 27, 2008 at 9:44 AM, Pierre GM wrote: > On Tuesday 26 August 2008 20:41:44 Charles R Harris wrote: > >> I've been thinking of reverting the changes rather than waiting for David >> to get back. > > As mentioned in a previous email, that fails on x86_64 AMD as well, same > place. Commenting out `trunc` in umathmodule.c.src does the trick. Yes, it is not arch specific, but runtime specific. trunc is not available from math.h without adding some flags at compilation with glibc and gcc, but is with mac os x C runtime. I did not expect it would have been different from ceil and other rouding methods, I should have checked. Since the change is reverted and we are in rc, I won't change it back now. cheers, David From stefan at sun.ac.za Thu Aug 28 03:59:48 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 28 Aug 2008 00:59:48 -0700 Subject: [Numpy-discussion] Array subclassing - question for Travis In-Reply-To: <1e2af89e0808272350v4d19f447lffe48fb31635f9ca@mail.gmail.com> References: <1e2af89e0808270303s632067b2nc0951290785f18cd@mail.gmail.com> <9457e7c80808272215t770ab998l21734ec5e43f2339@mail.gmail.com> <1e2af89e0808272350v4d19f447lffe48fb31635f9ca@mail.gmail.com> Message-ID: <9457e7c80808280059sbf95fcak6427ca991825f079@mail.gmail.com> 2008/8/28 Matthew Brett : > np.multiply.outer, always returns an ndarray type, regardless of the > subtypes it is passed. Sorry, I don't know what I was thinking. You are right! Cheers St?fan From nwagner at iam.uni-stuttgart.de Thu Aug 28 06:29:35 2008 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 28 Aug 2008 12:29:35 +0200 Subject: [Numpy-discussion] ImportError: No module named my_module Message-ID: ====================================================================== ERROR: Failure: ImportError (No module named my_module) ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/home/nwagner/local/lib/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/loader.py", line 364, in loadTestsFromName addr.filename, addr.module) File "/data/home/nwagner/local/lib/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/data/home/nwagner/local/lib/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/importer.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/doc/__init__.py", line 10, in __import__(__name__ + '.' + f) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/doc/example.py", line 35, in from my_module import my_func, other_func ImportError: No module named my_module From oliphant at enthought.com Thu Aug 28 10:48:16 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 28 Aug 2008 09:48:16 -0500 Subject: [Numpy-discussion] rc1 update In-Reply-To: <48B6A3B0.2060903@astraw.com> References: <9457e7c80808272218x6d7fe7e6ic54e58bb17001d31@mail.gmail.com> <9457e7c80808272252j6aad7244l74cc7023f0fd40fe@mail.gmail.com> <9457e7c80808280102u7f9690a9ud6267253f774af67@mail.gmail.com> <9457e7c80808280103g30540cf1sd3f7a05a58984956@mail.gmail.com> <48B6A3B0.2060903@astraw.com> Message-ID: <48B6BAB0.5010305@enthought.com> Andrew Straw wrote: > Joining the party late, here, so I don't know how much has already been > said... Particularly in favor of renaming NPY_VERSION. What's the benefit? > > I'm -1 on removing the name NPY_VERSION. It would cause unnecessary > breakage. > > I'm -0 on defining an additional NPY_ABI_VERSION to equal NPY_VERSION. > It just seems unnecessary to me. Won't a comment do, e.g. /* We wanted > to name this NPY_ABI_VERSION, but it was already NPY_VERSION since pre > 1.0, so we're sticking with it. (Until 1.2, it also served a dual role > as NPY_API_VERSION, when we added that, below.) */ ? > > I'm +0 on renaming the recently added NPY_FEATURE_VERSION to > NPY_API_VERSION. I don't care what it's called, really. > > Since A) this is a C developer thing only (aka people who should know > what they're doing), B) there is already IMO fairly good documentation > immediately above the NPY_VERSION and NPY_FEATURE_VERSION in the header > file, and C) very little impetus for anyone to actually directly use > NPY_VERSION anyway (there's a runtime ABI check during import_array() > which is the most important thing -- it checks at runtime if the compile > time ABI version of the caller and numpy itself are the same), I think > it's best to leave NPY_VERSION as NPY_VERSION. > > The only code that I know of that currently explicitly uses NPY_VERSION > is C preprocessor tests for which C API to use -- from now, new code > being written will check the new NPY_FEATURE_VERSION (or NPY_API_VERSION > or whatever it ends up being called) for API features. Anyone updating > old code will already have a bit of additional complexity to deal with, > and I want to minimize the amount of additional complexity. We don't > want a rat's nest of C preprossor #ifdefs as people track the numpy > version checking defines. Given the addition of NPY_FEATURE_VERSION, > code that's already been written with explicit NPY_VERSION checks for > API information is already going to become outdated, but as long as we > keep the NPY_VERSION name, at least it won't break or require another > level of complexity to maintain forward compatibility. > > +1 on what Andrew said. -Travis From millman at berkeley.edu Thu Aug 28 11:00:01 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Thu, 28 Aug 2008 08:00:01 -0700 Subject: [Numpy-discussion] rc1 update In-Reply-To: <48B6BAB0.5010305@enthought.com> References: <9457e7c80808272218x6d7fe7e6ic54e58bb17001d31@mail.gmail.com> <9457e7c80808272252j6aad7244l74cc7023f0fd40fe@mail.gmail.com> <9457e7c80808280102u7f9690a9ud6267253f774af67@mail.gmail.com> <9457e7c80808280103g30540cf1sd3f7a05a58984956@mail.gmail.com> <48B6A3B0.2060903@astraw.com> <48B6BAB0.5010305@enthought.com> Message-ID: On Thu, Aug 28, 2008 at 7:48 AM, Travis E. Oliphant wrote: > +1 on what Andrew said. I don't really care that much, but I do think API is better than FEATURE. I would think that there may be times when we change the API but not the features (e.g., renaming something). I won't bring this up again, so if no one else finds this compelling consider it dropped. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From strawman at astraw.com Thu Aug 28 11:50:54 2008 From: strawman at astraw.com (Andrew Straw) Date: Thu, 28 Aug 2008 08:50:54 -0700 Subject: [Numpy-discussion] rc1 update In-Reply-To: References: <9457e7c80808272218x6d7fe7e6ic54e58bb17001d31@mail.gmail.com> <9457e7c80808272252j6aad7244l74cc7023f0fd40fe@mail.gmail.com> <9457e7c80808280102u7f9690a9ud6267253f774af67@mail.gmail.com> <9457e7c80808280103g30540cf1sd3f7a05a58984956@mail.gmail.com> <48B6A3B0.2060903@astraw.com> <48B6BAB0.5010305@enthought.com> Message-ID: <48B6C95E.5060408@astraw.com> Jarrod Millman wrote: > On Thu, Aug 28, 2008 at 7:48 AM, Travis E. Oliphant > wrote: > >> +1 on what Andrew said. >> > > I don't really care that much, but I do think API is better than > FEATURE. I would think that there may be times when we change the API > but not the features (e.g., renaming something). I won't bring this > up again, so if no one else finds this compelling consider it dropped. > > As I said, I'm +0 on that aspect. So I guess that makes Travis +0, too. ;) From stefan at sun.ac.za Thu Aug 28 12:49:29 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 28 Aug 2008 09:49:29 -0700 Subject: [Numpy-discussion] rc1 update In-Reply-To: <48B6C95E.5060408@astraw.com> References: <9457e7c80808272218x6d7fe7e6ic54e58bb17001d31@mail.gmail.com> <9457e7c80808272252j6aad7244l74cc7023f0fd40fe@mail.gmail.com> <9457e7c80808280102u7f9690a9ud6267253f774af67@mail.gmail.com> <9457e7c80808280103g30540cf1sd3f7a05a58984956@mail.gmail.com> <48B6A3B0.2060903@astraw.com> <48B6BAB0.5010305@enthought.com> <48B6C95E.5060408@astraw.com> Message-ID: <9457e7c80808280949t86f7b42nf523b26cea284c81@mail.gmail.com> 2008/8/28 Andrew Straw : > Jarrod Millman wrote: >> On Thu, Aug 28, 2008 at 7:48 AM, Travis E. Oliphant >> wrote: >> >>> +1 on what Andrew said. >>> >> >> I don't really care that much, but I do think API is better than >> FEATURE. I would think that there may be times when we change the API >> but not the features (e.g., renaming something). I won't bring this >> up again, so if no one else finds this compelling consider it dropped. Feel free to change it -- I don't mind. Cheers St?fan From matthew.brett at gmail.com Thu Aug 28 13:21:16 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 28 Aug 2008 10:21:16 -0700 Subject: [Numpy-discussion] ImportError: No module named my_module In-Reply-To: References: Message-ID: <1e2af89e0808281021s6143e95dlb1e964931a40b868@mail.gmail.com> Hi Nils, I don't have an example.py in a checkout from a few seconds ago - it is possible it could be a stray file? Best, Matthew On Thu, Aug 28, 2008 at 3:29 AM, Nils Wagner wrote: > ====================================================================== > ERROR: Failure: ImportError (No module named my_module) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/data/home/nwagner/local/lib/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/loader.py", > line 364, in loadTestsFromName > addr.filename, addr.module) > File > "/data/home/nwagner/local/lib/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/importer.py", > line 39, in importFromPath > return self.importFromDir(dir_path, fqname) > File > "/data/home/nwagner/local/lib/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/importer.py", > line 84, in importFromDir > mod = load_module(part_fqname, fh, filename, desc) > File > "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/doc/__init__.py", > line 10, in > __import__(__name__ + '.' + f) > File > "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/doc/example.py", > line 35, in > from my_module import my_func, other_func > ImportError: No module named my_module > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From Chris.Barker at noaa.gov Thu Aug 28 13:36:27 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 28 Aug 2008 10:36:27 -0700 Subject: [Numpy-discussion] numpy and py2exe Message-ID: <48B6E21B.6070205@noaa.gov> Hi all, An issue just came up with numpy and py2exe on the wxPython list. A solution has been found, but I thought I'd post here, so that it could be on the numpy developers' radar when considering the structure of numpy for the future: > I'm compile a wxPython script using GUI2Exe, and am getting the > following when running the produced exe: > > $ ./dist/svm.exe > Traceback (most recent call last): > File "svm.py", line 28, in > import graphic_svm_data > File "graphic_svm_data.pyo", line 29, in > File "floatcanvas\NavCanvas.pyo", line 7, in > File "floatcanvas\FloatCanvas.pyo", line 7, in > File "numpy\__init__.pyo", line 93, in > File "numpy\add_newdocs.pyo", line 9, in > File "numpy\lib\__init__.pyo", line 19, in > File "numpy\lib\financial.pyo", line 78, in > TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str' > I tried to add numpy.lib.financial to the Excludes and Ignores list, but am still getting this error. > > Googling found that this problem is a known bug: http://article.gmane.org/gmane.comp.python.py2exe/2920 The solution was to: > 2) Use Optimize=1 or Optimize=0 in GUI2Exe and re-build your > executable. Uing Optimize=2 strips all docstrings in the executable, > which may cause troubles (as in your case); > [Ron Barak] YES!!! > With Optimize=1 my script finally compiles to exe. While this worked, I have a couple observations: 1) This broke on numpy.lib.financial. that had nothing to do with the issue, but it would be nice if people could use numpy with py2exe and the like without having to include all those unused packages. I think this ties in with the discussion about speeding up with importing process, and may be addressed in future versions if I understood the results of some of the discussion at SciPy right. 2) it would also be nice if one could use numpy with the docstrings removed, like in this case. I have no idea what the issues are with that. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From nwagner at iam.uni-stuttgart.de Thu Aug 28 13:46:00 2008 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 28 Aug 2008 19:46:00 +0200 Subject: [Numpy-discussion] ImportError: No module named my_module In-Reply-To: <1e2af89e0808281021s6143e95dlb1e964931a40b868@mail.gmail.com> References: <1e2af89e0808281021s6143e95dlb1e964931a40b868@mail.gmail.com> Message-ID: On Thu, 28 Aug 2008 10:21:16 -0700 "Matthew Brett" wrote: > Hi Nils, > > I don't have an example.py in a checkout from a few >seconds ago - it > is possible it could be a stray file? > > Best, > > Matthew > Running find . -name example.py in my svn/numpy directory yields ./doc/example.py ./doc/newdtype_example/example.py ls -l doc/example.py -rw-r--r-- 1 nwagner users 3497 2008-08-28 19:42 doc/example.py Nils From charlesr.harris at gmail.com Thu Aug 28 13:50:14 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 28 Aug 2008 11:50:14 -0600 Subject: [Numpy-discussion] numpy and py2exe In-Reply-To: <48B6E21B.6070205@noaa.gov> References: <48B6E21B.6070205@noaa.gov> Message-ID: On Thu, Aug 28, 2008 at 11:36 AM, Christopher Barker wrote: > Hi all, > > An issue just came up with numpy and py2exe on the wxPython list. A > solution has been found, but I thought I'd post here, so that it could > be on the numpy developers' radar when considering the structure of > numpy for the future: > > > I'm compile a wxPython script using GUI2Exe, and am getting the > > following when running the produced exe: > > > > $ ./dist/svm.exe > > Traceback (most recent call last): > > File "svm.py", line 28, in > > import graphic_svm_data > > File "graphic_svm_data.pyo", line 29, in > > File "floatcanvas\NavCanvas.pyo", line 7, in > > File "floatcanvas\FloatCanvas.pyo", line 7, in > > File "numpy\__init__.pyo", line 93, in > > File "numpy\add_newdocs.pyo", line 9, in > > File "numpy\lib\__init__.pyo", line 19, in > > File "numpy\lib\financial.pyo", line 78, in > > TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str' > > > I tried to add numpy.lib.financial to the Excludes and Ignores list, but > am still getting this error. > > > > Googling found that this problem is a known bug: > http://article.gmane.org/gmane.comp.python.py2exe/2920 > > The solution was to: This needs to be fixed in trunk, please open a ticket. There have been several other fixes for -OO recently and I don't think we are done with them yet. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Aug 28 13:55:41 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 28 Aug 2008 10:55:41 -0700 Subject: [Numpy-discussion] ImportError: No module named my_module In-Reply-To: References: <1e2af89e0808281021s6143e95dlb1e964931a40b868@mail.gmail.com> Message-ID: <1e2af89e0808281055u3f011b2coe7b6e57ec8ece746@mail.gmail.com> Hi, > find . -name example.py > > in my svn/numpy directory yields > > ./doc/example.py > ./doc/newdtype_example/example.py Ah - how strange - I just tried deleting my doc directory and checking it out again, and don't have those files or directory... Matthew From robert.kern at gmail.com Thu Aug 28 13:58:05 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 28 Aug 2008 12:58:05 -0500 Subject: [Numpy-discussion] ImportError: No module named my_module In-Reply-To: <1e2af89e0808281055u3f011b2coe7b6e57ec8ece746@mail.gmail.com> References: <1e2af89e0808281021s6143e95dlb1e964931a40b868@mail.gmail.com> <1e2af89e0808281055u3f011b2coe7b6e57ec8ece746@mail.gmail.com> Message-ID: <3d375d730808281058j462b0060qe865a00310bc3a20@mail.gmail.com> On Thu, Aug 28, 2008 at 12:55, Matthew Brett wrote: > Hi, > >> find . -name example.py >> >> in my svn/numpy directory yields >> >> ./doc/example.py >> ./doc/newdtype_example/example.py > > Ah - how strange - I just tried deleting my doc directory and checking > it out again, and don't have those files or directory... He's talking about the root trunk/doc/ directory (named svn/numpy/doc/ on his machine), not the trunk/numpy/doc/ directory. However, the file that is getting picked up earlier is installed into site-packages, meaning it is from an old installation. Nils, remove the site-packages/numpy/ directory and reinstall numpy. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From nwagner at iam.uni-stuttgart.de Thu Aug 28 14:16:38 2008 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 28 Aug 2008 20:16:38 +0200 Subject: [Numpy-discussion] ImportError: No module named my_module In-Reply-To: <3d375d730808281058j462b0060qe865a00310bc3a20@mail.gmail.com> References: <1e2af89e0808281021s6143e95dlb1e964931a40b868@mail.gmail.com> <1e2af89e0808281055u3f011b2coe7b6e57ec8ece746@mail.gmail.com> <3d375d730808281058j462b0060qe865a00310bc3a20@mail.gmail.com> Message-ID: On Thu, 28 Aug 2008 12:58:05 -0500 "Robert Kern" wrote: > On Thu, Aug 28, 2008 at 12:55, Matthew Brett > wrote: >> Hi, >> >>> find . -name example.py >>> >>> in my svn/numpy directory yields >>> >>> ./doc/example.py >>> ./doc/newdtype_example/example.py >> >> Ah - how strange - I just tried deleting my doc >>directory and checking >> it out again, and don't have those files or directory... > > He's talking about the root trunk/doc/ directory (named >svn/numpy/doc/ > on his machine), not the trunk/numpy/doc/ directory. >However, the file > that is getting picked up earlier is installed into >site-packages, > meaning it is from an old installation. > > Nils, remove the site-packages/numpy/ directory and >reinstall numpy. > > -- > Robert Kern > > "I have come to believe that the whole world is an >enigma, a harmless > enigma that is made terrible by our own mad attempt to >interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion Works for me. Thank you very much ! Nils Ran 1716 tests in 24.268s OK (SKIP=1) From charlesr.harris at gmail.com Thu Aug 28 17:43:47 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 28 Aug 2008 15:43:47 -0600 Subject: [Numpy-discussion] numpy and py2exe In-Reply-To: References: <48B6E21B.6070205@noaa.gov> Message-ID: On Thu, Aug 28, 2008 at 11:50 AM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Thu, Aug 28, 2008 at 11:36 AM, Christopher Barker < > Chris.Barker at noaa.gov> wrote: > >> Hi all, >> >> An issue just came up with numpy and py2exe on the wxPython list. A >> solution has been found, but I thought I'd post here, so that it could >> be on the numpy developers' radar when considering the structure of >> numpy for the future: >> >> > I'm compile a wxPython script using GUI2Exe, and am getting the >> > following when running the produced exe: >> > >> > $ ./dist/svm.exe >> > Traceback (most recent call last): >> > File "svm.py", line 28, in >> > import graphic_svm_data >> > File "graphic_svm_data.pyo", line 29, in >> > File "floatcanvas\NavCanvas.pyo", line 7, in >> > File "floatcanvas\FloatCanvas.pyo", line 7, in >> > File "numpy\__init__.pyo", line 93, in >> > File "numpy\add_newdocs.pyo", line 9, in >> > File "numpy\lib\__init__.pyo", line 19, in >> > File "numpy\lib\financial.pyo", line 78, in >> > TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str' >> >> > I tried to add numpy.lib.financial to the Excludes and Ignores list, but >> am still getting this error. >> > >> > Googling found that this problem is a known bug: >> http://article.gmane.org/gmane.comp.python.py2exe/2920 >> >> The solution was to: > > > > > This needs to be fixed in trunk, please open a ticket. There have been > several other fixes for -OO recently and I don't think we are done with them > yet. > I thought I had seen this before and indeed it has been reported. However, financial has been changed and this particular error should no longer be a problem. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Thu Aug 28 18:01:16 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 28 Aug 2008 15:01:16 -0700 Subject: [Numpy-discussion] numpy and py2exe In-Reply-To: References: <48B6E21B.6070205@noaa.gov> Message-ID: <48B7202C.801@noaa.gov> Charles R Harris wrote: > This needs to be fixed in trunk, please open a ticket. There have been > several other fixes for -OO recently and I don't think we are done with > them yet. already there: http://scipy.org/scipy/numpy/ticket/893 and: http://scipy.org/scipy/numpy/ticket/894 -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Thu Aug 28 20:26:24 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 28 Aug 2008 18:26:24 -0600 Subject: [Numpy-discussion] Where does _assert_func come from. Message-ID: Where does _assert_func come from in this fragment from a test class? class _GenericTest(object): def _test_equal(self, a, b): self._assert_func(a, b) I hope it isn't monkey patched in. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Thu Aug 28 20:48:24 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 28 Aug 2008 17:48:24 -0700 Subject: [Numpy-discussion] Where does _assert_func come from. In-Reply-To: References: Message-ID: On Thu, Aug 28, 2008 at 5:26 PM, Charles R Harris wrote: > Where does _assert_func come from in this fragment from a test class? > > class _GenericTest(object): > def _test_equal(self, a, b): > self._assert_func(a, b) > > I hope it isn't monkey patched in. It could be meant as a mix-in, case in which it would be ok for the class to be incomplete as written. A bit surprising, but not necessarily incorrect. If it's not meant to be a base/mix-in class, then it's a good ole'bug. Cheers, f From alan at ajackson.org Thu Aug 28 20:57:09 2008 From: alan at ajackson.org (Alan Jackson) Date: Thu, 28 Aug 2008 19:57:09 -0500 Subject: [Numpy-discussion] Advice on converting iterator into array efficiently Message-ID: <20080828195709.15deca69@ajackson.org> Looking for advice on a good way to handle this problem. I'm dealing with large tables (Gigabyte large). I would like to efficiently subset values from one column based on the values in another column, and get arrays out of the operation. For example, say I have 2 columns, "energy" and "collection". Collection is basically an index that flags values that go together, so all the energy values with a collection value of 18 belong together. I'd like to be able to set up an iterator on collection that would hand me an array of energy on each iteration : if table is all my data, then something like for c in table['collection'] : e = c['energy'] ... do array operations on e I've been playing with pytables, and they help, but I can't quite seem to get there. I can get an iterator for energy within a collection, but I can't figure out an efficient way to get an array out. What I have so far is for h in np.unique(table.col('collection')) : rows = table.where('collection == c') for row in rows : print c,' : ', row['energy'] but I really want to convert rows['energy'] to an array. I've thought about building a nasty set of pointers and whatnot - I did it once in perl - but I'm hoping to avoid that. -- ----------------------------------------------------------------------- | Alan K. Jackson | To see a World in a Grain of Sand | | alan at ajackson.org | And a Heaven in a Wild Flower, | | www.ajackson.org | Hold Infinity in the palm of your hand | | Houston, Texas | And Eternity in an hour. - Blake | ----------------------------------------------------------------------- From strawman at astraw.com Thu Aug 28 21:26:18 2008 From: strawman at astraw.com (Andrew Straw) Date: Thu, 28 Aug 2008 18:26:18 -0700 Subject: [Numpy-discussion] Advice on converting iterator into array efficiently In-Reply-To: <20080828195709.15deca69@ajackson.org> References: <20080828195709.15deca69@ajackson.org> Message-ID: <48B7503A.9040302@astraw.com> Alan Jackson wrote: > Looking for advice on a good way to handle this problem. > > I'm dealing with large tables (Gigabyte large). I would like to > efficiently subset values from one column based on the values in > another column, and get arrays out of the operation. For example, > say I have 2 columns, "energy" and "collection". Collection is > basically an index that flags values that go together, so all the > energy values with a collection value of 18 belong together. I'd > like to be able to set up an iterator on collection that would > hand me an array of energy on each iteration : > > if table is all my data, then something like > > for c in table['collection'] : > e = c['energy'] > ... do array operations on e > > I've been playing with pytables, and they help, but I can't quite > seem to get there. I can get an iterator for energy within a collection, > but I can't figure out an efficient way to get an array out. > > What I have so far is > > for h in np.unique(table.col('collection')) : > rows = table.where('collection == c') > for row in rows : > print c,' : ', row['energy'] > > but I really want to convert rows['energy'] to an array. > > I've thought about building a nasty set of pointers and whatnot - > I did it once in perl - but I'm hoping to avoid that. > > I do stuff like this all the time: t = table[:] # convert to structured array collections = np.unique(t['collection']) for collection in collections: cond = t['collection'] == collection energy_this_collection = t['energy'][cond] HTH, Andrew From charlesr.harris at gmail.com Thu Aug 28 21:39:02 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 28 Aug 2008 19:39:02 -0600 Subject: [Numpy-discussion] Where does _assert_func come from. In-Reply-To: References: Message-ID: On Thu, Aug 28, 2008 at 6:48 PM, Fernando Perez wrote: > On Thu, Aug 28, 2008 at 5:26 PM, Charles R Harris > wrote: > > Where does _assert_func come from in this fragment from a test class? > > > > class _GenericTest(object): > > def _test_equal(self, a, b): > > self._assert_func(a, b) > > > > I hope it isn't monkey patched in. > > It could be meant as a mix-in, case in which it would be ok for the > class to be incomplete as written. A bit surprising, but not > necessarily incorrect. > > If it's not meant to be a base/mix-in class, then it's a good ole'bug. > Turns out it's defined in a derived class, which I suppose makes it a mix-in class. Looks kinda like the python version of pure virtual functions in C++ base classes. It's a bit surprising to see a python pattern that's more obscure than C++ ;) I'm not sure I like its usage here -- implementation rather than is a -- but I'm not going to clean it up. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Thu Aug 28 21:54:37 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 28 Aug 2008 18:54:37 -0700 Subject: [Numpy-discussion] Where does _assert_func come from. In-Reply-To: References: Message-ID: On Thu, Aug 28, 2008 at 6:39 PM, Charles R Harris wrote: > Turns out it's defined in a derived class, which I suppose makes it a mix-in > class. Looks kinda like the python version of pure virtual functions in C++ > base classes. It's a bit surprising to see a python pattern that's more > obscure than C++ ;) I'm not sure I like its usage here -- implementation > rather than is a -- but I'm not going to clean it up. The typical way to do this in python would be to write in the base class you saw: def assert_func(self,a,b): "some explanation" raise NotImplementedError this makes the desired interface explicit, while indicating that the method is meant to be implemented by subclasses. In a multiple inheritance situation, however, this may be a bad idea if the class is meant to be mixed with another that provides the method (because it could overwrite the real one, depending on how the inheritance diagram is set up). In summary, my take on this is: - if it's meant to be an abstract base class that declares an interface, use the idiom above for clarity. - if you're writing a mixin meant to be overlaid on top of another class with a specific interface like this, then at the very least document that fact very well in the code. Just my 1e-2, f From charlesr.harris at gmail.com Thu Aug 28 22:07:56 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 28 Aug 2008 20:07:56 -0600 Subject: [Numpy-discussion] Where does _assert_func come from. In-Reply-To: References: Message-ID: On Thu, Aug 28, 2008 at 7:54 PM, Fernando Perez wrote: > On Thu, Aug 28, 2008 at 6:39 PM, Charles R Harris > wrote: > > > Turns out it's defined in a derived class, which I suppose makes it a > mix-in > > class. Looks kinda like the python version of pure virtual functions in > C++ > > base classes. It's a bit surprising to see a python pattern that's more > > obscure than C++ ;) I'm not sure I like its usage here -- implementation > > rather than is a -- but I'm not going to clean it up. > > The typical way to do this in python would be to write in the base > class you saw: > > def assert_func(self,a,b): > "some explanation" > raise NotImplementedError > > this makes the desired interface explicit, while indicating that the > method is meant to be implemented by subclasses. In a multiple > inheritance situation, however, this may be a bad idea if the class is > meant to be mixed with another that provides the method (because it > could overwrite the real one, depending on how the inheritance diagram > is set up). > > In summary, my take on this is: > > - if it's meant to be an abstract base class that declares an > interface, use the idiom above for clarity. > > - if you're writing a mixin meant to be overlaid on top of another > class with a specific interface like this, then at the very least > document that fact very well in the code. > > Just my 1e-2, > Thanks. It's always nice to learn something new. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Thu Aug 28 22:16:26 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 28 Aug 2008 19:16:26 -0700 Subject: [Numpy-discussion] Where does _assert_func come from. In-Reply-To: References: Message-ID: On Thu, Aug 28, 2008 at 7:07 PM, Charles R Harris wrote: > Thanks. It's always nice to learn something new. Well, take my comments with a grain of salt. Gael always chastises me for not having drunk enough of the computer science abstraction kool-aid, so my take on these things tends to be pretty pedestrian. Wait for someone more properly educated than myself to chime in before thanking me. Blind leading the blind and all that :) Cheers, f From david at ar.media.kyoto-u.ac.jp Thu Aug 28 22:47:27 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 29 Aug 2008 11:47:27 +0900 Subject: [Numpy-discussion] #882: PyArray_ConvertToCommon and 0 sized sequences Message-ID: <48B7633F.7070607@ar.media.kyoto-u.ac.jp> Hi, The error in #882 looks pretty straightforward: from numpy import numarray a = numarray.array(1) a.choose([]) core dumps because a.choose calls PyArray_ConvertCommonType with a 0-sized array, which is not handled correctly. Should the 0-sized case be handled in this function, or in PyArray_Choose (the caller of PyArray_ConvertCommonType) ? cheers, David From faltet at pytables.org Fri Aug 29 05:29:46 2008 From: faltet at pytables.org (Francesc Alted) Date: Fri, 29 Aug 2008 11:29:46 +0200 Subject: [Numpy-discussion] Advice on converting iterator into array efficiently In-Reply-To: <20080828195709.15deca69@ajackson.org> References: <20080828195709.15deca69@ajackson.org> Message-ID: <200808291129.47228.faltet@pytables.org> A Friday 29 August 2008, Alan Jackson escrigu?: > Looking for advice on a good way to handle this problem. > > I'm dealing with large tables (Gigabyte large). I would like to > efficiently subset values from one column based on the values in > another column, and get arrays out of the operation. For example, > say I have 2 columns, "energy" and "collection". Collection is > basically an index that flags values that go together, so all the > energy values with a collection value of 18 belong together. I'd > like to be able to set up an iterator on collection that would > hand me an array of energy on each iteration : > > if table is all my data, then something like > > for c in table['collection'] : > e = c['energy'] > ... do array operations on e > > I've been playing with pytables, and they help, but I can't quite > seem to get there. I can get an iterator for energy within a > collection, but I can't figure out an efficient way to get an array > out. > > What I have so far is > > for h in np.unique(table.col('collection')) : > rows = table.where('collection == c') > for row in rows : > print c,' : ', row['energy'] > > but I really want to convert rows['energy'] to an array. You may use a list to keep the values and then convert it to an array. Also, yoy can use a dictionary for keeping the unique collections. The next should do the trick: energies = {} for row in table: c = row['collection'] e = row['energy'] if c in energies: energies[c].append(e) else: energies[c] = [e] # Convert the lists in numpy arrays for key in energies: energies[key] = numpy.array(energies[key]) This solution is pretty optimal in that it avoids to have to load all the table in memory and besides only requires one table iteration to get the job done. Hope this helps, -- Francesc Alted From faltet at pytables.org Fri Aug 29 06:19:39 2008 From: faltet at pytables.org (Francesc Alted) Date: Fri, 29 Aug 2008 12:19:39 +0200 Subject: [Numpy-discussion] Advice on converting iterator into array efficiently In-Reply-To: <200808291129.47228.faltet@pytables.org> References: <20080828195709.15deca69@ajackson.org> <200808291129.47228.faltet@pytables.org> Message-ID: <200808291219.39670.faltet@pytables.org> A Friday 29 August 2008, Francesc Alted escrigu?: > A Friday 29 August 2008, Alan Jackson escrigu?: > > Looking for advice on a good way to handle this problem. > > > > I'm dealing with large tables (Gigabyte large). I would like to > > efficiently subset values from one column based on the values in > > another column, and get arrays out of the operation. For example, > > say I have 2 columns, "energy" and "collection". Collection is > > basically an index that flags values that go together, so all the > > energy values with a collection value of 18 belong together. I'd > > like to be able to set up an iterator on collection that would > > hand me an array of energy on each iteration : > > > > if table is all my data, then something like > > > > for c in table['collection'] : > > e = c['energy'] > > ... do array operations on e > > > > I've been playing with pytables, and they help, but I can't quite > > seem to get there. I can get an iterator for energy within a > > collection, but I can't figure out an efficient way to get an array > > out. > > > > What I have so far is > > > > for h in np.unique(table.col('collection')) : > > rows = table.where('collection == c') > > for row in rows : > > print c,' : ', row['energy'] > > > > but I really want to convert rows['energy'] to an array. > > You may use a list to keep the values and then convert it to an > array. Also, yoy can use a dictionary for keeping the unique > collections. The next should do the trick: > > energies = {} > for row in table: > c = row['collection'] > e = row['energy'] > if c in energies: > energies[c].append(e) > else: > energies[c] = [e] > > # Convert the lists in numpy arrays > for key in energies: > energies[key] = numpy.array(energies[key]) Er... I completely forgot about the fine ``Table.whereRead()`` method. By using it, you can do: for c in np.unique(table.col('collection')) : print c,' : ', table.readWhere('collection == c', field='energy') which is what you want. However, this solution does require to read the entire table for each value of the collection (and once more in order to read the column of collections at the beginning). Do your timings and choose whatever approach you prefer. Cheers, -- Francesc Alted From jjstarn at usgs.gov Fri Aug 29 09:45:30 2008 From: jjstarn at usgs.gov (Jeffrey Starn) Date: Fri, 29 Aug 2008 09:45:30 -0400 Subject: [Numpy-discussion] Problem with numpy in pyinstaller Message-ID: I have a program that imports numpy and scipy. When I try to use PyInstaller, I get the following error: File "", line 21, in NameError: name 'io' is not defined The code up to line 21 is (line numbers differ becasue of comments): from numpy import * from numpy import random from scipy import * from scipy.linalg import cholesky from scipy.io import write_array model=raw_input('MF2K or UCode? ') model=model.upper() fname=raw_input('Root name? ') NumRan=raw_input('Number of random realizations? ') NumRan=int(NumRan) outfile=fname+".ranvar" MvName=fname+"._mv" Mv=io.read_array(MvName,lines=(1,-1) I have seen related psots, but I really don't understand what the fix is. Thanks. --------------------------------------------------------------------------------------------------------- Jeff Starn, Ground-water Specialist U.S. Geological Survey 101 Pitkin Street East Hartford, CT 06108 (860) 291-6746 jjstarn at usgs.gov --------------------------------------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitrey.kroshko at scipy.org Fri Aug 29 13:19:23 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Fri, 29 Aug 2008 20:19:23 +0300 Subject: [Numpy-discussion] isn't it a bug in array.fill()? Message-ID: <48B82F9B.90600@scipy.org> hi all, isn't it a bug (latest numpy from svn, as well as my older version) from numpy import array print array((1,2,3)).fill(10) None Regards, D. From dmitrey.kroshko at scipy.org Fri Aug 29 13:22:11 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Fri, 29 Aug 2008 20:22:11 +0300 Subject: [Numpy-discussion] isn't it a bug in array.fill() Message-ID: <48B83043.8000702@scipy.org> sorry, it isn't a bug, it's my fault, fill() returns None and do in-place modification. D. From kwgoodman at gmail.com Fri Aug 29 13:27:51 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 29 Aug 2008 10:27:51 -0700 Subject: [Numpy-discussion] isn't it a bug in array.fill()? In-Reply-To: <48B82F9B.90600@scipy.org> References: <48B82F9B.90600@scipy.org> Message-ID: On Fri, Aug 29, 2008 at 10:19 AM, dmitrey wrote: > hi all, > isn't it a bug > (latest numpy from svn, as well as my older version) > > from numpy import array > print array((1,2,3)).fill(10) > None Yeah, I do stuff like that too. fill works in place so it returns None. >> x = np.array([1,2]) >> x.fill(10) >> x array([10, 10]) >> x = x.fill(10) # <-- Danger! >> print x None From cburns at berkeley.edu Fri Aug 29 13:44:19 2008 From: cburns at berkeley.edu (Christopher Burns) Date: Fri, 29 Aug 2008 10:44:19 -0700 Subject: [Numpy-discussion] np.test() Bus error on OSX. Reopened Ticket 816 Message-ID: <764e38540808291044l18379e01s28d9198a6643b6c4@mail.gmail.com> I reopened Ticket 816: http://scipy.org/scipy/numpy/ticket/816 Running numpy.test() causes a bus error on OSX. -- Christopher Burns Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From dmitrey.kroshko at scipy.org Fri Aug 29 13:42:01 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Fri, 29 Aug 2008 20:42:01 +0300 Subject: [Numpy-discussion] isn't it a bug in array.fill()? In-Reply-To: References: <48B82F9B.90600@scipy.org> Message-ID: <48B834E9.2050305@scipy.org> Keith Goodman wrote: > Yeah, I do stuff like that too. fill works in place so it returns None. > > >>> x = np.array([1,2]) >>> x.fill(10) >>> x >>> > array([10, 10]) > >>> x = x.fill(10) # <-- Danger! >>> print x >>> > None > Since result "None" is never used it would be better to return reference to the modified array, it would decrease number of bugs. The last expression can raise very seldom in untested cases, I have revealed one of this recently in my code: if some_seldom_cond: r = empty(n, bool).fill(True) else: r = None So, as you see, r was always None D. From kwgoodman at gmail.com Fri Aug 29 13:51:50 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 29 Aug 2008 10:51:50 -0700 Subject: [Numpy-discussion] isn't it a bug in array.fill()? In-Reply-To: <48B834E9.2050305@scipy.org> References: <48B82F9B.90600@scipy.org> <48B834E9.2050305@scipy.org> Message-ID: On Fri, Aug 29, 2008 at 10:42 AM, dmitrey wrote: > > Keith Goodman wrote: >> Yeah, I do stuff like that too. fill works in place so it returns None. >> >> >>>> x = np.array([1,2]) >>>> x.fill(10) >>>> x >>>> >> array([10, 10]) >> >>>> x = x.fill(10) # <-- Danger! >>>> print x >>>> >> None >> > Since result "None" is never used it would be better to return reference > to the modified array, ... I like that idea. A lot of numpy functions return a reference to the modified array when the output array (out) is specified. From kwgoodman at gmail.com Fri Aug 29 13:55:07 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 29 Aug 2008 10:55:07 -0700 Subject: [Numpy-discussion] isn't it a bug in array.fill()? In-Reply-To: References: <48B82F9B.90600@scipy.org> <48B834E9.2050305@scipy.org> Message-ID: On Fri, Aug 29, 2008 at 10:51 AM, Keith Goodman wrote: > On Fri, Aug 29, 2008 at 10:42 AM, dmitrey wrote: >> >> Keith Goodman wrote: >>> Yeah, I do stuff like that too. fill works in place so it returns None. >>> >>> >>>>> x = np.array([1,2]) >>>>> x.fill(10) >>>>> x >>>>> >>> array([10, 10]) >>> >>>>> x = x.fill(10) # <-- Danger! >>>>> print x >>>>> >>> None >>> >> Since result "None" is never used it would be better to return reference >> to the modified array, ... > > I like that idea. A lot of numpy functions return a reference to the > modified array when the output array (out) is specified. But python doesn't do that. For example, x.sort() returns None in python. Should it return None in numpy? From charlesr.harris at gmail.com Fri Aug 29 14:01:07 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 29 Aug 2008 12:01:07 -0600 Subject: [Numpy-discussion] isn't it a bug in array.fill()? In-Reply-To: References: <48B82F9B.90600@scipy.org> <48B834E9.2050305@scipy.org> Message-ID: On Fri, Aug 29, 2008 at 11:51 AM, Keith Goodman wrote: > On Fri, Aug 29, 2008 at 10:42 AM, dmitrey > wrote: > > > > Keith Goodman wrote: > >> Yeah, I do stuff like that too. fill works in place so it returns None. > >> > >> > >>>> x = np.array([1,2]) > >>>> x.fill(10) > >>>> x > >>>> > >> array([10, 10]) > >> > >>>> x = x.fill(10) # <-- Danger! > >>>> print x > >>>> > >> None > >> > > Since result "None" is never used it would be better to return reference > > to the modified array, ... > > I like that idea. A lot of numpy functions return a reference to the > modified array when the output array (out) is specified. Google up the various discussions of python sort to see why Guido doesn't like that sort of thing. We've had that discussion on this list also and pretty much decided to follow python in these matters. Not that I really agree with Guido here, but it's a small point and when in Rome... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitrey.kroshko at scipy.org Fri Aug 29 14:02:29 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Fri, 29 Aug 2008 21:02:29 +0300 Subject: [Numpy-discussion] isn't it a bug in array.fill()? In-Reply-To: References: <48B82F9B.90600@scipy.org> <48B834E9.2050305@scipy.org> Message-ID: <48B839B5.8050008@scipy.org> Keith Goodman wrote: > On Fri, Aug 29, 2008 at 10:51 AM, Keith Goodman wrote: > >> On Fri, Aug 29, 2008 at 10:42 AM, dmitrey wrote: >> >>> Keith Goodman wrote: >>> >>>> Yeah, I do stuff like that too. fill works in place so it returns None. >>>> >>>> >>>> >>>>>> x = np.array([1,2]) >>>>>> x.fill(10) >>>>>> x >>>>>> >>>>>> >>>> array([10, 10]) >>>> >>>> >>>>>> x = x.fill(10) # <-- Danger! >>>>>> print x >>>>>> >>>>>> >>>> None >>>> >>>> >>> Since result "None" is never used it would be better to return reference >>> to the modified array, ... >>> >> I like that idea. A lot of numpy functions return a reference to the >> modified array when the output array (out) is specified. >> > > But python doesn't do that. For example, x.sort() returns None in > python. Should it return None in numpy? > I think the value "None" that any method always returns is hardly used by anyone, so backward compatibility will not suffer much. Since in-place modification remains (along with new-style returning value that is reference to the array), there should be no backward incompatibilities at all. D. From aisaac at american.edu Fri Aug 29 14:08:08 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 29 Aug 2008 14:08:08 -0400 Subject: [Numpy-discussion] isn't it a bug in array.fill()? In-Reply-To: References: <48B82F9B.90600@scipy.org> <48B834E9.2050305@scipy.org> Message-ID: <48B83B08.8000607@american.edu> I suppose all the discussion on comp.lang.python about list methods (especially sort) is becoming relevant to this thread. Cheers, Alan Isaac From stefan at sun.ac.za Fri Aug 29 17:00:19 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 29 Aug 2008 23:00:19 +0200 Subject: [Numpy-discussion] isn't it a bug in array.fill()? In-Reply-To: References: <48B82F9B.90600@scipy.org> <48B834E9.2050305@scipy.org> Message-ID: <9457e7c80808291400g4a00a1b6p16557c430c366855@mail.gmail.com> 2008/8/29 Charles R Harris : >> I like that idea. A lot of numpy functions return a reference to the >> modified array when the output array (out) is specified. > > Google up the various discussions of python sort to see why Guido doesn't > like that sort of thing. We've had that discussion on this list also and > pretty much decided to follow python in these matters. Not that I really > agree with Guido here, but it's a small point and when in Rome... At first, I also thought it might be more intuitive to return the output array, but then I realised that it would make it more difficult to realise that the operation is being performed in-place. Maybe it is good to remind programmers of what happens under the hood, so that they are not surprised later when their data is "corrupted". St?fan From aisaac at american.edu Fri Aug 29 17:03:43 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 29 Aug 2008 17:03:43 -0400 Subject: [Numpy-discussion] isn't it a bug in array.fill()? In-Reply-To: <9457e7c80808291400g4a00a1b6p16557c430c366855@mail.gmail.com> References: <48B82F9B.90600@scipy.org> <48B834E9.2050305@scipy.org> <9457e7c80808291400g4a00a1b6p16557c430c366855@mail.gmail.com> Message-ID: <48B8642F.5000900@american.edu> St?fan van der Walt wrote: > At first, I also thought it might be more intuitive to return the > output array, but then I realised that it would make it more difficult > to realise that the operation is being performed in-place. Maybe it > is good to remind programmers of what happens under the hood, so that > they are not surprised later when their data is "corrupted". That is essentially the core reason for the behavior of list.sort(). Cheers, Alan From charlesr.harris at gmail.com Fri Aug 29 17:18:35 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 29 Aug 2008 15:18:35 -0600 Subject: [Numpy-discussion] isn't it a bug in array.fill()? In-Reply-To: <48B8642F.5000900@american.edu> References: <48B82F9B.90600@scipy.org> <48B834E9.2050305@scipy.org> <9457e7c80808291400g4a00a1b6p16557c430c366855@mail.gmail.com> <48B8642F.5000900@american.edu> Message-ID: On Fri, Aug 29, 2008 at 3:03 PM, Alan G Isaac wrote: > St?fan van der Walt wrote: > > At first, I also thought it might be more intuitive to return the > > output array, but then I realised that it would make it more difficult > > to realise that the operation is being performed in-place. Maybe it > > is good to remind programmers of what happens under the hood, so that > > they are not surprised later when their data is "corrupted". > > That is essentially the core reason for the behavior of list.sort(). > IIRC, it is also the case that Guido doesn't like the a.foo().bar().oops() style of programming, preferring one operation/line. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Fri Aug 29 17:35:59 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 29 Aug 2008 14:35:59 -0700 Subject: [Numpy-discussion] How do I do this? Message-ID: <48B86BBF.2080203@noaa.gov> HI all, I need to do something I thought would be simple -- set all the values of an array to some minimum. so I did this: >>> min_value = 2 >>> a = np.array((1, 2, 3, 4, 5,)) >>> np.maximum(a, min_value) array([2, 2, 3, 4, 5]) all was well... then I realized that a could have negative numbers in in, and I really wanted the absolute value to be greater than than minimum: >>> a = np.array((1, 2, 3, 4, -5,)) >>> np.maximum(a, min_value) array([2, 2, 3, 4, 2]) oops! so I added a sign() and abs(): >>> np.sign(a) * np.maximum(np.abs(a), min_value) array([ 2, 2, 3, 4, -5]) all was well. However it turns out a could contain a zero: >>> a = np.array((0, 1, 2, 3, 4, -5,)) >>> np.sign(a) * np.maximum(np.abs(a), min_value) array([ 0, 2, 2, 3, 4, -5]) Darn! I want that zero to become a 2, but sign(0) = 0, so that doesn't work. How can I do this without another line of code special casing the 0, which isn't that big I deal, but it seems kind of ugly... >>> a[a==0] = min_value >>> np.sign(a) * np.maximum(np.abs(a), min_value) array([ 2, 2, 2, 3, 4, -5]) thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From kwgoodman at gmail.com Fri Aug 29 17:52:47 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 29 Aug 2008 14:52:47 -0700 Subject: [Numpy-discussion] How do I do this? In-Reply-To: References: <48B86BBF.2080203@noaa.gov> Message-ID: On Fri, Aug 29, 2008 at 2:49 PM, Keith Goodman wrote: > On Fri, Aug 29, 2008 at 2:35 PM, Christopher Barker > wrote: >> HI all, >> >> I need to do something I thought would be simple -- set all the values >> of an array to some minimum. so I did this: >> >> >>> min_value = 2 >> >>> a = np.array((1, 2, 3, 4, 5,)) >> >>> np.maximum(a, min_value) >> array([2, 2, 3, 4, 5]) >> >> all was well... then I realized that a could have negative numbers in >> in, and I really wanted the absolute value to be greater than than minimum: >> >> >>> a = np.array((1, 2, 3, 4, -5,)) >> >>> np.maximum(a, min_value) >> array([2, 2, 3, 4, 2]) >> >> oops! >> >> so I added a sign() and abs(): >> >> >>> np.sign(a) * np.maximum(np.abs(a), min_value) >> array([ 2, 2, 3, 4, -5]) >> >> all was well. However it turns out a could contain a zero: >> >> >>> a = np.array((0, 1, 2, 3, 4, -5,)) >> >>> np.sign(a) * np.maximum(np.abs(a), min_value) >> array([ 0, 2, 2, 3, 4, -5]) >> >> Darn! I want that zero to become a 2, but sign(0) = 0, so that doesn't >> work. >> >> How can I do this without another line of code special casing the 0, >> which isn't that big I deal, but it seems kind of ugly... >> >> >>> a[a==0] = min_value >> >>> np.sign(a) * np.maximum(np.abs(a), min_value) >> array([ 2, 2, 2, 3, 4, -5]) > > Does this work? > >>> x > array([ 1., 2., -5., -1., 0.]) >>> np.sign(x) * np.clip(np.absolute(x), 2, np.inf) > array([ 2., 2., -5., -2., 0.]) Oh, I guess 0 is less than 2. From kwgoodman at gmail.com Fri Aug 29 17:49:38 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 29 Aug 2008 14:49:38 -0700 Subject: [Numpy-discussion] How do I do this? In-Reply-To: <48B86BBF.2080203@noaa.gov> References: <48B86BBF.2080203@noaa.gov> Message-ID: On Fri, Aug 29, 2008 at 2:35 PM, Christopher Barker wrote: > HI all, > > I need to do something I thought would be simple -- set all the values > of an array to some minimum. so I did this: > > >>> min_value = 2 > >>> a = np.array((1, 2, 3, 4, 5,)) > >>> np.maximum(a, min_value) > array([2, 2, 3, 4, 5]) > > all was well... then I realized that a could have negative numbers in > in, and I really wanted the absolute value to be greater than than minimum: > > >>> a = np.array((1, 2, 3, 4, -5,)) > >>> np.maximum(a, min_value) > array([2, 2, 3, 4, 2]) > > oops! > > so I added a sign() and abs(): > > >>> np.sign(a) * np.maximum(np.abs(a), min_value) > array([ 2, 2, 3, 4, -5]) > > all was well. However it turns out a could contain a zero: > > >>> a = np.array((0, 1, 2, 3, 4, -5,)) > >>> np.sign(a) * np.maximum(np.abs(a), min_value) > array([ 0, 2, 2, 3, 4, -5]) > > Darn! I want that zero to become a 2, but sign(0) = 0, so that doesn't > work. > > How can I do this without another line of code special casing the 0, > which isn't that big I deal, but it seems kind of ugly... > > >>> a[a==0] = min_value > >>> np.sign(a) * np.maximum(np.abs(a), min_value) > array([ 2, 2, 2, 3, 4, -5]) Does this work? >> x array([ 1., 2., -5., -1., 0.]) >> np.sign(x) * np.clip(np.absolute(x), 2, np.inf) array([ 2., 2., -5., -2., 0.]) From aisaac at american.edu Fri Aug 29 17:55:51 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 29 Aug 2008 17:55:51 -0400 Subject: [Numpy-discussion] How do I do this? In-Reply-To: <48B86BBF.2080203@noaa.gov> References: <48B86BBF.2080203@noaa.gov> Message-ID: <48B87067.3090604@american.edu> Christopher Barker wrote: > I need to do something I thought would be simple -- set > all the values of an array to some minimum. so I did this: Does this do what you want? idx = np.abs(a) References: <48B86BBF.2080203@noaa.gov> Message-ID: On Fri, Aug 29, 2008 at 2:52 PM, Keith Goodman wrote: > On Fri, Aug 29, 2008 at 2:49 PM, Keith Goodman wrote: >> On Fri, Aug 29, 2008 at 2:35 PM, Christopher Barker >> wrote: >>> HI all, >>> >>> I need to do something I thought would be simple -- set all the values >>> of an array to some minimum. so I did this: >>> >>> >>> min_value = 2 >>> >>> a = np.array((1, 2, 3, 4, 5,)) >>> >>> np.maximum(a, min_value) >>> array([2, 2, 3, 4, 5]) >>> >>> all was well... then I realized that a could have negative numbers in >>> in, and I really wanted the absolute value to be greater than than minimum: >>> >>> >>> a = np.array((1, 2, 3, 4, -5,)) >>> >>> np.maximum(a, min_value) >>> array([2, 2, 3, 4, 2]) >>> >>> oops! >>> >>> so I added a sign() and abs(): >>> >>> >>> np.sign(a) * np.maximum(np.abs(a), min_value) >>> array([ 2, 2, 3, 4, -5]) >>> >>> all was well. However it turns out a could contain a zero: >>> >>> >>> a = np.array((0, 1, 2, 3, 4, -5,)) >>> >>> np.sign(a) * np.maximum(np.abs(a), min_value) >>> array([ 0, 2, 2, 3, 4, -5]) >>> >>> Darn! I want that zero to become a 2, but sign(0) = 0, so that doesn't >>> work. >>> >>> How can I do this without another line of code special casing the 0, >>> which isn't that big I deal, but it seems kind of ugly... >>> >>> >>> a[a==0] = min_value >>> >>> np.sign(a) * np.maximum(np.abs(a), min_value) >>> array([ 2, 2, 2, 3, 4, -5]) >> >> Does this work? >> >>>> x >> array([ 1., 2., -5., -1., 0.]) >>>> np.sign(x) * np.clip(np.absolute(x), 2, np.inf) >> array([ 2., 2., -5., -2., 0.]) > > Oh, I guess 0 is less than 2. If you only have integers then >> x array([ 1, 2, -5, -1, 0]) >> np.sign(x+1e-16) * np.maximum(np.abs(x), 2) array([ 2., 2., -5., -2., 2.]) From cournape at gmail.com Fri Aug 29 18:08:08 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 30 Aug 2008 07:08:08 +0900 Subject: [Numpy-discussion] np.test() Bus error on OSX. Reopened Ticket 816 In-Reply-To: <764e38540808291044l18379e01s28d9198a6643b6c4@mail.gmail.com> References: <764e38540808291044l18379e01s28d9198a6643b6c4@mail.gmail.com> Message-ID: <5b8d13220808291508w410fa334h49a874826418ffe3@mail.gmail.com> On Sat, Aug 30, 2008 at 2:44 AM, Christopher Burns wrote: > I reopened Ticket 816: > http://scipy.org/scipy/numpy/ticket/816 > Hi Chris, Could you try to do a clean install, because when Travis and me (well, mostly Travis) look at that problem, we were both on mac os X, and I can't reproduce the error either. David From Chris.Barker at noaa.gov Fri Aug 29 18:08:25 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 29 Aug 2008 15:08:25 -0700 Subject: [Numpy-discussion] How do I do this? In-Reply-To: <48B87067.3090604@american.edu> References: <48B86BBF.2080203@noaa.gov> <48B87067.3090604@american.edu> Message-ID: <48B87359.4010805@noaa.gov> Alan G Isaac wrote: > Does this do what you want? > idx = np.abs(a) a[idx] = min_value yup, that's it. I had forgotten about that kind of indexing, even though I used it for: a[a==0] = min_value Keith Goodman wrote: > If you only have integers then > >>> x > array([ 1, 2, -5, -1, 0]) >>> np.sign(x+1e-16) * np.maximum(np.abs(x), 2) > array([ 2., 2., -5., -2., 2.]) that would work, though I like Alan's better. thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Fri Aug 29 18:13:36 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 29 Aug 2008 15:13:36 -0700 Subject: [Numpy-discussion] How do I do this? In-Reply-To: <48B87067.3090604@american.edu> References: <48B86BBF.2080203@noaa.gov> <48B87067.3090604@american.edu> Message-ID: <48B87490.3000300@noaa.gov> Alan G Isaac wrote: > Does this do what you want? > idx = np.abs(a) a[idx] = min_value out of interest, is there any performance difference between: a[np.abs(a) References: <48B86BBF.2080203@noaa.gov> <48B87067.3090604@american.edu> <48B87359.4010805@noaa.gov> Message-ID: On Fri, Aug 29, 2008 at 3:08 PM, Christopher Barker wrote: > Alan G Isaac wrote: >> Does this do what you want? >> idx = np.abs(a)> a[idx] = min_value > > yup, that's it. I had forgotten about that kind of indexing, even though > I used it for: a[a==0] = min_value > > Keith Goodman wrote: >> If you only have integers then >> >>>> x >> array([ 1, 2, -5, -1, 0]) >>>> np.sign(x+1e-16) * np.maximum(np.abs(x), 2) >> array([ 2., 2., -5., -2., 2.]) > > that would work, though I like Alan's better. I thought -1 should go to -2. If not, then my attempt doesn't work. From Chris.Barker at noaa.gov Fri Aug 29 21:09:25 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 29 Aug 2008 18:09:25 -0700 Subject: [Numpy-discussion] How do I do this? In-Reply-To: References: <48B86BBF.2080203@noaa.gov> <48B87067.3090604@american.edu> <48B87359.4010805@noaa.gov> Message-ID: <48B89DC5.3080004@noaa.gov> Keith Goodman wrote: >> Alan G Isaac wrote: >>> Does this do what you want? >>> idx = np.abs(a)>> a[idx] = min_value >> Keith Goodman wrote: >>> If you only have integers then >>> >>>>> x >>> array([ 1, 2, -5, -1, 0]) >>>>> np.sign(x+1e-16) * np.maximum(np.abs(x), 2) >>> array([ 2., 2., -5., -2., 2.]) >> that would work, though I like Alan's better. > > I thought -1 should go to -2. If not, then my attempt doesn't work. arrg! yes, you are quite right -- -1 should go to -2. That's what I get for trying to get home early before the holiday weekend.... honestly, in this case, it probably doesn't matter much. I'm using this in my wxPython data-visualization oriented 2-d drawing library: FloatCanvas. When you zoom out an a rectangle, at some point it gets very small and disappears. Often that is appropriate. however, in some users, folks want it to reach a minimum size, say a few pixels, and not get smaller. The trick is that rectangles can have a height that is negative due to swapping the y axis to be y-up, rather than y-down, like it usually is for graphics drawing. So if the user wants a minimum height of 2, and the rect has a height of -1 after scaling, it should really get a height of -2. as we're only talking a couple pixels, it probably doesn't matter, and that's why I didn't notice the error. thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception From aisaac at american.edu Fri Aug 29 21:28:58 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 29 Aug 2008 21:28:58 -0400 Subject: [Numpy-discussion] How do I do this? In-Reply-To: <48B89DC5.3080004@noaa.gov> References: <48B86BBF.2080203@noaa.gov> <48B87067.3090604@american.edu> <48B87359.4010805@noaa.gov> <48B89DC5.3080004@noaa.gov> Message-ID: <48B8A25A.4000206@american.edu> >>> Alan G Isaac wrote: >>>> Does this do what you want? >>>> idx = np.abs(a)>>> a[idx] = min_value Christopher Barker wrote: > -1 should go to -2 ok then: a[idx] = min_value*np.sign(a[idx]) I think that's what you are saying? Cheers, Alan Isaac From alan at ajackson.org Fri Aug 29 22:22:29 2008 From: alan at ajackson.org (Alan Jackson) Date: Fri, 29 Aug 2008 21:22:29 -0500 Subject: [Numpy-discussion] Advice on converting iterator into array efficiently In-Reply-To: <200808291219.39670.faltet@pytables.org> References: <20080828195709.15deca69@ajackson.org> <200808291129.47228.faltet@pytables.org> <200808291219.39670.faltet@pytables.org> Message-ID: <20080829212229.70762f8f@ajackson.org> I tested all three offered solutions : t = table[:] # convert to structured array collections = np.unique(t['collection']) for collection in collections: cond = t['collection'] == collection energy_this_collection = t['energy'][cond] ---------------------------------- energies = {} for row in table: c = row['collection'] e = row['energy'] if c in energies: energies[c].append(e) else: energies[c] = [e] # Convert the lists in numpy arrays for key in energies: energies[key] = numpy.array(energies[key]) --------------------------------- for c in np.unique(table.col('collection')) : print c,' : ', table.readWhere('collection == c', field='energy') and the timing results were rather dramatic : time 1 = 0.79 time 2 = 0.08 time 3 = 10.35 This was a test on a relatively small table. I'll have to try it out on something really big next and see how the memory usage works out. Thanks guys! - Alan -- ----------------------------------------------------------------------- | Alan K. Jackson | To see a World in a Grain of Sand | | alan at ajackson.org | And a Heaven in a Wild Flower, | | www.ajackson.org | Hold Infinity in the palm of your hand | | Houston, Texas | And Eternity in an hour. - Blake | ----------------------------------------------------------------------- From Chris.Barker at noaa.gov Sat Aug 30 00:02:59 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 29 Aug 2008 21:02:59 -0700 Subject: [Numpy-discussion] How do I do this? In-Reply-To: <48B8A25A.4000206@american.edu> References: <48B86BBF.2080203@noaa.gov> <48B87067.3090604@american.edu> <48B87359.4010805@noaa.gov> <48B89DC5.3080004@noaa.gov> <48B8A25A.4000206@american.edu> Message-ID: <48B8C673.2010901@noaa.gov> Alan G Isaac wrote: >>>> Alan G Isaac wrote: >>>>> Does this do what you want? >>>>> idx = np.abs(a)>>>> a[idx] = min_value > > Christopher Barker wrote: >> -1 should go to -2 > > ok then: > a[idx] = min_value*np.sign(a[idx]) > I think that's what you are saying? nope: >>> a = np.array((0, 1, -1, 2, 3, -5,)) >>> idx = np.abs(a)>> a = np.array((0, 1, -1, 2, 3, -5,)) >>> a array([ 0, 1, -1, 2, 3, -5]) 0 should go to 2 --it's not, because sign(0) == 0, so the 2 gets turned to zero, my original problem. Frankly, as I've only seen sign() used for this sort of thing, it makes me think that sign(0) really should be 1 -- the identity -- I know that 0 is neither positive nor negative, but as it is often used in this way, it would be more useful for sign(0) to be the identity. I'm not advocating a change in numpy, I'm sure this is established standard, but it does seem I need to special case zero. (Or fudge it by adding an epsilon, as Keith suggested. thanks all, -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception From stefan at sun.ac.za Sat Aug 30 02:46:28 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sat, 30 Aug 2008 08:46:28 +0200 Subject: [Numpy-discussion] How do I do this? In-Reply-To: <48B86BBF.2080203@noaa.gov> References: <48B86BBF.2080203@noaa.gov> Message-ID: <9457e7c80808292346k60467584t3dc14e413733171e@mail.gmail.com> 2008/8/29 Christopher Barker : > How can I do this without another line of code special casing the 0, > which isn't that big I deal, but it seems kind of ugly... > > >>> a[a==0] = min_value > >>> np.sign(a) * np.maximum(np.abs(a), min_value) > array([ 2, 2, 2, 3, 4, -5]) Maybe (np.sign(a) | 1) * np.maximum(np.abs(a), min_value) is a little bit easier on the eye. Cheers St?fan From aisaac at american.edu Sat Aug 30 09:05:52 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 30 Aug 2008 09:05:52 -0400 Subject: [Numpy-discussion] How do I do this? In-Reply-To: <48B8C673.2010901@noaa.gov> References: <48B86BBF.2080203@noaa.gov> <48B87067.3090604@american.edu> <48B87359.4010805@noaa.gov> <48B89DC5.3080004@noaa.gov> <48B8A25A.4000206@american.edu> <48B8C673.2010901@noaa.gov> Message-ID: <48B945B0.9050301@american.edu> Christopher Barker wrote: > 0 should go to 2 --it's not, because sign(0) == 0, so the 2 gets turned > to zero, my original problem. Odd asymmetry. Then you can use Keith's version. Or a[idx] = min_value*(np.sign(a[idx]) + (a[idx]==0)) Alan From aisaac at american.edu Sat Aug 30 09:24:27 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 30 Aug 2008 09:24:27 -0400 Subject: [Numpy-discussion] How do I do this? In-Reply-To: <9457e7c80808292346k60467584t3dc14e413733171e@mail.gmail.com> References: <48B86BBF.2080203@noaa.gov> <9457e7c80808292346k60467584t3dc14e413733171e@mail.gmail.com> Message-ID: <48B94A0B.9000207@american.edu> St?fan van der Walt wrote: > (np.sign(a) | 1) ... Ah, that's nice. How about idx = np.abs(a) References: <48B86BBF.2080203@noaa.gov> <9457e7c80808292346k60467584t3dc14e413733171e@mail.gmail.com> <48B94A0B.9000207@american.edu> Message-ID: On Sat, Aug 30, 2008 at 6:24 AM, Alan G Isaac wrote: > St?fan van der Walt wrote: >> (np.sign(a) | 1) ... > > Ah, that's nice. How about > idx = np.abs(a) a[idx] = min_value*(np.sign(a[idx]) | 1) Or, since he asked to do it in one line, >> min_value = 2 >> x array([ 1, 2, -5, -1, 0]) >> np.sign(x | 1) * np.maximum(np.abs(x), min_value) array([ 2, 2, -5, -2, 2]) From kwgoodman at gmail.com Sat Aug 30 10:39:08 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Sat, 30 Aug 2008 07:39:08 -0700 Subject: [Numpy-discussion] How do I do this? In-Reply-To: References: <48B86BBF.2080203@noaa.gov> <9457e7c80808292346k60467584t3dc14e413733171e@mail.gmail.com> <48B94A0B.9000207@american.edu> Message-ID: On Sat, Aug 30, 2008 at 7:29 AM, Keith Goodman wrote: > On Sat, Aug 30, 2008 at 6:24 AM, Alan G Isaac wrote: >> St?fan van der Walt wrote: >>> (np.sign(a) | 1) ... >> >> Ah, that's nice. How about >> idx = np.abs(a)> a[idx] = min_value*(np.sign(a[idx]) | 1) > > Or, since he asked to do it in one line, > >>> min_value = 2 >>> x > array([ 1, 2, -5, -1, 0]) >>> np.sign(x | 1) * np.maximum(np.abs(x), min_value) > array([ 2, 2, -5, -2, 2]) Why did I send that? Sorry. I saw St?fan's (np.sign(a) | 1), plugged it in and sent it. But St?fan already did that. An automated 5 minute delay on sending numpy email would cut the email you get from me in half. From aisaac at american.edu Sat Aug 30 15:48:24 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 30 Aug 2008 15:48:24 -0400 Subject: [Numpy-discussion] default index inconsistency? Message-ID: <48B9A408.5030804@american.edu> I think the question of the timing of removing the remaining default index inconsistencies in NumPy was raised by not answered? Anyway, what is the answer? I just ran into this again, this time with ``argsort``, which still has ``axis=-1`` as the default. But maybe it is intentional to keep this? E.g., a.sort(axis=None) is refused. I would have expected it to return the same as np.sort(a.flat), possibly (?) reshaped. Alan Isaac From Chris.Barker at noaa.gov Sat Aug 30 19:24:58 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Sat, 30 Aug 2008 16:24:58 -0700 Subject: [Numpy-discussion] How do I do this? In-Reply-To: <9457e7c80808292346k60467584t3dc14e413733171e@mail.gmail.com> References: <48B86BBF.2080203@noaa.gov> <9457e7c80808292346k60467584t3dc14e413733171e@mail.gmail.com> Message-ID: <48B9D6CA.4090800@noaa.gov> St?fan van der Walt wrote: > Maybe > > (np.sign(a) | 1) * np.maximum(np.abs(a), min_value) > > is a little bit easier on the eye. nice! that does it. somehow I never think of using or. Alan G Isaac wrote: > idx = np.abs(a) a[idx] = min_value*(np.sign(a[idx]) + (a[idx]==0)) that's do it too, but it's a bit more wordy, and I'm guessing slower, though that's really irrelevant here. thanks all, -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception From alan at ajackson.org Sat Aug 30 22:02:46 2008 From: alan at ajackson.org (Alan Jackson) Date: Sat, 30 Aug 2008 21:02:46 -0500 Subject: [Numpy-discussion] at my wits end over an error message... Message-ID: <20080830210246.7474c88e@ajackson.org> I been beating myself up over this bit of code for far too long now - I know I must be missing something really simple, but what is it? TYPICALLY_UINT_COLUMNS = ['Track', 'Bin', 'code', 'horizon'] ...... dtypes = [ ] for i in range(0, len(self.var_list)) : if TYPICALLY_UINT_COLUMNS.count(self.var_list[i]) > 0: dtypes.append((self.var_list[i], ' References: <12D2AD1D-7742-4248-9C5D-9EF3EE3CEE06@gmail.com> Message-ID: [ Sorry I never sent this, I just found it in my drafts folder. Just in case it's useful to the OP, here it is. ] On Thu, Jul 10, 2008 at 9:38 AM, Dan Lussier wrote: > r2 = numpy.power(r2,2).sum(axis=1) > > r2 = numpy.extract(r2 coord[i] = r2.shape[0]-1 > r2 = numpy.extract(r2 != 0, r2) # > avoid problems of inf when considering own atom. > r2 = numpy.divide(1.0,r2) > ljTotal[i] = numpy.multiply(2,(numpy.power(r2,6)- > numpy.power(r2,3))).sum(axis=0) That's a Lennard-Jones potential, and there seems to be a fair amount of work on FMM (fast multipole method) for that type of potential. Depending on what the final set of quantities you're after is, the FMM may be worth looking into. Cheers, f From zachary.pincus at yale.edu Sat Aug 30 23:10:10 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Sat, 30 Aug 2008 23:10:10 -0400 Subject: [Numpy-discussion] at my wits end over an error message... In-Reply-To: <20080830210246.7474c88e@ajackson.org> References: <20080830210246.7474c88e@ajackson.org> Message-ID: <05197C5B-52E9-43D1-BDE2-3D869DC42C4B@yale.edu> Hi Alan, > Traceback (most recent call last): > File "/usr/local/lib/python2.5/site-packages/enthought.traits-2.0.4- > py2.5-linux-i686.egg/enthought/traits/trait_notifiers.py", line 325, > in call_1 > self.handler( object ) > File "TrimMapl_1.py", line 98, in _Run_fired > outdata = np.array(outdata, dtype=dtypes) > TypeError: expected a readable buffer object This would make it appear that the problem is not with numpy per se, but with the traits library, or how you're using it... I'm not too familiar with traits, so I can't really provide any advice there. Can you disable whatever it is you're doing with traits for the time being, to see if that solves it? Maybe not have whatever notifier is presumably listening to outdata? Also, for the record, your inner loop could be rendered in more idiomatic python as: TYPICALLY_UINT_COLUMNS = set(['Track', 'Bin', 'code', 'horizon']) ...... dtypes = [] for var in self.var_list: if var in TYPICALLY_UINT_COLUMNS: dtypes.append(var, ' TYPICALLY_UINT_COLUMNS = ['Track', 'Bin', 'code', 'horizon'] > ...... > dtypes = [ ] > for i in range(0, len(self.var_list)) : > if TYPICALLY_UINT_COLUMNS.count(self.var_list[i]) > 0: > dtypes.append((self.var_list[i], ' else : > dtypes.append((self.var_list[i], ' print "dtypes = ", dtypes > print outdata[0] > outdata = np.array(outdata, dtype=dtypes) From alan at ajackson.org Sat Aug 30 23:14:40 2008 From: alan at ajackson.org (Alan Jackson) Date: Sat, 30 Aug 2008 22:14:40 -0500 Subject: [Numpy-discussion] at my wits end over an error message... In-Reply-To: <05197C5B-52E9-43D1-BDE2-3D869DC42C4B@yale.edu> References: <20080830210246.7474c88e@ajackson.org> <05197C5B-52E9-43D1-BDE2-3D869DC42C4B@yale.edu> Message-ID: <20080830221440.776c2331@ajackson.org> Thanks. I'm still struggling with learning the idiom. Too much perl to unlearn. 8-) On Sat, 30 Aug 2008 23:10:10 -0400 Zachary Pincus wrote: > Hi Alan, > > > Traceback (most recent call last): > > File "/usr/local/lib/python2.5/site-packages/enthought.traits-2.0.4- > > py2.5-linux-i686.egg/enthought/traits/trait_notifiers.py", line 325, > > in call_1 > > self.handler( object ) > > File "TrimMapl_1.py", line 98, in _Run_fired > > outdata = np.array(outdata, dtype=dtypes) > > TypeError: expected a readable buffer object > > This would make it appear that the problem is not with numpy per se, > but with the traits library, or how you're using it... I'm not too > familiar with traits, so I can't really provide any advice there. > > Can you disable whatever it is you're doing with traits for the time > being, to see if that solves it? Maybe not have whatever notifier is > presumably listening to outdata? > > Also, for the record, your inner loop could be rendered in more > idiomatic python as: > > TYPICALLY_UINT_COLUMNS = set(['Track', 'Bin', 'code', 'horizon']) > ...... > dtypes = [] > for var in self.var_list: > if var in TYPICALLY_UINT_COLUMNS: > dtypes.append(var, ' else : > dtypes.append(var, ' or even: > > TYPICALLY_UINT_COLUMNS = set(['Track', 'Bin', 'code', 'horizon']) > dtypes = [(var, ' var in self.var_list] > > if you really want to break out the new ternary operator (py 2.5 and > above). > > Not that it matters at all in this case, but (in addition to being > easier to read and write) loops like the above will be much faster > than the original code if you've got some larger parsing operations to > do. (Also, note that the 'in' operator works on lists and tuples as > well as sets and dicts, but for the former two, it's a linear search, > while for the latter, a hash lookup.) > > Zach > > > > TYPICALLY_UINT_COLUMNS = ['Track', 'Bin', 'code', 'horizon'] > > ...... > > dtypes = [ ] > > for i in range(0, len(self.var_list)) : > > if TYPICALLY_UINT_COLUMNS.count(self.var_list[i]) > 0: > > dtypes.append((self.var_list[i], ' > else : > > dtypes.append((self.var_list[i], ' > print "dtypes = ", dtypes > > print outdata[0] > > outdata = np.array(outdata, dtype=dtypes) > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -- ----------------------------------------------------------------------- | Alan K. Jackson | To see a World in a Grain of Sand | | alan at ajackson.org | And a Heaven in a Wild Flower, | | www.ajackson.org | Hold Infinity in the palm of your hand | | Houston, Texas | And Eternity in an hour. - Blake | ----------------------------------------------------------------------- From faltet at pytables.org Sun Aug 31 08:30:47 2008 From: faltet at pytables.org (Francesc Alted) Date: Sun, 31 Aug 2008 14:30:47 +0200 Subject: [Numpy-discussion] Advice on converting iterator into array efficiently In-Reply-To: <20080829212229.70762f8f@ajackson.org> References: <20080828195709.15deca69@ajackson.org> <200808291219.39670.faltet@pytables.org> <20080829212229.70762f8f@ajackson.org> Message-ID: <200808311430.47435.faltet@pytables.org> A Saturday 30 August 2008, Alan Jackson escrigu?: > I tested all three offered solutions : > > t = table[:] # convert to structured array > collections = np.unique(t['collection']) > for collection in collections: > cond = t['collection'] == collection > energy_this_collection = t['energy'][cond] > ---------------------------------- > > energies = {} > for row in table: > c = row['collection'] > e = row['energy'] > if c in energies: > energies[c].append(e) > else: > energies[c] = [e] > > # Convert the lists in numpy arrays > for key in energies: > energies[key] = numpy.array(energies[key]) > --------------------------------- > > for c in np.unique(table.col('collection')) : > print c,' : ', table.readWhere('collection == c', field='energy') > > and the timing results were rather dramatic : > > time 1 = 0.79 > time 2 = 0.08 > time 3 = 10.35 > > This was a test on a relatively small table. I'll have to try it out > on something really big next and see how the memory usage works out. Solution 1 is loading the entire table in memory (notice the ``[:]`` operator), so if your table is large this is probably not what you want. With solution 2, you will end loading only the energies in memory (first in list form and then as NumPy arrays), which, if your table has many other fields, can be a big win. Finally, solution 3 is the one that takes less memory, as it only requires to load in memory the collection column (I'm assuming that this column has a lighter datatype than the energy one). For what is worth, the most conservative solution in terms of memory usage would be a combination of 2 and 3: # Find the unique collections collections = [] for row in table: c = row['collection'] if c not in collections: collections.append(c) # Get the energy collections for c in sorted(collections): e = table.readWhere('collection == c', field='energy') print c,' : ', e Of course, and as I said in a previous message, this implies many reads of the table (this is why it is the slowest one). In case that you need both speed and minimum memory consumption, indexing the collection column with PyTables Pro could help, but as the bottleneck is probably in the sparse reads of the table (performed by the ``readWhere`` method), the gain shouldn't be too much in this case. Cheers, -- Francesc Alted From aisaac at american.edu Sun Aug 31 12:26:03 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 31 Aug 2008 12:26:03 -0400 Subject: [Numpy-discussion] sort documentation Message-ID: <48BAC61B.4060508@american.edu> I find this confusing: numpy.sort(a, axis=-1, kind='quicksort', order=None) Return copy of 'a' sorted along the given axis. Perform an inplace sort along the given axis using the algorithm specified by the kind keyword. I suppose the last bit is supposed to refer to the ``sort`` method rather than the function, but I do not see any signal that this is the case. Cheers, Alan Isaac From dmitrey.kroshko at scipy.org Sun Aug 31 12:33:00 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Sun, 31 Aug 2008 19:33:00 +0300 Subject: [Numpy-discussion] sort documentation In-Reply-To: <48BAC61B.4060508@american.edu> References: <48BAC61B.4060508@american.edu> Message-ID: <48BAC7BC.5050802@scipy.org> As for me I can't understand the general rule: when numpy funcs return copy and when reference? For example why x.fill() returns None (do inplace modification) while x.ravel(), x.flatten() returns copy? Why the latters don't do inplace modification, as should be expected? D. Alan G Isaac wrote: > I find this confusing: > > numpy.sort(a, axis=-1, kind='quicksort', order=None) > > Return copy of 'a' sorted along the given axis. > > Perform an inplace sort along the given axis using the algorithm > specified by the kind keyword. > > I suppose the last bit is supposed to refer to the ``sort`` > method rather than the function, but I do not see any signal > that this is the case. > > Cheers, > Alan Isaac > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > From aisaac at american.edu Sun Aug 31 13:19:58 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 31 Aug 2008 13:19:58 -0400 Subject: [Numpy-discussion] return value of array methods In-Reply-To: <48BAC7BC.5050802@scipy.org> References: <48BAC61B.4060508@american.edu> <48BAC7BC.5050802@scipy.org> Message-ID: <48BAD2BE.4020102@american.edu> dmitrey wrote: > As for me I can't understand the general rule: when numpy funcs return > copy and when reference? > > For example why x.fill() returns None (do inplace modification) while > x.ravel(), x.flatten() returns copy? Why the latters don't do inplace > modification, as should be expected? I take the rule to be to return the result when it is a copy or a change of view. So, these seem quite right, taken one at a time. E.g., ``a.fill()`` changes `a` in place, and so returns None. It does not make sense to think of ``ravel`` as doing an in-place modification, it is just a change of view (if possible). And ``flatten`` is specifically for insisting on a copy rather than a change of view. Or so I understand things. Alan Isaac From robert.kern at gmail.com Sun Aug 31 13:25:26 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 31 Aug 2008 10:25:26 -0700 Subject: [Numpy-discussion] sort documentation In-Reply-To: <48BAC7BC.5050802@scipy.org> References: <48BAC61B.4060508@american.edu> <48BAC7BC.5050802@scipy.org> Message-ID: <3d375d730808311025q30f2e8c2w8625a103acef7021@mail.gmail.com> On Sun, Aug 31, 2008 at 09:33, dmitrey wrote: > As for me I can't understand the general rule: when numpy funcs return > copy and when reference? It's driven by use cases and limitations of the particular functions. > For example why x.fill() returns None (do inplace modification) while > x.ravel(), x.flatten() returns copy? Why the latters don't do inplace > modification, as should be expected? That example does not go with the question that you asked, but the reason is that we need a way to quickly set all elements of an array to the same value. The entire use case for that feature is in-place modification. For example, suppose that I want an array of a given shape with all of its values being 2.0. x = numpy.empty(shape) x.fill(2.0) If .fill() were to make a copy, then I've just wasted memory, possibly quite a lot. For .ravel() and .flatten(), we can't modify the object in-place in many instances. Ravelling or flattening simply can't be done in-place for non-contiguous arrays. New memory needs to be allocated and data copied. That's intrinsically not an operation that can be done in-place. Now, to your original question, why some functions return views and some return copies. The difference between .ravel() and .flatten() is that .ravel() returns a view when it can and copies when it must while .flatten() always allocates new memory. The difference is driven by use cases. Sometimes, you will not be modifying the raveled array, and you just need an object with the right shape and data as fast as possible. So when possible, .ravel() returns a view. However, there are use cases when you do need to modify the flattened array afterwards without affecting the original, so we have .flatten(). It will consistently return new memory, so it can be modified freely. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Sun Aug 31 13:27:59 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 31 Aug 2008 10:27:59 -0700 Subject: [Numpy-discussion] sort documentation In-Reply-To: <48BAC61B.4060508@american.edu> References: <48BAC61B.4060508@american.edu> Message-ID: <3d375d730808311027r24428460w3b37baad188f5de1@mail.gmail.com> On Sun, Aug 31, 2008 at 09:26, Alan G Isaac wrote: > I find this confusing: > > numpy.sort(a, axis=-1, kind='quicksort', order=None) > > Return copy of 'a' sorted along the given axis. > > Perform an inplace sort along the given axis using the algorithm > specified by the kind keyword. > > I suppose the last bit is supposed to refer to the ``sort`` > method rather than the function, but I do not see any signal > that this is the case. The docstring is much more complete and correct in SVN. In [18]: sort? Type: function Base Class: Namespace: Interactive File: /Users/rkern/svn/grin/numpy/core/fromnumeric.py Definition: sort(a, axis=-1, kind='quicksort', order=None) Docstring: Return a sorted copy of an array. Parameters ---------- a : array-like Array to be sorted. axis : int, optional Axis along which to sort. If not specified, the flattened array is used. kind : {'quicksort', 'mergesort', 'heapsort'}, optional Sorting algorithm. order : list, optional When `a` is an ndarray with fields defined, this argument specifies which fields to compare first, second, etc. Not all fields need be specified. Returns ------- sorted_array : ndarray Array of same type and shape as `a`. See Also -------- argsort : Indirect sort. lexsort : Indirect stable sort on multiple keys. searchsorted : Find keys in sorted array. Notes ----- The various sorting algorithms are characterized by their average speed, worst case performance, work space size, and whether they are stable. A stable sort keeps items with the same key in the same relative order. The three available algorithms have the following properties: =========== ======= ============= ============ ======= kind speed worst case work space stable =========== ======= ============= ============ ======= 'quicksort' 1 O(n^2) 0 no 'mergesort' 2 O(n*log(n)) ~n/2 yes 'heapsort' 3 O(n*log(n)) 0 no =========== ======= ============= ============ ======= All the sort algorithms make temporary copies of the data when sorting along any but the last axis. Consequently, sorting along the last axis is faster and uses less space than sorting along any other axis. Examples -------- >>> a=np.array([[1,4],[3,1]]) >>> a.sort(1) >>> a array([[1, 4], [1, 3]]) >>> a.sort(0) >>> a array([[1, 3], [1, 4]]) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pav at iki.fi Sun Aug 31 13:37:31 2008 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 31 Aug 2008 17:37:31 +0000 (UTC) Subject: [Numpy-discussion] sort documentation References: <48BAC61B.4060508@american.edu> <3d375d730808311027r24428460w3b37baad188f5de1@mail.gmail.com> Message-ID: Sun, 31 Aug 2008 10:27:59 -0700, Robert Kern wrote: > On Sun, Aug 31, 2008 at 09:26, Alan G Isaac wrote: >> I find this confusing: >> >> numpy.sort(a, axis=-1, kind='quicksort', order=None) >> >> Return copy of 'a' sorted along the given axis. >> >> Perform an inplace sort along the given axis using the algorithm >> specified by the kind keyword. >> >> I suppose the last bit is supposed to refer to the ``sort`` method >> rather than the function, but I do not see any signal that this is the >> case. > > The docstring is much more complete and correct in SVN. Even more complete and correct docstring is here: http://sd-2116.dedibox.fr/pydocweb/doc/numpy.sort/ [clip] > >>> a=np.array([[1,4],[3,1]]) > >>> a.sort(1) > >>> a > array([[1, 4], > [1, 3]]) > >>> a.sort(0) > >>> a > array([[1, 3], > [1, 4]]) These examples were fixed a couple of weeks ago in the editor. -- Pauli Virtanen From aisaac at american.edu Sun Aug 31 13:50:49 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 31 Aug 2008 13:50:49 -0400 Subject: [Numpy-discussion] sort documentation In-Reply-To: References: <48BAC61B.4060508@american.edu> <3d375d730808311027r24428460w3b37baad188f5de1@mail.gmail.com> Message-ID: <48BAD9F9.7020101@american.edu> Thanks. So if I wish to determine the current state of documentation, is the right place to go: http://sd-2116.dedibox.fr/pydocweb/doc/ ? Alan From pav at iki.fi Sun Aug 31 14:21:06 2008 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 31 Aug 2008 18:21:06 +0000 (UTC) Subject: [Numpy-discussion] Updated Numpy reference guide Message-ID: Hi all, I finished the first iteration of incorporating material from Travis Oliphant's "Guide to Numpy" to the Sphinxy reference guide we were constructing in the Doc marathon. Result is here: (the PDF is a bit ugly, though, some content is almost randomly scattered there) http://www.iki.fi/pav/tmp/numpy-refguide/index.xhtml http://www.iki.fi/pav/tmp/numpy-refguide/NumPy.pdf Source is here: (St?fan, if it looks ok to you, could you pull and check if it builds for you when you have time?) https://code.launchpad.net/~pauli-virtanen/scipy/numpy-refguide What I did with the "Guide to Numpy" material was: - Collapsed each of the reference Chapters 3, 6, 8, 9 (ndarrays, scalars, dtypes, ufuncs) with the more introductory material in Chapter 2. - As this was supposed to be a reference guide, I tried to compress the text from Chapter 2 as much as possible, by sticking to definitions and dropping some more tutorial-oriented parts. This may have reduced readability at some points... - I added some small bits or rewrote parts in the above sections in places where I thought it would improve the result. - I did not include material that I thought was better to be put into appropriate docstrings in Numpy. What to do with class docstrings and obscure __xxx__ attributes was not so clear a decision, so what I did for these varies. - The sections about Ufuncs and array indexing are taken almost verbatim from the "Guide to Numpy". The ndarray, scalar and dtype sections somewhat follow the structure of the Guide, but the text is more heavily edited from the original. Some things to do: - Descriptions about constructing items with __new__ methods should probably still be clarified; I just replaced references to __new__ with references to the corresponding classes. - What to do with the material from numpy.doc.* should be decided, as the text there doesn't look like it should go into a reference manual. Some questions: - Is this good enough to go into Numpy SVN at some point? Or should we redo it and base the work closer to the original "Guide to Numpy"? - Does it build for you? (I'd recommend using the development 0.5 version of Sphinx, so that you get the nifty Inter-Sphinx links to the Python documentation.) We are unfortunately beating the Sphinx with a big stick to make it place the documentation of each function or class into a separate file, and to convert the Numpy docstring format to something the Sphinx can fathom. There's also some magic in place to make toctrees:: of function listings more pleasant to the eye. Any comments of what should be improved are welcome. (Even better: clone the bzr branch, make the changes yourself, and put the result somewhere available! E.g. as a bzr bundle or a branch on the launchpad.) -- Pauli Virtanen From stefan at sun.ac.za Sun Aug 31 14:22:56 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 31 Aug 2008 20:22:56 +0200 Subject: [Numpy-discussion] sort documentation In-Reply-To: <48BAD9F9.7020101@american.edu> References: <48BAC61B.4060508@american.edu> <3d375d730808311027r24428460w3b37baad188f5de1@mail.gmail.com> <48BAD9F9.7020101@american.edu> Message-ID: <9457e7c80808311122w68b697afha6437b9a825dc07e@mail.gmail.com> 2008/8/31 Alan G Isaac : > Thanks. So if I wish to determine the current > state of documentation, is the right place to go: > http://sd-2116.dedibox.fr/pydocweb/doc/ ? Jarrod, Could we set up a reverse proxy on doc.scipy.org to serve these pages (until we move the app over to that server permanently)? I'm worried about loading Gael's box. If that's too much work, maybe we should just make the plunge and do the move. Cheers St?fan From ckkart at hoc.net Sun Aug 31 15:14:33 2008 From: ckkart at hoc.net (Christian K.) Date: Sun, 31 Aug 2008 21:14:33 +0200 Subject: [Numpy-discussion] floating point char - bug? Message-ID: Hi, I just came across somethin I never noticed before. I cannot say whether this is due to an update of numpy but it is possible - I am running 1.1.1 on __german__ windows. Here is the observation: a = N.linspace(0,1,5) a array([ 0. , 0.25, 0.5 , 0.75, 1. ]) a.astype(float) array([ 0. , 0.25, 0.5 , 0.75, 1. ]) a[0].astype(float) 0.0 a[1].astype(float) 0,25 As you see in the last line, suddenly numpy picks up the german locale setting and converts the floating point into a comma. It does not affect the '0.0' in the line above and I believe to have seen some other numbers ending with '.0' where the point has not been replaced. I guess this is a bug. In fact I do not like the idea that repr() of a numpy float honours the locale settings. Reagrds, Christian From charlesr.harris at gmail.com Sun Aug 31 15:28:50 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 31 Aug 2008 13:28:50 -0600 Subject: [Numpy-discussion] floating point char - bug? In-Reply-To: References: Message-ID: On Sun, Aug 31, 2008 at 1:14 PM, Christian K. wrote: > Hi, > > I just came across somethin I never noticed before. I cannot say whether > this is due to an update of numpy but it is possible - I am running > 1.1.1 on __german__ windows. Here is the observation: > > a = N.linspace(0,1,5) > a > array([ 0. , 0.25, 0.5 , 0.75, 1. ]) > a.astype(float) > array([ 0. , 0.25, 0.5 , 0.75, 1. ]) > a[0].astype(float) > 0.0 > a[1].astype(float) > 0,25 > > As you see in the last line, suddenly numpy picks up the german locale > setting and converts the floating point into a comma. It does not affect > the '0.0' in the line above and I believe to have seen some other > numbers ending with '.0' where the point has not been replaced. > I guess this is a bug. In fact I do not like the idea that repr() of a > numpy float honours the locale settings. > I'd call it a bug, file a ticket. And we do need to have a policy on locales. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ckkart at hoc.net Sun Aug 31 16:29:21 2008 From: ckkart at hoc.net (Christian K.) Date: Sun, 31 Aug 2008 22:29:21 +0200 Subject: [Numpy-discussion] floating point char - bug? In-Reply-To: References: Message-ID: Charles R Harris schrieb: > > > On Sun, Aug 31, 2008 at 1:14 PM, Christian K. > wrote: > > Hi, > > I just came across somethin I never noticed before. I cannot say whether > this is due to an update of numpy but it is possible - I am running > 1.1.1 on __german__ windows. Here is the observation: > > a = N.linspace(0,1,5) > a > array([ 0. , 0.25, 0.5 , 0.75, 1. ]) > a.astype(float) > array([ 0. , 0.25, 0.5 , 0.75, 1. ]) > a[0].astype(float) > 0.0 > a[1].astype(float) > 0,25 > > As you see in the last line, suddenly numpy picks up the german locale > setting and converts the floating point into a comma. It does not affect > the '0.0' in the line above and I believe to have seen some other > numbers ending with '.0' where the point has not been replaced. > I guess this is a bug. In fact I do not like the idea that repr() of a > numpy float honours the locale settings. > > > I'd call it a bug, file a ticket. And we do need to have a policy on > locales. Done. http://scipy.org/scipy/numpy/ticket/902 Christian