From david at ar.media.kyoto-u.ac.jp Wed Apr 1 01:10:09 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 01 Apr 2009 14:10:09 +0900 Subject: [Numpy-discussion] Numpy 1.3.0 rc1 OS X Installer In-Reply-To: <49D28D7D.9050102@noaa.gov> References: <49CE2587.2050007@ar.media.kyoto-u.ac.jp> <5b8d13220903290753g224ee96am72c1cd93db88f650@mail.gmail.com> <5b8d13220903292356r5e954b90lf92c9f170ec01b13@mail.gmail.com> <49D0778C.5050706@ar.media.kyoto-u.ac.jp> <2BACA6B8-43A0-421E-859B-E6593B588FFF@post.harvard.edu> <5b8d13220903300822m682c7e26jb10975ecf1d9c723@mail.gmail.com> <49D0EA6D.6050601@noaa.gov> <5b8d13220903300919s13702969vd54c70c3a928fc0c@mail.gmail.com> <49D0FCF8.5040908@noaa.gov> <5b8d13220903301027n76634921hf4dc4b1818392ab3@mail.gmail.com> <49D11686.4010305@noaa.gov> <49D1A5A1.2060205@ar.media.kyoto-u.ac.jp> <49D28D7D.9050102@noaa.gov> Message-ID: <49D2F731.70006@ar.media.kyoto-u.ac.jp> Christopher Barker wrote: > It does, but we don't need a binary installer for a python that doesn't > have a binary installer. > Yes, not now - but I would prefer avoiding to have to change the process again when time comes. It may not look like it, but enabling a working process which works well on all platforms including windows took me several days to work properly. And I am in a hurry to go into this again :) > Hmmm -- I don't know virtualenv enough to know what the virtualenv knows > about how it was created... > > However, I'm not sure you need to do what your saying here. I imagine > this workflow: > > set up a virtualenv for, say numpy x.y.rc-z > > play around with it, get everything to build, etc. with plain old > setup.py build, setup.py install, etc. > > Once you are happy, run: > > /Library/Frameworks/Python.framework/Versions/2.5/bin/bdist_mpkg > This means building the same thing twice. Now, for numpy, it is not that a big deal, but for scipy, not so much. If/when we have a good, reliable build farm for mac os x, this point becomes moot, so. > By the way, if you run bdist_mpkg from a version installed into your > virtualenv, you will get an installer that will install into your > virtualenv, whit the path hard coded, so really useless. > That's exactly the problem in the current binary :) > > ah -- maybe that's the issue then -- darn. Are the docs included in the > .mpkg? Do they need to be built for that? > The docs are included in the .dmg, and yes, the doc needs to be built from the same installation (or more exactly the same source). > > yes, I am. > I have not tackled the uninstall part, but I already wrote this to "write in stone" my POV in the whole python packaging situation: http://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observations-cabal-for-a-solution/ > > True. In that case we could put the dylib somewhere obscure: > > /usr/local/lib/scipy1.6/lib/ > Hm, that's strange - why /usr/local/lib ? It is outside the scipy installation. > or even: > > /Library/Frameworks/Python.framework/Versions/2.5/lib/ > That's potentially dangerous: since this directory is likely to be in LIBDIR, it means libgfortran will be taken there or from /usr/local/lib if the user builds numpy/scipy after installing numpy. If it is incompatible with the user gfortran, it will lead to weird issues, hard to debug. This problem bit me on windows 64 bits recently: we did something similar (creating a libpython*.a and put in C:\python*\libs), but the installed library was not 64 bits compatible - I assumed this library was built by python itself, and I have wasted several hours looking elsewhere for a problem caused by numpy.distutils. If we install something like libgfortran, it should be installed privately - but dynamically linking against private libraries is hard, because that's very platform dependent (in particular on windows, I have yet to see a sane solution - everyone just copy the private .dll alongside the binaries, AFAICS). Now, if you bring me a solution to this problem, I would be *really* glad. > Actually, and I betray my ignorance here, but IIUC: > > - There are a bunch of different scipy extensions that use libgfortran > - Many of them are built more-or-less separately > - So each of them would get their own copy of the static libgfortran > AFAIK, statically linking a library does not mean the whole copy is put into the binary. I guess different binary formats do it differently, but for example, on linux: gfortran hello.f -> a.out is 8 kb gfortran hello.f -static-libgfortran -> a.out is 130 kb libgfortran.a -> ~ 1.3Mb Of course, this depends on the function you need to pull out from libgfortran - but I guess we do not pull so much, because we mainly use intrinsics (gfortran math functions, etc...) which should be very small. I don't think we use so much the IO fortran runtime - actually, we should explicitly avoid it since it cause trouble because the C and fortran runtimes would 'fight' each other with unspeakable consequences. And thinking about it: mac os x rather encourage big binaries - "fat binary", so I am not sure it is a big concern. > This is making me think solving the dynamic linking problem makes sense. > It makes sense for a whole lot of reasons, but it is hard. The problem is much bigger on windows (where almost *everything* is statically linked), and I tried to tackle this to ship only one numpy installer with 3 dynamically loaded atlas at runtime - I did not find a workable solution. > > Also, would it break anything if the libgfortran installed were properly > versioned: > > libgfortran.a.b.c > > Isn't that the point of versioned libs? > versioned libraries only make sense for shared libraries,I think. On Linux, the static library is not even publicly available (it is in /usr/lib/gcc/4.3.3). I wonder whether the mac os x gfortran binary did not make a mistake there, actually. cheers, David From novak at ucolick.org Wed Apr 1 11:26:45 2009 From: novak at ucolick.org (Greg Novak) Date: Wed, 1 Apr 2009 11:26:45 -0400 Subject: [Numpy-discussion] Out-of-RAM FFTs In-Reply-To: References: Message-ID: Hello, I'd like to do an FFT of a moderately large 3D cube, 1024^3. ?Looking at the run-time of smaller arrays, this is not a problem in terms of compute time, but the array doesn't fit in memory. ?So, several questions: 1) Numerical Recipes has an out-of-memory FFT algorithm, but looking through the numpy and scipy docs and modules, I didn't find a function that does the same thing. ?Did I miss it? ?Should I get to work typing it in? 2) I had high hopes for just memory-mapping the large array and passing it to the standard fft function. ?However, the memory-mapped region must fit into the address space, and I don't seem to be able to use more than 2 GB at a time. ?So memory mapping doesn't seem to help me at all. This last issue leads to another series of things that puzzle me. ?I have an iMac running OS X 10.5 with an Intel Core 2 duo processor and 4 GB of memory. ?As far as I've learned, the processor is 64 bit, the operating system is 64 bit, so I should be able to happily memory-map my entire disk if I want. ?However, Python seems to run out of steam when it's used 2 GB. ?This is true of both 2.5 and 2.6. ?What gives? Is this a Python issue? Thanks, Greg From dsdale24 at gmail.com Wed Apr 1 11:36:44 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Wed, 1 Apr 2009 11:36:44 -0400 Subject: [Numpy-discussion] bad line in setup.py Message-ID: In setup.py, svn_revision(), there is a line: log.warn("unrecognized .svn/entries format; skipping %s", base) log is not defined in setup.py. I'm using svn-1.6. Darren -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Wed Apr 1 11:26:24 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 02 Apr 2009 00:26:24 +0900 Subject: [Numpy-discussion] Out-of-RAM FFTs In-Reply-To: References: Message-ID: <49D387A0.60100@ar.media.kyoto-u.ac.jp> Greg Novak wrote: > 1) Numerical Recipes has an out-of-memory FFT algorithm, but looking > through the numpy and scipy docs and modules, I didn't find a function > that does the same thing. Did I miss it? I don't think so. > Should I get to work typing > it in? > Maybe :) > 2) I had high hopes for just memory-mapping the large array and > passing it to the standard fft function. However, the memory-mapped > region must fit into the address space, and I don't seem to be able to > use more than 2 GB at a time. So memory mapping doesn't seem to help > me at all. > > This last issue leads to another series of things that puzzle me. I > have an iMac running OS X 10.5 with an Intel Core 2 duo processor and > 4 GB of memory. As far as I've learned, the processor is 64 bit, the > operating system is 64 bit, so I should be able to happily memory-map > my entire disk if I want. However, Python seems to run out of steam > when it's used 2 GB. This is true of both 2.5 and 2.6. What gives? > Is this a Python issue? > Yes - official python binaries are 32 bits only. I don't know how advanced/usable is the 64 bits build, but I am afraid you will have to use an unofficial build or to build it by yourself. I don't know if the following can help you: http://developer.apple.com/documentation/Darwin/Conceptual/64bitPorting/intro/intro.html#//apple_ref/doc/uid/TP40001064-CH205-TPXREF101 cheers, David From dsdale24 at gmail.com Wed Apr 1 11:48:59 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Wed, 1 Apr 2009 11:48:59 -0400 Subject: [Numpy-discussion] trouble building docs with sphinx-0.6.1 Message-ID: This morning I upgraded to sphinx-0.6.1, hoping to take advantage of all the recent work that has been done to clean up and consolidate the web of sphinx extensions. I'm seeing segfaults when I try to build my own docs, or the h5py docs. I tried building the numpy documentation after applying the attached patch. After building html, updating the environment, reading sources, I see a slew of warnings and errors, followed by: preparing documents... done Exception occurred: 0%] contents File "/usr/lib64/python2.6/site-packages/docutils/nodes.py", line 471, in __getitem__ return self.attributes[key] KeyError: 'entries' The full traceback has been saved in /tmp/sphinx-err-RDe0NL.log, if you want to report the issue to the author. Please also report this if it was a user error, so that a better error message can be provided next time. Send reports to sphinx-dev at googlegroups.com. Thanks! Here are the contents of the referenced log file: Traceback (most recent call last): File "//usr/lib64/python2.6/site-packages/sphinx/cmdline.py", line 172, in main app.build(all_files, filenames) File "//usr/lib64/python2.6/site-packages/sphinx/application.py", line 129, in build self.builder.build_update() File "//usr/lib64/python2.6/site-packages/sphinx/builders/__init__.py", line 255, in build_update 'out of date' % len(to_build)) File "//usr/lib64/python2.6/site-packages/sphinx/builders/__init__.py", line 310, in build self.write(docnames, list(updated_docnames), method) File "//usr/lib64/python2.6/site-packages/sphinx/builders/__init__.py", line 348, in write doctree = self.env.get_and_resolve_doctree(docname, self) File "//usr/lib64/python2.6/site-packages/sphinx/environment.py", line 995, in get_and_resolve_doctree prune=prune_toctrees) File "//usr/lib64/python2.6/site-packages/sphinx/environment.py", line 1128, in resolve_toctree tocentries = _entries_from_toctree(toctree, separate=False) File "//usr/lib64/python2.6/site-packages/sphinx/environment.py", line 1109, in _entries_from_toctree subtree=True): File "//usr/lib64/python2.6/site-packages/sphinx/environment.py", line 1109, in _entries_from_toctree subtree=True): File "//usr/lib64/python2.6/site-packages/sphinx/environment.py", line 1109, in _entries_from_toctree subtree=True): File "//usr/lib64/python2.6/site-packages/sphinx/environment.py", line 1051, in _entries_from_toctree refs = [(e[0], str(e[1])) for e in toctreenode['entries']] File "/usr/lib64/python2.6/site-packages/docutils/nodes.py", line 471, in __getitem__ return self.attributes[key] KeyError: 'entries' Has anyone else tried building numpy's docs with sphinx-0.6.1? Is there any interest in sorting these issues out before 1.3 is released? Thanks, Darren -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: numpy_doc.patch Type: application/octet-stream Size: 1057 bytes Desc: not available URL: From matthew.brett at gmail.com Wed Apr 1 11:54:18 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 1 Apr 2009 08:54:18 -0700 Subject: [Numpy-discussion] Out-of-RAM FFTs In-Reply-To: References: Message-ID: <1e2af89e0904010854i34e6ca10h28e1522a0275e79a@mail.gmail.com> Hi, > 1) Numerical Recipes has an out-of-memory FFT algorithm, but looking > through the numpy and scipy docs and modules, I didn't find a function > that does the same thing. ?Did I miss it? ?Should I get to work typing > it in? No please don't do that; I'm afraid the Numerical Recipes book has a code license that is unusable for numpy / scipy, and it would be a very bad thing if any of their code ended up in the code base. Best, Matthew From nadavh at visionsense.com Wed Apr 1 12:10:39 2009 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed, 1 Apr 2009 19:10:39 +0300 Subject: [Numpy-discussion] trouble building docs with sphinx-0.6.1 References: Message-ID: <710F2847B0018641891D9A216027636029C4AD@ex3.envision.co.il> I tried, failed, and "downgraded" to 0.5.2 (and eventually succeeded). Nadav. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Darren Dale ????: ? 01-?????-09 17:48 ??: Discussion of Numerical Python ????: [Numpy-discussion] trouble building docs with sphinx-0.6.1 This morning I upgraded to sphinx-0.6.1, hoping to take advantage of all the recent work that has been done to clean up and consolidate the web of sphinx extensions. I'm seeing segfaults when I try to build my own docs, or the h5py docs. I tried building the numpy documentation after applying the attached patch. After building html, updating the environment, reading sources, I see a slew of warnings and errors, followed by: preparing documents... done Exception occurred: 0%] contents File "/usr/lib64/python2.6/site-packages/docutils/nodes.py", line 471, in __getitem__ return self.attributes[key] KeyError: 'entries' The full traceback has been saved in /tmp/sphinx-err-RDe0NL.log, if you want to report the issue to the author. Please also report this if it was a user error, so that a better error message can be provided next time. Send reports to sphinx-dev at googlegroups.com. Thanks! Here are the contents of the referenced log file: Traceback (most recent call last): File "//usr/lib64/python2.6/site-packages/sphinx/cmdline.py", line 172, in main app.build(all_files, filenames) File "//usr/lib64/python2.6/site-packages/sphinx/application.py", line 129, in build self.builder.build_update() File "//usr/lib64/python2.6/site-packages/sphinx/builders/__init__.py", line 255, in build_update 'out of date' % len(to_build)) File "//usr/lib64/python2.6/site-packages/sphinx/builders/__init__.py", line 310, in build self.write(docnames, list(updated_docnames), method) File "//usr/lib64/python2.6/site-packages/sphinx/builders/__init__.py", line 348, in write doctree = self.env.get_and_resolve_doctree(docname, self) File "//usr/lib64/python2.6/site-packages/sphinx/environment.py", line 995, in get_and_resolve_doctree prune=prune_toctrees) File "//usr/lib64/python2.6/site-packages/sphinx/environment.py", line 1128, in resolve_toctree tocentries = _entries_from_toctree(toctree, separate=False) File "//usr/lib64/python2.6/site-packages/sphinx/environment.py", line 1109, in _entries_from_toctree subtree=True): File "//usr/lib64/python2.6/site-packages/sphinx/environment.py", line 1109, in _entries_from_toctree subtree=True): File "//usr/lib64/python2.6/site-packages/sphinx/environment.py", line 1109, in _entries_from_toctree subtree=True): File "//usr/lib64/python2.6/site-packages/sphinx/environment.py", line 1051, in _entries_from_toctree refs = [(e[0], str(e[1])) for e in toctreenode['entries']] File "/usr/lib64/python2.6/site-packages/docutils/nodes.py", line 471, in __getitem__ return self.attributes[key] KeyError: 'entries' Has anyone else tried building numpy's docs with sphinx-0.6.1? Is there any interest in sorting these issues out before 1.3 is released? Thanks, Darren -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 4171 bytes Desc: not available URL: From david at ar.media.kyoto-u.ac.jp Wed Apr 1 11:57:28 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 02 Apr 2009 00:57:28 +0900 Subject: [Numpy-discussion] trouble building docs with sphinx-0.6.1 In-Reply-To: References: Message-ID: <49D38EE8.2070603@ar.media.kyoto-u.ac.jp> Darren Dale wrote: > This morning I upgraded to sphinx-0.6.1, hoping to take advantage of > all the recent work that has been done to clean up and consolidate the > web of sphinx extensions. I'm seeing segfaults when I try to build my > own docs, or the h5py docs. Segfaults ? In sphinx or numpy related ? > I tried building the numpy documentation after applying the attached > patch. After building html, updating the environment, reading sources, > I see a slew of warnings and errors, followed by: > > preparing documents... done > Exception occurred: 0%] contents > File "/usr/lib64/python2.6/site-packages/docutils/nodes.py", line > 471, in __getitem__ > return self.attributes[key] > KeyError: 'entries' > The full traceback has been saved in /tmp/sphinx-err-RDe0NL.log, if > you want to report the issue to the author. > Please also report this if it was a user error, so that a better error > message can be provided next time. > Send reports to sphinx-dev at googlegroups.com > . Thanks! This often happens for non-clean build. The only solution I got so far was to start the doc build from scratch... > Has anyone else tried building numpy's docs with sphinx-0.6.1? Is > there any interest in sorting these issues out before 1.3 is released? I am afraid it is too late for the 1.3 release, cheers, David From cournape at gmail.com Wed Apr 1 12:20:32 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 2 Apr 2009 01:20:32 +0900 Subject: [Numpy-discussion] bad line in setup.py In-Reply-To: References: Message-ID: <5b8d13220904010920q334d8f0v1301d9a824d1a996@mail.gmail.com> 2009/4/2 Darren Dale : > In setup.py, svn_revision(), there is a line: > > log.warn("unrecognized .svn/entries format; skipping %s", base) > > log is not defined in setup.py. I'm using svn-1.6. Damn - this should be fixed in r6830. Maybe I should have stayed with my solution instead of using setuptools version, cheers, David From dsdale24 at gmail.com Wed Apr 1 12:20:38 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Wed, 1 Apr 2009 12:20:38 -0400 Subject: [Numpy-discussion] trouble building docs with sphinx-0.6.1 In-Reply-To: <49D38EE8.2070603@ar.media.kyoto-u.ac.jp> References: <49D38EE8.2070603@ar.media.kyoto-u.ac.jp> Message-ID: On Wed, Apr 1, 2009 at 11:57 AM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Darren Dale wrote: > > This morning I upgraded to sphinx-0.6.1, hoping to take advantage of > > all the recent work that has been done to clean up and consolidate the > > web of sphinx extensions. I'm seeing segfaults when I try to build my > > own docs, or the h5py docs. > > Segfaults ? In sphinx or numpy related ? I just updated my numpy svn checkout, reinstalled, and now I can build the h5py documents without segfaulting. > I tried building the numpy documentation after applying the attached > > patch. After building html, updating the environment, reading sources, > > I see a slew of warnings and errors, followed by: > > > > preparing documents... done > > Exception occurred: 0%] contents > > File "/usr/lib64/python2.6/site-packages/docutils/nodes.py", line > > 471, in __getitem__ > > return self.attributes[key] > > KeyError: 'entries' > > The full traceback has been saved in /tmp/sphinx-err-RDe0NL.log, if > > you want to report the issue to the author. > > Please also report this if it was a user error, so that a better error > > message can be provided next time. > > Send reports to sphinx-dev at googlegroups.com > > . Thanks! > > This often happens for non-clean build. The only solution I got so far > was to start the doc build from scratch... > Do you mean to delete my doc/build directory and run "make html"? I've already tried that. Darren -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Wed Apr 1 12:07:02 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 02 Apr 2009 01:07:02 +0900 Subject: [Numpy-discussion] trouble building docs with sphinx-0.6.1 In-Reply-To: References: <49D38EE8.2070603@ar.media.kyoto-u.ac.jp> Message-ID: <49D39126.5080400@ar.media.kyoto-u.ac.jp> Darren Dale wrote: > > Do you mean to delete my doc/build directory and run "make html"? I've > already tried that. Yes, I meant that, and no, I don't have other suggestion :( David From pgmdevlist at gmail.com Wed Apr 1 12:26:11 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 1 Apr 2009 12:26:11 -0400 Subject: [Numpy-discussion] trouble building docs with sphinx-0.6.1 In-Reply-To: <49D38EE8.2070603@ar.media.kyoto-u.ac.jp> References: <49D38EE8.2070603@ar.media.kyoto-u.ac.jp> Message-ID: <50BFF37A-33D4-4F40-B1A7-6A39933F23B0@gmail.com> On Apr 1, 2009, at 11:57 AM, David Cournapeau wrote: >> >> preparing documents... done >> Exception occurred: 0%] contents >> File "/usr/lib64/python2.6/site-packages/docutils/nodes.py", line >> 471, in __getitem__ >> return self.attributes[key] >> KeyError: 'entries' >> The full traceback has been saved in /tmp/sphinx-err-RDe0NL.log, if >> you want to report the issue to the author. >> Please also report this if it was a user error, so that a better >> error >> message can be provided next time. >> Send reports to sphinx-dev at googlegroups.com >> . Thanks! > > This often happens for non-clean build. The only solution I got so far > was to start the doc build from scratch... David, won't work here, there's a bug indeed. Part of it comes from numpydoc, that isn't completely compatible w/ Sphin-0.6.1. In particular, the code doesn't know what to do w/ this 'entries' parameter. part of it comes from Sphinx. Georg said he made the 'entries' parameter optional, but that doesn't solve everything. Matt Knox actually came across the 'best' solution Edit Sphinx/environment.py, L1051: replace refs = [(e[0], str(e[1])) for e in toctreenode['entries'])] by refs = [(e[0], str(e[1])) for e in toctreenode.get('entries', [])] > >> Has anyone else tried building numpy's docs with sphinx-0.6.1? Is >> there any interest in sorting these issues out before 1.3 is >> released? > > I am afraid it is too late for the 1.3 release, > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From dsdale24 at gmail.com Wed Apr 1 12:44:36 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Wed, 1 Apr 2009 12:44:36 -0400 Subject: [Numpy-discussion] trouble building docs with sphinx-0.6.1 In-Reply-To: <50BFF37A-33D4-4F40-B1A7-6A39933F23B0@gmail.com> References: <49D38EE8.2070603@ar.media.kyoto-u.ac.jp> <50BFF37A-33D4-4F40-B1A7-6A39933F23B0@gmail.com> Message-ID: On Wed, Apr 1, 2009 at 12:26 PM, Pierre GM wrote: > > On Apr 1, 2009, at 11:57 AM, David Cournapeau wrote: > >> > >> preparing documents... done > >> Exception occurred: 0%] contents > >> File "/usr/lib64/python2.6/site-packages/docutils/nodes.py", line > >> 471, in __getitem__ > >> return self.attributes[key] > >> KeyError: 'entries' > >> The full traceback has been saved in /tmp/sphinx-err-RDe0NL.log, if > >> you want to report the issue to the author. > >> Please also report this if it was a user error, so that a better > >> error > >> message can be provided next time. > >> Send reports to sphinx-dev at googlegroups.com > >> . Thanks! > > > > This often happens for non-clean build. The only solution I got so far > > was to start the doc build from scratch... > > David, won't work here, there's a bug indeed. > Part of it comes from numpydoc, that isn't completely compatible w/ > Sphin-0.6.1. In particular, the code doesn't know what to do w/ this > 'entries' parameter. > part of it comes from Sphinx. Georg said he made the 'entries' > parameter optional, but that doesn't solve everything. Matt Knox > actually came across the 'best' solution > Edit Sphinx/environment.py, L1051: > replace > refs = [(e[0], str(e[1])) for e in toctreenode['entries'])] > by > refs = [(e[0], str(e[1])) for e in toctreenode.get('entries', [])] > Thanks Pierre, I think that did it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sienkiew at stsci.edu Wed Apr 1 12:56:05 2009 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Wed, 01 Apr 2009 12:56:05 -0400 Subject: [Numpy-discussion] 1.3.0 rc1 MATHLIB env variable / bad compiler flags Message-ID: <49D39CA5.5020008@stsci.edu> I have this configuration: numpy 1.3.0 rc1 Solaris 10 Python 2.5.4 compiled as a 64 bit executable When I try to install numpy, it says: C compiler: cc -DNDEBUG -O -xarch=native64 -xcode=pic32 compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/stsci/Python-2.5.4/include/python2.5 -c' cc: _configtest.c cc _configtest.o -lm -o _configtest ld: fatal: file _configtest.o: wrong ELF class: ELFCLASS64 ld: fatal: File processing errors. No output written to _configtest ld: fatal: file _configtest.o: wrong ELF class: ELFCLASS64 ld: fatal: File processing errors. No output written to _configtest failure. ... mathlibs = check_mathlib(config_cmd) File "numpy/core/setup.py", line 253, in check_mathlib raise EnvironmentError("math library missing; rerun " EnvironmentError: math library missing; rerun setup.py after setting the MATHLIB env variable Of course, the problem is that it is using the wrong compiler flags during the link phase, so nothing I set MATHLIB to can possibly work. I found that I can get it to compile by creating these shell scripts: % cat cc #!/bin/sh /opt/SUNWspro-6u2/bin/cc -xarch=native64 -xcode=pic32 $* % cat f90 #!/bin/sh /opt/SUNWspro-6u2/bin/f90 -xarch=native64 -xcode=pic32 $* I think this looks like a bug. I thought I might try to make a patch (since this is all about installing, so, in principle, you don't need to know much about numpy), but I did not get very far in figuring out how the install works. The good news is that once you get it to build, it seems to work. (IIRC, rc1 fails the same test that it had problems with in my other email. The fix on the trunk also works in 64 bit solaris.) Mark S. From tgrav at mac.com Wed Apr 1 11:59:00 2009 From: tgrav at mac.com (Tommy Grav) Date: Wed, 01 Apr 2009 11:59:00 -0400 Subject: [Numpy-discussion] [Announce] Numpy 1.3.0 rc1 In-Reply-To: <5b8d13220903292356r5e954b90lf92c9f170ec01b13@mail.gmail.com> References: <49CE2587.2050007@ar.media.kyoto-u.ac.jp> <5b8d13220903280904j6e4fc9f7of642ab2edc4ffde2@mail.gmail.com> <9EC23A11-4979-4DF7-AE6B-7C3BA61E6494@post.harvard.edu> <49CF2B55.3090601@ar.media.kyoto-u.ac.jp> <5b8d13220903290753g224ee96am72c1cd93db88f650@mail.gmail.com> <5b8d13220903292356r5e954b90lf92c9f170ec01b13@mail.gmail.com> Message-ID: On Mar 30, 2009, at 2:56 AM, David Cournapeau wrote: > On Mon, Mar 30, 2009 at 3:36 AM, Robert Pyle > wrote: > >> >> I just installed 2.5.4 from python.org, and the OS X installer still >> doesn't work. This is on a PPC G5; I haven't tried it on my Intel >> MacBook Pro. > > I think I got it. To build numpy, I use virtualenv to make a > "bootstrap" environment, but then the corresponding python path get > embedded in the .mpkg - so unless you have your python interpreter in > exactly the same path as my bootstrap (which is very unlikely), it > won't run at all. This would also explain why I never saw the problem. > This is exactly the problem. This is the error message that you get when running the .dmg and no hard drives are available for selection. You cannot install numpy 1.3.0rc1 on this volume. numpy requires /Users/david/src/dsp/numpy/1.3.x/bootstrap Python 2.5 to install. > I will prepare a new binary, Any idea when a new binary will be available on sourceforge.net? Cheers Tommy From charlesr.harris at gmail.com Wed Apr 1 13:06:01 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 1 Apr 2009 11:06:01 -0600 Subject: [Numpy-discussion] Out-of-RAM FFTs In-Reply-To: <49D387A0.60100@ar.media.kyoto-u.ac.jp> References: <49D387A0.60100@ar.media.kyoto-u.ac.jp> Message-ID: On Wed, Apr 1, 2009 at 9:26 AM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Greg Novak wrote: > > 1) Numerical Recipes has an out-of-memory FFT algorithm, but looking > > through the numpy and scipy docs and modules, I didn't find a function > > that does the same thing. Did I miss it? > > I don't think so. > > > Should I get to work typing > > it in? > > > > Maybe :) > > > 2) I had high hopes for just memory-mapping the large array and > > passing it to the standard fft function. However, the memory-mapped > > region must fit into the address space, and I don't seem to be able to > > use more than 2 GB at a time. So memory mapping doesn't seem to help > > me at all. > > > > This last issue leads to another series of things that puzzle me. I > > have an iMac running OS X 10.5 with an Intel Core 2 duo processor and > > 4 GB of memory. As far as I've learned, the processor is 64 bit, the > > operating system is 64 bit, so I should be able to happily memory-map > > my entire disk if I want. However, Python seems to run out of steam > > when it's used 2 GB. This is true of both 2.5 and 2.6. What gives? > > Is this a Python issue? > > > > Yes - official python binaries are 32 bits only. I don't know how > advanced/usable is the 64 bits build, but I am afraid you will have to > use an unofficial build or to build it by yourself. > > I don't know if the following can help you: > > > http://developer.apple.com/documentation/Darwin/Conceptual/64bitPorting/intro/intro.html#//apple_ref/doc/uid/TP40001064-CH205-TPXREF101 > There was a thread about this back when... Hereit is, note Michael Abshoff's directions on building 64 bit python on the mac. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Wed Apr 1 13:19:52 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 1 Apr 2009 19:19:52 +0200 Subject: [Numpy-discussion] Out-of-RAM FFTs In-Reply-To: References: Message-ID: Hi, In any case, the OS will have to swap a lot of your data : - if you use floats (32bits), you use 4GB for your input array - this does not fit inside your memory - it even fit less if you count on the fact that FFT needs a least one array as large, so at least 8 GB. So you should, in every case, split your data and load it on the fly, by hand (after each FFT, you could swap the array axis, but it may not be the best, the best being having something like 16GB RAM for floats). Matthieu 2009/4/1 Greg Novak : > Hello, > I'd like to do an FFT of a moderately large 3D cube, 1024^3. ?Looking > at the run-time of smaller arrays, this is not a problem in terms of > compute time, but the array doesn't fit in memory. ?So, several > questions: > > 1) Numerical Recipes has an out-of-memory FFT algorithm, but looking > through the numpy and scipy docs and modules, I didn't find a function > that does the same thing. ?Did I miss it? ?Should I get to work typing > it in? > 2) I had high hopes for just memory-mapping the large array and > passing it to the standard fft function. ?However, the memory-mapped > region must fit into the address space, and I don't seem to be able to > use more than 2 GB at a time. ?So memory mapping doesn't seem to help > me at all. > > This last issue leads to another series of things that puzzle me. ?I > have an iMac running OS X 10.5 with an Intel Core 2 duo processor and > 4 GB of memory. ?As far as I've learned, the processor is 64 bit, the > operating system is 64 bit, so I should be able to happily memory-map > my entire disk if I want. ?However, Python seems to run out of steam > when it's used 2 GB. ?This is true of both 2.5 and 2.6. ?What gives? > Is this a Python issue? > > Thanks, > Greg > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From Chris.Barker at noaa.gov Wed Apr 1 14:33:55 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 01 Apr 2009 11:33:55 -0700 Subject: [Numpy-discussion] Out-of-RAM FFTs In-Reply-To: References: Message-ID: <49D3B393.2080309@noaa.gov> Greg Novak wrote: > This last issue leads to another series of things that puzzle me. I > have an iMac running OS X 10.5 with an Intel Core 2 duo processor and > 4 GB of memory. As far as I've learned, the processor is 64 bit, the > operating system is 64 bit, so I should be able to happily memory-map > my entire disk if I want. However, Python seems to run out of steam > when it's used 2 GB. This is true of both 2.5 and 2.6. What gives? > Is this a Python issue? Not python per-se, but yes, the standard builds of Python on OS-X are all 32 bit. I'm pretty sure that python2.6 builds successfully as 64 on OS-X -- check the archives and/or send a note to the pythonmac list: http://mail.python.org/mailman/listinfo/pythonmac-sig to get more info. You will need to build python and all the extension packages you need to make it work, though. macports is worth checking out -- I think it can do 64 bit builds. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From pav at iki.fi Wed Apr 1 16:40:15 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 1 Apr 2009 20:40:15 +0000 (UTC) Subject: [Numpy-discussion] trouble building docs with sphinx-0.6.1 References: Message-ID: Wed, 01 Apr 2009 11:48:59 -0400, Darren Dale wrote: > This morning I upgraded to sphinx-0.6.1, hoping to take advantage of all > the recent work that has been done to clean up and consolidate the web > of sphinx extensions. I'm seeing segfaults when I try to build my own > docs, or the h5py docs. I tried building the numpy documentation after > applying the attached patch. After building html, updating the > environment, reading sources, I see a slew of warnings and errors, > followed by: [clip] It was an incompatibility of Numpy's autosummary extension and Sphinx >= 0.6. It should now be fixed in Numpy trunk. -- Pauli Virtanen From dsdale24 at gmail.com Wed Apr 1 17:20:20 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Wed, 1 Apr 2009 17:20:20 -0400 Subject: [Numpy-discussion] trouble building docs with sphinx-0.6.1 In-Reply-To: References: Message-ID: On Wed, Apr 1, 2009 at 4:40 PM, Pauli Virtanen wrote: > Wed, 01 Apr 2009 11:48:59 -0400, Darren Dale wrote: > > This morning I upgraded to sphinx-0.6.1, hoping to take advantage of all > > the recent work that has been done to clean up and consolidate the web > > of sphinx extensions. I'm seeing segfaults when I try to build my own > > docs, or the h5py docs. I tried building the numpy documentation after > > applying the attached patch. After building html, updating the > > environment, reading sources, I see a slew of warnings and errors, > > followed by: > [clip] > > It was an incompatibility of Numpy's autosummary extension and > Sphinx >= 0.6. It should now be fixed in Numpy trunk. > Thanks Pauli. I noticed some bad indentation in testing.decorators, maybe this patch could be considered: Index: numpy/testing/decorators.py =================================================================== --- numpy/testing/decorators.py (revision 6831) +++ numpy/testing/decorators.py (working copy) @@ -1,4 +1,5 @@ -"""Decorators for labeling test objects +""" +Decorators for labeling test objects Decorators that merely return a modified version of the original function object are straightforward. Decorators that return a new @@ -11,7 +12,8 @@ """ def slow(t): - """Labels a test as 'slow'. + """ + Labels a test as 'slow'. The exact definition of a slow test is obviously both subjective and hardware-dependent, but in general any individual test that requires more @@ -22,7 +24,8 @@ return t def setastest(tf=True): - ''' Signals to nose that this function is or is not a test + ''' + Signals to nose that this function is or is not a test Parameters ---------- @@ -47,7 +50,8 @@ return set_test def skipif(skip_condition, msg=None): - ''' Make function raise SkipTest exception if skip_condition is true + ''' + Make function raise SkipTest exception if skip_condition is true Parameters ---------- @@ -59,12 +63,12 @@ msg : string Message to give on raising a SkipTest exception - Returns - ------- - decorator : function - Decorator, which, when applied to a function, causes SkipTest - to be raised when the skip_condition was True, and the function - to be called normally otherwise. + Returns + ------- + decorator : function + Decorator, which, when applied to a function, causes SkipTest + to be raised when the skip_condition was True, and the function + to be called normally otherwise. Notes ----- @@ -86,9 +90,9 @@ def get_msg(func,msg=None): """Skip message with information about function being skipped.""" - if msg is None: + if msg is None: out = 'Test skipped due to test condition' - else: + else: out = '\n'+msg return "Skipping test: %s%s" % (func.__name__,out) @@ -115,32 +119,33 @@ skipper = skipper_gen else: skipper = skipper_func - + return nose.tools.make_decorator(f)(skipper) return skip_decorator def knownfailureif(fail_condition, msg=None): - ''' Make function raise KnownFailureTest exception if fail_condition is true + ''' + Make function raise KnownFailureTest exception if fail_condition is true Parameters ---------- fail_condition : bool or callable. Flag to determine whether to mark test as known failure (True) or not (False). If the condition is a callable, it is used at - runtime to dynamically make the decision. This is useful for + runtime to dynamically make the decision. This is useful for tests that may require costly imports, to delay the cost until the test suite is actually executed. msg : string Message to give on raising a KnownFailureTest exception - Returns - ------- - decorator : function - Decorator, which, when applied to a function, causes SkipTest - to be raised when the skip_condition was True, and the function - to be called normally otherwise. + Returns + ------- + decorator : function + Decorator, which, when applied to a function, causes SkipTest + to be raised when the skip_condition was True, and the function + to be called normally otherwise. Notes ----- -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Wed Apr 1 17:58:55 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 01 Apr 2009 14:58:55 -0700 Subject: [Numpy-discussion] Numpy 1.3.0 rc1 OS X Installer In-Reply-To: <49D2F731.70006@ar.media.kyoto-u.ac.jp> References: <49CE2587.2050007@ar.media.kyoto-u.ac.jp> <5b8d13220903290753g224ee96am72c1cd93db88f650@mail.gmail.com> <5b8d13220903292356r5e954b90lf92c9f170ec01b13@mail.gmail.com> <49D0778C.5050706@ar.media.kyoto-u.ac.jp> <2BACA6B8-43A0-421E-859B-E6593B588FFF@post.harvard.edu> <5b8d13220903300822m682c7e26jb10975ecf1d9c723@mail.gmail.com> <49D0EA6D.6050601@noaa.gov> <5b8d13220903300919s13702969vd54c70c3a928fc0c@mail.gmail.com> <49D0FCF8.5040908@noaa.gov> <5b8d13220903301027n76634921hf4dc4b1818392ab3@mail.gmail.com> <49D11686.4010305@noaa.gov> <49D1A5A1.2060205@ar.media.kyoto-u.ac.jp> <49D28D7D.9050102@noaa.gov> <49D2F731.70006@ar.media.kyoto-u.ac.jp> Message-ID: <49D3E39F.8020205@noaa.gov> David Cournapeau wrote: > Christopher Barker wrote: >> It does, but we don't need a binary installer for a python that doesn't >> have a binary installer. > > Yes, not now - but I would prefer avoiding to have to change the process > again when time comes. It may not look like it, but enabling a working > process which works well on all platforms including windows took me > several days to work properly. I'm not surprised it took that long -- that sounds short to me! Anyway, If there are new python builds that are 64-bit (quad?) you wont' have to change much -- "only" make sure that the libs you are linking to are 64 bit. I suppose you could try to get a quad-universal gfortran.a now, but I'd wait 'till you need it. >> However, I'm not sure you need to do what your saying here. I imagine >> this workflow: >> >> set up a virtualenv for, say numpy x.y.rc-z >> >> play around with it, get everything to build, etc. with plain old >> setup.py build, setup.py install, etc. >> >> Once you are happy, run: >> >> /Library/Frameworks/Python.framework/Versions/2.5/bin/bdist_mpkg >> > > This means building the same thing twice. does it? I just did test: I set up a virtualenv with nothing in it. I used that environment to do: setup.py build setup.py install and got an apparently working numpy in my virtualenv. Then I ran: /Library/Frameworks/Python.framework/Versions/2.5/bin/bdist_mpkg It did a lot, but I don't THINK is re-compiled everthing. I think the trick is that eveything it built was still in the build dir -- and it is the saem python, even though it's not living in the same place. I got what seems to be a functioning Universal Installer for the python.org python. Having said that, it looks like it may be very easy to hack a package built in the virtualenv to install in the right place. In: dist/numpy-1.3.0rc1-py2.5-macosx10.4.mpkg/Contents/Packages/ there are two mpkgs. IN each of those, there is: Contents/Info.plist which is an xml file. In there, there is: IFPkgFlagDefaultLocation which should be set to: /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages If it was built in a virtualenv, that would be the virtualenv path. It's a hack, but you could post-process the mpkg to change that value after building. > The docs are included in the .dmg, and yes, the doc needs to be built > from the same installation (or more exactly the same source). In my tests, that seems to work fine. >> True. In that case we could put the dylib somewhere obscure: >> >> /usr/local/lib/scipy1.6/lib/ >> > Hm, that's strange - why /usr/local/lib ? It is outside the scipy > installation. I'm not sure, really, except that that's where Robin Dunn puts stuff for wxPython -- I think one reason may be that he can then point to it from more than one Python installation -- for instance Apple's and python.orgs. And it could help with building from a virtualenv. >> or even: >> >> /Library/Frameworks/Python.framework/Versions/2.5/lib/ >> > That's potentially dangerous: since this directory is likely to be in > LIBDIR, it means libgfortran will be taken there or from /usr/local/lib > if the user builds numpy/scipy after installing numpy. If it is > incompatible with the user gfortran, it will lead to weird issues, hard > to debug. showing what a pain all of this is! Of course, you could put it in: /Library/Frameworks/Python.framework/Versions/2.5/lib/site_packages/scipy/lib/ or something that only scipy would know about. > If we install something like libgfortran, it should be installed > privately - but dynamically linking against private libraries is hard, > because that's very platform dependent yup -- on the Mac, it could work well, 'cause the paths to the libs are hard-coded when linked, so you WILL get the right one -- if it's there! Tough this could lead to trouble when you want to build from a virtualenv, and install to the system location. mocaholib gives you tools to re-write the locations, but someone would have to write that code. > gfortran hello.f -> a.out is 8 kb > gfortran hello.f -static-libgfortran -> a.out is 130 kb > libgfortran.a -> ~ 1.3Mb > And thinking about it: mac os x rather encourage big binaries - "fat > binary", so I am not sure it is a big concern. so I guess we're back to static linking -- I guess that's why it's generally "recommended" for distributing binaries. >> Also, would it break anything if the libgfortran installed were properly >> versioned: >> libgfortran.a.b.c > > versioned libraries only make sense for shared libraries,I think. right -- I meant for the dynamic lib option. > On > Linux, the static library is not even publicly available (it is in > /usr/lib/gcc/4.3.3). I wonder whether the mac os x gfortran binary did > not make a mistake there, actually. where are you getting that? I'm about to go on vacation, but maybe I could try a few things... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Wed Apr 1 18:13:39 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 01 Apr 2009 15:13:39 -0700 Subject: [Numpy-discussion] Numpy 1.3.0 rc1 OS X Installer In-Reply-To: <49D3E39F.8020205@noaa.gov> References: <49CE2587.2050007@ar.media.kyoto-u.ac.jp> <5b8d13220903290753g224ee96am72c1cd93db88f650@mail.gmail.com> <5b8d13220903292356r5e954b90lf92c9f170ec01b13@mail.gmail.com> <49D0778C.5050706@ar.media.kyoto-u.ac.jp> <2BACA6B8-43A0-421E-859B-E6593B588FFF@post.harvard.edu> <5b8d13220903300822m682c7e26jb10975ecf1d9c723@mail.gmail.com> <49D0EA6D.6050601@noaa.gov> <5b8d13220903300919s13702969vd54c70c3a928fc0c@mail.gmail.com> <49D0FCF8.5040908@noaa.gov> <5b8d13220903301027n76634921hf4dc4b1818392ab3@mail.gmail.com> <49D11686.4010305@noaa.gov> <49D1A5A1.2060205@ar.media.kyoto-u.ac.jp> <49D28D7D.9050102@noaa.gov> <49D2F731.70006@ar.media.kyoto-u.ac.jp> <49D3E39F.8020205@noaa.gov> Message-ID: <49D3E713.2020305@noaa.gov> Christopher Barker wrote: > Anyway, If there are new python builds that are 64-bit (quad?) you wont' > have to change much -- "only" make sure that the libs you are linking to > are 64 bit. I suppose you could try to get a quad-universal gfortran.a > now, Actually, it looks like the binary at: http://r.research.att.com/tools/ Is already a quad-universal build. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From stefan at sun.ac.za Wed Apr 1 19:19:53 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 2 Apr 2009 01:19:53 +0200 Subject: [Numpy-discussion] GSoC Proposals Due on the 3rd Message-ID: <9457e7c80904011619y68c31426x7a178c268acdb1a3@mail.gmail.com> Hi all Students interested in working on NumPy or SciPy for GSoC2009, please note that the deadline for proposals is the 3rd of April. http://socghop.appspot.com/document/show/program/google/gsoc2009/userguide#depth_studentapply Let's keep those applications coming! Cheers St?fan From david at ar.media.kyoto-u.ac.jp Thu Apr 2 01:18:43 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 02 Apr 2009 14:18:43 +0900 Subject: [Numpy-discussion] Numpy 1.3.0 rc1 OS X Installer In-Reply-To: <49D3E39F.8020205@noaa.gov> References: <49CE2587.2050007@ar.media.kyoto-u.ac.jp> <5b8d13220903290753g224ee96am72c1cd93db88f650@mail.gmail.com> <5b8d13220903292356r5e954b90lf92c9f170ec01b13@mail.gmail.com> <49D0778C.5050706@ar.media.kyoto-u.ac.jp> <2BACA6B8-43A0-421E-859B-E6593B588FFF@post.harvard.edu> <5b8d13220903300822m682c7e26jb10975ecf1d9c723@mail.gmail.com> <49D0EA6D.6050601@noaa.gov> <5b8d13220903300919s13702969vd54c70c3a928fc0c@mail.gmail.com> <49D0FCF8.5040908@noaa.gov> <5b8d13220903301027n76634921hf4dc4b1818392ab3@mail.gmail.com> <49D11686.4010305@noaa.gov> <49D1A5A1.2060205@ar.media.kyoto-u.ac.jp> <49D28D7D.9050102@noaa.gov> <49D2F731.70006@ar.media.kyoto-u.ac.jp> <49D3E39F.8020205@noaa.gov> Message-ID: <49D44AB3.7060901@ar.media.kyoto-u.ac.jp> Christopher Barker wrote: > > I'm not surprised it took that long -- that sounds short to me! > > Anyway, If there are new python builds that are 64-bit (quad?) you wont' > have to change much -- "only" make sure that the libs you are linking to > are 64 bit. I suppose you could try to get a quad-universal gfortran.a > now, but I'd wait 'till you need it. > My main concern was about making sure I always built against the same python interpreter. For now, I have hardcoded the python executable to use for the mpkg, but that's not good enough. > It did a lot, but I don't THINK is re-compiled everthing. I think the > trick is that eveything it built was still in the build dir -- and it is > the saem python, even though it's not living in the same place. > Well, yes, it could be a totally different, incompatible python that it would still do as you said - distutils cannot be trusted at all to do the right thing here, it has no real dependency system. The problem is to make sure that bdist_mpkg and the installed numpy in virtual env are built against the same python binary: - it makes sure the build is actually compatible - it also gives a sanity check about 'runnability' of the binary - at least, numpy can be imported (for the doc), and by building from scratch, I lose this. Of course, the real solution would be to automatically test for the built numpy - I unfortunately did not have the time to do this correctly, > IFPkgFlagDefaultLocation > > which should be set to: > > /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages > > If it was built in a virtualenv, that would be the virtualenv path. > > It's a hack, but you could post-process the mpkg to change that value > after building. > This has the same problem - it works as long as the virtual python and the 'python targetted by the binary installer are the same'. > > showing what a pain all of this is! Of course, you could put it in: > > /Library/Frameworks/Python.framework/Versions/2.5/lib/site_packages/scipy/lib/ > > or something that only scipy would know about. > That's the only correct solution I can see, yes. >> On >> Linux, the static library is not even publicly available (it is in >> /usr/lib/gcc/4.3.3). I wonder whether the mac os x gfortran binary did >> not make a mistake there, actually. >> > > where are you getting that? I'm about to go on vacation, but maybe I > could try a few things... > On linux, libgfortran.a is /usr/lib/gcc/i486-linux-gnu/4.3/libgfortran.a -> this is private On mac OS X, it is /usr/local/lib -> this is public Exactly the kind of 'small' things which end up with quite some headache when you care about reliability and repeatability. As a related problem, I am not a big fan about the Apple way of building fat binaries, I much prefer the approach one build/arch and merging the binaries after build with lipo. This is more compatible with all autoconf projects out there - and it means using/updating the compilers is easy (building multi-arch binaries the 'apple' way is really a pain - the scripts used by R, to build the universal gfortran, is quite complicated). I think this above problem would not have happened with a pristine gfortran built from sources, cheers, David From faltet at pytables.org Thu Apr 2 02:56:23 2009 From: faltet at pytables.org (Francesc Alted) Date: Thu, 2 Apr 2009 08:56:23 +0200 Subject: [Numpy-discussion] Out-of-RAM FFTs In-Reply-To: References: Message-ID: <200904020856.23811.faltet@pytables.org> A Wednesday 01 April 2009, Greg Novak escrigu?: > Hello, > I'd like to do an FFT of a moderately large 3D cube, 1024^3. ?Looking > at the run-time of smaller arrays, this is not a problem in terms of > compute time, but the array doesn't fit in memory. ?So, several > questions: > > 1) Numerical Recipes has an out-of-memory FFT algorithm, but looking > through the numpy and scipy docs and modules, I didn't find a > function that does the same thing. ?Did I miss it? ?Should I get to > work typing it in? > 2) I had high hopes for just memory-mapping the large array and > passing it to the standard fft function. ?However, the memory-mapped > region must fit into the address space, and I don't seem to be able > to use more than 2 GB at a time. ?So memory mapping doesn't seem to > help me at all. > > This last issue leads to another series of things that puzzle me. ?I > have an iMac running OS X 10.5 with an Intel Core 2 duo processor and > 4 GB of memory. ?As far as I've learned, the processor is 64 bit, the > operating system is 64 bit, so I should be able to happily memory-map > my entire disk if I want. ?However, Python seems to run out of steam > when it's used 2 GB. ?This is true of both 2.5 and 2.6. ?What gives? > Is this a Python issue? If your Python interpreter is 32-bit, one possibility is to use PyTables as a replacement of memory-mapped arrays that can address 64-bit arrays even on 32-bit platforms (I think h5py offers this possibility too). Of course, the access to data in the arrays is going to be slower than in-memory arrays, but no less than using memmap (and sometimes faster, specially for highly redundant datasets that can be compressed well). HTH, -- Francesc Alted "One would expect people to feel threatened by the 'giant brains or machines that think'. In fact, the fightening computer becomes less frightening if it is used only to simulate a familiar noncomputer." -- Edsger W. Dykstra "On the cruelty of really teaching computer science" From Chris.Barker at noaa.gov Thu Apr 2 17:15:43 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 02 Apr 2009 14:15:43 -0700 Subject: [Numpy-discussion] Numpy 1.3.0 rc1 OS X Installer In-Reply-To: <49D44AB3.7060901@ar.media.kyoto-u.ac.jp> References: <49CE2587.2050007@ar.media.kyoto-u.ac.jp> <5b8d13220903290753g224ee96am72c1cd93db88f650@mail.gmail.com> <5b8d13220903292356r5e954b90lf92c9f170ec01b13@mail.gmail.com> <49D0778C.5050706@ar.media.kyoto-u.ac.jp> <2BACA6B8-43A0-421E-859B-E6593B588FFF@post.harvard.edu> <5b8d13220903300822m682c7e26jb10975ecf1d9c723@mail.gmail.com> <49D0EA6D.6050601@noaa.gov> <5b8d13220903300919s13702969vd54c70c3a928fc0c@mail.gmail.com> <49D0FCF8.5040908@noaa.gov> <5b8d13220903301027n76634921hf4dc4b1818392ab3@mail.gmail.com> <49D11686.4010305@noaa.gov> <49D1A5A1.2060205@ar.media.kyoto-u.ac.jp> <49D28D7D.9050102@noaa.gov> <49D2F731.70006@ar.media.kyoto-u.ac.jp> <49D3E39F.8020205@noaa.gov> <49D44AB3.7060901@ar.media.kyoto-u.ac.jp> Message-ID: <49D52AFF.1060407@noaa.gov> David Cournapeau wrote: > Well, yes, it could be a totally different, incompatible python that it > would still do as you said - distutils cannot be trusted at all to do > the right thing here, quite true -- though hopefully you now what is on your system, so it can only go so wrong. However I just noticed that in a brand new virtualenv, there is ".Python" file, which is a symlink to: /Library/Frameworks/Python.framework/Versions/2.5/Python So you may be able to use that to ensure that the python you build the .mpkg with is the same as the one in the virtualenv. > - it also gives a sanity check about 'runnability' of the binary - > at least, numpy can be imported (for the doc) but it may not help with that. I think some of this could be improved by hacking bdist_mpkg, but someone would have to find the time to do that. It's still going to be a pain until core python supports some kind of versioning or virtualenv system, and all the tools then support that. >> IFPkgFlagDefaultLocation >> >> which should be set to: >> >> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages >> >> If it was built in a virtualenv, that would be the virtualenv path. > This has the same problem - it works as long as the virtual python and > the 'python targetted by the binary installer are the same'. so maybe the above could help. >> /Library/Frameworks/Python.framework/Versions/2.5/lib/site_packages/scipy/lib/ >> or something that only scipy would know about. >> > That's the only correct solution I can see, yes. So it's that or static libs -- I don't understand what the static lib problem is, so I'm not help there. >> where are you getting that? I'm about to go on vacation, but maybe I >> could try a few things... > > On linux, libgfortran.a is /usr/lib/gcc/i486-linux-gnu/4.3/libgfortran.a > -> this is private > On mac OS X, it is /usr/local/lib -> this is public I meant where did you get/how did you build gfortran in the first place. Are you using this one? http://r.research.att.com/tools/ They seem to be quad-architecture and have the static libs -- I haven't tried to build anything with it yet, though. > As a related problem, I am not a big fan about the Apple way of building > fat binaries, I much prefer the approach one build/arch and merging the > binaries after build with lipo. I don't know -- the biggest issues I've seen are with build systems that assume you are building for the architecture you are on, which would break either way, unless you build each piece on a different machine. I just discovered that mac port now support building Universal libs for at least some packages -- when it does, it seems to work great. NO gfortran, though.... I hope some of this helps. I really appreciate all you've done on this, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Thu Apr 2 18:46:52 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 2 Apr 2009 16:46:52 -0600 Subject: [Numpy-discussion] Bug in dstack? Message-ID: Note: In [133]: l = [[1,0,0],[1,1,0],[1,1,1]] In [134]: dstack(l) Out[134]: array([[[1, 1, 1], [0, 1, 1], [0, 0, 1]]]) In [135]: dstack(l).shape Out[135]: (1, 3, 3) Shouldn't the shape be (3,3)? Also, for generalized ufuncs and broadcasting I think a function that stacked along the first axis instead of the last would be useful. Maybe gstack or astack? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Apr 2 19:50:26 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 2 Apr 2009 19:50:26 -0400 Subject: [Numpy-discussion] Bug in dstack? In-Reply-To: References: Message-ID: <1cd32cbb0904021650m7cb91775m597a5d424ff04831@mail.gmail.com> On Thu, Apr 2, 2009 at 6:46 PM, Charles R Harris wrote: > Note: > > In [133]: l = [[1,0,0],[1,1,0],[1,1,1]] > > In [134]: dstack(l) > Out[134]: > array([[[1, 1, 1], > ??????? [0, 1, 1], > ??????? [0, 0, 1]]]) > > In [135]: dstack(l).shape > Out[135]: (1, 3, 3) > > > Shouldn't the shape be (3,3)? Also, for generalized ufuncs and broadcasting > I think a function that stacked along the first axis instead of the last > would be useful. Maybe gstack or astack? > > Chuck > I think dstack, vstack and hstack work exactly as advertised in the docs, at least that's how I interpret and use them Josef >>> l = [[1,0,0],[1,1,0],[1,1,1]] >>> np.dstack(l).shape (1, 3, 3) >>> np.vstack(l).shape (3, 3) >>> np.hstack(l).shape (9,) >>> np.vstack(l) array([[1, 0, 0], [1, 1, 0], [1, 1, 1]]) >>> np.dstack(l)[:,:,0] array([[1, 0, 0]]) >>> np.dstack(l)[:,:,1] array([[1, 1, 0]]) >>> np.dstack(l)[:,:,2] array([[1, 1, 1]]) From charlesr.harris at gmail.com Thu Apr 2 20:18:40 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 2 Apr 2009 18:18:40 -0600 Subject: [Numpy-discussion] Bug in dstack? In-Reply-To: <1cd32cbb0904021650m7cb91775m597a5d424ff04831@mail.gmail.com> References: <1cd32cbb0904021650m7cb91775m597a5d424ff04831@mail.gmail.com> Message-ID: On Thu, Apr 2, 2009 at 5:50 PM, wrote: > On Thu, Apr 2, 2009 at 6:46 PM, Charles R Harris > wrote: > > Note: > > > > In [133]: l = [[1,0,0],[1,1,0],[1,1,1]] > > > > In [134]: dstack(l) > > Out[134]: > > array([[[1, 1, 1], > > [0, 1, 1], > > [0, 0, 1]]]) > > > > In [135]: dstack(l).shape > > Out[135]: (1, 3, 3) > > > > > > Shouldn't the shape be (3,3)? Also, for generalized ufuncs and > broadcasting > > I think a function that stacked along the first axis instead of the last > > would be useful. Maybe gstack or astack? > > > > Chuck > > > > I think dstack, vstack and hstack work exactly as advertised in the > docs, at least that's how I interpret and use them > > Josef > > >>> l = [[1,0,0],[1,1,0],[1,1,1]] > >>> np.dstack(l).shape > (1, 3, 3) > >>> np.vstack(l).shape > (3, 3) > >>> np.hstack(l).shape > (9,) > >>> np.vstack(l) > array([[1, 0, 0], > [1, 1, 0], > [1, 1, 1]]) > >>> np.dstack(l)[:,:,0] > array([[1, 0, 0]]) > >>> np.dstack(l)[:,:,1] > array([[1, 1, 0]]) > >>> np.dstack(l)[:,:,2] > array([[1, 1, 1]]) > ________ But stacking 2D arrays gives the same number of dimensions as stack 1D vectors: In [1]: dstack([eye(3)]*3) Out[1]: array([[[ 1., 1., 1.], [ 0., 0., 0.], [ 0., 0., 0.]], [[ 0., 0., 0.], [ 1., 1., 1.], [ 0., 0., 0.]], [[ 0., 0., 0.], [ 0., 0., 0.], [ 1., 1., 1.]]]) In [2]: dstack([eye(3)]*3).shape Out[2]: (3, 3, 3) So dstack is turning 1D vectors iinto 1xn arrays so that In [8]: dstack([[1,0]]*3).shape Out[8]: (1, 2, 3) In [9]: dstack([[[1,0]]]*3).shape Out[9]: (1, 2, 3) Note the different bracket counts. This messes up general applications so that dstacking 1D vectors has to be treated differently than dstacking arrays. You can't sum along the last axis of the stack and get a 1D vector back, whereas for dstacked 2D arrays summing along the last axis gives back a 2D array. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Apr 2 21:30:30 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 2 Apr 2009 20:30:30 -0500 Subject: [Numpy-discussion] Bug in dstack? In-Reply-To: References: <1cd32cbb0904021650m7cb91775m597a5d424ff04831@mail.gmail.com> Message-ID: <3d375d730904021830h3cd18c18w99aef667a14b4c59@mail.gmail.com> On Thu, Apr 2, 2009 at 19:18, Charles R Harris wrote: > > > On Thu, Apr 2, 2009 at 5:50 PM, wrote: >> >> On Thu, Apr 2, 2009 at 6:46 PM, Charles R Harris >> wrote: >> > Note: >> > >> > In [133]: l = [[1,0,0],[1,1,0],[1,1,1]] >> > >> > In [134]: dstack(l) >> > Out[134]: >> > array([[[1, 1, 1], >> > ??????? [0, 1, 1], >> > ??????? [0, 0, 1]]]) >> > >> > In [135]: dstack(l).shape >> > Out[135]: (1, 3, 3) >> > >> > >> > Shouldn't the shape be (3,3)? Also, for generalized ufuncs and >> > broadcasting >> > I think a function that stacked along the first axis instead of the last >> > would be useful. Maybe gstack or astack? >> > >> > Chuck >> > >> >> I think dstack, vstack and hstack work exactly as advertised in the >> docs, at least that's how I interpret and use them >> >> Josef >> >> >>> l = [[1,0,0],[1,1,0],[1,1,1]] >> >>> np.dstack(l).shape >> (1, 3, 3) >> >>> np.vstack(l).shape >> (3, 3) >> >>> np.hstack(l).shape >> (9,) >> >>> np.vstack(l) >> array([[1, 0, 0], >> ? ? ? [1, 1, 0], >> ? ? ? [1, 1, 1]]) >> >>> np.dstack(l)[:,:,0] >> array([[1, 0, 0]]) >> >>> np.dstack(l)[:,:,1] >> array([[1, 1, 0]]) >> >>> np.dstack(l)[:,:,2] >> array([[1, 1, 1]]) >> ________ > > But stacking 2D arrays gives the same number of dimensions as stack 1D > vectors: > > In [1]: dstack([eye(3)]*3) > Out[1]: > array([[[ 1.,? 1.,? 1.], > ??????? [ 0.,? 0.,? 0.], > ??????? [ 0.,? 0.,? 0.]], > > ?????? [[ 0.,? 0.,? 0.], > ??????? [ 1.,? 1.,? 1.], > ??????? [ 0.,? 0.,? 0.]], > > ?????? [[ 0.,? 0.,? 0.], > ??????? [ 0.,? 0.,? 0.], > ??????? [ 1.,? 1.,? 1.]]]) > > In [2]: dstack([eye(3)]*3).shape > Out[2]: (3, 3, 3) > > So dstack is turning 1D vectors iinto 1xn arrays so that > > In [8]: dstack([[1,0]]*3).shape > Out[8]: (1, 2, 3) > > In [9]: dstack([[[1,0]]]*3).shape > Out[9]: (1, 2, 3) > > Note the different bracket counts. This messes up general applications so > that dstacking 1D vectors has to be treated differently than dstacking > arrays. You can't sum along the last axis of the stack and get a 1D vector > back, whereas for dstacked 2D arrays summing along the last axis gives back > a 2D array. Depth stack presumes at-least-3D arrays (hence "along the third axis" rather than "along the last axis") and "upcasts" input arrays with atleast_3d(). That's just what it does and has always done. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stefan at sun.ac.za Fri Apr 3 09:57:23 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 3 Apr 2009 15:57:23 +0200 Subject: [Numpy-discussion] Fwd: Fast numpy array visualization In-Reply-To: References: Message-ID: <9457e7c80904030657pdb9682fm714e286bab1b43d5@mail.gmail.com> Hi all, Nicolas Rougier is doing some fun things with Pyglet and IPython! Cheers St?fan ---------- Forwarded message ---------- From: Nicolas Rougier Date: 2009/4/3 Subject: Fast numpy array visualization To: pyglet-users Hi all, I've adapted the code from the ipython/python terminal to display 2d numpy arrays (using texture) directly within the terminal. Sources & screenshots: http://www.loria.fr/~rougier/glnumpy.html Nicolas From sturla at molden.no Fri Apr 3 10:14:19 2009 From: sturla at molden.no (Sturla Molden) Date: Fri, 03 Apr 2009 16:14:19 +0200 Subject: [Numpy-discussion] dpss windows Message-ID: <49D619BB.5070603@molden.no> NumPy has support for various window functions (von Hann, hamming, blackman, bartlet, kaiser). DPSS windows (e.g. used by multitaper spectral estimators) are missing. Is there any particular reason for this? (No this is not a request, I already have my own dpss code. I just notice while reading the NumPy and SciPy docs today that dpss windows are missing.) Sturla Molden From cimrman3 at ntc.zcu.cz Fri Apr 3 10:13:28 2009 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Fri, 03 Apr 2009 16:13:28 +0200 Subject: [Numpy-discussion] Fwd: Fast numpy array visualization In-Reply-To: <9457e7c80904030657pdb9682fm714e286bab1b43d5@mail.gmail.com> References: <9457e7c80904030657pdb9682fm714e286bab1b43d5@mail.gmail.com> Message-ID: <49D61988.6010608@ntc.zcu.cz> St?fan van der Walt wrote: > Hi all, > > Nicolas Rougier is doing some fun things with Pyglet and IPython! Awesome! r. > ---------- Forwarded message ---------- > From: Nicolas Rougier > Date: 2009/4/3 > Subject: Fast numpy array visualization > To: pyglet-users > > Hi all, > > I've adapted the code from the ipython/python terminal to display 2d > numpy arrays (using texture) directly within the terminal. > > Sources & screenshots: > http://www.loria.fr/~rougier/glnumpy.html > > > > Nicolas > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From stefan at sun.ac.za Fri Apr 3 10:21:38 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 3 Apr 2009 16:21:38 +0200 Subject: [Numpy-discussion] dpss windows In-Reply-To: <49D619BB.5070603@molden.no> References: <49D619BB.5070603@molden.no> Message-ID: <9457e7c80904030721k966a5cdx3a7053274f9a6020@mail.gmail.com> Hi Sturla 2009/4/3 Sturla Molden : > NumPy has support for various window functions (von Hann, hamming, > blackman, bartlet, kaiser). DPSS windows (e.g. used by multitaper > spectral estimators) are missing. Is there any particular reason for this? If things are missing, it probably means no one needed them thus far. Would you mind uploading your patch to Trac? I still owe you a review on your other patch; I can do that on Sunday evening. Cheers St?fan From beckers at orn.mpg.de Fri Apr 3 10:54:12 2009 From: beckers at orn.mpg.de (Gabriel Beckers) Date: Fri, 03 Apr 2009 16:54:12 +0200 Subject: [Numpy-discussion] dpss windows In-Reply-To: <9457e7c80904030721k966a5cdx3a7053274f9a6020@mail.gmail.com> References: <49D619BB.5070603@molden.no> <9457e7c80904030721k966a5cdx3a7053274f9a6020@mail.gmail.com> Message-ID: <1238770452.25582.22.camel@gabriel-desktop> I, for one, would be very grateful if DPSS code could be added to NumPy. Best, Gabriel On Fri, 2009-04-03 at 16:21 +0200, St?fan van der Walt wrote: > Hi Sturla > > 2009/4/3 Sturla Molden : > > NumPy has support for various window functions (von Hann, hamming, > > blackman, bartlet, kaiser). DPSS windows (e.g. used by multitaper > > spectral estimators) are missing. Is there any particular reason for this? > > If things are missing, it probably means no one needed them thus far. > Would you mind uploading your patch to Trac? > > I still owe you a review on your other patch; I can do that on Sunday evening. > > Cheers From david at ar.media.kyoto-u.ac.jp Fri Apr 3 11:18:18 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 04 Apr 2009 00:18:18 +0900 Subject: [Numpy-discussion] [ANN] Numpy 1.3.0rc2 Message-ID: <49D628BA.2000404@ar.media.kyoto-u.ac.jp> Hi, I am pleased to announce the release of the rc2 for numpy 1.3.0. I have decided to go for a rc2 instead of the release directly because of the serious mac os X issue. You can find source tarballs and installers for both Mac OS X and Windows on the sourceforge page: https://sourceforge.net/projects/numpy/ > The release note for the 1.3.0 release are below, the numpy developers. ========================= NumPy 1.3.0 Release Notes ========================= This minor includes numerous bug fixes, official python 2.6 support, and several new features such as generalized ufuncs. Highlights ========== Python 2.6 support ~~~~~~~~~~~~~~~~~~ Python 2.6 is now supported on all previously supported platforms, including windows. http://www.python.org/dev/peps/pep-0361/ Generalized ufuncs ~~~~~~~~~~~~~~~~~~ There is a general need for looping over not only functions on scalars but also over functions on vectors (or arrays), as explained on http://scipy.org/scipy/numpy/wiki/GeneralLoopingFunctions. We propose to realize this concept by generalizing the universal functions (ufuncs), and provide a C implementation that adds ~500 lines to the numpy code base. In current (specialized) ufuncs, the elementary function is limited to element-by-element operations, whereas the generalized version supports "sub-array" by "sub-array" operations. The Perl vector library PDL provides a similar functionality and its terms are re-used in the following. Each generalized ufunc has information associated with it that states what the "core" dimensionality of the inputs is, as well as the corresponding dimensionality of the outputs (the element-wise ufuncs have zero core dimensions). The list of the core dimensions for all arguments is called the "signature" of a ufunc. For example, the ufunc numpy.add has signature "(),()->()" defining two scalar inputs and one scalar output. Another example is (see the GeneralLoopingFunctions page) the function inner1d(a,b) with a signature of "(i),(i)->()". This applies the inner product along the last axis of each input, but keeps the remaining indices intact. For example, where a is of shape (3,5,N) and b is of shape (5,N), this will return an output of shape (3,5). The underlying elementary function is called 3*5 times. In the signature, we specify one core dimension "(i)" for each input and zero core dimensions "()" for the output, since it takes two 1-d arrays and returns a scalar. By using the same name "i", we specify that the two corresponding dimensions should be of the same size (or one of them is of size 1 and will be broadcasted). The dimensions beyond the core dimensions are called "loop" dimensions. In the above example, this corresponds to (3,5). The usual numpy "broadcasting" rules apply, where the signature determines how the dimensions of each input/output object are split into core and loop dimensions: While an input array has a smaller dimensionality than the corresponding number of core dimensions, 1's are pre-pended to its shape. The core dimensions are removed from all inputs and the remaining dimensions are broadcasted; defining the loop dimensions. The output is given by the loop dimensions plus the output core dimensions. Experimental Windows 64 bits support ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Numpy can now be built on windows 64 bits (amd64 only, not IA64), with both MS compilers and mingw-w64 compilers: This is *highly experimental*: DO NOT USE FOR PRODUCTION USE. See INSTALL.txt, Windows 64 bits section for more information on limitations and how to build it by yourself. New features ============ Formatting issues ~~~~~~~~~~~~~~~~~ Float formatting is now handled by numpy instead of the C runtime: this enables locale independent formatting, more robust fromstring and related methods. Special values (inf and nan) are also more consistent across platforms (nan vs IND/NaN, etc...), and more consistent with recent python formatting work (in 2.6 and later). Nan handling in max/min ~~~~~~~~~~~~~~~~~~~~~~~ The maximum/minimum ufuncs now reliably propagate nans. If one of the arguments is a nan, then nan is retured. This affects np.min/np.max, amin/amax and the array methods max/min. New ufuncs fmax and fmin have been added to deal with non-propagating nans. Nan handling in sign ~~~~~~~~~~~~~~~~~~~~ The ufunc sign now returns nan for the sign of anan. New ufuncs ~~~~~~~~~~ #. fmax - same as maximum for integer types and non-nan floats. Returns the non-nan argument if one argument is nan and returns nan if both arguments are nan. #. fmin - same as minimum for integer types and non-nan floats. Returns the non-nan argument if one argument is nan and returns nan if both arguments are nan. #. deg2rad - converts degrees to radians, same as the radians ufunc. #. rad2deg - converts radians to degrees, same as the degrees ufunc. #. log2 - base 2 logarithm. #. exp2 - base 2 exponential. #. trunc - truncate floats to nearest integer towards zero. #. logaddexp - add numbers stored as logarithms and return the logarithm of the result. #. logaddexp2 - add numbers stored as base 2 logarithms and return the base 2 logarithm of the result result. Masked arrays ~~~~~~~~~~~~~ Several new features and bug fixes, including: * structured arrays should now be fully supported by MaskedArray (r6463, r6324, r6305, r6300, r6294...) * Minor bug fixes (r6356, r6352, r6335, r6299, r6298) * Improved support for __iter__ (r6326) * made baseclass, sharedmask and hardmask accesible to the user (but read-only) * doc update gfortran support on windows ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Gfortran can now be used as a fortran compiler for numpy on windows, even when the C compiler is Visual Studio (VS 2005 and above; VS 2003 will NOT work). Gfortran + Visual studio does not work on windows 64 bits (but gcc + gfortran does). It is unclear whether it will be possible to use gfortran and visual studio at all on x64. Arch option for windows binary ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Automatic arch detection can now be bypassed from the command line for the superpack installed: numpy-1.3.0-superpack-win32.exe /arch=nosse will install a numpy which works on any x86, even if the running computer supports SSE set. Deprecated features =================== Histogram ~~~~~~~~~ The semantics of histogram has been modified to fix long-standing issues with outliers handling. The main changes concern #. the definition of the bin edges, now including the rightmost edge, and #. the handling of upper outliers, now ignored rather than tallied in the rightmost bin. The previous behavior is still accessible using `new=False`, but this is deprecated, and will be removed entirely in 1.4.0. Documentation changes ===================== A lot of documentation has been added. Both user guide and references can be built from sphinx. New C API ========= Multiarray API ~~~~~~~~~~~~~~ The following functions have been added to the multiarray C API: * PyArray_GetEndianness: to get runtime endianness Ufunc API ~~~~~~~~~~~~~~ The following functions have been added to the ufunc API: * PyUFunc_FromFuncAndDataAndSignature: to declare a more general ufunc (generalized ufunc). New defines ~~~~~~~~~~~ New public C defines are available for ARCH specific code through numpy/npy_cpu.h: * NPY_CPU_X86: x86 arch (32 bits) * NPY_CPU_AMD64: amd64 arch (x86_64, NOT Itanium) * NPY_CPU_PPC: 32 bits ppc * NPY_CPU_PPC64: 64 bits ppc * NPY_CPU_SPARC: 32 bits sparc * NPY_CPU_SPARC64: 64 bits sparc * NPY_CPU_S390: S390 * NPY_CPU_IA64: ia64 * NPY_CPU_PARISC: PARISC New macros for CPU endianness has been added as well (see internal changes below for details): * NPY_BYTE_ORDER: integer * NPY_LITTLE_ENDIAN/NPY_BIG_ENDIAN defines Those provide portable alternatives to glibc endian.h macros for platforms without it. Portable NAN, INFINITY, etc... ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ npy_math.h now makes available several portable macro to get NAN, INFINITY: * NPY_NAN: equivalent to NAN, which is a GNU extension * NPY_INFINITY: equivalent to C99 INFINITY * NPY_PZERO, NPY_NZERO: positive and negative zero respectively Corresponding single and extended precision macros are available as well. All references to NAN, or home-grown computation of NAN on the fly have been removed for consistency. Internal changes ================ numpy.core math configuration revamp ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This should make the porting to new platforms easier, and more robust. In particular, the configuration stage does not need to execute any code on the target platform, which is a first step toward cross-compilation. http://projects.scipy.org/numpy/browser/trunk/doc/neps/math_config_clean.txt umath refactor ~~~~~~~~~~~~~~ A lot of code cleanup for umath/ufunc code (charris). Improvements to build warnings ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Numpy can now build with -W -Wall without warnings http://projects.scipy.org/numpy/browser/trunk/doc/neps/warnfix.txt Separate core math library ~~~~~~~~~~~~~~~~~~~~~~~~~~ The core math functions (sin, cos, etc... for basic C types) have been put into a separate library; it acts as a compatibility layer, to support most C99 maths functions (real only for now). The library includes platform-specific fixes for various maths functions, such as using those versions should be more robust than using your platform functions directly. The API for existing functions is exactly the same as the C99 math functions API; the only difference is the npy prefix (npy_cos vs cos). The core library will be made available to any extension in 1.4.0. CPU arch detection ~~~~~~~~~~~~~~~~~~ npy_cpu.h defines numpy specific CPU defines, such as NPY_CPU_X86, etc... Those are portable across OS and toolchains, and set up when the header is parsed, so that they can be safely used even in the case of cross-compilation (the values is not set when numpy is built), or for multi-arch binaries (e.g. fat binaries on Max OS X). npy_endian.h defines numpy specific endianness defines, modeled on the glibc endian.h. NPY_BYTE_ORDER is equivalent to BYTE_ORDER, and one of NPY_LITTLE_ENDIAN or NPY_BIG_ENDIAN is defined. As for CPU archs, those are set when the header is parsed by the compiler, and as such can be used for cross-compilation and multi-arch binaries. Checksums ========= 55a2d2c9ba056133f61ad70e6445d2f9 release/installers/numpy-1.3.0rc2.tar.gz ea88cd161ac8b267bd85419dddde40a5 release/installers/numpy-1.3.0rc2.zip 77b5a1ed12f86353f7ab998e1ce7d4b9 release/installers/numpy-1.3.0rc2-win32-superpack-python2.5.exe 803425416d6e86ce5cc2704aab16eedf release/installers/numpy-1.3.0rc2-win32-superpack-python2.6.exe b00b22484080b58141961af9a9707b01 release/installers/numpy-1.3.0rc2-py2.5-macosx10.5.dmg From stefan at sun.ac.za Fri Apr 3 11:55:58 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 3 Apr 2009 17:55:58 +0200 Subject: [Numpy-discussion] [ANN] Numpy 1.3.0rc2 In-Reply-To: <49D628BA.2000404@ar.media.kyoto-u.ac.jp> References: <49D628BA.2000404@ar.media.kyoto-u.ac.jp> Message-ID: <9457e7c80904030855t7283e87en51de528c84ab7f62@mail.gmail.com> Hi David 2009/4/3 David Cournapeau : > I am pleased to announce the release of the rc2 for numpy 1.3.0. I have > decided to go for a rc2 instead of the release directly because of the > serious mac os X issue. You can find source tarballs and installers for > both Mac OS X and Windows on the sourceforge page: Thanks for all your effort! Also, thanks to the 3 people (out of the 300 who downloaded the binary) who reported that it was unusable :-) Cheers St?fan From tgrav at mac.com Fri Apr 3 12:01:07 2009 From: tgrav at mac.com (Tommy Grav) Date: Fri, 03 Apr 2009 12:01:07 -0400 Subject: [Numpy-discussion] Fwd: [ANN] Numpy 1.3.0rc2 References: <66C61B69-9EF4-4D73-9A0F-431F67C41643@pha.jhu.edu> Message-ID: On Apr 3, 2009, at 11:18 AM, David Cournapeau wrote: > Hi, > > I am pleased to announce the release of the rc2 for numpy 1.3.0. I > have > decided to go for a rc2 instead of the release directly because of the > serious mac os X issue. You can find source tarballs and installers > for > both Mac OS X and Windows on the sourceforge page: Installed on my PowerMac Intel running 10.5.6 and ActiveState 2.5.4.3 using the mpkg installer. Worked fine. tgrav% python ActivePython 2.5.4.3 (ActiveState Software Inc.) based on Python 2.5.4 (r254:67916, Jan 20 2009, 14:11:42) [GCC 4.0.1 (Apple Computer, Inc. build 5250)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> print numpy.__version__ 1.3.0rc2 >>> numpy.test() Running unit tests for numpy NumPy version 1.3.0rc2 NumPy is installed in /Library/Frameworks/Python.framework/Versions/ 2.5/lib/python2.5/site-packages/numpy Python version 2.5.4 (r254:67916, Jan 20 2009, 14:11:42) [GCC 4.0.1 (Apple Computer, Inc. build 5250)] nose version 0.10.4 ........................................................................ ........................................................................ ........................................................................ ........................................................................ ........................................................................ ........................................................................ ........................................................................ ................................................K....................... ........................................................................ ........................................................................ ........................................................................ ........................................................................ ........................................................................ ........................................................................ ........................................................................ .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S.............................................................................................................................................................................................................................................................................................................................................................................................................................. ---------------------------------------------------------------------- Ran 2030 tests in 7.663s OK (KNOWNFAIL=1, SKIP=1) >>> From rpyle at post.harvard.edu Fri Apr 3 12:44:09 2009 From: rpyle at post.harvard.edu (Robert Pyle) Date: Fri, 03 Apr 2009 12:44:09 -0400 Subject: [Numpy-discussion] [ANN] Numpy 1.3.0rc2 In-Reply-To: <49D628BA.2000404@ar.media.kyoto-u.ac.jp> References: <49D628BA.2000404@ar.media.kyoto-u.ac.jp> Message-ID: <80657AC3-D835-4659-98F8-C5AD43AC3EDF@post.harvard.edu> Hi David, On Apr 3, 2009, at 11:18 AM, David Cournapeau wrote: > > I am pleased to announce the release of the rc2 for numpy 1.3.0. I > have > decided to go for a rc2 instead of the release directly because of the > serious mac os X issue. You can find source tarballs and installers > for > both Mac OS X and Windows on the sourceforge page: > > https://sourceforge.net/projects/numpy/ PPC G5, Mac OS X 10.5.6 Installing from source for my usual python (EPD Py25 4.1.30101) went smoothly and numpy.test() passed. Installing from the OS X installer for python.org was uneventful, but when I tried to run the usual tests, I got ------------------------------------------------ ~ $ /Library/Frameworks/Python.framework/Versions/2.5/bin/python2.5 Python 2.5.4 (r254:67917, Dec 23 2008, 14:57:27) [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.__version__ '1.3.0rc2' >>> np.test() Running unit tests for numpy Traceback (most recent call last): File "", line 1, in File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/testing/nosetester.py", line 242, in test self._show_system_info() File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/testing/nosetester.py", line 149, in _show_system_info nose = import_nose() File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/testing/nosetester.py", line 49, in import_nose raise ImportError(msg) ImportError: Need nose >= 0.10.0 for tests - see http://somethingaboutorange.com/mrl/projects/nose >>> ------------------------------------------------ I guess EPD python already has nose, but python.org does not? Bob From millman at berkeley.edu Fri Apr 3 13:03:42 2009 From: millman at berkeley.edu (Jarrod Millman) Date: Fri, 3 Apr 2009 10:03:42 -0700 Subject: [Numpy-discussion] [ANN] Numpy 1.3.0rc2 In-Reply-To: <49D628BA.2000404@ar.media.kyoto-u.ac.jp> References: <49D628BA.2000404@ar.media.kyoto-u.ac.jp> Message-ID: On Fri, Apr 3, 2009 at 8:18 AM, David Cournapeau wrote: > I am pleased to announce the release of the rc2 for numpy 1.3.0. I have > decided to go for a rc2 instead of the release directly because of the > serious mac os X issue. You can find source tarballs and installers for > both Mac OS X and Windows on the sourceforge page: Thanks to David and everyone else who worked on this release! I am looking forward to 1.3.0 final. Jarrod From cournape at gmail.com Fri Apr 3 13:10:58 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 4 Apr 2009 02:10:58 +0900 Subject: [Numpy-discussion] [ANN] Numpy 1.3.0rc2 In-Reply-To: <80657AC3-D835-4659-98F8-C5AD43AC3EDF@post.harvard.edu> References: <49D628BA.2000404@ar.media.kyoto-u.ac.jp> <80657AC3-D835-4659-98F8-C5AD43AC3EDF@post.harvard.edu> Message-ID: <5b8d13220904031010i49964303tafad35248c771a6b@mail.gmail.com> On Sat, Apr 4, 2009 at 1:44 AM, Robert Pyle wrote: > > PPC G5, Mac OS X 10.5.6 > > Installing from source for my usual python (EPD Py25 4.1.30101) went > smoothly and numpy.test() passed. That's confusing. I have no idea why this works - or maybe EPD loads numpy as installed in the python.org binary ? > > I guess EPD python already has nose, but python.org does not? Indeed. The whole point of EPD is to package a whole suite of tools integrated together - you need to install node separately. cheers, David From aisaac at american.edu Fri Apr 3 13:37:07 2009 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 03 Apr 2009 13:37:07 -0400 Subject: [Numpy-discussion] [ANN] Numpy 1.3.0rc2 In-Reply-To: <9457e7c80904030855t7283e87en51de528c84ab7f62@mail.gmail.com> References: <49D628BA.2000404@ar.media.kyoto-u.ac.jp> <9457e7c80904030855t7283e87en51de528c84ab7f62@mail.gmail.com> Message-ID: <49D64943.10404@american.edu> Looks good. Cheers, Alan Isaac Python 2.6.1 (r261:67517, Dec 4 2008, 16:51:00) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.test() Running unit tests for numpy NumPy version 1.3.0rc2 NumPy is installed in C:\Python26\lib\site-packages\numpy Python version 2.6.1 (r261:67517, Dec 4 2008, 16:51:00) [MSC v.1500 32 bit (Intel)] nose version 0.11.0 [snip] Ran 1882 tests in 10.453s OK (KNOWNFAIL=3, SKIP=1) From atmyers at berkeley.edu Fri Apr 3 15:28:45 2009 From: atmyers at berkeley.edu (Andrew Myers) Date: Fri, 3 Apr 2009 12:28:45 -0700 Subject: [Numpy-discussion] "Can't install when cross-compiling" Message-ID: <327949C0-998D-48F9-8369-BFC87360E163@berkeley.edu> Hello All, I'm trying to install numpy 1.2.1 for 64-bit python 2.6.1 on Leopard. Python appears to have been installed correctly, and sys.maxint returns the 64 bit value. When I do sudo python-64 setup.py config_fc --fcompiler=gnu95 install I get the following error message: running build_scripts adding 'build/scripts.macosx-10.5-universal-2.6/f2py64-64' to scripts error: Can't install when cross-compiling What's weird is that I've got this working once before, but after I tried to erase and re-install I started getting the error. Chris Kees seems to have had this problem a few months back: http://www.nabble.com/Mac-OSX-4-way-universal-td18957309.html He found that after erasing "Library/Frameworks/Python64.framework" the build worked correctly. Apparently having some remnant of a previous install on your system tricks the installer into thinking you are cross-compiling. However, removing that directory doesn't work for me. Any ideas? Thanks! From theller at ctypes.org Fri Apr 3 15:38:18 2009 From: theller at ctypes.org (Thomas Heller) Date: Fri, 03 Apr 2009 21:38:18 +0200 Subject: [Numpy-discussion] numpy.ctypeslib.ndpointer and the restype attribute [patch] In-Reply-To: References: <6183458.298391237815382345.JavaMail.tomcat@pne-ps1-sn2> <1238067688.6464.29.camel@supraflex> <49CB804E.8060402@molden.no> Message-ID: Thomas Heller schrieb: >> Sturla Molden schrieb: >>> On 3/26/2009 12:41 PM, Jens Rantil wrote: >>> >>>> Wouldn't my code, or a tweak of it, be a nice feature in >>>> numpy.ctypeslib? Is this the wrong channel for proposing things like >>>> this? >>> >>> If you look at >>> >>> http://svn.scipy.org/svn/numpy/trunk/numpy/ctypeslib.py >>> >>> you will see that it does almost the same. I think it would be better to >>> work out why ndpointer fails as restype and patch that. >> > > Thomas Heller schrieb: > >> ndpointer(...), which returns an _nptr instance, does not work as restype >> because neither it is a base class of one of the ctypes base types like >> ctypes.c_void_p, also it is not callable with one argument. >> >> There are two ways to fix this. The first one is to make the _nptr callable > [...] >> >> The other way is to make _nptr a subclass of ctypes.c_void_p, >> the result that the foreign function call returns will then be >> an instance of this class. Unfortunately, ctypes will not call >> __new__() to create this instance; so a custom __new__() implementation >> cannot return a numpy array and we are left with the _nptr instance. >> The only way to create and access the numpy array is to construct >> and return one from a method call on the _nptr instance, or a property >> on the _nptr instance. >> Ok, .errcheck could call that method and return the result. >> > Well, looking into the ctypes sources trying to invent a new protocol for > the restype attribute I found out that THERE IS ALREADY a mechanism for it, > but I had totally forgotten that it exists. > > When the .restype attribute of a function is set to a SUBCLASS of a > ctypes type (c_void_p for example), an instance of this subclass is created. > After that, if this instance has a _check_retval_ method, this method is called > and the result of this call is returned. So, it is indeed possible to create > a class that can be assigned to .restype, and which can convert the return value > of a function to whatever we like. > > I will prepare a patch for numpy.ctypeslib. > It seems there isn't much interest in a patch - true? Anyway, before it gets totally forgotten, here is the patch. The _ndptr._check_retval_ method should probably be extended to handle the memory management of the result. diff -u c:\python25\lib\site-packages\numpy\ctypeslib.py.orig c:\python25\lib\site-packages\numpy\ctypeslib.py.new --- c:\python25\lib\site-packages\numpy\ctypeslib.py.orig Fri Apr 03 19:35:10 2009 +++ c:\python25\lib\site-packages\numpy\ctypeslib.py.new Fri Apr 03 19:35:10 2009 @@ -69,10 +69,12 @@ as_ctypes = _dummy as_array = _dummy from numpy import intp as c_intp + _ndptr_base = object else: import numpy.core._internal as nic c_intp = nic._getintp_ctype() del nic + _ndptr_base = ctypes.c_void_p # Adapted from Albert Strasheim def load_library(libname, loader_path): @@ -130,7 +132,24 @@ return res -class _ndptr(object): +class _ndptr(_ndptr_base): + + def _check_retval_(self): + """This method is called when this class is used as .restype + attribute for a dll function. It constructs a numpy array from + a void pointer.""" + return array(self) + + @property + def __array_interface__(self): + return {'descr': self._dtype_.descr, + '__ref': self, + 'strides': None, + 'shape': self._shape_, + 'version': 3, + 'typestr': self._dtype_.descr[0][1], + 'data': (self.value, False), + } def from_param(cls, obj): if not isinstance(obj, ndarray): -- Thanks, Thomas From rpyle at post.harvard.edu Fri Apr 3 16:26:28 2009 From: rpyle at post.harvard.edu (Robert Pyle) Date: Fri, 03 Apr 2009 16:26:28 -0400 Subject: [Numpy-discussion] [ANN] Numpy 1.3.0rc2 In-Reply-To: <5b8d13220904031010i49964303tafad35248c771a6b@mail.gmail.com> References: <49D628BA.2000404@ar.media.kyoto-u.ac.jp> <80657AC3-D835-4659-98F8-C5AD43AC3EDF@post.harvard.edu> <5b8d13220904031010i49964303tafad35248c771a6b@mail.gmail.com> Message-ID: <8413BD97-F989-4C0C-87AD-19D35670769F@post.harvard.edu> On Apr 3, 2009, at 1:10 PM, David Cournapeau wrote: > On Sat, Apr 4, 2009 at 1:44 AM, Robert Pyle > wrote: > >> >> PPC G5, Mac OS X 10.5.6 >> >> Installing from source for my usual python (EPD Py25 4.1.30101) went >> smoothly and numpy.test() passed. > > That's confusing. I have no idea why this works - or maybe EPD loads > numpy as installed in the python.org binary ? I should have been clearer: I performed two installations, from the source tarball for EPD, and from the OS X installer for the python.org 2.5 python. > >> >> I guess EPD python already has nose, but python.org does not? After I installed nose for the python.org 2.5, numpy.test() runs perfectly, that is, Ran 2030 tests in 13.710s OK (KNOWNFAIL=2, SKIP=1) Bob From david at ar.media.kyoto-u.ac.jp Sat Apr 4 05:51:52 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 04 Apr 2009 18:51:52 +0900 Subject: [Numpy-discussion] [ANN] Numpy 1.3.0rc2 In-Reply-To: <8413BD97-F989-4C0C-87AD-19D35670769F@post.harvard.edu> References: <49D628BA.2000404@ar.media.kyoto-u.ac.jp> <80657AC3-D835-4659-98F8-C5AD43AC3EDF@post.harvard.edu> <5b8d13220904031010i49964303tafad35248c771a6b@mail.gmail.com> <8413BD97-F989-4C0C-87AD-19D35670769F@post.harvard.edu> Message-ID: <49D72DB8.3010706@ar.media.kyoto-u.ac.jp> Robert Pyle wrote: > > I should have been clearer: I performed two installations, from the > source tarball for EPD, and from the OS X installer for the python.org > 2.5 python. > ok. Thanks for the report, David From oliphant at enthought.com Sat Apr 4 14:48:41 2009 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sat, 04 Apr 2009 13:48:41 -0500 Subject: [Numpy-discussion] numpy.ctypeslib.ndpointer and the restype attribute [patch] In-Reply-To: References: <6183458.298391237815382345.JavaMail.tomcat@pne-ps1-sn2> <1238067688.6464.29.camel@supraflex> <49CB804E.8060402@molden.no> Message-ID: <49D7AB89.6000708@enthought.com> Thomas Heller wrote: > Thomas Heller schrieb: > >>> Sturla Molden schrieb: >>> >>>> On 3/26/2009 12:41 PM, Jens Rantil wrote: >>>> >>>> >>>>> Wouldn't my code, or a tweak of it, be a nice feature in >>>>> numpy.ctypeslib? Is this the wrong channel for proposing things like >>>>> this? >>>>> >>>> If you look at >>>> >>>> http://svn.scipy.org/svn/numpy/trunk/numpy/ctypeslib.py >>>> >>>> you will see that it does almost the same. I think it would be better to >>>> work out why ndpointer fails as restype and patch that. >>>> >> Thomas Heller schrieb: >> >> >>> ndpointer(...), which returns an _nptr instance, does not work as restype >>> because neither it is a base class of one of the ctypes base types like >>> ctypes.c_void_p, also it is not callable with one argument. >>> >>> There are two ways to fix this. The first one is to make the _nptr callable >>> >> [...] >> >>> The other way is to make _nptr a subclass of ctypes.c_void_p, >>> the result that the foreign function call returns will then be >>> an instance of this class. Unfortunately, ctypes will not call >>> __new__() to create this instance; so a custom __new__() implementation >>> cannot return a numpy array and we are left with the _nptr instance. >>> The only way to create and access the numpy array is to construct >>> and return one from a method call on the _nptr instance, or a property >>> on the _nptr instance. >>> Ok, .errcheck could call that method and return the result. >>> >>> >> Well, looking into the ctypes sources trying to invent a new protocol for >> the restype attribute I found out that THERE IS ALREADY a mechanism for it, >> but I had totally forgotten that it exists. >> >> When the .restype attribute of a function is set to a SUBCLASS of a >> ctypes type (c_void_p for example), an instance of this subclass is created. >> After that, if this instance has a _check_retval_ method, this method is called >> and the result of this call is returned. So, it is indeed possible to create >> a class that can be assigned to .restype, and which can convert the return value >> of a function to whatever we like. >> >> I will prepare a patch for numpy.ctypeslib. >> >> > > It seems there isn't much interest in a patch - true? > No, I'm very interested in the patch. Thanks for it. -Travis From pgmdevlist at gmail.com Sat Apr 4 22:17:37 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Sat, 4 Apr 2009 22:17:37 -0400 Subject: [Numpy-discussion] Construct symmetric matrix Message-ID: <46FED3E5-4EFE-4EAE-B75A-DF01A80BE2CF@gmail.com> All, I'm trying to build a relatively complicated symmetric matrix. I can build the upper-right block without pb. What is the fastest way to get the LL corner from the UR one ? Thanks a lot in advance for any idea. P. From cournape at gmail.com Sun Apr 5 00:18:44 2009 From: cournape at gmail.com (David Cournapeau) Date: Sun, 5 Apr 2009 13:18:44 +0900 Subject: [Numpy-discussion] Construct symmetric matrix In-Reply-To: <46FED3E5-4EFE-4EAE-B75A-DF01A80BE2CF@gmail.com> References: <46FED3E5-4EFE-4EAE-B75A-DF01A80BE2CF@gmail.com> Message-ID: <5b8d13220904042118i673527dsf2b0ac83a9b542d9@mail.gmail.com> On Sun, Apr 5, 2009 at 11:17 AM, Pierre GM wrote: > All, > I'm trying to build a relatively complicated symmetric matrix. I can > build the upper-right block without pb. What is the fastest way to get > the LL corner from the UR one ? > Thanks a lot in advance for any idea. I may be missing something, but why not building it from scratch + a couple of elementary operations ? Is it too slow ? For example: # Assuming the lower part of a is zero before the operation a + a.T - np.diag(a.diagonal()) David From Chris.Barker at noaa.gov Sun Apr 5 00:29:17 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Sat, 04 Apr 2009 21:29:17 -0700 Subject: [Numpy-discussion] [ANN] Numpy 1.3.0rc2 In-Reply-To: <49D628BA.2000404@ar.media.kyoto-u.ac.jp> References: <49D628BA.2000404@ar.media.kyoto-u.ac.jp> Message-ID: <49D8339D.8060708@noaa.gov> > I am pleased to announce the release of the rc2 for numpy 1.3.0. OS-X 10.4 Intel MacBook. Python.org Python 2.5: Ran 2030 tests in 10.865s OK (KNOWNFAIL=1, SKIP=1) Looks good to me -- Thanks for all your work on this, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From nwagner at iam.uni-stuttgart.de Sun Apr 5 05:21:38 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Sun, 05 Apr 2009 11:21:38 +0200 Subject: [Numpy-discussion] cannot build numpy from trunk Message-ID: Hi all, I cannot build numpy from trunk compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.6 -c' gcc: _configtest.c _configtest.c:1: warning: conflicting types for built-in function ?asin? _configtest.c:2: warning: conflicting types for built-in function ?cos? _configtest.c:3: warning: conflicting types for built-in function ?log? _configtest.c:4: warning: conflicting types for built-in function ?fabs? _configtest.c:5: warning: conflicting types for built-in function ?tanh? _configtest.c:6: warning: conflicting types for built-in function ?atan? _configtest.c:7: warning: conflicting types for built-in function ?acos? _configtest.c:8: warning: conflicting types for built-in function ?floor? _configtest.c:9: warning: conflicting types for built-in function ?fmod? _configtest.c:10: warning: conflicting types for built-in function ?sqrt? _configtest.c:11: warning: conflicting types for built-in function ?cosh? _configtest.c:12: warning: conflicting types for built-in function ?modf? _configtest.c:13: warning: conflicting types for built-in function ?sinh? _configtest.c:14: warning: conflicting types for built-in function ?frexp? _configtest.c:15: warning: conflicting types for built-in function ?exp? _configtest.c:16: warning: conflicting types for built-in function ?tan? _configtest.c:17: warning: conflicting types for built-in function ?ceil? _configtest.c:18: warning: conflicting types for built-in function ?log10? _configtest.c:19: warning: conflicting types for built-in function ?sin? _configtest.c:20: warning: conflicting types for built-in function ?ldexp? /usr/bin/gcc _configtest.o -o _configtest _configtest.o: In function `main': /home/nwagner/svn/numpy/_configtest.c:44: undefined reference to `sin' /home/nwagner/svn/numpy/_configtest.c:45: undefined reference to `cos' /home/nwagner/svn/numpy/_configtest.c:46: undefined reference to `tan' /home/nwagner/svn/numpy/_configtest.c:47: undefined reference to `sinh' /home/nwagner/svn/numpy/_configtest.c:48: undefined reference to `cosh' /home/nwagner/svn/numpy/_configtest.c:49: undefined reference to `tanh' /home/nwagner/svn/numpy/_configtest.c:50: undefined reference to `fabs' /home/nwagner/svn/numpy/_configtest.c:51: undefined reference to `floor' /home/nwagner/svn/numpy/_configtest.c:52: undefined reference to `ceil' /home/nwagner/svn/numpy/_configtest.c:53: undefined reference to `sqrt' /home/nwagner/svn/numpy/_configtest.c:54: undefined reference to `log10' /home/nwagner/svn/numpy/_configtest.c:55: undefined reference to `log' /home/nwagner/svn/numpy/_configtest.c:56: undefined reference to `exp' /home/nwagner/svn/numpy/_configtest.c:57: undefined reference to `asin' /home/nwagner/svn/numpy/_configtest.c:58: undefined reference to `acos' /home/nwagner/svn/numpy/_configtest.c:59: undefined reference to `atan' /home/nwagner/svn/numpy/_configtest.c:60: undefined reference to `fmod' collect2: ld returned 1 exit status _configtest.o: In function `main': /home/nwagner/svn/numpy/_configtest.c:44: undefined reference to `sin' /home/nwagner/svn/numpy/_configtest.c:45: undefined reference to `cos' /home/nwagner/svn/numpy/_configtest.c:46: undefined reference to `tan' /home/nwagner/svn/numpy/_configtest.c:47: undefined reference to `sinh' /home/nwagner/svn/numpy/_configtest.c:48: undefined reference to `cosh' /home/nwagner/svn/numpy/_configtest.c:49: undefined reference to `tanh' /home/nwagner/svn/numpy/_configtest.c:50: undefined reference to `fabs' /home/nwagner/svn/numpy/_configtest.c:51: undefined reference to `floor' /home/nwagner/svn/numpy/_configtest.c:52: undefined reference to `ceil' /home/nwagner/svn/numpy/_configtest.c:53: undefined reference to `sqrt' /home/nwagner/svn/numpy/_configtest.c:54: undefined reference to `log10' /home/nwagner/svn/numpy/_configtest.c:55: undefined reference to `log' /home/nwagner/svn/numpy/_configtest.c:56: undefined reference to `exp' /home/nwagner/svn/numpy/_configtest.c:57: undefined reference to `asin' /home/nwagner/svn/numpy/_configtest.c:58: undefined reference to `acos' /home/nwagner/svn/numpy/_configtest.c:59: undefined reference to `atan' /home/nwagner/svn/numpy/_configtest.c:60: undefined reference to `fmod' collect2: ld returned 1 exit status failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 172, in setup_package() File "setup.py", line 165, in setup_package configuration=configuration ) File "/home/nwagner/svn/numpy/numpy/distutils/core.py", line 184, in setup return old_setup(**new_attr) File "/usr/lib64/python2.6/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib64/python2.6/distutils/dist.py", line 975, in run_commands self.run_command(cmd) File "/usr/lib64/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/home/nwagner/svn/numpy/numpy/distutils/command/install.py", line 49, in run r = old_install.run(self) File "/usr/lib64/python2.6/distutils/command/install.py", line 582, in run self.run_command('build') File "/usr/lib64/python2.6/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/lib64/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/home/nwagner/svn/numpy/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/usr/lib64/python2.6/distutils/command/build.py", line 134, in run self.run_command(cmd_name) File "/usr/lib64/python2.6/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/lib64/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/home/nwagner/svn/numpy/numpy/distutils/command/build_src.py", line 130, in run self.build_sources() File "/home/nwagner/svn/numpy/numpy/distutils/command/build_src.py", line 147, in build_sources self.build_extension_sources(ext) File "/home/nwagner/svn/numpy/numpy/distutils/command/build_src.py", line 250, in build_extension_sources sources = self.generate_sources(sources, ext) File "/home/nwagner/svn/numpy/numpy/distutils/command/build_src.py", line 307, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 297, in generate_config_h check_math_capabilities(config_cmd, moredefs, mathlibs) File "numpy/core/setup.py", line 126, in check_math_capabilities " available (the list is %s)." % str(MANDATORY_FUNCS)) SystemError: One of the required function to build numpy is not available (the list is ['sin', 'cos', 'tan', 'sinh', 'cosh', 'tanh', 'fabs', 'floor', 'ceil', 'sqrt', 'log10', 'log', 'exp', 'asin', 'acos', 'atan', 'fmod', 'modf', 'frexp', 'ldexp']). Nils From david at ar.media.kyoto-u.ac.jp Sun Apr 5 05:13:50 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sun, 05 Apr 2009 18:13:50 +0900 Subject: [Numpy-discussion] cannot build numpy from trunk In-Reply-To: References: Message-ID: <49D8764E.5060907@ar.media.kyoto-u.ac.jp> Nils Wagner wrote: > Hi all, > > I cannot build numpy from trunk > Yes, I broke something in the config stage, David From david at ar.media.kyoto-u.ac.jp Sun Apr 5 05:17:30 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sun, 05 Apr 2009 18:17:30 +0900 Subject: [Numpy-discussion] cannot build numpy from trunk In-Reply-To: <49D8764E.5060907@ar.media.kyoto-u.ac.jp> References: <49D8764E.5060907@ar.media.kyoto-u.ac.jp> Message-ID: <49D8772A.3000702@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > Nils Wagner wrote: > >> Hi all, >> >> I cannot build numpy from trunk >> >> > > Yes, I broke something in the config stage, > And it should be fixed now, David From david at ar.media.kyoto-u.ac.jp Sun Apr 5 06:04:39 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sun, 05 Apr 2009 19:04:39 +0900 Subject: [Numpy-discussion] [ANN] Numpy 1.3.0 released Message-ID: <49D88237.7050106@ar.media.kyoto-u.ac.jp> Hi, I am pleased to announce the release of numpy 1.3.0. You can find source tarballs and binary installers for both Mac OS X and Windows on the sourceforge page: https://sourceforge.net/projects/numpy/ The release note for the 1.3.0 release are below. There are no changes compared to the release candidate 2. Thank you to everyone involved in this release, developers, users who reported bugs, enjoy, the numpy developers. ---- ========================= NumPy 1.3.0 Release Notes ========================= This minor includes numerous bug fixes, official python 2.6 support, and several new features such as generalized ufuncs. Highlights ========== Python 2.6 support ~~~~~~~~~~~~~~~~~~ Python 2.6 is now supported on all previously supported platforms, including windows. http://www.python.org/dev/peps/pep-0361/ Generalized ufuncs ~~~~~~~~~~~~~~~~~~ There is a general need for looping over not only functions on scalars but also over functions on vectors (or arrays), as explained on http://scipy.org/scipy/numpy/wiki/GeneralLoopingFunctions. We propose to realize this concept by generalizing the universal functions (ufuncs), and provide a C implementation that adds ~500 lines to the numpy code base. In current (specialized) ufuncs, the elementary function is limited to element-by-element operations, whereas the generalized version supports "sub-array" by "sub-array" operations. The Perl vector library PDL provides a similar functionality and its terms are re-used in the following. Each generalized ufunc has information associated with it that states what the "core" dimensionality of the inputs is, as well as the corresponding dimensionality of the outputs (the element-wise ufuncs have zero core dimensions). The list of the core dimensions for all arguments is called the "signature" of a ufunc. For example, the ufunc numpy.add has signature "(),()->()" defining two scalar inputs and one scalar output. Another example is (see the GeneralLoopingFunctions page) the function inner1d(a,b) with a signature of "(i),(i)->()". This applies the inner product along the last axis of each input, but keeps the remaining indices intact. For example, where a is of shape (3,5,N) and b is of shape (5,N), this will return an output of shape (3,5). The underlying elementary function is called 3*5 times. In the signature, we specify one core dimension "(i)" for each input and zero core dimensions "()" for the output, since it takes two 1-d arrays and returns a scalar. By using the same name "i", we specify that the two corresponding dimensions should be of the same size (or one of them is of size 1 and will be broadcasted). The dimensions beyond the core dimensions are called "loop" dimensions. In the above example, this corresponds to (3,5). The usual numpy "broadcasting" rules apply, where the signature determines how the dimensions of each input/output object are split into core and loop dimensions: While an input array has a smaller dimensionality than the corresponding number of core dimensions, 1's are pre-pended to its shape. The core dimensions are removed from all inputs and the remaining dimensions are broadcasted; defining the loop dimensions. The output is given by the loop dimensions plus the output core dimensions. Experimental Windows 64 bits support ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Numpy can now be built on windows 64 bits (amd64 only, not IA64), with both MS compilers and mingw-w64 compilers: This is *highly experimental*: DO NOT USE FOR PRODUCTION USE. See INSTALL.txt, Windows 64 bits section for more information on limitations and how to build it by yourself. New features ============ Formatting issues ~~~~~~~~~~~~~~~~~ Float formatting is now handled by numpy instead of the C runtime: this enables locale independent formatting, more robust fromstring and related methods. Special values (inf and nan) are also more consistent across platforms (nan vs IND/NaN, etc...), and more consistent with recent python formatting work (in 2.6 and later). Nan handling in max/min ~~~~~~~~~~~~~~~~~~~~~~~ The maximum/minimum ufuncs now reliably propagate nans. If one of the arguments is a nan, then nan is retured. This affects np.min/np.max, amin/amax and the array methods max/min. New ufuncs fmax and fmin have been added to deal with non-propagating nans. Nan handling in sign ~~~~~~~~~~~~~~~~~~~~ The ufunc sign now returns nan for the sign of anan. New ufuncs ~~~~~~~~~~ #. fmax - same as maximum for integer types and non-nan floats. Returns the non-nan argument if one argument is nan and returns nan if both arguments are nan. #. fmin - same as minimum for integer types and non-nan floats. Returns the non-nan argument if one argument is nan and returns nan if both arguments are nan. #. deg2rad - converts degrees to radians, same as the radians ufunc. #. rad2deg - converts radians to degrees, same as the degrees ufunc. #. log2 - base 2 logarithm. #. exp2 - base 2 exponential. #. trunc - truncate floats to nearest integer towards zero. #. logaddexp - add numbers stored as logarithms and return the logarithm of the result. #. logaddexp2 - add numbers stored as base 2 logarithms and return the base 2 logarithm of the result result. Masked arrays ~~~~~~~~~~~~~ Several new features and bug fixes, including: * structured arrays should now be fully supported by MaskedArray (r6463, r6324, r6305, r6300, r6294...) * Minor bug fixes (r6356, r6352, r6335, r6299, r6298) * Improved support for __iter__ (r6326) * made baseclass, sharedmask and hardmask accesible to the user (but read-only) * doc update gfortran support on windows ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Gfortran can now be used as a fortran compiler for numpy on windows, even when the C compiler is Visual Studio (VS 2005 and above; VS 2003 will NOT work). Gfortran + Visual studio does not work on windows 64 bits (but gcc + gfortran does). It is unclear whether it will be possible to use gfortran and visual studio at all on x64. Arch option for windows binary ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Automatic arch detection can now be bypassed from the command line for the superpack installed: numpy-1.3.0-superpack-win32.exe /arch=nosse will install a numpy which works on any x86, even if the running computer supports SSE set. Deprecated features =================== Histogram ~~~~~~~~~ The semantics of histogram has been modified to fix long-standing issues with outliers handling. The main changes concern #. the definition of the bin edges, now including the rightmost edge, and #. the handling of upper outliers, now ignored rather than tallied in the rightmost bin. The previous behavior is still accessible using `new=False`, but this is deprecated, and will be removed entirely in 1.4.0. Documentation changes ===================== A lot of documentation has been added. Both user guide and references can be built from sphinx. New C API ========= Multiarray API ~~~~~~~~~~~~~~ The following functions have been added to the multiarray C API: * PyArray_GetEndianness: to get runtime endianness Ufunc API ~~~~~~~~~~~~~~ The following functions have been added to the ufunc API: * PyUFunc_FromFuncAndDataAndSignature: to declare a more general ufunc (generalized ufunc). New defines ~~~~~~~~~~~ New public C defines are available for ARCH specific code through numpy/npy_cpu.h: * NPY_CPU_X86: x86 arch (32 bits) * NPY_CPU_AMD64: amd64 arch (x86_64, NOT Itanium) * NPY_CPU_PPC: 32 bits ppc * NPY_CPU_PPC64: 64 bits ppc * NPY_CPU_SPARC: 32 bits sparc * NPY_CPU_SPARC64: 64 bits sparc * NPY_CPU_S390: S390 * NPY_CPU_IA64: ia64 * NPY_CPU_PARISC: PARISC New macros for CPU endianness has been added as well (see internal changes below for details): * NPY_BYTE_ORDER: integer * NPY_LITTLE_ENDIAN/NPY_BIG_ENDIAN defines Those provide portable alternatives to glibc endian.h macros for platforms without it. Portable NAN, INFINITY, etc... ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ npy_math.h now makes available several portable macro to get NAN, INFINITY: * NPY_NAN: equivalent to NAN, which is a GNU extension * NPY_INFINITY: equivalent to C99 INFINITY * NPY_PZERO, NPY_NZERO: positive and negative zero respectively Corresponding single and extended precision macros are available as well. All references to NAN, or home-grown computation of NAN on the fly have been removed for consistency. Internal changes ================ numpy.core math configuration revamp ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This should make the porting to new platforms easier, and more robust. In particular, the configuration stage does not need to execute any code on the target platform, which is a first step toward cross-compilation. http://projects.scipy.org/numpy/browser/trunk/doc/neps/math_config_clean.txt umath refactor ~~~~~~~~~~~~~~ A lot of code cleanup for umath/ufunc code (charris). Improvements to build warnings ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Numpy can now build with -W -Wall without warnings http://projects.scipy.org/numpy/browser/trunk/doc/neps/warnfix.txt Separate core math library ~~~~~~~~~~~~~~~~~~~~~~~~~~ The core math functions (sin, cos, etc... for basic C types) have been put into a separate library; it acts as a compatibility layer, to support most C99 maths functions (real only for now). The library includes platform-specific fixes for various maths functions, such as using those versions should be more robust than using your platform functions directly. The API for existing functions is exactly the same as the C99 math functions API; the only difference is the npy prefix (npy_cos vs cos). The core library will be made available to any extension in 1.4.0. CPU arch detection ~~~~~~~~~~~~~~~~~~ npy_cpu.h defines numpy specific CPU defines, such as NPY_CPU_X86, etc... Those are portable across OS and toolchains, and set up when the header is parsed, so that they can be safely used even in the case of cross-compilation (the values is not set when numpy is built), or for multi-arch binaries (e.g. fat binaries on Max OS X). npy_endian.h defines numpy specific endianness defines, modeled on the glibc endian.h. NPY_BYTE_ORDER is equivalent to BYTE_ORDER, and one of NPY_LITTLE_ENDIAN or NPY_BIG_ENDIAN is defined. As for CPU archs, those are set when the header is parsed by the compiler, and as such can be used for cross-compilation and multi-arch binaries. Checksums ========= e8d2b1f0d30416ee72bc29e3b6762fef release/installers/numpy-1.3.0-win32-superpack-python2.5.exe 3f7773ff0971a5ebb8591536d8ec7bd6 release/installers/numpy-1.3.0.tar.gz 18a7352f6ed8e6507f7cba1e56b4adc8 release/installers/numpy-1.3.0.zip 82e1cc25d945652c8408a48e00ba6726 release/installers/numpy-1.3.0-win32-superpack-python2.6.exe 0504197b83c7ed442aa684556bd0c519 release/installers/numpy-1.3.0-py2.5-macosx10.5.dmg 1a8089e34d7435d2b43bdda4f0a85443 release/installers/numpy-1.3.0.win-amd64-py2.6.msi From david at ar.media.kyoto-u.ac.jp Sun Apr 5 06:06:32 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sun, 05 Apr 2009 19:06:32 +0900 Subject: [Numpy-discussion] Registering numpy on Pypi Message-ID: <49D882A8.6050003@ar.media.kyoto-u.ac.jp> Hi, Just to mention that I could not register the 1.3.0 release on pypi, as I don't have rights to the numpy project. The release is tagged, so it is just a matter of running python setup.py register for anyone with corresponding privileges on pypi's numpy project, cheers, David From charlesr.harris at gmail.com Sun Apr 5 11:42:55 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 5 Apr 2009 09:42:55 -0600 Subject: [Numpy-discussion] [ANN] Numpy 1.3.0 released In-Reply-To: <49D88237.7050106@ar.media.kyoto-u.ac.jp> References: <49D88237.7050106@ar.media.kyoto-u.ac.jp> Message-ID: On Sun, Apr 5, 2009 at 4:04 AM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Hi, > > I am pleased to announce the release of numpy 1.3.0. You can find source > tarballs and binary installers for > both Mac OS X and Windows on the sourceforge page: > Good job! Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From timmichelsen at gmx-topmail.de Sun Apr 5 14:33:07 2009 From: timmichelsen at gmx-topmail.de (Tim Michelsen) Date: Sun, 05 Apr 2009 20:33:07 +0200 Subject: [Numpy-discussion] Registering numpy on Pypi In-Reply-To: <49D882A8.6050003@ar.media.kyoto-u.ac.jp> References: <49D882A8.6050003@ar.media.kyoto-u.ac.jp> Message-ID: to mention that I could not register the 1.3.0 release on pypi, > as I don't have rights to the numpy project. The release is tagged, so > it is just a matter of running python setup.py register for anyone with > corresponding privileges on pypi's numpy project, Sorry, I should have read here first before posting my message Re: [ANN] Numpy 1.3.0 released http://permalink.gmane.org/gmane.comp.python.scientific.user/20199 To scipy-user list. From pgmdevlist at gmail.com Sun Apr 5 15:40:25 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Sun, 5 Apr 2009 15:40:25 -0400 Subject: [Numpy-discussion] Construct symmetric matrix In-Reply-To: <5b8d13220904042118i673527dsf2b0ac83a9b542d9@mail.gmail.com> References: <46FED3E5-4EFE-4EAE-B75A-DF01A80BE2CF@gmail.com> <5b8d13220904042118i673527dsf2b0ac83a9b542d9@mail.gmail.com> Message-ID: <6AEC0C05-007D-49FB-9119-CA1E3764C270@gmail.com> On Apr 5, 2009, at 12:18 AM, David Cournapeau wrote: > On Sun, Apr 5, 2009 at 11:17 AM, Pierre GM > wrote: >> All, >> I'm trying to build a relatively complicated symmetric matrix. I can >> build the upper-right block without pb. What is the fastest way to >> get >> the LL corner from the UR one ? >> Thanks a lot in advance for any idea. > > I may be missing something, but why not building it from scratch + a > couple of elementary operations ? Is it too slow ? For example: > > # Assuming the lower part of a is zero before the operation > a + a.T - np.diag(a.diagonal()) Sweet! Works pretty well indeed. I didn't think about that (that was a bit late on a Sat. night for numpy). Thx again. From ckkart at hoc.net Mon Apr 6 09:01:51 2009 From: ckkart at hoc.net (Christian K.) Date: Mon, 6 Apr 2009 13:01:51 +0000 (UTC) Subject: [Numpy-discussion] replace voids in 2d dat with nearest neighbour value Message-ID: Hi, I am looking for an elegant and fast way to fill the voids of a 2d array with neighbouring values. The array's size can be up to (1000, 1000) and its values are slowly varying around a mean value. What I call voids are values which are far from the mean value (+- 80%). A void usually extends over some adjacent coordinates. Currently I am using tmp = N.ma.array(tmp, tmp Hi David, Thanks again for bundling in the architecture-specification flag into the numpy superpack installers: being able to choose sse vs. nosse installs is really helpful to me, and from what I hear, many others as well! Anyhow, I just noticed (sorry I didn't see this before the release) a minor erratum in the release note wrt this feature, which reads: > Automatic arch detection can now be bypassed from the command line > for the superpack installed: > > numpy-1.3.0-superpack-win32.exe /arch=nosse > > will install a numpy which works on any x86, even if the running > computer > supports SSE set. > However, the actual command-line should be: numpy-1.3.0-superpack-win32.exe /arch nosse Probably anyone advanced enough to use this feature would figure that out, but no harm being clear... Thanks, Zach From zachary.pincus at yale.edu Mon Apr 6 09:24:18 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Mon, 6 Apr 2009 09:24:18 -0400 Subject: [Numpy-discussion] replace voids in 2d dat with nearest neighbour value In-Reply-To: References: Message-ID: Hi Christian, Check out this discussion from a little while ago on a very similar issue (but in 3d): http://www.nabble.com/Help-with-interpolating-missing-values-from-a-3D-scanner-td21489413.html Most of the suggestions should be directly applicable. Zach On Apr 6, 2009, at 9:01 AM, Christian K. wrote: > Hi, > > I am looking for an elegant and fast way to fill the voids of a 2d > array with > neighbouring values. The array's size can be up to (1000, 1000) and > its values > are slowly varying around a mean value. What I call voids are values > which are > far from the mean value (+- 80%). A void usually extends over some > adjacent > coordinates. > > Currently I am using > > tmp = N.ma.array(tmp, tmp data[tmp.mask] = tmp.mean() > > which moves the voids closer to the mean value but which is still > far from > beeing a smooth interpolation. > > Regards, Christian > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From Jim.Vickroy at noaa.gov Mon Apr 6 12:42:57 2009 From: Jim.Vickroy at noaa.gov (Jim Vickroy) Date: Mon, 06 Apr 2009 10:42:57 -0600 Subject: [Numpy-discussion] replace voids in 2d dat with nearest neighbour value In-Reply-To: References: Message-ID: <49DA3111.3000709@noaa.gov> Zachary Pincus wrote: > Hi Christian, > > Check out this discussion from a little while ago on a very similar > issue (but in 3d): > http://www.nabble.com/Help-with-interpolating-missing-values-from-a-3D-scanner-td21489413.html > > Most of the suggestions should be directly applicable. > > Zach > Hi Christian, I'm in the early stages of testing the scipy.interpolate.Rbf(...) function (one of the approaches Robert Kern suggested in the discussion Zach mentioned) for a not too dissimilar application -- replacing the bad pixels regions in Solar X-ray images captured by a damaged detector. Preliminary results look promising for this application. -- jv > > On Apr 6, 2009, at 9:01 AM, Christian K. wrote: > > >> Hi, >> >> I am looking for an elegant and fast way to fill the voids of a 2d >> array with >> neighbouring values. The array's size can be up to (1000, 1000) and >> its values >> are slowly varying around a mean value. What I call voids are values >> which are >> far from the mean value (+- 80%). A void usually extends over some >> adjacent >> coordinates. >> >> Currently I am using >> >> tmp = N.ma.array(tmp, tmp> data[tmp.mask] = tmp.mean() >> >> which moves the voids closer to the mean value but which is still >> far from >> beeing a smooth interpolation. >> >> Regards, Christian >> >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ondrej at certik.cz Mon Apr 6 16:00:42 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 6 Apr 2009 13:00:42 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> References: <49CE3A2C.9000007@enthought.com> <5b8d13220903280852l17c578d7hadddbc40873f8c9f@mail.gmail.com> <49D136A3.80302@gmail.com> <5b8d13220903302041m71cea034pf21b4a61d39eb6f@mail.gmail.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> Message-ID: <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> On Mon, Mar 30, 2009 at 10:59 PM, David Cournapeau wrote: > Eric Firing wrote: >> >> I agree. ?The PEP does not show overwhelming superiority (or, arguably, >> even mild superiority) of any alternative > > I think this PEP was poorly written. You can't see any of the > advantage/differences of the different systems. Some people even said > they don't see the differences with svn. I think the reason partly is > that the PEP focused on existing python workflows, but the whole point, > at least for me, is to change the general workflow (for reviews, code > contributions, etc...). Stephen J. Turnbull sums it up nicely: > > http://mail.python.org/pipermail/python-dev/2009-March/087968.html > > FWIW, I tend to agree that Hg is less disruptive than git when coming > from svn, at least for the simple tasks (I don't know hg enough to have > a really informed opinion for more advanced workflows). Yes, git rocks. Ondrej From f.yw at hotmail.com Mon Apr 6 20:56:41 2009 From: f.yw at hotmail.com (frank wang) Date: Mon, 6 Apr 2009 18:56:41 -0600 Subject: [Numpy-discussion] replace voids in 2d dat with nearest neighbour value In-Reply-To: <49DA3111.3000709@noaa.gov> References: <49DA3111.3000709@noaa.gov> Message-ID: Hi, I have a big 2 column data file where the data are repeated either 5 or 6 times. Are there any quick way to remove the duplicated data? Thanks Frank _________________________________________________________________ Rediscover Hotmail?: Get e-mail storage that grows with you. http://windowslive.com/RediscoverHotmail?ocid=TXT_TAGLM_WL_HM_Rediscover_Storage1_042009 -------------- next part -------------- An HTML attachment was scrubbed... URL: From f.yw at hotmail.com Mon Apr 6 21:46:22 2009 From: f.yw at hotmail.com (frank wang) Date: Mon, 6 Apr 2009 19:46:22 -0600 Subject: [Numpy-discussion] how to delete the duplicated data in numpy? In-Reply-To: References: <49DA3111.3000709@noaa.gov> Message-ID: Hi, I have just noticed that I did not change the title of my question. So I resend it out. Sorry for the mistake. Here is my question. I have a big 2 column data file where the data are repeated either 5 or 6 times. Are there any quick way to remove the duplicated data? Thanks Frank Rediscover Hotmail?: Get e-mail storage that grows with you. Check it out. _________________________________________________________________ Windows Live?: Keep your life in sync. http://windowslive.com/explore?ocid=TXT_TAGLM_WL_allup_1a_explore_042009 -------------- next part -------------- An HTML attachment was scrubbed... URL: From nfaggian at unimelb.edu.au Mon Apr 6 21:53:16 2009 From: nfaggian at unimelb.edu.au (Nathan Faggian) Date: Tue, 7 Apr 2009 11:53:16 +1000 Subject: [Numpy-discussion] Construct a large n-dimensional complex matrix In-Reply-To: <5b8d13220904042118i673527dsf2b0ac83a9b542d9@mail.gmail.com> References: <46FED3E5-4EFE-4EAE-B75A-DF01A80BE2CF@gmail.com> <5b8d13220904042118i673527dsf2b0ac83a9b542d9@mail.gmail.com> Message-ID: HI, I want to construct a large complex matrix, I have the real and imaginary components as double vectors, is there a fast way to construct a complex vector in numpy? Cheers, Nathan. ---------------------------------------------------------------------------- Nathan Faggian, Ph.D. The Centre for Neuroscience / Howard Florey Institute The University of Melbourne Phone : +61 3 8344 1836 Email ; nfaggian at unimelb.edu.au ---------------------------------------------------------------------------- On 05/04/2009, at 2:18 PM, David Cournapeau wrote: > On Sun, Apr 5, 2009 at 11:17 AM, Pierre GM > wrote: >> All, >> I'm trying to build a relatively complicated symmetric matrix. I can >> build the upper-right block without pb. What is the fastest way to >> get >> the LL corner from the UR one ? >> Thanks a lot in advance for any idea. > > I may be missing something, but why not building it from scratch + a > couple of elementary operations ? Is it too slow ? For example: > > # Assuming the lower part of a is zero before the operation > a + a.T - np.diag(a.diagonal()) > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Apr 6 22:20:19 2009 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 6 Apr 2009 21:20:19 -0500 Subject: [Numpy-discussion] Construct a large n-dimensional complex matrix In-Reply-To: References: <46FED3E5-4EFE-4EAE-B75A-DF01A80BE2CF@gmail.com> <5b8d13220904042118i673527dsf2b0ac83a9b542d9@mail.gmail.com> Message-ID: <3d375d730904061920h49b19a5ale6fd482ee2eadc35@mail.gmail.com> On Mon, Apr 6, 2009 at 20:53, Nathan Faggian wrote: > HI, > I want to construct a large complex matrix, I have the real and imaginary > components as double vectors, is there a fast way to construct a complex > vector in numpy? C = np.empty((n,m), dtype=complex) C.real.flat[:] = real_values C.imag.flat[:] = imag_values -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From josef.pktd at gmail.com Mon Apr 6 23:05:14 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 6 Apr 2009 23:05:14 -0400 Subject: [Numpy-discussion] how to delete the duplicated data in numpy? In-Reply-To: References: <49DA3111.3000709@noaa.gov> Message-ID: <1cd32cbb0904062005q72f3b26dl29292aec0309d69c@mail.gmail.com> On Mon, Apr 6, 2009 at 9:46 PM, frank wang wrote: > > > Hi, > > I have just noticed that I did not change the title of my question. So I > resend it out. Sorry for the mistake. Here is my question. > > I have a big 2 column data file where the data are repeated either 5 or 6 > times. Are there any quick way to remove the duplicated data? > > Thanks > I'm still trying to figure out how to work with rows. The following is based on an answer that I got when I asked for sortrows. However, the last reshape shouldn't be necessary or I don't know why it is. See if the following works for you >>> xx2 array([[ 0., 1.], [ 0., 1.], [ 0., 1.], [ 1., 1.], [ 1., 1.], [ 1., 1.], [ 2., 1.], [ 2., 1.], [ 2., 1.]]) >>> xv = xx2.view([('',xx2.dtype)]*xx2.shape[1]) >>> xu = np.unique(xv) >>> xu.view('f8') array([ 0., 1., 1., 1., 2., 1.]) >>> xu.view('f8').reshape(-1,2) array([[ 0., 1.], [ 1., 1.], [ 2., 1.]]) or in one line >>> np.unique(xx2.view([('',xx2.dtype)]*xx2.shape[1])).view('f8').reshape(-1,xx2.shape[1]) array([[ 0., 1.], [ 1., 1.], [ 2., 1.]]) The idea is the the structured array keeps the rows together and it can be fed to unique, which works on 1d, but I have no idea about the internal structure, but it seems to work. Josef From josef.pktd at gmail.com Mon Apr 6 23:31:03 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 6 Apr 2009 23:31:03 -0400 Subject: [Numpy-discussion] views or copy Message-ID: <1cd32cbb0904062031kbb660f5n3b0953c1a2927746@mail.gmail.com> I ran again into a problem where numpy created a view (which I didn't realize) and an operation works differently on the view than if it were a copy. I try to construct an example array, which, however, is only a view >>> x,y = np.mgrid[0:3,0:3] >>> xx = np.vstack((x.flatten(), y.flatten(), np.ones(9))).T >>> xx array([[ 0., 0., 1.], [ 0., 1., 1.], [ 0., 2., 1.], [ 1., 0., 1.], [ 1., 1., 1.], [ 1., 2., 1.], [ 2., 0., 1.], [ 2., 1., 1.], [ 2., 2., 1.]]) >>> xx.flags C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False >>> xx.base array([[ 0., 0., 0., 1., 1., 1., 2., 2., 2.], [ 0., 1., 2., 0., 1., 2., 0., 1., 2.], [ 1., 1., 1., 1., 1., 1., 1., 1., 1.]]) >>> xx == xx.base False When I convert it to a view as a structured array, it produces a strange result. I didn't get what I thought I should get >>> xx.view([('',xx.dtype)]*xx.shape[1]) array([[(0.0, 0.0, 0.0), (0.0, 1.0, 2.0), (1.0, 1.0, 1.0)], [(1.0, 1.0, 1.0), (0.0, 1.0, 2.0), (1.0, 1.0, 1.0)], [(2.0, 2.0, 2.0), (0.0, 1.0, 2.0), (1.0, 1.0, 1.0)]], dtype=[('f0', '>> xx2 = xx.copy() >>> xx2.view([('',xx2.dtype)]*xx2.shape[1]) array([[(0.0, 0.0, 1.0)], [(0.0, 1.0, 1.0)], [(0.0, 2.0, 1.0)], [(1.0, 0.0, 1.0)], [(1.0, 1.0, 1.0)], [(1.0, 2.0, 1.0)], [(2.0, 0.0, 1.0)], [(2.0, 1.0, 1.0)], [(2.0, 2.0, 1.0)]], dtype=[('f0', ' References: <1cd32cbb0904062031kbb660f5n3b0953c1a2927746@mail.gmail.com> Message-ID: <3d375d730904062034x7cc4f0can6e1fb3d2d7d085a2@mail.gmail.com> On Mon, Apr 6, 2009 at 22:31, wrote: > I ran again into a problem where numpy created a view (which I didn't > realize) and an operation works differently on the view than if it > were a copy. > > I try to construct an example array, which, however, is only a view > >>>> x,y = np.mgrid[0:3,0:3] >>>> xx = np.vstack((x.flatten(), y.flatten(), np.ones(9))).T >>>> xx > array([[ 0., ?0., ?1.], > ? ? ? [ 0., ?1., ?1.], > ? ? ? [ 0., ?2., ?1.], > ? ? ? [ 1., ?0., ?1.], > ? ? ? [ 1., ?1., ?1.], > ? ? ? [ 1., ?2., ?1.], > ? ? ? [ 2., ?0., ?1.], > ? ? ? [ 2., ?1., ?1.], > ? ? ? [ 2., ?2., ?1.]]) >>>> xx.flags > ?C_CONTIGUOUS : False > ?F_CONTIGUOUS : True > ?OWNDATA : False > ?WRITEABLE : True > ?ALIGNED : True > ?UPDATEIFCOPY : False >>>> xx.base > array([[ 0., ?0., ?0., ?1., ?1., ?1., ?2., ?2., ?2.], > ? ? ? [ 0., ?1., ?2., ?0., ?1., ?2., ?0., ?1., ?2.], > ? ? ? [ 1., ?1., ?1., ?1., ?1., ?1., ?1., ?1., ?1.]]) >>>> xx == xx.base > False > > When I convert it to a view as a structured array, it produces a > strange result. I didn't get what I thought I should get > >>>> xx.view([('',xx.dtype)]*xx.shape[1]) > array([[(0.0, 0.0, 0.0), (0.0, 1.0, 2.0), (1.0, 1.0, 1.0)], > ? ? ? [(1.0, 1.0, 1.0), (0.0, 1.0, 2.0), (1.0, 1.0, 1.0)], > ? ? ? [(2.0, 2.0, 2.0), (0.0, 1.0, 2.0), (1.0, 1.0, 1.0)]], > ? ? ?dtype=[('f0', ' > if I make a copy and then construct a view as structured array, I get > what I want > >>>> xx2 = xx.copy() >>>> xx2.view([('',xx2.dtype)]*xx2.shape[1]) > array([[(0.0, 0.0, 1.0)], > ? ? ? [(0.0, 1.0, 1.0)], > ? ? ? [(0.0, 2.0, 1.0)], > ? ? ? [(1.0, 0.0, 1.0)], > ? ? ? [(1.0, 1.0, 1.0)], > ? ? ? [(1.0, 2.0, 1.0)], > ? ? ? [(2.0, 0.0, 1.0)], > ? ? ? [(2.0, 1.0, 1.0)], > ? ? ? [(2.0, 2.0, 1.0)]], > ? ? ?dtype=[('f0', ' > Are there rules for this behavior or a description in the docs, > because my mistakes in this are quite difficult to debug? Or do I have > to make a copy by default as in matlab? .view() makes a view onto the memory, not onto the memory-as-if-it-were-normalized-to-C-contiguous-order. Use ascontiguous() if you need to ensure the latter. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stefan at sun.ac.za Tue Apr 7 03:32:13 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 7 Apr 2009 09:32:13 +0200 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <5b8d13220903280852l17c578d7hadddbc40873f8c9f@mail.gmail.com> <49D136A3.80302@gmail.com> <5b8d13220903302041m71cea034pf21b4a61d39eb6f@mail.gmail.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> Message-ID: <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> 2009/4/6 Ondrej Certik : >> FWIW, I tend to agree that Hg is less disruptive than git when coming >> from svn, at least for the simple tasks (I don't know hg enough to have >> a really informed opinion for more advanced workflows). > > Yes, git rocks. Did Sympy move from Hg to Git? If so, could you give us an overview of the pros and cons you experienced? Cheers St?fan From Tobias.Lindberg at byggvetenskaper.lth.se Tue Apr 7 10:05:52 2009 From: Tobias.Lindberg at byggvetenskaper.lth.se (Tobias Lindberg) Date: Tue, 7 Apr 2009 16:05:52 +0200 Subject: [Numpy-discussion] f2py-problem fcompiler for f90-files missing In-Reply-To: References: Message-ID: Hi, I've tried to find anything in the archives to help me solve this problem but nothing of the suggested things seems to work. Background: I have installed the Python(xy) package (full) (numpy 1.2.1.2) both on my XP and Vista machine. On both these machines I have VS2008 installed as well. So the first problem I got was this; No module named msvccompiler in numpy.distutils; trying from distutils error: Python was built with Visual Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py. Then I read that one could write like this; f2py -m test -c test.f90 --compiler=mingw32 Then I got this error; error: f90 not supported by GnuFCompiler needed for fort_mod.f90 so then I check which fcompilers there were and got this; Fortran compilers found: --fcompiler=gnu GNU Fortran 77 compiler (3.4.5) Compilers available for this platform, but not found: --fcompiler=absoft Absoft Corp Fortran Compiler --fcompiler=compaqv DIGITAL or Compaq Visual Fortran Compiler --fcompiler=g95 G95 Fortran Compiler --fcompiler=gnu95 GNU Fortran 95 compiler --fcompiler=intelev Intel Visual Fortran Compiler for Itanium apps --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps and the thing is that I have this g95.py under c:\Python25\Lib\site-packages\numpy\distutils\fcompiler\ so why does not this thing wotk? And more important, how can I make it work? I hope someone can help me with this. Best regards /tobias -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Apr 7 10:25:19 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 7 Apr 2009 10:25:19 -0400 Subject: [Numpy-discussion] f2py-problem fcompiler for f90-files missing In-Reply-To: References: Message-ID: <1cd32cbb0904070725p558bea87n534cecfd3c3b8f33@mail.gmail.com> On Tue, Apr 7, 2009 at 10:05 AM, Tobias Lindberg wrote: > Hi, > > I've tried to find anything in the archives to help me solve this problem > but nothing of the suggested things seems to work. > Background: > I have installed the Python(xy) package (full) (numpy 1.2.1.2) both on my XP > and Vista machine. On both these machines I have VS2008 installed as well. > So the first problem I got was this; > > No module named msvccompiler in numpy.distutils; trying from distutils > error: Python was built with Visual Studio 2003; > extensions must be built with a compiler than can generate compatible > binaries. > Visual Studio 2003 was not found on this system. If you have Cygwin > installed, > you can try compiling with MingW32, by passing "-c mingw32" to setup.py. > > Then I read that one could write like this; > > f2py -m test -c test.f90 --compiler=mingw32 > > Then I got this error; > > error: f90 not supported by GnuFCompiler needed for fort_mod.f90 > > so then I check which fcompilers there were and got this; > > Fortran compilers found: > ? --fcompiler=gnu? GNU Fortran 77 compiler (3.4.5) > Compilers available for this platform, but not found: > ? --fcompiler=absoft?? Absoft Corp Fortran Compiler > ? --fcompiler=compaqv? DIGITAL or Compaq Visual Fortran Compiler > ? --fcompiler=g95????? G95 Fortran Compiler > ? --fcompiler=gnu95??? GNU Fortran 95 compiler > ? --fcompiler=intelev? Intel Visual Fortran Compiler for Itanium apps > ? --fcompiler=intelv?? Intel Visual Fortran Compiler for 32-bit apps > and the thing is that I have this g95.py under > c:\Python25\Lib\site-packages\numpy\distutils\fcompiler\ > > so why does? not this thing wotk? And more important, how can I make it > work? > > I hope someone can help me with this. > > Best regards > > /tobias I guess this means that in MingW you only have the fortran 77 compiler, and for f.90 you need a newer compiler (g95 or gfortran ?) Since I only use g77, I cannot help with the recommended update. Until I really need it, I stick with the official MingW release version. Josef From scott.sinclair.za at gmail.com Tue Apr 7 10:31:10 2009 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Tue, 7 Apr 2009 16:31:10 +0200 Subject: [Numpy-discussion] f2py-problem fcompiler for f90-files missing In-Reply-To: References: Message-ID: <6a17e9ee0904070731y4f35a481j8763672476f3632a@mail.gmail.com> > 2009/4/7 Tobias Lindberg : > Background: > I have installed the Python(xy) package (full) (numpy 1.2.1.2) both on my XP > and Vista machine. On both these machines I have VS2008 installed as well. > > Then I read that one could write like this; > > f2py -m test -c test.f90 --compiler=mingw32 > > Then I got this error; > > error: f90 not supported by GnuFCompiler needed for fort_mod.f90 > > so then I check which fcompilers there were and got this; > > Fortran compilers found: > ? --fcompiler=gnu? GNU Fortran 77 compiler (3.4.5) > Compilers available for this platform, but not found: > ? --fcompiler=absoft?? Absoft Corp Fortran Compiler > ? --fcompiler=compaqv? DIGITAL or Compaq Visual Fortran Compiler > ? --fcompiler=g95????? G95 Fortran Compiler > ? --fcompiler=gnu95??? GNU Fortran 95 compiler > ? --fcompiler=intelev? Intel Visual Fortran Compiler for Itanium apps > ? --fcompiler=intelv?? Intel Visual Fortran Compiler for 32-bit apps > and the thing is that I have this g95.py under > c:\Python25\Lib\site-packages\numpy\distutils\fcompiler\ > > so why does? not this thing wotk? And more important, how can I make it > work? g95.py isn't a FORTRAN compiler, it's some code to help NumPy find out what compilers are available on your system. Since you seem to have g77 installed, I assume this is part of either a mingw or cygwin installation on your machine. I suggest using the relevant package manager (cygwin or mingw setup.exe or whatever) to find and install g95 or gfortran. You should then be able to compile FORTRAN90 source code. If you make sure that you can compile a simple FORTRAN90 program at the command line before trying f2py, you'll probably have better luck.. Cheers, Scott From Tobias.Lindberg at byggvetenskaper.lth.se Tue Apr 7 10:59:39 2009 From: Tobias.Lindberg at byggvetenskaper.lth.se (Tobias Lindberg) Date: Tue, 7 Apr 2009 16:59:39 +0200 Subject: [Numpy-discussion] f2py-problem fcompiler for f90-files missing In-Reply-To: <6a17e9ee0904070731y4f35a481j8763672476f3632a@mail.gmail.com> References: , <6a17e9ee0904070731y4f35a481j8763672476f3632a@mail.gmail.com> Message-ID: g95.py isn't a FORTRAN compiler, it's some code to help NumPy find out what compilers are available on your system. Since you seem to have g77 installed, I assume this is part of either a mingw or cygwin installation on your machine. I suggest using the relevant package manager (cygwin or mingw setup.exe or whatever) to find and install g95 or gfortran. You should then be able to compile FORTRAN90 source code. If you make sure that you can compile a simple FORTRAN90 program at the command line before trying f2py, you'll probably have better luck.. Cheers, Scott Hi Scott, Thanx for the info! No wonder it won't work if the g95-files isn't what I thought it was :-) Well, you learn something new every day! Your suggestion about using a relevant package manager seems like a good idea. I will give it a try and hope for the best! Thank you! Best regards /tobias From klaus.kopec at tuebingen.mpg.de Tue Apr 7 11:56:21 2009 From: klaus.kopec at tuebingen.mpg.de (Klaus Kopec) Date: Tue, 07 Apr 2009 17:56:21 +0200 Subject: [Numpy-discussion] shift for optimal superimposition of two 3D matrices according to correlation computed using FFT Message-ID: Hello everybody, I have two 3D density maps (meaning volumetric data, each one essentially a IxJxK matrix containing real values that sum to one) and want to find translation of between the two that maximises correlation. This can be done by computing the correlation between the two (correlation theorem -> I use FFT for this) and then shifting one of the two maps by the vector that corresponds to the index of the maximum in the 3D correlation matrix. My problem with this method is, however, that the shift wraps around the edges of the matrix. e.g: for a 30x30x30 matrix that needs to be slightly shifted to fit the seconds 3D matrix, sometimes shift vectors like (0, 29, 0) appear, that actually mean a shift by (0, -1, 0). This means that the size of the matrix needs to be subtracted from some components of the shift vector in special cases. So far, I have been unable to determine the cutoff value, outside of which I need to subtract the matrix size from a component of the shift vector, as this cutoff seems to vary between different data sets. Can anybody help me out with this? Best regards, Klaus From josef.pktd at gmail.com Tue Apr 7 12:07:36 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 7 Apr 2009 12:07:36 -0400 Subject: [Numpy-discussion] views or copy In-Reply-To: <3d375d730904062034x7cc4f0can6e1fb3d2d7d085a2@mail.gmail.com> References: <1cd32cbb0904062031kbb660f5n3b0953c1a2927746@mail.gmail.com> <3d375d730904062034x7cc4f0can6e1fb3d2d7d085a2@mail.gmail.com> Message-ID: <1cd32cbb0904070907s257126dbs881eabd7ccecff03@mail.gmail.com> On Mon, Apr 6, 2009 at 11:34 PM, Robert Kern wrote: > On Mon, Apr 6, 2009 at 22:31, ? wrote: >> I ran again into a problem where numpy created a view (which I didn't >> realize) and an operation works differently on the view than if it >> were a copy. >> >> I try to construct an example array, which, however, is only a view >> >>>>> x,y = np.mgrid[0:3,0:3] >>>>> xx = np.vstack((x.flatten(), y.flatten(), np.ones(9))).T >>>>> xx >> array([[ 0., ?0., ?1.], >> ? ? ? [ 0., ?1., ?1.], >> ? ? ? [ 0., ?2., ?1.], >> ? ? ? [ 1., ?0., ?1.], >> ? ? ? [ 1., ?1., ?1.], >> ? ? ? [ 1., ?2., ?1.], >> ? ? ? [ 2., ?0., ?1.], >> ? ? ? [ 2., ?1., ?1.], >> ? ? ? [ 2., ?2., ?1.]]) >>>>> xx.flags >> ?C_CONTIGUOUS : False >> ?F_CONTIGUOUS : True >> ?OWNDATA : False >> ?WRITEABLE : True >> ?ALIGNED : True >> ?UPDATEIFCOPY : False >>>>> xx.base >> array([[ 0., ?0., ?0., ?1., ?1., ?1., ?2., ?2., ?2.], >> ? ? ? [ 0., ?1., ?2., ?0., ?1., ?2., ?0., ?1., ?2.], >> ? ? ? [ 1., ?1., ?1., ?1., ?1., ?1., ?1., ?1., ?1.]]) >>>>> xx == xx.base >> False >> >> When I convert it to a view as a structured array, it produces a >> strange result. I didn't get what I thought I should get >> >>>>> xx.view([('',xx.dtype)]*xx.shape[1]) >> array([[(0.0, 0.0, 0.0), (0.0, 1.0, 2.0), (1.0, 1.0, 1.0)], >> ? ? ? [(1.0, 1.0, 1.0), (0.0, 1.0, 2.0), (1.0, 1.0, 1.0)], >> ? ? ? [(2.0, 2.0, 2.0), (0.0, 1.0, 2.0), (1.0, 1.0, 1.0)]], >> ? ? ?dtype=[('f0', '> >> if I make a copy and then construct a view as structured array, I get >> what I want >> >>>>> xx2 = xx.copy() >>>>> xx2.view([('',xx2.dtype)]*xx2.shape[1]) >> array([[(0.0, 0.0, 1.0)], >> ? ? ? [(0.0, 1.0, 1.0)], >> ? ? ? [(0.0, 2.0, 1.0)], >> ? ? ? [(1.0, 0.0, 1.0)], >> ? ? ? [(1.0, 1.0, 1.0)], >> ? ? ? [(1.0, 2.0, 1.0)], >> ? ? ? [(2.0, 0.0, 1.0)], >> ? ? ? [(2.0, 1.0, 1.0)], >> ? ? ? [(2.0, 2.0, 1.0)]], >> ? ? ?dtype=[('f0', '> >> Are there rules for this behavior or a description in the docs, >> because my mistakes in this are quite difficult to debug? Or do I have >> to make a copy by default as in matlab? > > .view() makes a view onto the memory, not onto the > memory-as-if-it-were-normalized-to-C-contiguous-order. Use > ascontiguous() if you need to ensure the latter. > I guess that's np.ascontiguousarray(xx) Thanks, I'm still trying to get the trickier parts of numpy. I was trying this out some more in order to understand the view examples. In my other examples, numpy raises an exception of the array is not c-contiguous when constructing a view, see below. Shouldn't numpy raise an exception also for my original example, since also there the array wasn't c-contiguous instead of silently using the array base? At least it would reduce debugging time, since I am not used to having to think about memory layout when I just do some array and matrix calculations and the arrays are not what they appear to be. (coming from other, more restrictive matrix languages like matlab or gauss). Related to my original problem, is there a way to convert between structured arrays and plain ndarrays (in both directions) that forces a copy instead of just creating a view? Josef Josef >>> import numpy as np >>> xx = np.array([[ 0., 0., 1.], ... [ 0., 1., 1.], ... [ 0., 2., 1.], ... [ 1., 0., 1.], ... [ 1., 1., 1.], ... [ 1., 2., 1.], ... [ 2., 0., 1.], ... [ 2., 1., 1.]]) >>> xx.flags C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False >>> xx.view([('',float),('',float),('',float)]) array([[(0.0, 0.0, 1.0)], [(0.0, 1.0, 1.0)], [(0.0, 2.0, 1.0)], [(1.0, 0.0, 1.0)], [(1.0, 1.0, 1.0)], [(1.0, 2.0, 1.0)], [(2.0, 0.0, 1.0)], [(2.0, 1.0, 1.0)]], dtype=[('f0', '>> xx[:6,:].flags C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False >>> xx[:6,:].view([('',float),('',float),('',float)]) array([[(0.0, 0.0, 1.0)], [(0.0, 1.0, 1.0)], [(0.0, 2.0, 1.0)], [(1.0, 0.0, 1.0)], [(1.0, 1.0, 1.0)], [(1.0, 2.0, 1.0)]], dtype=[('f0', '>> xx[:,:2].flags C_CONTIGUOUS : False F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False >>> xx[:,:2].view([('',float),('',float)]) Traceback (most recent call last): File "", line 1, in xx[:,:2].view([('',float),('',float)]) ValueError: new type not compatible with array. >>> xx.T[:,:2].view([('',float),('',float)]) Traceback (most recent call last): File "", line 1, in xx.T[:,:2].view([('',float),('',float)]) ValueError: new type not compatible with array. >>> xx.T[:,:2].flags C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False the following also produces a view on the base, not the one that I expected, and no exception is raised xx.reshape(-1,2,order='F').view([('',float),('',float)]) From zachary.pincus at yale.edu Tue Apr 7 12:24:40 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Tue, 7 Apr 2009 12:24:40 -0400 Subject: [Numpy-discussion] shift for optimal superimposition of two 3D matrices according to correlation computed using FFT In-Reply-To: References: Message-ID: > I have two 3D density maps (meaning volumetric data, each one > essentially a IxJxK matrix containing real values that sum to one) and > want to find translation of between the two that maximises > correlation. > This can be done by computing the correlation between the two > (correlation theorem -> I use FFT for this) and then shifting one of > the > two maps by the vector that corresponds to the index of the maximum in > the 3D correlation matrix. > > My problem with this method is, however, that the shift wraps around > the > edges of the matrix. e.g: for a 30x30x30 matrix that needs to be > slightly shifted to fit the seconds 3D matrix, sometimes shift vectors > like (0, 29, 0) appear, that actually mean a shift by (0, -1, 0). This > means that the size of the matrix needs to be subtracted from some > components of the shift vector in special cases. > > So far, I have been unable to determine the cutoff value, outside of > which I need to subtract the matrix size from a component of the shift > vector, as this cutoff seems to vary between different data sets. Does it work to use a cutoff of half the size of the input arrays in each dimension? This is equivalent to calculating both shifts (the positive and negative) and using whichever has a smaller absolute value. Alternately, you could use numpy.roll to shift the data (one axis at a time). Since roll wraps around, you wouldn't need to bother figuring out which shift is "correct". Finally, you could not use FFTs but instead directly optimize a transformation between the two, using scipy.ndimage's affine transforms and scipy.optimize's numerical optimizers. Zach From ondrej at certik.cz Tue Apr 7 14:34:14 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Tue, 7 Apr 2009 11:34:14 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <5b8d13220903280852l17c578d7hadddbc40873f8c9f@mail.gmail.com> <49D136A3.80302@gmail.com> <5b8d13220903302041m71cea034pf21b4a61d39eb6f@mail.gmail.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> Message-ID: <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> 2009/4/7 St?fan van der Walt : > 2009/4/6 Ondrej Certik : >>> FWIW, I tend to agree that Hg is less disruptive than git when coming >>> from svn, at least for the simple tasks (I don't know hg enough to have >>> a really informed opinion for more advanced workflows). >> >> Yes, git rocks. > > Did Sympy move from Hg to Git? Yes. >?If so, could you give us an overview > of the pros and cons you experienced? Here are some my (subjective) observations: 1) I think all developers were able to learn git pretty easily. See also: http://wiki.sympy.org/wiki/Git_hg_rosetta_stone 2) git is faster 3) it boosted my productivity a lot, thanks to branches. Previously I was copying the whole directories with sympy, everytime someone sended a patch, just to try it. I know that hg has branches too, but imho working with them is a pain, as you can't rebase patches easily, no interactive rebase etc and also branches were in git from the beginning, so it's very polished. Hg is catching up I guess, but some things are still missing. 4) no need to learn mercurial queues, just learn git and git rebase -i, and that's all I ever need (and it's actually more powerful than MQ). So git is simpler. Ondrej From dlenski at gmail.com Tue Apr 7 14:44:58 2009 From: dlenski at gmail.com (Dan Lenski) Date: Tue, 7 Apr 2009 18:44:58 +0000 (UTC) Subject: [Numpy-discussion] using reducing functions without eliminating dimensions? Message-ID: Hi all, I often want to use some kind of dimension-reducing function (like min(), max(), sum(), mean()) on an array without actually removing the last dimension, so that I can then do operations broadcasting the reduced array back to the size of the full array. Full example: >> table.shape (47, 1814) >> table.min(axis=1).shape (47,) >> table - table.min(axis=1) ValueError: shape mismatch: objects cannot be broadcast to a single shape >> table - table.min(axis=1)[:, newaxis] I have to resort to ugly code with lots of stuff like "... axis=1)[:, newaxis]". Is there any way to get the reducing functions to leave a size-1 dummy dimension in place, to make this easier? Thanks! Dan From nathanielpeterson08 at gmail.com Tue Apr 7 15:19:24 2009 From: nathanielpeterson08 at gmail.com (Nathaniel Peterson) Date: Tue, 7 Apr 2009 15:19:24 -0400 Subject: [Numpy-discussion] Can numpy catch this error for me? Message-ID: <41d6e77b0904071219w4ab87400k519e842ad626db08@mail.gmail.com> import numpy as np import operator np.seterr(all='raise') a=np.arange(1)+1 print(a.dtype) # int32 for num in range(1,17): a=np.arange(num)+1 b=np.multiply.reduce(a) print('%s! = %s'%(num,b)) # c=reduce(operator.mul,range(1,num+1)) # assert(b==c) The code above outputs int32 1! = 1 2! = 2 3! = 6 4! = 24 5! = 120 6! = 720 7! = 5040 8! = 40320 9! = 362880 10! = 3628800 11! = 39916800 12! = 479001600 13! = 1932053504 14! = 1278945280 15! = 2004310016 16! = 2004189184 The results for 14! and above are wrong due to overflow of the int32 data type. Is there a way to setup numpy so it will raise an error when this occurs? -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Apr 7 16:03:45 2009 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 7 Apr 2009 15:03:45 -0500 Subject: [Numpy-discussion] Can numpy catch this error for me? In-Reply-To: <41d6e77b0904071219w4ab87400k519e842ad626db08@mail.gmail.com> References: <41d6e77b0904071219w4ab87400k519e842ad626db08@mail.gmail.com> Message-ID: <3d375d730904071303q6f7467ecwb5c93c696b5d7116@mail.gmail.com> On Tue, Apr 7, 2009 at 14:19, Nathaniel Peterson wrote: > > import numpy as np > import operator > np.seterr(all='raise') > a=np.arange(1)+1 > print(a.dtype) > # int32 > for num in range(1,17): > ??? a=np.arange(num)+1 > ??? b=np.multiply.reduce(a) > ??? print('%s! = %s'%(num,b)) > #???? c=reduce(operator.mul,range(1,num+1)) > #???? assert(b==c) > > The code above outputs > > int32 > 1! = 1 > 2! = 2 > 3! = 6 > 4! = 24 > 5! = 120 > 6! = 720 > 7! = 5040 > 8! = 40320 > 9! = 362880 > 10! = 3628800 > 11! = 39916800 > 12! = 479001600 > 13! = 1932053504 > 14! = 1278945280 > 15! = 2004310016 > 16! = 2004189184 > > The results for 14! and above are wrong due to overflow of the int32 data > type. > Is there a way to setup numpy so it will raise an error when this occurs? No, sorry. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From nathanielpeterson08 at gmail.com Tue Apr 7 16:27:05 2009 From: nathanielpeterson08 at gmail.com (Nathaniel Peterson) Date: Tue, 7 Apr 2009 16:27:05 -0400 Subject: [Numpy-discussion] Can numpy catch this error for me? In-Reply-To: <3d375d730904071303q6f7467ecwb5c93c696b5d7116@mail.gmail.com> References: <41d6e77b0904071219w4ab87400k519e842ad626db08@mail.gmail.com> <3d375d730904071303q6f7467ecwb5c93c696b5d7116@mail.gmail.com> Message-ID: <41d6e77b0904071327u7e81bb0fsbe3b0e20aef00966@mail.gmail.com> Thanks for the quick response. In http://www.scipy.org/Cookbook/Indexing I see >>> a = C[1,2,3] >>> a 23 >>> type(a) >>> type(int(a)) >>> a**a Warning: overflow encountered in long_scalars -1276351769 >>> int(a)**int(a) 20880467999847912034355032910567L This shows numpy can catch an overflow generated by a**a. Why is it that numpy can catch this overflow, but can not catch one generated by np.multiply.reduce? -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Apr 7 16:58:08 2009 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 7 Apr 2009 15:58:08 -0500 Subject: [Numpy-discussion] Can numpy catch this error for me? In-Reply-To: <41d6e77b0904071327u7e81bb0fsbe3b0e20aef00966@mail.gmail.com> References: <41d6e77b0904071219w4ab87400k519e842ad626db08@mail.gmail.com> <3d375d730904071303q6f7467ecwb5c93c696b5d7116@mail.gmail.com> <41d6e77b0904071327u7e81bb0fsbe3b0e20aef00966@mail.gmail.com> Message-ID: <3d375d730904071358v4b6042b2ued8fc0d11d23b9ec@mail.gmail.com> On Tue, Apr 7, 2009 at 15:27, Nathaniel Peterson wrote: > Thanks for the quick response. > In http://www.scipy.org/Cookbook/Indexing > I see > >>>> a = C[1,2,3] > >>>> a > 23 >>>> type(a) > > >>>> type(int(a)) > > >>>> a**a > > Warning: overflow encountered in long_scalars > > -1276351769 >>>> int(a)**int(a) > > 20880467999847912034355032910567L > > This shows numpy can catch an overflow generated by a**a. > Why is it that numpy can catch this overflow, > > but can not catch one generated by np.multiply.reduce? Unlike true floating point errors (where the hardware FPU sets a flag whenever it does an atomic operation that overflows), we need to implement the integer overflow detection ourselves. We do it on the scalars, but not arrays because it would be too slow to implement for every atomic operation on arrays. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From nathanielpeterson08 at gmail.com Tue Apr 7 17:05:13 2009 From: nathanielpeterson08 at gmail.com (Nathaniel Peterson) Date: Tue, 7 Apr 2009 17:05:13 -0400 Subject: [Numpy-discussion] Can numpy catch this error for me? In-Reply-To: <3d375d730904071358v4b6042b2ued8fc0d11d23b9ec@mail.gmail.com> References: <41d6e77b0904071219w4ab87400k519e842ad626db08@mail.gmail.com> <3d375d730904071303q6f7467ecwb5c93c696b5d7116@mail.gmail.com> <41d6e77b0904071327u7e81bb0fsbe3b0e20aef00966@mail.gmail.com> <3d375d730904071358v4b6042b2ued8fc0d11d23b9ec@mail.gmail.com> Message-ID: <41d6e77b0904071405j6689bf1bk7b2fca3a7250ebc9@mail.gmail.com> Okay; thank you very much for the explanation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh8912 at yahoo.com Tue Apr 7 17:31:24 2009 From: josh8912 at yahoo.com (JJ) Date: Tue, 7 Apr 2009 14:31:24 -0700 (PDT) Subject: [Numpy-discussion] using numpy array data in C++ matrix template library (MTL) matrices Message-ID: <593404.65364.qm@web54001.mail.re2.yahoo.com> Hello: I would like to make a numpy array accessible in C++ as a matrix template library (MTL) matrix object. Has anyone done this before? I see that there is a project numpy_boost that will make numpy arrays accessible as boost::multi_array objects. Any ideas on how I might do this, or even better, a simple example of passing a numpy array to C++, accessing it in a MatrixRow container, changing one element, and then passing it back to python/numpy? I'm a novice C++ programmer, so bear with me. Thanks John From Matthew.Partridge at barclaysglobal.com Tue Apr 7 17:49:50 2009 From: Matthew.Partridge at barclaysglobal.com (Partridge, Matthew BGI SYD) Date: Wed, 8 Apr 2009 07:49:50 +1000 Subject: [Numpy-discussion] Can numpy catch this error for me? In-Reply-To: <3d375d730904071303q6f7467ecwb5c93c696b5d7116@mail.gmail.com> References: <41d6e77b0904071219w4ab87400k519e842ad626db08@mail.gmail.com> <3d375d730904071303q6f7467ecwb5c93c696b5d7116@mail.gmail.com> Message-ID: <5EFCE9D6AE4DD9409FA6BDD0E02FA8D54ED56F@sydnte2k032.insidelive.net> > On Tue, Apr 7, 2009 at 14:19, Nathaniel Peterson > wrote: > > > > import numpy as np > > import operator > > np.seterr(all='raise') > > a=np.arange(1)+1 > > print(a.dtype) > > # int32 > > for num in range(1,17): > > a=np.arange(num)+1 > > b=np.multiply.reduce(a) > > print('%s! = %s'%(num,b)) > > # c=reduce(operator.mul,range(1,num+1)) > > # assert(b==c) > > > > The code above outputs > > > > int32 > > 1! = 1 > > 2! = 2 > > 3! = 6 > > 4! = 24 > > 5! = 120 > > 6! = 720 > > 7! = 5040 > > 8! = 40320 > > 9! = 362880 > > 10! = 3628800 > > 11! = 39916800 > > 12! = 479001600 > > 13! = 1932053504 > > 14! = 1278945280 > > 15! = 2004310016 > > 16! = 2004189184 > > > > The results for 14! and above are wrong due to overflow of > the int32 > > data type. > > Is there a way to setup numpy so it will raise an error > when this occurs? > > No, sorry. Of course one can get the int behaviour by specifying dtype=object. (This way you get the longer int behaviour and the *lack of speed* to go with it.) for num in range(1,17): a=np.arange(num, dtype=object)+1 b=np.multiply.reduce(a) print('%s! = %s'%(num,b)) 1! = 1 2! = 2 3! = 6 4! = 24 5! = 120 6! = 720 7! = 5040 8! = 40320 9! = 362880 10! = 3628800 11! = 39916800 12! = 479001600 13! = 6227020800 14! = 87178291200 15! = 1307674368000 16! = 20922789888000 matt -- This message and any attachments are confidential, proprietary, and may be privileged. If this message was misdirected, Barclays Global Investors (BGI) does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of BGI, unless the author is authorized by BGI to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by BGI. Although BGI operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed. From ndbecker2 at gmail.com Tue Apr 7 18:53:25 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 07 Apr 2009 18:53:25 -0400 Subject: [Numpy-discussion] using numpy array data in C++ matrix template library (MTL) matrices References: <593404.65364.qm@web54001.mail.re2.yahoo.com> Message-ID: JJ wrote: > > Hello: > I would like to make a numpy array accessible in C++ as a matrix template > library (MTL) matrix object. Has anyone done this before? I see that > there is a project numpy_boost that will make numpy arrays accessible as > boost::multi_array objects. Any ideas on how I might do this, or even > better, a simple example of passing a numpy array to C++, accessing it in > a MatrixRow container, changing one element, and then passing it back to > python/numpy? I'm a novice C++ programmer, so bear with me. Thanks John I've used boost::python to interface to several different c++ matrix template libs. My preferred approach now is pyublas. From cycomanic at gmail.com Tue Apr 7 20:10:08 2009 From: cycomanic at gmail.com (Jochen Schroeder) Date: Wed, 8 Apr 2009 12:10:08 +1200 Subject: [Numpy-discussion] Construct a large n-dimensional complex matrix In-Reply-To: <3d375d730904061920h49b19a5ale6fd482ee2eadc35@mail.gmail.com> References: <46FED3E5-4EFE-4EAE-B75A-DF01A80BE2CF@gmail.com> <5b8d13220904042118i673527dsf2b0ac83a9b542d9@mail.gmail.com> <3d375d730904061920h49b19a5ale6fd482ee2eadc35@mail.gmail.com> Message-ID: <20090408001007.GA31337@jochen.schroeder.phy.auckland.ac.nz> On 06/04/09 21:20, Robert Kern wrote: > On Mon, Apr 6, 2009 at 20:53, Nathan Faggian wrote: > > HI, > > I want to construct a large complex matrix, I have the real and imaginary > > components as double vectors, is there a fast way to construct a complex > > vector in numpy? > > C = np.empty((n,m), dtype=complex) > C.real.flat[:] = real_values > C.imag.flat[:] = imag_values > Just a question, is there any advantage to doing C = 1.j *imag_values + real_values ? Cheers Jochen > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From robert.kern at gmail.com Tue Apr 7 20:37:18 2009 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 7 Apr 2009 19:37:18 -0500 Subject: [Numpy-discussion] Construct a large n-dimensional complex matrix In-Reply-To: <20090408001007.GA31337@jochen.schroeder.phy.auckland.ac.nz> References: <46FED3E5-4EFE-4EAE-B75A-DF01A80BE2CF@gmail.com> <5b8d13220904042118i673527dsf2b0ac83a9b542d9@mail.gmail.com> <3d375d730904061920h49b19a5ale6fd482ee2eadc35@mail.gmail.com> <20090408001007.GA31337@jochen.schroeder.phy.auckland.ac.nz> Message-ID: <3d375d730904071737s4424f18bi967f789f060609ca@mail.gmail.com> On Tue, Apr 7, 2009 at 19:10, Jochen Schroeder wrote: > On 06/04/09 21:20, Robert Kern wrote: >> On Mon, Apr 6, 2009 at 20:53, Nathan Faggian wrote: >> > HI, >> > I want to construct a large complex matrix, I have the real and imaginary >> > components as double vectors, is there a fast way to construct a complex >> > vector in numpy? >> >> C = np.empty((n,m), dtype=complex) >> C.real.flat[:] = real_values >> C.imag.flat[:] = imag_values >> > Just a question, is there any advantage to doing > C = 1.j *imag_values + real_values > ? Conciseness at the expense of large temporaries and unnecessary computation. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Wed Apr 8 02:35:41 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 08 Apr 2009 15:35:41 +0900 Subject: [Numpy-discussion] using numpy array data in C++ matrix template library (MTL) matrices In-Reply-To: <593404.65364.qm@web54001.mail.re2.yahoo.com> References: <593404.65364.qm@web54001.mail.re2.yahoo.com> Message-ID: <49DC45BD.5060407@ar.media.kyoto-u.ac.jp> JJ wrote: > Hello: > I would like to make a numpy array accessible in C++ as a matrix template library (MTL) matrix object. Has anyone done this before? I see that there is a project numpy_boost that will make numpy arrays accessible as boost::multi_array objects. Any ideas on how I might do this, or even better, a simple example of passing a numpy array to C++, accessing it in a MatrixRow container, changing one element, and then passing it back to python/numpy? I'm a novice C++ programmer, so bear with me. > The details will depend on your matrix library, but the underlying numpy array object has a full C api, so you can do whatever you want with it in your C++ code. But it can get quite messy :) I don't know for MTL, and for C++, boost can be useful, like Neal suggested. One thing to keep in mind is that C arrays are much more general that most low level C++ array containers (numpy arrays may not be contiguous in memory, for example). The easy solution is to use the numpy C api to get C contiguous arrays, so that the underlying data pointer can be accessed as expected by a plain C array. At this point, it is no different than using an C pointer pointing to an array, which virtually every C++ array library can deal with. cheers, David From david at ar.media.kyoto-u.ac.jp Wed Apr 8 03:18:08 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 08 Apr 2009 16:18:08 +0900 Subject: [Numpy-discussion] using numpy array data in C++ matrix template library (MTL) matrices In-Reply-To: <49DC45BD.5060407@ar.media.kyoto-u.ac.jp> References: <593404.65364.qm@web54001.mail.re2.yahoo.com> <49DC45BD.5060407@ar.media.kyoto-u.ac.jp> Message-ID: <49DC4FB0.6010100@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > The details will depend on your matrix library, but the underlying numpy > array object has a full C api, so you can do whatever you want with it > in your C++ code. But it can get quite messy :) > > I don't know for MTL, and for C++, boost can be useful, like Neal > suggested. One thing to keep in mind is that C arrays are much more > general Sorry, this should read "keep in mind that *numpy* C array are much more general", cheers, David From gael.varoquaux at normalesup.org Wed Apr 8 03:56:12 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 8 Apr 2009 09:56:12 +0200 Subject: [Numpy-discussion] array of matrices In-Reply-To: References: <1238193504.12867.4.camel@pc2.cole.uklinux.net> <3d375d730903271543m23e3f6dcj39c59cd115dedfa2@mail.gmail.com> <3d375d730903280047h2195a468i108f963453bdb78d@mail.gmail.com> <7f9d599f0903280947p3c30614epb83b9266ae25ed6e@mail.gmail.com> <3d375d730903282132r431ac3e0i66808b4ae533df4b@mail.gmail.com> Message-ID: <20090408075612.GC9197@phare.normalesup.org> On Wed, Apr 01, 2009 at 01:40:54AM +0000, Hans-Andreas Engel wrote: > By the way, matrix multiplication is one of the testcases for the generalized > ufuncs in numpy 1.3 -- this makes playing around with it easy: > In [1]: N = 10; a = randn(N, 4, 4); b = randn(N, 4, 4) > In [2]: import numpy.core.umath_tests > In [3]: (numpy.core.umath_tests.matrix_multiply(a, b) == [dot(ai, bi) for (ai, > bi) in zip(a, b)]).all() > Out[3]: True Hey Hans-Andreas, I am jumping on your message to stress that generalized ufunc realy need a bit more documentation. I have been trying to push a few collegues to use them, but I am getting not traction, because they are unsure of what generalized ufuncs do. I wasn't aware of the example you mentioned. It certainly is useful. Maybe it would be useful to add it as an example to the generalized-ufunc documnetation embryo (http://docs.scipy.org/numpy/docs/numpy-docs/reference/generalized_ufuncs.rst), with a discussion of the example. If you, or anyone else who knows generalized ufuncs, register on the above webpage, I can give rights to edit the page, so that you can improve the docs. Cheers, Ga?l From gael.varoquaux at normalesup.org Wed Apr 8 03:59:50 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 8 Apr 2009 09:59:50 +0200 Subject: [Numpy-discussion] trouble building docs with sphinx-0.6.1 In-Reply-To: References: Message-ID: <20090408075950.GD9197@phare.normalesup.org> On Wed, Apr 01, 2009 at 08:40:15PM +0000, Pauli Virtanen wrote: > It was an incompatibility of Numpy's autosummary extension and > Sphinx >= 0.6. It should now be fixed in Numpy trunk. autosummary is now in Sphinx (>= 0.6). Shouldn't we be using Sphinx's version, and default to ours for versions of Sphinx lacking the extention. If we don't, I fear the same problem will arise again in the long run: we have too much tight coupling for a feature that is not essential to numpy. Ga?l From charlie.xia.fdu at gmail.com Wed Apr 8 04:05:00 2009 From: charlie.xia.fdu at gmail.com (charlie) Date: Wed, 8 Apr 2009 01:05:00 -0700 Subject: [Numpy-discussion] Got: "undefined symbol: PyUnicodeUCS2_FromUnicode" error Message-ID: <11c6cf4e0904080105s775e1162y7705a09304d2845f@mail.gmail.com> Hi All, I got the "undefined symbol: PyUnicodeUCS2_FromUnicode" error when importing numpy. I have my own non-root version of python 2.5.4 final installed with --prefix=$HOME/usr. PYTHONHOME=$HOME/usr; PYTHONPATH="$PYTHONHOME/lib:$PYTHONHOME/lib/python2.5/site-packages/" install and import other modules works fine. I install numpy-1.3.0rc2 from the svn repository with "python setup.py install" then import numpy results in following error: *Python 2.5 (release25-maint, Jul 23 2008, 17:54:01) [GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy Traceback (most recent call last): File "", line 1, in File "numpy/__init__.py", line 130, in import add_newdocs File "numpy/add_newdocs.py", line 9, in from lib import add_newdoc File "numpy/lib/__init__.py", line 4, in from type_check import * File "numpy/lib/type_check.py", line 8, in import numpy.core.numeric as _nx File "numpy/core/__init__.py", line 5, in import multiarray ImportError: numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_FromUnicode >>> import Bio >>> import sys* I am not sure where is trick is. As I checked the previous discussion, I found several people raised similar issue but no one has posted a final solution to this yet. So I can only ask for help here again! Thanks in advance! Charlie -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Apr 8 04:09:13 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 8 Apr 2009 03:09:13 -0500 Subject: [Numpy-discussion] Got: "undefined symbol: PyUnicodeUCS2_FromUnicode" error In-Reply-To: <11c6cf4e0904080105s775e1162y7705a09304d2845f@mail.gmail.com> References: <11c6cf4e0904080105s775e1162y7705a09304d2845f@mail.gmail.com> Message-ID: <3d375d730904080109g6a1cb122u58f4ffccbe3fbfbd@mail.gmail.com> On Wed, Apr 8, 2009 at 03:05, charlie wrote: > Hi All, > > I got the? "undefined symbol: PyUnicodeUCS2_FromUnicode" error when > importing numpy. > > I have my own non-root version of python 2.5.4 final installed with > --prefix=$HOME/usr. > PYTHONHOME=$HOME/usr; > PYTHONPATH="$PYTHONHOME/lib:$PYTHONHOME/lib/python2.5/site-packages/" > install and import other modules works fine. > I install numpy-1.3.0rc2 from the svn repository with "python setup.py > install" > then import numpy results in following error: Are you sure you are using the same python executable to build and run this installation of numpy? This problem is always caused by this. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Wed Apr 8 03:53:10 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 08 Apr 2009 16:53:10 +0900 Subject: [Numpy-discussion] Got: "undefined symbol: PyUnicodeUCS2_FromUnicode" error In-Reply-To: <11c6cf4e0904080105s775e1162y7705a09304d2845f@mail.gmail.com> References: <11c6cf4e0904080105s775e1162y7705a09304d2845f@mail.gmail.com> Message-ID: <49DC57E6.8030905@ar.media.kyoto-u.ac.jp> charlie wrote: > Hi All, > > I got the "undefined symbol: PyUnicodeUCS2_FromUnicode" error when > importing numpy. > > I have my own non-root version of python 2.5.4 final installed with > --prefix=$HOME/usr. > PYTHONHOME=$HOME/usr; > PYTHONPATH="$PYTHONHOME/lib:$PYTHONHOME/lib/python2.5/site-packages/" > install and import other modules works fine. > I install numpy-1.3.0rc2 from the svn repository with "python setup.py > install" > then import numpy results in following error: > /Python 2.5 (release25-maint, Jul 23 2008, 17:54:01) > / ^^^^ As you can see, this is the system -wide python. You built a python which is binary incompatible with the system python - which is almost always a bad idea unless you really know what you are doing. You should use the system wide python, or built a compatible python. On debian, this is done with the option --enable-unicode=ucs4 (from you error message, I guess your installed python was configured with --enable-unicode=ucs2). cheers, David From michael.abshoff at googlemail.com Wed Apr 8 04:10:17 2009 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Wed, 08 Apr 2009 01:10:17 -0700 Subject: [Numpy-discussion] Got: "undefined symbol: PyUnicodeUCS2_FromUnicode" error In-Reply-To: <11c6cf4e0904080105s775e1162y7705a09304d2845f@mail.gmail.com> References: <11c6cf4e0904080105s775e1162y7705a09304d2845f@mail.gmail.com> Message-ID: <49DC5BE9.1070901@gmail.com> charlie wrote: > Hi All, Hi Charlie, > I got the "undefined symbol: PyUnicodeUCS2_FromUnicode" error when > importing numpy. > > I have my own non-root version of python 2.5.4 final installed with > --prefix=$HOME/usr. > PYTHONHOME=$HOME/usr; > PYTHONPATH="$PYTHONHOME/lib:$PYTHONHOME/lib/python2.5/site-packages/" > install and import other modules works fine. > I install numpy-1.3.0rc2 from the svn repository with "python setup.py > install" > then import numpy results in following error: > *Python 2.5 (release25-maint, Jul 23 2008, 17:54:01) > [GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > PyUnicodeUCS2_FromUnicode numpy was build with a python configured with ucs2 while the python you have is build with ucs4. >>>> import Bio >>>> import sys* > I am not sure where is trick is. As I checked the previous discussion, I > found several people raised similar issue but no one has posted a final > solution to this yet. So I can only ask for help here again! Thanks in > advance! To fix this build python with ucs2, i.e. "check configure --help" and set python to use ucs2. > Charlie > Cheers, Michael > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From pav at iki.fi Wed Apr 8 05:56:35 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 8 Apr 2009 09:56:35 +0000 (UTC) Subject: [Numpy-discussion] trouble building docs with sphinx-0.6.1 References: <20090408075950.GD9197@phare.normalesup.org> Message-ID: Wed, 08 Apr 2009 09:59:50 +0200, Gael Varoquaux kirjoitti: > On Wed, Apr 01, 2009 at 08:40:15PM +0000, Pauli Virtanen wrote: >> It was an incompatibility of Numpy's autosummary extension and Sphinx >> >= 0.6. It should now be fixed in Numpy trunk. > > autosummary is now in Sphinx (>= 0.6). Shouldn't we be using Sphinx's > version, and default to ours for versions of Sphinx lacking the > extention. > > If we don't, I fear the same problem will arise again in the long run: > we have too much tight coupling for a feature that is not essential to > numpy. The version in Sphinx is incomplete; I've asked Georg to pull fixes from my branch, and after those get merged, we can switch. -- Pauli Virtanen From matthew.brett at gmail.com Wed Apr 8 15:44:02 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 8 Apr 2009 12:44:02 -0700 Subject: [Numpy-discussion] Blas / lapack / binary installers... Message-ID: <1e2af89e0904081244y25c98d38n7da88f103b3c46ca@mail.gmail.com> Hello, Summary: is it possible to distribute, optionally or not, the blas / lapack libraries that numpy is built against, with the numpy binary installers? We at the NIPY project have run into what seems like a recurring problem; we want to build our code against both numpy and lapack, on windows, linux and OS X. No problem of course if we've done a development install - we already needed to have blas/lapack. For a binary install, we've got a nasty problem, especially on windows. Of course, on windows, numpy.distutils dreams that there will be libraries still in the installed location for the builder's machine - I think by default at C:\local\lib\yop\sse3. So, to make - say - NIPY - build, we need to: 1) Download some binary blas/lapack libraries, hoping they match our compiler, for example from links here (for windows): http://www.scipy.org/Installing_SciPy/Windows 2) EITHER: make our own site.cfg specifying the library location and library files, and copy this same site.cfg to any other package we're building against numpy / lapack, OR: edit the site.cfg in the installed package distribution to point to these new libraries. So, I'm wondering, would it be beyond sense to: Proposal ------------- Make the binary installers for windows with the option to put the blas/lapack libraries in some predictable location and adapt the site.cfg file accordingly. Is it possible this would also make compiling scipy on windows a whole lot easier? Thanks a lot, Matthew From gael.varoquaux at normalesup.org Wed Apr 8 17:30:30 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 8 Apr 2009 23:30:30 +0200 Subject: [Numpy-discussion] Blas / lapack / binary installers... In-Reply-To: <1e2af89e0904081244y25c98d38n7da88f103b3c46ca@mail.gmail.com> References: <1e2af89e0904081244y25c98d38n7da88f103b3c46ca@mail.gmail.com> Message-ID: <20090408213030.GB12399@phare.normalesup.org> On Wed, Apr 08, 2009 at 12:44:02PM -0700, Matthew Brett wrote: > Summary: is it possible to distribute, optionally or not, the blas / > lapack libraries that numpy is built against, with the numpy binary > installers? You mean the unlinked libraries (.a or .so), and the corresponding headers, I believe. I would rephrase the problem in a more general way: would it be possible to have so way that a project using scipy or numpy at runtime could use numpy's blas (or even better, scipy's lapack, that way we are garantied to have a real lapack) in its C code. I am not sure at all what the right strategy to do this it, this is why I rephrased Matthew's question in a more general way. Ga?l From elaine.angelino at gmail.com Wed Apr 8 17:57:47 2009 From: elaine.angelino at gmail.com (Elaine Angelino) Date: Wed, 8 Apr 2009 17:57:47 -0400 Subject: [Numpy-discussion] dtype field renaming Message-ID: <901520e20904081457p29f35b73i97f70663411d4e48@mail.gmail.com> hi there -- for a numpy.recarray, is it possible to rename the fields in the dtype? thanks a bunch elaine -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Wed Apr 8 18:12:08 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 8 Apr 2009 18:12:08 -0400 Subject: [Numpy-discussion] dtype field renaming In-Reply-To: <901520e20904081457p29f35b73i97f70663411d4e48@mail.gmail.com> References: <901520e20904081457p29f35b73i97f70663411d4e48@mail.gmail.com> Message-ID: On Apr 8, 2009, at 5:57 PM, Elaine Angelino wrote: > hi there -- > > for a numpy.recarray, is it possible to rename the fields in the > dtype? Take a new view: >>> a = np.array([(1,1)],dtype=[('a',int),('b',int)]) >>> b = a.view([("A",int), ('b', int)]) or: use numpy.lib.recfunctions.rename_fields From stefan at sun.ac.za Wed Apr 8 18:18:03 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 9 Apr 2009 00:18:03 +0200 Subject: [Numpy-discussion] dtype field renaming In-Reply-To: References: <901520e20904081457p29f35b73i97f70663411d4e48@mail.gmail.com> Message-ID: <9457e7c80904081518n6380e24ex54046a17706a73d1@mail.gmail.com> 2009/4/9 Pierre GM : >> for a numpy.recarray, is it possible to rename the fields in the >> dtype? > > Take a new view: > ?>>> a = np.array([(1,1)],dtype=[('a',int),('b',int)]) > ?>>> b = a.view([("A",int), ('b', int)]) > > or: > > use numpy.lib.recfunctions.rename_fields Or change the names tuple: a.dtype.names = ('c', 'd') St?fan From pgmdevlist at gmail.com Wed Apr 8 18:29:21 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 8 Apr 2009 18:29:21 -0400 Subject: [Numpy-discussion] dtype field renaming In-Reply-To: <9457e7c80904081518n6380e24ex54046a17706a73d1@mail.gmail.com> References: <901520e20904081457p29f35b73i97f70663411d4e48@mail.gmail.com> <9457e7c80904081518n6380e24ex54046a17706a73d1@mail.gmail.com> Message-ID: On Apr 8, 2009, at 6:18 PM, St?fan van der Walt wrote: > 2009/4/9 Pierre GM : >>> for a numpy.recarray, is it possible to rename the fields in the >>> dtype? >> >> Take a new view: >> >>> a = np.array([(1,1)],dtype=[('a',int),('b',int)]) >> >>> b = a.view([("A",int), ('b', int)]) >> >> or: >> >> use numpy.lib.recfunctions.rename_fields > > Or change the names tuple: > > a.dtype.names = ('c', 'd') Now that's wicked neat trick ! I love it ! Faster than taking a view for sure. Note that rename_fields should work also w/ nested fields (not that common, true). From stefan at sun.ac.za Wed Apr 8 18:47:18 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 9 Apr 2009 00:47:18 +0200 Subject: [Numpy-discussion] dtype field renaming In-Reply-To: References: <901520e20904081457p29f35b73i97f70663411d4e48@mail.gmail.com> <9457e7c80904081518n6380e24ex54046a17706a73d1@mail.gmail.com> Message-ID: <9457e7c80904081547t52b08234tb5e8b34deca857a2@mail.gmail.com> 2009/4/9 Pierre GM : >> a.dtype.names = ('c', 'd') > > Now that's wicked neat trick ! I love it ! Faster than taking a view > for sure. > Note that rename_fields should work also w/ nested fields (not that > common, true). Yes, that is slightly more effort: In [30]: arr = np.array(((2,3),), dtype=[('x',[('a',int),('b',int)])]) In [31]: arr['x'].dtype.names = ('p','q') No all-in-one solution for renaming all the fields, but if you wanted to do that a view would probably be the cleanest. Cheers St?fan From geometrian at gmail.com Wed Apr 8 21:39:14 2009 From: geometrian at gmail.com (Ian Mallett) Date: Wed, 8 Apr 2009 18:39:14 -0700 Subject: [Numpy-discussion] Specially Constructed Arrays Message-ID: Hello, I want to make an array of size sqrt(n) by sqrt(n) by 3, filled with special values. The values range from 0.0 to 3.0, starting with 0.0 at one corner and ending at 3.0 in the opposite, increasing going row by row. The value is to be encoded in each color. Because this is somewhat abstract, here's a small example (n=25), generated using the attached code (it also multiplies the number by 255 to obtain a RGB color and not messy floats) to show the concept. The real version should be done by NumPy. This is where I need help; I have no idea how to even approach the problem. [[ 0, 0, 0],[ 32, 0, 0],[ 64, 0, 0],[ 96, 0, 0],[128, 0, 0], [159, 0, 0],[191, 0, 0],[223, 0, 0],[255, 0, 0],[255, 32, 0], [255, 64, 0],[255, 96, 0],[255,128, 0],[255,159, 0],[255,191, 0], [255,223, 0],[255,255, 0],[255,255, 32],[255,255, 64],[255,255, 96], [255,255,128],[255,255,159],[255,255,191],[255,255,223],[255,255,255]] Arrays like this need to be generated quite quickly, so the per-pixel method I presented will not work. How should I do it with NumPy? Thanks, Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: particlesurfgen.py Type: text/x-python Size: 870 bytes Desc: not available URL: From cournape at gmail.com Wed Apr 8 22:11:13 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 9 Apr 2009 11:11:13 +0900 Subject: [Numpy-discussion] Blas / lapack / binary installers... In-Reply-To: <1e2af89e0904081244y25c98d38n7da88f103b3c46ca@mail.gmail.com> References: <1e2af89e0904081244y25c98d38n7da88f103b3c46ca@mail.gmail.com> Message-ID: <5b8d13220904081911t67760efo37f53eed51985221@mail.gmail.com> Hi Matthew, On Thu, Apr 9, 2009 at 4:44 AM, Matthew Brett wrote: > Hello, > > Summary: is it possible to distribute, optionally or not, the blas / > lapack libraries that numpy is built against, with the numpy binary > installers? Yes, it is possible. > We at the NIPY project have run into what seems like a recurring > problem; we want to build our code against both numpy and lapack, on > windows, linux and OS X. > > No problem of course if we've done a development install - we already > needed to have blas/lapack. I am not sure I understand: why do you need blas/lapack to build projects ? Does NiPY itself uses blas/lapack ? > Proposal > ------------- > Make the binary installers for windows with the option to put the > blas/lapack libraries in some predictable location and adapt the > site.cfg file accordingly. It is possible, but it would make the installers quite big. As Gael said, a more general solution would be to use .so (well, dll), put them in a known location, so that applications can reuse it. That's something which would be great for a lot of reasons, but that's difficult to do on windows + distutils. cheers, David From matthew.brett at gmail.com Thu Apr 9 00:54:25 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 8 Apr 2009 21:54:25 -0700 Subject: [Numpy-discussion] Blas / lapack / binary installers... In-Reply-To: <5b8d13220904081911t67760efo37f53eed51985221@mail.gmail.com> References: <1e2af89e0904081244y25c98d38n7da88f103b3c46ca@mail.gmail.com> <5b8d13220904081911t67760efo37f53eed51985221@mail.gmail.com> Message-ID: <1e2af89e0904082154t92cadbem8653b204295a943e@mail.gmail.com> Hi, >> We at the NIPY project have run into what seems like a recurring >> problem; we want to build our code against both numpy and lapack, on >> windows, linux and OS X. >> >> No problem of course if we've done a development install - we already >> needed to have blas/lapack. > > I am not sure I understand: why do you need blas/lapack to build > projects ? Does NiPY itself uses blas/lapack ?? Yes it does... >> Proposal >> ------------- >> Make the binary installers for windows with the option to put the >> blas/lapack libraries in some predictable location and adapt the >> site.cfg file accordingly. > > It is possible, but it would make the installers quite big. As Gael > said, a more general solution would be to use .so (well, dll), put > them in a known location, so that applications can reuse it. That's > something which would be great for a lot of reasons, but that's > difficult to do on windows + distutils. Yes, I know, hence my suggestion of something more practical in the short term. I wonder whether the installer could be: by default, smallish, with just numpy, with the option of pulling down the lapack libraries from the web on installation or, possibly optionally, large, with the lapack libraries installed by default. I was thinking, that if that were the case, then installing scipy from source might go from being very nasty (needing a compiled blas / lapack), to only fairly nasty (needing fortran), making it more accessible. Thanks a lot, Matthew From gael.varoquaux at normalesup.org Thu Apr 9 01:17:49 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 9 Apr 2009 07:17:49 +0200 Subject: [Numpy-discussion] Blas / lapack / binary installers... In-Reply-To: <5b8d13220904081911t67760efo37f53eed51985221@mail.gmail.com> References: <1e2af89e0904081244y25c98d38n7da88f103b3c46ca@mail.gmail.com> <5b8d13220904081911t67760efo37f53eed51985221@mail.gmail.com> Message-ID: <20090409051749.GA13241@phare.normalesup.org> On Thu, Apr 09, 2009 at 11:11:13AM +0900, David Cournapeau wrote: > > No problem of course if we've done a development install - we already > > needed to have blas/lapack. > I am not sure I understand: why do you need blas/lapack to build > projects ? Does NiPY itself uses blas/lapack ? NiPy uses blas/lapack in the C code, and I feel this is not an abnormal situation: as soon as you starting wanting to do vector operations in C, you end up needing blas/lapack. > > Proposal > > ------------- > > Make the binary installers for windows with the option to put the > > blas/lapack libraries in some predictable location and adapt the > > site.cfg file accordingly. > It is possible, but it would make the installers quite big. As Gael > said, a more general solution would be to use .so (well, dll), put > them in a known location, so that applications can reuse it. That's > something which would be great for a lot of reasons, but that's > difficult to do on windows + distutils. OK, so a short term solution might be to have a 'dev' installer that would ship the lapack libraries used to build numpy (that would address the size issue), and a long term one, which would be to expose shared libraries and headers (.so and .dll) to the applications. Does that sound like a feasible plan? I'd be interested in helping you work on part 2 of the plan, but I can't spend any time on it before fall (a set of deadlines...). Ga?l From david at ar.media.kyoto-u.ac.jp Thu Apr 9 01:14:44 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 09 Apr 2009 14:14:44 +0900 Subject: [Numpy-discussion] Blas / lapack / binary installers... In-Reply-To: <1e2af89e0904082154t92cadbem8653b204295a943e@mail.gmail.com> References: <1e2af89e0904081244y25c98d38n7da88f103b3c46ca@mail.gmail.com> <5b8d13220904081911t67760efo37f53eed51985221@mail.gmail.com> <1e2af89e0904082154t92cadbem8653b204295a943e@mail.gmail.com> Message-ID: <49DD8444.2010909@ar.media.kyoto-u.ac.jp> Matthew Brett wrote: > > Yes it does... > Ok. > Yes, I know, hence my suggestion of something more practical in the > short term. I wonder whether the installer could be: > > by default, smallish, with just numpy, with the option of pulling down > the lapack libraries from the web on installation or, possibly > optionally, large, with the lapack libraries installed by default. > That's possible, but I don't know how much work it would be to make it work well. In particular, once you start using the web, you have all kind of problems with proxy and the likes. But the nsis installer certainly can support this: http://nsis.sourceforge.net/NSISdl_plug-in I would prefer that to a developer version, to avoid people having to choose which version they want. cheers, David From charlesr.harris at gmail.com Thu Apr 9 02:02:51 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 9 Apr 2009 00:02:51 -0600 Subject: [Numpy-discussion] using reducing functions without eliminating dimensions? In-Reply-To: References: Message-ID: On Tue, Apr 7, 2009 at 12:44 PM, Dan Lenski wrote: > Hi all, > > I often want to use some kind of dimension-reducing function (like min(), > max(), sum(), mean()) on an array without actually removing the last > dimension, so that I can then do operations broadcasting the reduced > array back to the size of the full array. Full example: > > >> table.shape > (47, 1814) > > >> table.min(axis=1).shape > (47,) > > >> table - table.min(axis=1) > ValueError: shape mismatch: objects cannot be broadcast to a single > shape > > >> table - table.min(axis=1)[:, newaxis] > > I have to resort to ugly code with lots of stuff like "... axis=1)[:, > newaxis]". > > Is there any way to get the reducing functions to leave a size-1 dummy > dimension in place, to make this easier? > Not at the moment. There was talk a while back of adding a keyword for this option, it would certainly make things easier for some common uses. It might be worth starting that conversation up again. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Thu Apr 9 02:04:37 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 9 Apr 2009 15:04:37 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <5b8d13220903280852l17c578d7hadddbc40873f8c9f@mail.gmail.com> <49D136A3.80302@gmail.com> <5b8d13220903302041m71cea034pf21b4a61d39eb6f@mail.gmail.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> Message-ID: <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> On Wed, Apr 8, 2009 at 3:34 AM, Ondrej Certik wrote: > Yes. Do you have any windows developers (I am sorry, I am not familiar at all with sympy)? My main concern with git are: - you may need the command line - the index can be confusing (you can avoid learning it at first, but still some errors won't make sense before you get it). - git is not discoverable (you need to read some documentation) I tend to think those problems are not that big, and the concern raised on python ML way overblown - but I learned the tool, and I don't think anyone wants to alienate any numpy/scipy developer. cheers, David From peridot.faceted at gmail.com Thu Apr 9 02:29:49 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 9 Apr 2009 02:29:49 -0400 Subject: [Numpy-discussion] using reducing functions without eliminating dimensions? In-Reply-To: References: Message-ID: 2009/4/9 Charles R Harris : > > > On Tue, Apr 7, 2009 at 12:44 PM, Dan Lenski wrote: >> >> Hi all, >> >> I often want to use some kind of dimension-reducing function (like min(), >> max(), sum(), mean()) on an array without actually removing the last >> dimension, so that I can then do operations broadcasting the reduced >> array back to the size of the full array. ?Full example: >> >> ?>> table.shape >> ?(47, 1814) >> >> ?>> table.min(axis=1).shape >> ?(47,) >> >> ?>> table - table.min(axis=1) >> ?ValueError: shape mismatch: objects cannot be broadcast to a single >> shape >> >> ?>> table - table.min(axis=1)[:, newaxis] >> >> I have to resort to ugly code with lots of stuff like "... axis=1)[:, >> newaxis]". >> >> Is there any way to get the reducing functions to leave a size-1 dummy >> dimension in place, to make this easier? > > Not at the moment. There was talk a while back of adding a keyword for this > option, it would certainly make things easier for some common uses. It might > be worth starting that conversation up again. What's wrong with np.amin(a,axis=-1)[...,np.newaxis]? Anne > Chuck > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From robert.kern at gmail.com Thu Apr 9 02:31:33 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 9 Apr 2009 01:31:33 -0500 Subject: [Numpy-discussion] using reducing functions without eliminating dimensions? In-Reply-To: References: Message-ID: <3d375d730904082331g55d21e5el517591127fc19384@mail.gmail.com> On Thu, Apr 9, 2009 at 01:29, Anne Archibald wrote: > What's wrong with np.amin(a,axis=-1)[...,np.newaxis]? It's cumbersome, particularly when you have axis=arbitrary_axis. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Thu Apr 9 02:43:03 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 9 Apr 2009 01:43:03 -0500 Subject: [Numpy-discussion] Specially Constructed Arrays In-Reply-To: References: Message-ID: <3d375d730904082343t4f3ef72aj409607cbfbdaf2df@mail.gmail.com> On Wed, Apr 8, 2009 at 20:39, Ian Mallett wrote: > Hello, > > I want to make an array of size sqrt(n) by sqrt(n) by 3, filled with special > values. > > The values range from 0.0 to 3.0, starting with 0.0 at one corner and ending > at 3.0 in the opposite, increasing going row by row.? The value is to be > encoded in each color.? Because this is somewhat abstract, here's a small > example (n=25), generated using the attached code (it also multiplies the > number by 255 to obtain a RGB color and not messy floats) to show the > concept.? The real version should be done by NumPy.? This is where I need > help; I have no idea how to even approach the problem. > > [[? 0,? 0,? 0],[ 32,? 0,? 0],[ 64,? 0,? 0],[ 96,? 0,? 0],[128,? 0,? 0], > ?[159,? 0,? 0],[191,? 0,? 0],[223,? 0,? 0],[255,? 0,? 0],[255, 32,? 0], > ?[255, 64,? 0],[255, 96,? 0],[255,128,? 0],[255,159,? 0],[255,191,? 0], > ?[255,223,? 0],[255,255,? 0],[255,255, 32],[255,255, 64],[255,255, 96], > ?[255,255,128],[255,255,159],[255,255,191],[255,255,223],[255,255,255]] > > Arrays like this need to be generated quite quickly, so the per-pixel method > I presented will not work.? How should I do it with NumPy? In [1]: from numpy import * In [2]: sqrtn = 5 In [3]: n = sqrtn**2 In [4]: x = linspace(0.0, 3.0, n) In [5]: y = column_stack([x, x-1, x-2]).clip(0, 1).reshape([sqrtn, sqrtn, 3]) In [6]: z = (y * 255).round().astype(uint8) In [7]: z Out[7]: array([[[ 0, 0, 0], [ 32, 0, 0], [ 64, 0, 0], [ 96, 0, 0], [128, 0, 0]], [[159, 0, 0], [191, 0, 0], [223, 0, 0], [255, 0, 0], [255, 32, 0]], [[255, 64, 0], [255, 96, 0], [255, 128, 0], [255, 159, 0], [255, 191, 0]], [[255, 223, 0], [255, 255, 0], [255, 255, 32], [255, 255, 64], [255, 255, 96]], [[255, 255, 128], [255, 255, 159], [255, 255, 191], [255, 255, 223], [255, 255, 255]]], dtype=uint8) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From klaus.kopec at tuebingen.mpg.de Thu Apr 9 04:56:36 2009 From: klaus.kopec at tuebingen.mpg.de (Klaus Kopec) Date: Thu, 09 Apr 2009 10:56:36 +0200 Subject: [Numpy-discussion] shift for optimal superimposition of two 3D matrices according to correlation computed using FFT In-Reply-To: References: Message-ID: > Does it work to use a cutoff of half the size of the input arrays in > each dimension? This is equivalent to calculating both shifts (the > positive and negative) and using whichever has a smaller absolute value. no, unfortunately the cutoff is not half of the dimensions. > Alternately, you could use numpy.roll to shift the data (one axis at a > time). Since roll wraps around, you wouldn't need to bother figuring > out which shift is "correct". Maybe I do not understand correctly what you mean by this, but as far as I get it this will not help me, given that I do not know the number of positions I need to roll. I think the parameter to the roll function would be the same value as the cutoff I need. If you wanted to suggest that I can check some property for every shift and hence could try every possible shift in each axis: this is computationally not feasable for me. I need to run time consuming operations to ensure the correctness and with a 200x200x200 matrix (which is very roughly as large as it gets for me), so this would ruin the speed benfits of using the FFT. > Finally, you could not use FFTs but instead directly optimize a > transformation between the two, using scipy.ndimage's affine > transforms and scipy.optimize's numerical optimizers. Sadly, this is no option for me, as this is a research project and I need to use FFT. One possible problem could be, that I am not sure about the benefits of using the complex valued FFT (numpy.fft.fftn) compared to the real valued version for my matrix of reals. At the moment I use numpy.fft.rfftn, as the real part is the same as for the imaginary FFT. Maybe (just guessing) some information about the cutoff is found in the imaginary result part?!? Does anybody have other suggestions? From ondrej at certik.cz Thu Apr 9 06:00:16 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 9 Apr 2009 03:00:16 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <5b8d13220903280852l17c578d7hadddbc40873f8c9f@mail.gmail.com> <49D136A3.80302@gmail.com> <5b8d13220903302041m71cea034pf21b4a61d39eb6f@mail.gmail.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> Message-ID: <85b5c3130904090300h5c47905i935f02024ffec847@mail.gmail.com> On Wed, Apr 8, 2009 at 11:04 PM, David Cournapeau wrote: > On Wed, Apr 8, 2009 at 3:34 AM, Ondrej Certik wrote: > >> Yes. > > Do you have any windows developers (I am sorry, I am not familiar at > all with sympy)? Not much. > > My main concern with git are: > ?- you may need the command line Oh, definitely. What else? :) > ?- the index can be confusing (you can avoid learning it at first, but > still some errors won't make sense before you get it). Yes, but in fact the staging area (if this is what you mean) is in every VCS, only it's hidden, except git, where it is made explicit. > ?- git is not discoverable (you need to read some documentation) Yes, but "git help command" is very detailed most of the time. > > I tend to think those problems are not that big, and the concern > raised on python ML way overblown - but I learned the tool, and I > don't think anyone wants to alienate any numpy/scipy developer. Which tool did you learn? Git? Ondrej From david at ar.media.kyoto-u.ac.jp Thu Apr 9 06:04:04 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 09 Apr 2009 19:04:04 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <85b5c3130904090300h5c47905i935f02024ffec847@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <5b8d13220903280852l17c578d7hadddbc40873f8c9f@mail.gmail.com> <49D136A3.80302@gmail.com> <5b8d13220903302041m71cea034pf21b4a61d39eb6f@mail.gmail.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <85b5c3130904090300h5c47905i935f02024ffec847@mail.gmail.com> Message-ID: <49DDC814.4040802@ar.media.kyoto-u.ac.jp> Ondrej Certik wrote: > > Yes, but in fact the staging area (if this is what you mean) is in > every VCS, only it's hidden, except git, where it is made explicit. > I am not sure the staging area concept is there in other vcs, because in git it is intrinsically linked to idea that git tracks content. It is powerful, and I really miss it in other DVCS, but it takes time to get used to - time that other people may not be willing to spend. I tried to write some basic instructions which totally bypass the index concept, but it can still bite you even in those simple cases: http://projects.scipy.org/numpy/wiki/GitWorkflow#Commonscenario > Which tool did you learn? Git? > git. Both Stefan and me really like git, but assuming we make the transition, we worry a bit about the transition cost for people who are happy with svn. cheers, David From ondrej at certik.cz Thu Apr 9 06:29:41 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 9 Apr 2009 03:29:41 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49DDC814.4040802@ar.media.kyoto-u.ac.jp> References: <49CE3A2C.9000007@enthought.com> <5b8d13220903302041m71cea034pf21b4a61d39eb6f@mail.gmail.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <85b5c3130904090300h5c47905i935f02024ffec847@mail.gmail.com> <49DDC814.4040802@ar.media.kyoto-u.ac.jp> Message-ID: <85b5c3130904090329q436fbedfia0c4623f2b454702@mail.gmail.com> On Thu, Apr 9, 2009 at 3:04 AM, David Cournapeau wrote: > Ondrej Certik wrote: >> >> Yes, but in fact the staging area (if this is what you mean) is in >> every VCS, only it's hidden, except git, where it is made explicit. >> > > I am not sure the staging area concept is there in other vcs, because in > git it is intrinsically linked to idea that git tracks content. It is If you do "hg add", you add to the staging area. If you do "hg commit", you commit it, together with automatically adding all changes to the staging area as well (and committing). I think it's the same with svn. > powerful, and I really miss it in other DVCS, but it takes time to get > used to - time that other people may not be willing to spend. I tried to > write some basic instructions which totally bypass the index concept, > but it can still bite you even in those simple cases: > > http://projects.scipy.org/numpy/wiki/GitWorkflow#Commonscenario > >> Which tool did you learn? Git? >> > > git. Both Stefan and me really like git, but assuming we make the > transition, we worry a bit about the transition cost for people who are > happy with svn. I think it won't. At least it didn't in the sympy case. Ondrej From gael.varoquaux at normalesup.org Thu Apr 9 07:22:59 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 9 Apr 2009 13:22:59 +0200 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <5b8d13220903280852l17c578d7hadddbc40873f8c9f@mail.gmail.com> <49D136A3.80302@gmail.com> <5b8d13220903302041m71cea034pf21b4a61d39eb6f@mail.gmail.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> Message-ID: <20090409112259.GA4091@phare.normalesup.org> On Thu, Apr 09, 2009 at 03:04:37PM +0900, David Cournapeau wrote: > - git is not discoverable (you need to read some documentation) Yes, I found that when I tried to use git to quickly get a few basic things done, I had a lot of stupid problems: $ git stat git: 'stat' is not a git-command The most commonly used git commands are: add Add file contents to the index [...] => WTF?? Every single VCS works with 'stat'. Why do I need to learn something new. $ git up git: 'up' is not a git-command [...] => Right, I am starting to like this soft $ git info git: 'info' is not a git-command [...] => Really, you are kidding me. $ git revert usage: git-revert [--edit | --no-edit] [-n] => fsck-you, Git, I want to revert. I don't understand this error message, I just want to revert. Seriously, I haven't figured out how to revert. I would need to read the doc (I haven't). I _really_ have an issue with the way git treats the user. It is pretty "Well, lad, I am the best VCS in the world, and you are going to have to relearn your VCS-foo to use me". Great way to loose developers, IMHO. I whish the guys could make some small usability efforts. My 2 cents, Ga?l From david at ar.media.kyoto-u.ac.jp Thu Apr 9 07:14:20 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 09 Apr 2009 20:14:20 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <20090409112259.GA4091@phare.normalesup.org> References: <49CE3A2C.9000007@enthought.com> <5b8d13220903280852l17c578d7hadddbc40873f8c9f@mail.gmail.com> <49D136A3.80302@gmail.com> <5b8d13220903302041m71cea034pf21b4a61d39eb6f@mail.gmail.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <20090409112259.GA4091@phare.normalesup.org> Message-ID: <49DDD88C.4070904@ar.media.kyoto-u.ac.jp> Gael Varoquaux wrote: > On Thu, Apr 09, 2009 at 03:04:37PM +0900, David Cournapeau wrote: > >> - git is not discoverable (you need to read some documentation) >> > > Yes, I found that when I tried to use git to quickly get a few basic > things done, I had a lot of stupid problems: > The argument could be made that in any DVCS, the commands *are* different even if the name is the same (commit and up certainly don't do the same in svn and in bzr, for example). David From gael.varoquaux at normalesup.org Thu Apr 9 07:33:28 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 9 Apr 2009 13:33:28 +0200 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49DDD88C.4070904@ar.media.kyoto-u.ac.jp> References: <49D136A3.80302@gmail.com> <5b8d13220903302041m71cea034pf21b4a61d39eb6f@mail.gmail.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <20090409112259.GA4091@phare.normalesup.org> <49DDD88C.4070904@ar.media.kyoto-u.ac.jp> Message-ID: <20090409113328.GB4091@phare.normalesup.org> On Thu, Apr 09, 2009 at 08:14:20PM +0900, David Cournapeau wrote: > Gael Varoquaux wrote: > > On Thu, Apr 09, 2009 at 03:04:37PM +0900, David Cournapeau wrote: > >> - git is not discoverable (you need to read some documentation) > > Yes, I found that when I tried to use git to quickly get a few basic > > things done, I had a lot of stupid problems: > The argument could be made that in any DVCS, the commands *are* > different even if the name is the same (commit and up certainly don't do > the same in svn and in bzr, for example). That may be true for up, in my previous examples, but not for stat, info and revert. Ga?l From david at ar.media.kyoto-u.ac.jp Thu Apr 9 07:26:26 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 09 Apr 2009 20:26:26 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <20090409113328.GB4091@phare.normalesup.org> References: <49D136A3.80302@gmail.com> <5b8d13220903302041m71cea034pf21b4a61d39eb6f@mail.gmail.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <20090409112259.GA4091@phare.normalesup.org> <49DDD88C.4070904@ar.media.kyoto-u.ac.jp> <20090409113328.GB4091@phare.normalesup.org> Message-ID: <49DDDB62.8080701@ar.media.kyoto-u.ac.jp> Gael Varoquaux wrote: > That may be true for up, in my previous examples, but not for stat, info > and revert. > hg does not have an info either. And for revert, st, ci, etc... what do you feel is missing from there ? http://projects.scipy.org/numpy/wiki/GitWorkflow Which is an attempt to alleviate the non "discoverability" thing in the context of numpy/scipy. cheers, David From josef.pktd at gmail.com Thu Apr 9 09:09:33 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 9 Apr 2009 09:09:33 -0400 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <85b5c3130904090300h5c47905i935f02024ffec847@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <49D136A3.80302@gmail.com> <5b8d13220903302041m71cea034pf21b4a61d39eb6f@mail.gmail.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <85b5c3130904090300h5c47905i935f02024ffec847@mail.gmail.com> Message-ID: <1cd32cbb0904090609q793279bdwd7118fbf44bafe72@mail.gmail.com> On Thu, Apr 9, 2009 at 6:00 AM, Ondrej Certik wrote: > On Wed, Apr 8, 2009 at 11:04 PM, David Cournapeau wrote: >> On Wed, Apr 8, 2009 at 3:34 AM, Ondrej Certik wrote: >> >>> Yes. >> >> Do you have any windows developers (I am sorry, I am not familiar at >> all with sympy)? > > Not much. > >> >> My main concern with git are: >> ?- you may need the command line > > Oh, definitely. What else? :) I never commit to svn from the command line, only with a gui that lets me easily browse any changes that I am about to commit. I use the command line for checkouts and updates. or else plugins to editors, eg. eclipse For those of us, that don't use git all the time, the amount that we have to learn and keep in mind, seems pretty high. With a gui, I can check the menus to see what the names of the necessary commands are and I don't have to memorize them or type help all the time to remind me. Josef > >> ?- the index can be confusing (you can avoid learning it at first, but >> still some errors won't make sense before you get it). > > Yes, but in fact the staging area (if this is what you mean) is in > every VCS, only it's hidden, except git, where it is made explicit. > >> ?- git is not discoverable (you need to read some documentation) > > Yes, but "git help command" is very detailed most of the time. > >> >> I tend to think those problems are not that big, and the concern >> raised on python ML way overblown - but I learned the tool, and I >> don't think anyone wants to alienate any numpy/scipy developer. > > Which tool did you learn? Git? > > Ondrej > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From chaos.proton at gmail.com Thu Apr 9 09:14:11 2009 From: chaos.proton at gmail.com (Grissiom) Date: Thu, 9 Apr 2009 21:14:11 +0800 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <1cd32cbb0904090609q793279bdwd7118fbf44bafe72@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <5b8d13220903302041m71cea034pf21b4a61d39eb6f@mail.gmail.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <85b5c3130904090300h5c47905i935f02024ffec847@mail.gmail.com> <1cd32cbb0904090609q793279bdwd7118fbf44bafe72@mail.gmail.com> Message-ID: On Thu, Apr 9, 2009 at 21:09, wrote: > On Thu, Apr 9, 2009 at 6:00 AM, Ondrej Certik wrote: > > On Wed, Apr 8, 2009 at 11:04 PM, David Cournapeau > wrote: > >> On Wed, Apr 8, 2009 at 3:34 AM, Ondrej Certik wrote: > >> > >>> Yes. > >> > >> Do you have any windows developers (I am sorry, I am not familiar at > >> all with sympy)? > > > > Not much. > > > >> > >> My main concern with git are: > >> - you may need the command line > > > > Oh, definitely. What else? :) > > > I never commit to svn from the command line, only with a gui that lets > me easily browse any changes that I am about to commit. I use the > command line for checkouts and updates. > > If you want to do visualized commit in git, try "git citool" ;) -- Cheers, Grissiom -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Thu Apr 9 09:38:23 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 9 Apr 2009 15:38:23 +0200 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <5b8d13220903280852l17c578d7hadddbc40873f8c9f@mail.gmail.com> <49D136A3.80302@gmail.com> <5b8d13220903302041m71cea034pf21b4a61d39eb6f@mail.gmail.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> Message-ID: 2009/4/9 David Cournapeau : > On Wed, Apr 8, 2009 at 3:34 AM, Ondrej Certik wrote: > >> Yes. > > Do you have any windows developers (I am sorry, I am not familiar at > all with sympy)? > > My main concern with git are: > ?- you may need the command line > ?- the index can be confusing (you can avoid learning it at first, but > still some errors won't make sense before you get it). > ?- git is not discoverable (you need to read some documentation) I tried to install git on my computer, as git-svn seemed so nice. I tried git-svn and it told me that some .pm files are missing. Why did it install git-svn if some files are missing? And why isn't it possible to get some information on how to install those missing files on the Internet. git seems nice, but lacks documentation and usability :| Matthieu -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From zachary.pincus at yale.edu Thu Apr 9 09:51:32 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 9 Apr 2009 09:51:32 -0400 Subject: [Numpy-discussion] shift for optimal superimposition of two 3D matrices according to correlation computed using FFT In-Reply-To: References: Message-ID: <764FA243-F189-4BF3-9B49-5552D9DBB12F@yale.edu> >> Does it work to use a cutoff of half the size of the input arrays in >> each dimension? This is equivalent to calculating both shifts (the >> positive and negative) and using whichever has a smaller absolute >> value. > no, unfortunately the cutoff is not half of the dimensions. Explain more about what you need from this "cutoff" then. Given that the peak is just an index into the array, it seems like the "cutoff" for using the peak index as the shift vs. subtracting the size of the array from the index should be a function of the size of the arrays. How is the cutoff value data-dependent? Can you provide some simple 1D examples? >> Alternately, you could use numpy.roll to shift the data (one axis >> at a >> time). Since roll wraps around, you wouldn't need to bother figuring >> out which shift is "correct". > Maybe I do not understand correctly what you mean by this, but as > far as > I get it this will not help me, given that I do not know the number of > positions I need to roll. I think the parameter to the roll > function would be the same value as the cutoff I need. > If you wanted to suggest that I can check some property for every > shift > and hence could try every possible shift in each axis: this is > computationally not feasable for me. I need to run time consuming > operations to ensure the correctness and with a 200x200x200 matrix > (which is very roughly as large as it gets for me), so this would ruin > the speed benfits of using the FFT. 1D: say that you have 30-element arrays and a correlation peak of 25. I assume this means you need to shift one array 5 positions backward or the other 5 forward. However (and this ought to be justifiable since the FFT assumes wrap-around as well) you could also just shift one array 25 positions forward, wrapping around as needed. No? numpy.roll shifts arrays with wrap-around. If this isn't what you need, you'll have to explain a bit more about your problem, and about what it is you need from the cutoff value. Zach From cournape at gmail.com Thu Apr 9 10:04:01 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 9 Apr 2009 23:04:01 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: References: <49CE3A2C.9000007@enthought.com> <49D136A3.80302@gmail.com> <5b8d13220903302041m71cea034pf21b4a61d39eb6f@mail.gmail.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> Message-ID: <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> On Thu, Apr 9, 2009 at 10:38 PM, Matthieu Brucher wrote: > 2009/4/9 David Cournapeau : >> On Wed, Apr 8, 2009 at 3:34 AM, Ondrej Certik wrote: >> >>> Yes. >> >> Do you have any windows developers (I am sorry, I am not familiar at >> all with sympy)? >> >> My main concern with git are: >> ?- you may need the command line >> ?- the index can be confusing (you can avoid learning it at first, but >> still some errors won't make sense before you get it). >> ?- git is not discoverable (you need to read some documentation) > > I tried to install git on my computer, as git-svn seemed so nice. I > tried git-svn and it told me that some .pm files are missing. Why did > it install git-svn if some files are missing? And why isn't it > possible to get some information on how to install those missing files > on the Internet. For which system did you install it ? I don't think git-svn work well or at all on windows. Otherwise, git has binary installers for every platform which matters. Installing from sources is a bit harder than bzr/hg, but certainly easier than svn, if you need to, cheers, David From chaos.proton at gmail.com Thu Apr 9 10:11:50 2009 From: chaos.proton at gmail.com (Grissiom) Date: Thu, 9 Apr 2009 22:11:50 +0800 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: References: <49CE3A2C.9000007@enthought.com> <49D136A3.80302@gmail.com> <5b8d13220903302041m71cea034pf21b4a61d39eb6f@mail.gmail.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> Message-ID: On Thu, Apr 9, 2009 at 21:38, Matthieu Brucher wrote: > 2009/4/9 David Cournapeau : > > On Wed, Apr 8, 2009 at 3:34 AM, Ondrej Certik wrote: > > > >> Yes. > > > > Do you have any windows developers (I am sorry, I am not familiar at > > all with sympy)? > > > > My main concern with git are: > > - you may need the command line > > - the index can be confusing (you can avoid learning it at first, but > > still some errors won't make sense before you get it). > > - git is not discoverable (you need to read some documentation) > > I tried to install git on my computer, as git-svn seemed so nice. I > tried git-svn and it told me that some .pm files are missing. Why did > it install git-svn if some files are missing? And why isn't it > possible to get some information on how to install those missing files > on the Internet. > > git seems nice, but lacks documentation and usability :| > > Matthieu > The saying that git lacks documentation and usability was a long-long ago history in my eyes. Although I'm not a sympy developer nor numpy developer but I use git for my own project and enjoy working with it. You may find some useful documentation here: http://git-scm.com/ and a tutorial: http://www.kernel.org/pub/software/scm/git/docs/gittutorial.html What really lacks is a little bit learning and using. -- Cheers, Grissiom -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Thu Apr 9 10:15:33 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 9 Apr 2009 16:15:33 +0200 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <5b8d13220903302041m71cea034pf21b4a61d39eb6f@mail.gmail.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> Message-ID: 2009/4/9 David Cournapeau : > On Thu, Apr 9, 2009 at 10:38 PM, Matthieu Brucher > wrote: >> 2009/4/9 David Cournapeau : >>> On Wed, Apr 8, 2009 at 3:34 AM, Ondrej Certik wrote: >>> >>>> Yes. >>> >>> Do you have any windows developers (I am sorry, I am not familiar at >>> all with sympy)? >>> >>> My main concern with git are: >>> ?- you may need the command line >>> ?- the index can be confusing (you can avoid learning it at first, but >>> still some errors won't make sense before you get it). >>> ?- git is not discoverable (you need to read some documentation) >> >> I tried to install git on my computer, as git-svn seemed so nice. I >> tried git-svn and it told me that some .pm files are missing. Why did >> it install git-svn if some files are missing? And why isn't it >> possible to get some information on how to install those missing files >> on the Internet. > > For which system did you install it ? I don't think git-svn work well > or at all on windows. > > Otherwise, git has binary installers for every platform which matters. > Installing from sources is a bit harder than bzr/hg, but certainly > easier than svn, if you need to, Installed from source on a RH4. It complains about missing SVN/Core.pm. Matthieu -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From cournape at gmail.com Thu Apr 9 11:56:08 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 10 Apr 2009 00:56:08 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: References: <49CE3A2C.9000007@enthought.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> Message-ID: <5b8d13220904090856m7be5ccf6r4c2b8f7dc96fa4cd@mail.gmail.com> On Thu, Apr 9, 2009 at 11:15 PM, Matthieu Brucher > > Installed from source on a RH4. It complains about missing SVN/Core.pm. you need the perl wrapper for subversion installed, and to set the equivalent of PYTHONPATH for perl to it (sorry, the name of the variable eludes me ATM, but I had to do this on a CENTOS workstation once). cheers, David From bsouthey at gmail.com Thu Apr 9 12:34:47 2009 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 9 Apr 2009 11:34:47 -0500 Subject: [Numpy-discussion] Numpy 1.3 on AMD64 import error Message-ID: Hi, I can not import numpy 1.3 on AMD64 on Windows Vista 64 business but it does import with the numpy-1.3.0b1. I get the first error below either via the command line or using IDLE. The second error occurs in IDLE if I try a second time to import numpy but probably not important. Bruce >>> import numpy Warning (from warnings module): File "C:\Python26\lib\site-packages\numpy\core\__init__.py", line 5 import multiarray Warning: Windows 64 bits support is experimental, and only available for testing. You are advised not to use it for production. CRASHES ARE TO BE EXPECTED - PLEASE REPORT THEM TO NUMPY DEVELOPERS Traceback (most recent call last): File "", line 1, in import numpy File "C:\Python26\lib\site-packages\numpy\__init__.py", line 130, in import add_newdocs File "C:\Python26\lib\site-packages\numpy\add_newdocs.py", line 9, in from lib import add_newdoc File "C:\Python26\lib\site-packages\numpy\lib\__init__.py", line 13, in from polynomial import * File "C:\Python26\lib\site-packages\numpy\lib\polynomial.py", line 18, in from numpy.linalg import eigvals, lstsq File "C:\Python26\lib\site-packages\numpy\linalg\__init__.py", line 47, in from linalg import * File "C:\Python26\lib\site-packages\numpy\linalg\linalg.py", line 22, in from numpy.linalg import lapack_lite ImportError: DLL load failed: The specified module could not be found. >>> import numpy Traceback (most recent call last): File "", line 1, in import numpy File "C:\Python26\lib\site-packages\numpy\__init__.py", line 130, in import add_newdocs File "C:\Python26\lib\site-packages\numpy\add_newdocs.py", line 9, in from lib import add_newdoc File "C:\Python26\lib\site-packages\numpy\lib\__init__.py", line 13, in from polynomial import * File "C:\Python26\lib\site-packages\numpy\lib\polynomial.py", line 11, in import numpy.core.numeric as NX AttributeError: 'module' object has no attribute 'core' From Chris.Barker at noaa.gov Thu Apr 9 13:29:00 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 09 Apr 2009 10:29:00 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <49D136A3.80302@gmail.com> <5b8d13220903302041m71cea034pf21b4a61d39eb6f@mail.gmail.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> Message-ID: <49DE305C.7000702@noaa.gov> David Cournapeau wrote: > I don't think git-svn work well > or at all on windows. This brings up a key issue for Python: git does not support Windows very well -- which makes sense, given its history with Linux kernel development. I personally use SVN primarily form the command line on all systems, though Tortoise is very nice on Windows. It looks like there is indeed a TortoiseGit -- anyone know hoe complete/stable it is? Anyway, the multi-platform and multiple tools support of SVN is amazing, and this is a big deal -- we're really better off with a system with good tool support on all platforms. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From ondrej at certik.cz Thu Apr 9 13:58:56 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 9 Apr 2009 10:58:56 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: References: <49CE3A2C.9000007@enthought.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> Message-ID: <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> On Thu, Apr 9, 2009 at 7:15 AM, Matthieu Brucher wrote: > 2009/4/9 David Cournapeau : >> On Thu, Apr 9, 2009 at 10:38 PM, Matthieu Brucher >> wrote: >>> 2009/4/9 David Cournapeau : >>>> On Wed, Apr 8, 2009 at 3:34 AM, Ondrej Certik wrote: >>>> >>>>> Yes. >>>> >>>> Do you have any windows developers (I am sorry, I am not familiar at >>>> all with sympy)? >>>> >>>> My main concern with git are: >>>> ?- you may need the command line >>>> ?- the index can be confusing (you can avoid learning it at first, but >>>> still some errors won't make sense before you get it). >>>> ?- git is not discoverable (you need to read some documentation) >>> >>> I tried to install git on my computer, as git-svn seemed so nice. I >>> tried git-svn and it told me that some .pm files are missing. Why did >>> it install git-svn if some files are missing? And why isn't it >>> possible to get some information on how to install those missing files >>> on the Internet. >> >> For which system did you install it ? I don't think git-svn work well >> or at all on windows. >> >> Otherwise, git has binary installers for every platform which matters. >> Installing from sources is a bit harder than bzr/hg, but certainly >> easier than svn, if you need to, > > Installed from source on a RH4. It complains about missing SVN/Core.pm. One thing about git-svn is that this is not really needed if you just use git and I installed git from source on many linuxes and clusters and it just works, as it is just pure C. I usually just use git-svn on my laptop/workstation, where I install the Debian/Ubuntu packages, and I create the git repository, upload to github.com or somewhere else and just work with the git repository. But I agree that if it installs git-svn and it doesn't just work, it's a big problem. Ondrej From matthieu.brucher at gmail.com Thu Apr 9 14:01:16 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 9 Apr 2009 20:01:16 +0200 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> Message-ID: > One thing about git-svn is that this is not really needed if you just > use git and I installed git from source on many linuxes and clusters > and it just works, as it is just pure C. I usually just use git-svn on > my laptop/workstation, where I install the Debian/Ubuntu packages, and > I create the git repository, upload to github.com or somewhere else > and just work with the git repository. > > But I agree that if it installs git-svn and it doesn't just work, it's > a big problem. I was inquiring the use of git with the use of one of our internal svn repositories, just to have a feeling about it :( Matthieu -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From strawman at astraw.com Thu Apr 9 14:16:28 2009 From: strawman at astraw.com (Andrew Straw) Date: Thu, 09 Apr 2009 11:16:28 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: References: <49CE3A2C.9000007@enthought.com> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> Message-ID: <49DE3B7C.6080207@astraw.com> Matthieu Brucher wrote: >> One thing about git-svn is that this is not really needed if you just >> use git and I installed git from source on many linuxes and clusters >> and it just works, as it is just pure C. I usually just use git-svn on >> my laptop/workstation, where I install the Debian/Ubuntu packages, and >> I create the git repository, upload to github.com or somewhere else >> and just work with the git repository. >> >> But I agree that if it installs git-svn and it doesn't just work, it's >> a big problem. > > I was inquiring the use of git with the use of one of our internal svn > repositories, just to have a feeling about it :( My opinion is that attempting to use git-svn to get a feeling of git is not a good idea. There's too much slowness of svn involved, too much pain of trying to learn git while also trying to learn git-svn (which itself has corner cases and such that pure git doesn't) and there's no bidirectional 1:1 mapping between branches (that I've found), eliminating a huge part of the joy of git -- cheap branches. Better to start developing on a pure git project to get a feel. You can't go wrong with sympy, for example. :) -Andrew From ondrej at certik.cz Thu Apr 9 14:53:36 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 9 Apr 2009 11:53:36 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49DE3B7C.6080207@astraw.com> References: <49CE3A2C.9000007@enthought.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> <49DE3B7C.6080207@astraw.com> Message-ID: <85b5c3130904091153p4b3912a1o9fa7f1efc4cd1215@mail.gmail.com> On Thu, Apr 9, 2009 at 11:16 AM, Andrew Straw wrote: > Matthieu Brucher wrote: >>> One thing about git-svn is that this is not really needed if you just >>> use git and I installed git from source on many linuxes and clusters >>> and it just works, as it is just pure C. I usually just use git-svn on >>> my laptop/workstation, where I install the Debian/Ubuntu packages, and >>> I create the git repository, upload to github.com or somewhere else >>> and just work with the git repository. >>> >>> But I agree that if it installs git-svn and it doesn't just work, it's >>> a big problem. >> >> I was inquiring the use of git with the use of one of our internal svn >> repositories, just to have a feeling about it :( > > My opinion is that attempting to use git-svn to get a feeling of git is > not a good idea. There's too much slowness of svn involved, too much > pain of trying to learn git while also trying to learn git-svn (which > itself has corner cases and such that pure git doesn't) and there's no > bidirectional 1:1 mapping between branches (that I've found), > eliminating a huge part of the joy of git -- cheap branches. > > Better to start developing on a pure git project to get a feel. You > can't go wrong with sympy, for example. :) Oh, definitely. :) Here is a list of easy to fix issues: http://code.google.com/p/sympy/issues/list?q=label:EasyToFix if you want to learn it on some easy, but real world example. :) Ondrej From matthieu.brucher at gmail.com Thu Apr 9 14:56:42 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 9 Apr 2009 20:56:42 +0200 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <85b5c3130904091153p4b3912a1o9fa7f1efc4cd1215@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> <49DE3B7C.6080207@astraw.com> <85b5c3130904091153p4b3912a1o9fa7f1efc4cd1215@mail.gmail.com> Message-ID: >>> I was inquiring the use of git with the use of one of our internal svn >>> repositories, just to have a feeling about it :( >> >> My opinion is that attempting to use git-svn to get a feeling of git is >> not a good idea. There's too much slowness of svn involved, too much >> pain of trying to learn git while also trying to learn git-svn (which >> itself has corner cases and such that pure git doesn't) and there's no >> bidirectional 1:1 mapping between branches (that I've found), >> eliminating a huge part of the joy of git -- cheap branches. >> >> Better to start developing on a pure git project to get a feel. You >> can't go wrong with sympy, for example. :) > > Oh, definitely. :) Here is a list of easy to fix issues: > > http://code.google.com/p/sympy/issues/list?q=label:EasyToFix > > if you want to learn it on some easy, but real world example. :) Unfortunately, I'm investigating it for my professional use, and I don't have a decent Internet access (i.e. without proxies). Matthieu -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From josef.pktd at gmail.com Thu Apr 9 16:30:29 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 9 Apr 2009 16:30:29 -0400 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> Message-ID: <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> On Thu, Apr 9, 2009 at 1:58 PM, Ondrej Certik wrote: > On Thu, Apr 9, 2009 at 7:15 AM, Matthieu Brucher > wrote: >> 2009/4/9 David Cournapeau : >>> On Thu, Apr 9, 2009 at 10:38 PM, Matthieu Brucher >>> wrote: >>>> 2009/4/9 David Cournapeau : >>>>> On Wed, Apr 8, 2009 at 3:34 AM, Ondrej Certik wrote: >>>>> >>>>>> Yes. >>>>> >>>>> Do you have any windows developers (I am sorry, I am not familiar at >>>>> all with sympy)? >>>>> >>>>> My main concern with git are: >>>>> ?- you may need the command line >>>>> ?- the index can be confusing (you can avoid learning it at first, but >>>>> still some errors won't make sense before you get it). >>>>> ?- git is not discoverable (you need to read some documentation) >>>> >>>> I tried to install git on my computer, as git-svn seemed so nice. I >>>> tried git-svn and it told me that some .pm files are missing. Why did >>>> it install git-svn if some files are missing? And why isn't it >>>> possible to get some information on how to install those missing files >>>> on the Internet. >>> >>> For which system did you install it ? I don't think git-svn work well >>> or at all on windows. >>> >>> Otherwise, git has binary installers for every platform which matters. >>> Installing from sources is a bit harder than bzr/hg, but certainly >>> easier than svn, if you need to, >> >> Installed from source on a RH4. It complains about missing SVN/Core.pm. > > One thing about git-svn is that this is not really needed if you just > use git and I installed git from source on many linuxes and clusters > and it just works, as it is just pure C. I usually just use git-svn on > my laptop/workstation, where I install the Debian/Ubuntu packages, and > I create the git repository, upload to github.com or somewhere else > and just work with the git repository. > > But I agree that if it installs git-svn and it doesn't just work, it's > a big problem. > > Ondrej > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > git-svn is supposed to be available again on windows as part of msysgit. I had it installed, but I didn't out git-svn. I removed msysgit again since 1 GByte for this on my notebook with scarce drive space was a waste. The installer version of git doesn't have git-svn included. git citool and git gui look pretty good In the file browser, I'm still missing change dates, tree view doesn't have dates to quickly see which files got changed last, e.g. http://projects.scipy.org/gitweb?p=scipy;a=tree;h=refs/heads/trunk;hb=trunk for the simple usage that I tried, the windows version seems to work pretty well. git clone --origin svn http://projects.scipy.org/git/scipy.git scipy ends with error: Failed connect to projects.scipy.org:80; No error (curl_result = 7, http_ code = 0, sha1 = 9e55c6b0809e66049f808448a2ce3a1131e318fe) error: Unable to find 9e55c6b0809e66049f808448a2ce3a1131e318fe under http://proj ects.scipy.org/git/scipy.git Cannot obtain needed commit 9e55c6b0809e66049f808448a2ce3a1131e318fe while processing commit 7a429837e7ea88547711e5b3b5b9e38569fad553. fatal: Fetch failed. git clone git://github.com/pv/scipy-work.git worked without problems. but just pushing buttons on the gui is not enough I created a branch but on my file system nothing happened. So I started to read the tutorial, which says that all branches are in the same directory. How can I edit files on the file system if I don't know what branch I'm in? Is there a way to have branches in different directories? I don't want to have to start git all the time when I'm editing some files on a branch? I'm able to clone in a different directory, but I didn't see an obvious way in the gui to link/compare with another clone. I didn't manage to get a simple diff between two clones, and the error messages are still "very" informative, e.g. response to failed fetch in the gui is "fatal: unable to fork" "Error: Command failed". or on the command line: fetch seems to have worked but then (following the tutorial: C:\Josef\work-oth\!oth\testbranches\copyofpv>git log -p master..pv/master fatal: ambiguous argument 'master..pv/master': unknown revision or path not in t he working tree. ok, finally a pull from one clone to another worked, I was confused about the different branches in the same repository with different names, but still no diff, ... Josef From cournape at gmail.com Thu Apr 9 23:13:27 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 10 Apr 2009 12:13:27 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> Message-ID: <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> On Fri, Apr 10, 2009 at 5:30 AM, wrote: > > In the file browser, I'm still missing change dates, tree view doesn't > have dates to quickly see which files got changed last, ?e.g. > > http://projects.scipy.org/gitweb?p=scipy;a=tree;h=refs/heads/trunk;hb=trunk Yes, you can't see this cheaply with git (as it does not track individual files), but I don't really understand what's useful in this view. What matters is the commits, not the last changed files which come from different commits. > Cannot obtain needed commit 9e55c6b0809e66049f808448a2ce3a1131e318fe > while processing commit 7a429837e7ea88547711e5b3b5b9e38569fad553. > fatal: Fetch failed. That may be a problem with the connection, or our own git configuration. But connection errors in svn can be pretty cryptic too. I agree the above message is bad, but not worse than "MKACTIVITY not authorized" and the likes I consistently get with svn behind corporate firewall/proxies. At least with git I can continue working. > I created a branch but on my file system nothing happened. So I > started to read the tutorial, which says that all branches are in the > same directory. How can I edit files on the file system if I don't > know what branch I'm in? This may be a limitation of the GUI - in the command line, you simply do "git branch" to list your local branches (git status also tell you the branch you are currently in). > > Is there a way to have branches in different directories? I don't think you should start with all this. That's why our wiki page does not introduce branches right away: http://projects.scipy.org/numpy/wiki/GitWorkflow It enables you to do what you usually do with svn, without branches and the likes. > I don't want > to have to start git all the time when I'm editing some files on a > branch? You have to if you need to change between branches, there is no way around it. > > C:\Josef\work-oth\!oth\testbranches\copyofpv>git log -p master..pv/master > fatal: ambiguous argument 'master..pv/master': unknown revision or path not in t > he working tree. That's because you don't have both branches in your repository. You need branches to compare in the same repository (directory). You can list branches with git branch -a, and then git log branch1..branch2 will work for any two branches listed. Same for any other command. But this is already a bit advanced usage (and again, it is not like doing branch stuff with svn is any easier - the svn documentation is very cryptic too on that matter). I think what is relevant is whether the simple svn workflow can be done with git. Trying to do branches as in svn won't work, as it is fundamentally different. cheers, David From cournape at gmail.com Thu Apr 9 23:18:02 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 10 Apr 2009 12:18:02 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: References: <49CE3A2C.9000007@enthought.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> <49DE3B7C.6080207@astraw.com> <85b5c3130904091153p4b3912a1o9fa7f1efc4cd1215@mail.gmail.com> Message-ID: <5b8d13220904092018w7e6852cfld17b16ba74df0c65@mail.gmail.com> On Fri, Apr 10, 2009 at 3:56 AM, Matthieu Brucher wrote: > > Unfortunately, I'm investigating it for my professional use, and I > don't have a decent Internet access (i.e. without proxies). If you are behind a proxy, you have to use the (slower) http protocol. That's actually a good argument in favor of mercurial I think, because the fast protocol is based on http (but I am not sure https work). Those proxy matters are indeed a real pain (I could not use svn at all at a former company, for example). I don't think there is any easy fix. cheers, David From cournape at gmail.com Thu Apr 9 23:35:22 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 10 Apr 2009 12:35:22 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49DE305C.7000702@noaa.gov> References: <49CE3A2C.9000007@enthought.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <49DE305C.7000702@noaa.gov> Message-ID: <5b8d13220904092035i2a3b1af8l8544584e6724fec1@mail.gmail.com> On Fri, Apr 10, 2009 at 2:29 AM, Christopher Barker wrote: > David Cournapeau wrote: >> I don't think git-svn work well >> or at all on windows. > > This brings up a key issue for Python: git does not support Windows very > well -- which makes sense, given its history with Linux kernel development. > > I personally use SVN primarily form the command line on all systems, > though Tortoise is very nice on Windows. It looks like there is indeed a > TortoiseGit -- anyone know hoe complete/stable it is? > > Anyway, the multi-platform and multiple tools support of SVN is amazing, > and this is a big deal -- we're really better off with a system with > good tool support on all platforms. Why ? We are not python, where many core developers work on windows. The git command line works well on windows (same as on other systems), I used it while testing things for numpy 1.3.0 (locally, though because of the git-svn issue). There will always be arguments for svn being more supported, but doesn't this sounds like matlab vs numpy/scipy ? There will always be people who will find the scipy stack not good, not integrated - but still, we use numpy/scipy. git is not better than svn in every way, but at least to me, the question is more "is git better than svn overall", and whether a majority of contributors would agree on the answer being the affirmative. cheers, David From geometrian at gmail.com Fri Apr 10 00:18:15 2009 From: geometrian at gmail.com (Ian Mallett) Date: Thu, 9 Apr 2009 21:18:15 -0700 Subject: [Numpy-discussion] Another Array Message-ID: Hello, With the help of this list, over the past two days, I have implemented a GPU particle system (picture here: http://img7.imageshack.us/img7/7589/image1olu.png). Every particle is updated entirely on the GPU, with texture data (arrays) being updated iteratively between two framebuffer objects. Here, 65,536 particles are drawn at ~50 fps. Currently, jitter is introduced into the system by means of a "random texture", which should store random unit 3D vectors encoded into the RGB channels (vectors point in all directions, positive and negative, so the vecotr is actually 0.5 long and added to 0.5). The random texture is constructed using Python's random number generation, which is noticeably slower. I also didn't find an efficient way of creating "random" vectors. All this contributes to the pattern artifact at the bottom of the screenshot. I'd like to use NumPy. The array should be constructed as n*n*3, with every vec3 being a normalized random vector with each component in the range [-1.0,1.0]. I could use numpy.random.random(), but this would not give the appropriate vectors. Every 3D vector should have the same probability of being chosen. I don't know how to do that in NumPy--that's where I need help. Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Apr 10 00:26:16 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 9 Apr 2009 23:26:16 -0500 Subject: [Numpy-discussion] Another Array In-Reply-To: References: Message-ID: <3d375d730904092126kb7fd796v4d9bf0dea9ca7285@mail.gmail.com> On Thu, Apr 9, 2009 at 23:18, Ian Mallett wrote: > Hello, > > With the help of this list, over the past two days, I have implemented a GPU > particle system (picture here: > http://img7.imageshack.us/img7/7589/image1olu.png).? Every particle is > updated entirely on the GPU, with texture data (arrays) being updated > iteratively between two framebuffer objects.? Here, 65,536 particles are > drawn at ~50 fps. > > Currently, jitter is introduced into the system by means of a "random > texture", which should store random unit 3D vectors encoded into the RGB > channels (vectors point in all directions, positive and negative, so the > vecotr is actually 0.5 long and added to 0.5).? The random texture is > constructed using Python's random number generation, which is noticeably > slower.? I also didn't find an efficient way of creating "random" vectors. > All this contributes to the pattern artifact at the bottom of the > screenshot.? I'd like to use NumPy. > > The array should be constructed as n*n*3, with every vec3 being a normalized > random vector with each component in the range [-1.0,1.0].? I could use > numpy.random.random(), but this would not give the appropriate vectors. > Every 3D vector should have the same probability of being chosen.? I don't > know how to do that in NumPy--that's where I need help. A quick trick, normalizing standard multivariate Gaussians yields uniformly sampled unit vectors: vecs = numpy.random.standard_normal(size=(n,n,3)) magnitudes = numpy.sqrt((vecs*vecs).sum(axis=-1)) uvecs = vecs / magnitudes[...,newaxis] rgb = (uvecs + 1.0) / 2.0 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From geometrian at gmail.com Fri Apr 10 00:37:43 2009 From: geometrian at gmail.com (Ian Mallett) Date: Thu, 9 Apr 2009 21:37:43 -0700 Subject: [Numpy-discussion] Another Array In-Reply-To: <3d375d730904092126kb7fd796v4d9bf0dea9ca7285@mail.gmail.com> References: <3d375d730904092126kb7fd796v4d9bf0dea9ca7285@mail.gmail.com> Message-ID: This works; thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Apr 10 01:19:15 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 10 Apr 2009 01:19:15 -0400 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> Message-ID: <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> On Thu, Apr 9, 2009 at 11:13 PM, David Cournapeau wrote: > On Fri, Apr 10, 2009 at 5:30 AM, ? wrote: > >> >> In the file browser, I'm still missing change dates, tree view doesn't >> have dates to quickly see which files got changed last, ?e.g. >> >> http://projects.scipy.org/gitweb?p=scipy;a=tree;h=refs/heads/trunk;hb=trunk > > Yes, you can't see this cheaply with git (as it does not track > individual files), but I don't really understand what's useful in this > view. What matters is the commits, not the last changed files which > come from different commits. > >> Cannot obtain needed commit 9e55c6b0809e66049f808448a2ce3a1131e318fe >> while processing commit 7a429837e7ea88547711e5b3b5b9e38569fad553. >> fatal: Fetch failed. > > That may be a problem with the connection, or our own git > configuration. But connection errors in svn can be pretty cryptic too. > I agree the above message is bad, but not worse than "MKACTIVITY not > authorized" and the likes I consistently get with svn behind corporate > firewall/proxies. > > At least with git I can continue working. > >> I created a branch but on my file system nothing happened. So I >> started to read the tutorial, which says that all branches are in the >> same directory. How can I edit files on the file system if I don't >> know what branch I'm in? > > This may be a limitation of the GUI - in the command line, you simply > do "git branch" to list your local branches (git status also tell you > the branch you are currently in). > >> >> Is there a way to have branches in different directories? > > I don't think you should start with all this. That's why our wiki page > does not introduce branches right away: > > http://projects.scipy.org/numpy/wiki/GitWorkflow > > It enables you to do what you usually do with svn, without branches > and the likes. > >> I don't want >> to have to start git all the time when I'm editing some files on a >> branch? > > You have to if you need to change between branches, there is no way around it. > >> >> C:\Josef\work-oth\!oth\testbranches\copyofpv>git log -p master..pv/master >> fatal: ambiguous argument 'master..pv/master': unknown revision or path not in t >> he working tree. > > That's because you don't have both branches in your repository. You > need branches to compare in the same repository (directory). You can > list branches with git branch -a, and then git log branch1..branch2 > will work for any two branches listed. > > Same for any other command. But this is already a bit advanced usage > (and again, it is not like doing branch stuff with svn is any easier - > the svn documentation is very cryptic too on that matter). I think > what is relevant is whether the simple svn workflow can be done with > git. Trying to do branches as in svn won't work, as it is > fundamentally different. > I was comparing this more to my short experience with bazar, after a few hours, I had a scipy checkout and several branches in different directories, and could relatively easily compare and merge and pull between the different branches. I'm very used to working out of windows explorer as my main file access, and when I quickly edit something, I don't need to tell bzr or svn which version I am working on, I know it from the directory structure and can add and commit at some later point. So, for my style, working with different clones instead of branches seems easier. And for example, pysvn workbench lets me browse the directory tree showing the change and commit status per directory and diff to head and branches for each file in the gui. The history browser seems much better in the git gui. I managed to finally also get a diff between clones on the command line. but in the gui, I still get the unable to fork error. The git gui seems to still need a bit of work to be useful. I'm not behind a proxy, just behind the wireless router, and I never had problems with svn. Josef From david at ar.media.kyoto-u.ac.jp Fri Apr 10 01:25:59 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 10 Apr 2009 14:25:59 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> Message-ID: <49DED867.5050206@ar.media.kyoto-u.ac.jp> josef.pktd at gmail.com wrote: > So, for my > style, working with different clones instead of branches seems easier. > Yes, it is. There is no denying that git makes it more difficult for this workflow, and that git is more difficult at first for easy tasks. I am interested in making it as easy as possible, and if it is indeed too difficult, then maybe git is out. > I'm not behind a proxy, just behind the wireless router, and I never > had problems with svn. > You're right, I tried on windows and Linux from my machine, and the error is windows only, and not related to a proxy. David From ondrej at certik.cz Fri Apr 10 01:56:17 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 9 Apr 2009 22:56:17 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49DED867.5050206@ar.media.kyoto-u.ac.jp> References: <49CE3A2C.9000007@enthought.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> Message-ID: <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> On Thu, Apr 9, 2009 at 10:25 PM, David Cournapeau wrote: > josef.pktd at gmail.com wrote: >> So, for my >> style, working with different clones instead of branches seems easier. >> > > Yes, it is. ?There is no denying that git makes it more difficult for > this workflow, and that git is more difficult at first for easy tasks. I > am interested in making it as easy as possible, and if it is indeed too > difficult, then maybe git is out. It is maybe easier to learn how to work with different clones, but once you start working with lots of patches and you need to reclone all the time, then it's the wrong approach to work, as it takes lots of time to copy the whole repository on the disk. I worked like that with mercurial and I hated it, especially once I started to have ~15 branches. Git branches are cheap and fast. Besides, imho, you can work with different clones with git too, if you want. Ondrej From david at ar.media.kyoto-u.ac.jp Fri Apr 10 01:45:05 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 10 Apr 2009 14:45:05 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> Message-ID: <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> Ondrej Certik wrote: > > It is maybe easier to learn how to work with different clones, but > once you start working with lots of patches and you need to reclone > all the time, then it's the wrong approach to work, as it takes lots > of time to copy the whole repository on the disk. Yes, *I* know how to use git, and I agree with you, I vastly prefer git branch handling to bzr branch handling. *I* find working with GUI for VCS a real PITA. But I am not the only numpy developer, that's why the feedback from people like Josef with a totally different workflow than me is valuable - much more than people like us who are unix geeks :) David From ondrej at certik.cz Fri Apr 10 02:14:53 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 9 Apr 2009 23:14:53 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> References: <49CE3A2C.9000007@enthought.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> Message-ID: <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> On Thu, Apr 9, 2009 at 10:45 PM, David Cournapeau wrote: > Ondrej Certik wrote: >> >> It is maybe easier to learn how to work with different clones, but >> once you start working with lots of patches and you need to reclone >> all the time, then it's the wrong approach to work, as it takes lots >> of time to copy the whole repository on the disk. > > Yes, *I* know how to use git, and I agree with you, I vastly prefer git > branch handling to bzr branch handling. *I* find working with GUI for > VCS a real PITA. But I am not the only numpy developer, that's why the > feedback from people like Josef with a totally different workflow than > me is valuable - much more than people like us who are unix geeks :) Yes, definitely. Ondrej From dlenski at gmail.com Fri Apr 10 02:36:08 2009 From: dlenski at gmail.com (Dan Lenski) Date: Fri, 10 Apr 2009 06:36:08 +0000 (UTC) Subject: [Numpy-discussion] using reducing functions without eliminating dimensions? References: <3d375d730904082331g55d21e5el517591127fc19384@mail.gmail.com> Message-ID: On Thu, 09 Apr 2009 01:31:33 -0500, Robert Kern wrote: > On Thu, Apr 9, 2009 at 01:29, Anne Archibald > wrote: > >> What's wrong with np.amin(a,axis=-1)[...,np.newaxis]? > > It's cumbersome, particularly when you have axis=arbitrary_axis. Quite right. It would nice to be able to say: np.amin(a, axiskeep=-1) or a.min(axiskeep=3) ... or something along those lines. Dan From geometrian at gmail.com Fri Apr 10 02:42:03 2009 From: geometrian at gmail.com (Ian Mallett) Date: Thu, 9 Apr 2009 23:42:03 -0700 Subject: [Numpy-discussion] Another Array In-Reply-To: References: <3d375d730904092126kb7fd796v4d9bf0dea9ca7285@mail.gmail.com> Message-ID: It gives a perfect parabolic shape that looks very nice, but somewhat unrealistic. I'd like to scale the unit vectors by a random length (which can just be a uniform distribution). I tried scaling the unit vector n*n*3 array by a random n*n array, but that didn't work, obviously. Help? -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Apr 10 02:46:47 2009 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 10 Apr 2009 01:46:47 -0500 Subject: [Numpy-discussion] Another Array In-Reply-To: References: <3d375d730904092126kb7fd796v4d9bf0dea9ca7285@mail.gmail.com> Message-ID: <3d375d730904092346l6850c03chca1962915263e79c@mail.gmail.com> On Fri, Apr 10, 2009 at 01:42, Ian Mallett wrote: > It gives a perfect parabolic shape that looks very nice, but somewhat > unrealistic. Parabolic? They should be spherical. > I'd like to scale the unit vectors by a random length (which > can just be a uniform distribution).? I tried scaling the unit vector n*n*3 > array by a random n*n array, but that didn't work, obviously. No, it's not obvious. Exactly what code did you try? What results did you get? What results were you expecting? > Help? Let's take a step back. What kind of distribution are you trying to achieve? You asked for uniformly distributed unit vectors. Now you are asking for something else, but I'm really not sure what. What standard are you comparing against when you say that the unit vectors look "unrealistic"? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From geometrian at gmail.com Fri Apr 10 02:58:21 2009 From: geometrian at gmail.com (Ian Mallett) Date: Thu, 9 Apr 2009 23:58:21 -0700 Subject: [Numpy-discussion] Another Array In-Reply-To: <3d375d730904092346l6850c03chca1962915263e79c@mail.gmail.com> References: <3d375d730904092126kb7fd796v4d9bf0dea9ca7285@mail.gmail.com> <3d375d730904092346l6850c03chca1962915263e79c@mail.gmail.com> Message-ID: On Thu, Apr 9, 2009 at 11:46 PM, Robert Kern wrote: > Parabolic? They should be spherical. The particle system in the last screenshot was affected by gravity. In the absence of gravity, the results should be spherical, yes. All the vectors are a unit length, which produces a perfectly smooth surface (unrealistic for such an effect). > No, it's not obvious. Exactly what code did you try? What results did > you get? What results were you expecting? It crashed. I have this code: vecs = Numeric.random.standard_normal(size=(self.size[0],self.size[1],3)) magnitudes = Numeric.sqrt((vecs*vecs).sum(axis=-1)) uvecs = vecs / magnitudes[...,Numeric.newaxis] randlen = Numeric.random.random((self.size[0],self.size[1])) randuvecs = uvecs*randlen #It crashes here with a dimension mismatch rgb = ((randvecs+1.0)/2.0)*255.0 I also tried randlen = Numeric.random.random((self.size[0],self.size[1],3)), but this does not scale each of the vector's components equally, producing artifacts again. Each needs to be scaled by the same random value for it to make sense. > Let's take a step back. What kind of distribution are you trying to > achieve? You asked for uniformly distributed unit vectors. Now you are > asking for something else, but I'm really not sure what. What standard > are you comparing against when you say that the unit vectors look > "unrealistic"? The vectors are used to "jitter" each particle's initial speed, so that the particles go in different directions instead of moving all as one. Using the unit vector causes the particles to make the smooth parabolic shape. The jitter vectors much then be of a random length, so that the particles go in all different directions at all different speeds, instead of just all in different directions. Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Apr 10 03:01:10 2009 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 10 Apr 2009 02:01:10 -0500 Subject: [Numpy-discussion] Another Array In-Reply-To: References: <3d375d730904092126kb7fd796v4d9bf0dea9ca7285@mail.gmail.com> <3d375d730904092346l6850c03chca1962915263e79c@mail.gmail.com> Message-ID: <3d375d730904100001w4cdb7bedi2b36bca97731ef8c@mail.gmail.com> On Fri, Apr 10, 2009 at 01:58, Ian Mallett wrote: > On Thu, Apr 9, 2009 at 11:46 PM, Robert Kern wrote: >> >> Parabolic? They should be spherical. > > The particle system in the last screenshot was affected by gravity.? In the > absence of gravity, the results should be spherical, yes.? All the vectors > are a unit length, which produces a perfectly smooth surface (unrealistic > for such an effect). >> >> No, it's not obvious. Exactly what code did you try? What results did >> you get? What results were you expecting? > > It crashed. > I have this code: > vecs = Numeric.random.standard_normal(size=(self.size[0],self.size[1],3)) > magnitudes = Numeric.sqrt((vecs*vecs).sum(axis=-1)) > uvecs = vecs / magnitudes[...,Numeric.newaxis] > randlen = Numeric.random.random((self.size[0],self.size[1])) > randuvecs = uvecs*randlen #It crashes here with a dimension mismatch > rgb = ((randvecs+1.0)/2.0)*255.0 > > I also tried randlen = Numeric.random.random((self.size[0],self.size[1],3)), > but this does not scale each of the vector's components equally, producing > artifacts again.? Each needs to be scaled by the same random value for it to > make sense. See how I did magnitudes[...,numpy.newaxis]? You have to do the same. >> Let's take a step back. What kind of distribution are you trying to >> achieve? You asked for uniformly distributed unit vectors. Now you are >> asking for something else, but I'm really not sure what. What standard >> are you comparing against when you say that the unit vectors look >> "unrealistic"? > > The vectors are used to "jitter" each particle's initial speed, so that the > particles go in different directions instead of moving all as one.? Using > the unit vector causes the particles to make the smooth parabolic shape. > The jitter vectors much then be of a random length, so that the particles go > in all different directions at all different speeds, instead of just all in > different directions. Ah, okay. That makes sense. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From geometrian at gmail.com Fri Apr 10 03:07:45 2009 From: geometrian at gmail.com (Ian Mallett) Date: Fri, 10 Apr 2009 00:07:45 -0700 Subject: [Numpy-discussion] Another Array In-Reply-To: <3d375d730904100001w4cdb7bedi2b36bca97731ef8c@mail.gmail.com> References: <3d375d730904092126kb7fd796v4d9bf0dea9ca7285@mail.gmail.com> <3d375d730904092346l6850c03chca1962915263e79c@mail.gmail.com> <3d375d730904100001w4cdb7bedi2b36bca97731ef8c@mail.gmail.com> Message-ID: This seems to work: vecs = Numeric.random.standard_normal(size=(self.size[0],self.size[1],3)) magnitudes = Numeric.sqrt((vecs*vecs).sum(axis=-1)) uvecs = vecs / magnitudes[...,Numeric.newaxis] randlen = Numeric.random.random((self.size[0],self.size[1])) randuvecs = uvecs*randlen[...,Numeric.newaxis] rgb = ((randuvecs+1.0)/2.0)*255.0 (I have "import numpy as Numeric" for other reasons, that's why there's Numeric there). Thanks, Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Fri Apr 10 03:17:58 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 10 Apr 2009 03:17:58 -0400 Subject: [Numpy-discussion] Another Array In-Reply-To: References: <3d375d730904092126kb7fd796v4d9bf0dea9ca7285@mail.gmail.com> <3d375d730904092346l6850c03chca1962915263e79c@mail.gmail.com> Message-ID: 2009/4/10 Ian Mallett : > The vectors are used to "jitter" each particle's initial speed, so that the > particles go in different directions instead of moving all as one.? Using > the unit vector causes the particles to make the smooth parabolic shape. > The jitter vectors much then be of a random length, so that the particles go > in all different directions at all different speeds, instead of just all in > different directions. Why not just skip the normalization? Then you'll get vectors with random direction and a natural distribution of lengths. And it'll be faster than tne unit vectors... Anne From roland at utk.edu Fri Apr 10 03:15:51 2009 From: roland at utk.edu (Roland Schulz) Date: Fri, 10 Apr 2009 03:15:51 -0400 Subject: [Numpy-discussion] frombuffer alignment for ctypes.Structure array In-Reply-To: References: Message-ID: Hi, I would like to create a ndarray view on an array of ctypes.Structure. (reason: see below). The problem is that the structure size is not falling on an alignment boundary and thus the size is bigger than its parts. A sample showing the problem: import numpy as N from ctypes import * class C(Structure): _fields_=[("a",c_int),("b",c_short)] c=(C*2)() _ctypes_to_numpy = {c_short : N.int16,c_int : N.int32} ty = [(x,_ctypes_to_numpy[y]) for x,y in C._fields_] x=N.frombuffer(c,ty) The is no align or aligned option to frombuffer. What is the best way to tell numpy to align the data as the C-struct/ctypes.Stucture array is? reason: I have an array which get allocated in the external C library and thus I get a pointer on structs from that C function. The only way I know how to create a ndarray view for pointers on data, is to first convert the ctypes pointer to a ctypes array. I convert the POINTER(t_X) to an ctypes array of the structre using the (t_X*N).from_address(addressof(X.contents)) trick, where t_X is the data type of the struct and X the pointer to the array. Thanks Roland -- ORNL/UT Center for Molecular Biophysics cmb.ornl.gov 865-241-1537, ORNL PO BOX 2008 MS6309 -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Apr 10 03:21:59 2009 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 10 Apr 2009 02:21:59 -0500 Subject: [Numpy-discussion] Another Array In-Reply-To: References: <3d375d730904092126kb7fd796v4d9bf0dea9ca7285@mail.gmail.com> <3d375d730904092346l6850c03chca1962915263e79c@mail.gmail.com> Message-ID: <3d375d730904100021r2fca2e84m4ea781e3a7c12eb5@mail.gmail.com> On Fri, Apr 10, 2009 at 02:17, Anne Archibald wrote: > 2009/4/10 Ian Mallett : > >> The vectors are used to "jitter" each particle's initial speed, so that the >> particles go in different directions instead of moving all as one.? Using >> the unit vector causes the particles to make the smooth parabolic shape. >> The jitter vectors much then be of a random length, so that the particles go >> in all different directions at all different speeds, instead of just all in >> different directions. > > Why not just skip the normalization? Then you'll get vectors with > random direction and a natural distribution of lengths. And it'll be > faster than tne unit vectors... They are going to be forced into a uint8 texture, so they have a limited dynamic range. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From roland at utk.edu Fri Apr 10 03:12:41 2009 From: roland at utk.edu (Roland Schulz) Date: Fri, 10 Apr 2009 03:12:41 -0400 Subject: [Numpy-discussion] frombuffer alignment for ctypes.Structure array Message-ID: Hi, I would like to create a ndarray view on an array of ctypes.Structure. (reason: see below). The problem is that the structure size is not falling on an alignment boundary and thus the size is bigger than its parts. A sample showing the problem: import numpy as N from ctypes import * class C(Structure): _fields_=[("a",c_int),("b",c_short)] c=(C*2)() _ctypes_to_numpy = {c_short : N.int16,c_int : N.int32} ty = [(x,_ctypes_to_numpy[y]) for x,y in C._fields_] x=N.frombuffer(c,ty) The is no align or aligned option to frombuffer. What is the best way to tell numpy to align the data as the C-struct/ctypes.Stucture array is? reason: I have an array which get allocated in the external C library and thus I get a pointer on structs from that C function. The only way I know how to create a ndarray view for pointers on data, is to first convert the ctypes pointer to a ctypes array. I convert the POINTER(t_X) to an ctypes array of the structre using the (t_X*N).from_address(addressof(X.contents)) trick, where t_X is the data type of the struct and X the pointer to the array. Thanks Roland -- ORNL/UT Center for Molecular Biophysics cmb.ornl.gov 865-241-1537, ORNL PO BOX 2008 MS6309 -------------- next part -------------- An HTML attachment was scrubbed... URL: From drnlmuller+scipy at gmail.com Fri Apr 10 03:35:56 2009 From: drnlmuller+scipy at gmail.com (Neil Muller) Date: Fri, 10 Apr 2009 09:35:56 +0200 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> Message-ID: On Fri, Apr 10, 2009 at 7:56 AM, Ondrej Certik wrote: > It is maybe easier to learn how to work with different clones, but > once you start working with lots of patches and you need to reclone > all the time, then it's the wrong approach to work, as it takes lots > of time to copy the whole repository on the disk. I worked like that > with mercurial and I hated it, especially once I started to have ~15 > branches. There are also tools like git-new-workdir (in git's contrib section) that allow you to keep separate working directories for different branches, without needing multiple clones. It's written in shell, but, AFAIK, it works fine on windows using cygwin. yap (yet another porcelain) also provides this functionality. easy-git doesn't, but as a thin layer, it should work fine with git-new-workdir -- Neil Muller drnlmuller at gmail.com I've got a gmail account. Why haven't I become cool? From efiring at hawaii.edu Fri Apr 10 03:43:25 2009 From: efiring at hawaii.edu (Eric Firing) Date: Thu, 09 Apr 2009 21:43:25 -1000 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> Message-ID: <49DEF89D.7000605@hawaii.edu> Ondrej Certik wrote: > On Thu, Apr 9, 2009 at 10:45 PM, David Cournapeau > wrote: >> Ondrej Certik wrote: >>> It is maybe easier to learn how to work with different clones, but >>> once you start working with lots of patches and you need to reclone >>> all the time, then it's the wrong approach to work, as it takes lots >>> of time to copy the whole repository on the disk. This is simply wrong. Mercurial uses hard links for cloning a repo that is on the same disk, so it is faster and much more space-efficient than copying the files. But if you do want named branches in a given repo, you can have that also with hg. Granted, it has not always been part of hg, but it is now. Same with rebasing and transplanting. Now, I know that your (speaking to Ondrej) project switched from hg to git, and you have provided some useful insight as to why. I also understand that there are substantive differences between the two, with advantages and disadvantages. But I don't think it follows that numpy (or matplotlib, eventually, I hope) necessarily should move to git if/when a move to a DVCS is decided upon. It is possible that hg might be a better fit--a better compromise--for present *and* *potential* *future* contributors to numpy, scipy, and matplotlib. >> Yes, *I* know how to use git, and I agree with you, I vastly prefer git >> branch handling to bzr branch handling. *I* find working with GUI for >> VCS a real PITA. But I am not the only numpy developer, that's why the >> feedback from people like Josef with a totally different workflow than >> me is valuable - much more than people like us who are unix geeks :) > Speaking to David: is git branch handling vastly preferable to both of the branch styles available in hg? Speaking to Josef: does tortoise-hg provide a satisfactory windows gui, from your standpoint? Eric > Yes, definitely. > > Ondrej > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From ondrej at certik.cz Fri Apr 10 04:03:52 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 10 Apr 2009 01:03:52 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49DEF89D.7000605@hawaii.edu> References: <49CE3A2C.9000007@enthought.com> <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> Message-ID: <85b5c3130904100103t64c968ebm24e53d59c0b7238e@mail.gmail.com> On Fri, Apr 10, 2009 at 12:43 AM, Eric Firing wrote: > Ondrej Certik wrote: >> On Thu, Apr 9, 2009 at 10:45 PM, David Cournapeau >> wrote: >>> Ondrej Certik wrote: >>>> It is maybe easier to learn how to work with different clones, but >>>> once you start working with lots of patches and you need to reclone >>>> all the time, then it's the wrong approach to work, as it takes lots >>>> of time to copy the whole repository on the disk. > > This is simply wrong. ?Mercurial uses hard links for cloning a repo that On my laptop, recloning the whole repository (with hardlinks) takes considerably longer than creating a new branch in the same directory, that's a pure fact. > is on the same disk, so it is faster and much more space-efficient than > copying the files. But if you do want named branches in a given repo, > you can have that also with hg. ?Granted, it has not always been part of > hg, but it is now. ?Same with rebasing and transplanting. As far as I know mercurial doesn't have interactive rebase. Besides, the default package in Ubuntu Jaunty doesn't even have the (noninteractive) rebase enabled. So I think it will still take quite some (lot?) of time until mercurial has all those tools available by default. > > Now, I know that your (speaking to Ondrej) project switched from hg to > git, and you have provided some useful insight as to why. ?I also > understand that there are substantive differences between the two, with > advantages and disadvantages. But I don't think it follows that numpy > (or matplotlib, eventually, I hope) necessarily should move to git I never said numpy should move to git because I like git. > if/when a move to a DVCS is decided upon. ?It is possible that hg might > be a better fit--a better compromise--for present *and* *potential* > *future* contributors to numpy, scipy, and matplotlib. Yes, it may be possible. Ondrej From david at ar.media.kyoto-u.ac.jp Fri Apr 10 04:07:46 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 10 Apr 2009 17:07:46 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49DEF89D.7000605@hawaii.edu> References: <49CE3A2C.9000007@enthought.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> Message-ID: <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> Eric Firing wrote: > > This is simply wrong. Mercurial uses hard links for cloning a repo that > is on the same disk, so it is faster and much more space-efficient than > copying the files. Yes, but maybe Ondrej talks about an older hg version ? Hg could not handle NTFS hardlink for some time, but it does now if you have pywin32. And still, switching between branches is faster in git than in hg, for some technical reasons I can't really explain (but can be found on the ML archives). But as I said previously, speed is not really an argument for me. If hg is fast enough for python, it is obviously fast enough for numpy and scipy. As long as it does not takes minutes to merge/review the 5 lines difference between two branches as is the case in svn right now, I am happier :) > But if you do want named branches in a given repo, > you can have that also with hg. Granted, it has not always been part of > hg, but it is now. Same with rebasing and transplanting. > As I understand, and correct me if I am wrong, the problems with named branches are: - you can't remove them later, it is in the repository 'forever' - it is not easy to make them publicly available One big potential with the way git does branching is review - gerrit (the code review tool used by Google for android) looks really nice. Unfortunately, to see whether this is indeed true requires trying it, because git-svn cannot be used to that effect (because of branches). What would be great is to have git-svnserver, like git-cvsserver (which is a gateway for cvs, that is people who don't want to bother can use cvs to do as they do normally, and other people can use git proper). I find it surprising that such a tool does not exist. > It is possible that hg might > be a better fit--a better compromise--for present *and* *potential* > *future* contributors to numpy, scipy, and matplotlib. > Yes, that may be the case. > Speaking to David: is git branch handling vastly preferable to both of > the branch styles available in hg? > Besides git, I am mostly familiar with bzr, which has the "branch is a different directory" concept (but no named branches). I think bzr is very confusing for advanced workflows in relation to this UI (I am too lazy to restate my reasons, see here: http://cournape.wordpress.com/2008/10/30/going-away-from-bzr-toward-git). When I tried hg (version 0.8 maybe ?), there was little support for named branches, but this has changed since I believe. When you have different repositories for different branches, comparisons between branches are difficult. This was the case in bzr. Is this the case in hg ? For example, can you log a subdirectory between two branches, easily review files changes between branches/tags/etc... For the release process, I find this very useful - but this can be somewhat alleviated with git-svn without forcing anyone using it. If people think that at least trying hg should be possible, I think it should not be too difficult to have a hg mirror of svn - but I am not sure whether you can use it to commit back to svn, like git-svn. David From david at ar.media.kyoto-u.ac.jp Fri Apr 10 04:28:23 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 10 Apr 2009 17:28:23 +0900 Subject: [Numpy-discussion] frombuffer alignment for ctypes.Structure array In-Reply-To: References: Message-ID: <49DF0327.5010209@ar.media.kyoto-u.ac.jp> Roland Schulz wrote: > The is no align or aligned option to frombuffer. What is the best way > to tell numpy to align the data as the C-struct/ctypes.Stucture array is? You could add a 'fake' field in between to get the right alignment, maybe ? import numpy as N from ctypes import * class C(Structure): _fields_=[("a",c_int),("b",c_short), ('', c_short)] c=(C*2)() _ctypes_to_numpy = {c_short : N.int16,c_int : N.int32} ty = N.dtype([(x,_ctypes_to_numpy[y]) for x,y in C._fields_]) x=N.frombuffer(c,ty) You may use a different type if you need another alignment, or using something like 'VN' in the dtype, e.g.: a = N.dtype([('a', np.int), ('b', np.short), ('', 'V2')]) David From ondrej at certik.cz Fri Apr 10 04:54:15 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 10 Apr 2009 01:54:15 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> References: <49CE3A2C.9000007@enthought.com> <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> Message-ID: <85b5c3130904100154l797d062fn90ea9389bc2da0b@mail.gmail.com> On Fri, Apr 10, 2009 at 1:07 AM, David Cournapeau wrote: > Eric Firing wrote: >> >> This is simply wrong. ?Mercurial uses hard links for cloning a repo that >> is on the same disk, so it is faster and much more space-efficient than >> copying the files. > > Yes, but maybe Ondrej talks about an older hg version ? Hg could not > handle NTFS hardlink for some time, but it does now if you have pywin32. > > And still, switching between branches is faster in git than in hg, for > some technical reasons I can't really explain (but can be found on the > ML archives). But as I said previously, speed is not really an argument > for me. If hg is fast enough for python, it is obviously fast enough for > numpy and scipy. As long as it does not takes minutes to merge/review > the 5 lines difference between two branches as is the case in svn right > now, I am happier :) > >> ?But if you do want named branches in a given repo, >> you can have that also with hg. ?Granted, it has not always been part of >> hg, but it is now. ?Same with rebasing and transplanting. >> > > As I understand, and correct me if I am wrong, the problems with named > branches are: > ? ?- you can't remove them later, it is in the repository 'forever' > ? ?- it is not easy to make them publicly available Plus with git, you can fetch the remote repository with all the branches and browse them locally in your remote branches, when you are offline. And merge them with your own branches. In mercurial, it seems the only way to see what changes are there and which branch and which commits I want to merge is to use "hg in", but that requires the internet connection, so it's basically like svn. I don't know if mercurial has improved in this lately, but at least for me, that's a major problem with mercurial. But since python now moved to mercurial too, maybe they will help fix this. :) It seems to me like the python distutils vs cmake discussion. I also prefer the "unpythonic" cmake. Ondrej From david at ar.media.kyoto-u.ac.jp Fri Apr 10 04:44:23 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 10 Apr 2009 17:44:23 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49DEF89D.7000605@hawaii.edu> References: <49CE3A2C.9000007@enthought.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> Message-ID: <49DF06E7.9040401@ar.media.kyoto-u.ac.jp> Eric Firing wrote: > Speaking to Josef: does tortoise-hg provide a satisfactory windows gui, > from your standpoint? > Another solution may be eclipse integration. I don't know if that would work for Josef, but there is a git plugin for eclipse, and I can at least clone branches from a remote repository, and work with it. Is there a hg eclipse plugin ? I am not very knowledgeable about IDE, David From matthieu.brucher at gmail.com Fri Apr 10 05:13:59 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Fri, 10 Apr 2009 11:13:59 +0200 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49DF06E7.9040401@ar.media.kyoto-u.ac.jp> References: <49CE3A2C.9000007@enthought.com> <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DF06E7.9040401@ar.media.kyoto-u.ac.jp> Message-ID: 2009/4/10 David Cournapeau : > Eric Firing wrote: >> Speaking to Josef: does tortoise-hg provide a satisfactory windows gui, >> from your standpoint? >> > > Another solution may be eclipse integration. I don't know if that would > work for Josef, but there is a git plugin for eclipse, and I can at > least clone branches from a remote repository, and work with it. > > Is there a hg eclipse plugin ? I am not very knowledgeable about IDE, Yes, there is MercurialEclipse. I don't know how it handles branches. I use BzrEclipse for my work, and it doesn't handle branches at all, you have to fall back to the command line. Matthieu -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From vincent.thierion at ema.fr Fri Apr 10 07:14:32 2009 From: vincent.thierion at ema.fr (Vincent Thierion) Date: Fri, 10 Apr 2009 13:14:32 +0200 Subject: [Numpy-discussion] numpy import on x86_64 arch Message-ID: Hello, I import numpy module (numpy-1.0.4) on a x86_64 machine (on which I don't have any root privileges) after having install it thanks to python "setup.py install --prefix=../numpy". In this manner, I obtain a 64 bits compatible numpy library. (the "numpy" folder used for install is created just before install process) I had the following error : * import numpy File "../numpy/lib64/python2.3/site-packages/numpy/__init__.py", line 43, in ? File "../numpy/lib64/python2.3/site-packages/numpy/linalg/__init__.py", line 4, in ? File "../numpy/lib64/python2.3/site-packages/numpy/linalg/linalg.py", line 25, in ? ImportError: liblapack.so.3: cannot open shared object file: No such file or directory *I looked for liblapack.so.3 and find it in /usr/*lib*/liblapack.so.3. It seems it doesn't work because numpy 64 bits version seems to need /usr/* lib64*/liblapack.so.3. That I tested in other machine and it works when numpy can find /usr/*lib64*/liblapack.so.3. So is there a solution to use numpy 64 bits version with a /usr/*lib*/liblapack.so.3 or is it absolutly necessary to install lapack 64 bits version ? Thank you Vincent -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Fri Apr 10 07:02:13 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 10 Apr 2009 20:02:13 +0900 Subject: [Numpy-discussion] numpy import on x86_64 arch In-Reply-To: References: Message-ID: <49DF2735.4080708@ar.media.kyoto-u.ac.jp> Vincent Thierion wrote: > Hello, > > I import numpy module (numpy-1.0.4) on a x86_64 machine (on which I > don't have any root privileges) after having install it thanks to > python "setup.py install --prefix=../numpy". In this manner, I obtain > a 64 bits compatible numpy library. > (the "numpy" folder used for install is created just before install > process) You should really use a newer version if you install from sources. numpy 1.0.4 is one year and a half old. If you have to use pytyhon 2.3, you should use numpy 1.1.*. If you have python 2.4 or later, you should use numpy 1.3.0. > > I had the following error : > > / import numpy > File "../numpy/lib64/python2.3/site-packages/numpy/__init__.py", > line 43, in ? > File > "../numpy/lib64/python2.3/site-packages/numpy/linalg/__init__.py", > line 4, in ? > File > "../numpy/lib64/python2.3/site-packages/numpy/linalg/linalg.py", line > 25, in ? > ImportError: liblapack.so.3: cannot open shared object file: No such > file or directory > / What does ldd ../numpy/lib64/python2.3/site-packages/numpy/linalg/lapack_lite.so say ? > So is there a solution to use numpy 64 bits version with a > /usr/*lib*/liblapack.so.3 or is it absolutly necessary to install > lapack 64 bits version ? It is certainly mandatory to have a 64 bits version of lapack when building numpy for a 64 bits python, but it does not have to be in /usr/lib64. You can also build numpy without lapack: LAPACK=None python setup.py .... cheers, David From frank at gis4weather.com Fri Apr 10 10:18:52 2009 From: frank at gis4weather.com (Frank Peacock) Date: Fri, 10 Apr 2009 15:18:52 +0100 Subject: [Numpy-discussion] Replacing colours in numpy array In-Reply-To: References: Message-ID: <003001c9b9e7$4660c220$d3224660$@com> Hello I have a numpy array that I obtained from a converted RGBA gif image. I have tried to replace some colours with different ones using the where condition but have a problem with dimensions. If I use b=where(a==0,255,a) where a is a numpy array from an image it does replace components of the RGB values in each pixel but it fails if I try b=where(a==(0,0,0,255),(255,255,255,255),a) with incorrect dimension error. Could you please help? Thanks Frank -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of numpy-discussion-request at scipy.org Sent: 10 April 2009 08:18 To: numpy-discussion at scipy.org Subject: Numpy-discussion Digest, Vol 31, Issue 24 Send Numpy-discussion mailing list submissions to numpy-discussion at scipy.org To subscribe or unsubscribe via the World Wide Web, visit http://mail.scipy.org/mailman/listinfo/numpy-discussion or, via email, send a message with subject or body 'help' to numpy-discussion-request at scipy.org You can reach the person managing the list at numpy-discussion-owner at scipy.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Numpy-discussion digest..." Today's Topics: 1. Re: DVCS at PyCon (Ondrej Certik) 2. Re: using reducing functions without eliminating dimensions? (Dan Lenski) 3. Re: Another Array (Ian Mallett) 4. Re: Another Array (Robert Kern) 5. Re: Another Array (Ian Mallett) 6. Re: Another Array (Robert Kern) 7. Re: Another Array (Ian Mallett) 8. Re: Another Array (Anne Archibald) ---------------------------------------------------------------------- Message: 1 Date: Thu, 9 Apr 2009 23:14:53 -0700 From: Ondrej Certik Subject: Re: [Numpy-discussion] DVCS at PyCon To: Discussion of Numerical Python Message-ID: <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf at mail.gmail.com> Content-Type: text/plain; charset=UTF-8 On Thu, Apr 9, 2009 at 10:45 PM, David Cournapeau wrote: > Ondrej Certik wrote: >> >> It is maybe easier to learn how to work with different clones, but >> once you start working with lots of patches and you need to reclone >> all the time, then it's the wrong approach to work, as it takes lots >> of time to copy the whole repository on the disk. > > Yes, *I* know how to use git, and I agree with you, I vastly prefer git > branch handling to bzr branch handling. *I* find working with GUI for > VCS a real PITA. But I am not the only numpy developer, that's why the > feedback from people like Josef with a totally different workflow than > me is valuable - much more than people like us who are unix geeks :) Yes, definitely. Ondrej ------------------------------ Message: 2 Date: Fri, 10 Apr 2009 06:36:08 +0000 (UTC) From: Dan Lenski Subject: Re: [Numpy-discussion] using reducing functions without eliminating dimensions? To: numpy-discussion at scipy.org Message-ID: Content-Type: text/plain; charset=UTF-8 On Thu, 09 Apr 2009 01:31:33 -0500, Robert Kern wrote: > On Thu, Apr 9, 2009 at 01:29, Anne Archibald > wrote: > >> What's wrong with np.amin(a,axis=-1)[...,np.newaxis]? > > It's cumbersome, particularly when you have axis=arbitrary_axis. Quite right. It would nice to be able to say: np.amin(a, axiskeep=-1) or a.min(axiskeep=3) ... or something along those lines. Dan ------------------------------ Message: 3 Date: Thu, 9 Apr 2009 23:42:03 -0700 From: Ian Mallett Subject: Re: [Numpy-discussion] Another Array To: Discussion of Numerical Python Message-ID: Content-Type: text/plain; charset="iso-8859-1" It gives a perfect parabolic shape that looks very nice, but somewhat unrealistic. I'd like to scale the unit vectors by a random length (which can just be a uniform distribution). I tried scaling the unit vector n*n*3 array by a random n*n array, but that didn't work, obviously. Help? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20090409/e8fea8 42/attachment-0001.html ------------------------------ Message: 4 Date: Fri, 10 Apr 2009 01:46:47 -0500 From: Robert Kern Subject: Re: [Numpy-discussion] Another Array To: Discussion of Numerical Python Message-ID: <3d375d730904092346l6850c03chca1962915263e79c at mail.gmail.com> Content-Type: text/plain; charset=UTF-8 On Fri, Apr 10, 2009 at 01:42, Ian Mallett wrote: > It gives a perfect parabolic shape that looks very nice, but somewhat > unrealistic. Parabolic? They should be spherical. > I'd like to scale the unit vectors by a random length (which > can just be a uniform distribution).? I tried scaling the unit vector n*n*3 > array by a random n*n array, but that didn't work, obviously. No, it's not obvious. Exactly what code did you try? What results did you get? What results were you expecting? > Help? Let's take a step back. What kind of distribution are you trying to achieve? You asked for uniformly distributed unit vectors. Now you are asking for something else, but I'm really not sure what. What standard are you comparing against when you say that the unit vectors look "unrealistic"? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ------------------------------ Message: 5 Date: Thu, 9 Apr 2009 23:58:21 -0700 From: Ian Mallett Subject: Re: [Numpy-discussion] Another Array To: Discussion of Numerical Python Message-ID: Content-Type: text/plain; charset="iso-8859-1" On Thu, Apr 9, 2009 at 11:46 PM, Robert Kern wrote: > Parabolic? They should be spherical. The particle system in the last screenshot was affected by gravity. In the absence of gravity, the results should be spherical, yes. All the vectors are a unit length, which produces a perfectly smooth surface (unrealistic for such an effect). > No, it's not obvious. Exactly what code did you try? What results did > you get? What results were you expecting? It crashed. I have this code: vecs = Numeric.random.standard_normal(size=(self.size[0],self.size[1],3)) magnitudes = Numeric.sqrt((vecs*vecs).sum(axis=-1)) uvecs = vecs / magnitudes[...,Numeric.newaxis] randlen = Numeric.random.random((self.size[0],self.size[1])) randuvecs = uvecs*randlen #It crashes here with a dimension mismatch rgb = ((randvecs+1.0)/2.0)*255.0 I also tried randlen = Numeric.random.random((self.size[0],self.size[1],3)), but this does not scale each of the vector's components equally, producing artifacts again. Each needs to be scaled by the same random value for it to make sense. > Let's take a step back. What kind of distribution are you trying to > achieve? You asked for uniformly distributed unit vectors. Now you are > asking for something else, but I'm really not sure what. What standard > are you comparing against when you say that the unit vectors look > "unrealistic"? The vectors are used to "jitter" each particle's initial speed, so that the particles go in different directions instead of moving all as one. Using the unit vector causes the particles to make the smooth parabolic shape. The jitter vectors much then be of a random length, so that the particles go in all different directions at all different speeds, instead of just all in different directions. Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20090409/8d6bd9 31/attachment-0001.html ------------------------------ Message: 6 Date: Fri, 10 Apr 2009 02:01:10 -0500 From: Robert Kern Subject: Re: [Numpy-discussion] Another Array To: Discussion of Numerical Python Message-ID: <3d375d730904100001w4cdb7bedi2b36bca97731ef8c at mail.gmail.com> Content-Type: text/plain; charset=UTF-8 On Fri, Apr 10, 2009 at 01:58, Ian Mallett wrote: > On Thu, Apr 9, 2009 at 11:46 PM, Robert Kern wrote: >> >> Parabolic? They should be spherical. > > The particle system in the last screenshot was affected by gravity.? In the > absence of gravity, the results should be spherical, yes.? All the vectors > are a unit length, which produces a perfectly smooth surface (unrealistic > for such an effect). >> >> No, it's not obvious. Exactly what code did you try? What results did >> you get? What results were you expecting? > > It crashed. > I have this code: > vecs = Numeric.random.standard_normal(size=(self.size[0],self.size[1],3)) > magnitudes = Numeric.sqrt((vecs*vecs).sum(axis=-1)) > uvecs = vecs / magnitudes[...,Numeric.newaxis] > randlen = Numeric.random.random((self.size[0],self.size[1])) > randuvecs = uvecs*randlen #It crashes here with a dimension mismatch > rgb = ((randvecs+1.0)/2.0)*255.0 > > I also tried randlen = Numeric.random.random((self.size[0],self.size[1],3)), > but this does not scale each of the vector's components equally, producing > artifacts again.? Each needs to be scaled by the same random value for it to > make sense. See how I did magnitudes[...,numpy.newaxis]? You have to do the same. >> Let's take a step back. What kind of distribution are you trying to >> achieve? You asked for uniformly distributed unit vectors. Now you are >> asking for something else, but I'm really not sure what. What standard >> are you comparing against when you say that the unit vectors look >> "unrealistic"? > > The vectors are used to "jitter" each particle's initial speed, so that the > particles go in different directions instead of moving all as one.? Using > the unit vector causes the particles to make the smooth parabolic shape. > The jitter vectors much then be of a random length, so that the particles go > in all different directions at all different speeds, instead of just all in > different directions. Ah, okay. That makes sense. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ------------------------------ Message: 7 Date: Fri, 10 Apr 2009 00:07:45 -0700 From: Ian Mallett Subject: Re: [Numpy-discussion] Another Array To: Discussion of Numerical Python Message-ID: Content-Type: text/plain; charset="iso-8859-1" This seems to work: vecs = Numeric.random.standard_normal(size=(self.size[0],self.size[1],3)) magnitudes = Numeric.sqrt((vecs*vecs).sum(axis=-1)) uvecs = vecs / magnitudes[...,Numeric.newaxis] randlen = Numeric.random.random((self.size[0],self.size[1])) randuvecs = uvecs*randlen[...,Numeric.newaxis] rgb = ((randuvecs+1.0)/2.0)*255.0 (I have "import numpy as Numeric" for other reasons, that's why there's Numeric there). Thanks, Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20090410/84e6aa a1/attachment-0001.html ------------------------------ Message: 8 Date: Fri, 10 Apr 2009 03:17:58 -0400 From: Anne Archibald Subject: Re: [Numpy-discussion] Another Array To: Discussion of Numerical Python Message-ID: Content-Type: text/plain; charset=UTF-8 2009/4/10 Ian Mallett : > The vectors are used to "jitter" each particle's initial speed, so that the > particles go in different directions instead of moving all as one.? Using > the unit vector causes the particles to make the smooth parabolic shape. > The jitter vectors much then be of a random length, so that the particles go > in all different directions at all different speeds, instead of just all in > different directions. Why not just skip the normalization? Then you'll get vectors with random direction and a natural distribution of lengths. And it'll be faster than tne unit vectors... Anne ------------------------------ _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion End of Numpy-discussion Digest, Vol 31, Issue 24 ************************************************ From josef.pktd at gmail.com Fri Apr 10 10:57:09 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 10 Apr 2009 10:57:09 -0400 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: References: <49CE3A2C.9000007@enthought.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DF06E7.9040401@ar.media.kyoto-u.ac.jp> Message-ID: <1cd32cbb0904100757k7b72ae11v9be75b38fdf0f747@mail.gmail.com> On Fri, Apr 10, 2009 at 5:13 AM, Matthieu Brucher wrote: > 2009/4/10 David Cournapeau : >> Eric Firing wrote: >>> Speaking to Josef: does tortoise-hg provide a satisfactory windows gui, >>> from your standpoint? >>> >> >> Another solution may be eclipse integration. I don't know if that would >> work for Josef, but there is a git plugin for eclipse, and I can at >> least clone branches from a remote repository, and work with it. >> >> Is there a hg eclipse plugin ? I am not very knowledgeable about IDE, > > Yes, there is MercurialEclipse. I don't know how it handles branches. > I use BzrEclipse for my work, and it doesn't handle branches at all, > you have to fall back to the command line. > I tried out mercurial one year ago, including the eclipse plugin, but it didn't work very well compared to the svn plugin. And since at that time mercurial to svn connection wasn't very good, I gave up (I have all my work in svn). I haven't used it since except for checking out some repositories. It's time consuming to keep track of 4 different version control systems, and for my main use, I'm quite ok with svn and minimal use of bzr. I never tried tortoise, because I prefer standalone programs or program plugins that don't mess with my windows installation or registry, if I'm not sure I use it long-term. Now that I see the differences in the branching concept between git and the other ones, I can understand that for reviewing patches, the git way of branches in the same directory is much faster. But how do you actually run the programs in python? How does python know which version to access? Or does git change the content of the files whenever you switch a branch? Answering my own question, for the record, after some more playing: `git checkout branchname` changes files in directory to the branch version, updates changed time stamp to date of checkout (i.e. modification time of a file in git is useless information). I think there is a difference in the usage pattern for development and for reviewing patches. When writing or rewriting code, I just need a few stable branches (clones) and directory access and information based on file structure is more convenient, which is also more consistent with the workspace concept of eclipse. For reviewing and trying out patches, the patch orientation of git is more useful, I imagine. I haven't tried ssh with git yet, with bzr and launchpad, half a year ago it took several hours of trial and error and googling to get it setup, (that was quite a few versions of bzr ago). With svn, authorization to commit to the scipy repository required filling out name and password in the svn gui and it worked. Josef From david at ar.media.kyoto-u.ac.jp Fri Apr 10 11:06:18 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 11 Apr 2009 00:06:18 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <1cd32cbb0904100757k7b72ae11v9be75b38fdf0f747@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DF06E7.9040401@ar.media.kyoto-u.ac.jp> <1cd32cbb0904100757k7b72ae11v9be75b38fdf0f747@mail.gmail.com> Message-ID: <49DF606A.2040502@ar.media.kyoto-u.ac.jp> Hi Josef, josef.pktd at gmail.com wrote: > > I tried out mercurial one year ago, including the eclipse plugin, but > it didn't work very well compared to the svn plugin. And since at that > time mercurial to svn connection wasn't very good, I gave up (I have > all my work in svn). I haven't used it since except for checking out > some repositories. It's time consuming to keep track of 4 different > version control systems, and for my main use, I'm quite ok with svn > and minimal use of bzr. I never tried tortoise, because I prefer > standalone programs or program plugins that don't mess with my windows > installation or registry, if I'm not sure I use it long-term. > (I am trying different tools, in particular tortoisesvn, and eclipse on windows, to get a 'feeling' of your workflow, and how that can work out - or not - for git/hg. I will update the wiki accordingly). > Now that I see the differences in the branching concept between git > and the other ones, I can understand that for reviewing patches, the > git way of branches in the same directory is much faster. But how do > you actually run the programs in python? How does python know which > version to access? Or does git change the content of the files > whenever you switch a branch? > Yes. That's what checkout is about in git (git checkout is somewhat equivalent to svn switch - also again, the comparison breaks quickly because svn switch can be used to switch repositories and or branches, which does not make sense in git). For example, to give you a feeling about why I enjoy git very much: # add branch published by Josef somewhere for review git remote add stats_branch url_from_josef_useful_stats # give me all the changes between the master branch (trunk) and your branch git diff master..stats_branch scipy/stats # I want to build your changes, so I create a locate branch which tracks your branch git branch local_stats stats_branch/master # Now I 'checkout' the branch, which means updating the working tree (the files) from the local_stats branch git co local_stats && build it # I do some modification to your branch, maybe - I then see a bug in scipy stats independent of your changes git stash # put away all the changes in a stash # Fix the bug hack.... # Return to your changes, checkout the branch, and revert the changes I was working on previously git co local_stats git stash pop For the release process, something I have used a lot for numpy 1.3.0 through git-svn (if using git, it would be easier, git-svn can be quirky) # Log for the last few days git log --since='one week' # Get an idea of the modified files in the 1.3.x branch: number of lines modified/file git diff --stat svn/branches/1.3.x # Review changes between the beta and the rc git diff 1.3.0b1..1.3.0rc1 git log 1.3.0b1..1.3.0rc1 # Something fails building in the rc1, or maybe some tests - check that the beta did not have this problem by rebuilding # from scratch git co 1.3.0b1 && make test_scratch All this is excruciating on svn. Switching branches is not reliable, and takes ages (it is of the order of minutes for every single command). Diffing/log of branches is even worse, and the command line UI is really painful (I can't understand why I have to use full URL for branches in svn). One problem which does not concern many people, but which is quite acute at least for me: I often need to check some changes on several platforms at the same time (windows/linux, generally). With svn, I have no choice but to commit every change to svn, and svn acts as the gateway - making my own local gateway is just too much work. With git, that's relatively easy (but bzr is easier - that's one thing I really miss from bzr), I can just make sure everything work instead of making tons of useless and broken commits. > Answering my own question, for the record, after some more playing: > `git checkout branchname` changes files in directory to the branch > version, updates changed time stamp to date of checkout (i.e. > modification time of a file in git is useless information) > Yes, git tracks content (that is the whole tree). I did not think about this, because I never use this information even on svn - I use the command line. You have to use GUI to notice this problem :) > I haven't tried ssh with git yet, with bzr and launchpad, half a year > ago it took several hours of trial and error and googling to get it > setup, (that was quite a few versions of bzr ago). With svn, > authorization to commit to the scipy repository required filling out > name and password in the svn gui and it worked. > Yes, this would take some time (from people involved in the eventual transition) to get right. Launchpad was not very good at it. But this is independent of the DVCS: either bzr, hg or git would need to use a ssh-based mechanism I think (so the good news is that you have already done all the work for launchpad :) ). cheers, David From Chris.Barker at noaa.gov Fri Apr 10 12:43:28 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 10 Apr 2009 09:43:28 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <5b8d13220904092035i2a3b1af8l8544584e6724fec1@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <49D1AF7C.3050802@hawaii.edu> <49D1B12D.3000300@ar.media.kyoto-u.ac.jp> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <49DE305C.7000702@noaa.gov> <5b8d13220904092035i2a3b1af8l8544584e6724fec1@mail.gmail.com> Message-ID: <49DF7730.3010505@noaa.gov> David Cournapeau wrote: >> we're really better off with a system with >> good tool support on all platforms. > > Why ? We are not python, where many core developers work on windows. As I understand it there is a dearth of Python developers on Windows, too... But anyway, we probably want MORE Windows folks contributing, not fewer. Another key is the "core developers" concept -- I'm not a core developer on any major OS project, but I do contribute bits and pieces here and there to quite a few ( numpy, scipy, MPL, wxPython, MacPython, ... ). I think we want to encourage folks to be able to do that - learning a new VCS to test and contribute a small patch is large barrier to entry. > The git command line works well on windows (same as on other systems), > I used it while testing things for numpy 1.3.0 (locally, though > because of the git-svn issue). Somehow, command-line anything is kind of painful on Windows, and lots of folks don't like using it. I'm kind of a command-line guy, but do most of my work on Linux or OS-X, and only use Windows when I have to, and then I tend to use GUI tools there -- TortoiseSVN, for instance, even though I use command line SVN on *nix systems. > There will always be arguments for svn being more supported, but > doesn't this sounds like matlab vs numpy/scipy ? There will always be > people who will find the scipy stack not good, not integrated - right -- and there are a LOT more Matlab users, even though it costs a lot, and is a very limited language. > still, we use numpy/scipy. git is not better than svn in every way, > but at least to me, the question is more "is git better than svn > overall" To carry the analogy above further, I use python/numpy/scipy rather than Matlab for a few major reasons: * cost/licensing -- we're only considering open source VCS here anyway. * numpy is more powerful/flexible -- That's why we're considering moving to a DVCS from SVN at all -- I think it's almost a foregone conclusion that we'll make that move -- the question is not if, but when and which one? * I use Python for a bunch of other stuff Matlab is not suitable for -- This is my argument about usability and tool support. A few years back, CVS was a standard, now SVN is. I like that I can use the same tool to contribute to a whole bunch of OS projects, and I use it to manage all my projects as work. It seems many in the OS world are moving to a DVCS -- but there are three major ones in play: git, Hg and bzr -- I don't know enough about any of them to say what I prefer, but I really don't want to have to learn all three! And if I do, it would be nice if there were semi-similar interfaces for them all: tortoise, for instance. Anyway, it's not a deal-breaker, but tool support across platforms is important, and should be considered. And I do think that Windows is just as important to numpy/scipy as it is to Python. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From ondrej at certik.cz Fri Apr 10 13:29:29 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 10 Apr 2009 10:29:29 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49DF7730.3010505@noaa.gov> References: <49CE3A2C.9000007@enthought.com> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <49DE305C.7000702@noaa.gov> <5b8d13220904092035i2a3b1af8l8544584e6724fec1@mail.gmail.com> <49DF7730.3010505@noaa.gov> Message-ID: <85b5c3130904101029r6d6f85efp8bd7b233f642b450@mail.gmail.com> > * I use Python for a bunch of other stuff Matlab is not suitable for -- > This is my argument about usability and tool support. A few years back, > CVS was a standard, now SVN is. I like that I can use the same tool to > contribute to a whole bunch of OS projects, and I use it to manage all > my projects as work. It seems many in the OS world are moving to a DVCS > -- but there are three major ones in play: git, Hg and bzr -- I don't > know enough about any of them to say what I prefer, but I really don't > want to have to learn all three! And if I do, it would be nice if there > were semi-similar interfaces for them all: tortoise, for instance. Unfortunately, one has to learn all 3 (I need to use bzr for ipython, hg for Sage, maybe soon some other projects and git for everything else). But it's not difficult. I think it's like learning bash and then couple commands from tcsh and zsh just to be able to survive. Ondrej From praxbaffle at hotmail.com Fri Apr 10 13:30:18 2009 From: praxbaffle at hotmail.com (John Seales) Date: Fri, 10 Apr 2009 10:30:18 -0700 Subject: [Numpy-discussion] help getting started Message-ID: Intel mac. Python 2.6. I'm trying to start using numpy and scipy, and am having trouble. I'm trying to install numpy following the instructions on http://scipy.org/Installing_SciPy/Mac_OS_X . when I give the python setup.py build command, it doesn't work. Here's my interaction at the terminal: john-seales-computer:/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy johnseales$ python setup.py build This is the wrong setup.py file to run john-seales-computer:/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy Anybody have any ideas how I can get up and runnng? _________________________________________________________________ Rediscover Hotmail?: Now available on your iPhone or BlackBerry http://windowslive.com/RediscoverHotmail?ocid=TXT_TAGLM_WL_HM_Rediscover_Mobile1_042009 -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Fri Apr 10 13:42:24 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 11 Apr 2009 02:42:24 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49DF7730.3010505@noaa.gov> References: <49CE3A2C.9000007@enthought.com> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <49DE305C.7000702@noaa.gov> <5b8d13220904092035i2a3b1af8l8544584e6724fec1@mail.gmail.com> <49DF7730.3010505@noaa.gov> Message-ID: <5b8d13220904101042p445997d5xade98f7b694c79b7@mail.gmail.com> On Sat, Apr 11, 2009 at 1:43 AM, Christopher Barker wrote: > David Cournapeau wrote: >>> we're really better off with a system with >>> good tool support on all platforms. >> >> Why ? We are not python, where many core developers work on windows. > > As I understand it there is a dearth of Python developers on Windows, > too... Yes, but several highly visible python committers contributed a lot of windows specific changes, or are windows users only. And a lot of people who integrate python are windows shops. That was an argument stated by several significant contributors against git. Today, I think Josef is the main contributor who mostly uses windows - there is me, too, actually :) > But anyway, we probably want MORE Windows folks contributing, not > fewer. Yes, sure. But the fact is that there are not many windows developers. And that's kind of inherent to the platform I believe. For many open source projects which are cross-platform, there are very few windows developers. For numpy/scipy, you almost must install mingw compilers, all this being command line. I wish there was a developer who would handle windows specifics for me :) > Another key is the "core developers" concept -- I'm not a core developer > on any major OS project, but I do contribute bits and pieces here and > there to quite a few ( numpy, scipy, MPL, wxPython, MacPython, ... ). I > think we want to encourage folks to be able to do that - learning a new > VCS to test and contribute a small patch is large barrier to entry. Yes, I agree "core developer" is a bit artificial and dangerous concept. But for open source contribution, I think it is pretty clear you will have to learn git. There are just so many projects which use it now (all the freedesktop stuff, ror, gnome, perl, vlc, etc...). Also, several people stated that they would be more willing to contribute to numpy/scipy if it were under git. Now, granted, that's just talk and guesses - but not more than the influence of git on windows developers. > Somehow, command-line anything is kind of painful on Windows, and lots > of folks don't like using it. I'm kind of a command-line guy, but do > most of my work on Linux or OS-X, and only use Windows when I have to, > and then I tend to use GUI tools there -- TortoiseSVN, for instance, > even though I use command line SVN on *nix systems. Yes, command line on windows is not great, for various reasons. But I think that's a good argument for svn and against *any* DVCS - because frankly, all those GUI tools suck on windows ATM, be it hg, bzr or git. I've just tried tortoisehg, and it is still really rough, not really better than tortoisegit for what I can see. Heck, I find tortoisesvn really rough, compared to things I have seen under good IDE (I use very few GUI tools - but when I do, I expect top notch stuff - which open source is rarely good at as far as GUI are concerned). David From efiring at hawaii.edu Fri Apr 10 13:59:59 2009 From: efiring at hawaii.edu (Eric Firing) Date: Fri, 10 Apr 2009 07:59:59 -1000 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49DF06E7.9040401@ar.media.kyoto-u.ac.jp> References: <49CE3A2C.9000007@enthought.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DF06E7.9040401@ar.media.kyoto-u.ac.jp> Message-ID: <49DF891F.90904@hawaii.edu> David Cournapeau wrote: > Eric Firing wrote: >> Speaking to Josef: does tortoise-hg provide a satisfactory windows gui, >> from your standpoint? >> > > Another solution may be eclipse integration. I don't know if that would > work for Josef, but there is a git plugin for eclipse, and I can at > least clone branches from a remote repository, and work with it. > > Is there a hg eclipse plugin ? I am not very knowledgeable about IDE, I'm not, either, but yes, there is a mercurial plugin for eclipse. Eclipse has devoted fans, but in the exploration my colleague and I have done, we have found it very heavy-weight and cumbersome; it doesn't look like we will adopt it. Eric > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From cournape at gmail.com Fri Apr 10 14:10:07 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 11 Apr 2009 03:10:07 +0900 Subject: [Numpy-discussion] help getting started In-Reply-To: References: Message-ID: <5b8d13220904101110n5f39f7cdwd5a5fb85c1d1c405@mail.gmail.com> On Sat, Apr 11, 2009 at 2:30 AM, John Seales wrote: > Intel mac. > Python 2.6. > > I'm trying to start using numpy and scipy, and am having trouble. > > I'm trying to install numpy following the instructions on > http://scipy.org/Installing_SciPy/Mac_OS_X . > > when I give the python setup.py build command, it doesn't work. Here's my > interaction at the terminal: > > john-seales-computer:/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy > johnseales$ python setup.py build > > This is the wrong setup.py file to run > john-seales-computer:/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy > > Anybody have any ideas how I can get up and runnng? nothing :) It looks like you have already installed numpy (site-packages is the directory where all python packages like numpy get installed by default). You just have to run the python interpreter, and import numpy. That is, if you can do: python -c "import numpy" It means you have numpy installed. David From zachary.pincus at yale.edu Fri Apr 10 14:23:35 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Fri, 10 Apr 2009 14:23:35 -0400 Subject: [Numpy-discussion] help getting started In-Reply-To: References: Message-ID: <2B1D6995-8A1C-4B21-B0D2-78CDF46E5739@yale.edu> Hi John, First, did you build your own Python 2.6 or install from a binary? When you type "python" at the command prompt, which python runs? (You can find this out by running "which python" from the command line.) Second, it appears that numpy is *already installed* for a non-apple python 2.5 on your computer. (Either you manually built that python, or installed it from an installer, and then you built or installed numpy for that python?) I can tell this because you have a directory: /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- packages/numpy which normally gets created during the process of installing numpy for Python 2.5. Unless you manually created this directory yourself somehow? Anyhow, that directory is *not* the source-code directory that you should be running numpy's setup.py from; it is the directory that the installation process for numpy copies all of the necessary files into! Anyhow, how did you come to be executing commands from that directory? The numpy install instructions definitely don't have a "cd /Library/ Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/ numpy" step... So, if you want to use numpy with python 2.5, you should be all set already. If you need to use it with 2.6, then you will want to download the source code into a directory somewhere -- say in your home folder. Then execute "python setup.py install" from *this* directory. (Note: if typing "python" does not run python 2.6, but instead runs python 2.5, you will want to do "python2.6 setup.py install"... Feel free to ask any more questions -- I'm heading out the door, so this might be a bit terse. Zach On Apr 10, 2009, at 1:30 PM, John Seales wrote: > Intel mac. > Python 2.6. > > I'm trying to start using numpy and scipy, and am having trouble. > > I'm trying to install numpy following the instructions on http://scipy.org/Installing_SciPy/Mac_OS_X > . > > when I give the python setup.py build command, it doesn't work. > Here's my interaction at the terminal: > > john-seales-computer:/Library/Frameworks/Python.framework/Versions/ > 2.5/lib/python2.5/site-packages/numpy johnseales$ python setup.py > build > > This is the wrong setup.py file to run > john-seales-computer:/Library/Frameworks/Python.framework/Versions/ > 2.5/lib/python2.5/site-packages/numpy > > Anybody have any ideas how I can get up and runnng? > > > > Rediscover Hotmail?: Now available on your iPhone or BlackBerry > Check it out._______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From gael.varoquaux at normalesup.org Fri Apr 10 16:04:50 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 10 Apr 2009 22:04:50 +0200 Subject: [Numpy-discussion] LARS algorithm Message-ID: <20090410200450.GA3641@phare.normalesup.org> Has anybody implemented the LARS[1] (least angle regression) algorithm for regularized regression in Python or in C binded in Python? I am about to start in such endeaviour, but I wanted to check if someone was willing to share code under a BSD-compatible license. Ga?l [1] http://en.wikipedia.org/wiki/Least-angle_regression PS: Sorry for the cross-post, I just wanted to be really sure that if anybody was willing to share such code, I would reach him :) From praxbaffle at hotmail.com Fri Apr 10 16:19:49 2009 From: praxbaffle at hotmail.com (John Seales) Date: Fri, 10 Apr 2009 13:19:49 -0700 Subject: [Numpy-discussion] help getting started In-Reply-To: <5b8d13220904101110n5f39f7cdwd5a5fb85c1d1c405@mail.gmail.com> References: <5b8d13220904101110n5f39f7cdwd5a5fb85c1d1c405@mail.gmail.com> Message-ID: loading the numpy or scipy module doesn't work, whether I do it at the terminal command line or at the python interpreter. Anyone know what else might be wrong? > Date: Sat, 11 Apr 2009 03:10:07 +0900 > From: cournape at gmail.com > To: numpy-discussion at scipy.org > Subject: Re: [Numpy-discussion] help getting started > > On Sat, Apr 11, 2009 at 2:30 AM, John Seales wrote: > > Intel mac. > > Python 2.6. > > > > I'm trying to start using numpy and scipy, and am having trouble. > > > > I'm trying to install numpy following the instructions on > > http://scipy.org/Installing_SciPy/Mac_OS_X . > > > > when I give the python setup.py build command, it doesn't work. Here's my > > interaction at the terminal: > > > > john-seales-computer:/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy > > johnseales$ python setup.py build > > > > This is the wrong setup.py file to run > > john-seales-computer:/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy > > > > Anybody have any ideas how I can get up and runnng? > > nothing :) It looks like you have already installed numpy > (site-packages is the directory where all python packages like numpy > get installed by default). > > You just have to run the python interpreter, and import numpy. That > is, if you can do: > > python -c "import numpy" > > It means you have numpy installed. > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion _________________________________________________________________ Rediscover Hotmail?: Get e-mail storage that grows with you. http://windowslive.com/RediscoverHotmail?ocid=TXT_TAGLM_WL_HM_Rediscover_Storage1_042009 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Fri Apr 10 16:42:11 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 10 Apr 2009 13:42:11 -0700 Subject: [Numpy-discussion] help getting started In-Reply-To: References: <5b8d13220904101110n5f39f7cdwd5a5fb85c1d1c405@mail.gmail.com> Message-ID: <49DFAF23.5030100@noaa.gov> John Seales wrote: > loading the numpy or scipy module doesn't work, whether I do it at the > terminal command line or at the python interpreter. > > Anyone know what else might be wrong? we're going to have to know a lot more about what you've done so far, but; a) unless you have a strong reason to use Python2.6, I'd use 2.5: - install the python 2.5 form the python,org web site (it looks like you have done this - install the binary for numpy for python 2.5: http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 You're done. b) if you really need 2.6, you'll have to build numpy yourself: - Install the binary of Python2.6 from the python web site - make sure that's the one you get when you type "python": . "which python" may help here . If it's not, edit your .bash_profile to get it on your PATH. - download the source tarball for numpy . unpack it into somewhere in your home directory - Make sure you have the Apple development tools: XCodeTools - cd into the numpy dir, and type: $ python setup.py build $ python setup.py install - run python, and type "import numpy" Last I checked, the binary for ScipPy was broken, you'll need to get gfortran and compile it yourself, as the page tells you. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Fri Apr 10 16:44:30 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 10 Apr 2009 13:44:30 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <5b8d13220904101042p445997d5xade98f7b694c79b7@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <85b5c3130904061300g78e4c7f8w23e2fc72c38e13aa@mail.gmail.com> <9457e7c80904070032l232beb7brfab644a17b4991d4@mail.gmail.com> <85b5c3130904071134s2826ce6v239f25fe724fe3bf@mail.gmail.com> <5b8d13220904082304i39d2e2aard3b37cb8435eebc7@mail.gmail.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <49DE305C.7000702@noaa.gov> <5b8d13220904092035i2a3b1af8l8544584e6724fec1@mail.gmail.com> <49DF7730.3010505@noaa.gov> <5b8d13220904101042p445997d5xade98f7b694c79b7@mail.gmail.com> Message-ID: <49DFAFAE.1010407@noaa.gov> Well, there is a LOT to consider here, and I have virtually no experience with any of the DVCSs, so I don't have any conclusions to offer, but: Windows support matters. Tool support matters. Those should be taken into consideration when making a choice. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From emanuele at relativita.com Fri Apr 10 16:42:01 2009 From: emanuele at relativita.com (emanuele at relativita.com) Date: Fri, 10 Apr 2009 22:42:01 +0200 (CEST) Subject: [Numpy-discussion] [SciPy-user] LARS algorithm In-Reply-To: <20090410200450.GA3641@phare.normalesup.org> References: <20090410200450.GA3641@phare.normalesup.org> Message-ID: Per Sedeberg wrapped LARS (through RPy) in PyMVPA. Later he mentioned about a C implementation he found and it seems he's going to work on it: http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2009q1/000404.html I guess you should contact him. Emanuele On Fri, April 10, 2009 10:04 pm, Gael Varoquaux wrote: > Has anybody implemented the LARS[1] (least angle regression) algorithm for > regularized regression in Python or in C binded in Python? > > I am about to start in such endeaviour, but I wanted to check if someone > was willing to share code under a BSD-compatible license. > > Ga?l > > [1] http://en.wikipedia.org/wiki/Least-angle_regression > > PS: Sorry for the cross-post, I just wanted to be really sure that if > anybody was willing to share such code, I would reach him :) > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From oliner at stanford.edu Fri Apr 10 17:09:00 2009 From: oliner at stanford.edu (Adam Oliner) Date: Fri, 10 Apr 2009 14:09:00 -0700 Subject: [Numpy-discussion] numpy.ma.arg{min,max} Message-ID: Hi there, The documentation for numpy.ma.argmin says: "Returns array of indices of the maximum values along the given axis." Aside from probably meaning to say the 'minimum' values, it also doesn't seem to return an array: >>> a array([1, 1, 1, 5, 5]) >>> numpy.ma.argmax(a) 3 >>> numpy.ma.argmin(a) 0 I was expecting to get more than one index for both calls. Am I doing something wrong or misreading the documentation? - Adam J. Oliner oliner at cs.stanford.edu From efiring at hawaii.edu Fri Apr 10 17:16:22 2009 From: efiring at hawaii.edu (Eric Firing) Date: Fri, 10 Apr 2009 11:16:22 -1000 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> References: <49CE3A2C.9000007@enthought.com> <5b8d13220904090704n7e298577lf2b46eb6d850fc65@mail.gmail.com> <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> Message-ID: <49DFB726.3000002@hawaii.edu> David Cournapeau wrote: > Eric Firing wrote: >> This is simply wrong. Mercurial uses hard links for cloning a repo that >> is on the same disk, so it is faster and much more space-efficient than >> copying the files. > > Yes, but maybe Ondrej talks about an older hg version ? Hg could not > handle NTFS hardlink for some time, but it does now if you have pywin32. > > And still, switching between branches is faster in git than in hg, for > some technical reasons I can't really explain (but can be found on the efiring at manini:~/programs/py/mpl/mpl_hgtestclone3$ hg up v0_98_5_maint 1957 files updated, 0 files merged, 90 files removed, 0 files unresolved efiring at manini:~/programs/py/mpl/mpl_hgtestclone3$ hg up default 262 files updated, 0 files merged, 1785 files removed, 0 files unresolved On my laptop, switching back and forth between the two active branches of mpl takes about 3 s for the first and 2 s for the second, timed by counting in my head. > ML archives). But as I said previously, speed is not really an argument > for me. If hg is fast enough for python, it is obviously fast enough for > numpy and scipy. As long as it does not takes minutes to merge/review > the 5 lines difference between two branches as is the case in svn right > now, I am happier :) > >> But if you do want named branches in a given repo, >> you can have that also with hg. Granted, it has not always been part of >> hg, but it is now. Same with rebasing and transplanting. >> > > As I understand, and correct me if I am wrong, the problems with named > branches are: I'm on thin ice here, because for my own work I have not been using named branches. > - you can't remove them later, it is in the repository 'forever' They can be removed with the strip command which is in the mq extension, but one must identify the root of the branch and supply that to strip. There may be some limits and gotchas. > - it is not easy to make them publicly available Maybe I'm missing something, but I don't see this problem. For example, see http://currents.soest.hawaii.edu/hg/hgwebdir.cgi/matplotlib_mirror/ which is a mirror of the mpl svn repo using hgsubversion. The branches are there, and show up when you clone it. A problem is that the web interface does not allow one to select a single branch. The log command, however, does. For very lightweight local branching there are bookmarks, which allow one to make a local, changeable label for a given head within a repo. Again, such a local branch can be eliminated via strip--although I am not sure there is much point in doing so. To go this route, I suspect one would first commit a tag, set the bookmark, make whatever changes and commits are desired, etc. The point of the tag would be to make it easy to tell strip where to start stripping. Certainly, hg is simplest when used as it was originally designed, with separate directories for branches. How well it can emulate some git-style workflows, I don't know--probably never perfectly. There are tradeoffs. Overall, it appears to me that for someone who has never used a VCS, or for someone used to svn, hg has a lower barrier to entry than git. > > One big potential with the way git does branching is review - gerrit > (the code review tool used by Google for android) looks really nice. How is the code review related to the branching options in git? > Unfortunately, to see whether this is indeed true requires trying it, > because git-svn cannot be used to that effect (because of branches). > > What would be great is to have git-svnserver, like git-cvsserver (which > is a gateway for cvs, that is people who don't want to bother can use > cvs to do as they do normally, and other people can use git proper). I > find it surprising that such a tool does not exist. > >> It is possible that hg might >> be a better fit--a better compromise--for present *and* *potential* >> *future* contributors to numpy, scipy, and matplotlib. >> > > Yes, that may be the case. > >> Speaking to David: is git branch handling vastly preferable to both of >> the branch styles available in hg? >> > > Besides git, I am mostly familiar with bzr, which has the "branch is a > different directory" concept (but no named branches). I think bzr is > very confusing for advanced workflows in relation to this UI (I am too > lazy to restate my reasons, see here: > http://cournape.wordpress.com/2008/10/30/going-away-from-bzr-toward-git). > > When I tried hg (version 0.8 maybe ?), there was little support for > named branches, but this has changed since I believe. When you have > different repositories for different branches, comparisons between > branches are difficult. This was the case in bzr. Is this the case in hg > ? For example, can you log a subdirectory between two branches, easily > review files changes between branches/tags/etc... For the release > process, I find this very useful - but this can be somewhat alleviated > with git-svn without forcing anyone using it. You can easily do diffs between changesets within a repo, regardless of whether they are on different named branches, and the diffs can be specific to a file or subdirectory. Changesets can be identified by branchnames, tags, bookmarks, sequential numbers, or hash ID. For diffs between directories, there is an rdiff extension that is not distributed with hg. I haven't tried it. It seems like it should be standard--I have no idea why it has not been brought into the distribution. > > If people think that at least trying hg should be possible, I think it > should not be too difficult to have a hg mirror of svn - but I am not > sure whether you can use it to commit back to svn, like git-svn. svn interoperability is an area where hg has acknowledged lagging behind git-svn. There have been several options. What seems to be the best is hgsubversion, (http://www.selenic.com/mercurial/wiki/index.cgi/HgSubversion) which is what I am using to make the mpl mirror. I haven't tried committing back, however--but it "should work". Cloning does not maintain the connection to svn; the extra metadata is not propagated. When/if VCS changes are made, I would prefer to see a clean break. I think that all of the gateways are likely to be more trouble than they are worth. For someone used to svn, doing the same basic operations with hg--but with the advantage of the full history on the local machine--just isn't that hard to learn. (Yes, all the git proponents say the same thing, and maybe they are right--but I think it is a little harder.) Eric > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From matthew.brett at gmail.com Fri Apr 10 17:47:28 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 10 Apr 2009 14:47:28 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49DFB726.3000002@hawaii.edu> References: <49CE3A2C.9000007@enthought.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> Message-ID: <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> Hi, I enjoyed this quote from http://www.eecs.harvard.edu/~cduan/technical/git/ Summary: You can only really use Git if you understand how Git works. "When I first started using Git, I read plenty of tutorials, as well as the user manual. Though I picked up the basic usage patterns and commands, I never felt like I grasped what was going on ?under the hood?, so to speak. Frequently this resulted in cryptic error messages, caused by my random guessing at the right command to use at a given time. These difficulties worsened as I began to need more advanced (and less well documented) features. After a few months, I started to understand those under-the-hood concepts. Once I did, suddenly everything made sense. I could understand the manual pages and perform all sorts of source control tasks. Everything that seemed so cryptic and obscure now was perfectly clear. Understanding Git The conclusion I draw from this is that you can only really use Git if you understand how Git works. Merely memorizing which commands you should run at what times will work in the short run, but it?s only a matter of time before you get stuck or, worse, break something." Best, Matthew From frank at gis4weather.com Fri Apr 10 18:02:45 2009 From: frank at gis4weather.com (Frank Peacock) Date: Fri, 10 Apr 2009 23:02:45 +0100 Subject: [Numpy-discussion] Replacing colours in numpy array 2 Message-ID: <000601c9ba28$1409bb10$3c1d3130$@com> Hello Apologies for the incorrect posting before. The problem was that I was not using an array for the true condition and instead was using a scalar. Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank at gis4weather.com Fri Apr 10 18:15:10 2009 From: frank at gis4weather.com (Frank Peacock) Date: Fri, 10 Apr 2009 23:15:10 +0100 Subject: [Numpy-discussion] Constant array of tuples In-Reply-To: References: Message-ID: <000b01c9ba29$d02d6070$70882150$@com> Hello I would like to know whether there is a simple way to construct a constant array of tuples: How do I construct an array of size (width*height) where each element is (w,x,y,z) Frank -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of numpy-discussion-request at scipy.org Sent: 10 April 2009 15:19 To: numpy-discussion at scipy.org Subject: Numpy-discussion Digest, Vol 31, Issue 26 Send Numpy-discussion mailing list submissions to numpy-discussion at scipy.org To subscribe or unsubscribe via the World Wide Web, visit http://mail.scipy.org/mailman/listinfo/numpy-discussion or, via email, send a message with subject or body 'help' to numpy-discussion-request at scipy.org You can reach the person managing the list at numpy-discussion-owner at scipy.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Numpy-discussion digest..." Today's Topics: 1. Re: frombuffer alignment for ctypes.Structure array (David Cournapeau) 2. Re: DVCS at PyCon (Ondrej Certik) 3. Re: DVCS at PyCon (David Cournapeau) 4. Re: DVCS at PyCon (Matthieu Brucher) 5. numpy import on x86_64 arch (Vincent Thierion) 6. Re: numpy import on x86_64 arch (David Cournapeau) 7. Replacing colours in numpy array (Frank Peacock) ---------------------------------------------------------------------- Message: 1 Date: Fri, 10 Apr 2009 17:28:23 +0900 From: David Cournapeau Subject: Re: [Numpy-discussion] frombuffer alignment for ctypes.Structure array To: Discussion of Numerical Python Message-ID: <49DF0327.5010209 at ar.media.kyoto-u.ac.jp> Content-Type: text/plain; charset=ISO-8859-1 Roland Schulz wrote: > The is no align or aligned option to frombuffer. What is the best way > to tell numpy to align the data as the C-struct/ctypes.Stucture array is? You could add a 'fake' field in between to get the right alignment, maybe ? import numpy as N from ctypes import * class C(Structure): _fields_=[("a",c_int),("b",c_short), ('', c_short)] c=(C*2)() _ctypes_to_numpy = {c_short : N.int16,c_int : N.int32} ty = N.dtype([(x,_ctypes_to_numpy[y]) for x,y in C._fields_]) x=N.frombuffer(c,ty) You may use a different type if you need another alignment, or using something like 'VN' in the dtype, e.g.: a = N.dtype([('a', np.int), ('b', np.short), ('', 'V2')]) David ------------------------------ Message: 2 Date: Fri, 10 Apr 2009 01:54:15 -0700 From: Ondrej Certik Subject: Re: [Numpy-discussion] DVCS at PyCon To: Discussion of Numerical Python Message-ID: <85b5c3130904100154l797d062fn90ea9389bc2da0b at mail.gmail.com> Content-Type: text/plain; charset=UTF-8 On Fri, Apr 10, 2009 at 1:07 AM, David Cournapeau wrote: > Eric Firing wrote: >> >> This is simply wrong. ?Mercurial uses hard links for cloning a repo that >> is on the same disk, so it is faster and much more space-efficient than >> copying the files. > > Yes, but maybe Ondrej talks about an older hg version ? Hg could not > handle NTFS hardlink for some time, but it does now if you have pywin32. > > And still, switching between branches is faster in git than in hg, for > some technical reasons I can't really explain (but can be found on the > ML archives). But as I said previously, speed is not really an argument > for me. If hg is fast enough for python, it is obviously fast enough for > numpy and scipy. As long as it does not takes minutes to merge/review > the 5 lines difference between two branches as is the case in svn right > now, I am happier :) > >> ?But if you do want named branches in a given repo, >> you can have that also with hg. ?Granted, it has not always been part of >> hg, but it is now. ?Same with rebasing and transplanting. >> > > As I understand, and correct me if I am wrong, the problems with named > branches are: > ? ?- you can't remove them later, it is in the repository 'forever' > ? ?- it is not easy to make them publicly available Plus with git, you can fetch the remote repository with all the branches and browse them locally in your remote branches, when you are offline. And merge them with your own branches. In mercurial, it seems the only way to see what changes are there and which branch and which commits I want to merge is to use "hg in", but that requires the internet connection, so it's basically like svn. I don't know if mercurial has improved in this lately, but at least for me, that's a major problem with mercurial. But since python now moved to mercurial too, maybe they will help fix this. :) It seems to me like the python distutils vs cmake discussion. I also prefer the "unpythonic" cmake. Ondrej ------------------------------ Message: 3 Date: Fri, 10 Apr 2009 17:44:23 +0900 From: David Cournapeau Subject: Re: [Numpy-discussion] DVCS at PyCon To: Discussion of Numerical Python Message-ID: <49DF06E7.9040401 at ar.media.kyoto-u.ac.jp> Content-Type: text/plain; charset=ISO-8859-1 Eric Firing wrote: > Speaking to Josef: does tortoise-hg provide a satisfactory windows gui, > from your standpoint? > Another solution may be eclipse integration. I don't know if that would work for Josef, but there is a git plugin for eclipse, and I can at least clone branches from a remote repository, and work with it. Is there a hg eclipse plugin ? I am not very knowledgeable about IDE, David ------------------------------ Message: 4 Date: Fri, 10 Apr 2009 11:13:59 +0200 From: Matthieu Brucher Subject: Re: [Numpy-discussion] DVCS at PyCon To: Discussion of Numerical Python Message-ID: Content-Type: text/plain; charset=ISO-8859-1 2009/4/10 David Cournapeau : > Eric Firing wrote: >> Speaking to Josef: does tortoise-hg provide a satisfactory windows gui, >> from your standpoint? >> > > Another solution may be eclipse integration. I don't know if that would > work for Josef, but there is a git plugin for eclipse, and I can at > least clone branches from a remote repository, and work with it. > > Is there a hg eclipse plugin ? I am not very knowledgeable about IDE, Yes, there is MercurialEclipse. I don't know how it handles branches. I use BzrEclipse for my work, and it doesn't handle branches at all, you have to fall back to the command line. Matthieu -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher ------------------------------ Message: 5 Date: Fri, 10 Apr 2009 13:14:32 +0200 From: Vincent Thierion Subject: [Numpy-discussion] numpy import on x86_64 arch To: Discussion of Numerical Python Message-ID: Content-Type: text/plain; charset="iso-8859-1" Hello, I import numpy module (numpy-1.0.4) on a x86_64 machine (on which I don't have any root privileges) after having install it thanks to python "setup.py install --prefix=../numpy". In this manner, I obtain a 64 bits compatible numpy library. (the "numpy" folder used for install is created just before install process) I had the following error : * import numpy File "../numpy/lib64/python2.3/site-packages/numpy/__init__.py", line 43, in ? File "../numpy/lib64/python2.3/site-packages/numpy/linalg/__init__.py", line 4, in ? File "../numpy/lib64/python2.3/site-packages/numpy/linalg/linalg.py", line 25, in ? ImportError: liblapack.so.3: cannot open shared object file: No such file or directory *I looked for liblapack.so.3 and find it in /usr/*lib*/liblapack.so.3. It seems it doesn't work because numpy 64 bits version seems to need /usr/* lib64*/liblapack.so.3. That I tested in other machine and it works when numpy can find /usr/*lib64*/liblapack.so.3. So is there a solution to use numpy 64 bits version with a /usr/*lib*/liblapack.so.3 or is it absolutly necessary to install lapack 64 bits version ? Thank you Vincent -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20090410/98240a bd/attachment-0001.html ------------------------------ Message: 6 Date: Fri, 10 Apr 2009 20:02:13 +0900 From: David Cournapeau Subject: Re: [Numpy-discussion] numpy import on x86_64 arch To: Discussion of Numerical Python Message-ID: <49DF2735.4080708 at ar.media.kyoto-u.ac.jp> Content-Type: text/plain; charset=ISO-8859-1 Vincent Thierion wrote: > Hello, > > I import numpy module (numpy-1.0.4) on a x86_64 machine (on which I > don't have any root privileges) after having install it thanks to > python "setup.py install --prefix=../numpy". In this manner, I obtain > a 64 bits compatible numpy library. > (the "numpy" folder used for install is created just before install > process) You should really use a newer version if you install from sources. numpy 1.0.4 is one year and a half old. If you have to use pytyhon 2.3, you should use numpy 1.1.*. If you have python 2.4 or later, you should use numpy 1.3.0. > > I had the following error : > > / import numpy > File "../numpy/lib64/python2.3/site-packages/numpy/__init__.py", > line 43, in ? > File > "../numpy/lib64/python2.3/site-packages/numpy/linalg/__init__.py", > line 4, in ? > File > "../numpy/lib64/python2.3/site-packages/numpy/linalg/linalg.py", line > 25, in ? > ImportError: liblapack.so.3: cannot open shared object file: No such > file or directory > / What does ldd ../numpy/lib64/python2.3/site-packages/numpy/linalg/lapack_lite.so say ? > So is there a solution to use numpy 64 bits version with a > /usr/*lib*/liblapack.so.3 or is it absolutly necessary to install > lapack 64 bits version ? It is certainly mandatory to have a 64 bits version of lapack when building numpy for a 64 bits python, but it does not have to be in /usr/lib64. You can also build numpy without lapack: LAPACK=None python setup.py .... cheers, David ------------------------------ Message: 7 Date: Fri, 10 Apr 2009 15:18:52 +0100 From: "Frank Peacock" Subject: [Numpy-discussion] Replacing colours in numpy array To: Message-ID: <003001c9b9e7$4660c220$d3224660$@com> Content-Type: text/plain; charset="us-ascii" Hello I have a numpy array that I obtained from a converted RGBA gif image. I have tried to replace some colours with different ones using the where condition but have a problem with dimensions. If I use b=where(a==0,255,a) where a is a numpy array from an image it does replace components of the RGB values in each pixel but it fails if I try b=where(a==(0,0,0,255),(255,255,255,255),a) with incorrect dimension error. Could you please help? Thanks Frank -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of numpy-discussion-request at scipy.org Sent: 10 April 2009 08:18 To: numpy-discussion at scipy.org Subject: Numpy-discussion Digest, Vol 31, Issue 24 Send Numpy-discussion mailing list submissions to numpy-discussion at scipy.org To subscribe or unsubscribe via the World Wide Web, visit http://mail.scipy.org/mailman/listinfo/numpy-discussion or, via email, send a message with subject or body 'help' to numpy-discussion-request at scipy.org You can reach the person managing the list at numpy-discussion-owner at scipy.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Numpy-discussion digest..." Today's Topics: 1. Re: DVCS at PyCon (Ondrej Certik) 2. Re: using reducing functions without eliminating dimensions? (Dan Lenski) 3. Re: Another Array (Ian Mallett) 4. Re: Another Array (Robert Kern) 5. Re: Another Array (Ian Mallett) 6. Re: Another Array (Robert Kern) 7. Re: Another Array (Ian Mallett) 8. Re: Another Array (Anne Archibald) ---------------------------------------------------------------------- Message: 1 Date: Thu, 9 Apr 2009 23:14:53 -0700 From: Ondrej Certik Subject: Re: [Numpy-discussion] DVCS at PyCon To: Discussion of Numerical Python Message-ID: <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf at mail.gmail.com> Content-Type: text/plain; charset=UTF-8 On Thu, Apr 9, 2009 at 10:45 PM, David Cournapeau wrote: > Ondrej Certik wrote: >> >> It is maybe easier to learn how to work with different clones, but >> once you start working with lots of patches and you need to reclone >> all the time, then it's the wrong approach to work, as it takes lots >> of time to copy the whole repository on the disk. > > Yes, *I* know how to use git, and I agree with you, I vastly prefer git > branch handling to bzr branch handling. *I* find working with GUI for > VCS a real PITA. But I am not the only numpy developer, that's why the > feedback from people like Josef with a totally different workflow than > me is valuable - much more than people like us who are unix geeks :) Yes, definitely. Ondrej ------------------------------ Message: 2 Date: Fri, 10 Apr 2009 06:36:08 +0000 (UTC) From: Dan Lenski Subject: Re: [Numpy-discussion] using reducing functions without eliminating dimensions? To: numpy-discussion at scipy.org Message-ID: Content-Type: text/plain; charset=UTF-8 On Thu, 09 Apr 2009 01:31:33 -0500, Robert Kern wrote: > On Thu, Apr 9, 2009 at 01:29, Anne Archibald > wrote: > >> What's wrong with np.amin(a,axis=-1)[...,np.newaxis]? > > It's cumbersome, particularly when you have axis=arbitrary_axis. Quite right. It would nice to be able to say: np.amin(a, axiskeep=-1) or a.min(axiskeep=3) ... or something along those lines. Dan ------------------------------ Message: 3 Date: Thu, 9 Apr 2009 23:42:03 -0700 From: Ian Mallett Subject: Re: [Numpy-discussion] Another Array To: Discussion of Numerical Python Message-ID: Content-Type: text/plain; charset="iso-8859-1" It gives a perfect parabolic shape that looks very nice, but somewhat unrealistic. I'd like to scale the unit vectors by a random length (which can just be a uniform distribution). I tried scaling the unit vector n*n*3 array by a random n*n array, but that didn't work, obviously. Help? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20090409/e8fea8 42/attachment-0001.html ------------------------------ Message: 4 Date: Fri, 10 Apr 2009 01:46:47 -0500 From: Robert Kern Subject: Re: [Numpy-discussion] Another Array To: Discussion of Numerical Python Message-ID: <3d375d730904092346l6850c03chca1962915263e79c at mail.gmail.com> Content-Type: text/plain; charset=UTF-8 On Fri, Apr 10, 2009 at 01:42, Ian Mallett wrote: > It gives a perfect parabolic shape that looks very nice, but somewhat > unrealistic. Parabolic? They should be spherical. > I'd like to scale the unit vectors by a random length (which > can just be a uniform distribution).? I tried scaling the unit vector n*n*3 > array by a random n*n array, but that didn't work, obviously. No, it's not obvious. Exactly what code did you try? What results did you get? What results were you expecting? > Help? Let's take a step back. What kind of distribution are you trying to achieve? You asked for uniformly distributed unit vectors. Now you are asking for something else, but I'm really not sure what. What standard are you comparing against when you say that the unit vectors look "unrealistic"? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ------------------------------ Message: 5 Date: Thu, 9 Apr 2009 23:58:21 -0700 From: Ian Mallett Subject: Re: [Numpy-discussion] Another Array To: Discussion of Numerical Python Message-ID: Content-Type: text/plain; charset="iso-8859-1" On Thu, Apr 9, 2009 at 11:46 PM, Robert Kern wrote: > Parabolic? They should be spherical. The particle system in the last screenshot was affected by gravity. In the absence of gravity, the results should be spherical, yes. All the vectors are a unit length, which produces a perfectly smooth surface (unrealistic for such an effect). > No, it's not obvious. Exactly what code did you try? What results did > you get? What results were you expecting? It crashed. I have this code: vecs = Numeric.random.standard_normal(size=(self.size[0],self.size[1],3)) magnitudes = Numeric.sqrt((vecs*vecs).sum(axis=-1)) uvecs = vecs / magnitudes[...,Numeric.newaxis] randlen = Numeric.random.random((self.size[0],self.size[1])) randuvecs = uvecs*randlen #It crashes here with a dimension mismatch rgb = ((randvecs+1.0)/2.0)*255.0 I also tried randlen = Numeric.random.random((self.size[0],self.size[1],3)), but this does not scale each of the vector's components equally, producing artifacts again. Each needs to be scaled by the same random value for it to make sense. > Let's take a step back. What kind of distribution are you trying to > achieve? You asked for uniformly distributed unit vectors. Now you are > asking for something else, but I'm really not sure what. What standard > are you comparing against when you say that the unit vectors look > "unrealistic"? The vectors are used to "jitter" each particle's initial speed, so that the particles go in different directions instead of moving all as one. Using the unit vector causes the particles to make the smooth parabolic shape. The jitter vectors much then be of a random length, so that the particles go in all different directions at all different speeds, instead of just all in different directions. Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20090409/8d6bd9 31/attachment-0001.html ------------------------------ Message: 6 Date: Fri, 10 Apr 2009 02:01:10 -0500 From: Robert Kern Subject: Re: [Numpy-discussion] Another Array To: Discussion of Numerical Python Message-ID: <3d375d730904100001w4cdb7bedi2b36bca97731ef8c at mail.gmail.com> Content-Type: text/plain; charset=UTF-8 On Fri, Apr 10, 2009 at 01:58, Ian Mallett wrote: > On Thu, Apr 9, 2009 at 11:46 PM, Robert Kern wrote: >> >> Parabolic? They should be spherical. > > The particle system in the last screenshot was affected by gravity.? In the > absence of gravity, the results should be spherical, yes.? All the vectors > are a unit length, which produces a perfectly smooth surface (unrealistic > for such an effect). >> >> No, it's not obvious. Exactly what code did you try? What results did >> you get? What results were you expecting? > > It crashed. > I have this code: > vecs = Numeric.random.standard_normal(size=(self.size[0],self.size[1],3)) > magnitudes = Numeric.sqrt((vecs*vecs).sum(axis=-1)) > uvecs = vecs / magnitudes[...,Numeric.newaxis] > randlen = Numeric.random.random((self.size[0],self.size[1])) > randuvecs = uvecs*randlen #It crashes here with a dimension mismatch > rgb = ((randvecs+1.0)/2.0)*255.0 > > I also tried randlen = Numeric.random.random((self.size[0],self.size[1],3)), > but this does not scale each of the vector's components equally, producing > artifacts again.? Each needs to be scaled by the same random value for it to > make sense. See how I did magnitudes[...,numpy.newaxis]? You have to do the same. >> Let's take a step back. What kind of distribution are you trying to >> achieve? You asked for uniformly distributed unit vectors. Now you are >> asking for something else, but I'm really not sure what. What standard >> are you comparing against when you say that the unit vectors look >> "unrealistic"? > > The vectors are used to "jitter" each particle's initial speed, so that the > particles go in different directions instead of moving all as one.? Using > the unit vector causes the particles to make the smooth parabolic shape. > The jitter vectors much then be of a random length, so that the particles go > in all different directions at all different speeds, instead of just all in > different directions. Ah, okay. That makes sense. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ------------------------------ Message: 7 Date: Fri, 10 Apr 2009 00:07:45 -0700 From: Ian Mallett Subject: Re: [Numpy-discussion] Another Array To: Discussion of Numerical Python Message-ID: Content-Type: text/plain; charset="iso-8859-1" This seems to work: vecs = Numeric.random.standard_normal(size=(self.size[0],self.size[1],3)) magnitudes = Numeric.sqrt((vecs*vecs).sum(axis=-1)) uvecs = vecs / magnitudes[...,Numeric.newaxis] randlen = Numeric.random.random((self.size[0],self.size[1])) randuvecs = uvecs*randlen[...,Numeric.newaxis] rgb = ((randuvecs+1.0)/2.0)*255.0 (I have "import numpy as Numeric" for other reasons, that's why there's Numeric there). Thanks, Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20090410/84e6aa a1/attachment-0001.html ------------------------------ Message: 8 Date: Fri, 10 Apr 2009 03:17:58 -0400 From: Anne Archibald Subject: Re: [Numpy-discussion] Another Array To: Discussion of Numerical Python Message-ID: Content-Type: text/plain; charset=UTF-8 2009/4/10 Ian Mallett : > The vectors are used to "jitter" each particle's initial speed, so that the > particles go in different directions instead of moving all as one.? Using > the unit vector causes the particles to make the smooth parabolic shape. > The jitter vectors much then be of a random length, so that the particles go > in all different directions at all different speeds, instead of just all in > different directions. Why not just skip the normalization? Then you'll get vectors with random direction and a natural distribution of lengths. And it'll be faster than tne unit vectors... Anne ------------------------------ _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion End of Numpy-discussion Digest, Vol 31, Issue 24 ************************************************ ------------------------------ _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion End of Numpy-discussion Digest, Vol 31, Issue 26 ************************************************ From Chris.Barker at noaa.gov Fri Apr 10 18:16:06 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 10 Apr 2009 15:16:06 -0700 Subject: [Numpy-discussion] numpy.ma.arg{min,max} In-Reply-To: References: Message-ID: <49DFC526.4090602@noaa.gov> Adam Oliner wrote: > The documentation for numpy.ma.argmin says: > "Returns array of indices of the maximum values along the given axis." > > Aside from probably meaning to say the 'minimum' values, that's a typo... > it also doesn't seem to return an array: it does if you use a higher rank array, and specify an axis: >>> a array([[ 0.643502 , 0.83867769, 0.97762954, 0.49686553], [ 0.37798796, 0.69300396, 0.06683333, 0.3457077 ], [ 0.51357262, 0.29095034, 0.42811108, 0.72644421], [ 0.5977396 , 0.64992292, 0.53068753, 0.46386281]]) >>> np.argmin(a, 0) array([1, 2, 1, 1]) >>> np.argmin(a, 1) array([3, 2, 1, 3]) argmin(a, axis=None) Return array of indices to the minimum values along the given axis. Parameters ---------- a : {array_like} Array to look in. axis : {None, integer} If None, the index is into the flattened array, otherwise along the specified axis -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliner at stanford.edu Fri Apr 10 18:46:35 2009 From: oliner at stanford.edu (Adam Oliner) Date: Fri, 10 Apr 2009 15:46:35 -0700 Subject: [Numpy-discussion] numpy.ma.arg{min,max} In-Reply-To: <49DFC526.4090602@noaa.gov> References: <49DFC526.4090602@noaa.gov> Message-ID: Christoper, I see, thanks. Then the documentation should say that it returns an array of the _smallest_ indices of the minimum values along the given axis. If the minimum value is not unique, as in my examples, is there an efficient way to get all of these indices? It seems that I can get roughly the behavior I want with 'where': >>> a array([1, 1, 1, 5, 5]) >>> numpy.ma.where(a==numpy.ma.max(a))[0] array([3, 4]) >>> numpy.ma.where(a==numpy.ma.min(a))[0] array([0, 1, 2]) - Adam J. Oliner oliner at cs.stanford.edu On Apr 10, 2009, at 3:16 PM, Christopher Barker wrote: > Adam Oliner wrote: >> The documentation for numpy.ma.argmin says: >> "Returns array of indices of the maximum values along the given >> axis." >> >> Aside from probably meaning to say the 'minimum' values, > > that's a typo... > > >> it also doesn't seem to return an array: > > it does if you use a higher rank array, and specify an axis: > >>>> a > array([[ 0.643502 , 0.83867769, 0.97762954, 0.49686553], > [ 0.37798796, 0.69300396, 0.06683333, 0.3457077 ], > [ 0.51357262, 0.29095034, 0.42811108, 0.72644421], > [ 0.5977396 , 0.64992292, 0.53068753, 0.46386281]]) > >>>> np.argmin(a, 0) > array([1, 2, 1, 1]) > >>>> np.argmin(a, 1) > > array([3, 2, 1, 3]) > > > argmin(a, axis=None) > Return array of indices to the minimum values along the given > axis. > > Parameters > ---------- > a : {array_like} > Array to look in. > axis : {None, integer} > If None, the index is into the flattened array, otherwise > along > the specified axis > > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From Chris.Barker at noaa.gov Fri Apr 10 19:11:19 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 10 Apr 2009 16:11:19 -0700 Subject: [Numpy-discussion] Constant array of tuples In-Reply-To: <000b01c9ba29$d02d6070$70882150$@com> References: <000b01c9ba29$d02d6070$70882150$@com> Message-ID: <49DFD217.4060502@noaa.gov> Frank Peacock wrote: > Hello > > I would like to know whether there is a simple way to construct a constant > array of tuples: > > How do I construct an array of size (width*height) where each element is > (w,x,y,z) Is this what you want? a = np.empty((5,6), dtype=np.object) >>> for i in range(len(a.flat)): ... a.flat[i] = (1,2,3,4) ... >>> a array([[(1, 2, 3, 4), (1, 2, 3, 4), (1, 2, 3, 4), (1, 2, 3, 4), (1, 2, 3, 4), (1, 2, 3, 4)], [(1, 2, 3, 4), (1, 2, 3, 4), (1, 2, 3, 4), (1, 2, 3, 4), (1, 2, 3, 4), (1, 2, 3, 4)], [(1, 2, 3, 4), (1, 2, 3, 4), (1, 2, 3, 4), (1, 2, 3, 4), (1, 2, 3, 4), (1, 2, 3, 4)], [(1, 2, 3, 4), (1, 2, 3, 4), (1, 2, 3, 4), (1, 2, 3, 4), (1, 2, 3, 4), (1, 2, 3, 4)], [(1, 2, 3, 4), (1, 2, 3, 4), (1, 2, 3, 4), (1, 2, 3, 4), (1, 2, 3, 4), (1, 2, 3, 4)]], dtype=object) Interestingly: >>> a[:,:] = (1,2,3,4) Traceback (most recent call last): File "", line 1, in ValueError: shape mismatch: objects cannot be broadcast to a single shape Doesn't work - even though (1,2,3,4) is a tuple, which is a single object. Anyway, I suspect that that's not really what you want, but rather, you might be looking for: >>> a = np.zeros((w,h,4), np.float) >>> a[:,:] = (1,2,3,4) >>> a array([[[ 1., 2., 3., 4.], [ 1., 2., 3., 4.], [ 1., 2., 3., 4.], [ 1., 2., 3., 4.], [ 1., 2., 3., 4.]], [[ 1., 2., 3., 4.], [ 1., 2., 3., 4.], [ 1., 2., 3., 4.], [ 1., 2., 3., 4.], [ 1., 2., 3., 4.]], [[ 1., 2., 3., 4.], [ 1., 2., 3., 4.], [ 1., 2., 3., 4.], [ 1., 2., 3., 4.], [ 1., 2., 3., 4.]]]) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From frank at gis4weather.com Fri Apr 10 20:28:13 2009 From: frank at gis4weather.com (Frank Peacock) Date: Sat, 11 Apr 2009 01:28:13 +0100 Subject: [Numpy-discussion] Using where with tuples Message-ID: <000001c9ba3c$66781680$33684380$@com> Hello I am trying to use the where function on a numpy array which was obtained from an image using the asarray function. The code is as follows: from numpy import * from Image import * im=open("a.gif") im2=im.convert("RGBA") a2 = asarray(im2,uint8) c = zeros((140,90,4),uint8) c[:,:] = [0,0,0,255] b=where(a2[:,:]==(255,0,0,255),c,a2) c=fromarray(b,"RGBA") c.save("c.gif") I am trying to convert only red colours to black but although this works it also converts other tuples as well. How do I ensure that the tuple condition is enforced exactly, i.e. only on red colours? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Apr 10 21:30:52 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 10 Apr 2009 21:30:52 -0400 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> Message-ID: <1cd32cbb0904101830w26058c27i1c9ecdb1cd87f0a9@mail.gmail.com> On Fri, Apr 10, 2009 at 5:47 PM, Matthew Brett wrote: > Hi, > > I enjoyed this quote from http://www.eecs.harvard.edu/~cduan/technical/git/ > > Summary: You can only really use Git if you understand how Git works. > > "When I first started using Git, I read plenty of tutorials, as well > as the user manual. Though I picked up the basic usage patterns and > commands, I never felt like I grasped what was going on ?under the > hood?, so to speak. Frequently this resulted in cryptic error > messages, caused by my random guessing at the right command to use at > a given time. These difficulties worsened as I began to need more > advanced (and less well documented) features. > > After a few months, I started to understand those under-the-hood > concepts. Once I did, suddenly everything made sense. I could > understand the manual pages and perform all sorts of source control > tasks. Everything that seemed so cryptic and obscure now was perfectly > clear. > Understanding Git > > The conclusion I draw from this is that you can only really use Git if > you understand how Git works. Merely memorizing which commands you > should run at what times will work in the short run, but it?s only a > matter of time before you get stuck or, worse, break something." > break something: I updated my eclipse so I can try out the git eclipse plugin. Except for a description how to clone a github repository and push back to it, I didn't find much information on the internet. cloning from a remote repository (pymvpa), setting it up for local version control in eclipse and converting to a pydev project went very easily. Creating local branches and a clone in a parallel directory tree also worked. >From then on its working mostly blind. Working with branches in one repository doesn't clearly indicate which branch I'm in and which branch my files belong to. `checkout` doesn't appear to change the visible files on the hard drive as with the command line, switching a branch with `reset` can be soft, mixed or hard. `hard` changes the files with a warning that all edits will be overwritten, but it doesn't tell whether there are any uncommitted changes in files. Fetching between clones seemed to work, there are still some things that I need to figure out, it wasn't always clear whether my commits actually happened or not. But finally with switching around, I managed to get a "circular reference" in my clone, and now I cannot access it as git repository any more. On the command line, git says this is not a repository. So I guess it's gone with all the changes that are not currently on the file system. I still miss a git directory browser with status information for each file, and I still didn't manage to get diffs between branches or clones. I can fetch or push, but I cannot see directly what it is that I'm pulling or pushing, except maybe a long alphanumeric status code. Without a help file, I wasn't sure what I was doing, but, for sure, git eclipse is not foolproof and not very self-explanatory. For someone used to a real and not a virtual file system, I will, for now, stay away from saving any of my edits in branches, that might disappear when I make a mistake. And both git gui and the eclipse plugin seem to require still some work before they are really useful, unless there are some working examples and recipes to proof my impression wrong. But from the webpages of the gui and eclipse plugin, there seems to be currently a lot of work going on. Simple cloning from and committing to a real remote repository seems to work easily enough, but then there isn't much reason to switch away from svn, if maintaining several local branches is a risky pain. So for users who don't "get it", the branch system of bazar and mercurial is a lot more intuitive than git. Josef From efiring at hawaii.edu Fri Apr 10 21:59:28 2009 From: efiring at hawaii.edu (Eric Firing) Date: Fri, 10 Apr 2009 15:59:28 -1000 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> Message-ID: <49DFF980.8040407@hawaii.edu> Matthew Brett wrote: > Hi, > > I enjoyed this quote from http://www.eecs.harvard.edu/~cduan/technical/git/ > > Summary: You can only really use Git if you understand how Git works. Matthew, Nice link, thank you. Another couple of quotes from that tutorial: Important note: if there are any uncommitted changes when you run git checkout, Git will behave very strangely. The strangeness is predictable and sometimes useful, but it is best to avoid it. All you need to do, of course, is commit all the new changes before checking out the new head. Important note: Git can get very confused if there are uncommitted changes in the files when you ask it to perform a merge. So make sure to commit whatever changes you have made so far before you merge. Sounds like booby traps for occasional users, or for people who are multitasking a little too much. The hg UI approach is different: it blocks that sort of thing with a warning, and provides override options when needed. And for comic relief, here is a line from a git man page: The state you are in while your HEAD is detached is not recorded by any branch... It hurts to even think about such a state... Eric From cournape at gmail.com Fri Apr 10 22:00:13 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 11 Apr 2009 11:00:13 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <1cd32cbb0904101830w26058c27i1c9ecdb1cd87f0a9@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <1cd32cbb0904101830w26058c27i1c9ecdb1cd87f0a9@mail.gmail.com> Message-ID: <5b8d13220904101900o6c90132fyfe3afb71de6e83f@mail.gmail.com> On Sat, Apr 11, 2009 at 10:30 AM, wrote: > > I updated my eclipse so I can try out the git eclipse plugin. Except > for a description how to clone a github repository and push back to > it, I didn't find much information on the internet. FWIW, I tried the eclipse plugin yesterday, and I did not understand anything at all about it either. I assumed it was because I was not familiar with eclipse, but from what you say, it looks like it is seriously broken. David From cournape at gmail.com Fri Apr 10 22:17:56 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 11 Apr 2009 11:17:56 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49DFF980.8040407@hawaii.edu> References: <49CE3A2C.9000007@enthought.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> Message-ID: <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> On Sat, Apr 11, 2009 at 10:59 AM, Eric Firing wrote: > > Important note: if there are any uncommitted changes when you run git > checkout, Git will behave very strangely. The strangeness is predictable > and sometimes useful, but it is best to avoid it. All you need to do, of > course, is commit all the new changes before checking out the new head. If by strange and confused, the OP means refuse to change branch, then yes. Otherwise, I have no idea what he is talking about. Maybe an old git version - he does not say which one he is using. Also, it is said in the introduction that the OP was using git but did not understand what was happening under the hood. So his conclusion is you have to understand git internals to understand git internals ? Sounds tautological to me. I am all for hearing git bashing by people - but I would prefer if it was coming after actual use of the tool. > > The state you are in while your HEAD is detached is not recorded by any > ? ? ? ?branch... The only occasion it happened to me was with git-svn - which I would no advocate using as a transition path, indeed. It is great for individual people (I use it all the time for numpy) but it is definitely strange UI-wise sometimes. David From praxbaffle at hotmail.com Fri Apr 10 22:33:10 2009 From: praxbaffle at hotmail.com (John Seales) Date: Fri, 10 Apr 2009 19:33:10 -0700 Subject: [Numpy-discussion] FFTW?? Do I need it to do Fourier transforms? In-Reply-To: <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> Message-ID: The link to FFTW on http://www.scipy.org/Installing_SciPy/Mac_OS_X is broken. Is it needed to do Fourier Transforms? My main motivation for using numpy and scipy is to do spectral analysis of sound. _________________________________________________________________ Rediscover Hotmail?: Get quick friend updates right in your inbox. http://windowslive.com/RediscoverHotmail?ocid=TXT_TAGLM_WL_HM_Rediscover_Updates1_042009 -------------- next part -------------- An HTML attachment was scrubbed... URL: From efiring at hawaii.edu Fri Apr 10 22:53:53 2009 From: efiring at hawaii.edu (Eric Firing) Date: Fri, 10 Apr 2009 16:53:53 -1000 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> Message-ID: <49E00641.5050903@hawaii.edu> David Cournapeau wrote: > On Sat, Apr 11, 2009 at 10:59 AM, Eric Firing wrote: > >> Important note: if there are any uncommitted changes when you run git >> checkout, Git will behave very strangely. The strangeness is predictable >> and sometimes useful, but it is best to avoid it. All you need to do, of >> course, is commit all the new changes before checking out the new head. > > If by strange and confused, the OP means refuse to change branch, then > yes. Otherwise, I have no idea what he is talking about. Maybe an old > git version - he does not say which one he is using. Could be. All of these tools are moving targets. The tutorial was last modified in June, 2008. > > Also, it is said in the introduction that the OP was using git but did > not understand what was happening under the hood. So his conclusion is > you have to understand git internals to understand git internals ? > Sounds tautological to me. No, I think the point of Duan's introduction was that *initially* he had trouble *using* git because he did not understand its internals; therefore he was writing a tutorial that would explain the usage in terms of what the commands do internally, instead of simply providing lists of commands to accomplish a given set of tasks. The description of internals that he provides is pretty minimal, though. > > I am all for hearing git bashing by people - but I would prefer if it > was coming after actual use of the tool. Please understand, I am not trying to bash git--it is clearly an enormously powerful and well-made tool--and I apologize if my posts have appeared to tend in that direction. I guess I am trying to keep hg from being dismissed too quickly, and I am trying to understand what the similarities and differences are, and what consequences the choice of one or the other would be likely to have. The quotes were from a tutorial http://www.eecs.harvard.edu/~cduan/technical/git/ referenced by Matthew. The author of the tutorial appears to have been writing after actual use of git. I certainly am not, and will say no more about it. > >> The state you are in while your HEAD is detached is not recorded by any >> branch... > I included the above quote because it made me laugh, and I hoped it would do the same for others. That's all. Eric > The only occasion it happened to me was with git-svn - which I would > no advocate using as a transition path, indeed. It is great for > individual people (I use it all the time for numpy) but it is > definitely strange UI-wise sometimes. > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From cournape at gmail.com Fri Apr 10 22:56:40 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 11 Apr 2009 11:56:40 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49DFB726.3000002@hawaii.edu> References: <49CE3A2C.9000007@enthought.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> Message-ID: <5b8d13220904101956v15cc4184r7c8144a519c369ea@mail.gmail.com> On Sat, Apr 11, 2009 at 6:16 AM, Eric Firing wrote: > > On my laptop, switching back and forth between the two active branches > of mpl takes about 3 s for the first and 2 s for the second, timed by > counting in my head. I think Ondrej cares more about being faster than most of us - he is just too fast for most of us :) I don't think speed should be an argument, because both are fast enough. > > I'm on thin ice here, because for my own work I have not been using > named branches. > >> ? ? - you can't remove them later, it is in the repository 'forever' > > They can be removed with the strip command which is in the mq extension, > but one must identify the root of the branch and supply that to strip. > There may be some limits and gotchas. Ok, will look at it more closely (I have just made a hg repo from the numpy git one on my machine). I am still confused about how named branches are supposed to work (the hg book is a a bit light on this). For example, let's say I want to create a branch from a given revision/tag, how should I do it ? Since hg branch does not take an -r revision, I tried to create the branch first, and then use hg up -r N, but it did not commit as I expected from a git perspective: hg branch foo hg up -r 500 hg log # show me last revision independently from the branch changes. Not so surprising given that the branch was created from the tip of default. hg ci -m "bla" hg branches # no branch foo ? What am I doing wrong ? > For very lightweight local branching there are bookmarks, which allow > one to make a local, changeable label for a given head within a repo. > Again, such a local branch can be eliminated via strip--although I am > not sure there is much point in doing so. ?To go this route, I suspect > one would first commit a tag, set the bookmark, make whatever changes > and commits are desired, etc. ?The point of the tag would be to make it > easy to tell strip where to start stripping. Ah, I think I was confused between named branches and bookmarks. From the description of bookmarks, this actually looks nearer to the git cheap branch concept. I should really try it to get a good understanding, though. > Overall, it appears to me that for someone who has never > used a VCS, or for someone used to svn, hg has a lower barrier to entry > than git. It may be true for branches - but the basic (commit/log/pull/push/clone) is almost exactly the same between the two of them (and for bzr as well). So for someone who keeps a svn-like workflow, I don't see big differences between the tools. >> >> One big potential with the way git does branching is review - gerrit >> (the code review tool used by Google for android) looks really nice. > > How is the code review related to the branching options in git? A good explanation is there: http://groups.google.com/group/repo-discuss/browse_thread/thread/f8365b4217157cd4/2c3c729082bdd1b7?#2c3c729082bdd1b7 If you have used rietveld (from which gerrit was forked), it is a bit similar UI-wise, but git makes some things more automatic (if the review is good, the changes are automatically merged, for example). But again, I said potentially because we would need to really try it with other people for review to get a real idea. If there is one thing that I learned why using different tools, it is that the only way to know about them is to stop reading about them, and actually use them :) > You can easily do diffs between changesets within a repo, regardless of > whether they are on different named branches, and the diffs can be > specific to a file or subdirectory. ?Changesets can be identified by > branchnames, tags, bookmarks, sequential numbers, or hash ID. For diffs > between directories, there is an rdiff extension that is not distributed > with hg. ?I haven't tried it. Yes, I have just tried this as well, and it works as I expected - once I understand that what mercurial calls a revision includes branch name as well. It is a bit slower than git, but really nothing prohibitive. David From cournape at gmail.com Fri Apr 10 23:05:55 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 11 Apr 2009 12:05:55 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49E00641.5050903@hawaii.edu> References: <49CE3A2C.9000007@enthought.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> <49E00641.5050903@hawaii.edu> Message-ID: <5b8d13220904102005y21503770r32ac48c97c412849@mail.gmail.com> On Sat, Apr 11, 2009 at 11:53 AM, Eric Firing wrote: > > Please understand, I am not trying to bash git--it is clearly an > enormously powerful and well-made tool--and I apologize if my posts have > appeared to tend in that direction. No need to apologize, I think I used the work bashing inappropriately - I just wanted to say that the only way to understand the differences between the tools is to use them. Reading about them will only confuse you in my own experience. For example, I tried git once a long time ago (during an infamous discussion between git and bzr developers on the bzr M), could not understand a thing about it, and did not understand any point in it except speed. Then I was forced to use git because of bzr-svn constant frustrations - and I ended up to really like it. At last scipy conference, I tried to "sell" git advantages to Stefan (a long time bzr user as well), who was far from convinced from my explanations and my ranting. Then later he used it and liked it. Of course, the logical conclusion could be that I am just very bad at explaining why I like git :) >?I guess I am trying to keep hg from > being dismissed too quickly I am not dismissing anything - certainly if many people hate git, it will be out of the possible choices. David From charlesr.harris at gmail.com Fri Apr 10 23:37:58 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 10 Apr 2009 21:37:58 -0600 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> Message-ID: On Fri, Apr 10, 2009 at 8:17 PM, David Cournapeau wrote: > On Sat, Apr 11, 2009 at 10:59 AM, Eric Firing wrote: > > > > > Important note: if there are any uncommitted changes when you run git > > checkout, Git will behave very strangely. The strangeness is predictable > > and sometimes useful, but it is best to avoid it. All you need to do, of > > course, is commit all the new changes before checking out the new head. > > If by strange and confused, the OP means refuse to change branch, then > yes. Otherwise, I have no idea what he is talking about. Maybe an old > git version - he does not say which one he is using. > > Also, it is said in the introduction that the OP was using git but did > not understand what was happening under the hood. So his conclusion is > you have to understand git internals to understand git internals ? > Sounds tautological to me. > I think he meant that it is easier to correctly use the knobs and levers of git if you understand what they relate to the repository, and that means knowing how the depository tracks things. I found his exposition helpful. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Apr 10 23:40:44 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 10 Apr 2009 21:40:44 -0600 Subject: [Numpy-discussion] FFTW?? Do I need it to do Fourier transforms? In-Reply-To: References: <49CE3A2C.9000007@enthought.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> Message-ID: On Fri, Apr 10, 2009 at 8:33 PM, John Seales wrote: > The link to FFTW on http://www.scipy.org/Installing_SciPy/Mac_OS_X is > broken. > > Is it needed to do Fourier Transforms? My main motivation for using numpy > and scipy is to do spectral analysis of sound. > > No, it isn't needed. The fft is available in numpy itself as well as scipy. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Fri Apr 10 23:49:35 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 11 Apr 2009 12:49:35 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: References: <49CE3A2C.9000007@enthought.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> Message-ID: <5b8d13220904102049w52a5926aha48be7fc1843201e@mail.gmail.com> On Sat, Apr 11, 2009 at 12:37 PM, Charles R Harris wrote: > > > On Fri, Apr 10, 2009 at 8:17 PM, David Cournapeau > wrote: >> >> On Sat, Apr 11, 2009 at 10:59 AM, Eric Firing wrote: >> >> > >> > Important note: if there are any uncommitted changes when you run git >> > checkout, Git will behave very strangely. The strangeness is predictable >> > and sometimes useful, but it is best to avoid it. All you need to do, of >> > course, is commit all the new changes before checking out the new head. >> >> If by strange and confused, the OP means refuse to change branch, then >> yes. Otherwise, I have no idea what he is talking about. Maybe an old >> git version - he does not say which one he is using. >> >> Also, it is said in the introduction that the OP was using git but did >> not understand what was happening under the hood. So his conclusion is >> you have to understand git internals to understand git internals ? >> Sounds tautological to me. > > I think he meant that it is easier to correctly use the knobs and levers of > git if you understand what they relate to the repository, and that means > knowing how the depository tracks things. I found his exposition helpful. Yes, his exposition is well written otherwise - much better than what you find on more "google visible" git expositions (I happen to think that git is badly "marketed" by some vocal people). But does this really look harder than hg ? I can't see a big difference. You need to know about heads, branches, parents and the likes in any DVCS if you use branches, no ? And the ton of my previous answer was not appropriate anyway, I apologize. David From cournape at gmail.com Fri Apr 10 23:50:46 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 11 Apr 2009 12:50:46 +0900 Subject: [Numpy-discussion] help getting started In-Reply-To: <49DFAF23.5030100@noaa.gov> References: <5b8d13220904101110n5f39f7cdwd5a5fb85c1d1c405@mail.gmail.com> <49DFAF23.5030100@noaa.gov> Message-ID: <5b8d13220904102050l4ee0e7fdhb9b3b23d9e21a133@mail.gmail.com> On Sat, Apr 11, 2009 at 5:42 AM, Christopher Barker wrote: > > Last I checked, the binary for ScipPy was broken, you'll need to get > gfortran and compile it yourself, as the page tells you. This should really be fixed, it is not acceptable to have a broken binary. I will generate a new one, David From charlesr.harris at gmail.com Sat Apr 11 00:00:12 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 10 Apr 2009 22:00:12 -0600 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <5b8d13220904102049w52a5926aha48be7fc1843201e@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> <5b8d13220904102049w52a5926aha48be7fc1843201e@mail.gmail.com> Message-ID: On Fri, Apr 10, 2009 at 9:49 PM, David Cournapeau wrote: > On Sat, Apr 11, 2009 at 12:37 PM, Charles R Harris > wrote: > > > > > > On Fri, Apr 10, 2009 at 8:17 PM, David Cournapeau > > wrote: > >> > >> On Sat, Apr 11, 2009 at 10:59 AM, Eric Firing > wrote: > >> > >> > > >> > Important note: if there are any uncommitted changes when you run git > >> > checkout, Git will behave very strangely. The strangeness is > predictable > >> > and sometimes useful, but it is best to avoid it. All you need to do, > of > >> > course, is commit all the new changes before checking out the new > head. > >> > >> If by strange and confused, the OP means refuse to change branch, then > >> yes. Otherwise, I have no idea what he is talking about. Maybe an old > >> git version - he does not say which one he is using. > >> > >> Also, it is said in the introduction that the OP was using git but did > >> not understand what was happening under the hood. So his conclusion is > >> you have to understand git internals to understand git internals ? > >> Sounds tautological to me. > > > > I think he meant that it is easier to correctly use the knobs and levers > of > > git if you understand what they relate to the repository, and that means > > knowing how the depository tracks things. I found his exposition helpful. > > Yes, his exposition is well written otherwise - much better than what > you find on more "google visible" git expositions (I happen to think > that git is badly "marketed" by some vocal people). But does this > really look harder than hg ? I can't see a big difference. You need to > know about heads, branches, parents and the likes in any DVCS if you > use branches, no ? > I think hg works in a similar manner to git. At least Linus said so in that old google talk ;) But hg doesn't/didn't have the same superstructure built on top of the basic repository idea. However, I'm not familiar with the hg internals... > And the ton of my previous answer was not appropriate anyway, I apologize. > Heh. And on good Friday too ;) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Sat Apr 11 00:14:11 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 10 Apr 2009 21:14:11 -0700 Subject: [Numpy-discussion] help getting started In-Reply-To: <5b8d13220904102050l4ee0e7fdhb9b3b23d9e21a133@mail.gmail.com> References: <5b8d13220904101110n5f39f7cdwd5a5fb85c1d1c405@mail.gmail.com> <49DFAF23.5030100@noaa.gov> <5b8d13220904102050l4ee0e7fdhb9b3b23d9e21a133@mail.gmail.com> Message-ID: <49E01913.1030805@noaa.gov> David Cournapeau wrote: > wrote: >> Last I checked, the binary for ScipPy was broken, you'll need to get >> gfortran and compile it yourself, as the page tells you. > > This should really be fixed, it is not acceptable to have a broken > binary. I will generate a new one, That would be great. IIRC, there were two issues: one was intel-only binaries generated by gfortran, and the other was libgfortran missing. By the way, how id you end up building the new numpy binary? Any good ideas on the virtualenv issues? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception From cournape at gmail.com Sat Apr 11 00:47:47 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 11 Apr 2009 13:47:47 +0900 Subject: [Numpy-discussion] help getting started In-Reply-To: <49E01913.1030805@noaa.gov> References: <5b8d13220904101110n5f39f7cdwd5a5fb85c1d1c405@mail.gmail.com> <49DFAF23.5030100@noaa.gov> <5b8d13220904102050l4ee0e7fdhb9b3b23d9e21a133@mail.gmail.com> <49E01913.1030805@noaa.gov> Message-ID: <5b8d13220904102147h3620a9d8lcd7af04ed9269a04@mail.gmail.com> On Sat, Apr 11, 2009 at 1:14 PM, Christopher Barker wrote: > David Cournapeau wrote: >> wrote: >>> Last I checked, the binary for ScipPy was broken, you'll need to get >>> gfortran and compile it yourself, as the page tells you. >> >> This should really be fixed, it is not acceptable to have a broken >> binary. I will generate a new one, > > That would be great. IIRC, there were two issues: one was intel-only > binaries generated by gfortran, and the other was libgfortran missing. ATM, I am dealing with a third one: scipy.test() crashes right away when importing scipy.cluster :( > > By the way, how id you end up building the new numpy binary? Any good > ideas on the virtualenv issues? I solved it the most expedient way for now: hardcoding the python interpreter to use, and building from scratch, David From efiring at hawaii.edu Sat Apr 11 00:50:36 2009 From: efiring at hawaii.edu (Eric Firing) Date: Fri, 10 Apr 2009 18:50:36 -1000 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <5b8d13220904101956v15cc4184r7c8144a519c369ea@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <5b8d13220904101956v15cc4184r7c8144a519c369ea@mail.gmail.com> Message-ID: <49E0219C.1050308@hawaii.edu> David Cournapeau wrote: > On Sat, Apr 11, 2009 at 6:16 AM, Eric Firing wrote: > >> On my laptop, switching back and forth between the two active branches >> of mpl takes about 3 s for the first and 2 s for the second, timed by >> counting in my head. > > I think Ondrej cares more about being faster than most of us - he is > just too fast for most of us :) I don't think speed should be an > argument, because both are fast enough. > >> I'm on thin ice here, because for my own work I have not been using >> named branches. >> >>> - you can't remove them later, it is in the repository 'forever' >> They can be removed with the strip command which is in the mq extension, >> but one must identify the root of the branch and supply that to strip. >> There may be some limits and gotchas. Also, if you just want to remove the branch from the list of branches, as opposed to expunging the changesets it contains, then you can use hg commit --close_branch. Strangely, though, you then have to use the -a option of branches to exclude it from the list; and you can still go back to it and resume work on it. > > Ok, will look at it more closely (I have just made a hg repo from the > numpy git one on my machine). I am still confused about how named > branches are supposed to work (the hg book is a a bit light on this). > For example, let's say I want to create a branch from a given > revision/tag, how should I do it ? Since hg branch does not take an -r > revision, I tried to create the branch first, and then use hg up -r N, > but it did not commit as I expected from a git perspective: > > hg branch foo > hg up -r 500 > hg log # show me last revision independently from the branch changes. > Not so surprising given that the branch was created from the tip of > default. > hg ci -m "bla" > hg branches # no branch foo ? > > What am I doing wrong ? Try something like this sequence (substituting your editor for "z" and adding random junk each time): 504 mkdir hgtest 505 cd hgtest 506 hg init 507 z test.txt 509 hg add 510 hg commit -m first 511 z test.txt 512 hg commit -m second 513 z test.txt 514 hg commit -m third 515 hg log 516 hg up -r 1 517 hg branch from1 518 hg tag from1tag 519 hg status 520 hg log 521 z test.txt 522 hg commit -m "first change after tag on from1" 523 hg up -r default 524 cat test.txt 525 history 526 hg branches 527 hg branch 528 hg up -r from1 529 hg branch hg branch foo saves the name foo to be used as the branch name *starting* with the next commit. I arbitrarily made that next commit a tag to identify the base of the new branch, and then made additional commits. hg up -r foo switches the working directory to the tip (head) of branch foo; "hg up" does all changing of location within the repo, regardless of whether the location is identified by a branch, a tag, etc. hg branches lists branches; with -a it omits closed branches. One oddity you may notice in the example above is that the changeset resulting from the tag command is *after* the changeset that gets tagged. Tags are just entries in a revision-controlled file, so the tag command changes that file, and then commits the change. Any revision can be tagged at any time. > >> For very lightweight local branching there are bookmarks, which allow >> one to make a local, changeable label for a given head within a repo. >> Again, such a local branch can be eliminated via strip--although I am >> not sure there is much point in doing so. To go this route, I suspect >> one would first commit a tag, set the bookmark, make whatever changes >> and commits are desired, etc. The point of the tag would be to make it >> easy to tell strip where to start stripping. > > Ah, I think I was confused between named branches and bookmarks. From > the description of bookmarks, this actually looks nearer to the git > cheap branch concept. I should really try it to get a good > understanding, though. I would not be surprised if bookmarks were directly inspired by that. There is also a localbranch extension, but it is not included in the distribution and hasn't gotten much attention. I have the sense that bookmarks were designed to accomplish the same thing. Eric From charlesr.harris at gmail.com Sat Apr 11 01:09:03 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 10 Apr 2009 23:09:03 -0600 Subject: [Numpy-discussion] Bug or feature? Message-ID: Ticket #1083 , In [3]: np.array([324938], dtype=np.uint8) Out[3]: array([74], dtype=uint8) i.e., 324938 is silently downcast. This is common numpy behavior, but I wonder if this case shouldn't be an exception. Or in general, if conversion from out of range python numbers should raise an error. It does for the int32 case on 32 bit machines: In [27]: np.array([2**32 - 1], dtype=np.uint32) Out[27]: array([4294967295], dtype=uint32) In [28]: np.array([2**32], dtype=np.uint32) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) /home/charris/ in () ValueError: setting an array element with a sequence. Although the error message seems singularly inappropriate. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sat Apr 11 01:25:53 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 11 Apr 2009 14:25:53 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: References: <49CE3A2C.9000007@enthought.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> <5b8d13220904102049w52a5926aha48be7fc1843201e@mail.gmail.com> Message-ID: <5b8d13220904102225sa2a5e82ra335bfcb6a856ca0@mail.gmail.com> On Sat, Apr 11, 2009 at 1:00 PM, Charles R Harris wrote: > I think hg works in a similar manner to git. At least Linus said so in that > old google talk ;) Yes, compared to svn, hg, git and bzr are quite similar in a way. I think the differences still matter, though. > But hg doesn't/didn't have the same superstructure built > on top of the basic repository idea. Yes, it means you can build a lot of things around git, which is very flexible. You have the repo tool to deal with many modules, gerrit for code review, etc... But this flexibility comes with a price at the UI level - price which has diminished a lot, but is still not zero. I think this cost is nearly 0 for simple workflows, but it looks like I am relatively alone on this :) I could see (but still not entirely convinced) that hg could be simpler but from the hg book, it seems that that's only because it hides or prevent some advanced things (what Teo Tso' means with git has more legs http://tytso.livejournal.com/29467.html). And it has some things similar to bzr which I personally dislike deeply (simple revision). But that's a personal preference which does not weight as much for numpy in general. I would really need to use hg on a daily basis to get a good idea, specially w.r.t branch handling - that takes time in any VCS. That's the only way - it took me time to appreciate git branches compared to bz's way. David From charlesr.harris at gmail.com Sat Apr 11 01:54:59 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 10 Apr 2009 23:54:59 -0600 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <5b8d13220904102225sa2a5e82ra335bfcb6a856ca0@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> <5b8d13220904102049w52a5926aha48be7fc1843201e@mail.gmail.com> <5b8d13220904102225sa2a5e82ra335bfcb6a856ca0@mail.gmail.com> Message-ID: On Fri, Apr 10, 2009 at 11:25 PM, David Cournapeau wrote: > On Sat, Apr 11, 2009 at 1:00 PM, Charles R Harris > wrote: > > > I think hg works in a similar manner to git. At least Linus said so in > that > > old google talk ;) > > Yes, compared to svn, hg, git and bzr are quite similar in a way. I > think the differences still matter, though. > > > But hg doesn't/didn't have the same superstructure built > > on top of the basic repository idea. > > Yes, it means you can build a lot of things around git, which is very > flexible. You have the repo tool to deal with many modules, gerrit for > code review, etc... But this flexibility comes with a price at the UI > level - price which has diminished a lot, but is still not zero. I > think this cost is nearly 0 for simple workflows, but it looks like I > am relatively alone on this :) > I use git myself and find it quite nice. The lack of a good gui on windows is what keeps me from recommending it for general use. I also think it is a good idea to move slowly on major changes like choice of source control. A project like kde can move faster because it is mostly unix specific and the developers aren't likely to complain about the command line. It seems that designing an intuitive gui that fits a dvcs isn't all that easy, and git is a bit of a jump because of the way it treats branches and the irrelevence of time as opposed to change set sequence. I think these issues will be worked out because dvcs *is* the wave of the future. And someday numpy will make the switch. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wright at esrf.fr Sat Apr 11 02:47:44 2009 From: wright at esrf.fr (Jon Wright) Date: Sat, 11 Apr 2009 08:47:44 +0200 Subject: [Numpy-discussion] FFTW?? Do I need it to do Fourier transforms? In-Reply-To: References: <49CE3A2C.9000007@enthought.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> Message-ID: <49E03D10.3060609@esrf.fr> Charles R Harris wrote: > On Fri, Apr 10, 2009 at 8:33 PM, John Seales > wrote: > > The link to FFTW on http://www.scipy.org/Installing_SciPy/Mac_OS_X > is broken. > > Is it needed to do Fourier Transforms? My main motivation for using > numpy and scipy is to do spectral analysis of sound. > > > No, it isn't needed. The fft is available in numpy itself as well as scipy. You can also get ctypes based fftw wrappers from: http://developer.berlios.de/projects/pyfftw/ These worked for me, but do behave differently to the numpy fft (factors of N, speed and planning). Best, Jon From gael.varoquaux at normalesup.org Sat Apr 11 05:52:50 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 11 Apr 2009 11:52:50 +0200 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <5b8d13220904102005y21503770r32ac48c97c412849@mail.gmail.com> References: <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> <49E00641.5050903@hawaii.edu> <5b8d13220904102005y21503770r32ac48c97c412849@mail.gmail.com> Message-ID: <20090411095250.GB8496@phare.normalesup.org> On Sat, Apr 11, 2009 at 12:05:55PM +0900, David Cournapeau wrote: > On Sat, Apr 11, 2009 at 11:53 AM, Eric Firing wrote: > No need to apologize, I think I used the work bashing inappropriately > - I just wanted to say that the only way to understand the differences > between the tools is to use them. Reading about them will only confuse > you in my own experience. For example, I tried git once a long time > ago (during an infamous discussion between git and bzr developers on > the bzr M), could not understand a thing about it, and did not > understand any point in it except speed. Then I was forced to use git > because of bzr-svn constant frustrations - and I ended up to really > like it. > At last scipy conference, I tried to "sell" git advantages to Stefan > (a long time bzr user as well), who was far from convinced from my > explanations and my ranting. Then later he used it and liked it. Of > course, the logical conclusion could be that I am just very bad at > explaining why I like git :) I am pretty convinced that git is an excellent tool, but it forces people to invest a fare amount of time to learn it. I struggle quite a lot to have people use _any_ VCS. I just whish they'd make a usability effort. They could. There are a lot of low hanging fruits. But they don't care. It is geeks programming for geeks, not for normal users, IMHO. Just to make it clear: I could learn git. I wouldn't mind. I just don't think raising the bar too high to new contributors is good. You have to start somewhere, and the best way to get new contributors (new in general, and not only new for scipy) is to make it easy for them. My 2 cents, Ga?l From gael.varoquaux at normalesup.org Sat Apr 11 05:55:06 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 11 Apr 2009 11:55:06 +0200 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: References: <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> <5b8d13220904102049w52a5926aha48be7fc1843201e@mail.gmail.com> <5b8d13220904102225sa2a5e82ra335bfcb6a856ca0@mail.gmail.com> Message-ID: <20090411095506.GC8496@phare.normalesup.org> On Fri, Apr 10, 2009 at 11:54:59PM -0600, Charles R Harris wrote: > It seems that designing an intuitive gui that fits a dvcs isn't all that > easy, and git is a bit of a jump because of the way it treats branches and > the irrelevence of time as opposed to change set sequence. I think these > issues will be worked out because dvcs *is* the wave of the future. And > someday numpy will make the switch. Well said! Ga?l From cournape at gmail.com Sat Apr 11 06:38:42 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 11 Apr 2009 19:38:42 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <20090411095250.GB8496@phare.normalesup.org> References: <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> <49E00641.5050903@hawaii.edu> <5b8d13220904102005y21503770r32ac48c97c412849@mail.gmail.com> <20090411095250.GB8496@phare.normalesup.org> Message-ID: <5b8d13220904110338h1a249ddfi38d31fa82b5a2559@mail.gmail.com> On Sat, Apr 11, 2009 at 6:52 PM, Gael Varoquaux wrote: > On Sat, Apr 11, 2009 at 12:05:55PM +0900, David Cournapeau wrote: >> On Sat, Apr 11, 2009 at 11:53 AM, Eric Firing wrote: >> No need to apologize, I think I used the work bashing inappropriately >> - I just wanted to say that the only way to understand the differences >> between the tools is to use them. Reading about them will only confuse >> you in my own experience. For example, I tried git once a long time >> ago (during an infamous discussion between git and bzr developers on >> the bzr M), could not understand a thing about it, and did not >> understand any point in it except speed. Then I was forced to use git >> because of bzr-svn constant frustrations - and I ended up to really >> like it. > >> At last scipy conference, I tried to "sell" git advantages to Stefan >> (a long time bzr user as well), who was far from convinced from my >> explanations and my ranting. Then later he used it and liked it. Of >> course, the logical conclusion could be that I am just very bad at >> explaining why I like git :) > > I am pretty convinced that git is an excellent tool, but it forces people > to invest a fare amount of time to learn it. I struggle quite a lot to > have people use _any_ VCS. I just whish they'd make a usability effort. > They could. There are a lot of low hanging fruits. But they don't care. > It is geeks programming for geeks, not for normal users, IMHO. But that's a development tool. I think it is expected that people have to somewhat learn something about it. I agree we should care about barrier of entry - if there is no way to make Josef happy, for example, that would be a really strong argument against git. But at the risk of being obvious, we should also care about people who work on numpy and scipy codebase on a quasi daily basis, no ? cheers, David From josef.pktd at gmail.com Sat Apr 11 08:25:01 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 11 Apr 2009 08:25:01 -0400 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <5b8d13220904110338h1a249ddfi38d31fa82b5a2559@mail.gmail.com> References: <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> <49E00641.5050903@hawaii.edu> <5b8d13220904102005y21503770r32ac48c97c412849@mail.gmail.com> <20090411095250.GB8496@phare.normalesup.org> <5b8d13220904110338h1a249ddfi38d31fa82b5a2559@mail.gmail.com> Message-ID: <1cd32cbb0904110525q700a37bfu11ab93b3447e4426@mail.gmail.com> On Sat, Apr 11, 2009 at 6:38 AM, David Cournapeau wrote: > On Sat, Apr 11, 2009 at 6:52 PM, Gael Varoquaux > wrote: >> On Sat, Apr 11, 2009 at 12:05:55PM +0900, David Cournapeau wrote: >>> On Sat, Apr 11, 2009 at 11:53 AM, Eric Firing wrote: >>> No need to apologize, I think I used the work bashing inappropriately >>> - I just wanted to say that the only way to understand the differences >>> between the tools is to use them. Reading about them will only confuse >>> you in my own experience. For example, I tried git once a long time >>> ago (during an infamous discussion between git and bzr developers on >>> the bzr M), could not understand a thing about it, and did not >>> understand any point in it except speed. Then I was forced to use git >>> because of bzr-svn constant frustrations - and I ended up to really >>> like it. >> >>> At last scipy conference, I tried to "sell" git advantages to Stefan >>> (a long time bzr user as well), who was far from convinced from my >>> explanations and my ranting. Then later he used it and liked it. Of >>> course, the logical conclusion could be that I am just very bad at >>> explaining why I like git :) >> >> I am pretty convinced that git is an excellent tool, but it forces people >> to invest a fare amount of time to learn it. I struggle quite a lot to >> have people use _any_ VCS. I just whish they'd make a usability effort. >> They could. There are a lot of low hanging fruits. But they don't care. >> It is geeks programming for geeks, not for normal users, IMHO. > > But that's a development tool. I think it is expected that people have > to somewhat learn something about it. I agree we should care about > barrier of entry - if there is no way to make Josef happy, for > example, that would be a really strong argument against git. > > But at the risk of being obvious, we should also care about people who > work on numpy and scipy codebase on a quasi daily basis, no ? > David, if your current git svn interface/workflow works well enough then I think we could move pretty slowly with switching to a dvcs. I updated my bzr and hg, with tortoise, (hg I haven't tried yet since it requires a computer restart) the svn support in bzr has improved considerably in the last half year, now bzr-svn uses its own svn bindings instead of pysvn. I checked out the scikits and scipy svn without errors with bzr. bzr-svn doesn't get stuck anymore on the end-of-line errors. And tortoise-bzr and the gui gbzr are easy to understand and are similar to svn in the presentation. I expect the new version of hg are the same, since the hg gui where already ahead last year and tortoise-bzr is an offspring of tortoise-hg. What is also very helpful in bzr compared to the git gui, are, for example, the explanations for different types of checkouts, branches or repositories are directly build into the windows for branching/checkout. mercurials two way support still seems weaker. According to the hgsubversion webpage the code is not "production ready". But, I haven't tried it out yet. But if both git and bzr support of local svn branches works well enough, then it is not so "urgent" that we switch away from svn. hg might still require a clearer break since it appears less interoperable with the other cvs. windows and gui support for git seems to be improving pretty fast, so it might be worth the wait. But the conceptional switch from file based to patch based interaction seems to be a big jump, that might always create some dissonance for those of us who think that code is a collection of files and modules and not a history of changes. Josef From gael.varoquaux at normalesup.org Sat Apr 11 08:49:10 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 11 Apr 2009 14:49:10 +0200 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <5b8d13220904110338h1a249ddfi38d31fa82b5a2559@mail.gmail.com> References: <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> <49E00641.5050903@hawaii.edu> <5b8d13220904102005y21503770r32ac48c97c412849@mail.gmail.com> <20090411095250.GB8496@phare.normalesup.org> <5b8d13220904110338h1a249ddfi38d31fa82b5a2559@mail.gmail.com> Message-ID: <20090411124910.GF8496@phare.normalesup.org> On Sat, Apr 11, 2009 at 07:38:42PM +0900, David Cournapeau wrote: > But that's a development tool. I think it is expected that people have > to somewhat learn something about it. I agree we should care about > barrier of entry - if there is no way to make Josef happy, for > example, that would be a really strong argument against git. I remeber years ago, when I wanted to help out in projects, not understanding CVS was a big issue, and slowed my involvement by probably a year at least. Developping in Python is fairly easy, at least for simple things. We can coach someone into writing a simple pure-Python numerical algorithm for scipy, and teach him tests and coding convention. It would be a pitty if the workflow made it hard. You have spent a huge amount of time improving the build system. That's an important part. The VCS is another. The easiest it is for a grad student to download the development version of scipy, build it, and do small modifications to it, propagated upstream, the better it is for the project. And remember, most people are never taught version control. Ga?l From bsouthey at gmail.com Sat Apr 11 11:05:16 2009 From: bsouthey at gmail.com (Bruce Southey) Date: Sat, 11 Apr 2009 10:05:16 -0500 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <5b8d13220904110338h1a249ddfi38d31fa82b5a2559@mail.gmail.com> References: <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> <49E00641.5050903@hawaii.edu> <5b8d13220904102005y21503770r32ac48c97c412849@mail.gmail.com> <20090411095250.GB8496@phare.normalesup.org> <5b8d13220904110338h1a249ddfi38d31fa82b5a2559@mail.gmail.com> Message-ID: On Sat, Apr 11, 2009 at 5:38 AM, David Cournapeau wrote: > On Sat, Apr 11, 2009 at 6:52 PM, Gael Varoquaux > wrote: >> On Sat, Apr 11, 2009 at 12:05:55PM +0900, David Cournapeau wrote: >>> On Sat, Apr 11, 2009 at 11:53 AM, Eric Firing wrote: >>> No need to apologize, I think I used the work bashing inappropriately >>> - I just wanted to say that the only way to understand the differences >>> between the tools is to use them. Reading about them will only confuse >>> you in my own experience. For example, I tried git once a long time >>> ago (during an infamous discussion between git and bzr developers on >>> the bzr M), could not understand a thing about it, and did not >>> understand any point in it except speed. Then I was forced to use git >>> because of bzr-svn constant frustrations - and I ended up to really >>> like it. >> >>> At last scipy conference, I tried to "sell" git advantages to Stefan >>> (a long time bzr user as well), who was far from convinced from my >>> explanations and my ranting. Then later he used it and liked it. Of >>> course, the logical conclusion could be that I am just very bad at >>> explaining why I like git :) >> >> I am pretty convinced that git is an excellent tool, but it forces people >> to invest a fare amount of time to learn it. I struggle quite a lot to >> have people use _any_ VCS. I just whish they'd make a usability effort. >> They could. There are a lot of low hanging fruits. But they don't care. >> It is geeks programming for geeks, not for normal users, IMHO. > > But that's a development tool. I think it is expected that people have > to somewhat learn something about it. I agree we should care about > barrier of entry - if there is no way to make Josef happy, for > example, that would be a really strong argument against git. How good is the conversion between git, bzr and hg? Rather can the selected system sufficiently support the other systems? > > But at the risk of being obvious, we should also care about people who > work on numpy and scipy codebase on a quasi daily basis, no ? > There are at least two components: 1) Developers 2) Users like myself that test and use developmental versions and provide the odd error report and patch. Under the distributed approach, a developmental version does not exist. Is there going to be a way to address this or will we have to wait to release candidates to appears? Also, comments like 'get the latest version from the trunk' does become rather meaningless. Bruce From cournape at gmail.com Sat Apr 11 11:20:05 2009 From: cournape at gmail.com (David Cournapeau) Date: Sun, 12 Apr 2009 00:20:05 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: References: <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> <49E00641.5050903@hawaii.edu> <5b8d13220904102005y21503770r32ac48c97c412849@mail.gmail.com> <20090411095250.GB8496@phare.normalesup.org> <5b8d13220904110338h1a249ddfi38d31fa82b5a2559@mail.gmail.com> Message-ID: <5b8d13220904110820l2219a29bgaa6cbff643e3a650@mail.gmail.com> On Sun, Apr 12, 2009 at 12:05 AM, Bruce Southey wrote: > On Sat, Apr 11, 2009 at 5:38 AM, David Cournapeau wrote: >> On Sat, Apr 11, 2009 at 6:52 PM, Gael Varoquaux >> wrote: >>> On Sat, Apr 11, 2009 at 12:05:55PM +0900, David Cournapeau wrote: >>>> On Sat, Apr 11, 2009 at 11:53 AM, Eric Firing wrote: >>>> No need to apologize, I think I used the work bashing inappropriately >>>> - I just wanted to say that the only way to understand the differences >>>> between the tools is to use them. Reading about them will only confuse >>>> you in my own experience. For example, I tried git once a long time >>>> ago (during an infamous discussion between git and bzr developers on >>>> the bzr M), could not understand a thing about it, and did not >>>> understand any point in it except speed. Then I was forced to use git >>>> because of bzr-svn constant frustrations - and I ended up to really >>>> like it. >>> >>>> At last scipy conference, I tried to "sell" git advantages to Stefan >>>> (a long time bzr user as well), who was far from convinced from my >>>> explanations and my ranting. Then later he used it and liked it. Of >>>> course, the logical conclusion could be that I am just very bad at >>>> explaining why I like git :) >>> >>> I am pretty convinced that git is an excellent tool, but it forces people >>> to invest a fare amount of time to learn it. I struggle quite a lot to >>> have people use _any_ VCS. I just whish they'd make a usability effort. >>> They could. There are a lot of low hanging fruits. But they don't care. >>> It is geeks programming for geeks, not for normal users, IMHO. >> >> But that's a development tool. I think it is expected that people have >> to somewhat learn something about it. I agree we should care about >> barrier of entry - if there is no way to make Josef happy, for >> example, that would be a really strong argument against git. > > How good is the conversion between git, bzr and hg? git <-> bzr works well for one time import, because they share a common stream format for exchange. I don't know for hg (there is hg->git, but I don't know the other direction - I used the convert extension to try hg named branches from my git import of numpy). > Rather can the selected system sufficiently support the other systems? I think it would only lead to more complication, weird errors and confusion. I know at least gnome thought about this, and quickly gave up. > There are at least two components: > 1) Developers > 2) Users like myself that test and use developmental versions and > provide the odd error report and patch. > > Under the distributed approach, a developmental version does not > exist. It is exactly the same as before - there is a "reference" branch for main development that people would use. > Also, comments like 'get the > latest version from the trunk' does become rather meaningless. It is meaningless because there is nothing to do :) If you look at the actual git mirror http://projects.scipy.org/git/?p=numpy;a=summary The snapshot link gives you a tarball for every revision (this is not specific to git, at least hg web portal has the same capability). David From ndbecker2 at gmail.com Sat Apr 11 11:30:32 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Sat, 11 Apr 2009 11:30:32 -0400 Subject: [Numpy-discussion] documentation suggestion Message-ID: numpy.ndarray says: buffer : object exposing buffer interface, optional Used to fill the array with data. I found this misleading. It isn't just used to _fill_ the array - it is the actual storage for the array. If set to an object exporting the buffer interface, any updates to the ndarray will change the value of this object. As an example: from numpy import * import eos import mmap import os fd = os.open ('test.dat', os.O_RDWR|os.O_CREAT) size = 128 os.ftruncate (fd, size) m = mmap.mmap (fd, size, prot=mmap.PROT_READ|mmap.PROT_WRITE, flags=mmap.MAP_SHARED) u = ndarray (shape=(size,), dtype=uint8, buffer=m) u[:] = 24 mmap exports buffer interface (Also not mentioned in the documentation for mmap!). Updating the value of u (last line above) will change the file on disk. From charlesr.harris at gmail.com Sat Apr 11 11:58:53 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 11 Apr 2009 09:58:53 -0600 Subject: [Numpy-discussion] Python version dependency Message-ID: Hi All, Sturla Molden has a patch adding offset support to memmap. However... the feature requires Python 2.5 and above. The Python version is checked for in the code but I am wondering if we want to do that sort of thing. Using advanced features here and there doesn't bother me too much as long as they are marked with a grepable signature like "REMOVE_IN_FUTURE" so the code doesn't accumulate too much crud. On the other hand, maybe we should just wait until python 2.5 becomes a requirement of numpy. Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Sat Apr 11 12:45:23 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 11 Apr 2009 09:45:23 -0700 Subject: [Numpy-discussion] Python version dependency In-Reply-To: References: Message-ID: <1e2af89e0904110945h72724b5dg75bc67490a7b9090@mail.gmail.com> Hi, > Sturla Molden has a patch adding offset support to memmap. However... the > feature requires Python 2.5 and above. The Python version is checked for in > the code but I am wondering if we want to do that sort of thing. Using > advanced features here and there doesn't bother me too much as long as they > are marked with a grepable signature like "REMOVE_IN_FUTURE" so the code > doesn't accumulate too much crud. On the other hand, maybe we should just > wait until python 2.5 becomes a requirement of numpy. I think it's OK to do conditional stuff. We used to have the occasional check for set() for example. I don't think cruft is a big worry, because there's not much of it, and it's usually pretty obvious to someone maintaining the code, but a good signature comment would be useful. Matthew From nadavh at visionsense.com Sat Apr 11 14:03:28 2009 From: nadavh at visionsense.com (Nadav Horesh) Date: Sat, 11 Apr 2009 21:03:28 +0300 Subject: [Numpy-discussion] Using where with tuples References: <000001c9ba3c$66781680$33684380$@com> Message-ID: <710F2847B0018641891D9A216027636029C4C4@ex3.envision.co.il> b = a2.copy() base256 = 256**arange(4) idx = dot(a2, base256) from = dot((255,0,0,255), base256) to = dot((0,0,0,255), base256) b[idx == from] = to Nadav -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Frank Peacock ????: ? 11-?????-09 02:28 ??: numpy-discussion at scipy.org ????: [Numpy-discussion] Using where with tuples Hello I am trying to use the where function on a numpy array which was obtained from an image using the asarray function. The code is as follows: from numpy import * from Image import * im=open("a.gif") im2=im.convert("RGBA") a2 = asarray(im2,uint8) c = zeros((140,90,4),uint8) c[:,:] = [0,0,0,255] b=where(a2[:,:]==(255,0,0,255),c,a2) c=fromarray(b,"RGBA") c.save("c.gif") I am trying to convert only red colours to black but although this works it also converts other tuples as well. How do I ensure that the tuple condition is enforced exactly, i.e. only on red colours? Thanks -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3127 bytes Desc: not available URL: From ndbecker2 at gmail.com Sat Apr 11 14:44:52 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Sat, 11 Apr 2009 14:44:52 -0400 Subject: [Numpy-discussion] DVCS at PyCon References: <49CE3A2C.9000007@enthought.com> <85b5c3130904091058v55d53522i65e81e7076808636@mail.gmail.com> <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <85b5c3130904100103t64c968ebm24e53d59c0b7238e@mail.gmail.com> Message-ID: Ondrej Certik wrote: > On Fri, Apr 10, 2009 at 12:43 AM, Eric Firing wrote: >> Ondrej Certik wrote: >>> On Thu, Apr 9, 2009 at 10:45 PM, David Cournapeau >>> wrote: >>>> Ondrej Certik wrote: >>>>> It is maybe easier to learn how to work with different clones, but >>>>> once you start working with lots of patches and you need to reclone >>>>> all the time, then it's the wrong approach to work, as it takes lots >>>>> of time to copy the whole repository on the disk. >> >> This is simply wrong. Mercurial uses hard links for cloning a repo that > > On my laptop, recloning the whole repository (with hardlinks) takes > considerably longer than creating a new branch in the same directory, > that's a pure fact. You can clone a repo using: cp -al old new That should be very fast. As long as you then use an editor that behaves correctly with hard links. If you use emacs you can configure this behavior. From ondrej at certik.cz Sat Apr 11 16:02:28 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Sat, 11 Apr 2009 13:02:28 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: References: <49CE3A2C.9000007@enthought.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <85b5c3130904100103t64c968ebm24e53d59c0b7238e@mail.gmail.com> Message-ID: <85b5c3130904111302k23738a58m4710f9524f163cc7@mail.gmail.com> On Sat, Apr 11, 2009 at 11:44 AM, Neal Becker wrote: > Ondrej Certik wrote: > >> On Fri, Apr 10, 2009 at 12:43 AM, Eric Firing wrote: >>> Ondrej Certik wrote: >>>> On Thu, Apr 9, 2009 at 10:45 PM, David Cournapeau >>>> wrote: >>>>> Ondrej Certik wrote: >>>>>> It is maybe easier to learn how to work with different clones, but >>>>>> once you start working with lots of patches and you need to reclone >>>>>> all the time, then it's the wrong approach to work, as it takes lots >>>>>> of time to copy the whole repository on the disk. >>> >>> This is simply wrong. ?Mercurial uses hard links for cloning a repo that >> >> On my laptop, recloning the whole repository (with hardlinks) takes >> considerably longer than creating a new branch in the same directory, >> that's a pure fact. > > You can clone a repo using: > cp -al old new > > That should be very fast. > > As long as you then use an editor that behaves correctly with hard links. > If you use emacs you can configure this behavior. Ok, this seems to be pretty fast: $ time cp -al sympy-git.hg/ new real 0m0.129s user 0m0.020s sys 0m0.084s e.g. this was the mercurial repo. Creating a new branch in git: $ time git co -b new2 Switched to a new branch "new" real 0m0.048s user 0m0.020s sys 0m0.016s is faster, but I agree, that 0.1s is not an issue for me. Is this going to work on windows? I thought windows don't have hardlinks. In any case, I would prefer to use standard mercurial tools for the job, so if I do: $ time hg clone sympy-git.hg new updating working directory 566 files updated, 0 files merged, 0 files removed, 0 files unresolved real 0m1.156s user 0m0.948s sys 0m0.164s it still takes over a second. That's too much for me, as this operation of creating new branches is something that I do almost all the time. The way out is to use branches in on repository, and imho that's the correct way, both in git and in mercurial. However, is here anyone who actually uses branches in mercurial? If so, try this: hg clone http://hg.sympy.org/sympy-git.hg cd sympy-git.hg hg branch new2 vim README # do some changes hg ci hg up -C default hg vi and the hg vi doesn't even show your branch names and which branch contains what. let's edit README in the default branch: vim README hg ci now if you do: hg vi it shows the new2 branch, and it shows the main branch diverged, so it doesn't look as nice as in gitk, but it is possible to use. Now let's try mercurial rebase: $ hg up -C new2 $ hg rebase -d default merging README saving bundle to /tmp/ab/hg/sympy-git.hg/.hg/strip-backup/536215160a1c-temp adding branch adding changesets adding manifests adding file changes added 2 changesets with 2 changes to 1 files abort: 00changelog.i at 536215160a1c: no node! Oops, something went wrong. But commits are still there, so I guess I can safely ignore the error message (could someone clarify?). Now let's say I would like to merge the top two patches, since they both modify readme and I would like to only send one patch. In git, I just do "git rebase -i HEAD~2" tell it in vim which patches to squash and I am done. In mercurial, it's a hell of a job: $ hg qimport -r tip $ hg qimport -r qparent $ hg qpop now at: 2146.diff $ hg qfold 2147.diff $ hg qdelete -r qbase:qtip And I am still not done! I forgot to change the log (git asks you this automatically during the rebase), so we need to import the patch to MQ again: $ hg qimport -r tip $ hg qrefresh -e $ hg qdelete -r qbase:qtip And I am finally done. Now let's say some of the patches in MQ didn't apply after changing the order or some other changes. Then I am in deep troubles, because "hg qpush" fails and I need to modify the patch by hand (that really sucks!). With git, you only use rebase, and rebase is pretty powerful tool that can handle most of the conflicts itself, and if it can't, it asks you to resolve it, I assume just like mercurial rebase, but unfortunately mercurial rebase can't be used to mangle patches or history. I would like to ask mercurial proponents on this list to please correct me above and do it the right way. :) So that it's not biased. Also, read this nice article, that imho nicely compares git and mercurial: http://rg03.wordpress.com/2009/04/07/mercurial-vs-git/ Ondrej From ndbecker2 at gmail.com Sat Apr 11 17:12:17 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Sat, 11 Apr 2009 17:12:17 -0400 Subject: [Numpy-discussion] DVCS at PyCon References: <49CE3A2C.9000007@enthought.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <85b5c3130904100103t64c968ebm24e53d59c0b7238e@mail.gmail.com> <85b5c3130904111302k23738a58m4710f9524f163cc7@mail.gmail.com> Message-ID: Ondrej Certik wrote: > On Sat, Apr 11, 2009 at 11:44 AM, Neal Becker wrote: >> Ondrej Certik wrote: >> >>> On Fri, Apr 10, 2009 at 12:43 AM, Eric Firing >>> wrote: >>>> Ondrej Certik wrote: >>>>> On Thu, Apr 9, 2009 at 10:45 PM, David Cournapeau >>>>> wrote: >>>>>> Ondrej Certik wrote: >>>>>>> It is maybe easier to learn how to work with different clones, but >>>>>>> once you start working with lots of patches and you need to reclone >>>>>>> all the time, then it's the wrong approach to work, as it takes lots >>>>>>> of time to copy the whole repository on the disk. >>>> >>>> This is simply wrong. Mercurial uses hard links for cloning a repo >>>> that >>> >>> On my laptop, recloning the whole repository (with hardlinks) takes >>> considerably longer than creating a new branch in the same directory, >>> that's a pure fact. >> >> You can clone a repo using: >> cp -al old new >> >> That should be very fast. >> >> As long as you then use an editor that behaves correctly with hard links. >> If you use emacs you can configure this behavior. > > Ok, this seems to be pretty fast: > > $ time cp -al sympy-git.hg/ new > > real 0m0.129s > user 0m0.020s > sys 0m0.084s > > > e.g. this was the mercurial repo. > > Creating a new branch in git: > > $ time git co -b new2 > Switched to a new branch "new" > > real 0m0.048s > user 0m0.020s > sys 0m0.016s > > > is faster, but I agree, that 0.1s is not an issue for me. Is this > going to work on windows? I thought windows don't have hardlinks. > In any case, I would prefer to use standard mercurial tools for the > job, so if I do: > > $ time hg clone sympy-git.hg new > updating working directory > 566 files updated, 0 files merged, 0 files removed, 0 files unresolved > > real 0m1.156s > user 0m0.948s > sys 0m0.164s > > > it still takes over a second. That's too much for me, as this > operation of creating new branches is something that I do almost all > the time. > > The way out is to use branches in on repository, and imho that's the > correct way, both in git and in mercurial. > > However, is here anyone who actually uses branches in mercurial? If > so, try this: > > hg clone http://hg.sympy.org/sympy-git.hg > cd sympy-git.hg > hg branch new2 > vim README # do some changes > hg ci > hg up -C default > hg vi > > and the hg vi doesn't even show your branch names and which branch > contains what. > > let's edit README in the default branch: > > vim README > hg ci > > now if you do: > > hg vi > > it shows the new2 branch, and it shows the main branch diverged, so it > doesn't look as nice as in gitk, but it is possible to use. Now let's > try mercurial rebase: > > $ hg up -C new2 > $ hg rebase -d default > merging README > saving bundle to > /tmp/ab/hg/sympy-git.hg/.hg/strip-backup/536215160a1c-temp adding branch > adding changesets > adding manifests > adding file changes > added 2 changesets with 2 changes to 1 files > abort: 00changelog.i at 536215160a1c: no node! > > > Oops, something went wrong. But commits are still there, so I guess I > can safely ignore the error message (could someone clarify?). > > Now let's say I would like to merge the top two patches, since they > both modify readme and I would like to only send one patch. In git, I > just do "git rebase -i HEAD~2" tell it in vim which patches to squash > and I am done. In mercurial, it's a hell of a job: > > $ hg qimport -r tip > $ hg qimport -r qparent > $ hg qpop > now at: 2146.diff > $ hg qfold 2147.diff > $ hg qdelete -r qbase:qtip > > And I am still not done! I forgot to change the log (git asks you this > automatically during the rebase), so we need to import the patch to MQ > again: > > $ hg qimport -r tip > $ hg qrefresh -e > $ hg qdelete -r qbase:qtip > > > And I am finally done. Now let's say some of the patches in MQ didn't > apply after changing the order or some other changes. Then I am in > deep troubles, because "hg qpush" fails and I need to modify the patch > by hand (that really sucks!). With git, you only use rebase, and > rebase is pretty powerful tool that can handle most of the conflicts > itself, and if it can't, it asks you to resolve it, I assume just like > mercurial rebase, but unfortunately mercurial rebase can't be used to > mangle patches or history. > > I would like to ask mercurial proponents on this list to please > correct me above and do it the right way. :) So that it's not biased. > > Also, read this nice article, that imho nicely compares git and mercurial: > > http://rg03.wordpress.com/2009/04/07/mercurial-vs-git/ > Why not try asking on mercurial at selenic.com (gmane.comp.version- control.mercurial.general) From ondrej at certik.cz Sat Apr 11 19:29:45 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Sat, 11 Apr 2009 16:29:45 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: References: <49CE3A2C.9000007@enthought.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <85b5c3130904100103t64c968ebm24e53d59c0b7238e@mail.gmail.com> <85b5c3130904111302k23738a58m4710f9524f163cc7@mail.gmail.com> Message-ID: <85b5c3130904111629v77c35945n87ee3a8d30537a06@mail.gmail.com> On Sat, Apr 11, 2009 at 2:12 PM, Neal Becker wrote: > > Why not try asking on mercurial at selenic.com (gmane.comp.version- > control.mercurial.general) Done: http://www.selenic.com/pipermail/mercurial/2009-April/025131.html O. From stefan at sun.ac.za Sat Apr 11 21:57:26 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 12 Apr 2009 03:57:26 +0200 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <20090411095250.GB8496@phare.normalesup.org> References: <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> <49E00641.5050903@hawaii.edu> <5b8d13220904102005y21503770r32ac48c97c412849@mail.gmail.com> <20090411095250.GB8496@phare.normalesup.org> Message-ID: <9457e7c80904111857x28cf93edm14f0e8d2c924410b@mail.gmail.com> Hi all, DVCS discussions are notoriously unproductive, aren't they :) The two systems under discussion, git and hg, are both *almost* what we want, which means we have to make a choice and compromise. Here are some of the pros and cons I've picked up, and I'm sure you can add more: Git has well-integrated in-place branches, allows you to easily do "patch carving" (i.e. manipulation of patches to get them Just Right (TM)) and is very fast. Mercurial has a more intuitive interface / less confusing error messages in places, has a very good http serving protocol, is written in and used by Python. Bzr wasn't included in the discussion, but its patience diff algorithm and good user interface is worth mentioning. We have to decide which of these attributes are more important to the developers and users of NumPy and SciPy, and then go with that system. Alternatively, we can simply ignore the arguments and pick one randomly. Having any DVCS in place would improve the situation and, besides, it's easy to switch to another system later if we make an enormous error in judgement. Regards St?fan From charlesr.harris at gmail.com Sat Apr 11 22:19:39 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 11 Apr 2009 20:19:39 -0600 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <9457e7c80904111857x28cf93edm14f0e8d2c924410b@mail.gmail.com> References: <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> <49E00641.5050903@hawaii.edu> <5b8d13220904102005y21503770r32ac48c97c412849@mail.gmail.com> <20090411095250.GB8496@phare.normalesup.org> <9457e7c80904111857x28cf93edm14f0e8d2c924410b@mail.gmail.com> Message-ID: 2009/4/11 St?fan van der Walt > Hi all, > > DVCS discussions are notoriously unproductive, aren't they :) > > The two systems under discussion, git and hg, are both *almost* what > we want, which means we have to make a choice and compromise. > > > Here are some of the pros and cons I've picked up, and I'm sure you > can add more: > > Git has well-integrated in-place branches, allows you to easily do > "patch carving" (i.e. manipulation of patches to get them Just Right > (TM)) and is very fast. Mercurial has a more intuitive interface / > less confusing error messages in places, has a very good http serving > protocol, is written in and used by Python. Bzr wasn't included in > the discussion, but its patience diff algorithm and good user > interface is worth mentioning. > > > We have to decide which of these attributes are more important to the > developers and users of NumPy and SciPy, and then go with that system. > > Alternatively, we can simply ignore the arguments and pick one > randomly. Having any DVCS in place would improve the situation and, > besides, it's easy to switch to another system later if we make an > enormous error in judgement. > I vote that we don't even think about a decision until next year. The current system works well enough, I think. Meanwhile, the endless discussions are fun and pass the time. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Sat Apr 11 22:36:49 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 12 Apr 2009 04:36:49 +0200 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: References: <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> <49E00641.5050903@hawaii.edu> <5b8d13220904102005y21503770r32ac48c97c412849@mail.gmail.com> <20090411095250.GB8496@phare.normalesup.org> <9457e7c80904111857x28cf93edm14f0e8d2c924410b@mail.gmail.com> Message-ID: <9457e7c80904111936o7b4d37dcuc0342c35238cf748@mail.gmail.com> 2009/4/12 Charles R Harris : > I vote that we don't even think about a decision until next year. The > current system works well enough, I think. Meanwhile, the endless > discussions are fun and pass the time. I'll take that with a pinch of salt, as I assume it was intended :) >From my POV, the current system is very unproductive and, while git-svn makes life a bit easier, it comes with its own set of headaches. Especially now that we are evaluating different work-flows, we need the right kind of vcs to back it up. Cheers St?fan From stefan at sun.ac.za Sat Apr 11 22:43:54 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 12 Apr 2009 04:43:54 +0200 Subject: [Numpy-discussion] documentation suggestion In-Reply-To: References: Message-ID: <9457e7c80904111943hddd3b20h6cc49266fd3d31e0@mail.gmail.com> Hi Neal 2009/4/11 Neal Becker : > I found this misleading. ?It isn't just used to _fill_ the array - it is the Please create an account on docs.scipy.org, then I'll give you priviledges to edit documentation. Thanks! St?fan From stefan at sun.ac.za Sat Apr 11 23:16:14 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 12 Apr 2009 05:16:14 +0200 Subject: [Numpy-discussion] numpy.ma.arg{min,max} In-Reply-To: References: <49DFC526.4090602@noaa.gov> Message-ID: <9457e7c80904112016v786adf43p68e75a4553fa78e@mail.gmail.com> Hi Adam 2009/4/11 Adam Oliner : > I see, thanks. Then the documentation should say that it returns an > array of the _smallest_ indices of the minimum values along the given > axis. Please sign up on http://docs.scipy.org, then I'll give you acess to the documentation editor. Regards St?fan From fperez.net at gmail.com Sat Apr 11 23:28:53 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 11 Apr 2009 20:28:53 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <9457e7c80904111936o7b4d37dcuc0342c35238cf748@mail.gmail.com> References: <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <1e2af89e0904101447x42f06757lc2db42908a218b0@mail.gmail.com> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> <49E00641.5050903@hawaii.edu> <5b8d13220904102005y21503770r32ac48c97c412849@mail.gmail.com> <20090411095250.GB8496@phare.normalesup.org> <9457e7c80904111857x28cf93edm14f0e8d2c924410b@mail.gmail.com> <9457e7c80904111936o7b4d37dcuc0342c35238cf748@mail.gmail.com> Message-ID: 2009/4/11 St?fan van der Walt : > From my POV, the current system is very unproductive and, while > git-svn makes life a bit easier, it comes with its own set of > headaches. ?Especially now that we are evaluating different > work-flows, we need the right kind of vcs to back it up. Please take the following with a big grain of salt, because I haven't used either git nor hg beyond trivial cloning. But I do have by now pretty extensive experience with bzr (ipython and nipy), so I think I've developed a decent intuition on what's important/useful in the long run from DVCS workflows. I have also been reading a fair amount about git and hg, in particular about the core of their internal models. This, it seems to me, is actually far more important in the long run than their top-layer polish. >From that perspective, right now my intuition (and yes, it's only that for now) tells me that git has some really appealing features for a dvcs, which as far as I can see are not present in hg (seeing as unless I've grossly misunderstood it, it is much closer to bzr in its internal model than to git). To me the key point in git of fundamental value is its direct manipulation of the commit DAG and history: this is something that I think one only comes to appreciate after using a DVCS for *a reasonably long time* on a *reasonably complex project* with multiple developers, branches and merges. I stress this because I think these points really only become apparent under such conditions, at least I didn't really think of these things until I used bzr extensively for ipython. Let me elaborate a bit. One of the main benefits of a DVCS is that it enables all developers to be aggressive locally, to experiment on crazy ideas and to use the VCS as their safety line in their experimentation. You are free to try crazy things, commit as often and finely-grained as you want, and if things go wrong, you can backtrack easily. But in general what happens is that things don't simply go wrong: you often end up making things work, it's just that the intermediate history can look totally crazy, with tons of intermediate commits that are really of no interest anymore to anyone. With git, there is a way of saying "merge all this into a single commit (or a few)" so that it goes into the upstream project into chunks that make logical sense and not just that reflect your tiptoeing during a tricky part of the development. In bzr (and as far as I see, also in hg), this kind of history rewriting is near impossible, so the best you can do is make a merge commit and leave all that history in there, visible in the 'second level' log (indented in the text view). As a project grows many developers, having all this history merged back into the main project tree gets unwieldy. >From my (now reasonably extensive) experience with bzr, it really feels like a system that took the centralized VCS model and put 'a little svn server everywhere you need one'. That is, the repository/branch/history model retains the rigidity of a centralized VCS, it's just that you can have it anywhere, and it can track branching and merging intelligently. There's certainly a lot of value in that, I am not denying it in the least bit. However, git seems to really make the key conceptual jump of saying: once you have a truly distributed development process, that rigid model just breaks down and should be abandoned. What you need to accept is that the core objects you should manipulate are the atomic change units needed to reconstruct the state of the project, and the connectivity between those units. If you have tools to manipulate said entities, you'll be able to really integrate the work that many people may be doing on the same objects in disconnected ways, back into a single coherent entity. Sorry if this seems a bit in the air, but I've been thinking about this for the past couple of days, and I figured I'd share. I don't mean this to be a bashing of hg or bzr (which we'll continue using for ipython at least for a long while, since now is not the time for yet another workflow change for us). But from *my* perspective, git offers really the correct abstractions to think about distributed collaborative workflows, while the other systems simply seem to offer tools to distribute the workflow of a rigid development history (a la CVS) to multiple developers. There's a fundamental difference between thosee two approaches, and I think it's a critically important one. As for what numpy/scipy should do, I'll leave that to those who actually contribute to the projects :) I just hope that this view is useful to anyone who wants to think about the problem from an angle different to that of specific commands, benchmarks or features :) All the best, f From the.minjae at gmail.com Sat Apr 11 23:54:52 2009 From: the.minjae at gmail.com (Minjae Kim) Date: Sat, 11 Apr 2009 23:54:52 -0400 Subject: [Numpy-discussion] Preserving trailing zeros when printing an array Message-ID: Hello, When I work with arrays, I would like to print trailing zeros also. Somehow, Numpy refuses to print trailing zeros of a number within an array. For example, if I set the print precision to 5 decimal digits (set_printoptions(5)) and try to print array([0.10000001], it gives me 0.1 instead of 0.10000 . I would like to see all trailing zeros when I print it on the shell (up to a predefined number of digits). (This is confirmed with Numpy functions like array2string and array_str) Any help is welcome. Best, Minjae -------------- next part -------------- An HTML attachment was scrubbed... URL: From ondrej at certik.cz Sat Apr 11 23:58:51 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Sat, 11 Apr 2009 20:58:51 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: References: <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> <49E00641.5050903@hawaii.edu> <5b8d13220904102005y21503770r32ac48c97c412849@mail.gmail.com> <20090411095250.GB8496@phare.normalesup.org> <9457e7c80904111857x28cf93edm14f0e8d2c924410b@mail.gmail.com> <9457e7c80904111936o7b4d37dcuc0342c35238cf748@mail.gmail.com> Message-ID: <85b5c3130904112058i21856490t96da38e2d9322c6d@mail.gmail.com> On Sat, Apr 11, 2009 at 8:28 PM, Fernando Perez wrote: > In bzr (and as far as I see, also in hg), ?this kind of history > rewriting is near impossible, so the best you can do is make a merge > commit and leave all that history in there, visible in the 'second > level' log (indented in the text view). ?As a project grows many > developers, having all this history merged back into the main project > tree gets unwieldy. Great email. Exactly, that's my experience with hg, it makes it nearly impossible, without misusing mercurial queues. I am just surprised, you have exactly the same experience with bzr, I thought it's only hg. As an amusing story, my roommate here at UNR also switched from svn to hg and was telling me, you know, git is too unfriendly, hg does everything I ever need. I was telling myself, well, after you really start using it, you'll maybe switch to git. Then after couple months he once told me -- "I had to rewrite history, so I tried MQ and that's a disaster, the interface to it is horrible. So I switched to git." I was thinking about Fernando just yesterday, asking myself, hm, I wonder how he is satisfied with bzr. So this email really made me laugh. :) > As for what numpy/scipy should do, I'll leave that to those who > actually contribute to the projects :) ?I just hope that this view is Exactly, this decision should be done by numpy developers, not me either. > useful to anyone who wants to think about the problem from an angle > different to that of specific commands, benchmarks or features :) Ondrej From stefan at sun.ac.za Sun Apr 12 04:54:37 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 12 Apr 2009 10:54:37 +0200 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: References: <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <49DFF980.8040407@hawaii.edu> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> <49E00641.5050903@hawaii.edu> <5b8d13220904102005y21503770r32ac48c97c412849@mail.gmail.com> <20090411095250.GB8496@phare.normalesup.org> <9457e7c80904111857x28cf93edm14f0e8d2c924410b@mail.gmail.com> <9457e7c80904111936o7b4d37dcuc0342c35238cf748@mail.gmail.com> Message-ID: <9457e7c80904120154x7973a5f4v993f0d3d0419fa7f@mail.gmail.com> 2009/4/12 Fernando Perez : > What you need to > accept is that the core objects you should manipulate are the atomic > change units needed to reconstruct the state of the project, and the > connectivity between those units. ?If you have tools to manipulate > said entities, ?you'll be able to really integrate the work that many > people may be doing on the same objects in disconnected ways, back > into a single coherent entity. Your paragraph above very neatly sums up the conclusion I have been converging towards during the last couple of months. Back at Sage Days 8, when we discussed the different systems, I underestimated the value of this type of manipulation, and of having a clearly structured and easily traversable history. Of course, back then git wasn't even an option, being somewhat crude, but the playing field has levelled since. With git, I have discovered the joys of "patch carving" ("patch manipulation"), and contributing well polished patches to any project, including those you don't have repository access to, becomes painless. Thanks for taking the time to write. It's great to hear from more experienced DVCS users! Regards St?fan From cournape at gmail.com Sun Apr 12 09:38:17 2009 From: cournape at gmail.com (David Cournapeau) Date: Sun, 12 Apr 2009 22:38:17 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <85b5c3130904112058i21856490t96da38e2d9322c6d@mail.gmail.com> References: <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> <49E00641.5050903@hawaii.edu> <5b8d13220904102005y21503770r32ac48c97c412849@mail.gmail.com> <20090411095250.GB8496@phare.normalesup.org> <9457e7c80904111857x28cf93edm14f0e8d2c924410b@mail.gmail.com> <9457e7c80904111936o7b4d37dcuc0342c35238cf748@mail.gmail.com> <85b5c3130904112058i21856490t96da38e2d9322c6d@mail.gmail.com> Message-ID: <5b8d13220904120638w10a5c970w7eba468293326681@mail.gmail.com> On Sun, Apr 12, 2009 at 12:58 PM, Ondrej Certik wrote: > > I am just surprised, you have exactly the same experience with bzr, I > thought it's only hg. I agree that simplicity is a bit overrated. Sure, bzr is simpler than git - at the beginning. But once you start thinking about shared repository, stacked branches, checkout, looms (bzr equivalent of mercurial queues), etc... is it still so simple ? Git is more "in your face", but you don't need much more than that after. I don't think I use more than 10-12 commands or so. David From general.mooney at googlemail.com Sun Apr 12 09:59:48 2009 From: general.mooney at googlemail.com (=?ISO-8859-1?Q?Ciar=E1n_Mooney?=) Date: Sun, 12 Apr 2009 14:59:48 +0100 Subject: [Numpy-discussion] [Annoucement] Europython 2009 Message-ID: <3e4e51a80904120659o520851fbp4394465c10a7ab57@mail.gmail.com> Hi, There has been a inkling of interest in a Scipy/Numpy tutorial on the Europython 2009 lists. But we need someone to lead it! So even though the deadline for papers has passed, they may still be open to an offer. If not, please come along anyway to mingle, meet and network with other python-ista's. The following is from the Europython Commitee. ====== On behalf of the EuroPython 2009 organisation it is my privilege and honour to announce the 'Call for Participation' for EuroPython 2009! EuroPython is the conference for the communities around Python, including the Django, Zope and Plone communities. This years conference will be held in Birmingham, UK from Monday 29th June to Saturday 4th July 2009. Talk & Themes Do you have something you wish to present at EuroPython? Go to http://www.europython.eu/talks/cfp/ for this years themes and submissions criteria, the deadline is on 5th April 2009. Other Talks, Activities and Events Have you got something which does not fit the above? Visit http://www.europython.eu/talks/ Help Us Out We could use a hand - any contribution is welcome. Please take a look at http://www.europython.eu/contact/ . Sponsors An unique opportunity to affiliate with the prestigious EuroPython conference! http://www.europython.eu/sponsors/ Spread the Word Improve our publicity by distributing this announcement in your corner of the community, please coordinate this with the organizers: http://www.europython.eu/contact/ General Information For more information about the conference, please visit http://www.europython.eu/ Looking forward to seeing you! The EuroPython Team =========== Regards, Ciar?n From pav at iki.fi Sun Apr 12 11:07:50 2009 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 12 Apr 2009 15:07:50 +0000 (UTC) Subject: [Numpy-discussion] DVCS at PyCon References: <49CE3A2C.9000007@enthought.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <5b8d13220904101956v15cc4184r7c8144a519c369ea@mail.gmail.com> <49E0219C.1050308@hawaii.edu> Message-ID: Fri, 10 Apr 2009 18:50:36 -1000, Eric Firing wrote: [clip] >> Ah, I think I was confused between named branches and bookmarks. From >> the description of bookmarks, this actually looks nearer to the git >> cheap branch concept. I should really try it to get a good >> understanding, though. > > I would not be surprised if bookmarks were directly inspired by that. > There is also a localbranch extension, but it is not included in the > distribution and hasn't gotten much attention. I have the sense that > bookmarks were designed to accomplish the same thing. Hg bookmarks cannot be easily shared remotely, similarly to Git's branches. This is okay for topic branches, but is a drawback if you want to share multiple lines of work with other people. Also, based on a quick try, hg rebase + hg bookmarks don't play completely well together, as rebasing tends to move also the "trunk" bookmark around. -- Pauli Virtanen From sedwards2 at cinci.rr.com Sun Apr 12 12:19:13 2009 From: sedwards2 at cinci.rr.com (Stuart Edwards) Date: Sun, 12 Apr 2009 12:19:13 -0400 Subject: [Numpy-discussion] Leopard install Message-ID: <0EAFD24D-078E-4EAD-AEF0-BC5B55789CBD@cinci.rr.com> Hi I am trying to install numpy 1.3.0 on Leopard 10.5.6 and at the point in the install process where I select a destination, my boot disc is excluded with the message: " You cannot install numpy 1.3.0 on this volume. numpy requires System Python 2.5 to install." I'm not sure what 'System Python 2.5' is as compared to 'Python 2.5' but in the terminal when I type 'python' I get: "Python 2.5.1 (r251:54863, Jan 13 2009, 10:26:13) [GCC 4.0.1 (Apple Inc. build 5465)] on darwin" so the Python 2.5 requirement seems to be met. Any ideas on what is happening here and why the installer can't see my python installation? (I notice that this issue was raised 3/28 also, but no resolution yet) Thanks for any assistance Stu From ctw at cogsci.info Sun Apr 12 13:43:03 2009 From: ctw at cogsci.info (Christoph T. Weidemann) Date: Sun, 12 Apr 2009 13:43:03 -0400 Subject: [Numpy-discussion] converting from scipy.io.fread Message-ID: I noticed that scipy.io.fread is deprecated and wanted to change some code I got from somebody else accordingly. This code contains statements like: fread(fid, 1, 'h') fread(fid, 1, 'l') fread(fid, 1, 'd') fread(fid, 6, 'B') and variations of those with different number and letters. The docstring suggested that numpy.fromfile may be a good alternative, but I'm having some trouble mapping the fread commands to appropriate fromfile commands. Particularly, I could not figure out how to map the read_type values (which the fread docstring suggests are PyArray types -- 'h', however, is not listed as a possible value in the docstring) to the dtypes in fromfile and how to translate the fread Num variable into the counts required for fromfile. I would appreciate any help converting these fread commands to non-deprecated commands (using fromfile or any other method that may be more appropriate). Unfortunately I know very little about how exactly the data is stored in the binary file, so it would be great if the conversion could make do with the information from the call to fread. From nwagner at iam.uni-stuttgart.de Sun Apr 12 14:17:18 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Sun, 12 Apr 2009 20:17:18 +0200 Subject: [Numpy-discussion] numpy.test() errors r6862 Message-ID: ====================================================================== ERROR: test suite ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/suite.py", line 154, in run self.setUp() File "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/suite.py", line 180, in setUp if not self: File "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/suite.py", line 65, in __nonzero__ test = self.test_generator.next() File "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/loader.py", line 221, in generate for test in g(): File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_format.py", line 440, in test_memmap_roundtrip shape=arr.shape, fortran_order=fortran_order) File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/format.py", line 484, in open_memmap mode=mode, offset=offset) File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/core/memmap.py", line 231, in __new__ mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=offset) error: [Errno 22] Invalid argument ====================================================================== ERROR: test_mmap (test_io.TestSaveLoad) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/decorators.py", line 169, in knownfailer return f(*args, **kwargs) File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_io.py", line 92, in test_mmap self.roundtrip(a, file_on_disk=True, load_kwds={'mmap_mode': 'r'}) File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_io.py", line 100, in roundtrip RoundtripTest.roundtrip(self, np.save, *args, **kwargs) File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_io.py", line 67, in roundtrip arr_reloaded = np.load(load_file, **load_kwds) File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/io.py", line 193, in load return format.open_memmap(file, mode=mmap_mode) File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/format.py", line 484, in open_memmap mode=mode, offset=offset) File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/core/memmap.py", line 231, in __new__ mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=offset) error: [Errno 22] Invalid argument ---------------------------------------------------------------------- Ran 1889 tests in 12.656s FAILED (KNOWNFAIL=1, errors=2) From charlesr.harris at gmail.com Sun Apr 12 14:52:13 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 12 Apr 2009 12:52:13 -0600 Subject: [Numpy-discussion] numpy.test() errors r6862 In-Reply-To: References: Message-ID: On Sun, Apr 12, 2009 at 12:17 PM, Nils Wagner wrote: > ====================================================================== > ERROR: test suite > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > > "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/suite.py", > line 154, in run > self.setUp() > File > > "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/suite.py", > line 180, in setUp > if not self: > File > > "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/suite.py", > line 65, in __nonzero__ > test = self.test_generator.next() > File > > "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/loader.py", > line 221, in generate > for test in g(): > File > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_format.py", > line 440, in test_memmap_roundtrip > shape=arr.shape, fortran_order=fortran_order) > File > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/format.py", > line 484, in open_memmap > mode=mode, offset=offset) > File > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/core/memmap.py", > line 231, in __new__ > mm = mmap.mmap(fid.fileno(), bytes, access=acc, > offset=offset) > error: [Errno 22] Invalid argument > > ====================================================================== > ERROR: test_mmap (test_io.TestSaveLoad) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/decorators.py", > line 169, in knownfailer > return f(*args, **kwargs) > File > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_io.py", > line 92, in test_mmap > self.roundtrip(a, file_on_disk=True, > load_kwds={'mmap_mode': 'r'}) > File > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_io.py", > line 100, in roundtrip > RoundtripTest.roundtrip(self, np.save, *args, > **kwargs) > File > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_io.py", > line 67, in roundtrip > arr_reloaded = np.load(load_file, **load_kwds) > File > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/io.py", > line 193, in load > return format.open_memmap(file, mode=mmap_mode) > File > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/format.py", > line 484, in open_memmap > mode=mode, offset=offset) > File > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/core/memmap.py", > line 231, in __new__ > mm = mmap.mmap(fid.fileno(), bytes, access=acc, > offset=offset) > error: [Errno 22] Invalid argument > > ---------------------------------------------------------------------- > Ran 1889 tests in 12.656s > > FAILED (KNOWNFAIL=1, errors=2) > Hmm, I'll guess that the problem is this: *offset* must be a multiple of the ALLOCATIONGRANULARITY. This conflicts with the current intent of offset. Looks like we need to fix up the patch to use the nearest multiple of ALLOCATIONGRANULARITY and then offset the usual way. Or, since ALLOCATIONGRANULARITY is likely to be platform dependent, maybe we should just revert the patch. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Apr 12 14:56:46 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 12 Apr 2009 12:56:46 -0600 Subject: [Numpy-discussion] numpy.test() errors r6862 In-Reply-To: References: Message-ID: On Sun, Apr 12, 2009 at 12:52 PM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Sun, Apr 12, 2009 at 12:17 PM, Nils Wagner < > nwagner at iam.uni-stuttgart.de> wrote: > >> ====================================================================== >> ERROR: test suite >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File >> >> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/suite.py", >> line 154, in run >> self.setUp() >> File >> >> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/suite.py", >> line 180, in setUp >> if not self: >> File >> >> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/suite.py", >> line 65, in __nonzero__ >> test = self.test_generator.next() >> File >> >> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/loader.py", >> line 221, in generate >> for test in g(): >> File >> >> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_format.py", >> line 440, in test_memmap_roundtrip >> shape=arr.shape, fortran_order=fortran_order) >> File >> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/format.py", >> line 484, in open_memmap >> mode=mode, offset=offset) >> File >> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/core/memmap.py", >> line 231, in __new__ >> mm = mmap.mmap(fid.fileno(), bytes, access=acc, >> offset=offset) >> error: [Errno 22] Invalid argument >> >> ====================================================================== >> ERROR: test_mmap (test_io.TestSaveLoad) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File >> >> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/decorators.py", >> line 169, in knownfailer >> return f(*args, **kwargs) >> File >> >> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_io.py", >> line 92, in test_mmap >> self.roundtrip(a, file_on_disk=True, >> load_kwds={'mmap_mode': 'r'}) >> File >> >> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_io.py", >> line 100, in roundtrip >> RoundtripTest.roundtrip(self, np.save, *args, >> **kwargs) >> File >> >> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_io.py", >> line 67, in roundtrip >> arr_reloaded = np.load(load_file, **load_kwds) >> File >> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/io.py", >> line 193, in load >> return format.open_memmap(file, mode=mmap_mode) >> File >> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/format.py", >> line 484, in open_memmap >> mode=mode, offset=offset) >> File >> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/core/memmap.py", >> line 231, in __new__ >> mm = mmap.mmap(fid.fileno(), bytes, access=acc, >> offset=offset) >> error: [Errno 22] Invalid argument >> >> ---------------------------------------------------------------------- >> Ran 1889 tests in 12.656s >> >> FAILED (KNOWNFAIL=1, errors=2) >> > > Hmm, I'll guess that the problem is this: > > *offset* must be a multiple of the ALLOCATIONGRANULARITY. > > This conflicts with the current intent of offset. Looks like we need to fix > up the patch to use the nearest multiple of ALLOCATIONGRANULARITY and then > offset the usual way. Or, since ALLOCATIONGRANULARITY is likely to be > platform dependent, maybe we should just revert the patch. > Can you import mmap, then do a dir(mmap) and see if ALLOCATIONGRANULARITY is available? TIA, Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Sun Apr 12 17:32:31 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Sun, 12 Apr 2009 17:32:31 -0400 Subject: [Numpy-discussion] documentation suggestion References: <9457e7c80904111943hddd3b20h6cc49266fd3d31e0@mail.gmail.com> Message-ID: St?fan van der Walt wrote: > Hi Neal > > 2009/4/11 Neal Becker : >> I found this misleading. It isn't just used to _fill_ the array - it is >> the > > Please create an account on docs.scipy.org, then I'll give you > priviledges to edit documentation. > > Thanks! > St?fan Done, but can someone check that what I wrote is accurate? I wrote that changes to the ndarray will change the underlying buffer object. But, the buffer protocol allows for read-only buffers. Not sure what ndarray would do if you tried to write in that case. From mg at lazybytes.net Sun Apr 12 18:10:40 2009 From: mg at lazybytes.net (Martin Geisler) Date: Mon, 13 Apr 2009 00:10:40 +0200 Subject: [Numpy-discussion] DVCS at PyCon References: <49CE3A2C.9000007@enthought.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <5b8d13220904101956v15cc4184r7c8144a519c369ea@mail.gmail.com> Message-ID: <87bpr11q1r.fsf@hbox.dyndns.org> David Cournapeau writes: > Ok, will look at it more closely (I have just made a hg repo from the > numpy git one on my machine). I am still confused about how named > branches are supposed to work (the hg book is a a bit light on this). > For example, let's say I want to create a branch from a given > revision/tag, how should I do it ? You update to the revision, edit and make a commit. Please see below. > Since hg branch does not take an -r revision, I tried to create the > branch first, and then use hg up -r N, but it did not commit as I > expected from a git perspective: > > hg branch foo > hg up -r 500 > hg log # show me last revision independently from the branch changes. > Not so surprising given that the branch was created from the tip of > default. > hg ci -m "bla" > hg branches # no branch foo ? > > What am I doing wrong ? As we pointed out to Ondrej Certik on the Mercurial list, 'hg branch' is very different from 'git branch'. In Mercurial it creates a *named branch*, which are imbedded in the history and are used for long-term development branches. Names of Git branches are not recorded in the history and are thus well-suited for local experiments. In Mercurial you can create an unnamed branch from any point in the changeset graph by updating to that point and making a commit: hg update 123 # edit files hg commit -m 'My change.' If you make a commit from a changeset which already has other child changesets, then you've created an unnamed (anonymous) branch. You can see this with 'hg heads' -- it shows you the changesets without children, and having more than one of these means that there are some unmerged branches. The bookmarks extension let you assign names to the heads, it is inspired by the way Git branches work. Unfortunately the bookmarks are not yet transferred over the network when you do push/pull, but this is of course a planned feature. If you want to let others know about a branch you're working on, you can always put up a separate clone with that branch -- then just delete the clone if you want to throw away the branch. If you have multiple heads in a repository, it is easy to separate them into separate clones: hg clone -r some-rev repo repo-with-some-rev This makes repo-with-some-rev contain some-rev and all ancestor revisions -- but no other heads. You can merge such splitted clones again by pulling in changesets from another clone. -- Martin Geisler VIFF (Virtual Ideal Functionality Framework) brings easy and efficient SMPC (Secure Multiparty Computation) to Python. See: http://viff.dk/. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available URL: From cournape at gmail.com Sun Apr 12 19:02:11 2009 From: cournape at gmail.com (David Cournapeau) Date: Mon, 13 Apr 2009 08:02:11 +0900 Subject: [Numpy-discussion] Leopard install In-Reply-To: <0EAFD24D-078E-4EAD-AEF0-BC5B55789CBD@cinci.rr.com> References: <0EAFD24D-078E-4EAD-AEF0-BC5B55789CBD@cinci.rr.com> Message-ID: <5b8d13220904121602u265a2418j18541fe7479aabde@mail.gmail.com> On Mon, Apr 13, 2009 at 1:19 AM, Stuart Edwards wrote: > Hi > > I am trying to install numpy 1.3.0 on Leopard 10.5.6 and at the point > in the install process where I select a destination, my boot disc is > excluded with the message: > I think you need to install python from python.org (version 2.5) to install the numpy binary, David From stefan at sun.ac.za Sun Apr 12 19:11:47 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 13 Apr 2009 01:11:47 +0200 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <87bpr11q1r.fsf@hbox.dyndns.org> References: <49CE3A2C.9000007@enthought.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <5b8d13220904101956v15cc4184r7c8144a519c369ea@mail.gmail.com> <87bpr11q1r.fsf@hbox.dyndns.org> Message-ID: <9457e7c80904121611u44d060f7i62eec5d35a637ea3@mail.gmail.com> Hi Martin 2009/4/13 Martin Geisler : > In Mercurial it creates a *named > branch*, which are imbedded in the history and are used for long-term > development branches. Names of Git branches are not recorded in the > history and are thus well-suited for local experiments. [...] > The bookmarks extension let you assign names to the heads, it is > inspired by the way Git branches work. Unfortunately the bookmarks are > not yet transferred over the network when you do push/pull, but this is > of course a planned feature. Thanks for the info. Maybe you can also help me out with the following questions: I'd like to know how to duplicate git's branch workflow in Mercurial. - How do you create a new head when you are working on top of the current head? Do you have to make a dummy commit and then commit on top of (TIP - 1) whenever you need a new "branch"? - Is it possible to remove a head without merging it, discarding the children? Thanks, St?fan From charlesr.harris at gmail.com Sun Apr 12 19:34:13 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 12 Apr 2009 17:34:13 -0600 Subject: [Numpy-discussion] numpy.test() errors r6862 In-Reply-To: References: Message-ID: Hi Nils, On Sun, Apr 12, 2009 at 12:17 PM, Nils Wagner wrote: > ====================================================================== > ERROR: test suite > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > > "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/suite.py", > line 154, in run > self.setUp() > File > > "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/suite.py", > line 180, in setUp > if not self: > File > > "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/suite.py", > line 65, in __nonzero__ > test = self.test_generator.next() > File > > "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/loader.py", > line 221, in generate > for test in g(): > File > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_format.py", > line 440, in test_memmap_roundtrip > shape=arr.shape, fortran_order=fortran_order) > File > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/format.py", > line 484, in open_memmap > mode=mode, offset=offset) > File > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/core/memmap.py", > line 231, in __new__ > mm = mmap.mmap(fid.fileno(), bytes, access=acc, > offset=offset) > error: [Errno 22] Invalid argument > > ====================================================================== > ERROR: test_mmap (test_io.TestSaveLoad) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/decorators.py", > line 169, in knownfailer > return f(*args, **kwargs) > File > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_io.py", > line 92, in test_mmap > self.roundtrip(a, file_on_disk=True, > load_kwds={'mmap_mode': 'r'}) > File > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_io.py", > line 100, in roundtrip > RoundtripTest.roundtrip(self, np.save, *args, > **kwargs) > File > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_io.py", > line 67, in roundtrip > arr_reloaded = np.load(load_file, **load_kwds) > File > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/io.py", > line 193, in load > return format.open_memmap(file, mode=mmap_mode) > File > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/format.py", > line 484, in open_memmap > mode=mode, offset=offset) > File > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/core/memmap.py", > line 231, in __new__ > mm = mmap.mmap(fid.fileno(), bytes, access=acc, > offset=offset) > error: [Errno 22] Invalid argument > > ---------------------------------------------------------------------- > Ran 1889 tests in 12.656s > > FAILED (KNOWNFAIL=1, errors=2) > Should be fixed in r6864. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From efiring at hawaii.edu Sun Apr 12 20:26:55 2009 From: efiring at hawaii.edu (Eric Firing) Date: Sun, 12 Apr 2009 14:26:55 -1000 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <9457e7c80904121611u44d060f7i62eec5d35a637ea3@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <5b8d13220904101956v15cc4184r7c8144a519c369ea@mail.gmail.com> <87bpr11q1r.fsf@hbox.dyndns.org> <9457e7c80904121611u44d060f7i62eec5d35a637ea3@mail.gmail.com> Message-ID: <49E286CF.80901@hawaii.edu> St?fan, or other git-users, One feature of hg that I use frequently is "hg serve", the builtin web server. I use it for two purposes: for temporary local publishing (e.g., in place of using ssh--sometimes it is quicker and easier), and for providing access to the very nice hgwebdir.cgi browsing capability on local repos. I have looked through git commands etc., and have not found an equivalent; am I missing something? The browsing capability of hgwebdir.cgi is much better than any gui interface I have seen for git or for hg. I realize there is a gitweb.cgi, but having that cgi is not the same as being able to publish locally with a single command, and then browse. Thanks. Eric From cournape at gmail.com Sun Apr 12 20:46:49 2009 From: cournape at gmail.com (David Cournapeau) Date: Mon, 13 Apr 2009 09:46:49 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49E286CF.80901@hawaii.edu> References: <49CE3A2C.9000007@enthought.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <5b8d13220904101956v15cc4184r7c8144a519c369ea@mail.gmail.com> <87bpr11q1r.fsf@hbox.dyndns.org> <9457e7c80904121611u44d060f7i62eec5d35a637ea3@mail.gmail.com> <49E286CF.80901@hawaii.edu> Message-ID: <5b8d13220904121746l7d46fde7kcff876f55a2ad0fa@mail.gmail.com> On Mon, Apr 13, 2009 at 9:26 AM, Eric Firing wrote: > > St?fan, or other git-users, > > One feature of hg that I use frequently is "hg serve", the builtin web > server. ?I use it for two purposes: for temporary local publishing > (e.g., in place of using ssh--sometimes it is quicker and easier), and > for providing access to the very nice hgwebdir.cgi browsing capability > on local repos. There is git daemon, but it does not provide 'web publishing' AFAIK, cheers, David From cournape at gmail.com Sun Apr 12 21:10:48 2009 From: cournape at gmail.com (David Cournapeau) Date: Mon, 13 Apr 2009 10:10:48 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <5b8d13220904121746l7d46fde7kcff876f55a2ad0fa@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <5b8d13220904101956v15cc4184r7c8144a519c369ea@mail.gmail.com> <87bpr11q1r.fsf@hbox.dyndns.org> <9457e7c80904121611u44d060f7i62eec5d35a637ea3@mail.gmail.com> <49E286CF.80901@hawaii.edu> <5b8d13220904121746l7d46fde7kcff876f55a2ad0fa@mail.gmail.com> Message-ID: <5b8d13220904121810n4baa684akfe51148b9eff637a@mail.gmail.com> On Mon, Apr 13, 2009 at 9:46 AM, David Cournapeau wrote: > On Mon, Apr 13, 2009 at 9:26 AM, Eric Firing wrote: >> >> St?fan, or other git-users, >> >> One feature of hg that I use frequently is "hg serve", the builtin web >> server. ?I use it for two purposes: for temporary local publishing >> (e.g., in place of using ssh--sometimes it is quicker and easier), and >> for providing access to the very nice hgwebdir.cgi browsing capability >> on local repos. > > There is git daemon, but it does not provide 'web publishing' AFAIK, But you have this which seems to work (I just checked that gitserve gives something which looks like a webpage, nothing more :) ): http://github.com/jezdez/git-serve/tree/master I guess this does not work on windows, though David From efiring at hawaii.edu Sun Apr 12 22:11:53 2009 From: efiring at hawaii.edu (Eric Firing) Date: Sun, 12 Apr 2009 16:11:53 -1000 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <5b8d13220904121810n4baa684akfe51148b9eff637a@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <5b8d13220904101956v15cc4184r7c8144a519c369ea@mail.gmail.com> <87bpr11q1r.fsf@hbox.dyndns.org> <9457e7c80904121611u44d060f7i62eec5d35a637ea3@mail.gmail.com> <49E286CF.80901@hawaii.edu> <5b8d13220904121746l7d46fde7kcff876f55a2ad0fa@mail.gmail.com> <5b8d13220904121810n4baa684akfe51148b9eff637a@mail.gmail.com> Message-ID: <49E29F69.6090008@hawaii.edu> David Cournapeau wrote: > On Mon, Apr 13, 2009 at 9:46 AM, David Cournapeau wrote: >> On Mon, Apr 13, 2009 at 9:26 AM, Eric Firing wrote: >>> St?fan, or other git-users, >>> >>> One feature of hg that I use frequently is "hg serve", the builtin web >>> server. I use it for two purposes: for temporary local publishing >>> (e.g., in place of using ssh--sometimes it is quicker and easier), and >>> for providing access to the very nice hgwebdir.cgi browsing capability >>> on local repos. >> There is git daemon, but it does not provide 'web publishing' AFAIK, > > But you have this which seems to work (I just checked that gitserve > gives something which looks like a webpage, nothing more :) ): > > http://github.com/jezdez/git-serve/tree/master Sure enough, that is what I was looking for. (gitweb doesn't seem to have the annotate [or blame, in git-speak] option, or the graph.) > > I guess this does not work on windows, though It probably does--it is written in python. Thanks. Eric From david at ar.media.kyoto-u.ac.jp Sun Apr 12 22:13:17 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 13 Apr 2009 11:13:17 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49E29F69.6090008@hawaii.edu> References: <49CE3A2C.9000007@enthought.com> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <5b8d13220904101956v15cc4184r7c8144a519c369ea@mail.gmail.com> <87bpr11q1r.fsf@hbox.dyndns.org> <9457e7c80904121611u44d060f7i62eec5d35a637ea3@mail.gmail.com> <49E286CF.80901@hawaii.edu> <5b8d13220904121746l7d46fde7kcff876f55a2ad0fa@mail.gmail.com> <5b8d13220904121810n4baa684akfe51148b9eff637a@mail.gmail.com> <49E29F69.6090008@hawaii.edu> Message-ID: <49E29FBD.3000804@ar.media.kyoto-u.ac.jp> Eric Firing wrote: > > Sure enough, that is what I was looking for. (gitweb doesn't seem to > have the annotate [or blame, in git-speak] option, or the graph.) > It indeed does not support blame, which is a bit of a shame, as git blame is so much better than svn blame (it automatically detects some code moves). > > It probably does--it is written in python. > Yes, but it is just a script to call git-daemon. I am somewhat doubtful that it would work on windows, but I would love being proven wrong :) David From strawman at astraw.com Sun Apr 12 22:32:02 2009 From: strawman at astraw.com (Andrew Straw) Date: Sun, 12 Apr 2009 19:32:02 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49E29F69.6090008@hawaii.edu> References: <49CE3A2C.9000007@enthought.com> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <5b8d13220904101956v15cc4184r7c8144a519c369ea@mail.gmail.com> <87bpr11q1r.fsf@hbox.dyndns.org> <9457e7c80904121611u44d060f7i62eec5d35a637ea3@mail.gmail.com> <49E286CF.80901@hawaii.edu> <5b8d13220904121746l7d46fde7kcff876f55a2ad0fa@mail.gmail.com> <5b8d13220904121810n4baa684akfe51148b9eff637a@mail.gmail.com> <49E29F69.6090008@hawaii.edu> Message-ID: <49E2A422.3070606@astraw.com> Eric Firing wrote: > Sure enough, that is what I was looking for. (gitweb doesn't seem to > have the annotate [or blame, in git-speak] option, or the graph.) > gitweb does, you have to turn it on, though... You need to add this to your gitweb.conf, though: $feature{'blame'}{'default'} = [1]; $feature{'blame'}{'override'} = 1; I also find pickaxe and snapshot useful: $feature{'pickaxe'}{'default'} = [1]; $feature{'pickaxe'}{'override'} = 1; $feature{'snapshot'}{'default'} = ['zip', 'tgz']; $feature{'snapshot'}{'override'} = 1; I don't know about the graph. (You mean the "gitk --all" kind of view? I saw one JavaScript-y web app that did that, but it was slow and ugly for any non-trivial repo.) From ondrej at certik.cz Sun Apr 12 23:19:38 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Sun, 12 Apr 2009 20:19:38 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49E29FBD.3000804@ar.media.kyoto-u.ac.jp> References: <49CE3A2C.9000007@enthought.com> <49DFB726.3000002@hawaii.edu> <5b8d13220904101956v15cc4184r7c8144a519c369ea@mail.gmail.com> <87bpr11q1r.fsf@hbox.dyndns.org> <9457e7c80904121611u44d060f7i62eec5d35a637ea3@mail.gmail.com> <49E286CF.80901@hawaii.edu> <5b8d13220904121746l7d46fde7kcff876f55a2ad0fa@mail.gmail.com> <5b8d13220904121810n4baa684akfe51148b9eff637a@mail.gmail.com> <49E29F69.6090008@hawaii.edu> <49E29FBD.3000804@ar.media.kyoto-u.ac.jp> Message-ID: <85b5c3130904122019t2424d167tcaa8b7f956c0302d@mail.gmail.com> On Sun, Apr 12, 2009 at 7:13 PM, David Cournapeau wrote: > Eric Firing wrote: >> It probably does--it is written in python. >> > > Yes, but it is just a script to call git-daemon. I am somewhat doubtful > that it would work on windows, but I would love being proven wrong :) It uses os.fork() which doesn't work on windows. Ondrej From josef.pktd at gmail.com Sun Apr 12 23:26:27 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 12 Apr 2009 23:26:27 -0400 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49E29FBD.3000804@ar.media.kyoto-u.ac.jp> References: <49CE3A2C.9000007@enthought.com> <49DFB726.3000002@hawaii.edu> <5b8d13220904101956v15cc4184r7c8144a519c369ea@mail.gmail.com> <87bpr11q1r.fsf@hbox.dyndns.org> <9457e7c80904121611u44d060f7i62eec5d35a637ea3@mail.gmail.com> <49E286CF.80901@hawaii.edu> <5b8d13220904121746l7d46fde7kcff876f55a2ad0fa@mail.gmail.com> <5b8d13220904121810n4baa684akfe51148b9eff637a@mail.gmail.com> <49E29F69.6090008@hawaii.edu> <49E29FBD.3000804@ar.media.kyoto-u.ac.jp> Message-ID: <1cd32cbb0904122026p53eb3b60j8fc9e7907c9c811a@mail.gmail.com> On Sun, Apr 12, 2009 at 10:13 PM, David Cournapeau wrote: > Eric Firing wrote: >> >> Sure enough, that is what I was looking for. ?(gitweb doesn't seem to >> have the annotate [or blame, in git-speak] option, or the graph.) >> > > It indeed does not support blame, which is a bit of a shame, as git > blame is so much better than svn blame (it ?automatically detects some > code moves). > >> >> It probably does--it is written in python. >> > > Yes, but it is just a script to call git-daemon. I am somewhat doubtful > that it would work on windows, but I would love being proven wrong :) > I looked briefly at the source of gitserve and it's posix only, too many functions that won't work on windows including fork, and the cgi script is in perl. hg serve works well even in my 1 year old mercurial install, however I didn't find history by subtree which is my favorite in trac, e.g. http://projects.scipy.org/scipy/log/trunk/scipy/stats Josef From mg at lazybytes.net Mon Apr 13 03:21:45 2009 From: mg at lazybytes.net (Martin Geisler) Date: Mon, 13 Apr 2009 09:21:45 +0200 Subject: [Numpy-discussion] DVCS at PyCon References: <49CE3A2C.9000007@enthought.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <5b8d13220904101956v15cc4184r7c8144a519c369ea@mail.gmail.com> <87bpr11q1r.fsf@hbox.dyndns.org> <9457e7c80904121611u44d060f7i62eec5d35a637ea3@mail.gmail.com> Message-ID: <87iql9m31y.fsf@hbox.dyndns.org> St?fan van der Walt writes: > Hi Martin > > 2009/4/13 Martin Geisler : >> In Mercurial it creates a *named >> branch*, which are imbedded in the history and are used for long-term >> development branches. Names of Git branches are not recorded in the >> history and are thus well-suited for local experiments. > > [...] > >> The bookmarks extension let you assign names to the heads, it is >> inspired by the way Git branches work. Unfortunately the bookmarks are >> not yet transferred over the network when you do push/pull, but this is >> of course a planned feature. > > Thanks for the info. Maybe you can also help me out with the following > questions: I'd like to know how to duplicate git's branch workflow in > Mercurial. > > - How do you create a new head when you are working on top of the > current head? Do you have to make a dummy commit and then commit on > top of (TIP - 1) whenever you need a new "branch"? Yeah, you can create multiple heads like that. In general, let's say the current tip of the tree is T: ... --- T You've just cloned this tree from the Numpy server. To make your head you will just do a commit: ... --- T --- S In Mercurial there is no notion of "my head" and "your head" and (Git-style) branches are just implicitly made from how the changeset graph develops. So if I have also started from T and made a commit my tree will look like this: ... --- T --- M and so far neither of us knows about the other guys branch. We can both tell that we've done something by using 'hg outgoing', which will tell you that the S changeset is missing at Numpy, and tell me that the M changeset is missing. Now imagine I push my change to Numpy -- if you do 'hg incoming' you'll see that there is a missing changeset, and if you do 'hg pull' your tree will end up like this: S / ... --- T \ M The files in your working copy will still be at the S changeset ('hg parents' will say S), but you can now see that your changeset has created a branch with regard to the Numpy repository. But you can continue working and make U, V, etc: S --- U --- V / ... --- T \ M In the mean time I might commit N and O to my tree: ... --- T --- M --- N --- O and push this to Numpy. If you pull from Numpy you'll now see S --- U --- V / ... --- T \ M --- N --- O At some point you can decide to merge your branch into the main branch and get a merge changeset X: S --- U --- V / \ ... --- T X \ / M --- N---- O > - Is it possible to remove a head without merging it, discarding the > children? Or your can decide to forget about your branch by making a local clone where you only ask for the O head: hg clone -r O repo repo-with-O The new repo-with-O clone will look like this: ... --- T --- M --- N --- O i.e., it will be exactly like what's on Numpy at this point in time. The strip command from the mq extension gives an alternative way to do this from within a single clone: hg strip -r S will remove S and all child changesets, i.e., S, U, V and the result will again be exactly what is on Numpy. -- Martin Geisler VIFF (Virtual Ideal Functionality Framework) brings easy and efficient SMPC (Secure Multiparty Computation) to Python. See: http://viff.dk/. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available URL: From stefan at sun.ac.za Mon Apr 13 03:22:49 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 13 Apr 2009 09:22:49 +0200 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49E286CF.80901@hawaii.edu> References: <49CE3A2C.9000007@enthought.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <5b8d13220904101956v15cc4184r7c8144a519c369ea@mail.gmail.com> <87bpr11q1r.fsf@hbox.dyndns.org> <9457e7c80904121611u44d060f7i62eec5d35a637ea3@mail.gmail.com> <49E286CF.80901@hawaii.edu> Message-ID: <9457e7c80904130022h7f995db3veef9fa666ce0e7a8@mail.gmail.com> 2009/4/13 Eric Firing : > > St?fan, or other git-users, > > One feature of hg that I use frequently is "hg serve", the builtin web > server. ?I use it for two purposes: for temporary local publishing > (e.g., in place of using ssh--sometimes it is quicker and easier), and > for providing access to the very nice hgwebdir.cgi browsing capability > on local repos. ?I have looked through git commands etc., and have not > found an equivalent; am I missing something? ?The browsing capability of > hgwebdir.cgi is much better than any gui interface I have seen for git > or for hg. ?I realize there is a gitweb.cgi, but having that cgi is not > the same as being able to publish locally with a single command, and > then browse. The command is git instaweb --httpd=webrick Cheers St?fan From ondrej at certik.cz Mon Apr 13 03:42:06 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 13 Apr 2009 00:42:06 -0700 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <9457e7c80904130022h7f995db3veef9fa666ce0e7a8@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <5b8d13220904101956v15cc4184r7c8144a519c369ea@mail.gmail.com> <87bpr11q1r.fsf@hbox.dyndns.org> <9457e7c80904121611u44d060f7i62eec5d35a637ea3@mail.gmail.com> <49E286CF.80901@hawaii.edu> <9457e7c80904130022h7f995db3veef9fa666ce0e7a8@mail.gmail.com> Message-ID: <85b5c3130904130042v6d70f339pc669da6daca91c99@mail.gmail.com> 2009/4/13 St?fan van der Walt : > 2009/4/13 Eric Firing : >> >> St?fan, or other git-users, >> >> One feature of hg that I use frequently is "hg serve", the builtin web >> server. ?I use it for two purposes: for temporary local publishing >> (e.g., in place of using ssh--sometimes it is quicker and easier), and >> for providing access to the very nice hgwebdir.cgi browsing capability >> on local repos. ?I have looked through git commands etc., and have not >> found an equivalent; am I missing something? ?The browsing capability of >> hgwebdir.cgi is much better than any gui interface I have seen for git >> or for hg. ?I realize there is a gitweb.cgi, but having that cgi is not >> the same as being able to publish locally with a single command, and >> then browse. > > The command is > > git instaweb --httpd=webrick Ah, that's nice, thanks for sharing it. I didn't know about it. Works fine for me. Ondrej From nwagner at iam.uni-stuttgart.de Mon Apr 13 03:56:42 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Mon, 13 Apr 2009 09:56:42 +0200 Subject: [Numpy-discussion] numpy.test() errors r6862 In-Reply-To: References: Message-ID: On Sun, 12 Apr 2009 12:56:46 -0600 Charles R Harris wrote: > On Sun, Apr 12, 2009 at 12:52 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Sun, Apr 12, 2009 at 12:17 PM, Nils Wagner < >> nwagner at iam.uni-stuttgart.de> wrote: >> >>> ====================================================================== >>> ERROR: test suite >>> ---------------------------------------------------------------------- >>> Traceback (most recent call last): >>> File >>> >>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/suite.py", >>> line 154, in run >>> self.setUp() >>> File >>> >>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/suite.py", >>> line 180, in setUp >>> if not self: >>> File >>> >>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/suite.py", >>> line 65, in __nonzero__ >>> test = self.test_generator.next() >>> File >>> >>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/loader.py", >>> line 221, in generate >>> for test in g(): >>> File >>> >>> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_format.py", >>> line 440, in test_memmap_roundtrip >>> shape=arr.shape, fortran_order=fortran_order) >>> File >>> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/format.py", >>> line 484, in open_memmap >>> mode=mode, offset=offset) >>> File >>> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/core/memmap.py", >>> line 231, in __new__ >>> mm = mmap.mmap(fid.fileno(), bytes, access=acc, >>> offset=offset) >>> error: [Errno 22] Invalid argument >>> >>> ====================================================================== >>> ERROR: test_mmap (test_io.TestSaveLoad) >>> ---------------------------------------------------------------------- >>> Traceback (most recent call last): >>> File >>> >>> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/decorators.py", >>> line 169, in knownfailer >>> return f(*args, **kwargs) >>> File >>> >>> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_io.py", >>> line 92, in test_mmap >>> self.roundtrip(a, file_on_disk=True, >>> load_kwds={'mmap_mode': 'r'}) >>> File >>> >>> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_io.py", >>> line 100, in roundtrip >>> RoundtripTest.roundtrip(self, np.save, *args, >>> **kwargs) >>> File >>> >>> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_io.py", >>> line 67, in roundtrip >>> arr_reloaded = np.load(load_file, **load_kwds) >>> File >>> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/io.py", >>> line 193, in load >>> return format.open_memmap(file, mode=mmap_mode) >>> File >>> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/format.py", >>> line 484, in open_memmap >>> mode=mode, offset=offset) >>> File >>> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/core/memmap.py", >>> line 231, in __new__ >>> mm = mmap.mmap(fid.fileno(), bytes, access=acc, >>> offset=offset) >>> error: [Errno 22] Invalid argument >>> >>> ---------------------------------------------------------------------- >>> Ran 1889 tests in 12.656s >>> >>> FAILED (KNOWNFAIL=1, errors=2) >>> >> >> Hmm, I'll guess that the problem is this: >> >> *offset* must be a multiple of the >>ALLOCATIONGRANULARITY. >> >> This conflicts with the current intent of offset. Looks >>like we need to fix >> up the patch to use the nearest multiple of >>ALLOCATIONGRANULARITY and then >> offset the usual way. Or, since ALLOCATIONGRANULARITY is >>likely to be >> platform dependent, maybe we should just revert the >>patch. >> > > Can you import mmap, then do a dir(mmap) and see if > ALLOCATIONGRANULARITY > is available? > > TIA, > > Chuck Hi Chuck, >>> import mmap >>> dir (mmap) ['ACCESS_COPY', 'ACCESS_READ', 'ACCESS_WRITE', 'ALLOCATIONGRANULARITY', 'MAP_ANON', 'MAP_ANONYMOUS', 'MAP_DENYWRITE', 'MAP_EXECUTABLE', 'MAP_PRIVATE', 'MAP_SHARED', 'PAGESIZE', 'PROT_EXEC', 'PROT_READ', 'PROT_WRITE', '__doc__', '__file__', '__name__', '__package__', 'error', 'mmap'] All tests pass with '1.4.0.dev6864'. Thank you very much. Cheers, Nils From nwagner at iam.uni-stuttgart.de Mon Apr 13 04:00:50 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Mon, 13 Apr 2009 10:00:50 +0200 Subject: [Numpy-discussion] survey of freely available software for the solution of linear algebra problems Message-ID: FWIW, From: Jack Dongarra Date: Tue, 7 Apr 2009 12:00:01 -0400 Subject: Survey of linear algebra software We have updated the survey of freely available software for the solution of linear algebra problems. Send us comments if you see a problem. http://www.netlib.org/utk/people/JackDongarra/la-sw.html Regards, Jack and Hatem From efiring at hawaii.edu Mon Apr 13 04:01:47 2009 From: efiring at hawaii.edu (Eric Firing) Date: Sun, 12 Apr 2009 22:01:47 -1000 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <9457e7c80904130022h7f995db3veef9fa666ce0e7a8@mail.gmail.com> References: <49CE3A2C.9000007@enthought.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <49DFB726.3000002@hawaii.edu> <5b8d13220904101956v15cc4184r7c8144a519c369ea@mail.gmail.com> <87bpr11q1r.fsf@hbox.dyndns.org> <9457e7c80904121611u44d060f7i62eec5d35a637ea3@mail.gmail.com> <49E286CF.80901@hawaii.edu> <9457e7c80904130022h7f995db3veef9fa666ce0e7a8@mail.gmail.com> Message-ID: <49E2F16B.7060303@hawaii.edu> St?fan van der Walt wrote: > 2009/4/13 Eric Firing : >> St?fan, or other git-users, >> >> One feature of hg that I use frequently is "hg serve", the builtin web >> server. I use it for two purposes: for temporary local publishing >> (e.g., in place of using ssh--sometimes it is quicker and easier), and >> for providing access to the very nice hgwebdir.cgi browsing capability >> on local repos. I have looked through git commands etc., and have not >> found an equivalent; am I missing something? The browsing capability of >> hgwebdir.cgi is much better than any gui interface I have seen for git >> or for hg. I realize there is a gitweb.cgi, but having that cgi is not >> the same as being able to publish locally with a single command, and >> then browse. > > The command is > > git instaweb --httpd=webrick Well, sort of--but one must already have one of the three supported web servers installed, so it is a bit heavier-weight. The gitserve program is closer to "hg serve" because all it requires is git and python (and *nix, unlike hg serve). Eric > > Cheers > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From mg at lazybytes.net Mon Apr 13 04:03:47 2009 From: mg at lazybytes.net (Martin Geisler) Date: Mon, 13 Apr 2009 10:03:47 +0200 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <85b5c3130904100154l797d062fn90ea9389bc2da0b@mail.gmail.com> (Ondrej Certik's message of "Fri, 10 Apr 2009 01:54:15 -0700") References: <49CE3A2C.9000007@enthought.com> <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <85b5c3130904100154l797d062fn90ea9389bc2da0b@mail.gmail.com> Message-ID: <878wm5m13w.fsf@hbox.dyndns.org> Ondrej Certik writes: > Plus with git, you can fetch the remote repository with all the > branches and browse them locally in your remote branches, when you are > offline. And merge them with your own branches. In mercurial, it seems > the only way to see what changes are there and which branch and which > commits I want to merge is to use "hg in", but that requires the > internet connection, so it's basically like svn. No, no, ... I think you're misunderstanding how the history graph works in Mercurial. It works like it does in Git -- changesets are in a parent-child relationship. So if I cloned a repository at changeset T: ... --- T I'm free to make new commits: ... --- T --- A --- B And you can do the same: ... --- T --- X --- Y --- Z I can pull in your changesets without disrupting my own work: X --- Y --- Z / ... --- T --- A --- B Your changesets will be attached to the graph at the point where you made them, the same for my changesets. I don't have to merge at this point, instead I can continue with my work even though the repository has multiple heads (changesets with no childre). So I make C: X / ... --- T --- A --- B --- C I might now decide that I like your X, Y changesets but not Z, so I merge them Y into my line of work to create D: X --- Y --- Z / \ ... --- T `---- D \ / A --- B --- C or I might decide that I don't need your changesets and discard them by cloning or by the strip command from mq (one or the other): hg clone -r C repo repo-with-C hg strip X The result is a repository that has only the history up to C: ... --- T --- A --- B --- C So I think it's nonsense to say that Mercurial is like Subversion here: you pull in changesets when online and you can make new commits, merge commits, delete commits while offline. Git has the advantage that it names these branches in a nice way, and you can work with these names across the network. The bookmarks extension for Mercurial is inspired by this and lets you attach local names to heads in the graph. > I don't know if mercurial has improved in this lately, but at least > for me, that's a major problem with mercurial. There has been no improvement lately since Mercurial has worked like this all the time :-) -- Martin Geisler VIFF (Virtual Ideal Functionality Framework) brings easy and efficient SMPC (Secure Multiparty Computation) to Python. See: http://viff.dk/. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available URL: From david at ar.media.kyoto-u.ac.jp Mon Apr 13 04:10:02 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 13 Apr 2009 17:10:02 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <878wm5m13w.fsf@hbox.dyndns.org> References: <49CE3A2C.9000007@enthought.com> <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <85b5c3130904100154l797d062fn90ea9389bc2da0b@mail.gmail.com> <878wm5m13w.fsf@hbox.dyndns.org> Message-ID: <49E2F35A.2010704@ar.media.kyoto-u.ac.jp> Martin Geisler wrote: > Ondrej Certik writes: > >> Plus with git, you can fetch the remote repository with all the >> branches and browse them locally in your remote branches, when you are >> offline. And merge them with your own branches. In mercurial, it seems >> the only way to see what changes are there and which branch and which >> commits I want to merge is to use "hg in", but that requires the >> internet connection, so it's basically like svn. > > No, no, ... I think you're misunderstanding how the history graph works > in Mercurial. It works like it does in Git -- changesets are in a > parent-child relationship. So if I cloned a repository at changeset T: > > ... --- T > > I'm free to make new commits: > > ... --- T --- A --- B > > And you can do the same: > > ... --- T --- X --- Y --- Z > > I can pull in your changesets without disrupting my own work: > > X --- Y --- Z > / > ... --- T --- A --- B > > Your changesets will be attached to the graph at the point where you > made them, the same for my changesets. I don't have to merge at this > point, instead I can continue with my work even though the repository > has multiple heads (changesets with no childre). So I make C: > > X > / > ... --- T --- A --- B --- C > > I might now decide that I like your X, Y changesets but not Z, so I > merge them Y into my line of work to create D: > > X --- Y --- Z > / \ > ... --- T `---- D > \ / > A --- B --- C > > or I might decide that I don't need your changesets and discard them by > cloning or by the strip command from mq (one or the other): > > hg clone -r C repo repo-with-C > hg strip X > > The result is a repository that has only the history up to C: > > ... --- T --- A --- B --- C > > So I think it's nonsense to say that Mercurial is like Subversion here: > you pull in changesets when online and you can make new commits, merge > commits, delete commits while offline. > > Git has the advantage that it names these branches in a nice way, and > you can work with these names across the network. The bookmarks > extension for Mercurial is inspired by this and lets you attach local > names to heads in the graph. Thanks a lot for this information, that's really useful. I am still a bit confused about how this works at the UI level, though, w.r.t one clone/branch. In bzr, there is one branch and at most one working tree / repository (at least that's how it is used generally). It looks like hg, while being based on one branch / repository, is a more flexible. One thing which converted me to git from bzr was the UI for branch comparison. My basic reference is for release process. Currently, in numpy, we have the trunk, and potentially several release branches, which would be one head each if I get the vocabulary right: ------ A ------------ B --------------- C (1.2.x head) / trunk ------ Release 1.2.0 -------- Release 1.3.0 -------- (mainline head) \ D--------- E ------- F -------- (1.3.x head) How do you refer to different branches from one clone ? For example, if I want to see the differences between mainline and 1.3.x branch, cherry-pick things from mainline into both 1.3.x and 1.2.x, etc... How does it work with bookmarks ? Also, do we have to agree on everyone using bookmark to communicate each other (one repository for everything on main numpy web repository), or can people still use clones for every branch, and "putting things back into bookmarks" when they push things in the official repo ? cheers, David From mg at lazybytes.net Mon Apr 13 05:22:21 2009 From: mg at lazybytes.net (Martin Geisler) Date: Mon, 13 Apr 2009 11:22:21 +0200 Subject: [Numpy-discussion] DVCS at PyCon References: <49CE3A2C.9000007@enthought.com> <1cd32cbb0904091330g57fec446j2e3c4745cc304273@mail.gmail.com> <5b8d13220904092013s72651e5aj357f2c3493a3291@mail.gmail.com> <1cd32cbb0904092219y7932c0d3ha51ac78524f31894@mail.gmail.com> <49DED867.5050206@ar.media.kyoto-u.ac.jp> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <85b5c3130904100154l797d062fn90ea9389bc2da0b@mail.gmail.com> <878wm5m13w.fsf@hbox.dyndns.org> <49E2F35A.2010704@ar.media.kyoto-u.ac.jp> Message-ID: <87y6u4lxgy.fsf@hbox.dyndns.org> David Cournapeau writes: Hi David x 2 :-) I've put the David Soria on Cc since he wrote the bookmarks extension, maybe he can give additional information. The thread can be found here: http://thread.gmane.org/gmane.comp.python.numeric.general/29117 > Martin Geisler wrote: > >> [...] changesets are in a parent-child relationship. So if I cloned a >> repository at changeset T: >> >> ... --- T >> >> I'm free to make new commits: >> >> ... --- T --- A --- B >> >> And you can do the same: >> >> ... --- T --- X --- Y --- Z >> >> I can pull in your changesets without disrupting my own work: >> >> X --- Y --- Z >> / >> ... --- T --- A --- B >> >> Your changesets will be attached to the graph at the point where you >> made them, the same for my changesets. I don't have to merge at this >> point, instead I can continue with my work even though the repository >> has multiple heads (changesets with no childre). So I make C: >> >> X >> / >> ... --- T --- A --- B --- C >> >> I might now decide that I like your X, Y changesets but not Z, so I >> merge them Y into my line of work to create D: >> >> X --- Y --- Z >> / \ >> ... --- T `---- D >> \ / >> A --- B --- C >> >> or I might decide that I don't need your changesets and discard them by >> cloning or by the strip command from mq (one or the other): >> >> hg clone -r C repo repo-with-C >> hg strip X >> >> The result is a repository that has only the history up to C: >> >> ... --- T --- A --- B --- C >> >> So I think it's nonsense to say that Mercurial is like Subversion here: >> you pull in changesets when online and you can make new commits, merge >> commits, delete commits while offline. >> >> Git has the advantage that it names these branches in a nice way, and >> you can work with these names across the network. The bookmarks >> extension for Mercurial is inspired by this and lets you attach local >> names to heads in the graph. > > Thanks a lot for this information, that's really useful. Great! I trust that you guys will let me know when/if you get tired of this discussion :-) > I am still a bit confused about how this works at the UI level, > though, w.r.t one clone/branch. In bzr, there is one branch and at > most one working tree / repository (at least that's how it is used > generally). It looks like hg, while being based on one branch / > repository, is a more flexible. One thing which converted me to git > from bzr was the UI for branch comparison. My basic reference is for > release process. Currently, in numpy, we have the trunk, and > potentially several release branches, which would be one head each if > I get the vocabulary right: > > --- A --- B --- C (1.2.x head) > / > trunk --- Release 1.2.0 --- Release 1.3.0 --- (mainline head) > \ > D --- E --- F --- (1.3.x head) > > How do you refer to different branches from one clone ? For example, > if I want to see the differences between mainline and 1.3.x branch, > cherry-pick things from mainline into both 1.3.x and 1.2.x, etc... How > does it work with bookmarks ? You write things like hg diff -r F -r tip where 'tip' is a built-in name that always point to the newest revision in a repository. If you have a bookmark named 'numpy-1.2.x' on F you could write: hg diff -r numpy-1.2.x -r tip As for cherry-picking the recommended way (in both Git and Mercurial, as far as I know) is to do the commit on the oldest relevant branch and then merge this branch into younger branches and finally into the mainline. This way each branch is a strict subset of the next branch, and mainline contains all commits on all branches. But of course one might realise too late that a changeset on mainline should have been made on, say, the 1.3.x branch. In case one can apply the change as a new changeset by exporting it from mainline and importing it on 1.3.x: hg export tip > tip.patch hg update -C 1.3.x hg import tip.patch The transplant extension for Mercurial can help with this, probably with better handling of merges, but I don't have any real experience with it. > Also, do we have to agree on everyone using bookmark to communicate > each other (one repository for everything on main numpy web > repository), or can people still use clones for every branch, and > "putting things back into bookmarks" when they push things in the > official repo ? The bookmarks is a convenience layer on top of the basic functionality in Mercurial. They let you attach a name to a changeset just like a tag, but unlike tags the names move around: if you make a commit based on a changeset F with 'numpy-1.3.x', the child changeset will now have the bookmark 'numpy-1.3.x'. This way bookmarks become moveable pointers into the history, similarly to how Git heads point to changesets in the history. At the moment, bookmarks are even local to a repository, but it is planned to add support for looking them up remotely and for transferring them on push/pull. But right now you cannot tell if I'm using bookmarks to keep track of the heads in my clone. -- Martin Geisler VIFF (Virtual Ideal Functionality Framework) brings easy and efficient SMPC (Secure Multiparty Computation) to Python. See: http://viff.dk/. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available URL: From ndbecker2 at gmail.com Mon Apr 13 07:10:40 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 13 Apr 2009 07:10:40 -0400 Subject: [Numpy-discussion] [OT] read data from pdf Message-ID: Anyone know of software that can assist with reading data points from a pdf version of a 2-d line graph? From nadavh at visionsense.com Mon Apr 13 07:33:41 2009 From: nadavh at visionsense.com (Nadav Horesh) Date: Mon, 13 Apr 2009 14:33:41 +0300 Subject: [Numpy-discussion] [OT] read data from pdf References: Message-ID: <710F2847B0018641891D9A216027636029C4C7@ex3.envision.co.il> Do you mean a manual digitation? You can use g3data after converting the plot to a bitmap. Nadav -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Neal Becker ????: ? 13-?????-09 13:10 ??: numpy-discussion at scipy.org ????: [Numpy-discussion] [OT] read data from pdf Anyone know of software that can assist with reading data points from a pdf version of a 2-d line graph? _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 2791 bytes Desc: not available URL: From jsilva at fc.up.pt Mon Apr 13 07:36:10 2009 From: jsilva at fc.up.pt (=?ISO-8859-1?Q?Jo=E3o_Lu=EDs_Silva?=) Date: Mon, 13 Apr 2009 12:36:10 +0100 Subject: [Numpy-discussion] [OT] read data from pdf In-Reply-To: References: Message-ID: Neal Becker wrote: > Anyone know of software that can assist with reading data points from a pdf > version of a 2-d line graph? There are programs to help convert a graphic image to data points, such as http://plotdigitizer.sourceforge.net/ From gruben at bigpond.net.au Mon Apr 13 07:56:13 2009 From: gruben at bigpond.net.au (Gary Ruben) Date: Mon, 13 Apr 2009 21:56:13 +1000 Subject: [Numpy-discussion] [OT] read data from pdf In-Reply-To: References: Message-ID: <49E3285D.4090800@bigpond.net.au> My friend has used this successfully: Looks like this will do it too: Gary R. Jo?o Lu?s Silva wrote: > Neal Becker wrote: >> Anyone know of software that can assist with reading data points from a pdf >> version of a 2-d line graph? > > There are programs to help convert a graphic image to data points, such > as http://plotdigitizer.sourceforge.net/ From gael.varoquaux at normalesup.org Mon Apr 13 08:02:46 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 13 Apr 2009 14:02:46 +0200 Subject: [Numpy-discussion] [OT] read data from pdf In-Reply-To: References: Message-ID: <20090413120246.GD13133@phare.normalesup.org> On Mon, Apr 13, 2009 at 07:10:40AM -0400, Neal Becker wrote: > Anyone know of software that can assist with reading data points from a pdf > version of a 2-d line graph? I know domeone who had a lot of success with engauge-digitizer (packaged in Ubuntu). Ga?l From cournape at gmail.com Mon Apr 13 09:03:40 2009 From: cournape at gmail.com (David Cournapeau) Date: Mon, 13 Apr 2009 22:03:40 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <87y6u4lxgy.fsf@hbox.dyndns.org> References: <49CE3A2C.9000007@enthought.com> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <85b5c3130904100154l797d062fn90ea9389bc2da0b@mail.gmail.com> <878wm5m13w.fsf@hbox.dyndns.org> <49E2F35A.2010704@ar.media.kyoto-u.ac.jp> <87y6u4lxgy.fsf@hbox.dyndns.org> Message-ID: <5b8d13220904130603p4d746811qc47a8eee00706c9d@mail.gmail.com> On Mon, Apr 13, 2009 at 6:22 PM, Martin Geisler wrote: > > ?hg diff -r F -r tip > > where 'tip' is a built-in name that always point to the newest revision > in a repository. If you have a bookmark named 'numpy-1.2.x' on F you > could write: > > ?hg diff -r numpy-1.2.x -r tip Ok, so bookmarks are like named branches in that regard. But what if they happened to be in different repositories ? If you have two clones, how do you refer one clone from the other ? That's where bzr UI for branch handling broke IMHO, so I was wondering about hg ? > > As for cherry-picking the recommended way (in both Git and Mercurial, as > far as I know) is to do the commit on the oldest relevant branch and > then merge this branch into younger branches and finally into the > mainline. This way each branch is a strict subset of the next branch, > and mainline contains all commits on all branches. > > But of course one might realise too late that a changeset on mainline > should have been made on, say, the 1.3.x branch. In case one can apply > the change as a new changeset by exporting it from mainline and > importing it on 1.3.x: > > ?hg export tip > tip.patch > ?hg update -C 1.3.x > ?hg import tip.patch > > The transplant extension for Mercurial can help with this, probably with > better handling of merges, but I don't have any real experience with it. Ok. > > The bookmarks is a convenience layer on top of the basic functionality > in Mercurial. They let you attach a name to a changeset just like a tag, > but unlike tags the names move around: if you make a commit based on a > changeset F with 'numpy-1.3.x', the child changeset will now have the > bookmark 'numpy-1.3.x'. This way bookmarks become moveable pointers into > the history, similarly to how Git heads point to changesets in the > history. > > At the moment, bookmarks are even local to a repository, but it is > planned to add support for looking them up remotely and for transferring > them on push/pull. But right now you cannot tell if I'm using bookmarks > to keep track of the heads in my clone. So it means that we would have to keep clones for each branch on the server at the moment, right ? Can I import this into bookmarks ? Say, we have three numpy branches on scipy.org: numpy-mainline numpy-1.2.x numpy-1.3.x and I want to keep everything into one repository, each branch being bookmark. It looks like it is possible internally from my understanding on how it works in hg, but does the UI supports it ? David From mg at lazybytes.net Mon Apr 13 10:02:24 2009 From: mg at lazybytes.net (Martin Geisler) Date: Mon, 13 Apr 2009 16:02:24 +0200 Subject: [Numpy-discussion] DVCS at PyCon References: <49CE3A2C.9000007@enthought.com> <85b5c3130904092256w5e25e07fw1af14bc58804e7b2@mail.gmail.com> <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <85b5c3130904100154l797d062fn90ea9389bc2da0b@mail.gmail.com> <878wm5m13w.fsf@hbox.dyndns.org> <49E2F35A.2010704@ar.media.kyoto-u.ac.jp> <87y6u4lxgy.fsf@hbox.dyndns.org> <5b8d13220904130603p4d746811qc47a8eee00706c9d@mail.gmail.com> Message-ID: <87ocv0lki7.fsf@hbox.dyndns.org> David Cournapeau writes: > On Mon, Apr 13, 2009 at 6:22 PM, Martin Geisler wrote: > >> ?hg diff -r F -r tip >> >> where 'tip' is a built-in name that always point to the newest >> revision in a repository. If you have a bookmark named 'numpy-1.2.x' >> on F you could write: >> >> ?hg diff -r numpy-1.2.x -r tip > > Ok, so bookmarks are like named branches in that regard. Yeah, sort of. Bookmarks are actually more like tags: both concepts allow you to give a changeset a friendly name. Named branches are slightly different: they allow you to stamp a friendly name on a changeset, but not on just one changeset -- all changesets in the named branch get the stamp. The branch name can be used in operations like 'hg update', 'hg diff', etc, and will resolve to the tip-most changeset on the branch. > But what if they happened to be in different repositories ? If you > have two clones, how do you refer one clone from the other ? That's > where bzr UI for branch handling broke IMHO, so I was wondering about > hg ? You're right, the UI is not so good. In particular, you cannot use 'hg diff' to compare repositories. The rdiff extension does this, though: http://www.selenic.com/mercurial/wiki/index.cgi/RdiffExtension Using stock Mercurial you can use 'hg incoming -p' to see the patches of the incoming (missing) changesets, or you can pull in the changesets and then look at them -- if you don't like them you can remove them again. >> At the moment, bookmarks are even local to a repository, but it is >> planned to add support for looking them up remotely and for >> transferring them on push/pull. But right now you cannot tell if I'm >> using bookmarks to keep track of the heads in my clone. > > So it means that we would have to keep clones for each branch on the > server at the moment, right ? Yes, that would be easiest. The bookmarks are stored in a plain text file (.hg/bookmarks) which can be copied around, but the bookmarks will only update themselves upon commit when they point to a head changeset. > Can I import this into bookmarks ? Say, we have three numpy branches > on scipy.org: > > numpy-mainline > numpy-1.2.x > numpy-1.3.x > > and I want to keep everything into one repository, each branch being > bookmark. It looks like it is possible internally from my > understanding on how it works in hg, but does the UI supports it ? Right, it is supported -- you would just pull the changesets from each clone into one clone and this will merge the acyclic graphs into one graph with all changesets. You could do this in just your clone, without requiring the numpy server to do the same. If you do this you'll probably want to attach three bookmarks in your clone in order to make it easy for yourself to update From one head to another. Or you could use named branches for the numpy-1.2.x and numpy-1.3.x branches -- then everybody would see the branch names when they clone the combined repository. -- Martin Geisler VIFF (Virtual Ideal Functionality Framework) brings easy and efficient SMPC (Secure Multiparty Computation) to Python. See: http://viff.dk/. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available URL: From stefan at sun.ac.za Mon Apr 13 11:12:13 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 13 Apr 2009 17:12:13 +0200 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <9457e7c80904120154x7973a5f4v993f0d3d0419fa7f@mail.gmail.com> References: <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <5b8d13220904101917t16fe7aebt9c5e310b42ab7ba4@mail.gmail.com> <49E00641.5050903@hawaii.edu> <5b8d13220904102005y21503770r32ac48c97c412849@mail.gmail.com> <20090411095250.GB8496@phare.normalesup.org> <9457e7c80904111857x28cf93edm14f0e8d2c924410b@mail.gmail.com> <9457e7c80904111936o7b4d37dcuc0342c35238cf748@mail.gmail.com> <9457e7c80904120154x7973a5f4v993f0d3d0419fa7f@mail.gmail.com> Message-ID: <9457e7c80904130812s55697f2bp1239de8428733d35@mail.gmail.com> 2009/4/12 St?fan van der Walt : > I underestimated the > value of this type of manipulation, and of having a clearly structured > and easily traversable history. I read that Bram Cohen of Codeville / "patience diff" fame doesn't agree with me, so I'll give his opinion too: """ Don't bother with a pretty history. The history of a branch is hardly ever looked at. Making it look pretty for the historians is just a waste of time. The beauty of 3-way merge is that you can always clean stuff up later and never worry about the past mess ever again. In particular, don't go to great lengths to make sure that there's a coherent local image of the entire repository exactly as it appeared on your local machine after every new feature. There are very rare projects which maintain a level of reliability and testing which warrant such behavior, and yours isn't one of them. """ http://bramcohen.livejournal.com/52148.html I look at the SciPy history a lot, so I'm not convinced, but there it is. Cheers St?fan From tgrav at mac.com Mon Apr 13 12:39:15 2009 From: tgrav at mac.com (Tommy Grav) Date: Mon, 13 Apr 2009 12:39:15 -0400 Subject: [Numpy-discussion] Leopard install In-Reply-To: <5b8d13220904121602u265a2418j18541fe7479aabde@mail.gmail.com> References: <0EAFD24D-078E-4EAD-AEF0-BC5B55789CBD@cinci.rr.com> <5b8d13220904121602u265a2418j18541fe7479aabde@mail.gmail.com> Message-ID: On Apr 12, 2009, at 7:02 PM, David Cournapeau wrote: > On Mon, Apr 13, 2009 at 1:19 AM, Stuart Edwards > wrote: >> Hi >> >> I am trying to install numpy 1.3.0 on Leopard 10.5.6 and at the point >> in the install process where I select a destination, my boot disc is >> excluded with the message: >> > > I think you need to install python from python.org (version 2.5) to > install the numpy binary, Or you can alternatively use the ActiveState python binary (v2.5), which also seems to work. Cheers Tommy From pav at iki.fi Mon Apr 13 13:17:26 2009 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 13 Apr 2009 17:17:26 +0000 (UTC) Subject: [Numpy-discussion] documentation suggestion References: <9457e7c80904111943hddd3b20h6cc49266fd3d31e0@mail.gmail.com> Message-ID: Sun, 12 Apr 2009 17:32:31 -0400, Neal Becker wrote: [clip] > Done, but can someone check that what I wrote is accurate? I wrote that > changes to the ndarray will change the underlying buffer object. But, > the buffer protocol allows for read-only buffers. Not sure what ndarray > would do if you tried to write in that case. For read-only buffers, the new array will not be writeable: >>> x="abcde" >>> z=np.ndarray((5,), buffer=x, dtype='1S') >>> z[0] = 'x' Traceback (most recent call last): File "", line 1, in RuntimeError: array is not writeable However, you can override this, and misusing it potentially leads to nasty things: >>> x="abcde" >>> y="abcde" >>> z=np.ndarray((5,), buffer=x, dtype='1S') >>> z.flags.writeable = True >>> z[0] = 'x' >>> x 'xbcde' >>> y 'xbcde' >>> hash(x) -1332677140 >>> hash("abcde") -1332677140 Hashing breaks, and the assumed immutability of strings causes nonlocal effects! *** However, for some reason, I don't see your contribution in the change list: http://docs.scipy.org/numpy/changes/ Can you check if the change you made went through? What page did you edit? -- Pauli Virtanen From rowen at u.washington.edu Mon Apr 13 15:58:54 2009 From: rowen at u.washington.edu (Russell E. Owen) Date: Mon, 13 Apr 2009 12:58:54 -0700 Subject: [Numpy-discussion] survey of freely available software for the solution of linear algebra problems References: Message-ID: In article , "Nils Wagner" wrote: > http://www.netlib.org/utk/people/JackDongarra/la-sw.html You might add Eigen: We are finding it to be a very nice package (though the name is unfortunate from the perspective of internet search engines). It is a pure C++ template library, which is brilliant. That makes it much easier to build a package using Eigen than one using lapack/blas/etc.. -- Russell From charlie.xia.fdu at gmail.com Mon Apr 13 17:16:13 2009 From: charlie.xia.fdu at gmail.com (charlie) Date: Mon, 13 Apr 2009 14:16:13 -0700 Subject: [Numpy-discussion] Got: "undefined symbol: PyUnicodeUCS2_FromUnicode" error In-Reply-To: <49DC5BE9.1070901@gmail.com> References: <11c6cf4e0904080105s775e1162y7705a09304d2845f@mail.gmail.com> <49DC5BE9.1070901@gmail.com> Message-ID: <11c6cf4e0904131416y212f4e55l3ea8a0341c068580@mail.gmail.com> Thanks all! I was mixed with the system python version and mine. And I got the problem solved by configuring the python environmental. On Wed, Apr 8, 2009 at 1:10 AM, Michael Abshoff < michael.abshoff at googlemail.com> wrote: > charlie wrote: > > Hi All, > > Hi Charlie, > > > I got the "undefined symbol: PyUnicodeUCS2_FromUnicode" error when > > importing numpy. > > > > I have my own non-root version of python 2.5.4 final installed with > > --prefix=$HOME/usr. > > PYTHONHOME=$HOME/usr; > > PYTHONPATH="$PYTHONHOME/lib:$PYTHONHOME/lib/python2.5/site-packages/" > > install and import other modules works fine. > > I install numpy-1.3.0rc2 from the svn repository with "python setup.py > > install" > > then import numpy results in following error: > > *Python 2.5 (release25-maint, Jul 23 2008, 17:54:01) > > [GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2 > > Type "help", "copyright", "credits" or "license" for more information. > > > > > PyUnicodeUCS2_FromUnicode > > numpy was build with a python configured with ucs2 while the python you > have is build with ucs4. > > >>>> import Bio > >>>> import sys* > > I am not sure where is trick is. As I checked the previous discussion, I > > found several people raised similar issue but no one has posted a final > > solution to this yet. So I can only ask for help here again! Thanks in > > advance! > > To fix this build python with ucs2, i.e. "check configure --help" and > set python to use ucs2. > > > Charlie > > > > Cheers, > > Michael > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From myeates at jpl.nasa.gov Mon Apr 13 17:59:20 2009 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Mon, 13 Apr 2009 14:59:20 -0700 Subject: [Numpy-discussion] polyfit on multiple data points Message-ID: <49E3B5B8.1060702@jpl.nasa.gov> Hi, I understand how to fit the points (x1,y1) (x2,y2),(x3,y3) with a line using polyfit. But, what if I want to perform this task on every row of an array? For instance [[x1,x2,x3], [s1,s2,s3]] [[y1,y2,y3,], [r1,r2,r3]] and I want the results to be the coefficients [a,b,c] and [d,e,f] where [a,b,c] fits the points (x1,y1) (x2,y2),(x3,y3) and [d,e,f] fits the points (s1,r1) (s2,r2),(s3,r3) I realize I could use "apply_along_axis" but I'm afraid of the performance penalty. Is there a way to do this without resorting to a function call for each row? Mathew From brennan.williams at visualreservoir.com Mon Apr 13 20:28:42 2009 From: brennan.williams at visualreservoir.com (Brennan Williams) Date: Tue, 14 Apr 2009 12:28:42 +1200 Subject: [Numpy-discussion] recommendations on goodness of fit functions? Message-ID: <49E3D8BA.7060505@visualreservoir.com> Hi numpy/scipy users, I'm looking to add some basic goodness-of-fit functions/plots to my app. I have a set of simulated y vs time data and a set of observed y vs time data. The time values aren't always the same, i.e. there are often fewer observed data points. Some variables will be in a 0.0...1.0 range, others in a 0.0.....1.0e+12 range. I'm also hoping to update the calculated goodness of fit value at each simulated timestep, the idea being to allow the user to set a tolerance level which if exceeded stops the simulation (which otherwise can keep running for many hours/days). Thanks Brennan From charlesr.harris at gmail.com Mon Apr 13 21:01:09 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 13 Apr 2009 19:01:09 -0600 Subject: [Numpy-discussion] recommendations on goodness of fit functions? In-Reply-To: <49E3D8BA.7060505@visualreservoir.com> References: <49E3D8BA.7060505@visualreservoir.com> Message-ID: On Mon, Apr 13, 2009 at 6:28 PM, Brennan Williams < brennan.williams at visualreservoir.com> wrote: > Hi numpy/scipy users, > > I'm looking to add some basic goodness-of-fit functions/plots to my app. > > I have a set of simulated y vs time data and a set of observed y vs time > data. > > The time values aren't always the same, i.e. there are often fewer > observed data points. > > Some variables will be in a 0.0...1.0 range, others in a 0.0.....1.0e+12 > range. > > I'm also hoping to update the calculated goodness of fit value at each > simulated timestep, the idea being to allow the user to set a tolerance > level which if exceeded stops the simulation (which otherwise can keep > running for many hours/days). > Some questions. 1) What kind of fit are you doing? 2) What is the measurement model? 3) What do you know apriori about the measurement errors? 4) How is the time series modeled? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From chaos.proton at gmail.com Mon Apr 13 21:40:11 2009 From: chaos.proton at gmail.com (Grissiom) Date: Tue, 14 Apr 2009 09:40:11 +0800 Subject: [Numpy-discussion] dimension along axis? Message-ID: Hi all, It there a convenience way to get dimension along an axis? Say I have two ndarray: li1 = np.array([2,3,4]) li2 = np.array([[2,3,4],[5,6,7]]) I know my list is in C order so the two array is the same in someway. But li1.shape will give (3, ) and li2.shape will give (2,3). "3" appear in different position so it's inconvenient to identify them. Is there anyway to get dimension along axis? (In this case should be axis0) -- Cheers, Grissiom -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Apr 13 21:47:37 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 13 Apr 2009 19:47:37 -0600 Subject: [Numpy-discussion] dimension along axis? In-Reply-To: References: Message-ID: On Mon, Apr 13, 2009 at 7:40 PM, Grissiom wrote: > Hi all, > > It there a convenience way to get dimension along an axis? Say I have two > ndarray: > > li1 = np.array([2,3,4]) > li2 = np.array([[2,3,4],[5,6,7]]) > > I know my list is in C order so the two array is the same in someway. But > li1.shape will give (3, ) and li2.shape will give (2,3). "3" appear in > different position so it's inconvenient to identify them. Is there anyway to > get dimension along axis? (In this case should be axis0) > You mean something like this? In [1]: li1 = np.array([2,3,4]) In [2]: li1[np.newaxis,:].shape Out[2]: (1, 3) Or maybe like this? In [3]: li1 = np.array([[2,3,4]]) In [4]: li1.shape Out[4]: (1, 3) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From chaos.proton at gmail.com Mon Apr 13 21:50:15 2009 From: chaos.proton at gmail.com (Grissiom) Date: Tue, 14 Apr 2009 09:50:15 +0800 Subject: [Numpy-discussion] dimension along axis? In-Reply-To: References: Message-ID: On Tue, Apr 14, 2009 at 09:47, Charles R Harris wrote: > You mean something like this? > > In [1]: li1 = np.array([2,3,4]) > > In [2]: li1[np.newaxis,:].shape > Out[2]: (1, 3) > > Or maybe like this? > > In [3]: li1 = np.array([[2,3,4]]) > > In [4]: li1.shape > Out[4]: (1, 3) > > Chuck > > This is exactly what I want. Thanks ;) -- Cheers, Grissiom -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Apr 13 22:06:45 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 13 Apr 2009 22:06:45 -0400 Subject: [Numpy-discussion] polyfit on multiple data points In-Reply-To: <49E3B5B8.1060702@jpl.nasa.gov> References: <49E3B5B8.1060702@jpl.nasa.gov> Message-ID: <1cd32cbb0904131906r1f8cb689ie117f08651afdec6@mail.gmail.com> On Mon, Apr 13, 2009 at 5:59 PM, Mathew Yeates wrote: > Hi, > I understand how to fit ?the points (x1,y1) (x2,y2),(x3,y3) with a line > using polyfit. But, what if I want to perform this task on every row of > an array? > For instance > > [[x1,x2,x3], > ?[s1,s2,s3]] > > [[y1,y2,y3,], > ?[r1,r2,r3]] > > and I want the results to be the coefficients ?[a,b,c] ?and [d,e,f] where > [a,b,c] fits the points (x1,y1) (x2,y2),(x3,y3) and > [d,e,f] fits the points (s1,r1) (s2,r2),(s3,r3) > > I realize I could use "apply_along_axis" but I'm afraid of the > performance penalty. Is there a way to do this without resorting to a > function call for each row? > > Mathew Which order of polynomial are you fitting to each 3 points? If you want to have independent polynomials, the only other way I can see is building a block design matrix, essentially stacking all independent regression into one big least squares problem. The performance will depend on the number of points you have, but I doubt it will be faster to build the design matrix and solve the large least squares problem then the loop. For a low order polynomial, it might be possible to get an explicit solution, easy for linear, I never tried for ols with two regressors, maybe sympy would help. If you really only have 3 points, then with two regressors and a constant, you would already have an exact solution with a second order polynomial, 3 equations in 3 unknowns. In this case, you can write an explicit solution where it might be possible to use only vectorized array manipulation. If you need the loop and you have fixed axis, doing the loop yourself should be faster then "apply_along_axis". I haven't tried any of this myself, but that's where I would be looking, if runtime is more expensive than developer time and you don't need a solution for the general case with more points and higher order polynomials. ------ Rereading your email, I interpret you want a line, 1st order polynomial, and the predicted values of the dependent variable. That's just a linear regression, you can just use the formula for slope and constant, see for example stats.linregress. Something like the following: assuming x, and y is mean corrected array with your 3 points in columns, and rows are different cases, both are (n,3) arrays, then the slope coefficient should be b = (x*y).sum(1) / (x,x).sum(1) constant is something like: a = y.mean(axis=1) - b * x.mean(1) predicted points ypred = a[:,np.newaxis] + x*a[:,np.newaxis] (should be (n,3) array) warning: not tested, written from memory Josef From Chris.Barker at noaa.gov Mon Apr 13 23:43:17 2009 From: Chris.Barker at noaa.gov (Chris.Barker at noaa.gov) Date: Mon, 13 Apr 2009 20:43:17 -0700 Subject: [Numpy-discussion] Leopard install Message-ID: David Cournapeau wrote: > I think you need to install python from python.org (version 2.5) to > install the numpy binary, yes, that's it -- "system Python" is a misnomer. I really should figure out how to change that message. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From brennan.williams at visualreservoir.com Tue Apr 14 00:47:17 2009 From: brennan.williams at visualreservoir.com (Brennan Williams) Date: Tue, 14 Apr 2009 16:47:17 +1200 Subject: [Numpy-discussion] recommendations on goodness of fit functions? In-Reply-To: References: <49E3D8BA.7060505@visualreservoir.com> Message-ID: <49E41555.3040204@visualreservoir.com> Charles R Harris wrote: > > > On Mon, Apr 13, 2009 at 6:28 PM, Brennan Williams > > wrote: > > Hi numpy/scipy users, > > I'm looking to add some basic goodness-of-fit functions/plots to > my app. > > I have a set of simulated y vs time data and a set of observed y > vs time > data. > > The time values aren't always the same, i.e. there are often fewer > observed data points. > > Some variables will be in a 0.0...1.0 range, others in a > 0.0.....1.0e+12 > range. > > I'm also hoping to update the calculated goodness of fit value at each > simulated timestep, the idea being to allow the user to set a > tolerance > level which if exceeded stops the simulation (which otherwise can keep > running for many hours/days). > > Before I try and answer the following, attached is an example of a suggested GOF function. > Some questions. > > 1) What kind of fit are you doing? > 2) What is the measurement model? > 3) What do you know apriori about the measurement errors? > 4) How is the time series modeled? > The simulated data are output by a oil reservoir simulator. Time series is typically monthly or annual timesteps over anything from 5-30 years but it could also be in some cases 10 minute timesteps over 24 hours The timesteps output by the simulator are controlled by the user and are not always even, e.g. for a simulation over 30 years you may have annual timesteps from year 0 to year 25 and then 3 monthly from year 26-29 and then monthly for the most recent year. Not sure about measurement errors - the older the data the higher the errors due to changes in oil field measurement technology. And the error range varies depending on the data type as well, e.g. error range for a water meter is likely to be higher than that for an oil or gas meter. I'll try and find out more about that. Brennan > Chuck > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- A non-text attachment was scrubbed... Name: goodnessoffit.png Type: image/png Size: 6395 bytes Desc: not available URL: From david at ar.media.kyoto-u.ac.jp Tue Apr 14 00:58:15 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 14 Apr 2009 13:58:15 +0900 Subject: [Numpy-discussion] Leopard install In-Reply-To: References: Message-ID: <49E417E7.9050201@ar.media.kyoto-u.ac.jp> Chris.Barker at noaa.gov wrote: > David Cournapeau wrote: > >> I think you need to install python from python.org (version 2.5) to >> install the numpy binary, >> > > yes, that's it -- "system Python" is a misnomer. I really should figure > out how to change that message. > It is set-up in the info.plist as built by bdist_mpkg, so I guess changing this file should be enough - this should be easy to test :) Maybe we should do the packaging ourselves instead of relying on the bdist_mpkg command, to control all this better (we could reuse some code from the bdist_mpkg command, of course) ? cheers, David From cournape at gmail.com Tue Apr 14 02:14:05 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 14 Apr 2009 15:14:05 +0900 Subject: [Numpy-discussion] help getting started In-Reply-To: <5b8d13220904102147h3620a9d8lcd7af04ed9269a04@mail.gmail.com> References: <5b8d13220904101110n5f39f7cdwd5a5fb85c1d1c405@mail.gmail.com> <49DFAF23.5030100@noaa.gov> <5b8d13220904102050l4ee0e7fdhb9b3b23d9e21a133@mail.gmail.com> <49E01913.1030805@noaa.gov> <5b8d13220904102147h3620a9d8lcd7af04ed9269a04@mail.gmail.com> Message-ID: <5b8d13220904132314i3a0d4ae5n907e41233ce01f1@mail.gmail.com> On Sat, Apr 11, 2009 at 1:47 PM, David Cournapeau wrote: > > ATM, I am dealing with a third one: scipy.test() crashes right away > when importing scipy.cluster Ok, ended up being a problem with incompatible matplotlib imported by scipy.cluster, and specific to my machine. Anyway, problem solved. Christopher, would you mind trying the following binary ? http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/scipy-0.7.0-py2.5-macosx10.5.mpkg.tar It should solve both universal build and statically linking gfortran issues, David From cournape at gmail.com Tue Apr 14 02:26:56 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 14 Apr 2009 15:26:56 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <9457e7c80904130812s55697f2bp1239de8428733d35@mail.gmail.com> References: <49DEDCE1.7050201@ar.media.kyoto-u.ac.jp> <49E00641.5050903@hawaii.edu> <5b8d13220904102005y21503770r32ac48c97c412849@mail.gmail.com> <20090411095250.GB8496@phare.normalesup.org> <9457e7c80904111857x28cf93edm14f0e8d2c924410b@mail.gmail.com> <9457e7c80904111936o7b4d37dcuc0342c35238cf748@mail.gmail.com> <9457e7c80904120154x7973a5f4v993f0d3d0419fa7f@mail.gmail.com> <9457e7c80904130812s55697f2bp1239de8428733d35@mail.gmail.com> Message-ID: <5b8d13220904132326v6ee0675cy700ee98f2d66a29c@mail.gmail.com> 2009/4/14 St?fan van der Walt : > 2009/4/12 St?fan van der Walt : >> I underestimated the >> value of this type of manipulation, and of having a clearly structured >> and easily traversable history. > > I read that Bram Cohen of Codeville / "patience diff" fame doesn't > agree with me, so I'll give his opinion too: > > """ > Don't bother with a pretty history. > > The history of a branch is hardly ever looked at. Making it look > pretty for the historians is just a waste of time. The beauty of 3-way > merge is that you can always clean stuff up later and never worry > about the past mess ever again. In particular, don't go to great > lengths to make sure that there's a coherent local image of the entire > repository exactly as it appeared on your local machine after every > new feature. There are very rare projects which maintain a level of > reliability and testing which warrant such behavior, and yours isn't > one of them. > """ > > http://bramcohen.livejournal.com/52148.html > > I look at the SciPy history a lot, so I'm not convinced, but there it is. I think it depends on the context. In open source software, the history is quite important - when I digged into distutils, I wished svn were better at this to understand where some code came from (the information is almost always lost when changes cross files). In a company, or more exactly when people working on the code are physically near from each other, you (hopefully) have people who know a lot of the 'black art' behind things, and who are available to transmit this. Concerning branches, he said he had no experience in code review. So it is to be taken with some perspective. I think ultimately, only experience can really tell which things are better than others on those issues. cheers, David From cournape at gmail.com Tue Apr 14 02:34:25 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 14 Apr 2009 15:34:25 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <49E339D9.5050801@gmx.net> References: <49CE3A2C.9000007@enthought.com> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <85b5c3130904100154l797d062fn90ea9389bc2da0b@mail.gmail.com> <878wm5m13w.fsf@hbox.dyndns.org> <49E2F35A.2010704@ar.media.kyoto-u.ac.jp> <87y6u4lxgy.fsf@hbox.dyndns.org> <5b8d13220904130603p4d746811qc47a8eee00706c9d@mail.gmail.com> <49E339D9.5050801@gmx.net> Message-ID: <5b8d13220904132334k33eb96d0q357ed19ea2fa1d25@mail.gmail.com> On Mon, Apr 13, 2009 at 10:10 PM, David Soria Parra wrote: > > You can have multiple heads in a repository and if you created a bookmark > on the remote site you can refer to it by using ssh://url#bookmarkname . > It is not possible to get a list of bookmarks at the moment, nor is it > possible to push them. I would recommend to use one repository per > branch until bookmarks are pushable which we hope to happen in either hg > 1.3 or 1.4. Ok. So do I understand right that bookmark will be integrated in hg proper for 1.3/1.4 ? Reading a bit the hg ML to know more about some hg issues I am wondering about, I also came across tasks, which look neared to git branches ? David From cournape at gmail.com Tue Apr 14 02:39:54 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 14 Apr 2009 15:39:54 +0900 Subject: [Numpy-discussion] DVCS at PyCon In-Reply-To: <87ocv0lki7.fsf@hbox.dyndns.org> References: <49CE3A2C.9000007@enthought.com> <85b5c3130904092314o7512cdbfjd2f43b98e65f48cf@mail.gmail.com> <49DEF89D.7000605@hawaii.edu> <49DEFE52.6060408@ar.media.kyoto-u.ac.jp> <85b5c3130904100154l797d062fn90ea9389bc2da0b@mail.gmail.com> <878wm5m13w.fsf@hbox.dyndns.org> <49E2F35A.2010704@ar.media.kyoto-u.ac.jp> <87y6u4lxgy.fsf@hbox.dyndns.org> <5b8d13220904130603p4d746811qc47a8eee00706c9d@mail.gmail.com> <87ocv0lki7.fsf@hbox.dyndns.org> Message-ID: <5b8d13220904132339n4663d6aaidfcd23513137cb59@mail.gmail.com> On Mon, Apr 13, 2009 at 11:02 PM, Martin Geisler wrote: > You're right, the UI is not so good. In particular, you cannot use 'hg > diff' to compare repositories. The rdiff extension does this, though: > > ?http://www.selenic.com/mercurial/wiki/index.cgi/RdiffExtension In git, it is not just about diff, but about every command where you would want to compare branches. From a UI POV, this vastly outweighs the possible confusion caused by multi-branches / repo ala git. This means everything is done within the same concept of a branch (review, tasks, release branches, task branches, patches). > > Using stock Mercurial you can use 'hg incoming -p' to see the patches of > the incoming (missing) changesets, or you can pull in the changesets and > then look at them -- if you don't like them you can remove them again. Ok, I would have to try this to get a better idea as well. > > Or you could use named branches for the numpy-1.2.x and numpy-1.3.x > branches -- then everybody would see the branch names when they clone > the combined repository. Hm, ok. For release branches, named branches is the obvious choice - it is very unlikely that you would need to delete them. David From sebastian.walter at gmail.com Tue Apr 14 04:16:55 2009 From: sebastian.walter at gmail.com (Sebastian Walter) Date: Tue, 14 Apr 2009 10:16:55 +0200 Subject: [Numpy-discussion] survey of freely available software for the solution of linear algebra problems In-Reply-To: References: Message-ID: I'm looking for a library that has linear algebra routines on objects of a self-written class (with overloaded operators +,-,*,/). An example would be truncated Taylor polynomials. One can perform all operations +,-,*,/ on truncated Taylor polynomials. Often it is also necessary to perform linear algebra routines on matrices where the elements are such truncated Taylor polynomials. Does Eigen2 support such custom datatypes? On Mon, Apr 13, 2009 at 9:58 PM, Russell E. Owen wrote: > In article , > "Nils Wagner" wrote: > >> http://www.netlib.org/utk/people/JackDongarra/la-sw.html > > You might add Eigen: > > We are finding it to be a very nice package (though the name is > unfortunate from the perspective of internet search engines). It is a > pure C++ template library, which is brilliant. That makes it much easier > to build a package using Eigen than one using lapack/blas/etc.. > > -- Russell > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From Tobias.Lindberg at byggvetenskaper.lth.se Tue Apr 14 04:28:41 2009 From: Tobias.Lindberg at byggvetenskaper.lth.se (Tobias Lindberg) Date: Tue, 14 Apr 2009 10:28:41 +0200 Subject: [Numpy-discussion] passing dynamic arrays from fortran 90 to python Message-ID: Hi, Is there a way to pass dynamic arrays generated in a fortran 90- module to a python script? Best regards /tobias -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Tue Apr 14 08:11:35 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 14 Apr 2009 08:11:35 -0400 Subject: [Numpy-discussion] [OT] read data from pdf References: <20090413120246.GD13133@phare.normalesup.org> Message-ID: Gael Varoquaux wrote: > On Mon, Apr 13, 2009 at 07:10:40AM -0400, Neal Becker wrote: >> Anyone know of software that can assist with reading data points from a >> pdf version of a 2-d line graph? > > I know domeone who had a lot of success with engauge-digitizer (packaged > in Ubuntu). > > Ga?l Thanks! I tried this one (engauge-digitizer), although I'm not really familiar with it, it seemed to do the job! From bsouthey at gmail.com Tue Apr 14 10:13:51 2009 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 14 Apr 2009 09:13:51 -0500 Subject: [Numpy-discussion] recommendations on goodness of fit functions? In-Reply-To: <49E41555.3040204@visualreservoir.com> References: <49E3D8BA.7060505@visualreservoir.com> <49E41555.3040204@visualreservoir.com> Message-ID: <49E49A1F.4000302@gmail.com> Brennan Williams wrote: > Charles R Harris wrote: >> >> >> On Mon, Apr 13, 2009 at 6:28 PM, Brennan Williams >> > > wrote: >> >> Hi numpy/scipy users, >> >> I'm looking to add some basic goodness-of-fit functions/plots to >> my app. >> >> I have a set of simulated y vs time data and a set of observed y >> vs time >> data. >> >> The time values aren't always the same, i.e. there are often fewer >> observed data points. >> >> Some variables will be in a 0.0...1.0 range, others in a >> 0.0.....1.0e+12 >> range. Ignoring the time scale, if time is linear with respect to the parameters then you should be okay. If these cause numerical issues, you probably want to scale or standardize the time scale first. If your model has polynomials then you probably want to select an orthogonal type that has good properties for what you want. Otherwise, you make either transform the data or fit an appropriate non-linear model. >> >> I'm also hoping to update the calculated goodness of fit value at >> each >> simulated timestep, the idea being to allow the user to set a >> tolerance >> level which if exceeded stops the simulation (which otherwise can >> keep >> running for many hours/days). Exactly how does this work? I would have thought that you have simulated model that has set parameters and simulates the data based on the inputs that include a time range. For example, if I have a simple linear model Y=Xb, I would estimate parameters b and then allow the user to provide their X so it should stop relatively quickly depending on the model complexity and dimensions of X. It appears like you are doing some search over some values to maximize the parameters b or some type of 'what if' or sensitivity scenarios. In the first case you probably should use one of the optimization algorithms in scipy. In the second case, the simulation would stop when it exceeds the parameter space. >> >> > Before I try and answer the following, attached is an example of a > suggested GOF function. > >> Some questions. >> >> 1) What kind of fit are you doing? >> 2) What is the measurement model? >> 3) What do you know apriori about the measurement errors? >> 4) How is the time series modeled? >> > The simulated data are output by a oil reservoir simulator. So much depends on the simulator model - for simple linear models assuming normality you could get away with R2 or mean square error or mean absolute deviance. But that really doesn't cut it with complex models. If you are trying different models, then you should look at model comparison techniques. > > Time series is typically monthly or annual timesteps over anything > from 5-30 years > but it could also be in some cases 10 minute timesteps over 24 hours > > The timesteps output by the simulator are controlled by the user and > are not always even, e.g. for a simulation over 30 years you may > have annual timesteps from year 0 to year 25 and then 3 monthly from > year 26-29 and then monthly for the most recent year. If the data is not measured on the same regular interval across the complete period (say every month) you have a potential problem of selecting the correct time scale and making suitable assumptions for missing data points (like assuming that the value from one year is equal to the value at 1/4 of a year). If you can not make suitable assumptions, you probably can not mix the different time scales so you probably need a piece-wise solution to handle the different periods. > > Not sure about measurement errors - the older the data the higher the > errors due to changes in oil field measurement technology. > And the error range varies depending on the data type as well, e.g. > error range for a water meter is likely to be higher than that for an > oil or gas meter. > This implies heterogeneity of variance over time which further complicates things. But again this is something that should be addressed when creating the simulator. Bruce From myeates at jpl.nasa.gov Tue Apr 14 18:16:07 2009 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Tue, 14 Apr 2009 15:16:07 -0700 Subject: [Numpy-discussion] problem using optimized libraries Message-ID: <49E50B27.1020008@jpl.nasa.gov> Hi The line from _dotblas import dot ..... is giving me an import error when I looked at the symbols in _dotblas.so I only see things like CFLOAT_dot. When I trace the system calls I see that my optimized ATLAS libraries are being accessed, but immediately after opening libatlas.so I get an Import Error resulting in an unoptimized "dot" Anyone have any ideas? Mathew From Chris.Barker at noaa.gov Tue Apr 14 22:31:21 2009 From: Chris.Barker at noaa.gov (Chris.Barker at noaa.gov) Date: Tue, 14 Apr 2009 19:31:21 -0700 Subject: [Numpy-discussion] Leopard install Message-ID: <9fa2d0e339b64be8.49e4e489@noaa.gov> David Cournapeau wrote: >> yes, that's it -- "system Python" is a misnomer. I really should figure >> out how to change that message. >> > It is set-up in the info.plist as built by bdist_mpkg, so I guess > changing this file should be enough - this should be easy to test :) yup. It's pretty easy to find where it gets put in the bdist_mpkg code, but I didn't get as far as making a patch and submitting it. I should have. > Maybe we should do the packaging ourselves instead of relying on the > bdist_mpkg command, to control all this better (we could reuse some code > from the bdist_mpkg command, of course) ? I think I'd prefer to patch bdist_mpkg, rather than re-writing or forking. Others need it too. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Tue Apr 14 22:31:27 2009 From: Chris.Barker at noaa.gov (Chris.Barker at noaa.gov) Date: Tue, 14 Apr 2009 19:31:27 -0700 Subject: [Numpy-discussion] help getting started Message-ID: David Cournapeau wrote: > Christopher, would you mind trying the following binary ? > > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/scipy-0.7.0-py2.5-macosx10.5.mpkg.tar I'm out of town and away from my Mac the whole week. Hopefully someone else can give it a try. Thanks for working on it, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From josef.pktd at gmail.com Tue Apr 14 23:02:08 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 14 Apr 2009 23:02:08 -0400 Subject: [Numpy-discussion] recommendations on goodness of fit functions? In-Reply-To: <49E49A1F.4000302@gmail.com> References: <49E3D8BA.7060505@visualreservoir.com> <49E41555.3040204@visualreservoir.com> <49E49A1F.4000302@gmail.com> Message-ID: <1cd32cbb0904142002n38c39b9fvca508249518ded70@mail.gmail.com> On Tue, Apr 14, 2009 at 10:13 AM, Bruce Southey wrote: > Brennan Williams wrote: >> Charles R Harris wrote: >>> >>> >>> On Mon, Apr 13, 2009 at 6:28 PM, Brennan Williams >>> >> > wrote: >>> >>> ? ? Hi numpy/scipy users, >>> >>> ? ? I'm looking to add some basic goodness-of-fit functions/plots to >>> ? ? my app. >>> >>> ? ? I have a set of simulated y vs time data and a set of observed y >>> ? ? vs time >>> ? ? data. >>> >>> ? ? The time values aren't always the same, i.e. there are often fewer >>> ? ? observed data points. >>> >>> ? ? Some variables will be in a 0.0...1.0 range, others in a >>> ? ? 0.0.....1.0e+12 >>> ? ? range. > Ignoring the time scale, if time is linear with respect to the > parameters then you should be okay. If these cause numerical issues, you > probably want to scale or standardize the time scale first. If your > model has polynomials then you probably want to select an orthogonal > type that has good properties for what you want. Otherwise, you make > either transform the data or fit an appropriate non-linear model. > >>> >>> ? ? I'm also hoping to update the calculated goodness of fit value at >>> each >>> ? ? simulated timestep, the idea being to allow the user to ?set a >>> ? ? tolerance >>> ? ? level which if exceeded stops the simulation (which otherwise can >>> keep >>> ? ? running for many hours/days). > Exactly how does this work? > > I would have thought that you have simulated model that has set > parameters and simulates the data based on the inputs that include a > time range. For example, if I have a simple linear model Y=Xb, I would > estimate parameters b and then allow the user to provide their X so it > should stop relatively quickly depending on the model complexity and > dimensions of X. > > It appears like you are doing some search over some values to maximize > the parameters b or some type of 'what if' or sensitivity scenarios. In > the first case you probably should use one of the optimization > algorithms in scipy. In the second case, the simulation would stop when > it exceeds the parameter space. > >>> >>> >> Before I try and answer the following, attached is an example of a >> suggested GOF function. >> >>> Some questions. >>> >>> 1) What kind of fit are you doing? >>> 2) What is the measurement model? >>> 3) What do you know apriori about the measurement errors? >>> 4) How is the time series modeled? >>> >> The simulated data are output by a oil reservoir simulator. > So much depends on the simulator model - for simple linear models > assuming normality you could get away with R2 or mean square error or > mean absolute deviance. But that really doesn't cut it with complex > models. If you are trying different models, then you should look at > model comparison techniques. >> >> Time series is typically monthly or annual timesteps over anything >> from 5-30 years >> but it could also be in some cases 10 minute timesteps over 24 hours >> >> The timesteps output by the simulator are controlled by the user and >> are not always even, e.g. for a simulation over 30 years you may >> have annual timesteps from year 0 to year 25 and then 3 monthly from >> year 26-29 and then monthly for the most recent year. > If the data is not measured on the same regular interval across the > complete period (say every month) you have a potential problem of > selecting the correct time scale and making suitable assumptions for > missing data points (like assuming that the value from one year is equal > to the value at 1/4 of a year). If you can not make suitable > assumptions, you probably can not mix the different time scales so you > probably need a piece-wise solution to handle the different periods. > >> >> Not sure about measurement errors - the older the data the higher the >> errors due to changes in oil field measurement technology. >> And the error range varies depending on the data type as well, e.g. >> error range for a water meter is likely to be higher than that for an >> oil or gas meter. >> > This implies heterogeneity of variance over time which further > complicates things. But again this is something that should be addressed > when creating the simulator. Just some ideas (assuming you have the simulator as black box): * I would consider the difference between simulation results and data as a univariate time series and try to estimate the simulation error variance, either moving, depreciated or conditional, using some approach from this literature. * scikits time series has some statistical window functions, with either moving window or geometric depreciation, which might be useful either directly or as examples * In your goodness of fit formula, I would use p=1, i.e. mean absolute deviation, if you have some measurement or prediction outliers or fat tails, and p=2, i.e. RMSE, if the distribution of simulation errors look more like a normal distribution. * unequal spaced observation time points: I would start with a high frequency model (the highest frequency for that simulation e.g. monthly) with intertemporal correlation of the simulation error, and then derive the correlation between two observed time points for weighting the variance estimate. There is quite a literature on irregularly observed time series, with which I'm not very familiar. Often it starts with an assumption on an underlying continuous time model with a selection process for the time periods at which the process is observed. In your case, when you don't have a large number of observation, I would just start with a simple AR(1) and calculate the prediction confidence interval as goodness of fit. (The problem is that this assumes stationarity and it wouldn't get any depreciation in the information. If there are large deviations from stationarity, then a messier model with, for example, conditional or time varying variance or mean would be necessary.) * to pull the plug for simulations that go off the map: this could also be based on some tests of stationarity, e.g. a simple test whether there was a structural break in the mean 5 observations ago. With only a few observations the power might not be very good and other tests like unit root test won't have enough power, but it would kill bad simulations. (I usually get complex numbers for prices and so on, when I screw up in the simulation.) * if the physical measurement errors are small compared to the simulation errors, I would just ignore them. If you have significant changes over time, then extra information about this is required, the s_i in your goodness of fit formula. It all depends on how sophisticated you want to get in this. In finance, where investor bet large amounts of money on this, the statistical models for this can be very complex, but maybe you can get away just with the RMSE or prediction confidence interval. Josef From jh at physics.ucf.edu Wed Apr 15 00:56:31 2009 From: jh at physics.ucf.edu (Joe Harrington) Date: Wed, 15 Apr 2009 00:56:31 -0400 Subject: [Numpy-discussion] JOB: write numpy docs In-Reply-To: (message from Joe Harrington on Mon, 30 Mar 2009 12:26:39 -0400) References: Message-ID: So far I have no takers for the numpy doc job (below), and the lack of activity on docs.scipy.org indicates nobody new is preparing an application. This job is open (as a subcontract) to international applicants. It could be part-time, if a suitable full-time person is not available. It could be a volunteer job, if you're gainfully employed and would prefer it that way. If you have substantial free time this summer and you want to have full docs for numpy, please contact me. Thanks, --jh-- Date: Mon, 30 Mar 2009 12:26:39 -0400 To: Scipy Developers , Scipy Users , Numpy Discussion Subject: JOB: write numpy docs From: Joe Harrington CC: jh at physics.ucf.edu Reply-to: jh at physics.ucf.edu Last year's Doc Marathon got us off to a great start on documenting NumPy! But, there's still much work to be done, and SciPy after that. It's time to gear up for doing it again. Critical to last year's success was Stefan van der Walt's committed time, but he will be unable to play that role this year. So, I am looking to hire someone to write NumPy docs and help coordinate the doc project and its volunteers. The job includes working with me, the doc team, doc volunteers, and developers to: write and review a lot of docs, mainly those that others don't want to write help define milestones organize campaigns and volunteer teams to meet them research the NumPy and SciPy source codes to help plan: the eventual SciPy documentation the writing of a good User Manual work with the packaging team to meet their release deadlines perform other duties as assigned I am seeking someone to work full time if possible, and at least half time, from mid-April (or soon thereafter) through at least the (northern) summer. Candidates must be experienced NumPy and SciPy programmers; familiarity under the hood is a strong plus. They must also demonstrate their ability to produce excellent docs on the docs.SciPy.org wiki. Having contributed at a high level to an open-source community, especially to SciPy, is a big plus. Ability to take direction, work with and lead a team, and to work for extended periods without direct supervision on a list of assigned tasks are all critical. The applicant must be able to function well in a Linux environment; familiarity with multiple platforms is a plus. Please reply directly to me by email only. Include the following (PDF or ASCII formats strongly preferred): CV Statement of interest, qualifications per requirements above, availability, and wage expectations. Contact info for at least 3 professional references. Links to doc wiki pages for which you wrote the initial draft Links to doc wiki pages started by others to which you contributed significantly (edited, reviewed, proofed) The position is open until filled; candidates with complete applications by April 15 will receive full consideration. This is an open posting. Candidates who have not written any pages on the doc wiki yet have several weeks in which to do so. Pay will be commensurate with experience (up to a point). Relocation is not necessary. Candidates will need to provide their own computer and internet access. The University of Central Florida is an equal opportunity, equal access, affirmative action employer. --jh-- Prof. Joseph Harrington Department of Physics MAP 414 4000 Central Florida Blvd. University of Central Florida Orlando, FL 32816-2385 (407) 823-3416 voice (407) 823-5112 fax (407) 823-2325 physics office jh at physics.ucf.edu From gael.varoquaux at normalesup.org Wed Apr 15 01:19:23 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 15 Apr 2009 07:19:23 +0200 Subject: [Numpy-discussion] Leopard install In-Reply-To: <9fa2d0e339b64be8.49e4e489@noaa.gov> References: <9fa2d0e339b64be8.49e4e489@noaa.gov> Message-ID: <20090415051923.GD28627@phare.normalesup.org> On Tue, Apr 14, 2009 at 07:31:21PM -0700, Chris.Barker at noaa.gov wrote: > I think I'd prefer to patch bdist_mpkg, rather than re-writing or > forking. Others need it too. Tarek Ziade, one of the distutils maintainer, has been fairly responsive lately. And I know he runs a Mac. So I believe that if you write good code, you'll get a patch in (one advice: he loves tests). Ga?l From robert.kern at gmail.com Wed Apr 15 01:22:07 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 15 Apr 2009 00:22:07 -0500 Subject: [Numpy-discussion] Leopard install In-Reply-To: <20090415051923.GD28627@phare.normalesup.org> References: <9fa2d0e339b64be8.49e4e489@noaa.gov> <20090415051923.GD28627@phare.normalesup.org> Message-ID: <3d375d730904142222r58f1f235i58def1c0b05461dd@mail.gmail.com> On Wed, Apr 15, 2009 at 00:19, Gael Varoquaux wrote: > On Tue, Apr 14, 2009 at 07:31:21PM -0700, Chris.Barker at noaa.gov wrote: >> I ?think I'd prefer to patch bdist_mpkg, rather than re-writing or >> forking. Others need it too. > > Tarek Ziade, one of the distutils maintainer, has been fairly responsive > lately. And I know he runs a Mac. So I believe that if you write good > code, you'll get a patch in (one advice: he loves tests). bdist_mpkg is a separate project. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From gael.varoquaux at normalesup.org Wed Apr 15 01:37:32 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 15 Apr 2009 07:37:32 +0200 Subject: [Numpy-discussion] Leopard install In-Reply-To: <3d375d730904142222r58f1f235i58def1c0b05461dd@mail.gmail.com> References: <9fa2d0e339b64be8.49e4e489@noaa.gov> <20090415051923.GD28627@phare.normalesup.org> <3d375d730904142222r58f1f235i58def1c0b05461dd@mail.gmail.com> Message-ID: <20090415053731.GF28627@phare.normalesup.org> On Wed, Apr 15, 2009 at 12:22:07AM -0500, Robert Kern wrote: > On Wed, Apr 15, 2009 at 00:19, Gael Varoquaux > wrote: > > On Tue, Apr 14, 2009 at 07:31:21PM -0700, Chris.Barker at noaa.gov wrote: > >> I ?think I'd prefer to patch bdist_mpkg, rather than re-writing or > >> forking. Others need it too. > > Tarek Ziade, one of the distutils maintainer, has been fairly responsive > > lately. And I know he runs a Mac. So I believe that if you write good > > code, you'll get a patch in (one advice: he loves tests). > bdist_mpkg is a separate project. Thanks for correcting me. G. From david at ar.media.kyoto-u.ac.jp Wed Apr 15 02:10:20 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 15 Apr 2009 15:10:20 +0900 Subject: [Numpy-discussion] Leopard install In-Reply-To: <9fa2d0e339b64be8.49e4e489@noaa.gov> References: <9fa2d0e339b64be8.49e4e489@noaa.gov> Message-ID: <49E57A4C.1060107@ar.media.kyoto-u.ac.jp> Chris.Barker at noaa.gov wrote: > I think I'd prefer to patch bdist_mpkg, rather than re-writing or > forking. > If you manage to do what you want as a command, then yes, forking does not make any sense. It is just that I am tired of wasting my time with distutils-related codebase :) cheers, David From rpyle at post.harvard.edu Wed Apr 15 11:50:29 2009 From: rpyle at post.harvard.edu (Robert Pyle) Date: Wed, 15 Apr 2009 11:50:29 -0400 Subject: [Numpy-discussion] help getting started In-Reply-To: References: Message-ID: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> On Apr 14, 2009, at 10:31 PM, Chris.Barker at noaa.gov wrote: > David Cournapeau wrote: >> Christopher, would you mind trying the following binary ? >> >> http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/scipy-0.7.0-py2.5-macosx10.5.mpkg.tar > > I'm out of town and away from my Mac the whole week. Hopefully someone > else can give it a try. Now that the tax return is (almost) done.... The mpkg installed without a hitch. I ran scipy.test(). Lots of warning messages; I don't know what is "normal", so I'm attaching a text file with the results of scipy.test(). Bob -------------- next part -------------- A non-text attachment was scrubbed... Name: scipy_test_results Type: application/octet-stream Size: 10494 bytes Desc: not available URL: -------------- next part -------------- From Fadhley.Salim at uk.calyon.com Wed Apr 15 12:17:45 2009 From: Fadhley.Salim at uk.calyon.com (Fadhley Salim) Date: Wed, 15 Apr 2009 17:17:45 +0100 Subject: [Numpy-discussion] Compiling for free on Windows32 References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> Message-ID: <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> I've been asked to provide Numpy & Scipy as python egg files. Unfortunately Numpy and Scipy do not make official releases of their product in .egg form for a Win32 platform - that means if I want eggs then I have to compile them myself. At the moment my employer provides Visual Studio.Net 2003, which will compile no version of Numpy later than 1.1.1 - every version released subsequently cannot be compiled with VS2003. I've tried. What I'd really like is some other compiler I can use, perhaps for free, but at a push as a free time-limited trial. I would use that to compile the eggs. Is anybody aware of another compiler that I can download and use without paying anything and will definitely compile Numpy on Windows? So in summary, what I need is: * A compiler for Windows * Which will definitely compile numpy 1.3.0 * Which I can use for free (legally) Please only suggest something if you know for a fact that it works - no speculation! Thanks! -------------- next part -------------- This email does not create a legal relationship between any member of the Cr=E9dit Agricole group and the recipient or constitute investment advice. The content of this email should not be copied or disclosed (in whole or part) to any other person. It may contain information which is confidential, privileged or otherwise protected from disclosure. If you are not the intended recipient, you should notify us and delete it from your system. Emails may be monitored, are not secure and may be amended, destroyed or contain viruses and in communicating with us such conditions are accepted. Any content which does not relate to business matters is not endorsed by us. Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et des Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. Calyon is incorporated in France with limited liability and registered in England & Wales. Registration number: FC008194. Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. This message and/or any attachments is intended for the sole use of its addressee. If you are not the addressee, please immediately notify the sender and then destroy the message. As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. All rights reserved. Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. Tous droits r?serv?s. From matthieu.brucher at gmail.com Wed Apr 15 12:44:42 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 15 Apr 2009 18:44:42 +0200 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> Message-ID: Hi, There is a Python limitation for the compiler. if you have 2.5 or less, you're stuck with VS2003, if you have 2.6, you can use 2008 (or 2005, I don't remember). If you use numscons, you can go with any of them (at your own risks). You can also use mingw. For scipy, you have to use mingw, because there is no other free Fortran compiler. Matthieu 2009/4/15 Fadhley Salim : > I've been asked to provide Numpy & Scipy as python egg files. > Unfortunately Numpy and Scipy do not make official releases of their > product in .egg form for a Win32 platform - that means if I want eggs > then I have to compile them myself. > > At the moment my employer provides Visual Studio.Net 2003, which will > compile no version of Numpy later than 1.1.1 - every version released > subsequently cannot be compiled with VS2003. I've tried. > > What I'd really like is some other compiler I can use, perhaps for free, > but at a push as a free time-limited trial. I would use that to compile > the eggs. Is anybody aware of another compiler that I can download and > use without paying anything and will definitely compile Numpy on > Windows? > > So in summary, what I need is: > > * A compiler for Windows > * Which will definitely compile numpy 1.3.0 > * Which I can use for free (legally) > > Please only suggest something if you know for a fact that it works - no > speculation! > > Thanks! > > This email does not create a legal relationship between any member of the > Cr=E9dit Agricole group and the recipient or constitute investment advice. > The content of this email should not be copied or disclosed (in whole or > part) to any other person. It may contain information which is > confidential, privileged or otherwise protected from disclosure. If you > are not the intended recipient, you should notify us and delete it from > your system. Emails may be monitored, are not secure and may be amended, > destroyed or contain viruses and in communicating with us such conditions > are accepted. Any content which does not relate to business matters is not > endorsed by us. > > Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et des > Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire > in France and subject to limited regulation by the Financial Services Authority. > Details about the extent of our regulation by the Financial Services Authority > are available from us on request. Calyon is incorporated in France with limited > liability and registered in England & Wales. Registration number: FC008194. > Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. > This message and/or any ?attachments is intended for the sole use of its addressee. > If you are not the addressee, please immediately notify the sender and then destroy the message. > As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. > All rights reserved. > > Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. > Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. > Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. > Tous droits r?serv?s. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From Fadhley.Salim at uk.calyon.com Wed Apr 15 12:52:15 2009 From: Fadhley.Salim at uk.calyon.com (Fadhley Salim) Date: Wed, 15 Apr 2009 17:52:15 +0100 Subject: [Numpy-discussion] Compiling for free on Windows32 References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu><7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> Message-ID: <7F347D91614EBC48AA7540A2A76D3BB20448406D@MXCU10MX1.MSX.CIB> I'm compiling for Python 2.4, so I guess that means that I'm never going to be able to use anything more than visual studio 2003? What about compilors from other vendors, for example Intel or Borland? Thanks -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Matthieu Brucher Sent: 15 April 2009 17:45 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Compiling for free on Windows32 Hi, There is a Python limitation for the compiler. if you have 2.5 or less, you're stuck with VS2003, if you have 2.6, you can use 2008 (or 2005, I don't remember). If you use numscons, you can go with any of them (at your own risks). You can also use mingw. For scipy, you have to use mingw, because there is no other free Fortran compiler. Matthieu 2009/4/15 Fadhley Salim : > I've been asked to provide Numpy & Scipy as python egg files. > Unfortunately Numpy and Scipy do not make official releases of their > product in .egg form for a Win32 platform - that means if I want eggs > then I have to compile them myself. > > At the moment my employer provides Visual Studio.Net 2003, which will > compile no version of Numpy later than 1.1.1 - every version released > subsequently cannot be compiled with VS2003. I've tried. > > What I'd really like is some other compiler I can use, perhaps for > free, but at a push as a free time-limited trial. I would use that to > compile the eggs. Is anybody aware of another compiler that I can > download and use without paying anything and will definitely compile > Numpy on Windows? > > So in summary, what I need is: > > * A compiler for Windows > * Which will definitely compile numpy 1.3.0 > * Which I can use for free (legally) > > Please only suggest something if you know for a fact that it works - > no speculation! > > Thanks! > > This email does not create a legal relationship between any member of > the Cr=E9dit Agricole group and the recipient or constitute investment advice. > The content of this email should not be copied or disclosed (in whole > or > part) to any other person. It may contain information which is > confidential, privileged or otherwise protected from disclosure. If > you are not the intended recipient, you should notify us and delete it > from your system. Emails may be monitored, are not secure and may be > amended, destroyed or contain viruses and in communicating with us > such conditions are accepted. Any content which does not relate to > business matters is not endorsed by us. > > Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et > des Entreprises d'Investissement (CECEI) and supervised by the > Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. > Details about the extent of our regulation by the Financial Services > Authority are available from us on request. Calyon is incorporated in > France with limited liability and registered in England & Wales. Registration number: FC008194. > Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. > This message and/or any ?attachments is intended for the sole use of its addressee. > If you are not the addressee, please immediately notify the sender and then destroy the message. > As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. > All rights reserved. > > Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. > Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. > Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. > Tous droits r?serv?s. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion This email does not create a legal relationship between any member of the Cr=E9dit Agricole group and the recipient or constitute investment advice. The content of this email should not be copied or disclosed (in whole or part) to any other person. It may contain information which is confidential, privileged or otherwise protected from disclosure. If you are not the intended recipient, you should notify us and delete it from your system. Emails may be monitored, are not secure and may be amended, destroyed or contain viruses and in communicating with us such conditions are accepted. Any content which does not relate to business matters is not endorsed by us. Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et des Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. Calyon is incorporated in France with limited liability and registered in England & Wales. Registration number: FC008194. Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. This message and/or any attachments is intended for the sole use of its addressee. If you are not the addressee, please immediately notify the sender and then destroy the message. As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. All rights reserved. Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. Tous droits r?serv?s. From matthieu.brucher at gmail.com Wed Apr 15 13:01:48 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 15 Apr 2009 19:01:48 +0200 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <7F347D91614EBC48AA7540A2A76D3BB20448406D@MXCU10MX1.MSX.CIB> References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB20448406D@MXCU10MX1.MSX.CIB> Message-ID: Borland -> don't even try them, they are not worth it. Intel -> you may try, but I don't know if numpy or scipy was compiled with them. As there is a C compiler, as well as a C++ and a Fortran one, you may try. David compiles with mingw, you should give it a try (although compiling them yourself will lead to worse performance, as David provides atlas with numpy and scipy). Matthieu 2009/4/15 Fadhley Salim : > I'm compiling for Python 2.4, so I guess that means that I'm never going to be able to use anything more than visual studio 2003? > > What about compilors from other vendors, for example Intel or Borland? > > Thanks > > -----Original Message----- > From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Matthieu Brucher > Sent: 15 April 2009 17:45 > To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion] Compiling for free on Windows32 > > Hi, > > There is a Python limitation for the compiler. if you have 2.5 or less, you're stuck with VS2003, if you have 2.6, you can use 2008 (or 2005, I don't remember). If you use numscons, you can go with any of them (at your own risks). > You can also use mingw. > > For scipy, you have to use mingw, because there is no other free Fortran compiler. > > Matthieu > > 2009/4/15 Fadhley Salim : >> I've been asked to provide Numpy & Scipy as python egg files. >> Unfortunately Numpy and Scipy do not make official releases of their >> product in .egg form for a Win32 platform - that means if I want eggs >> then I have to compile them myself. >> >> At the moment my employer provides Visual Studio.Net 2003, which will >> compile no version of Numpy later than 1.1.1 - every version released >> subsequently cannot be compiled with VS2003. I've tried. >> >> What I'd really like is some other compiler I can use, perhaps for >> free, but at a push as a free time-limited trial. I would use that to >> compile the eggs. Is anybody aware of another compiler that I can >> download and use without paying anything and will definitely compile >> Numpy on Windows? >> >> So in summary, what I need is: >> >> * A compiler for Windows >> * Which will definitely compile numpy 1.3.0 >> * Which I can use for free (legally) >> >> Please only suggest something if you know for a fact that it works - >> no speculation! >> >> Thanks! >> >> This email does not create a legal relationship between any member of >> the Cr=E9dit Agricole group and the recipient or constitute investment advice. >> The content of this email should not be copied or disclosed (in whole >> or >> part) to any other person. It may contain information which is >> confidential, privileged or otherwise protected from disclosure. If >> you are not the intended recipient, you should notify us and delete it >> from your system. Emails may be monitored, are not secure and may be >> amended, destroyed or contain viruses and in communicating with us >> such conditions are accepted. Any content which does not relate to >> business matters is not endorsed by us. >> >> Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et >> des Entreprises d'Investissement (CECEI) and supervised by the >> Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. >> Details about the extent of our regulation by the Financial Services >> Authority are available from us on request. Calyon is incorporated in >> France with limited liability and registered in England & Wales. Registration number: FC008194. >> Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. >> This message and/or any ?attachments is intended for the sole use of its addressee. >> If you are not the addressee, please immediately notify the sender and then destroy the message. >> As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. >> All rights reserved. >> >> Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. >> Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. >> Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. >> Tous droits r?serv?s. >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > > -- > Information System Engineer, Ph.D. > Website: http://matthieu-brucher.developpez.com/ > Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn: http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > This email does not create a legal relationship between any member of the > Cr=E9dit Agricole group and the recipient or constitute investment advice. > The content of this email should not be copied or disclosed (in whole or > part) to any other person. It may contain information which is > confidential, privileged or otherwise protected from disclosure. If you > are not the intended recipient, you should notify us and delete it from > your system. Emails may be monitored, are not secure and may be amended, > destroyed or contain viruses and in communicating with us such conditions > are accepted. Any content which does not relate to business matters is not > endorsed by us. > > Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et des > Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire > in France and subject to limited regulation by the Financial Services Authority. > Details about the extent of our regulation by the Financial Services Authority > are available from us on request. Calyon is incorporated in France with limited > liability and registered in England & Wales. Registration number: FC008194. > Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. > This message and/or any ?attachments is intended for the sole use of its addressee. > If you are not the addressee, please immediately notify the sender and then destroy the message. > As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. > All rights reserved. > > Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. > Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. > Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. > Tous droits r?serv?s. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From Fadhley.Salim at uk.calyon.com Wed Apr 15 13:22:01 2009 From: Fadhley.Salim at uk.calyon.com (Fadhley Salim) Date: Wed, 15 Apr 2009 18:22:01 +0100 Subject: [Numpy-discussion] Compiling for free on Windows32 References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu><7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB><7F347D91614EBC48AA7540A2A76D3BB20448406D@MXCU10MX1.MSX.CIB> Message-ID: <7F347D91614EBC48AA7540A2A76D3BB20448406E@MXCU10MX1.MSX.CIB> So what is the official way to compile Numpy 1.3.0 for Python 2.4.x? If Visual Studio 2003 will not compile it and nothing else is intended to work, then surely nothing will work? That cannot be right. PS. Perhaps there is another way to do it... compiling for me is just a means to an end. We are users of Numpy not developers of Numpy. What I want most of all to just be able to pick up a .egg file for Numpy & Scipy pre-built for Win32 by somebody else. Even better if it were an official or quasi-official distribution of the product. Does anybody know if I can just download and use Numpy + Scipy in pre-compiled in egg form? Thanks -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Matthieu Brucher Sent: 15 April 2009 18:02 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Compiling for free on Windows32 Borland -> don't even try them, they are not worth it. Intel -> you may try, but I don't know if numpy or scipy was compiled with them. As there is a C compiler, as well as a C++ and a Fortran one, you may try. David compiles with mingw, you should give it a try (although compiling them yourself will lead to worse performance, as David provides atlas with numpy and scipy). Matthieu 2009/4/15 Fadhley Salim : > I'm compiling for Python 2.4, so I guess that means that I'm never going to be able to use anything more than visual studio 2003? > > What about compilors from other vendors, for example Intel or Borland? > > Thanks > > -----Original Message----- > From: numpy-discussion-bounces at scipy.org > [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Matthieu > Brucher > Sent: 15 April 2009 17:45 > To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion] Compiling for free on Windows32 > > Hi, > > There is a Python limitation for the compiler. if you have 2.5 or less, you're stuck with VS2003, if you have 2.6, you can use 2008 (or 2005, I don't remember). If you use numscons, you can go with any of them (at your own risks). > You can also use mingw. > > For scipy, you have to use mingw, because there is no other free Fortran compiler. > > Matthieu > > 2009/4/15 Fadhley Salim : >> I've been asked to provide Numpy & Scipy as python egg files. >> Unfortunately Numpy and Scipy do not make official releases of their >> product in .egg form for a Win32 platform - that means if I want eggs >> then I have to compile them myself. >> >> At the moment my employer provides Visual Studio.Net 2003, which will >> compile no version of Numpy later than 1.1.1 - every version released >> subsequently cannot be compiled with VS2003. I've tried. >> >> What I'd really like is some other compiler I can use, perhaps for >> free, but at a push as a free time-limited trial. I would use that to >> compile the eggs. Is anybody aware of another compiler that I can >> download and use without paying anything and will definitely compile >> Numpy on Windows? >> >> So in summary, what I need is: >> >> * A compiler for Windows >> * Which will definitely compile numpy 1.3.0 >> * Which I can use for free (legally) >> >> Please only suggest something if you know for a fact that it works - >> no speculation! >> >> Thanks! >> >> This email does not create a legal relationship between any member of >> the Cr=E9dit Agricole group and the recipient or constitute investment advice. >> The content of this email should not be copied or disclosed (in whole >> or >> part) to any other person. It may contain information which is >> confidential, privileged or otherwise protected from disclosure. If >> you are not the intended recipient, you should notify us and delete >> it from your system. Emails may be monitored, are not secure and may >> be amended, destroyed or contain viruses and in communicating with us >> such conditions are accepted. Any content which does not relate to >> business matters is not endorsed by us. >> >> Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit >> et des Entreprises d'Investissement (CECEI) and supervised by the >> Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. >> Details about the extent of our regulation by the Financial Services >> Authority are available from us on request. Calyon is incorporated in >> France with limited liability and registered in England & Wales. Registration number: FC008194. >> Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. >> This message and/or any ?attachments is intended for the sole use of its addressee. >> If you are not the addressee, please immediately notify the sender and then destroy the message. >> As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. >> All rights reserved. >> >> Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. >> Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. >> Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. >> Tous droits r?serv?s. >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > > -- > Information System Engineer, Ph.D. > Website: http://matthieu-brucher.developpez.com/ > Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn: http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > This email does not create a legal relationship between any member of > the Cr=E9dit Agricole group and the recipient or constitute investment advice. > The content of this email should not be copied or disclosed (in whole > or > part) to any other person. It may contain information which is > confidential, privileged or otherwise protected from disclosure. If > you are not the intended recipient, you should notify us and delete it > from your system. Emails may be monitored, are not secure and may be > amended, destroyed or contain viruses and in communicating with us > such conditions are accepted. Any content which does not relate to > business matters is not endorsed by us. > > Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et > des Entreprises d'Investissement (CECEI) and supervised by the > Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. > Details about the extent of our regulation by the Financial Services > Authority are available from us on request. Calyon is incorporated in > France with limited liability and registered in England & Wales. Registration number: FC008194. > Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. > This message and/or any ?attachments is intended for the sole use of its addressee. > If you are not the addressee, please immediately notify the sender and then destroy the message. > As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. > All rights reserved. > > Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. > Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. > Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. > Tous droits r?serv?s. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion This email does not create a legal relationship between any member of the Cr=E9dit Agricole group and the recipient or constitute investment advice. The content of this email should not be copied or disclosed (in whole or part) to any other person. It may contain information which is confidential, privileged or otherwise protected from disclosure. If you are not the intended recipient, you should notify us and delete it from your system. Emails may be monitored, are not secure and may be amended, destroyed or contain viruses and in communicating with us such conditions are accepted. Any content which does not relate to business matters is not endorsed by us. Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et des Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. Calyon is incorporated in France with limited liability and registered in England & Wales. Registration number: FC008194. Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. This message and/or any attachments is intended for the sole use of its addressee. If you are not the addressee, please immediately notify the sender and then destroy the message. As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. All rights reserved. Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. Tous droits r?serv?s. From theller at ctypes.org Wed Apr 15 13:48:27 2009 From: theller at ctypes.org (Thomas Heller) Date: Wed, 15 Apr 2009 19:48:27 +0200 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <7F347D91614EBC48AA7540A2A76D3BB20448406E@MXCU10MX1.MSX.CIB> References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu><7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB><7F347D91614EBC48AA7540A2A76D3BB20448406D@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB20448406E@MXCU10MX1.MSX.CIB> Message-ID: Fadhley Salim schrieb: > So what is the official way to compile Numpy 1.3.0 for Python 2.4.x? > If Visual Studio 2003 will not compile it and nothing else is > intended to work, then surely nothing will work? That cannot be > right. > > PS. Perhaps there is another way to do it... compiling for me is just > a means to an end. We are users of Numpy not developers of Numpy. > What I want most of all to just be able to pick up a .egg file for > Numpy & Scipy pre-built for Win32 by somebody else. Even better if it > were an official or quasi-official distribution of the product. Does > anybody know if I can just download and use Numpy + Scipy in > pre-compiled in egg form? What I've heard is that easy_install can install from bdist_wininst installers, I guess it converts them into egg format. Is that what you want? -- Thanks, Thomas From Fadhley.Salim at uk.calyon.com Wed Apr 15 13:58:43 2009 From: Fadhley.Salim at uk.calyon.com (Fadhley Salim) Date: Wed, 15 Apr 2009 18:58:43 +0100 Subject: [Numpy-discussion] Compiling for free on Windows32 References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu><7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB><7F347D91614EBC48AA7540A2A76D3BB20448406D@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB20448406E@MXCU10MX1.MSX.CIB> Message-ID: <7F347D91614EBC48AA7540A2A76D3BB204484070@MXCU10MX1.MSX.CIB> Thomasm, What want is the current latest Numpy as a win32 .egg file for Python 2.4.4. I'm not bothered how I get there. We've been able to compile numpy 1.1.1 on visual studio 2003 with no problems at all. I've not yet been able to get anything newer to work. At the moment I'm trying to build using the setupegg.py file which comes with the Numpy 1.3.0 distribution. After about 5 minutes of compiling it abruptly like this: http://pastebin.com/d67dd871 Previously I was told that this is because of a bug in VC++ 2003, however I've just been informed that it's not possible to use a later version of VC++ if I am building for python 2.4. So what *do* I use to to compile Numpy for Python 2.4.4? In summary my requirments: * Latest Numpy + Scipy * Each as a .egg file * Works with python.org's Cpython 2.4.4 for windows 32bit * Dont care how I make it as long as it works! :-) -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Thomas Heller Sent: 15 April 2009 18:48 To: numpy-discussion at scipy.org Subject: Re: [Numpy-discussion] Compiling for free on Windows32 Fadhley Salim schrieb: > So what is the official way to compile Numpy 1.3.0 for Python 2.4.x? > If Visual Studio 2003 will not compile it and nothing else is intended > to work, then surely nothing will work? That cannot be right. > > PS. Perhaps there is another way to do it... compiling for me is just > a means to an end. We are users of Numpy not developers of Numpy. > What I want most of all to just be able to pick up a .egg file for > Numpy & Scipy pre-built for Win32 by somebody else. Even better if it > were an official or quasi-official distribution of the product. Does > anybody know if I can just download and use Numpy + Scipy in > pre-compiled in egg form? What I've heard is that easy_install can install from bdist_wininst installers, I guess it converts them into egg format. Is that what you want? -- Thanks, Thomas _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- This email does not create a legal relationship between any member of the Cr=E9dit Agricole group and the recipient or constitute investment advice. The content of this email should not be copied or disclosed (in whole or part) to any other person. It may contain information which is confidential, privileged or otherwise protected from disclosure. If you are not the intended recipient, you should notify us and delete it from your system. Emails may be monitored, are not secure and may be amended, destroyed or contain viruses and in communicating with us such conditions are accepted. Any content which does not relate to business matters is not endorsed by us. Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et des Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. Calyon is incorporated in France with limited liability and registered in England & Wales. Registration number: FC008194. Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. This message and/or any attachments is intended for the sole use of its addressee. If you are not the addressee, please immediately notify the sender and then destroy the message. As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. All rights reserved. Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. Tous droits r?serv?s. From matthieu.brucher at gmail.com Wed Apr 15 14:14:24 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 15 Apr 2009 20:14:24 +0200 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <7F347D91614EBC48AA7540A2A76D3BB20448406E@MXCU10MX1.MSX.CIB> References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB20448406D@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB20448406E@MXCU10MX1.MSX.CIB> Message-ID: 2009/4/15 Fadhley Salim : > So what is the official way to compile Numpy 1.3.0 for Python 2.4.x? If Visual Studio 2003 will not compile it and nothing else is intended to work, then surely nothing will work? That cannot be right. Mingw is used to compile numpy and scipy AFAIK. > PS. Perhaps there is another way to do it... compiling for me is just a means to an end. We are users of Numpy not developers of Numpy. What I want most of all to just be able to pick up a .egg file for Numpy & Scipy pre-built for Win32 by somebody else. Even better if it were an official or quasi-official distribution of the product. Does anybody know if I can just download and use Numpy + Scipy in pre-compiled in egg form? Why is exe files a problem ? -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From strawman at astraw.com Wed Apr 15 14:31:29 2009 From: strawman at astraw.com (Andrew Straw) Date: Wed, 15 Apr 2009 11:31:29 -0700 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <7F347D91614EBC48AA7540A2A76D3BB204484070@MXCU10MX1.MSX.CIB> References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu><7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB><7F347D91614EBC48AA7540A2A76D3BB20448406D@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB20448406E@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB204484070@MXCU10MX1.MSX.CIB> Message-ID: <49E62801.6060201@astraw.com> Fadhley Salim wrote: > Thomasm, > > What want is the current latest Numpy as a win32 .egg file for Python > 2.4.4. I'm not bothered how I get there. We've been able to compile > * Dont care how I make it as long as it works! Are you aware that the reason numpy is not distributed as an .egg is so that the installer can determine the correct architecture (no SSE, SSE, SSE2) and install the appropriate files? This is not possible with .egg, so you'll either have to ship the no SSE version or ensure that it will be installed on only computers with the appropriate SIMD processor. -Andrew From charlesr.harris at gmail.com Wed Apr 15 14:32:20 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 15 Apr 2009 12:32:20 -0600 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <7F347D91614EBC48AA7540A2A76D3BB204484070@MXCU10MX1.MSX.CIB> References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB20448406D@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB20448406E@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB204484070@MXCU10MX1.MSX.CIB> Message-ID: On Wed, Apr 15, 2009 at 11:58 AM, Fadhley Salim wrote: > Thomasm, > > What want is the current latest Numpy as a win32 .egg file for Python > 2.4.4. I'm not bothered how I get there. We've been able to compile > numpy 1.1.1 on visual studio 2003 with no problems at all. I've not yet > been able to get anything newer to work. > > At the moment I'm trying to build using the setupegg.py file which comes > with the Numpy 1.3.0 distribution. After about 5 minutes of compiling it > abruptly like this: > > http://pastebin.com/d67dd871 > Ah yes. That looks to be from the documentation merge in r5610. It's a bug that shows up in Microsoft compilers because they can't handle long strings. You should file a ticket, it needs to be fixed. As a quick (it's all relative) workaround, you could break up the long strings in the mtrand.c file into string per line and use line continuations. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Fadhley.Salim at uk.calyon.com Wed Apr 15 14:33:48 2009 From: Fadhley.Salim at uk.calyon.com (Fadhley Salim) Date: Wed, 15 Apr 2009 19:33:48 +0100 Subject: [Numpy-discussion] Compiling for free on Windows32 References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu><7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB><7F347D91614EBC48AA7540A2A76D3BB20448406D@MXCU10MX1.MSX.CIB><7F347D91614EBC48AA7540A2A76D3BB20448406E@MXCU10MX1.MSX.CIB> Message-ID: <7F347D91614EBC48AA7540A2A76D3BB204484071@MXCU10MX1.MSX.CIB> > Mingw is used to compile numpy and scipy AFAIK. So if I just grab Mingw I should be able to compile the latest stuff with no problems? Will that work perfectly with my standard Python.org Cpython distribution? > Why is exe files a problem ? We deploy on a very large number of computers in 5 countries. PEAK's Setuptools gives us the ability to do all this deployment automatically. If we can package all of our dependancies as egg files then setuptools automatically deploys the egg files from a bunch of web-servers we have around the globe. It would be an enormous pain if we had to go to every single machine and run an exe installer. -------------- next part -------------- This email does not create a legal relationship between any member of the Cr=E9dit Agricole group and the recipient or constitute investment advice. The content of this email should not be copied or disclosed (in whole or part) to any other person. It may contain information which is confidential, privileged or otherwise protected from disclosure. If you are not the intended recipient, you should notify us and delete it from your system. Emails may be monitored, are not secure and may be amended, destroyed or contain viruses and in communicating with us such conditions are accepted. Any content which does not relate to business matters is not endorsed by us. Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et des Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. Calyon is incorporated in France with limited liability and registered in England & Wales. Registration number: FC008194. Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. This message and/or any attachments is intended for the sole use of its addressee. If you are not the addressee, please immediately notify the sender and then destroy the message. As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. All rights reserved. Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. Tous droits r?serv?s. From Fadhley.Salim at uk.calyon.com Wed Apr 15 14:35:51 2009 From: Fadhley.Salim at uk.calyon.com (Fadhley Salim) Date: Wed, 15 Apr 2009 19:35:51 +0100 Subject: [Numpy-discussion] Compiling for free on Windows32 References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu><7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB><7F347D91614EBC48AA7540A2A76D3BB20448406D@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB20448406E@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB204484070@MXCU10MX1.MSX.CIB> <49E62801.6060201@astraw.com> Message-ID: <7F347D91614EBC48AA7540A2A76D3BB204484072@MXCU10MX1.MSX.CIB> Good point - we do not require ultimate optimization. I'd be happy if I could custom build an egg that is optimized for our PCs which are all Xeons or Pentium 4's built in the last 2 years. -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Andrew Straw Sent: 15 April 2009 19:31 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Compiling for free on Windows32 Fadhley Salim wrote: > Thomasm, > > What want is the current latest Numpy as a win32 .egg file for Python > 2.4.4. I'm not bothered how I get there. We've been able to compile > * Dont care how I make it as long as it works! Are you aware that the reason numpy is not distributed as an .egg is so that the installer can determine the correct architecture (no SSE, SSE, SSE2) and install the appropriate files? This is not possible with .egg, so you'll either have to ship the no SSE version or ensure that it will be installed on only computers with the appropriate SIMD processor. -Andrew _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- This email does not create a legal relationship between any member of the Cr=E9dit Agricole group and the recipient or constitute investment advice. The content of this email should not be copied or disclosed (in whole or part) to any other person. It may contain information which is confidential, privileged or otherwise protected from disclosure. If you are not the intended recipient, you should notify us and delete it from your system. Emails may be monitored, are not secure and may be amended, destroyed or contain viruses and in communicating with us such conditions are accepted. Any content which does not relate to business matters is not endorsed by us. Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et des Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. Calyon is incorporated in France with limited liability and registered in England & Wales. Registration number: FC008194. Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. This message and/or any attachments is intended for the sole use of its addressee. If you are not the addressee, please immediately notify the sender and then destroy the message. As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. All rights reserved. Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. Tous droits r?serv?s. From Fadhley.Salim at uk.calyon.com Wed Apr 15 14:37:56 2009 From: Fadhley.Salim at uk.calyon.com (Fadhley Salim) Date: Wed, 15 Apr 2009 19:37:56 +0100 Subject: [Numpy-discussion] Compiling for free on Windows32 References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu><7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB><7F347D91614EBC48AA7540A2A76D3BB20448406D@MXCU10MX1.MSX.CIB><7F347D91614EBC48AA7540A2A76D3BB20448406E@MXCU10MX1.MSX.CIB><7F347D91614EBC48AA7540A2A76D3BB204484070@MXCU10MX1.MSX.CIB> Message-ID: <7F347D91614EBC48AA7540A2A76D3BB204484073@MXCU10MX1.MSX.CIB> Chuck, I'm not all that familiar with the bug reporting process but if you were to point me to the relevant bug tracker I'd be delighted to file an issue. If I could get somebody to fix this upstream I'd be so happy! Thanks! Sal ________________________________ From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Charles R Harris Sent: 15 April 2009 19:32 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Compiling for free on Windows32 On Wed, Apr 15, 2009 at 11:58 AM, Fadhley Salim wrote: Thomasm, What want is the current latest Numpy as a win32 .egg file for Python 2.4.4. I'm not bothered how I get there. We've been able to compile numpy 1.1.1 on visual studio 2003 with no problems at all. I've not yet been able to get anything newer to work. At the moment I'm trying to build using the setupegg.py file which comes with the Numpy 1.3.0 distribution. After about 5 minutes of compiling it abruptly like this: http://pastebin.com/d67dd871 Ah yes. That looks to be from the documentation merge in r5610. It's a bug that shows up in Microsoft compilers because they can't handle long strings. You should file a ticket, it needs to be fixed. As a quick (it's all relative) workaround, you could break up the long strings in the mtrand.c file into string per line and use line continuations. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- This email does not create a legal relationship between any member of the Cr=E9dit Agricole group and the recipient or constitute investment advice. The content of this email should not be copied or disclosed (in whole or part) to any other person. It may contain information which is confidential, privileged or otherwise protected from disclosure. If you are not the intended recipient, you should notify us and delete it from your system. Emails may be monitored, are not secure and may be amended, destroyed or contain viruses and in communicating with us such conditions are accepted. Any content which does not relate to business matters is not endorsed by us. Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et des Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. Calyon is incorporated in France with limited liability and registered in England & Wales. Registration number: FC008194. Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. This message and/or any attachments is intended for the sole use of its addressee. If you are not the addressee, please immediately notify the sender and then destroy the message. As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. All rights reserved. Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. Tous droits r?serv?s. From charlesr.harris at gmail.com Wed Apr 15 15:00:45 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 15 Apr 2009 13:00:45 -0600 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <7F347D91614EBC48AA7540A2A76D3BB204484073@MXCU10MX1.MSX.CIB> References: <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB20448406D@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB20448406E@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB204484070@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB204484073@MXCU10MX1.MSX.CIB> Message-ID: Hi Sal, On Wed, Apr 15, 2009 at 12:37 PM, Fadhley Salim wrote: > Chuck, > > I'm not all that familiar with the bug reporting process but if you were to > point me to the relevant bug tracker I'd be delighted to file an issue. If I > could get somebody to fix this upstream I'd be so happy! > > Thanks! > > Go to SciPy and hit the bug button at the top of the page. You should include a sample of the error output that you get in the report. I'll do it myself later if you don't get to it, but I'm a pusher and filing bug reports is a gateway drug. The file in question is a pyrex generated file, so I regard this as a pyrex bug. If cython doesn't have this problem then that is an argument for cythonizing random. It is also possible that we should be adding the document strings after the module is loaded, as we do with other c code. It needs to be looked at. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Fadhley.Salim at uk.calyon.com Wed Apr 15 15:03:54 2009 From: Fadhley.Salim at uk.calyon.com (Fadhley Salim) Date: Wed, 15 Apr 2009 20:03:54 +0100 Subject: [Numpy-discussion] Compiling for free on Windows32 References: <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB><7F347D91614EBC48AA7540A2A76D3BB20448406D@MXCU10MX1.MSX.CIB><7F347D91614EBC48AA7540A2A76D3BB20448406E@MXCU10MX1.MSX.CIB><7F347D91614EBC48AA7540A2A76D3BB204484070@MXCU10MX1.MSX.CIB><7F347D91614EBC48AA7540A2A76D3BB204484073@MXCU10MX1.MSX.CIB> Message-ID: <7F347D91614EBC48AA7540A2A76D3BB204484075@MXCU10MX1.MSX.CIB> http://projects.scipy.org/numpy/ticket/1086 I filed this under numpy since that's what I am trying to compile. Please feel free to correct if you think I've mis-described the fault. I'm heading home now. My sinceerest thanks to everybody in the group who's helped me with this today. Sal ________________________________ From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Charles R Harris Sent: 15 April 2009 20:01 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Compiling for free on Windows32 Hi Sal, On Wed, Apr 15, 2009 at 12:37 PM, Fadhley Salim wrote: Chuck, I'm not all that familiar with the bug reporting process but if you were to point me to the relevant bug tracker I'd be delighted to file an issue. If I could get somebody to fix this upstream I'd be so happy! Thanks! Go to SciPy and hit the bug button at the top of the page. You should include a sample of the error output that you get in the report. I'll do it myself later if you don't get to it, but I'm a pusher and filing bug reports is a gateway drug. The file in question is a pyrex generated file, so I regard this as a pyrex bug. If cython doesn't have this problem then that is an argument for cythonizing random. It is also possible that we should be adding the document strings after the module is loaded, as we do with other c code. It needs to be looked at. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- This email does not create a legal relationship between any member of the Cr=E9dit Agricole group and the recipient or constitute investment advice. The content of this email should not be copied or disclosed (in whole or part) to any other person. It may contain information which is confidential, privileged or otherwise protected from disclosure. If you are not the intended recipient, you should notify us and delete it from your system. Emails may be monitored, are not secure and may be amended, destroyed or contain viruses and in communicating with us such conditions are accepted. Any content which does not relate to business matters is not endorsed by us. Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et des Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. Calyon is incorporated in France with limited liability and registered in England & Wales. Registration number: FC008194. Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. This message and/or any attachments is intended for the sole use of its addressee. If you are not the addressee, please immediately notify the sender and then destroy the message. As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. All rights reserved. Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. Tous droits r?serv?s. From charlesr.harris at gmail.com Wed Apr 15 15:15:23 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 15 Apr 2009 13:15:23 -0600 Subject: [Numpy-discussion] Numpy ticket notifications not being mailed. Message-ID: The subject line says it all... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Apr 15 15:16:22 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 15 Apr 2009 13:16:22 -0600 Subject: [Numpy-discussion] Numpy ticket notifications not being mailed. In-Reply-To: References: Message-ID: On Wed, Apr 15, 2009 at 1:15 PM, Charles R Harris wrote: > The subject line says it all... Chuck > Seems to be fixed now. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Wed Apr 15 15:17:26 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 16 Apr 2009 04:17:26 +0900 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> Message-ID: <5b8d13220904151217n53499a3eo317e6b8baa12d719@mail.gmail.com> On Thu, Apr 16, 2009 at 1:17 AM, Fadhley Salim wrote: > I've been asked to provide Numpy & Scipy as python egg files. > Unfortunately Numpy and Scipy do not make official releases of their > product in .egg form for a Win32 platform - that means if I want eggs > then I have to compile them myself. Yes. > > At the moment my employer provides Visual Studio.Net 2003, which will > compile no version of Numpy later than 1.1.1 - every version released > subsequently cannot be compiled with VS2003. I've tried. Yes, this is a visual studio bug. > > What I'd really like is some other compiler I can use, perhaps for free, > but at a push as a free time-limited trial. I would use that to compile > the eggs. Is anybody aware of another compiler that I can download and > use without paying anything and will definitely compile Numpy on > Windows? The compilers that are regularly used to compile python on windows are the mingw compilers. If you only need numpy and not scipy, there are no dependencies, and only the C compiler is needed. It should compile out of the box. Otherwise, you will need to compile your own blas/lapack, which is complicated on windows. The blas/lapack sources are to be downloaded from netlib.org: http://netlib.org/lapack/ You should not use the version 3.2 (3.1.1 is fine, and is the version used to build the numpy binaries). cheers, David From josef.pktd at gmail.com Wed Apr 15 15:42:05 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 15 Apr 2009 15:42:05 -0400 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <5b8d13220904151217n53499a3eo317e6b8baa12d719@mail.gmail.com> References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> <5b8d13220904151217n53499a3eo317e6b8baa12d719@mail.gmail.com> Message-ID: <1cd32cbb0904151242qd70eec6r78b570e87215d028@mail.gmail.com> On Wed, Apr 15, 2009 at 3:17 PM, David Cournapeau wrote: > On Thu, Apr 16, 2009 at 1:17 AM, Fadhley Salim > wrote: >> I've been asked to provide Numpy & Scipy as python egg files. >> Unfortunately Numpy and Scipy do not make official releases of their >> product in .egg form for a Win32 platform - that means if I want eggs >> then I have to compile them myself. > > Yes. > >> >> At the moment my employer provides Visual Studio.Net 2003, which will >> compile no version of Numpy later than 1.1.1 - every version released >> subsequently cannot be compiled with VS2003. I've tried. > > Yes, this is a visual studio bug. > >> >> What I'd really like is some other compiler I can use, perhaps for free, >> but at a push as a free time-limited trial. I would use that to compile >> the eggs. Is anybody aware of another compiler that I can download and >> use without paying anything and will definitely compile Numpy on >> Windows? > > The compilers that are regularly used to compile python on windows are > the mingw compilers. If you only need numpy and not scipy, there are > no dependencies, and only the C compiler is needed. It should compile > out of the box. > > Otherwise, you will need to compile your own blas/lapack, which is > complicated on windows. The blas/lapack sources are to be downloaded > from netlib.org: > > http://netlib.org/lapack/ > > You should not use the version 3.2 (3.1.1 is fine, and is the version > used to build the numpy binaries). > > cheers, > > David numpy compiles fine with mingw, but I never managed to build an egg. I'm not able to get the --compiler=mingw32 to be recognized. an error like No module named msvccompiler in numpy.distutils; trying from distutils error: Python was built with Visual Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py. but -c mingw32 is not a valid command line option: python setupegg.py bdist_egg --compiler=mingw32 error: option --compiler not recognized It also doesn't take the compiler specified in pythons distutils.cfg. No problems with build, and bdist, but it doesn't work with setuptools. The problem that the combination of setuptools and numpy distutils doesn't honor -c mingw32, I had now for several packages, but as usual I'm not sure whether it's my setup or a bug. Josef From charlesr.harris at gmail.com Wed Apr 15 15:46:30 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 15 Apr 2009 13:46:30 -0600 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <5b8d13220904151217n53499a3eo317e6b8baa12d719@mail.gmail.com> References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> <5b8d13220904151217n53499a3eo317e6b8baa12d719@mail.gmail.com> Message-ID: On Wed, Apr 15, 2009 at 1:17 PM, David Cournapeau wrote: > On Thu, Apr 16, 2009 at 1:17 AM, Fadhley Salim > wrote: > > I've been asked to provide Numpy & Scipy as python egg files. > > Unfortunately Numpy and Scipy do not make official releases of their > > product in .egg form for a Win32 platform - that means if I want eggs > > then I have to compile them myself. > > Yes. > > > > > At the moment my employer provides Visual Studio.Net 2003, which will > > compile no version of Numpy later than 1.1.1 - every version released > > subsequently cannot be compiled with VS2003. I've tried. > > Yes, this is a visual studio bug. > Looks like a pyrex bug to me, the documentation lines are too long. Remember those 'ol ufunc docstrings? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Wed Apr 15 15:52:54 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 15 Apr 2009 19:52:54 +0000 (UTC) Subject: [Numpy-discussion] Ill-defined in-place operations (#1085) Message-ID: Mixing views and mutating (eg. in-place) operations can cause surprising and ill-defined results. Consider http://projects.scipy.org/numpy/ticket/1085: >>> import numpy as np >>> x = np.array([[1,2], [3,4]]) >>> x array([[1, 2], [3, 4]]) >>> x += x.T >>> x array([[2, 5], [8, 8]]) >>> y = np.array([[1,2], [3,4]], order='F') >>> y array([[1, 2], [3, 4]]) >>> y += y.T >>> y array([[2, 7], [5, 8]]) The result depends on the order in which the elements happen to lie in the memory. Predicting the outcome is nearly impossible. (Also, I think Numpy tries to optimize the order of the loops, making it even more difficult?) This is a sort of a pitfall. Should Numpy issue a warning every time a mutating operation is performed on an array, with input data that is a view on the same array? Some alternatives: a) Raise warning for all arrays, even 1-D. b) Raise warning only for multidimensional arrays. (Assume the user can predict what happens in 1-D.) c) Don't raise any warnings, just educate users in the documentation. This is not the first time this has been discussed on this ML, but the last discussion did not (IIRC) lead to any conclusions. (But I think it was established that detecting when views really overlap is a problem with no cheap solution. So walking up ->base pointers or comparing ->data pointers seemed nearly the only feasible way.) -- Pauli Virtanen From pav at iki.fi Wed Apr 15 15:54:30 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 15 Apr 2009 19:54:30 +0000 (UTC) Subject: [Numpy-discussion] Compiling for free on Windows32 References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> <5b8d13220904151217n53499a3eo317e6b8baa12d719@mail.gmail.com> <1cd32cbb0904151242qd70eec6r78b570e87215d028@mail.gmail.com> Message-ID: Wed, 15 Apr 2009 15:42:05 -0400, josef.pktd wrote: > On Wed, Apr 15, 2009 at 3:17 PM, David Cournapeau > wrote: >> On Thu, Apr 16, 2009 at 1:17 AM, Fadhley Salim >> wrote: >>> I've been asked to provide Numpy & Scipy as python egg files. >>> Unfortunately Numpy and Scipy do not make official releases of their >>> product in .egg form for a Win32 platform - that means if I want eggs >>> then I have to compile them myself. >> >> Yes. >> >> >>> At the moment my employer provides Visual Studio.Net 2003, which will >>> compile no version of Numpy later than 1.1.1 - every version released >>> subsequently cannot be compiled with VS2003. I've tried. >> >> Yes, this is a visual studio bug. >> >> >>> What I'd really like is some other compiler I can use, perhaps for >>> free, but at a push as a free time-limited trial. I would use that to >>> compile the eggs. Is anybody aware of another compiler that I can >>> download and use without paying anything and will definitely compile >>> Numpy on Windows? >> >> The compilers that are regularly used to compile python on windows are >> the mingw compilers. If you only need numpy and not scipy, there are no >> dependencies, and only the C compiler is needed. It should compile out >> of the box. >> >> Otherwise, you will need to compile your own blas/lapack, which is >> complicated on windows. The blas/lapack sources are to be downloaded >> from netlib.org: >> >> http://netlib.org/lapack/ >> >> You should not use the version 3.2 (3.1.1 is fine, and is the version >> used to build the numpy binaries). >> >> cheers, >> >> David > > numpy compiles fine with mingw, but I never managed to build an egg. I'm > not able to get the --compiler=mingw32 to be recognized. I think it might work if you separate the compiling and bdist_egg steps: setupegg.py build --compiler=mingw32 bdist_egg Does this help? -- Pauli Virtanen From josef.pktd at gmail.com Wed Apr 15 16:06:22 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 15 Apr 2009 16:06:22 -0400 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> <5b8d13220904151217n53499a3eo317e6b8baa12d719@mail.gmail.com> <1cd32cbb0904151242qd70eec6r78b570e87215d028@mail.gmail.com> Message-ID: <1cd32cbb0904151306v76b1db2doe741c6db7274c648@mail.gmail.com> On Wed, Apr 15, 2009 at 3:54 PM, Pauli Virtanen wrote: > Wed, 15 Apr 2009 15:42:05 -0400, josef.pktd wrote: > >> On Wed, Apr 15, 2009 at 3:17 PM, David Cournapeau >> wrote: >>> On Thu, Apr 16, 2009 at 1:17 AM, Fadhley Salim >>> wrote: >>>> I've been asked to provide Numpy & Scipy as python egg files. >>>> Unfortunately Numpy and Scipy do not make official releases of their >>>> product in .egg form for a Win32 platform - that means if I want eggs >>>> then I have to compile them myself. >>> >>> Yes. >>> >>> >>>> At the moment my employer provides Visual Studio.Net 2003, which will >>>> compile no version of Numpy later than 1.1.1 - every version released >>>> subsequently cannot be compiled with VS2003. I've tried. >>> >>> Yes, this is a visual studio bug. >>> >>> >>>> What I'd really like is some other compiler I can use, perhaps for >>>> free, but at a push as a free time-limited trial. I would use that to >>>> compile the eggs. Is anybody aware of another compiler that I can >>>> download and use without paying anything and will definitely compile >>>> Numpy on Windows? >>> >>> The compilers that are regularly used to compile python on windows are >>> the mingw compilers. If you only need numpy and not scipy, there are no >>> dependencies, and only the C compiler is needed. It should compile out >>> of the box. >>> >>> Otherwise, you will need to compile your own blas/lapack, which is >>> complicated on windows. The blas/lapack sources are to be downloaded >>> from netlib.org: >>> >>> http://netlib.org/lapack/ >>> >>> You should not use the version 3.2 (3.1.1 is fine, and is the version >>> used to build the numpy binaries). >>> >>> cheers, >>> >>> David >> >> numpy compiles fine with mingw, but I never managed to build an egg. I'm >> not able to get the --compiler=mingw32 to be recognized. > > I think it might work if you separate the compiling and bdist_egg steps: > > ? ? ? ?setupegg.py build --compiler=mingw32 bdist_egg > > Does this help? Perfect, I thought I had tried all possible combinations. (but still strange) Josef From charlesr.harris at gmail.com Wed Apr 15 16:10:57 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 15 Apr 2009 14:10:57 -0600 Subject: [Numpy-discussion] Ill-defined in-place operations (#1085) In-Reply-To: References: Message-ID: On Wed, Apr 15, 2009 at 1:52 PM, Pauli Virtanen wrote: > > Mixing views and mutating (eg. in-place) operations can cause surprising > and ill-defined results. Consider > http://projects.scipy.org/numpy/ticket/1085: > > >>> import numpy as np > >>> x = np.array([[1,2], [3,4]]) > >>> x > array([[1, 2], > [3, 4]]) > >>> x += x.T > >>> x > array([[2, 5], > [8, 8]]) > >>> y = np.array([[1,2], [3,4]], order='F') > >>> y > array([[1, 2], > [3, 4]]) > >>> y += y.T > >>> y > array([[2, 7], > [5, 8]]) > > The result depends on the order in which the elements happen to lie in > the memory. Predicting the outcome is nearly impossible. (Also, I think > Numpy tries to optimize the order of the loops, making it even more > difficult?) > > This is a sort of a pitfall. Should Numpy issue a warning every time a > mutating operation is performed on an array, with input data that is a > view on the same array? > My first thought is yes, just because the result is hard to predict. > > Some alternatives: > > a) Raise warning for all arrays, even 1-D. > Yes again. Are there examples where one would not want a warning? Examples that couldn't be implemented otherwise? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Wed Apr 15 16:34:49 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 15 Apr 2009 22:34:49 +0200 Subject: [Numpy-discussion] Ill-defined in-place operations (#1085) In-Reply-To: References: Message-ID: I think this is unavoidable. If we state clearly that every numpy function assums non-pointer aliasing, then people have to care about this themselves (like they would in Fortran, or in C++ for object-like arrays). Matthieu 2009/4/15 Charles R Harris : > > > On Wed, Apr 15, 2009 at 1:52 PM, Pauli Virtanen wrote: >> >> Mixing views and mutating (eg. in-place) operations can cause surprising >> and ill-defined results. Consider >> http://projects.scipy.org/numpy/ticket/1085: >> >> >>> import numpy as np >> >>> x = np.array([[1,2], [3,4]]) >> >>> x >> array([[1, 2], >> ? ? ? [3, 4]]) >> >>> x += x.T >> >>> x >> array([[2, 5], >> ? ? ? [8, 8]]) >> >>> y = np.array([[1,2], [3,4]], order='F') >> >>> y >> array([[1, 2], >> ? ? ? [3, 4]]) >> >>> y += y.T >> >>> y >> array([[2, 7], >> ? ? ? [5, 8]]) >> >> The result depends on the order in which the elements happen to lie in >> the memory. Predicting the outcome is nearly impossible. (Also, I think >> Numpy tries to optimize the order of the loops, making it even more >> difficult?) >> >> This is a sort of a pitfall. Should Numpy issue a warning every time a >> mutating operation is performed on an array, with input data that is a >> view on the same array? > > My first thought is yes, just because the result is hard to predict. > >> >> Some alternatives: >> >> ?a) Raise warning for all arrays, even 1-D. > > Yes again. Are there examples where one would not want a warning? Examples > that couldn't be implemented otherwise? > > Chuck > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From josef.pktd at gmail.com Wed Apr 15 16:36:09 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 15 Apr 2009 16:36:09 -0400 Subject: [Numpy-discussion] Ill-defined in-place operations (#1085) In-Reply-To: References: Message-ID: <1cd32cbb0904151336k3a016e62qffdcca50efad665@mail.gmail.com> On Wed, Apr 15, 2009 at 4:10 PM, Charles R Harris wrote: > > > On Wed, Apr 15, 2009 at 1:52 PM, Pauli Virtanen wrote: >> >> Mixing views and mutating (eg. in-place) operations can cause surprising >> and ill-defined results. Consider >> http://projects.scipy.org/numpy/ticket/1085: >> >> >>> import numpy as np >> >>> x = np.array([[1,2], [3,4]]) >> >>> x >> array([[1, 2], >> ? ? ? [3, 4]]) >> >>> x += x.T >> >>> x >> array([[2, 5], >> ? ? ? [8, 8]]) >> >>> y = np.array([[1,2], [3,4]], order='F') >> >>> y >> array([[1, 2], >> ? ? ? [3, 4]]) >> >>> y += y.T >> >>> y >> array([[2, 7], >> ? ? ? [5, 8]]) >> >> The result depends on the order in which the elements happen to lie in >> the memory. Predicting the outcome is nearly impossible. (Also, I think >> Numpy tries to optimize the order of the loops, making it even more >> difficult?) >> >> This is a sort of a pitfall. Should Numpy issue a warning every time a >> mutating operation is performed on an array, with input data that is a >> view on the same array? > > My first thought is yes, just because the result is hard to predict. > >> >> Some alternatives: >> >> ?a) Raise warning for all arrays, even 1-D. > > Yes again. Are there examples where one would not want a warning? Examples > that couldn't be implemented otherwise? > > Chuck I had a similar problem with changing the view or dtype of an array to a structured array, that also worked directly on the base and only threw an exception if the base was inconsistent with the change in dtype. Is there a list of gotchas for working with views that might produce unexpected results? Josef From josef.pktd at gmail.com Wed Apr 15 16:40:30 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 15 Apr 2009 16:40:30 -0400 Subject: [Numpy-discussion] Ill-defined in-place operations (#1085) In-Reply-To: References: Message-ID: <1cd32cbb0904151340h1f1c977r847347eb35f33a88@mail.gmail.com> On Wed, Apr 15, 2009 at 4:34 PM, Matthieu Brucher wrote: > I think this is unavoidable. If we state clearly that every numpy > function assums non-pointer aliasing, then people have to care about > this themselves (like they would in Fortran, or in C++ for object-like > arrays). "non-pointer aliasing" this might be clear for those that come from fortran or c++. But for those that come from matlab or gauss, the intuition is "what you see is what you get", and I have no idea what "non-pointer aliasing" is. Josef From pav at iki.fi Wed Apr 15 16:41:16 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 15 Apr 2009 20:41:16 +0000 (UTC) Subject: [Numpy-discussion] Ill-defined in-place operations (#1085) References: <1cd32cbb0904151336k3a016e62qffdcca50efad665@mail.gmail.com> Message-ID: Wed, 15 Apr 2009 16:36:09 -0400, josef.pktd wrote: [clip] > Is there a list of gotchas for working with views that might produce > unexpected results? I don't think there is. Also, the reference manual would probably need a separate section about views. -- Pauli Virtanen From matthieu.brucher at gmail.com Wed Apr 15 16:43:56 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 15 Apr 2009 22:43:56 +0200 Subject: [Numpy-discussion] Ill-defined in-place operations (#1085) In-Reply-To: <1cd32cbb0904151340h1f1c977r847347eb35f33a88@mail.gmail.com> References: <1cd32cbb0904151340h1f1c977r847347eb35f33a88@mail.gmail.com> Message-ID: 2009/4/15 : > On Wed, Apr 15, 2009 at 4:34 PM, Matthieu Brucher > wrote: >> I think this is unavoidable. If we state clearly that every numpy >> function assums non-pointer aliasing, then people have to care about >> this themselves (like they would in Fortran, or in C++ for object-like >> arrays). > > > "non-pointer aliasing" > > this might be clear for those that come from fortran or c++. > But for those that come from matlab or gauss, the intuition is ?"what > you see is what you get", and I have no idea what "non-pointer > aliasing" is. It happens when an output pointer is also used as an input (with a potential offset). And even Fortran, C or C++ programmers don't know what it is :| -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From william.ratcliff at gmail.com Wed Apr 15 16:46:53 2009 From: william.ratcliff at gmail.com (william ratcliff) Date: Wed, 15 Apr 2009 16:46:53 -0400 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <1cd32cbb0904151306v76b1db2doe741c6db7274c648@mail.gmail.com> References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> <5b8d13220904151217n53499a3eo317e6b8baa12d719@mail.gmail.com> <1cd32cbb0904151242qd70eec6r78b570e87215d028@mail.gmail.com> <1cd32cbb0904151306v76b1db2doe741c6db7274c648@mail.gmail.com> Message-ID: <827183970904151346h64518f1dqbf31615c669b0b9d@mail.gmail.com> Hi! What I do is make a distutils.cfg file and put in the $PYTHONHOME/Lib/distutils directory (for me, c:\python25\lib\distutils) and then it will use mingw32 for the compilation. I've attached the one I use. Cheers, William On Wed, Apr 15, 2009 at 4:06 PM, wrote: > On Wed, Apr 15, 2009 at 3:54 PM, Pauli Virtanen wrote: > > Wed, 15 Apr 2009 15:42:05 -0400, josef.pktd wrote: > > > >> On Wed, Apr 15, 2009 at 3:17 PM, David Cournapeau > >> wrote: > >>> On Thu, Apr 16, 2009 at 1:17 AM, Fadhley Salim > >>> wrote: > >>>> I've been asked to provide Numpy & Scipy as python egg files. > >>>> Unfortunately Numpy and Scipy do not make official releases of their > >>>> product in .egg form for a Win32 platform - that means if I want eggs > >>>> then I have to compile them myself. > >>> > >>> Yes. > >>> > >>> > >>>> At the moment my employer provides Visual Studio.Net 2003, which will > >>>> compile no version of Numpy later than 1.1.1 - every version released > >>>> subsequently cannot be compiled with VS2003. I've tried. > >>> > >>> Yes, this is a visual studio bug. > >>> > >>> > >>>> What I'd really like is some other compiler I can use, perhaps for > >>>> free, but at a push as a free time-limited trial. I would use that to > >>>> compile the eggs. Is anybody aware of another compiler that I can > >>>> download and use without paying anything and will definitely compile > >>>> Numpy on Windows? > >>> > >>> The compilers that are regularly used to compile python on windows are > >>> the mingw compilers. If you only need numpy and not scipy, there are no > >>> dependencies, and only the C compiler is needed. It should compile out > >>> of the box. > >>> > >>> Otherwise, you will need to compile your own blas/lapack, which is > >>> complicated on windows. The blas/lapack sources are to be downloaded > >>> from netlib.org: > >>> > >>> http://netlib.org/lapack/ > >>> > >>> You should not use the version 3.2 (3.1.1 is fine, and is the version > >>> used to build the numpy binaries). > >>> > >>> cheers, > >>> > >>> David > >> > >> numpy compiles fine with mingw, but I never managed to build an egg. I'm > >> not able to get the --compiler=mingw32 to be recognized. > > > > I think it might work if you separate the compiling and bdist_egg steps: > > > > setupegg.py build --compiler=mingw32 bdist_egg > > > > Does this help? > > Perfect, I thought I had tried all possible combinations. > (but still strange) > > Josef > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: distutils.cfg Type: application/octet-stream Size: 55 bytes Desc: not available URL: From pav at iki.fi Wed Apr 15 16:51:50 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 15 Apr 2009 20:51:50 +0000 (UTC) Subject: [Numpy-discussion] Ill-defined in-place operations (#1085) References: Message-ID: Wed, 15 Apr 2009 22:34:49 +0200, Matthieu Brucher wrote: > I think this is unavoidable. If we state clearly that every numpy > function assums non-pointer aliasing, then people have to care about > this themselves (like they would in Fortran, or in C++ for object-like > arrays). Nevertheless, this is an easy way to shoot yourself into foot. Also, (as indicated by the x.T example), it's doubtful anyone actually relies on the current behavior for N-D arrays, since it depends on order of data in memory, etc. It rather likely is possible to cheaply detect potentially troublesome cases and raise a warning. (The user can silence these, if the behavior is really relied on.) I don't know what is the performance impact of Python warnings, though. The alternative is to have really good documentation on views... -- Pauli Virtanen From kbasye1 at jhu.edu Wed Apr 15 17:21:59 2009 From: kbasye1 at jhu.edu (Ken Basye) Date: Wed, 15 Apr 2009 17:21:59 -0400 Subject: [Numpy-discussion] Numpy Trac site redirecting in a loop? Message-ID: <49E64FF7.3050804@jhu.edu> Hi all, Just now when I tried to access http://projects.scipy.org/numpy, Firefox complains (after a bit) that there's a redirect loop. Is anyone else seeing this problem, and/or does anyone know how I can fix my side of things to get me to the site? Thanks, Ken From charlesr.harris at gmail.com Wed Apr 15 17:27:21 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 15 Apr 2009 15:27:21 -0600 Subject: [Numpy-discussion] Question for Pauli or Stefan about doc editor. Message-ID: Hi Pauli, I'm going to move the docstrings from mtrand.pyx to add_newdocs.py. Will this require any fixes to the doc editor so that the docs don't get readded to mtrand? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Apr 15 17:39:09 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 15 Apr 2009 15:39:09 -0600 Subject: [Numpy-discussion] Ill-defined in-place operations (#1085) In-Reply-To: References: Message-ID: On Wed, Apr 15, 2009 at 2:51 PM, Pauli Virtanen wrote: > Wed, 15 Apr 2009 22:34:49 +0200, Matthieu Brucher wrote: > > > I think this is unavoidable. If we state clearly that every numpy > > function assums non-pointer aliasing, then people have to care about > > this themselves (like they would in Fortran, or in C++ for object-like > > arrays). > > Nevertheless, this is an easy way to shoot yourself into foot. Also, (as > indicated by the x.T example), it's doubtful anyone actually relies on > the current behavior for N-D arrays, since it depends on order of data in > memory, etc. > > It rather likely is possible to cheaply detect potentially troublesome > cases and raise a warning. (The user can silence these, if the behavior > is really relied on.) I don't know what is the performance impact of > Python warnings, though. > > The alternative is to have really good documentation on views... > Asking the average user to understand memory issues is probably a bit much. Better to raise a warning and let the sophisticated users override it if they really need to. Checking if the same memory chunk is used should be sufficient (I think that can be done). Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Apr 15 18:00:17 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 15 Apr 2009 18:00:17 -0400 Subject: [Numpy-discussion] Ill-defined in-place operations (#1085) In-Reply-To: References: Message-ID: <1cd32cbb0904151500o4a921bb7m288ac9b6dd8f8703@mail.gmail.com> On Wed, Apr 15, 2009 at 5:39 PM, Charles R Harris wrote: > > > On Wed, Apr 15, 2009 at 2:51 PM, Pauli Virtanen wrote: >> >> Wed, 15 Apr 2009 22:34:49 +0200, Matthieu Brucher wrote: >> >> > I think this is unavoidable. If we state clearly that every numpy >> > function assums non-pointer aliasing, then people have to care about >> > this themselves (like they would in Fortran, or in C++ for object-like >> > arrays). >> >> Nevertheless, this is an easy way to shoot yourself into foot. Also, (as >> indicated by the x.T example), it's doubtful anyone actually relies on >> the current behavior for N-D arrays, since it depends on order of data in >> memory, etc. >> >> It rather likely is possible to cheaply detect potentially troublesome >> cases and raise a warning. (The user can silence these, if the behavior >> is really relied on.) I don't know what is the performance impact of >> Python warnings, though. >> >> The alternative is to have really good documentation on views... > > Asking the average user to understand memory issues is probably a bit much. > Better to raise a warning and let the sophisticated users override it if > they really need to. Checking if the same memory chunk is used should be > sufficient (I think that can be done). > It is also often difficult to figure out if an operation creates a view or a copy, and it would be safer to at least get a warning. The more convenient solution for users like me, would be to do also a copy in those cases where the operation is "inconsistent" with the view. Since I realized that there is this type of problem, I started to add more .copy() to my calculations. Josef From pav at iki.fi Wed Apr 15 18:17:52 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 15 Apr 2009 22:17:52 +0000 (UTC) Subject: [Numpy-discussion] Question for Pauli or Stefan about doc editor. References: Message-ID: Wed, 15 Apr 2009 15:27:21 -0600, Charles R Harris wrote: > I'm going to move the docstrings from mtrand.pyx to add_newdocs.py. Will > this require any fixes to the doc editor so that the docs don't get > readded to mtrand? Go ahead, nothing should break. -- Pauli Virtanen From pav at iki.fi Wed Apr 15 18:18:52 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 15 Apr 2009 22:18:52 +0000 (UTC) Subject: [Numpy-discussion] Numpy Trac site redirecting in a loop? References: <49E64FF7.3050804@jhu.edu> Message-ID: Wed, 15 Apr 2009 17:21:59 -0400, Ken Basye wrote: > Just now when I tried to access http://projects.scipy.org/numpy, > Firefox complains (after a bit) that there's a redirect loop. Is anyone > else seeing this problem, and/or does anyone know how I can fix my side > of things to get me to the site? I don't see this. Can you try emptying your browser's cache? -- Pauli Virtanen From robert.kern at gmail.com Wed Apr 15 18:50:13 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 15 Apr 2009 17:50:13 -0500 Subject: [Numpy-discussion] Question for Pauli or Stefan about doc editor. In-Reply-To: References: Message-ID: <3d375d730904151550n664d6e72i8a356ad96eafbf1a@mail.gmail.com> On Wed, Apr 15, 2009 at 16:27, Charles R Harris wrote: > Hi Pauli, > > I'm going to move the docstrings from mtrand.pyx to add_newdocs.py. Will > this require any fixes to the doc editor so that the docs don't get readded > to mtrand? IIRC, David has told me that recent Cythons will do the right thing with the docstrings. It would be easier simply to build mtrand.c with Cython instead of Pyrex. It would be cleaner to keep the docstrings attached to the source. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From kbasye1 at jhu.edu Wed Apr 15 18:54:02 2009 From: kbasye1 at jhu.edu (Ken Basye) Date: Wed, 15 Apr 2009 18:54:02 -0400 Subject: [Numpy-discussion] Numpy Trac site redirecting in a loop? In-Reply-To: References: Message-ID: <49E6658A.1090004@jhu.edu> Hi Pauli, and thanks. I tried clearing the cache (no luck), and then logging in from a completely different machine with a completely re-initialized browser. I can get to the site, but as soon as I try to log in, the same thing (redirect loop) happens. I should point out that just before this happened I had the site reset my password. The URL it seems to be trying to get to after the login dialog is dismissed is: http://projects.scipy.org/numpy/prefs/account Back on the original machine, I was able to get the main page loading again by clearing all the cookies and stored passwords from scipy.org, but the same thing happened when I tried to log into the site. Thanks again, Ken > > Just now when I tried to access http://projects.scipy.org/numpy, > > Firefox complains (after a bit) that there's a redirect loop. Is anyone > > else seeing this problem, and/or does anyone know how I can fix my side > > of things to get me to the site? I don't see this. Can you try emptying your browser's cache? -- Pauli Virtanen Wed, 15 Apr 2009 17:21:59 -0400, Ken Basye wrote: -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Apr 15 20:03:02 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 15 Apr 2009 18:03:02 -0600 Subject: [Numpy-discussion] Question for Pauli or Stefan about doc editor. In-Reply-To: <3d375d730904151550n664d6e72i8a356ad96eafbf1a@mail.gmail.com> References: <3d375d730904151550n664d6e72i8a356ad96eafbf1a@mail.gmail.com> Message-ID: On Wed, Apr 15, 2009 at 4:50 PM, Robert Kern wrote: > On Wed, Apr 15, 2009 at 16:27, Charles R Harris > wrote: > > Hi Pauli, > > > > I'm going to move the docstrings from mtrand.pyx to add_newdocs.py. Will > > this require any fixes to the doc editor so that the docs don't get > readded > > to mtrand? > > IIRC, David has told me that recent Cythons will do the right thing > with the docstrings. It would be easier simply to build mtrand.c with > Cython instead of Pyrex. It would be cleaner to keep the docstrings > attached to the source. > Are Cython and Pyrex still close enough to be interchangeable? I also like the idea of having all the docs in add_newdocs so that no one has to go looking for them. I would even prefer having the python function docs there ;) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Apr 15 20:05:49 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 15 Apr 2009 19:05:49 -0500 Subject: [Numpy-discussion] Question for Pauli or Stefan about doc editor. In-Reply-To: References: <3d375d730904151550n664d6e72i8a356ad96eafbf1a@mail.gmail.com> Message-ID: <3d375d730904151705s20dd4397i71e5cf1369bc69e2@mail.gmail.com> On Wed, Apr 15, 2009 at 19:03, Charles R Harris wrote: > > On Wed, Apr 15, 2009 at 4:50 PM, Robert Kern wrote: >> >> On Wed, Apr 15, 2009 at 16:27, Charles R Harris >> wrote: >> > Hi Pauli, >> > >> > I'm going to move the docstrings from mtrand.pyx to add_newdocs.py. Will >> > this require any fixes to the doc editor so that the docs don't get >> > readded >> > to mtrand? >> >> IIRC, David has told me that recent Cythons will do the right thing >> with the docstrings. It would be easier simply to build mtrand.c with >> Cython instead of Pyrex. It would be cleaner to keep the docstrings >> attached to the source. > > Are Cython and Pyrex still close enough to be interchangeable? David and others have built it so before. > I also like > the idea of having all the docs in add_newdocs so that no one has to go > looking for them. I would even prefer having the python function docs there > ;) Ugh. I really, really don't. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Wed Apr 15 20:13:44 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 15 Apr 2009 18:13:44 -0600 Subject: [Numpy-discussion] Question for Pauli or Stefan about doc editor. In-Reply-To: <3d375d730904151705s20dd4397i71e5cf1369bc69e2@mail.gmail.com> References: <3d375d730904151550n664d6e72i8a356ad96eafbf1a@mail.gmail.com> <3d375d730904151705s20dd4397i71e5cf1369bc69e2@mail.gmail.com> Message-ID: On Wed, Apr 15, 2009 at 6:05 PM, Robert Kern wrote: > On Wed, Apr 15, 2009 at 19:03, Charles R Harris > wrote: > > > > On Wed, Apr 15, 2009 at 4:50 PM, Robert Kern > wrote: > >> > >> On Wed, Apr 15, 2009 at 16:27, Charles R Harris > >> wrote: > >> > Hi Pauli, > >> > > >> > I'm going to move the docstrings from mtrand.pyx to add_newdocs.py. > Will > >> > this require any fixes to the doc editor so that the docs don't get > >> > readded > >> > to mtrand? > >> > >> IIRC, David has told me that recent Cythons will do the right thing > >> with the docstrings. It would be easier simply to build mtrand.c with > >> Cython instead of Pyrex. It would be cleaner to keep the docstrings > >> attached to the source. > > > > Are Cython and Pyrex still close enough to be interchangeable? > > David and others have built it so before. > > > I also like > > the idea of having all the docs in add_newdocs so that no one has to go > > looking for them. I would even prefer having the python function docs > there > > ;) > > Ugh. I really, really don't. > OK, I just ran mtrand through cython-0.9.8.1.1, which is probably horribly outdated, and it does break up the strings. So that is a possibility. Anyone else want to express a preference as to where the docs should go? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Wed Apr 15 20:15:31 2009 From: Chris.Barker at noaa.gov (Chris.Barker at noaa.gov) Date: Wed, 15 Apr 2009 17:15:31 -0700 Subject: [Numpy-discussion] Leopard install Message-ID: <9471188db05a201.49e61633@noaa.gov> David Cournapeau wrote: > Chris.Barker at noaa.gov wrote: >> I think I'd prefer to patch bdist_mpkg, rather than re-writing or >> forking. > If you manage to do what you want as a command, then yes, forking does > not make any sense. It is just that I am tired of wasting my time with > distutils-related codebase :) well, this is py2app code, not distutils, thought I don't know that that's any better ;-). Anyway, a few small changes are easy -- I don't know that'll I'll have time or motivation for much else anyway. - Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Wed Apr 15 21:28:13 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 15 Apr 2009 19:28:13 -0600 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <7F347D91614EBC48AA7540A2A76D3BB204484075@MXCU10MX1.MSX.CIB> References: <7F347D91614EBC48AA7540A2A76D3BB20448406D@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB20448406E@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB204484070@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB204484073@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB204484075@MXCU10MX1.MSX.CIB> Message-ID: Hi Sal, On Wed, Apr 15, 2009 at 1:03 PM, Fadhley Salim wrote: > http://projects.scipy.org/numpy/ticket/1086 > > I filed this under numpy since that's what I am trying to compile. Please > feel free to correct if you think I've mis-described the fault. I'm heading > home now. > > My sinceerest thanks to everybody in the group who's helped me with this > today. > > I've regenerated mtrand.c using cython so that the comment lines are broken up. The svn revision is r6871, so could you download that and give it a shot? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Wed Apr 15 22:19:44 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 16 Apr 2009 11:19:44 +0900 Subject: [Numpy-discussion] Question for Pauli or Stefan about doc editor. In-Reply-To: References: <3d375d730904151550n664d6e72i8a356ad96eafbf1a@mail.gmail.com> <3d375d730904151705s20dd4397i71e5cf1369bc69e2@mail.gmail.com> Message-ID: <5b8d13220904151919j16c6e499qd20ebb71c6c7a7d4@mail.gmail.com> On Thu, Apr 16, 2009 at 9:13 AM, Charles R Harris wrote: > > OK, I just ran mtrand through cython-0.9.8.1.1,? which is probably horribly > outdated, and it does break up the strings. Please use cython 0.10.4 or above - below versions have a bug/limitation which breaks 64 bits python 2.4. I don't know whether it applies to mtrand., but it causes segfaults with scipy.distance. cheers, David From charlesr.harris at gmail.com Wed Apr 15 22:42:01 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 15 Apr 2009 20:42:01 -0600 Subject: [Numpy-discussion] Question for Pauli or Stefan about doc editor. In-Reply-To: <5b8d13220904151919j16c6e499qd20ebb71c6c7a7d4@mail.gmail.com> References: <3d375d730904151550n664d6e72i8a356ad96eafbf1a@mail.gmail.com> <3d375d730904151705s20dd4397i71e5cf1369bc69e2@mail.gmail.com> <5b8d13220904151919j16c6e499qd20ebb71c6c7a7d4@mail.gmail.com> Message-ID: On Wed, Apr 15, 2009 at 8:19 PM, David Cournapeau wrote: > On Thu, Apr 16, 2009 at 9:13 AM, Charles R Harris > wrote: > > > > > OK, I just ran mtrand through cython-0.9.8.1.1, which is probably > horribly > > outdated, and it does break up the strings. > > Please use cython 0.10.4 or above - below versions have a > bug/limitation which breaks 64 bits python 2.4. I don't know whether > it applies to mtrand., but it causes segfaults with scipy.distance. > OK, I just committed a version generated with cython 0.11. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Thu Apr 16 01:45:27 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 16 Apr 2009 07:45:27 +0200 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <7F347D91614EBC48AA7540A2A76D3BB204484071@MXCU10MX1.MSX.CIB> References: <7F347D91614EBC48AA7540A2A76D3BB204484071@MXCU10MX1.MSX.CIB> Message-ID: <20090416054527.GF3274@phare.normalesup.org> On Wed, Apr 15, 2009 at 07:33:48PM +0100, Fadhley Salim wrote: > We deploy on a very large number of computers in 5 countries. > PEAK's Setuptools gives us the ability to do all this deployment > automatically. If we can package all of our dependancies as egg files > then setuptools automatically deploys the egg files from a bunch of > web-servers we have around the globe. It would be an enormous pain if we > had to go to every single machine and run an exe installer. PEAK's setuptools is fragile with complex packages. This is the core reason numpy is not distributed as an egg, as the implications on what numpy would have to do to 'fit' in an egg compromise numpy's instal quality (read: random crashes). David would tell you more about this, since he went through all this pain. Now, if you really want eggs to deploy (in lack of decent packaging solution), you can talk to Enthought. They deploy their EPD packages using eggs. It is going to cost you money, and AFAIK, they use a modified version of setuptools to do deployement. I haven't see any of the Enthought guys answer to your request, so I hope I have'nt missed something about their product, or company policy, that make it impossible to answer your request. HTH, Ga?l From robert.kern at gmail.com Thu Apr 16 01:50:59 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 16 Apr 2009 00:50:59 -0500 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <20090416054527.GF3274@phare.normalesup.org> References: <7F347D91614EBC48AA7540A2A76D3BB204484071@MXCU10MX1.MSX.CIB> <20090416054527.GF3274@phare.normalesup.org> Message-ID: <3d375d730904152250s7af963f5o28c7238dbbcc042d@mail.gmail.com> On Thu, Apr 16, 2009 at 00:45, Gael Varoquaux wrote: > On Wed, Apr 15, 2009 at 07:33:48PM +0100, Fadhley Salim wrote: >> We deploy on a very large number of computers in 5 countries. > >> PEAK's Setuptools gives us the ability to do all this deployment >> automatically. If we can package all of our dependancies as egg files >> then setuptools automatically deploys the egg files from a bunch of >> web-servers we have around the globe. It would be an enormous pain if we >> had to go to every single machine and run an exe installer. > > PEAK's setuptools is fragile with complex packages. This is the core > reason numpy is not distributed as an egg, as the implications on what > numpy would have to do to 'fit' in an egg compromise numpy's instal > quality (read: random crashes). David would tell you more about this, > since he went through all this pain. Nonsense. There is no reason that numpy *has* to be built with an optimized BLAS. We try hard to ensure that. It is David's desire to distribute numpy builds with optimized BLASes that does not fit into eggs, not numpy. Plain numpy eggs are really straightforward. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Thu Apr 16 02:27:24 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 16 Apr 2009 15:27:24 +0900 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <3d375d730904152250s7af963f5o28c7238dbbcc042d@mail.gmail.com> References: <7F347D91614EBC48AA7540A2A76D3BB204484071@MXCU10MX1.MSX.CIB> <20090416054527.GF3274@phare.normalesup.org> <3d375d730904152250s7af963f5o28c7238dbbcc042d@mail.gmail.com> Message-ID: <49E6CFCC.3080003@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > It is David's desire to > distribute numpy builds with optimized BLASes that does not fit into > eggs, not numpy. Plain numpy eggs are really straightforward. > There may be a misunderstanding: I would not mind distributing something without optimized BLAS. Not using atlas at all would be easier for me, but I think some people were against that when we discussed the matter some time ago. But I think it would be confusing to have an optimized numpy from .exe and an unoptimized numpy from eggs on the same machine. I am afraid we would have people complaining about this instead of complaining about the lack of eggs. David From robert.kern at gmail.com Thu Apr 16 02:54:56 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 16 Apr 2009 01:54:56 -0500 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <49E6CFCC.3080003@ar.media.kyoto-u.ac.jp> References: <7F347D91614EBC48AA7540A2A76D3BB204484071@MXCU10MX1.MSX.CIB> <20090416054527.GF3274@phare.normalesup.org> <3d375d730904152250s7af963f5o28c7238dbbcc042d@mail.gmail.com> <49E6CFCC.3080003@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730904152354k161794ect3798eeb55c76c490@mail.gmail.com> On Thu, Apr 16, 2009 at 01:27, David Cournapeau wrote: > Robert Kern wrote: >> It is David's desire to >> distribute numpy builds with optimized BLASes that does not fit into >> eggs, not numpy. Plain numpy eggs are really straightforward. > > There may be a misunderstanding: I would not mind distributing something > without optimized BLAS. Not using atlas at all would be easier for me, > but I think some people were against that when we discussed the matter > some time ago. But I think it would be confusing to have an optimized > numpy from .exe and an unoptimized numpy from eggs on the same machine. > I am afraid we would have people complaining about this instead of > complaining about the lack of eggs. I have found that people are more willing to accept that they have to do something different to get a technically chalenging feature (i.e. use a particular installer to get a safe, optimized BLAS) than to accept that a relatively straightforward feature is not available at all (i.e. egg binaries on PyPI). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Thu Apr 16 02:41:43 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 16 Apr 2009 15:41:43 +0900 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> Message-ID: <49E6D327.9040108@ar.media.kyoto-u.ac.jp> Fadhley Salim wrote: > I've been asked to provide Numpy & Scipy as python egg files. > Unfortunately Numpy and Scipy do not make official releases of their > product in .egg form for a Win32 platform - that means if I want eggs > then I have to compile them myself. > I think having a simple .exe installer that easy_install could run would be enough for your purpose (handling install_require dependencies). In that case, you just need to extract the arch specific installer from our official installer (the superpack .exe is compressed with 7zip), choose the most appropriate installer, and add this on your server. I think easy_install should then be able to automatically use the .exe to handle dependencies, cheers, David From gael.varoquaux at normalesup.org Thu Apr 16 03:03:08 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 16 Apr 2009 09:03:08 +0200 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <3d375d730904152250s7af963f5o28c7238dbbcc042d@mail.gmail.com> References: <7F347D91614EBC48AA7540A2A76D3BB204484071@MXCU10MX1.MSX.CIB> <20090416054527.GF3274@phare.normalesup.org> <3d375d730904152250s7af963f5o28c7238dbbcc042d@mail.gmail.com> Message-ID: <20090416070308.GB10698@phare.normalesup.org> On Thu, Apr 16, 2009 at 12:50:59AM -0500, Robert Kern wrote: > > PEAK's setuptools is fragile with complex packages. This is the core > > reason numpy is not distributed as an egg, as the implications on what > > numpy would have to do to 'fit' in an egg compromise numpy's instal > > quality (read: random crashes). David would tell you more about this, > > since he went through all this pain. > Nonsense. There is no reason that numpy *has* to be built with an > optimized BLAS. We try hard to ensure that. It is David's desire to > distribute numpy builds with optimized BLASes that does not fit into > eggs, not numpy. Plain numpy eggs are really straightforward. Fair enough. Ga?l From david at ar.media.kyoto-u.ac.jp Thu Apr 16 02:50:29 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 16 Apr 2009 15:50:29 +0900 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <3d375d730904152354k161794ect3798eeb55c76c490@mail.gmail.com> References: <7F347D91614EBC48AA7540A2A76D3BB204484071@MXCU10MX1.MSX.CIB> <20090416054527.GF3274@phare.normalesup.org> <3d375d730904152250s7af963f5o28c7238dbbcc042d@mail.gmail.com> <49E6CFCC.3080003@ar.media.kyoto-u.ac.jp> <3d375d730904152354k161794ect3798eeb55c76c490@mail.gmail.com> Message-ID: <49E6D535.3020506@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > > I have found that people are more willing to accept that they have to > do something different to get a technically chalenging feature (i.e. > use a particular installer to get a safe, optimized BLAS) than to > accept that a relatively straightforward feature is not available at > all (i.e. egg binaries on PyPI). > In that case, would a simple bdist_wininst-based installer available on pypi be enough ? Or do eggs have other benefit compared to .exe ? David From david at ar.media.kyoto-u.ac.jp Thu Apr 16 02:52:50 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 16 Apr 2009 15:52:50 +0900 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <49E6D535.3020506@ar.media.kyoto-u.ac.jp> References: <7F347D91614EBC48AA7540A2A76D3BB204484071@MXCU10MX1.MSX.CIB> <20090416054527.GF3274@phare.normalesup.org> <3d375d730904152250s7af963f5o28c7238dbbcc042d@mail.gmail.com> <49E6CFCC.3080003@ar.media.kyoto-u.ac.jp> <3d375d730904152354k161794ect3798eeb55c76c490@mail.gmail.com> <49E6D535.3020506@ar.media.kyoto-u.ac.jp> Message-ID: <49E6D5C2.1030203@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > Robert Kern wrote: > >> I have found that people are more willing to accept that they have to >> do something different to get a technically chalenging feature (i.e. >> use a particular installer to get a safe, optimized BLAS) than to >> accept that a relatively straightforward feature is not available at >> all (i.e. egg binaries on PyPI). >> >> > > In that case, would a simple bdist_wininst-based installer available on > pypi be enough ? Or do eggs have other benefit compared to .exe ? > (I meant in the context of easy_install, where numpy would be a dependency) David From robert.kern at gmail.com Thu Apr 16 03:14:19 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 16 Apr 2009 02:14:19 -0500 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <49E6D5C2.1030203@ar.media.kyoto-u.ac.jp> References: <7F347D91614EBC48AA7540A2A76D3BB204484071@MXCU10MX1.MSX.CIB> <20090416054527.GF3274@phare.normalesup.org> <3d375d730904152250s7af963f5o28c7238dbbcc042d@mail.gmail.com> <49E6CFCC.3080003@ar.media.kyoto-u.ac.jp> <3d375d730904152354k161794ect3798eeb55c76c490@mail.gmail.com> <49E6D535.3020506@ar.media.kyoto-u.ac.jp> <49E6D5C2.1030203@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730904160014m49c4b86ata6866ab825b9745f@mail.gmail.com> On Thu, Apr 16, 2009 at 01:52, David Cournapeau wrote: > David Cournapeau wrote: >> Robert Kern wrote: >> >>> I have found that people are more willing to accept that they have to >>> do something different to get a technically chalenging feature (i.e. >>> use a particular installer to get a safe, optimized BLAS) than to >>> accept that a relatively straightforward feature is not available at >>> all (i.e. egg binaries on PyPI). >> >> In that case, would a simple bdist_wininst-based installer available on >> pypi be enough ? Or do eggs have other benefit compared to .exe ? > > (I meant in the context of easy_install, where numpy would be a dependency) easy_install is documented to be able to find and convert a bdist_wininst .exe on the fly, so I believe that should be sufficient. It might be tricky ensuring that it finds the correct one on the Sourceforge page, though. You may need to ensure that the names of the optimized .exes won't be interpreted by easy_install as being bdist_wininst installers that it can use. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Thu Apr 16 03:02:01 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 16 Apr 2009 16:02:01 +0900 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <3d375d730904160014m49c4b86ata6866ab825b9745f@mail.gmail.com> References: <7F347D91614EBC48AA7540A2A76D3BB204484071@MXCU10MX1.MSX.CIB> <20090416054527.GF3274@phare.normalesup.org> <3d375d730904152250s7af963f5o28c7238dbbcc042d@mail.gmail.com> <49E6CFCC.3080003@ar.media.kyoto-u.ac.jp> <3d375d730904152354k161794ect3798eeb55c76c490@mail.gmail.com> <49E6D535.3020506@ar.media.kyoto-u.ac.jp> <49E6D5C2.1030203@ar.media.kyoto-u.ac.jp> <3d375d730904160014m49c4b86ata6866ab825b9745f@mail.gmail.com> Message-ID: <49E6D7E9.6000405@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > > easy_install is documented to be able to find and convert a > bdist_wininst .exe on the fly, so I believe that should be sufficient. > Yes, I've tried locally on a bdist_wininst exe, and easy_install could install it. I have not tested it from the network. > It might be tricky ensuring that it finds the correct one on the > Sourceforge page, though. You may need to ensure that the names of the > optimized .exes won't be interpreted by easy_install as being > bdist_wininst installers that it can use. > What about just pushing the non optimized bdist_wininst installer on pypi ? Is it guaranteed that easy_install would look there before anything else ? When running easy_install in virtual env, I can see it looks on pypi at first, but I don't know if that's something which I am supposed to rely on, David From gael.varoquaux at normalesup.org Thu Apr 16 03:22:11 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 16 Apr 2009 09:22:11 +0200 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <49E6D7E9.6000405@ar.media.kyoto-u.ac.jp> References: <7F347D91614EBC48AA7540A2A76D3BB204484071@MXCU10MX1.MSX.CIB> <20090416054527.GF3274@phare.normalesup.org> <3d375d730904152250s7af963f5o28c7238dbbcc042d@mail.gmail.com> <49E6CFCC.3080003@ar.media.kyoto-u.ac.jp> <3d375d730904152354k161794ect3798eeb55c76c490@mail.gmail.com> <49E6D535.3020506@ar.media.kyoto-u.ac.jp> <49E6D5C2.1030203@ar.media.kyoto-u.ac.jp> <3d375d730904160014m49c4b86ata6866ab825b9745f@mail.gmail.com> <49E6D7E9.6000405@ar.media.kyoto-u.ac.jp> Message-ID: <20090416072211.GD10698@phare.normalesup.org> On Thu, Apr 16, 2009 at 04:02:01PM +0900, David Cournapeau wrote: > What about just pushing the non optimized bdist_wininst installer on > pypi ? With a clear note saying that they are non optimised. G. From robert.kern at gmail.com Thu Apr 16 03:23:17 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 16 Apr 2009 02:23:17 -0500 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <49E6D7E9.6000405@ar.media.kyoto-u.ac.jp> References: <7F347D91614EBC48AA7540A2A76D3BB204484071@MXCU10MX1.MSX.CIB> <20090416054527.GF3274@phare.normalesup.org> <3d375d730904152250s7af963f5o28c7238dbbcc042d@mail.gmail.com> <49E6CFCC.3080003@ar.media.kyoto-u.ac.jp> <3d375d730904152354k161794ect3798eeb55c76c490@mail.gmail.com> <49E6D535.3020506@ar.media.kyoto-u.ac.jp> <49E6D5C2.1030203@ar.media.kyoto-u.ac.jp> <3d375d730904160014m49c4b86ata6866ab825b9745f@mail.gmail.com> <49E6D7E9.6000405@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730904160023o1d54a074xda27ae6d7cc5e21@mail.gmail.com> On Thu, Apr 16, 2009 at 02:02, David Cournapeau wrote: > Robert Kern wrote: >> >> easy_install is documented to be able to find and convert a >> bdist_wininst .exe on the fly, so I believe that should be sufficient. > > Yes, I've tried locally on a bdist_wininst exe, and easy_install could > install it. I have not tested it from the network. > >> It might be tricky ensuring that it finds the correct one on the >> Sourceforge page, though. You may need to ensure that the names of the >> optimized .exes won't be interpreted by easy_install as being >> bdist_wininst installers that it can use. > > What about just pushing the non optimized bdist_wininst installer on > pypi ? Is it guaranteed that easy_install would look there before > anything else ? When running easy_install in virtual env, I can see it > looks on pypi at first, but I don't know if that's something which I am > supposed to rely on, I think it will try to collect all of the possibilities by looking at PyPI and the download link, then decide on the best one. But it could be that if there are files on PyPI, it will only consider those. I don't know. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dmitrey.kroshko at scipy.org Thu Apr 16 05:14:45 2009 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Thu, 16 Apr 2009 02:14:45 -0700 (PDT) Subject: [Numpy-discussion] best way to get vector representation in a basis? Message-ID: <72f5c48d-28ad-4db3-bc84-a11eb31aa82e@g19g2000yql.googlegroups.com> Hi all, I have orthonormal set of vectors B = [b_0, b_1,..., b_k-1], b_i from R^n (k may be less than n), and vector a from R^n What is most efficient way in numpy to get r from R^n and c_0, ..., c_k-1 from R: a = c_0*b_0+...+c_k-1*b_k-1 + r (r is rest) Thank you in advance, D. From dmitrey.kroshko at scipy.org Thu Apr 16 05:06:18 2009 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Thu, 16 Apr 2009 02:06:18 -0700 (PDT) Subject: [Numpy-discussion] best way to get vector representation in a basis? Message-ID: <89257878-3e52-4628-9ef8-0184b1343213@h2g2000yqg.googlegroups.com> Hi all, I have orthonormal set of vectors B = [b_0, b_1,..., b_k-1], b_i from R^n (k may be less than n), and vector a from R^n What is most efficient way in numpy to get r from R^n and c_0, ..., c_k-1 from R: a = c_0*b_0+...+c_k-1*b_k-1 + r (r is rest) Thank you in advance, D. From aisaac at american.edu Thu Apr 16 10:33:02 2009 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 16 Apr 2009 10:33:02 -0400 Subject: [Numpy-discussion] best way to get vector representation in a basis? In-Reply-To: <89257878-3e52-4628-9ef8-0184b1343213@h2g2000yqg.googlegroups.com> References: <89257878-3e52-4628-9ef8-0184b1343213@h2g2000yqg.googlegroups.com> Message-ID: <49E7419E.7040906@american.edu> On 4/16/2009 5:06 AM dmitrey apparently wrote: > I have orthonormal set of vectors B = [b_0, b_1,..., b_k-1], > b_i from R^n (k may be less than n), and vector a from R^n > > What is most efficient way in numpy to get r from R^n and c_0, ..., > c_k-1 from R: > a = c_0*b_0+...+c_k-1*b_k-1 + r > (r is rest) I must misunderstand this problem: efficient would be to set c=0 and r=a. I would have said to use np.linalg.lstsq, if you would like r to be small, but if B is really orthonormal, you can just put c=B'a (since B'B is a k dimensional identity). But I suspect I have failed to understand what you are after. Alan Isaac From Fadhley.Salim at uk.calyon.com Thu Apr 16 10:50:44 2009 From: Fadhley.Salim at uk.calyon.com (Fadhley Salim) Date: Thu, 16 Apr 2009 15:50:44 +0100 Subject: [Numpy-discussion] Compiling for free on Windows32 References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu><7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB><5b8d13220904151217n53499a3eo317e6b8baa12d719@mail.gmail.com><1cd32cbb0904151242qd70eec6r78b570e87215d028@mail.gmail.com> Message-ID: <7F347D91614EBC48AA7540A2A76D3BB204484079@MXCU10MX1.MSX.CIB> I just installed the latest stable mingw, and made sure that the mingw bin directory is in my PATH. I used the command you suggested and got the following output: http://pastebin.com/m4aea512c Any suggestions as to what I might be doing wrong? I'm using standard cpython 2.4.4 from Python.org -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Pauli Virtanen Sent: 15 April 2009 20:55 To: numpy-discussion at scipy.org Subject: Re: [Numpy-discussion] Compiling for free on Windows32 I think it might work if you separate the compiling and bdist_egg steps: setupegg.py build --compiler=mingw32 bdist_egg Does this help? -- Pauli Virtanen _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- This email does not create a legal relationship between any member of the Cr=E9dit Agricole group and the recipient or constitute investment advice. The content of this email should not be copied or disclosed (in whole or part) to any other person. It may contain information which is confidential, privileged or otherwise protected from disclosure. If you are not the intended recipient, you should notify us and delete it from your system. Emails may be monitored, are not secure and may be amended, destroyed or contain viruses and in communicating with us such conditions are accepted. Any content which does not relate to business matters is not endorsed by us. Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et des Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. Calyon is incorporated in France with limited liability and registered in England & Wales. Registration number: FC008194. Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. This message and/or any attachments is intended for the sole use of its addressee. If you are not the addressee, please immediately notify the sender and then destroy the message. As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. All rights reserved. Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. Tous droits r?serv?s. From Fadhley.Salim at uk.calyon.com Thu Apr 16 11:03:05 2009 From: Fadhley.Salim at uk.calyon.com (Fadhley Salim) Date: Thu, 16 Apr 2009 16:03:05 +0100 Subject: [Numpy-discussion] Compiling for free on Windows32 References: <7F347D91614EBC48AA7540A2A76D3BB20448406D@MXCU10MX1.MSX.CIB><7F347D91614EBC48AA7540A2A76D3BB20448406E@MXCU10MX1.MSX.CIB><7F347D91614EBC48AA7540A2A76D3BB204484070@MXCU10MX1.MSX.CIB><7F347D91614EBC48AA7540A2A76D3BB204484073@MXCU10MX1.MSX.CIB><7F347D91614EBC48AA7540A2A76D3BB204484075@MXCU10MX1.MSX.CIB> Message-ID: <7F347D91614EBC48AA7540A2A76D3BB20448407A@MXCU10MX1.MSX.CIB> No joy yet - this time a completely different error. I just tried r6871 and r6872... http://pastebin.com/d4d240b36 ________________________________ From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Charles R Harris Sent: 16 April 2009 02:28 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Compiling for free on Windows32 Hi Sal, On Wed, Apr 15, 2009 at 1:03 PM, Fadhley Salim wrote: http://projects.scipy.org/numpy/ticket/1086 I filed this under numpy since that's what I am trying to compile. Please feel free to correct if you think I've mis-described the fault. I'm heading home now. My sinceerest thanks to everybody in the group who's helped me with this today. I've regenerated mtrand.c using cython so that the comment lines are broken up. The svn revision is r6871, so could you download that and give it a shot? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- This email does not create a legal relationship between any member of the Cr=E9dit Agricole group and the recipient or constitute investment advice. The content of this email should not be copied or disclosed (in whole or part) to any other person. It may contain information which is confidential, privileged or otherwise protected from disclosure. If you are not the intended recipient, you should notify us and delete it from your system. Emails may be monitored, are not secure and may be amended, destroyed or contain viruses and in communicating with us such conditions are accepted. Any content which does not relate to business matters is not endorsed by us. Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et des Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. Calyon is incorporated in France with limited liability and registered in England & Wales. Registration number: FC008194. Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. This message and/or any attachments is intended for the sole use of its addressee. If you are not the addressee, please immediately notify the sender and then destroy the message. As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. All rights reserved. Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. Tous droits r?serv?s. From josef.pktd at gmail.com Thu Apr 16 11:10:17 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 16 Apr 2009 11:10:17 -0400 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <7F347D91614EBC48AA7540A2A76D3BB20448407A@MXCU10MX1.MSX.CIB> References: <7F347D91614EBC48AA7540A2A76D3BB20448406E@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB204484070@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB204484073@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB204484075@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB20448407A@MXCU10MX1.MSX.CIB> Message-ID: <1cd32cbb0904160810q68fd4948t62638329fe0ce52d@mail.gmail.com> On Thu, Apr 16, 2009 at 11:03 AM, Fadhley Salim wrote: > No joy yet - this time a completely different error. I just tried r6871 and > r6872... > > http://pastebin.com/d4d240b36 do you have g77 included in mingw and on your path? Josef From Fadhley.Salim at uk.calyon.com Thu Apr 16 11:12:11 2009 From: Fadhley.Salim at uk.calyon.com (Fadhley Salim) Date: Thu, 16 Apr 2009 16:12:11 +0100 Subject: [Numpy-discussion] Compiling for free on Windows32 References: <7F347D91614EBC48AA7540A2A76D3BB204484071@MXCU10MX1.MSX.CIB> <20090416054527.GF3274@phare.normalesup.org> <3d375d730904152250s7af963f5o28c7238dbbcc042d@mail.gmail.com> Message-ID: <7F347D91614EBC48AA7540A2A76D3BB20448407B@MXCU10MX1.MSX.CIB> I agree with Robert There should be no reason on earth why you cannot use an Egg to package Numpy. Setuptools is not fragile, it's very stable but requires a bit of understanding. The problem I have is not packaging numpy but compiling it on Windows. If I could compile the C I'd have built the egg in moments. The fact that I cannot compile the current version of Numpy is almost certainly not the fault of setuptools! Regarding the stability of Numpy in an Egg-wrapper - we run numpy 1.1.1 as an egg on about 100 workstations, 20 servers and countless grid CPUs with no problems at all. It's exactly as stable as Numpy installed from an exe, only we do not have to pay somebody to manually install it on all those computers. -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Robert Kern Sent: 16 April 2009 06:51 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Compiling for free on Windows32 On Thu, Apr 16, 2009 at 00:45, Gael Varoquaux wrote: > On Wed, Apr 15, 2009 at 07:33:48PM +0100, Fadhley Salim wrote: >> We deploy on a very large number of computers in 5 countries. > >> PEAK's Setuptools gives us the ability to do all this deployment >> automatically. If we can package all of our dependancies as egg files >> then setuptools automatically deploys the egg files from a bunch of >> web-servers we have around the globe. It would be an enormous pain if >> we had to go to every single machine and run an exe installer. > > PEAK's setuptools is fragile with complex packages. This is the core > reason numpy is not distributed as an egg, as the implications on what > numpy would have to do to 'fit' in an egg compromise numpy's instal > quality (read: random crashes). David would tell you more about this, > since he went through all this pain. Nonsense. There is no reason that numpy *has* to be built with an optimized BLAS. We try hard to ensure that. It is David's desire to distribute numpy builds with optimized BLASes that does not fit into eggs, not numpy. Plain numpy eggs are really straightforward. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- This email does not create a legal relationship between any member of the Cr=E9dit Agricole group and the recipient or constitute investment advice. The content of this email should not be copied or disclosed (in whole or part) to any other person. It may contain information which is confidential, privileged or otherwise protected from disclosure. If you are not the intended recipient, you should notify us and delete it from your system. Emails may be monitored, are not secure and may be amended, destroyed or contain viruses and in communicating with us such conditions are accepted. Any content which does not relate to business matters is not endorsed by us. Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et des Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. Calyon is incorporated in France with limited liability and registered in England & Wales. Registration number: FC008194. Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. This message and/or any attachments is intended for the sole use of its addressee. If you are not the addressee, please immediately notify the sender and then destroy the message. As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. All rights reserved. Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. Tous droits r?serv?s. From cournape at gmail.com Thu Apr 16 11:13:27 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 17 Apr 2009 00:13:27 +0900 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <1cd32cbb0904160810q68fd4948t62638329fe0ce52d@mail.gmail.com> References: <7F347D91614EBC48AA7540A2A76D3BB204484070@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB204484073@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB204484075@MXCU10MX1.MSX.CIB> <7F347D91614EBC48AA7540A2A76D3BB20448407A@MXCU10MX1.MSX.CIB> <1cd32cbb0904160810q68fd4948t62638329fe0ce52d@mail.gmail.com> Message-ID: <5b8d13220904160813k759dcebfn434535ccee4bae2f@mail.gmail.com> On Fri, Apr 17, 2009 at 12:10 AM, wrote: > On Thu, Apr 16, 2009 at 11:03 AM, Fadhley Salim > wrote: >> No joy yet - this time a completely different error. I just tried r6871 and >> r6872... >> >> http://pastebin.com/d4d240b36 > > do you have g77 included in mingw and on your path? g77 should not be necessary, and is unlikely the cause of the current problem. The problem is that the configuration header is not built for some reason - I don't understand why exactly, and it should have failed much before. David From cournape at gmail.com Thu Apr 16 11:19:31 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 17 Apr 2009 00:19:31 +0900 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <7F347D91614EBC48AA7540A2A76D3BB20448407B@MXCU10MX1.MSX.CIB> References: <7F347D91614EBC48AA7540A2A76D3BB204484071@MXCU10MX1.MSX.CIB> <20090416054527.GF3274@phare.normalesup.org> <3d375d730904152250s7af963f5o28c7238dbbcc042d@mail.gmail.com> <7F347D91614EBC48AA7540A2A76D3BB20448407B@MXCU10MX1.MSX.CIB> Message-ID: <5b8d13220904160819g6f37b99do7503dbf529e17b70@mail.gmail.com> On Fri, Apr 17, 2009 at 12:12 AM, Fadhley Salim wrote: > I agree with Robert > > There should be no reason on earth why you cannot use an Egg to package > Numpy. Actually, there is, although it is not really an egg deficiency. We use ATLAS as blas/lapack, and ATLAS binaries are tuned for one architecture. So we have a so called superpack binary installer which contains several binaries, selects the best one on a given computer, and installs it. > Setuptools is not fragile, it's very stable but requires a bit of > understanding. setuptools and eggs are different things - the egg file format is fine, and should not cause any instability whatsoever compared to a normal binary install. David From Fadhley.Salim at uk.calyon.com Thu Apr 16 11:21:30 2009 From: Fadhley.Salim at uk.calyon.com (Fadhley Salim) Date: Thu, 16 Apr 2009 16:21:30 +0100 Subject: [Numpy-discussion] Compiling for free on Windows32 References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu><7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> <49E6D327.9040108@ar.media.kyoto-u.ac.jp> Message-ID: <7F347D91614EBC48AA7540A2A76D3BB20448407C@MXCU10MX1.MSX.CIB> As I said before we have a very big set up. The problem is not the difficulty in making eggs. Eggs are very easy things to make if you can make the C++ compile first. Setuptools is a very good and stable project whose purpose is to help automate very large deployments of Python dependancies. Your .exe approach is not a solution. Let me explain why. In a bank, every bit of software we run has to be certified by the relevant auhtorities. That means one project might require numpy 1.3.0.1 weheras another might require 1.1.1. We are not authorized to run any combination of dependancies that is not certified. But some projects that need to run on the same grid might have different Numpy dependancies. We need to be able to run both projects concurrently on the same grid. Setuptools + eggs allows this to happen. If we used .exe installers then we could only have one single version of any given dependancy at any time and so we would not be able to run the two projects in paralell. In summary: .exe files are fine for single-users and simple deployments but not suitable for very big deployments. Eggs may not be perfect but they are by far the best soltion we have. -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of David Cournapeau Sent: 16 April 2009 07:42 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Compiling for free on Windows32 I think having a simple .exe installer that easy_install could run would be enough for your purpose (handling install_require dependencies). In that case, you just need to extract the arch specific installer from our official installer (the superpack .exe is compressed with 7zip), choose the most appropriate installer, and add this on your server. I think easy_install should then be able to automatically use the .exe to handle dependencies, -------------- next part -------------- This email does not create a legal relationship between any member of the Cr=E9dit Agricole group and the recipient or constitute investment advice. The content of this email should not be copied or disclosed (in whole or part) to any other person. It may contain information which is confidential, privileged or otherwise protected from disclosure. If you are not the intended recipient, you should notify us and delete it from your system. Emails may be monitored, are not secure and may be amended, destroyed or contain viruses and in communicating with us such conditions are accepted. Any content which does not relate to business matters is not endorsed by us. Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et des Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. Calyon is incorporated in France with limited liability and registered in England & Wales. Registration number: FC008194. Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. This message and/or any attachments is intended for the sole use of its addressee. If you are not the addressee, please immediately notify the sender and then destroy the message. As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. All rights reserved. Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. Tous droits r?serv?s. From Fadhley.Salim at uk.calyon.com Thu Apr 16 11:25:52 2009 From: Fadhley.Salim at uk.calyon.com (Fadhley Salim) Date: Thu, 16 Apr 2009 16:25:52 +0100 Subject: [Numpy-discussion] Compiling for free on Windows32 References: <7F347D91614EBC48AA7540A2A76D3BB204484071@MXCU10MX1.MSX.CIB><20090416054527.GF3274@phare.normalesup.org> <3d375d730904152250s7af963f5o28c7238dbbcc042d@mail.gmail.com><49E6CFCC.3080003@ar.media.kyoto-u.ac.jp><3d375d730904152354k161794ect3798eeb55c76c490@mail.gmail.com> <49E6D535.3020506@ar.media.kyoto-u.ac.jp> Message-ID: <7F347D91614EBC48AA7540A2A76D3BB20448407D@MXCU10MX1.MSX.CIB> Eggs are still beneficial: * People who do not have access to PyPi can still use it (think banks) * If you can build eggs easily then so can people like me... it becomes easy to produce optimized eggs for all our Xeon processors. * Deploying an egg is nothing more than copying a file to the relevant folder - it's fast and very reliable. Since we deploy to end-user's PCs we need something that works very quickly. Making eggs is really really easy - the hard part is compiling the C++ stuff in Numpy reliably. :-) -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of David Cournapeau Sent: 16 April 2009 07:50 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Compiling for free on Windows32 Robert Kern wrote: > > I have found that people are more willing to accept that they have to > do something different to get a technically chalenging feature (i.e. > use a particular installer to get a safe, optimized BLAS) than to > accept that a relatively straightforward feature is not available at > all (i.e. egg binaries on PyPI). > In that case, would a simple bdist_wininst-based installer available on pypi be enough ? Or do eggs have other benefit compared to .exe ? David _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- This email does not create a legal relationship between any member of the Cr=E9dit Agricole group and the recipient or constitute investment advice. The content of this email should not be copied or disclosed (in whole or part) to any other person. It may contain information which is confidential, privileged or otherwise protected from disclosure. If you are not the intended recipient, you should notify us and delete it from your system. Emails may be monitored, are not secure and may be amended, destroyed or contain viruses and in communicating with us such conditions are accepted. Any content which does not relate to business matters is not endorsed by us. Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et des Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. Calyon is incorporated in France with limited liability and registered in England & Wales. Registration number: FC008194. Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. This message and/or any attachments is intended for the sole use of its addressee. If you are not the addressee, please immediately notify the sender and then destroy the message. As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. All rights reserved. Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. Tous droits r?serv?s. From cournape at gmail.com Thu Apr 16 11:25:54 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 17 Apr 2009 00:25:54 +0900 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <7F347D91614EBC48AA7540A2A76D3BB20448407C@MXCU10MX1.MSX.CIB> References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> <49E6D327.9040108@ar.media.kyoto-u.ac.jp> <7F347D91614EBC48AA7540A2A76D3BB20448407C@MXCU10MX1.MSX.CIB> Message-ID: <5b8d13220904160825n54f62c82xff4bb7a41c2d5929@mail.gmail.com> > We need to be able to run both projects concurrently on the same grid. > Setuptools + eggs allows this to happen. If we used .exe installers then > we could only have one single version of any given dependancy at any > time and so we would not be able to run the two projects in paralell. I think you are misunderstanding my suggestion - it would be strictly equivalent to installing an egg, because easy_install can build an egg from the .exe automatically. Numpy is installed as an egg from the .exe if you use easy_install. In particular you can use parallel multi-versions. cheers, David From cournape at gmail.com Thu Apr 16 11:34:18 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 17 Apr 2009 00:34:18 +0900 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <7F347D91614EBC48AA7540A2A76D3BB204484079@MXCU10MX1.MSX.CIB> References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> <5b8d13220904151217n53499a3eo317e6b8baa12d719@mail.gmail.com> <1cd32cbb0904151242qd70eec6r78b570e87215d028@mail.gmail.com> <7F347D91614EBC48AA7540A2A76D3BB204484079@MXCU10MX1.MSX.CIB> Message-ID: <5b8d13220904160834g60876bc4hc4e9271280c814c1@mail.gmail.com> On Thu, Apr 16, 2009 at 11:50 PM, Fadhley Salim wrote: > I just installed the latest stable mingw, and made sure that the mingw > bin directory is in my PATH. I used the command you suggested and got > the following output: > > http://pastebin.com/m4aea512c Could you make sure to remove entirely the build directory (rd /s /q build) and try again ? The problem is that some headers are not generated for some reasons, and that is often caused by distutils not being good at detecting obsolete sources. David From sccolbert at gmail.com Thu Apr 16 12:29:31 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Thu, 16 Apr 2009 12:29:31 -0400 Subject: [Numpy-discussion] problem using optimized libraries In-Reply-To: <49E50B27.1020008@jpl.nasa.gov> References: <49E50B27.1020008@jpl.nasa.gov> Message-ID: <7f014ea60904160929k3c6dfae4jcdb0462f787afc31@mail.gmail.com> if you built numpy from source with a site.cfg file pointing to you atlas libraries, numpy.dot() will use that library natively. no need to import _dotblas. Chris On Tue, Apr 14, 2009 at 6:16 PM, Mathew Yeates wrote: > Hi > The line > from _dotblas import dot ..... is giving me an import error when I > looked at the symbols in _dotblas.so I only see things like CFLOAT_dot. > When I trace the system calls I see that my optimized ATLAS libraries > are being accessed, but immediately after opening libatlas.so I get an > Import Error resulting in an unoptimized "dot" > > Anyone have any ideas? > > Mathew > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chanley at stsci.edu Thu Apr 16 12:59:40 2009 From: chanley at stsci.edu (Christopher Hanley) Date: Thu, 16 Apr 2009 12:59:40 -0400 Subject: [Numpy-discussion] Tabs in numpy-1.3 Message-ID: <49E763FC.1080700@stsci.edu> Hi, Our nightly build system has been detecting tabs in the recent versions of numpy. The following files appear to have issues: Checked out revision 6870. svn checkout ok PYTHON FILES INDENTED WITH TABS: ./numpy/numpy/distutils/fcompiler/compaq.py ./numpy/numpy/distutils/command/build_ext.py ./numpy/numpy/distutils/misc_util.py ./numpy/numpy/core/tests/test_scalarmath.py ./numpy/numpy/core/setup.py ./numpy/pavement.py tab check failed Chris -- Christopher Hanley Senior Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338 From stefan at sun.ac.za Thu Apr 16 13:32:17 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 16 Apr 2009 19:32:17 +0200 Subject: [Numpy-discussion] Tabs in numpy-1.3 In-Reply-To: <49E763FC.1080700@stsci.edu> References: <49E763FC.1080700@stsci.edu> Message-ID: <9457e7c80904161032q32b9cd51q8015bb968c1b5d30@mail.gmail.com> Thanks, Chris -- I fixed it in trunk. This is the kind of check we should be running on the buildbot. Regards St?fan 2009/4/16 Christopher Hanley : > Hi, > > Our nightly build system has been detecting tabs in the recent versions > of numpy. ?The following files appear to have issues: > > Checked out revision 6870. > svn checkout ok > PYTHON FILES INDENTED WITH TABS: > ./numpy/numpy/distutils/fcompiler/compaq.py > ./numpy/numpy/distutils/command/build_ext.py > ./numpy/numpy/distutils/misc_util.py > ./numpy/numpy/core/tests/test_scalarmath.py > ./numpy/numpy/core/setup.py > ./numpy/pavement.py > tab check failed From rowen at u.washington.edu Thu Apr 16 15:55:11 2009 From: rowen at u.washington.edu (Russell E. Owen) Date: Thu, 16 Apr 2009 12:55:11 -0700 Subject: [Numpy-discussion] numpy Mac binary for Python 2.6 Message-ID: Does anyone have a binary installer for numpy 1.3.0 and Python 2.6? I've been able to install from source and all tests passed, but I prefer official binaries because I have some confidence that there are no hidden dependencies (important for distributing self-contained apps). I tried to build a binary myself with bdist_mpkg, but it failed with the following (I appended the full traceback, this is just a summary): "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-pac kages/bdist_mpkg/tools.py", line 90, in get_gid raise ValueError('group %s not found' % (name,)) ValueError: group admin not found This is with bdist_mpkg 0.4.3 (official release), machllib and py2app from svn trunk (but same error with the official releases), svn 1.6.1, MacOS X 10.5.6 and export MACOSX_DEPLOYMENT_TARGET=10.4. Not sure why get_gid would be failing. The admin group certainly appears to exist and is the owner of /Library/Frameworks. -- Russell /usr/bin/mkbom build/bdist.macosx-10.3-fat/mpkg/platlib/Library/Frameworks/Python.framew ork/Versions/2.6/lib/python2.6/site-packages dist/numpy-1.3.0-py2.6-macosx10.5.mpkg/./Contents/Packages/numpy-platlib- 1.3.0-py2.6-macosx10.5.pkg/Contents/Archive.bom sh: /usr/bin/nidump: No such file or directory Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.6/bin/bdist_mpkg", line 8, in load_entry_point('bdist-mpkg==0.4.3', 'console_scripts', 'bdist_mpkg')() File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-pac kages/bdist_mpkg/script_bdist_mpkg.py", line 27, in main execfile(sys.argv[0], g, g) File "setup.py", line 172, in setup_package() File "setup.py", line 165, in setup_package configuration=configuration ) File "/Users/rowen/Archives?/PythonPackages/numpy-1.3.0/numpy/distutils/core.p y", line 184, in setup return old_setup(**new_attr) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutil s/core.py", line 152, in setup dist.run_commands() File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutil s/dist.py", line 975, in run_commands self.run_command(cmd) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutil s/dist.py", line 995, in run_command cmd_obj.run() File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-pac kages/bdist_mpkg/cmd_bdist_mpkg.py", line 443, in run self.make_scheme_package(scheme) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-pac kages/bdist_mpkg/cmd_bdist_mpkg.py", line 385, in make_scheme_package self.get_scheme_description(scheme), File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-pac kages/bdist_mpkg/pkg.py", line 157, in make_package admin = tools.admin_writable(prefix) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-pac kages/bdist_mpkg/tools.py", line 106, in admin_writable gid = get_gid('admin') File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-pac kages/bdist_mpkg/tools.py", line 90, in get_gid raise ValueError('group %s not found' % (name,)) ValueError: group admin not found From cournape at gmail.com Thu Apr 16 21:42:09 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 17 Apr 2009 10:42:09 +0900 Subject: [Numpy-discussion] numpy Mac binary for Python 2.6 In-Reply-To: References: Message-ID: <5b8d13220904161842k2f2f76c9v1dde62f4655c2e57@mail.gmail.com> On Fri, Apr 17, 2009 at 4:55 AM, Russell E. Owen wrote: > Does anyone have a binary installer for numpy 1.3.0 and Python 2.6? > > I've been able to install from source and all tests passed, but I prefer > official binaries because I have some confidence that there are no > hidden dependencies (important for distributing self-contained apps). > > I tried to build a binary myself with bdist_mpkg, but it failed with the > following (I appended the full traceback, this is just a summary): It is a bug of bdist_mpkg on leopard (the error message is a bit misleading - if you look at the code, you will see it calls for a command line utility which does not exist on leopard). See: http://www.pymvpa.org/devguide.html David From cournape at gmail.com Thu Apr 16 21:43:42 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 17 Apr 2009 10:43:42 +0900 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <5b8d13220904160825n54f62c82xff4bb7a41c2d5929@mail.gmail.com> References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> <49E6D327.9040108@ar.media.kyoto-u.ac.jp> <7F347D91614EBC48AA7540A2A76D3BB20448407C@MXCU10MX1.MSX.CIB> <5b8d13220904160825n54f62c82xff4bb7a41c2d5929@mail.gmail.com> Message-ID: <5b8d13220904161843n6b050d01ved165508aeeecf1d@mail.gmail.com> On Fri, Apr 17, 2009 at 12:25 AM, David Cournapeau wrote: >> We need to be able to run both projects concurrently on the same grid. >> Setuptools + eggs allows this to happen. If we used .exe installers then >> we could only have one single version of any given dependancy at any >> time and so we would not be able to run the two projects in paralell. > > I think you are misunderstanding my suggestion - it would be strictly > equivalent to installing an egg, because easy_install can build an egg > from the .exe automatically. Numpy is installed as an egg from the > .exe if you use easy_install. In particular you can use parallel > multi-versions. For the record, Fadhley could use this method and reported privately that it worked as expected once he extracted the bdist_wininst .exe from the superpack, David From charlesr.harris at gmail.com Fri Apr 17 00:07:35 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 16 Apr 2009 22:07:35 -0600 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <5b8d13220904161843n6b050d01ved165508aeeecf1d@mail.gmail.com> References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> <49E6D327.9040108@ar.media.kyoto-u.ac.jp> <7F347D91614EBC48AA7540A2A76D3BB20448407C@MXCU10MX1.MSX.CIB> <5b8d13220904160825n54f62c82xff4bb7a41c2d5929@mail.gmail.com> <5b8d13220904161843n6b050d01ved165508aeeecf1d@mail.gmail.com> Message-ID: On Thu, Apr 16, 2009 at 7:43 PM, David Cournapeau wrote: > On Fri, Apr 17, 2009 at 12:25 AM, David Cournapeau > wrote: > >> We need to be able to run both projects concurrently on the same grid. > >> Setuptools + eggs allows this to happen. If we used .exe installers then > >> we could only have one single version of any given dependancy at any > >> time and so we would not be able to run the two projects in paralell. > > > > I think you are misunderstanding my suggestion - it would be strictly > > equivalent to installing an egg, because easy_install can build an egg > > from the .exe automatically. Numpy is installed as an egg from the > > .exe if you use easy_install. In particular you can use parallel > > multi-versions. > > For the record, Fadhley could use this method and reported privately > that it worked as expected once he extracted the bdist_wininst .exe > from the superpack, > I'm curious if trunk compiled for him with VC++2003? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Fri Apr 17 00:37:05 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 17 Apr 2009 13:37:05 +0900 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <3d375d730904160023o1d54a074xda27ae6d7cc5e21@mail.gmail.com> References: <20090416054527.GF3274@phare.normalesup.org> <3d375d730904152250s7af963f5o28c7238dbbcc042d@mail.gmail.com> <49E6CFCC.3080003@ar.media.kyoto-u.ac.jp> <3d375d730904152354k161794ect3798eeb55c76c490@mail.gmail.com> <49E6D535.3020506@ar.media.kyoto-u.ac.jp> <49E6D5C2.1030203@ar.media.kyoto-u.ac.jp> <3d375d730904160014m49c4b86ata6866ab825b9745f@mail.gmail.com> <49E6D7E9.6000405@ar.media.kyoto-u.ac.jp> <3d375d730904160023o1d54a074xda27ae6d7cc5e21@mail.gmail.com> Message-ID: <5b8d13220904162137o46b988dcs14831f136d7b327f@mail.gmail.com> On Thu, Apr 16, 2009 at 4:23 PM, Robert Kern wrote: > > I think it will try to collect all of the possibilities by looking at > PyPI and the download link, then decide on the best one. But it could > be that if there are files on PyPI, it will only consider those. I > don't know. I ended up pushing the binary on pypi, and it looks like it is working, easy_install can retrieve it. At least from the output, easy_install looks on both pypi and sourceforge, but it picks up the binary on Pypi (the superpack installer is explicitly named such as to be incompatible with easy_install to avoid weird errors). cheers, David From robert.kern at gmail.com Fri Apr 17 00:38:35 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 16 Apr 2009 23:38:35 -0500 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <5b8d13220904162137o46b988dcs14831f136d7b327f@mail.gmail.com> References: <3d375d730904152250s7af963f5o28c7238dbbcc042d@mail.gmail.com> <49E6CFCC.3080003@ar.media.kyoto-u.ac.jp> <3d375d730904152354k161794ect3798eeb55c76c490@mail.gmail.com> <49E6D535.3020506@ar.media.kyoto-u.ac.jp> <49E6D5C2.1030203@ar.media.kyoto-u.ac.jp> <3d375d730904160014m49c4b86ata6866ab825b9745f@mail.gmail.com> <49E6D7E9.6000405@ar.media.kyoto-u.ac.jp> <3d375d730904160023o1d54a074xda27ae6d7cc5e21@mail.gmail.com> <5b8d13220904162137o46b988dcs14831f136d7b327f@mail.gmail.com> Message-ID: <3d375d730904162138k7895bf4ta3d1ef52d4d79b8c@mail.gmail.com> On Thu, Apr 16, 2009 at 23:37, David Cournapeau wrote: > On Thu, Apr 16, 2009 at 4:23 PM, Robert Kern wrote: > >> I think it will try to collect all of the possibilities by looking at >> PyPI and the download link, then decide on the best one. But it could >> be that if there are files on PyPI, it will only consider those. I >> don't know. > > I ended up pushing the binary on pypi, and it looks like it is > working, easy_install can retrieve it. At least from the output, > easy_install looks on both pypi and sourceforge, but it picks up the > binary on Pypi (the superpack installer is explicitly named such as to > be incompatible with easy_install to avoid weird errors). Great. Thanks! -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cournape at gmail.com Fri Apr 17 00:45:43 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 17 Apr 2009 13:45:43 +0900 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <3d375d730904162138k7895bf4ta3d1ef52d4d79b8c@mail.gmail.com> References: <49E6CFCC.3080003@ar.media.kyoto-u.ac.jp> <3d375d730904152354k161794ect3798eeb55c76c490@mail.gmail.com> <49E6D535.3020506@ar.media.kyoto-u.ac.jp> <49E6D5C2.1030203@ar.media.kyoto-u.ac.jp> <3d375d730904160014m49c4b86ata6866ab825b9745f@mail.gmail.com> <49E6D7E9.6000405@ar.media.kyoto-u.ac.jp> <3d375d730904160023o1d54a074xda27ae6d7cc5e21@mail.gmail.com> <5b8d13220904162137o46b988dcs14831f136d7b327f@mail.gmail.com> <3d375d730904162138k7895bf4ta3d1ef52d4d79b8c@mail.gmail.com> Message-ID: <5b8d13220904162145m397c26ecm2a4de68cd4795887@mail.gmail.com> On Fri, Apr 17, 2009 at 1:38 PM, Robert Kern wrote: > On Thu, Apr 16, 2009 at 23:37, David Cournapeau wrote: >> On Thu, Apr 16, 2009 at 4:23 PM, Robert Kern wrote: >> >>> I think it will try to collect all of the possibilities by looking at >>> PyPI and the download link, then decide on the best one. But it could >>> be that if there are files on PyPI, it will only consider those. I >>> don't know. >> >> I ended up pushing the binary on pypi, and it looks like it is >> working, easy_install can retrieve it. At least from the output, >> easy_install looks on both pypi and sourceforge, but it picks up the >> binary on Pypi (the superpack installer is explicitly named such as to >> be incompatible with easy_install to avoid weird errors). > > Great. Thanks! You're welcome. Since I am not an egg user myself, I wonder whether it would be useful to make numpy zip-safe ? Can it be done without numpy relying on setuptools, or do we have to use setuptools for pkg_resources (maybe pkg_resources can be used without using the whole setuptools machinery ?) cheers, David From robert.kern at gmail.com Fri Apr 17 00:48:15 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 16 Apr 2009 23:48:15 -0500 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <5b8d13220904162145m397c26ecm2a4de68cd4795887@mail.gmail.com> References: <3d375d730904152354k161794ect3798eeb55c76c490@mail.gmail.com> <49E6D535.3020506@ar.media.kyoto-u.ac.jp> <49E6D5C2.1030203@ar.media.kyoto-u.ac.jp> <3d375d730904160014m49c4b86ata6866ab825b9745f@mail.gmail.com> <49E6D7E9.6000405@ar.media.kyoto-u.ac.jp> <3d375d730904160023o1d54a074xda27ae6d7cc5e21@mail.gmail.com> <5b8d13220904162137o46b988dcs14831f136d7b327f@mail.gmail.com> <3d375d730904162138k7895bf4ta3d1ef52d4d79b8c@mail.gmail.com> <5b8d13220904162145m397c26ecm2a4de68cd4795887@mail.gmail.com> Message-ID: <3d375d730904162148k39f080ccjfdc07e9f9649fef1@mail.gmail.com> On Thu, Apr 16, 2009 at 23:45, David Cournapeau wrote: > Since I am not an egg user myself, I wonder whether it would be useful > to make numpy zip-safe ? Can it be done without numpy relying on > setuptools, or do we have to use setuptools for pkg_resources (maybe > pkg_resources can be used without using the whole setuptools machinery > ?) You don't have to use setuptools to build numpy in order to use pkg_resources, but setuptools would be a runtime dependency. I don't think it's particularly important, though. I wouldn't spend any time on it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Fadhley.Salim at uk.calyon.com Fri Apr 17 03:57:48 2009 From: Fadhley.Salim at uk.calyon.com (Fadhley Salim) Date: Fri, 17 Apr 2009 08:57:48 +0100 Subject: [Numpy-discussion] Compiling for free on Windows32 References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu><7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB><49E6D327.9040108@ar.media.kyoto-u.ac.jp><7F347D91614EBC48AA7540A2A76D3BB20448407C@MXCU10MX1.MSX.CIB><5b8d13220904160825n54f62c82xff4bb7a41c2d5929@mail.gmail.com><5b8d13220904161843n6b050d01ved165508aeeecf1d@mail.gmail.com> Message-ID: <7F347D91614EBC48AA7540A2A76D3BB204484085@MXCU10MX1.MSX.CIB> Yes, Thanks for all your support, it works a treat. My conclusion is that the .exe release for Windows are sufficient for our needs. ________________________________ From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Charles R Harris Sent: 17 April 2009 05:08 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Compiling for free on Windows32 On Thu, Apr 16, 2009 at 7:43 PM, David Cournapeau wrote: On Fri, Apr 17, 2009 at 12:25 AM, David Cournapeau wrote: >> We need to be able to run both projects concurrently on the same grid. >> Setuptools + eggs allows this to happen. If we used .exe installers then >> we could only have one single version of any given dependancy at any >> time and so we would not be able to run the two projects in paralell. > > I think you are misunderstanding my suggestion - it would be strictly > equivalent to installing an egg, because easy_install can build an egg > from the .exe automatically. Numpy is installed as an egg from the > .exe if you use easy_install. In particular you can use parallel > multi-versions. For the record, Fadhley could use this method and reported privately that it worked as expected once he extracted the bdist_wininst .exe from the superpack, I'm curious if trunk compiled for him with VC++2003? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- This email does not create a legal relationship between any member of the Cr=E9dit Agricole group and the recipient or constitute investment advice. The content of this email should not be copied or disclosed (in whole or part) to any other person. It may contain information which is confidential, privileged or otherwise protected from disclosure. If you are not the intended recipient, you should notify us and delete it from your system. Emails may be monitored, are not secure and may be amended, destroyed or contain viruses and in communicating with us such conditions are accepted. Any content which does not relate to business matters is not endorsed by us. Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et des Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. Calyon is incorporated in France with limited liability and registered in England & Wales. Registration number: FC008194. Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. This message and/or any attachments is intended for the sole use of its addressee. If you are not the addressee, please immediately notify the sender and then destroy the message. As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. All rights reserved. Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. Tous droits r?serv?s. From Fadhley.Salim at uk.calyon.com Fri Apr 17 04:02:54 2009 From: Fadhley.Salim at uk.calyon.com (Fadhley Salim) Date: Fri, 17 Apr 2009 09:02:54 +0100 Subject: [Numpy-discussion] Compiling for free on Windows32 References: <49E6CFCC.3080003@ar.media.kyoto-u.ac.jp><3d375d730904152354k161794ect3798eeb55c76c490@mail.gmail.com><49E6D535.3020506@ar.media.kyoto-u.ac.jp><49E6D5C2.1030203@ar.media.kyoto-u.ac.jp><3d375d730904160014m49c4b86ata6866ab825b9745f@mail.gmail.com><49E6D7E9.6000405@ar.media.kyoto-u.ac.jp><3d375d730904160023o1d54a074xda27ae6d7cc5e21@mail.gmail.com><5b8d13220904162137o46b988dcs14831f136d7b327f@mail.gmail.com><3d375d730904162138k7895bf4ta3d1ef52d4d79b8c@mail.gmail.com> <5b8d13220904162145m397c26ecm2a4de68cd4795887@mail.gmail.com> Message-ID: <7F347D91614EBC48AA7540A2A76D3BB204484086@MXCU10MX1.MSX.CIB> You actually need the package unpacked to use scipy.weave and make C extensions which use Numpy's .h files. We have found that Numpy + Scipy are perfectly zip-safe for users but not for developers. -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of David Cournapeau Sent: 17 April 2009 05:46 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Compiling for free on Windows32 On Fri, Apr 17, 2009 at 1:38 PM, Robert Kern wrote: > On Thu, Apr 16, 2009 at 23:37, David Cournapeau wrote: >> On Thu, Apr 16, 2009 at 4:23 PM, Robert Kern wrote: >> >>> I think it will try to collect all of the possibilities by looking >>> at PyPI and the download link, then decide on the best one. But it >>> could be that if there are files on PyPI, it will only consider >>> those. I don't know. >> >> I ended up pushing the binary on pypi, and it looks like it is >> working, easy_install can retrieve it. At least from the output, >> easy_install looks on both pypi and sourceforge, but it picks up the >> binary on Pypi (the superpack installer is explicitly named such as >> to be incompatible with easy_install to avoid weird errors). > > Great. Thanks! You're welcome. Since I am not an egg user myself, I wonder whether it would be useful to make numpy zip-safe ? Can it be done without numpy relying on setuptools, or do we have to use setuptools for pkg_resources (maybe pkg_resources can be used without using the whole setuptools machinery ?) cheers, David _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- This email does not create a legal relationship between any member of the Cr=E9dit Agricole group and the recipient or constitute investment advice. The content of this email should not be copied or disclosed (in whole or part) to any other person. It may contain information which is confidential, privileged or otherwise protected from disclosure. If you are not the intended recipient, you should notify us and delete it from your system. Emails may be monitored, are not secure and may be amended, destroyed or contain viruses and in communicating with us such conditions are accepted. Any content which does not relate to business matters is not endorsed by us. Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et des Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. Calyon is incorporated in France with limited liability and registered in England & Wales. Registration number: FC008194. Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. This message and/or any attachments is intended for the sole use of its addressee. If you are not the addressee, please immediately notify the sender and then destroy the message. As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. All rights reserved. Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. Tous droits r?serv?s. From sturla at molden.no Fri Apr 17 04:46:28 2009 From: sturla at molden.no (Sturla Molden) Date: Fri, 17 Apr 2009 10:46:28 +0200 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> Message-ID: <49E841E4.6040803@molden.no> On 4/15/2009 6:44 PM, Matthieu Brucher wrote: > There is a Python limitation for the compiler. There is a common misunderstanding that only VS2003 can be used to compile extension objects for Python 2.5. Don't believe it. There is no Python limitation for the compiler. There is a Python limitation for the C runtime (crt), but only if you share crt objects with Python. That is, if you e.g. open a FILE* pointer in C, create a Python file object from the pointer, and then read that file in Python. In this case Python would do a fread on the pointer, and it is your responsibility to make sure the same crt DLL does the fopen, fread and fclose. If you don't share any crt resources with Python, there is no limitation on the crt either. If you do a malloc and a corresponding free in C, it does not matter what crt you use. It may even be used by Python objects in the meanwhile. mingw (gcc, gfortran) can be used freely, and afaik is used to compile the official NumPy and SciPy releases for Windows. The gfortran team has made a binary installer for version 4.4 available in their Wiki. When using mingw you have to link with the the same crt as Python if crt resources are shared. That is -lmsvcr71 for Python 2.5 and -lmsvcr90 for Python 2.6. If crt resources are unshared, link with whatever crt you want. Sturla Molden From david at ar.media.kyoto-u.ac.jp Fri Apr 17 04:50:06 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 17 Apr 2009 17:50:06 +0900 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <49E841E4.6040803@molden.no> References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> <49E841E4.6040803@molden.no> Message-ID: <49E842BE.5060207@ar.media.kyoto-u.ac.jp> Sturla Molden wrote: > On 4/15/2009 6:44 PM, Matthieu Brucher wrote: > > >> There is a Python limitation for the compiler. >> > > There is a common misunderstanding that only VS2003 can be used to > compile extension objects for Python 2.5. Don't believe it. There is no > Python limitation for the compiler. > I think Matthieu meant you have to use VS2003 as a MS compiler. Mingw is obviously fine, since that's how numpy binaries are built for quite a long time (every single installer I built myself was built with mingw, and it was already the case before). It is somewhat a python limitation, in the sense that things can be done such as to limit the problems. That's exactly what has happened for py3k, where most IO-related code does not use the C runtime anymore, but the 'native' low-level, unbuffered system API (read/write and co on unix, equivalent on windows). You can see more details here: http://www.artima.com/weblogs/viewpost.jsp?thread=208549 or better if you have time on this video http://video.google.com/videoplay?docid=-6459339159268485356. It also means that python does not rely on the hopelessly broken implementation of the C library on windows anymore. Maybe things will be better on the whole C runtime business once we pull off numpy for py3k. > > If you don't share any crt resources with Python, there is no limitation > on the crt either. If you do a malloc and a corresponding free in C, it > does not matter what crt you use. If both happen in the same dll, then yes, malloc/free are ok. But it will obviously break otherwise - that problem is certainly not specific to windows, though. For file handles, it happens that almost every unix implementation of open returns the same number as used internally by the kernel, so they are trivially 'shareable' between libraries. That's not true on windows; on windows, win32 handles, as returned from CreateFile and the likes are shareable, but file id as returned by C open and the likes are not. So you have to use the win32 specific API to avoid this issue. > The gfortran team has > made a binary installer for version 4.4 available in their Wiki. > gfortran 4.4 is pre-released software, though, so take care. I know for sure that gfortran + MS compiler can mix on 32 bits (at least with VS 2005/VS2008 - There are some mingw bugs for VS 2003, which I have a workaround for, but it needs some more distutils fu to work correctly so it is not in the trunk yet). David From sturla at molden.no Fri Apr 17 05:48:57 2009 From: sturla at molden.no (Sturla Molden) Date: Fri, 17 Apr 2009 11:48:57 +0200 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <49E842BE.5060207@ar.media.kyoto-u.ac.jp> References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> <49E841E4.6040803@molden.no> <49E842BE.5060207@ar.media.kyoto-u.ac.jp> Message-ID: <49E85089.1050702@molden.no> On 4/17/2009 10:50 AM, David Cournapeau wrote: > I think Matthieu meant you have to use VS2003 as a MS compiler. Mingw is > obviously fine, since that's how numpy binaries are built for quite a > long time That is what I thought he meant as well, and it seems to be a widespread belief. The fact that numpy is built with mingw proves it wrong, obviously. I just wanted to point out what the real limitation is: don't mix and blend CRTs and CRT resources. This is obviously not a Windows or Python specific problem. > gfortran 4.4 is pre-released software, though, so take care. I know for > sure that gfortran + MS compiler can mix on 32 bits http://gcc.gnu.org/wiki/GFortranBinaries I use this 32 bit mingw binary to build my Cython and f2py extensions. I works like a charm. I have licenses for Intel compilers at work, but I prefer gfortran 4.4. Sturla Molden From gael.varoquaux at normalesup.org Fri Apr 17 05:50:36 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 17 Apr 2009 11:50:36 +0200 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <49E85089.1050702@molden.no> References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> <49E841E4.6040803@molden.no> <49E842BE.5060207@ar.media.kyoto-u.ac.jp> <49E85089.1050702@molden.no> Message-ID: <20090417095036.GE24080@phare.normalesup.org> On Fri, Apr 17, 2009 at 11:48:57AM +0200, Sturla Molden wrote: > I use this 32 bit mingw binary to build my Cython and f2py extensions. I > works like a charm. I have licenses for Intel compilers at work, but I > prefer gfortran 4.4. Could you elaborate on your reason. Ga?l From sturla at molden.no Fri Apr 17 06:06:06 2009 From: sturla at molden.no (Sturla Molden) Date: Fri, 17 Apr 2009 12:06:06 +0200 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <20090417095036.GE24080@phare.normalesup.org> References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> <49E841E4.6040803@molden.no> <49E842BE.5060207@ar.media.kyoto-u.ac.jp> <49E85089.1050702@molden.no> <20090417095036.GE24080@phare.normalesup.org> Message-ID: <49E8548E.3010508@molden.no> On 4/17/2009 11:50 AM, Gael Varoquaux wrote: > Could you elaborate on your reason. Probably silly reasons though... I have a distutils.cfg file and a build system set up that works. I don't want to bother setting up a different build system when I already have one that works. I can use the Intel compilers at home, but only if I have an open VPN connection to work. I think just using a free gcc compiler is easier. I don't want to remember switches for two compilers. Syntax for inline assembly is different (on Windows), requiring preprocessor magic and code duplication. Object files and libraries are named differently (on Windows). There are numerous resons, but they all boil down to me being lacy. S.M. From Gerry.Talbot at amd.com Fri Apr 17 06:20:40 2009 From: Gerry.Talbot at amd.com (Talbot, Gerry) Date: Fri, 17 Apr 2009 05:20:40 -0500 Subject: [Numpy-discussion] Indexing Message-ID: Hi, I stumbled across this anomaly, the assignment to ncycle[i][b] fails, yet ncycle[:][b] works and the final print works. Is this a known bug or limitation ? I am running numpy-1.2.1 with python 2.5.4 Gerry from numpy import * ncycle = arange(10) i = array([0, 1, 2, 3, 4, 5, 6, 7]) b = array([True,False]*2) ncycle[i][b] = 99 print ncycle ncycle[:][b] = 99 print ncycle print ncycle[i][b] -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Fri Apr 17 08:41:52 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 17 Apr 2009 14:41:52 +0200 Subject: [Numpy-discussion] Indexing In-Reply-To: References: Message-ID: <9457e7c80904170541k5a12c11fh7bef1eebe6d81a8e@mail.gmail.com> Hi Gerry 2009/4/17 Talbot, Gerry : > I stumbled across this anomaly, the assignment to ncycle[i][b] fails, yet > ncycle[:][b] works and the final print works.? Is this a known bug or > limitation ?? I am running numpy-1.2.1 with python 2.5.4 The fancy indexing (indexing with an array) in ncycle[i] makes a copy, so you are assigning to a temporary copy which disappears soon after. Regards St?fan From dan.s.towell+numpy at gmail.com Fri Apr 17 11:25:25 2009 From: dan.s.towell+numpy at gmail.com (Dan S) Date: Fri, 17 Apr 2009 16:25:25 +0100 Subject: [Numpy-discussion] Solving a memory leak in a numpy extension; PyArray_ContiguousFromObject Message-ID: <286e6b7c0904170825t56c44869u2fcc2eeeb8e1fda2@mail.gmail.com> Hi - I have written a numpy extension which works fine but has a memory leak. It takes a single array argument and returns a single scalar. After reducing the code down in order to chase the problem, I have the following: static PyObject * kdpee_pycall(PyObject *self, PyObject *args) { PyObject *input; PyArrayObject *array; int n, numdims; if (!PyArg_ParseTuple(args, "O", &input)) return NULL; // Ensure we have contiguous, 2D, floating-point data: array = (PyArrayObject*) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if(array==NULL){ printf("kdpee_py: nullness!\n"); return NULL; } PyArray_XDECREF(array); Py_DECREF(array); // destroy the contig array Py_DECREF(input); // is this needed? doc says no, but it seems to fix a reference-count-leak return PyFloat_FromDouble(3.1415927); // temporary } The test code is from numpy import * from kdpee import * import sys a = array( [[1,2,3,4,5,1,2,3,4,5], [6,7,8,9,0,6,7,8,9,0]]) sys.getrefcount(a) for i in range(5000): tmp = kdpee(a) sys.getrefcount(a) Every time I run this code I get "2" back from both calls to sys.getrefcount(a), but I'm still getting a memory leak of about 0.4 MB. What am I doing wrong? I'll be grateful for any suggestions you can offer. Thanks Dan P.S. My system is python 2.5.4 (from macports) on intel Mac 10.4.11, numpy provided by macports package "py25-numpy" v1.2.1 From Chris.Barker at noaa.gov Fri Apr 17 11:40:40 2009 From: Chris.Barker at noaa.gov (Chris Barker) Date: Fri, 17 Apr 2009 08:40:40 -0700 Subject: [Numpy-discussion] Compiling for free on Windows32 In-Reply-To: <7F347D91614EBC48AA7540A2A76D3BB20448407C@MXCU10MX1.MSX.CIB> References: <86444799-B2E5-4DC7-81A9-486E4A0C45E5@post.harvard.edu> <7F347D91614EBC48AA7540A2A76D3BB20448406C@MXCU10MX1.MSX.CIB> <49E6D327.9040108@ar.media.kyoto-u.ac.jp> <7F347D91614EBC48AA7540A2A76D3BB20448407C@MXCU10MX1.MSX.CIB> Message-ID: <49E8A2F8.1030602@noaa.gov> I know this is the wrong forum, but: Yet another reason Python itself needs a standard, supported way to handle package versioning. *sigh* -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Gerry.Talbot at amd.com Fri Apr 17 09:14:00 2009 From: Gerry.Talbot at amd.com (Talbot, Gerry) Date: Fri, 17 Apr 2009 08:14:00 -0500 Subject: [Numpy-discussion] Indexing In-Reply-To: <9457e7c80904170541k5a12c11fh7bef1eebe6d81a8e@mail.gmail.com> References: <9457e7c80904170541k5a12c11fh7bef1eebe6d81a8e@mail.gmail.com> Message-ID: Got it - Thanks Gerry -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of St?fan van der Walt Sent: Friday, April 17, 2009 8:42 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Indexing Hi Gerry 2009/4/17 Talbot, Gerry : > I stumbled across this anomaly, the assignment to ncycle[i][b] fails, yet > ncycle[:][b] works and the final print works.? Is this a known bug or > limitation ?? I am running numpy-1.2.1 with python 2.5.4 The fancy indexing (indexing with an array) in ncycle[i] makes a copy, so you are assigning to a temporary copy which disappears soon after. Regards St?fan _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From rowen at u.washington.edu Fri Apr 17 17:39:44 2009 From: rowen at u.washington.edu (Russell E. Owen) Date: Fri, 17 Apr 2009 14:39:44 -0700 Subject: [Numpy-discussion] numpy Mac binary for Python 2.6 References: <5b8d13220904161842k2f2f76c9v1dde62f4655c2e57@mail.gmail.com> Message-ID: In article <5b8d13220904161842k2f2f76c9v1dde62f4655c2e57 at mail.gmail.com>, David Cournapeau wrote: > On Fri, Apr 17, 2009 at 4:55 AM, Russell E. Owen > wrote: > > Does anyone have a binary installer for numpy 1.3.0 and Python 2.6? > > > > I've been able to install from source and all tests passed, but I prefer > > official binaries because I have some confidence that there are no > > hidden dependencies (important for distributing self-contained apps). > > > > I tried to build a binary myself with bdist_mpkg, but it failed with the > > following (I appended the full traceback, this is just a summary): > > It is a bug of bdist_mpkg on leopard (the error message is a bit > misleading - if you look at the code, you will see it calls for a > command line utility which does not exist on leopard). > > See: > > http://www.pymvpa.org/devguide.html > > David Thank you very much for the patch. -- Russell From rowen at u.washington.edu Fri Apr 17 18:43:04 2009 From: rowen at u.washington.edu (Russell E. Owen) Date: Fri, 17 Apr 2009 15:43:04 -0700 Subject: [Numpy-discussion] numpy Mac binary for Python 2.6 References: <5b8d13220904161842k2f2f76c9v1dde62f4655c2e57@mail.gmail.com> Message-ID: > In article > <5b8d13220904161842k2f2f76c9v1dde62f4655c2e57 at mail.gmail.com>, > David Cournapeau wrote: > > > It is a bug of bdist_mpkg on leopard (the error message is a bit > > misleading - if you look at the code, you will see it calls for a > > command line utility which does not exist on leopard). > > > > See: > > > > http://www.pymvpa.org/devguide.html The patch at the end of this document worked. I have a numpy 1.3.0 binary installer for Mac Python 2.6; if anyone wants to test it or serve it please contact me via email. -- Russell From dalcinl at gmail.com Fri Apr 17 20:36:38 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 17 Apr 2009 21:36:38 -0300 Subject: [Numpy-discussion] Solving a memory leak in a numpy extension; PyArray_ContiguousFromObject In-Reply-To: <286e6b7c0904170825t56c44869u2fcc2eeeb8e1fda2@mail.gmail.com> References: <286e6b7c0904170825t56c44869u2fcc2eeeb8e1fda2@mail.gmail.com> Message-ID: 1) Calling both PyArray_XDECREF(array) and Py_DECREF(array) is likely wrong. 2) Py_DECREF(input) should never be done. On Fri, Apr 17, 2009 at 12:25 PM, Dan S wrote: > Hi - > > I have written a numpy extension which works fine but has a memory > leak. It takes a single array argument and returns a single scalar. > After reducing the code down in order to chase the problem, I have the > following: > > static PyObject * kdpee_pycall(PyObject *self, PyObject *args) > { > ? ? ? ?PyObject *input; > ? ? ? ?PyArrayObject *array; > ? ? ? ?int n, numdims; > > ? ? ? ?if (!PyArg_ParseTuple(args, "O", &input)) > ? ? ? ? ? ? ? ?return NULL; > > ? ? ? ?// Ensure we have contiguous, 2D, floating-point data: > ? ? ? ?array = (PyArrayObject*) PyArray_ContiguousFromObject(input, > PyArray_DOUBLE, 2, 2); > > ? ? ? ?if(array==NULL){ > ? ? ? ? ? ? ? ?printf("kdpee_py: nullness!\n"); > ? ? ? ? ? ? ? ?return NULL; > ? ? ? ?} > ? ? ? ?PyArray_XDECREF(array); > ? ? ? ?Py_DECREF(array); // destroy the contig array > ? ? ? ?Py_DECREF(input); // is this needed? doc says no, but it seems to fix > a reference-count-leak > ? ? ? ?return PyFloat_FromDouble(3.1415927); // temporary > } > > > The test code is > > ? ? ?from numpy import * > ? ? ?from kdpee import * > ? ? ?import sys > ? ? ?a = array( [[1,2,3,4,5,1,2,3,4,5], [6,7,8,9,0,6,7,8,9,0]]) > ? ? ?sys.getrefcount(a) > ? ? ?for i in range(5000): tmp = kdpee(a) > > ? ? ?sys.getrefcount(a) > > Every time I run this code I get "2" back from both calls to > sys.getrefcount(a), but I'm still getting a memory leak of about 0.4 > MB. What am I doing wrong? > > I'll be grateful for any suggestions you can offer. > > Thanks > Dan > > P.S. My system is python 2.5.4 (from macports) on intel Mac 10.4.11, > numpy provided by macports package "py25-numpy" v1.2.1 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From charlesr.harris at gmail.com Fri Apr 17 21:37:47 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 17 Apr 2009 19:37:47 -0600 Subject: [Numpy-discussion] Solving a memory leak in a numpy extension; PyArray_ContiguousFromObject In-Reply-To: <286e6b7c0904170825t56c44869u2fcc2eeeb8e1fda2@mail.gmail.com> References: <286e6b7c0904170825t56c44869u2fcc2eeeb8e1fda2@mail.gmail.com> Message-ID: On Fri, Apr 17, 2009 at 9:25 AM, Dan S > wrote: > Hi - > > I have written a numpy extension which works fine but has a memory > leak. It takes a single array argument and returns a single scalar. > After reducing the code down in order to chase the problem, I have the > following: > > static PyObject * kdpee_pycall(PyObject *self, PyObject *args) > { > PyObject *input; > PyArrayObject *array; > int n, numdims; > > if (!PyArg_ParseTuple(args, "O", &input)) > return NULL; > > // Ensure we have contiguous, 2D, floating-point data: > array = (PyArrayObject*) PyArray_ContiguousFromObject(input, > PyArray_DOUBLE, 2, 2); > > if(array==NULL){ > printf("kdpee_py: nullness!\n"); > return NULL; > } > PyArray_XDECREF(array); ^^^^^^^^^^^^^^^^^^^ Not needed. > > Py_DECREF(array); // destroy the contig array > Py_DECREF(input); // is this needed? ^^^^^^^^^^^^^^^^^^^^ Shouldn't be, but there might be a bug somewhere which causes the reference count of input to be double incremented. Does the reference count in the test script increase without this line? Do array and input point to the same object? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Apr 17 21:44:28 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 17 Apr 2009 19:44:28 -0600 Subject: [Numpy-discussion] Solving a memory leak in a numpy extension; PyArray_ContiguousFromObject In-Reply-To: References: <286e6b7c0904170825t56c44869u2fcc2eeeb8e1fda2@mail.gmail.com> Message-ID: On Fri, Apr 17, 2009 at 7:37 PM, Charles R Harris wrote: > > > On Fri, Apr 17, 2009 at 9:25 AM, Dan S > > wrote: > >> Hi - >> >> I have written a numpy extension which works fine but has a memory >> leak. It takes a single array argument and returns a single scalar. >> After reducing the code down in order to chase the problem, I have the >> following: >> >> static PyObject * kdpee_pycall(PyObject *self, PyObject *args) >> { >> PyObject *input; >> PyArrayObject *array; >> int n, numdims; >> >> if (!PyArg_ParseTuple(args, "O", &input)) >> return NULL; >> >> // Ensure we have contiguous, 2D, floating-point data: >> array = (PyArrayObject*) PyArray_ContiguousFromObject(input, >> PyArray_DOUBLE, 2, 2); >> >> if(array==NULL){ >> printf("kdpee_py: nullness!\n"); >> return NULL; >> } >> PyArray_XDECREF(array); > > ^^^^^^^^^^^^^^^^^^^ > Not needed. > >> >> Py_DECREF(array); // destroy the contig array >> Py_DECREF(input); // is this needed? > > ^^^^^^^^^^^^^^^^^^^^ > > Shouldn't be, but there might be a bug somewhere which causes the reference > count of input to be double incremented. Does the reference count in the > test script increase without this line? Do array and input point to the same > object? > Oops, input in array are different types, so they won't point to the same thing. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Apr 17 21:47:47 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 17 Apr 2009 19:47:47 -0600 Subject: [Numpy-discussion] Solving a memory leak in a numpy extension; PyArray_ContiguousFromObject In-Reply-To: <286e6b7c0904170825t56c44869u2fcc2eeeb8e1fda2@mail.gmail.com> References: <286e6b7c0904170825t56c44869u2fcc2eeeb8e1fda2@mail.gmail.com> Message-ID: On Fri, Apr 17, 2009 at 9:25 AM, Dan S > wrote: > Hi - > > I have written a numpy extension which works fine but has a memory > leak. It takes a single array argument and returns a single scalar. > After reducing the code down in order to chase the problem, I have the > following: > > static PyObject * kdpee_pycall(PyObject *self, PyObject *args) > { > PyObject *input; > PyArrayObject *array; > int n, numdims; > > if (!PyArg_ParseTuple(args, "O", &input)) > return NULL; > > // Ensure we have contiguous, 2D, floating-point data: > array = (PyArrayObject*) PyArray_ContiguousFromObject(input, > PyArray_DOUBLE, 2, 2); > > if(array==NULL){ > printf("kdpee_py: nullness!\n"); > return NULL; > } > PyArray_XDECREF(array); > Py_DECREF(array); // destroy the contig array > Py_DECREF(input); // is this needed? doc says no, but it seems to > fix > a reference-count-leak > return PyFloat_FromDouble(3.1415927); // temporary > } > > > The test code is > > from numpy import * > from kdpee import * > import sys > a = array( [[1,2,3,4,5,1,2,3,4,5], [6,7,8,9,0,6,7,8,9,0]]) > sys.getrefcount(a) > for i in range(5000): tmp = kdpee(a) > > sys.getrefcount(a) > > Every time I run this code I get "2" back from both calls to > sys.getrefcount(a), but I'm still getting a memory leak of about 0.4 > MB. What am I doing wrong? > So about 80 bytes/iteration? How accurate is that .4 MB? does it change with the size of the input array? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Sat Apr 18 03:11:19 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 18 Apr 2009 16:11:19 +0900 Subject: [Numpy-discussion] Solving a memory leak in a numpy extension; PyArray_ContiguousFromObject In-Reply-To: <286e6b7c0904170825t56c44869u2fcc2eeeb8e1fda2@mail.gmail.com> References: <286e6b7c0904170825t56c44869u2fcc2eeeb8e1fda2@mail.gmail.com> Message-ID: <49E97D17.2090309@ar.media.kyoto-u.ac.jp> Dan S wrote: > Hi - > > I have written a numpy extension which works fine but has a memory > leak. It takes a single array argument and returns a single scalar. > After reducing the code down in order to chase the problem, I have the > following: > It does not answer your original question, but I would advise you to use cython to make your own extension, unless cython cannot do/is not practical to do what you want. Refcounting is quite difficult to get right, and refcount bugs are difficult to trackdown (they are even more difficult to track down than malloc/free problems in my experience). David From dmitrey.kroshko at scipy.org Sat Apr 18 06:29:33 2009 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Sat, 18 Apr 2009 03:29:33 -0700 (PDT) Subject: [Numpy-discussion] does numpy/scipy have solver for Ax=b, L_inf (Chebyshev norm)? Message-ID: <7b4daef6-3bb9-4aa3-9706-b32799481b33@g37g2000yqn.googlegroups.com> Hi all, does numpy/scipy, or maybe wrapper for a lapack routine have solver for Ax=b, L_inf (Chebyshev norm, i.e. max |Ax-b| -> min)? If there are several ones, which ones are most suitable for large-scale, maybe ill- conditioned problems? Thank you in advance, D. P.S. Currently I 'm not interested in translating the problem to LP here, I search for more specialized solvers. From gruben at bigpond.net.au Sat Apr 18 07:03:48 2009 From: gruben at bigpond.net.au (Gary Ruben) Date: Sat, 18 Apr 2009 21:03:48 +1000 Subject: [Numpy-discussion] help with applying a transform Message-ID: <49E9B394.8080809@bigpond.net.au> I'm trying to work out how to apply a 2x2 transform matrix to a spinor, e.g. [psi1'] [a b][psi1] [ ] = [ ][ ] [psi2'] [c d][psi2] where [[a,b],[c,d]] is a transform matrix and psi1 and psi2 are i x j x k complex arrays representing complex scalar field data. I worked that one way to do it with 2D fields (psi1 and psi2 being 2D, i x j arrays) is def transform(tx_matrix, psi1, psi2): psi = np.dot(tx_matrix, np.rollaxis(np.dstack((psi1,psi2)),2,1)) return psi[0], psi[1] or, equivalently def transform(tx_matrix, psi1, psi2): psi = np.dot(tx_matrix, np.rollaxis( np.concatenate((psi1[newaxis],psi2[newaxis])),1)) return psi[0], psi[1] but, as seems usual for me, I'm confused with trying to extend this to the next higher dimension; 3D in this case. It seems to me that there might be a neater way to do this, i.e. some built-in feature of an existing numpy function. How do I extend this for 3D psi1 and psi2 arrays? Are there any general tips or guides for helping to minimise confusion when it comes to manipulating axes like this, such as standard ways to extend 2D recipes to 3D? Gary R. From gruben at bigpond.net.au Sat Apr 18 07:17:01 2009 From: gruben at bigpond.net.au (Gary Ruben) Date: Sat, 18 Apr 2009 21:17:01 +1000 Subject: [Numpy-discussion] help with applying a transform In-Reply-To: <49E9B394.8080809@bigpond.net.au> References: <49E9B394.8080809@bigpond.net.au> Message-ID: <49E9B6AD.2080905@bigpond.net.au> I think I've answered my own question - I remembered tensordot, and the following seems to work: def transform(tx_matrix, psi1, psi2): psi = np.tensordot(tx_matrix, np.concatenate((psi1[newaxis],psi2[newaxis])),axes=1)) return psi[0], psi[1] sorry for the noise, Gary Gary Ruben wrote: > I'm trying to work out how to apply a 2x2 transform matrix to a spinor, e.g. > > [psi1'] [a b][psi1] > [ ] = [ ][ ] > [psi2'] [c d][psi2] > > where [[a,b],[c,d]] is a transform matrix and psi1 and psi2 are > i x j x k complex arrays representing complex scalar field data. I > worked that one way to do it with 2D fields (psi1 and psi2 being 2D, > i x j arrays) is > > def transform(tx_matrix, psi1, psi2): > psi = np.dot(tx_matrix, np.rollaxis(np.dstack((psi1,psi2)),2,1)) > return psi[0], psi[1] > > or, equivalently > > def transform(tx_matrix, psi1, psi2): > psi = np.dot(tx_matrix, np.rollaxis( > np.concatenate((psi1[newaxis],psi2[newaxis])),1)) > return psi[0], psi[1] > > but, as seems usual for me, I'm confused with trying to extend this to > the next higher dimension; 3D in this case. It seems to me that there > might be a neater way to do this, i.e. some built-in feature of an > existing numpy function. How do I extend this for 3D psi1 and psi2 > arrays? Are there any general tips or guides for helping to minimise > confusion when it comes to manipulating axes like this, such as standard > ways to extend 2D recipes to 3D? > > Gary R. > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From dagss at student.matnat.uio.no Sat Apr 18 09:36:36 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 18 Apr 2009 15:36:36 +0200 Subject: [Numpy-discussion] Struct alignment Message-ID: <49E9D764.8020602@student.matnat.uio.no> I'm looking at fixing up support for NumPy record arrays in Cython, and realize my knowledge of NumPy is a bit too limited. Searching the docs didn't enlighten me, so: Does NumPy always use packed alignment for nested dtypes? In [35]: T Out[35]: dtype([('f0', ' References: <7b4daef6-3bb9-4aa3-9706-b32799481b33@g37g2000yqn.googlegroups.com> Message-ID: On Sat, Apr 18, 2009 at 4:29 AM, dmitrey wrote: > Hi all, > does numpy/scipy, or maybe wrapper for a lapack routine have solver > for Ax=b, L_inf (Chebyshev norm, i.e. max |Ax-b| -> min)? If there are > several ones, which ones are most suitable for large-scale, maybe ill- > conditioned problems? The Remez algorithm will solve such problems if the columns of A form a Chebyshev system, but the only version of that algorithm in scipy is specialized to FIR filter design. I have a python version floating about somewhere from many years ago. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sat Apr 18 11:53:52 2009 From: cournape at gmail.com (David Cournapeau) Date: Sun, 19 Apr 2009 00:53:52 +0900 Subject: [Numpy-discussion] Struct alignment In-Reply-To: <49E9D764.8020602@student.matnat.uio.no> References: <49E9D764.8020602@student.matnat.uio.no> Message-ID: <5b8d13220904180853u62ff8415gbf71762c9fc7beb5@mail.gmail.com> On Sat, Apr 18, 2009 at 10:36 PM, Dag Sverre Seljebotn wrote: > I'm looking at fixing up support for NumPy record arrays in Cython, and > realize my knowledge of NumPy is a bit too limited. Searching the docs > didn't enlighten me, so: > > Does NumPy always use packed alignment for nested dtypes? No, you may have to pad "manually" depending on what you want. Sometimes, the align argument is enough, though: a = dtype([('f0', ' I updated my Thinkpad to Ubuntu 9.04 release candidate, 32-bit, got a fresh svn checkout of numpy trunk, built, and installed. Running numpy.test() yields 16 errors (output attached) in linear algebra on complex arguments. Is this expected? Eric -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy_test URL: From charlesr.harris at gmail.com Sat Apr 18 16:21:01 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 18 Apr 2009 14:21:01 -0600 Subject: [Numpy-discussion] svn test failures: linalg with complex In-Reply-To: <49EA2CE8.5030602@hawaii.edu> References: <49EA2CE8.5030602@hawaii.edu> Message-ID: On Sat, Apr 18, 2009 at 1:41 PM, Eric Firing wrote: > I updated my Thinkpad to Ubuntu 9.04 release candidate, 32-bit, got a fresh > svn checkout of numpy trunk, built, and installed. Running numpy.test() > yields 16 errors (output attached) in linear algebra on complex arguments. > Is this expected? No, there is something wrong with your build, LAPACK, ATLAS, or lapack-lite isn't working right. What compiler version is used? Did you install ATLAS from ubuntu? It might help if you compress the build log and attach it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sat Apr 18 16:34:35 2009 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 18 Apr 2009 20:34:35 +0000 (UTC) Subject: [Numpy-discussion] Buildbot unavailable Message-ID: Hi, buildbot.scipy.org says: Service Temporarily Unavailable The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later. -- Pauli Virtanen From efiring at hawaii.edu Sat Apr 18 17:10:03 2009 From: efiring at hawaii.edu (Eric Firing) Date: Sat, 18 Apr 2009 11:10:03 -1000 Subject: [Numpy-discussion] svn test failures: linalg with complex In-Reply-To: References: <49EA2CE8.5030602@hawaii.edu> Message-ID: <49EA41AB.10907@hawaii.edu> Charles R Harris wrote: > > > On Sat, Apr 18, 2009 at 1:41 PM, Eric Firing > wrote: > > I updated my Thinkpad to Ubuntu 9.04 release candidate, 32-bit, got > a fresh svn checkout of numpy trunk, built, and installed. Running > numpy.test() yields 16 errors (output attached) in linear algebra on > complex arguments. Is this expected? > > > > > No, there is something wrong with your build, LAPACK, ATLAS, or > lapack-lite isn't working right. What compiler version is used? Did you > install ATLAS from ubuntu? It might help if you compress the build log > and attach it. Attached is the redirected stdout. As far as I can see, all libraries are the ubuntu versions, and compilers are the ubuntu defaults. The Atlas library appears to have been built long ago. Eric > > Chuck > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: build.log.gz Type: application/x-gzip Size: 5845 bytes Desc: not available URL: From pav at iki.fi Sat Apr 18 17:46:01 2009 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 18 Apr 2009 21:46:01 +0000 (UTC) Subject: [Numpy-discussion] svn test failures: linalg with complex References: <49EA2CE8.5030602@hawaii.edu> Message-ID: Sat, 18 Apr 2009 09:41:28 -1000, Eric Firing wrote: > I updated my Thinkpad to Ubuntu 9.04 release candidate, 32-bit, got a > fresh svn checkout of numpy trunk, built, and installed. Running > numpy.test() yields 16 errors (output attached) in linear algebra on > complex arguments. Is this expected? The Numpy test suite should run without any failures. The failures here likely indicate that something is wrong with LAPACK/BLAS/ATLAS on Ubuntu 9.04. Apparently, at least ATLAS's complex single-precision routines are completely hosed on that platforms. I filed a bug here: https://bugs.launchpad.net/ubuntu/+source/atlas/+bug/363510 Seems specific to the SSE version of Atlas. Could you comment on the bug tracker if you see the same thing on your machine? Consequently, on 9.04 I get the following failures, but for some reason these are not the same as yours: Running unit tests for numpy-1.4.0.dev6826.4e95223c11c627aa8f4878b40953d75ea8228add-py2.5-linux- i686.egg.numpy NumPy version 1.4.0.dev6826.4e95223c11c627aa8f4878b40953d75ea8228add NumPy is installed in /home/pauli/koodi/proj/scipy/numpy.git/dist/linux/ lib/python2.5/site-packages/ numpy-1.4.0.dev6826.4e95223c11c627aa8f4878b40953d75ea8228add-py2.5-linux- i686.egg/numpy Python version 2.5.4 (r254:67916, Apr 4 2009, 17:55:16) [GCC 4.3.3] nose version 0.10.4 ....... ====================================================================== FAIL: test_csingle (test_linalg.TestLstsq) ---------------------------------------------------------------------- Traceback (most recent call last): File ".../numpy/linalg/tests/test_linalg.py", line 39, in test_csingle self.do(a, b) File ".../numpy/linalg/tests/test_linalg.py", line 141, in do assert_almost_equal(b, dot(a, x)) File ".../numpy/linalg/tests/test_linalg.py", line 23, in assert_almost_equal old_assert_almost_equal(a, b, decimal=decimal, **kw) File ".../numpy/testing/utils.py", line 262, in assert_almost_equal return assert_array_almost_equal(actual, desired, decimal, err_msg) File ".../numpy/testing/utils.py", line 537, in assert_array_almost_equal header='Arrays are not almost equal') File ".../numpy/testing/utils.py", line 395, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal (mismatch 100.0%) x: array([ 2.+1.j, 1.+2.j], dtype=complex64) y: array([ 7.22953510+1.74995685j, 10.88255692+1.7498703j ], dtype=complex64) ====================================================================== FAIL: test_csingle (test_linalg.TestSolve) ---------------------------------------------------------------------- Traceback (most recent call last): File ".../numpy/linalg/tests/test_linalg.py", line 39, in test_csingle self.do(a, b) File ".../numpy/linalg/tests/test_linalg.py", line 76, in do assert_almost_equal(b, dot(a, x)) File ".../numpy/linalg/tests/test_linalg.py", line 23, in assert_almost_equal old_assert_almost_equal(a, b, decimal=decimal, **kw) File ".../numpy/testing/utils.py", line 262, in assert_almost_equal return assert_array_almost_equal(actual, desired, decimal, err_msg) File ".../numpy/testing/utils.py", line 537, in assert_array_almost_equal header='Arrays are not almost equal') File ".../numpy/testing/utils.py", line 395, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal (mismatch 100.0%) x: array([ 2.+1.j, 1.+2.j], dtype=complex64) y: array([ 7.22953510+1.74995685j, 10.88255692+1.7498703j ], dtype=complex64) ---------------------------------------------------------------------- Ran 2034 tests in 41.154s FAILED (KNOWNFAIL=1, failures=2) -- Pauli Virtanen From efiring at hawaii.edu Sat Apr 18 19:19:39 2009 From: efiring at hawaii.edu (Eric Firing) Date: Sat, 18 Apr 2009 13:19:39 -1000 Subject: [Numpy-discussion] svn test failures: linalg with complex In-Reply-To: References: <49EA2CE8.5030602@hawaii.edu> Message-ID: <49EA600B.5090601@hawaii.edu> Pauli Virtanen wrote: > Sat, 18 Apr 2009 09:41:28 -1000, Eric Firing wrote: > >> I updated my Thinkpad to Ubuntu 9.04 release candidate, 32-bit, got a >> fresh svn checkout of numpy trunk, built, and installed. Running >> numpy.test() yields 16 errors (output attached) in linear algebra on >> complex arguments. Is this expected? > > The Numpy test suite should run without any failures. The failures here > likely indicate that something is wrong with LAPACK/BLAS/ATLAS on > Ubuntu 9.04. > > Apparently, at least ATLAS's complex single-precision routines are > completely hosed on that platforms. I filed a bug here: > > https://bugs.launchpad.net/ubuntu/+source/atlas/+bug/363510 > > Seems specific to the SSE version of Atlas. Could you comment on the bug > tracker if you see the same thing on your machine? Done. A difference between your setup and mine is that I am using sse2, not sse. Maybe that is why I see even more failures. There is a libatlas-test package. I installed it, ran the tests, and sure enough, two failed. (See my second comment on your bug report.) This is a real pain. What is the simplest work-around? Eric > > Consequently, on 9.04 I get the following failures, but for some reason > these are not the same as yours: > > > Running unit tests for > numpy-1.4.0.dev6826.4e95223c11c627aa8f4878b40953d75ea8228add-py2.5-linux- > i686.egg.numpy > NumPy version 1.4.0.dev6826.4e95223c11c627aa8f4878b40953d75ea8228add > NumPy is installed in /home/pauli/koodi/proj/scipy/numpy.git/dist/linux/ > lib/python2.5/site-packages/ > numpy-1.4.0.dev6826.4e95223c11c627aa8f4878b40953d75ea8228add-py2.5-linux- > i686.egg/numpy > Python version 2.5.4 (r254:67916, Apr 4 2009, 17:55:16) [GCC 4.3.3] > nose version 0.10.4 > ....... > ====================================================================== > FAIL: test_csingle (test_linalg.TestLstsq) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File ".../numpy/linalg/tests/test_linalg.py", line 39, in test_csingle > self.do(a, b) > File ".../numpy/linalg/tests/test_linalg.py", line 141, in do > assert_almost_equal(b, dot(a, x)) > File ".../numpy/linalg/tests/test_linalg.py", line 23, in > assert_almost_equal > old_assert_almost_equal(a, b, decimal=decimal, **kw) > File ".../numpy/testing/utils.py", line 262, in assert_almost_equal > return assert_array_almost_equal(actual, desired, decimal, err_msg) > File ".../numpy/testing/utils.py", line 537, in > assert_array_almost_equal > header='Arrays are not almost equal') > File ".../numpy/testing/utils.py", line 395, in assert_array_compare > raise AssertionError(msg) > AssertionError: > Arrays are not almost equal > > (mismatch 100.0%) > x: array([ 2.+1.j, 1.+2.j], dtype=complex64) > y: array([ 7.22953510+1.74995685j, 10.88255692+1.7498703j ], > dtype=complex64) > > ====================================================================== > FAIL: test_csingle (test_linalg.TestSolve) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File ".../numpy/linalg/tests/test_linalg.py", line 39, in test_csingle > self.do(a, b) > File ".../numpy/linalg/tests/test_linalg.py", line 76, in do > assert_almost_equal(b, dot(a, x)) > File ".../numpy/linalg/tests/test_linalg.py", line 23, in > assert_almost_equal > old_assert_almost_equal(a, b, decimal=decimal, **kw) > File ".../numpy/testing/utils.py", line 262, in assert_almost_equal > return assert_array_almost_equal(actual, desired, decimal, err_msg) > File ".../numpy/testing/utils.py", line 537, in > assert_array_almost_equal > header='Arrays are not almost equal') > File ".../numpy/testing/utils.py", line 395, in assert_array_compare > raise AssertionError(msg) > AssertionError: > Arrays are not almost equal > > (mismatch 100.0%) > x: array([ 2.+1.j, 1.+2.j], dtype=complex64) > y: array([ 7.22953510+1.74995685j, 10.88255692+1.7498703j ], > dtype=complex64) > > ---------------------------------------------------------------------- > Ran 2034 tests in 41.154s > > FAILED (KNOWNFAIL=1, failures=2) > From dyamins at gmail.com Sat Apr 18 19:40:05 2009 From: dyamins at gmail.com (Dan Yamins) Date: Sat, 18 Apr 2009 19:40:05 -0400 Subject: [Numpy-discussion] building 64bit Message-ID: <15e4667e0904181640k296c7749s8227ffc5a7e0b5f5@mail.gmail.com> Hi, I'm trying to build scipy 64bit binaries. I'm on OS10.5.6, and using python2.6 built as 64bit universal from macports. When I run: python2.6 setup.py install the build doesn't fail along the way. However, when I then go into python and try something, e.g. "from scipy import linalg", I get the error: Python 2.6.2 (r262:71600, Apr 18 2009, 15:53:12) [GCC 4.0.1 (Apple Inc. build 5490)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from scipy import linalg Traceback (most recent call last): File "", line 1, in File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/__init__.py", line 8, in from basic import * File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/basic.py", line 17, in from lapack import get_lapack_funcs File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/lapack.py", line 17, in from scipy.linalg import flapack ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so, 2): no suitable image found. Did find: /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so: no matching architecture in universal wrapper Evidently flapack.so has not been build as a 64bit. Why is this, and how do I solve it? Thanks, Dan -----Below are some more details from my own investigation of the problem, for those interested in helping: If I run file /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so this yields the output: /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so: Mach-O universal binary with 2 architectures /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so (for architecture ppc7400): Mach-O bundle ppc /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so (for architecture i386): Mach-O bundle i386 So obviously it really isn't built right. Now, I think I see when this is happening. If I look back at the output during the building of scipy, I see commands like: usr/local/bin/gfortran -Wall -arch ppc -arch i686 -Wall -undefined dynamic_lookup -bundle build/temp.macosx-10.5-universal-2.6/build/src.macosx-10.5-universal- 2.6/scipy/linalg/_flinalgmodule.o build/temp.macosx-10.5-universal-2.6/build/src.macosx-10.5-universal-2.6/fortranobject.o build/temp.macosx-10.5-universal-2.6/scipy/linalg/src/det.o build/temp.macosx-10.5-universal-2.6/scipy/linalg/src/lu.o -Lbuild/temp.macosx-10.5-universal-2.6 -lgfortran -o build/lib.macosx-10.5-universal-2.6/scipy/linalg/_flinalg.so -Wl,-framework -Wl,Accelerate e.g. the fortran compiler is not being given the flag -arch x86_64 (though reading the output shows that the C compiler IS.) Morevoer, if I do: python2.6 setup.py config_fc --help-fcompiler then the output is: archiver = ['/usr/local/bin/gfortran', '-cr'] compile_switch = '-c' compiler_f77 = ['/usr/local/bin/gfortran', '-Wall', '-ffixed-form', '- fno-second-underscore', '-arch', 'ppc', '-arch', 'i686', ' -fPIC', '-O3', '-funroll-loops'] compiler_f90 = ['/usr/local/bin/gfortran', '-Wall', '-fno-second- underscore', '-arch', 'ppc', '-arch', 'i686', '-fPIC', '- O3', '-funroll-loops'] compiler_fix = ['/usr/local/bin/gfortran', '-Wall', '-ffixed-form', '- fno-second-underscore', '-Wall', '-fno-second-underscore', '-arch', 'ppc', '-arch', 'i686', '-fPIC', '-O3', '-funroll -loops'] So I have three questions: A) is this lack of proper flag to the fortran compiler the reason my flapack is not being build properly? B) If so, how do I change the compiler options to the fortran compiler when I run setup.py? C) If not, what is the reason, and what do I do about it? Thanks for reading this long thing , Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sat Apr 18 19:46:29 2009 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 18 Apr 2009 23:46:29 +0000 (UTC) Subject: [Numpy-discussion] svn test failures: linalg with complex References: <49EA2CE8.5030602@hawaii.edu> <49EA600B.5090601@hawaii.edu> Message-ID: Sat, 18 Apr 2009 13:19:39 -1000, Eric Firing wrote: [clip] > This is a real pain. What is the simplest work-around? apt-get install libatlas3gf-base export LD_LIBRARY_PATH=/usr/lib/atlas Or, just uninstall libatlas3gf-sse2 in favor of the *-base version. -- Pauli Virtanen From efiring at hawaii.edu Sat Apr 18 20:21:41 2009 From: efiring at hawaii.edu (Eric Firing) Date: Sat, 18 Apr 2009 14:21:41 -1000 Subject: [Numpy-discussion] svn test failures: linalg with complex In-Reply-To: References: <49EA2CE8.5030602@hawaii.edu> <49EA600B.5090601@hawaii.edu> Message-ID: <49EA6E95.5040502@hawaii.edu> Pauli Virtanen wrote: > Sat, 18 Apr 2009 13:19:39 -1000, Eric Firing wrote: > [clip] >> This is a real pain. What is the simplest work-around? > > apt-get install libatlas3gf-base > export LD_LIBRARY_PATH=/usr/lib/atlas > > Or, just uninstall libatlas3gf-sse2 in favor of the *-base version. > Uninstalling libatlas3gf-sse2* worked fine. I didn't realize it simply shadowed *-base. Thanks. Eric From stefan at sun.ac.za Sun Apr 19 04:35:14 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 19 Apr 2009 10:35:14 +0200 Subject: [Numpy-discussion] Buildbot unavailable In-Reply-To: References: Message-ID: <9457e7c80904190135l2540e02g94374508a334639c@mail.gmail.com> Thanks, should be fixed now. 2009/4/18 Pauli Virtanen : > Hi, > > buildbot.scipy.org says: > > ? ? ? ?Service Temporarily Unavailable > > ? ? ? ?The server is temporarily unable to service your request due to > ? ? ? ?maintenance downtime or capacity problems. Please try again later. From sccolbert at gmail.com Sun Apr 19 11:32:02 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Sun, 19 Apr 2009 11:32:02 -0400 Subject: [Numpy-discussion] svn test failures: linalg with complex In-Reply-To: <49EA6E95.5040502@hawaii.edu> References: <49EA2CE8.5030602@hawaii.edu> <49EA600B.5090601@hawaii.edu> <49EA6E95.5040502@hawaii.edu> Message-ID: <7f014ea60904190832n639effe2i2fef8d22dc6ef65c@mail.gmail.com> you could always build atlas from source, the benefit of that is that you can build it with threading enabled, vs the ubuntu packages which dont scale across cores (at least on my machine they didnt). The build instructions on scipy.org are fairly complete, if you give it a go and have any trouble, shoot me an email. I just went through this process a couple weeks ago. Chris On Sat, Apr 18, 2009 at 8:21 PM, Eric Firing wrote: > Pauli Virtanen wrote: > > Sat, 18 Apr 2009 13:19:39 -1000, Eric Firing wrote: > > [clip] > >> This is a real pain. What is the simplest work-around? > > > > apt-get install libatlas3gf-base > > export LD_LIBRARY_PATH=/usr/lib/atlas > > > > Or, just uninstall libatlas3gf-sse2 in favor of the *-base version. > > > > Uninstalling libatlas3gf-sse2* worked fine. I didn't realize it simply > shadowed *-base. > > Thanks. > > Eric > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cburns at berkeley.edu Sun Apr 19 15:18:48 2009 From: cburns at berkeley.edu (Christopher Burns) Date: Sun, 19 Apr 2009 12:18:48 -0700 Subject: [Numpy-discussion] Leopard install In-Reply-To: <49E417E7.9050201@ar.media.kyoto-u.ac.jp> References: <49E417E7.9050201@ar.media.kyoto-u.ac.jp> Message-ID: <764e38540904191218p3be39e17recd286dc59dd67a6@mail.gmail.com> When I built the numpy binary last year I used the svn trunk of bdist_mpkg and changed the module that builds the info.plist (diff below). This was much easier then modifying the package after it's built. Chris Index: bdist_mpkg/plists.py =================================================================== --- bdist_mpkg/plists.py (revision 18) +++ bdist_mpkg/plists.py (working copy) @@ -70,7 +70,7 @@ FRIENDLY_PREFIX = { os.path.expanduser(u'~/Library/Frameworks') : u'User', u'/System/Library/Frameworks' : u'Apple', - u'/Library/Frameworks' : u'System', + u'/Library/Frameworks' : u'python.org Framework Build', u'/opt/local' : u'DarwinPorts', u'/usr/local' : u'Unix', u'/sw' : u'Fink', On Mon, Apr 13, 2009 at 9:58 PM, David Cournapeau wrote: > It is set-up in the info.plist as built by bdist_mpkg, so I guess > changing this file should be enough - this should be easy to test :) > > Maybe we should do the packaging ourselves instead of relying on the > bdist_mpkg command, to control all this better (we could reuse some code > from the bdist_mpkg command, of course) ? From praxbaffle at hotmail.com Sun Apr 19 23:31:32 2009 From: praxbaffle at hotmail.com (John Seales) Date: Sun, 19 Apr 2009 20:31:32 -0700 Subject: [Numpy-discussion] can't import pylab In-Reply-To: <49EA6E95.5040502@hawaii.edu> References: <49EA2CE8.5030602@hawaii.edu> <49EA600B.5090601@hawaii.edu> <49EA6E95.5040502@hawaii.edu> Message-ID: I can't import pylab!! it's a problem with ft2font. I checked for the existence of the files it's looking for, the libfreetype.6.dylib, and ft2font.so ? they exist in the places they're supposed to. I don't really know what 'image not found' exactly means. If anyone has any suggestions, I will be very grateful. >>> import pylab Traceback (most recent call last): File "", line 1, in File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pylab.py", line 1, in from matplotlib.pylab import * File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/matplotlib/pylab.py", line 206, in from matplotlib import mpl # pulls in most modules File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/matplotlib/mpl.py", line 2, in from matplotlib import axis File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/matplotlib/axis.py", line 9, in import matplotlib.font_manager as font_manager File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/matplotlib/font_manager.py", line 52, in from matplotlib import ft2font ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/matplotlib/ft2font.so, 2): Library not loaded: /usr/X11R6/lib/libfreetype.6.dylib Referenced from: /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/matplotlib/ft2font.so Reason: image not found >>> _________________________________________________________________ Rediscover Hotmail?: Now available on your iPhone or BlackBerry http://windowslive.com/RediscoverHotmail?ocid=TXT_TAGLM_WL_HM_Rediscover_Mobile2_042009 -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Apr 19 23:32:28 2009 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 19 Apr 2009 22:32:28 -0500 Subject: [Numpy-discussion] can't import pylab In-Reply-To: References: <49EA2CE8.5030602@hawaii.edu> <49EA600B.5090601@hawaii.edu> <49EA6E95.5040502@hawaii.edu> Message-ID: <3d375d730904192032h109d5e18k82fa85905af94d9d@mail.gmail.com> On Sun, Apr 19, 2009 at 22:31, John Seales wrote: > I can't import pylab!! > > it's a problem with ft2font. I checked for the existence of the files it's > looking for, the libfreetype.6.dylib, and? ft2font.so ? they exist in the > places they're supposed to. I don't really know what 'image not found' > exactly means. > > If anyone has any suggestions, I will be very grateful. Ask on the matplotlib-users mailing list: https://lists.sourceforge.net/lists/listinfo/matplotlib-users -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dan.s.towell+numpy at gmail.com Mon Apr 20 06:44:14 2009 From: dan.s.towell+numpy at gmail.com (Dan S) Date: Mon, 20 Apr 2009 11:44:14 +0100 Subject: [Numpy-discussion] Solving a memory leak in a numpy extension; PyArray_ContiguousFromObject In-Reply-To: References: <286e6b7c0904170825t56c44869u2fcc2eeeb8e1fda2@mail.gmail.com> Message-ID: <286e6b7c0904200344p1442913dpfb424fb742c6cdbd@mail.gmail.com> Thanks all for responses. Continuation below: 2009/4/18 Charles R Harris : > > > On Fri, Apr 17, 2009 at 9:25 AM, Dan S wrote: >> >> Hi - >> >> I have written a numpy extension which works fine but has a memory >> leak. It takes a single array argument and returns a single scalar. >> After reducing the code down in order to chase the problem, I have the >> following: >> >> static PyObject * kdpee_pycall(PyObject *self, PyObject *args) >> { >> ? ? ? ?PyObject *input; >> ? ? ? ?PyArrayObject *array; >> ? ? ? ?int n, numdims; >> >> ? ? ? ?if (!PyArg_ParseTuple(args, "O", &input)) >> ? ? ? ? ? ? ? ?return NULL; >> >> ? ? ? ?// Ensure we have contiguous, 2D, floating-point data: >> ? ? ? ?array = (PyArrayObject*) PyArray_ContiguousFromObject(input, >> PyArray_DOUBLE, 2, 2); >> >> ? ? ? ?if(array==NULL){ >> ? ? ? ? ? ? ? ?printf("kdpee_py: nullness!\n"); >> ? ? ? ? ? ? ? ?return NULL; >> ? ? ? ?} >> ? ? ? ?PyArray_XDECREF(array); >> ? ? ? ?Py_DECREF(array); // destroy the contig array >> ? ? ? ?Py_DECREF(input); // is this needed? doc says no, but it seems to fix > ^^^^^^^^^^^^^^^^^^^^ > > Shouldn't be, but there might be a bug somewhere which causes the reference > count of input to be double incremented. Does the reference count in the > test script increase without this line? Yes - if I comment out Py_DECREF(input), then sys.getrefcount(a) goes from 2, to 5002, to 10002, every time I run that little 5000-fold test iteration. (Py_DECREF(input) is not suggested in the docs - I tried it in desperation and for some reason it seems to reduce the problem, since it gets rid of the refleak, although it doesn't solve the memleak.) >> ? ? ? ?return PyFloat_FromDouble(3.1415927); // temporary >> } >> >> >> The test code is >> >> ? ? ?from numpy import * >> ? ? ?from kdpee import * >> ? ? ?import sys >> ? ? ?a = array( [[1,2,3,4,5,1,2,3,4,5], [6,7,8,9,0,6,7,8,9,0]]) >> ? ? ?sys.getrefcount(a) >> ? ? ?for i in range(5000): tmp = kdpee(a) >> >> ? ? ?sys.getrefcount(a) >> >> Every time I run this code I get "2" back from both calls to >> sys.getrefcount(a), but I'm still getting a memory leak of about 0.4 >> MB. What am I doing wrong? > > So about 80 bytes/iteration? How accurate is that .4 MB? does it change with > the size of the input array? OK, pinning it down more precisely I get around 86--90 bytes per iteration (using ps, which gives me Kb resolution). It does not change if I double or quadruple the size of the input array. Today I tried using "heapy": using the heap() and heapu() methods, I find no evidence of anything suspicious - there is no garbage left lying on the heap of python objects. So it's presumably something deeper. But as you can see, my C code doesn't perform any malloc() or suchlike, so I'm stumped. I'd be grateful for any further thoughts. Thanks Dan From sole at esrf.fr Mon Apr 20 07:39:07 2009 From: sole at esrf.fr (=?ISO-8859-1?Q?=22V=2E_Armando_Sol=E9=22?=) Date: Mon, 20 Apr 2009 13:39:07 +0200 Subject: [Numpy-discussion] Solving a memory leak in a numpy extension; PyArray_ContiguousFromObject In-Reply-To: <286e6b7c0904200344p1442913dpfb424fb742c6cdbd@mail.gmail.com> References: <286e6b7c0904170825t56c44869u2fcc2eeeb8e1fda2@mail.gmail.com> <286e6b7c0904200344p1442913dpfb424fb742c6cdbd@mail.gmail.com> Message-ID: <49EC5EDB.8050806@esrf.fr> Dan S wrote: > But as you can see, my C code doesn't perform any malloc() or > suchlike, so I'm stumped. > > I'd be grateful for any further thoughts Could it be your memory leak is in: return PyFloat_FromDouble(3.1415927); // temporary You are creating a "python float object" from something. What if you return Py_None instead of your float?: Py_INCREF(Py_None); return PyNone; I do not know if it will help you but I guess it falls in the "any further thought" category :-) Best regards, Armando From dan.s.towell+numpy at gmail.com Mon Apr 20 09:08:57 2009 From: dan.s.towell+numpy at gmail.com (Dan S) Date: Mon, 20 Apr 2009 14:08:57 +0100 Subject: [Numpy-discussion] Solving a memory leak in a numpy extension; PyArray_ContiguousFromObject In-Reply-To: <49EC5EDB.8050806@esrf.fr> References: <286e6b7c0904170825t56c44869u2fcc2eeeb8e1fda2@mail.gmail.com> <286e6b7c0904200344p1442913dpfb424fb742c6cdbd@mail.gmail.com> <49EC5EDB.8050806@esrf.fr> Message-ID: <286e6b7c0904200608s157f70a4l401b3930ba1c4be7@mail.gmail.com> 2009/4/20 "V. Armando Sol?" : > Dan S wrote: >> But as you can see, my C code doesn't perform any malloc() or >> suchlike, so I'm stumped. >> >> I'd be grateful for any further thoughts > Could it be your memory leak is in: > > return PyFloat_FromDouble(3.1415927); // temporary > > > You are creating a "python float object" from something. What if you > return Py_None instead of your float?: > > Py_INCREF(Py_None); > return PyNone; > > I do not know if it will help you but I guess it falls in the "any > further thought" category ?:-) Thanks :) but that doesn't alter things. In fact: I found the real source of the problem. Namely: I was doing #include "Numeric/arrayobject.h" when I should have been doing #include "numpy/arrayobject.h" Therefore I was compiling against the old deprecated version of the lib, which presumably had a memory leak inside itself which got fixed in the intervening years. Using the up-to-date lib, no leak. Thanks all, anyway! Dan From david at ar.media.kyoto-u.ac.jp Mon Apr 20 08:51:21 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 20 Apr 2009 21:51:21 +0900 Subject: [Numpy-discussion] Linker script, smaller source files and symbol visibility Message-ID: <49EC6FC9.4090009@ar.media.kyoto-u.ac.jp> Hi, For quite a long time I have been bothered by the very large files needed for python extensions. In particular for numpy.core, which consists in a few files which are ~ 1 Mb, I find this a pretty high barrier of entry for newcomers, and it has quite a big impact on the code organization. I think I have found a way to split things on common platforms (this includes at least windows, mac os x, linux and solaris), without impacting other potentially less capable platforms, or static linking of numpy. Assuming my idea is technically sound and that I can demonstrate it works on say Linux without impacting other platforms (see example below), would that be considered useful ? cheers, David Technical details ================== The rationale for doing things as they are is a C limitation related to symbol visibility being limited to file scope, i.e. if you want to share a function into several files without making it public in the binary, you have to tag the function static, and include all .c files which use this function into one giant .c file. That's how we do it in numpy. Many binary format (elf, coff and Mach-O) have a mechanism to limit the symbol visibility, so that we can explicitly set the functions we do want to export. With a couple of defines, we could either include every files and tag the implementation functions as static, or link every file together and limit symbol visibility with some linker magic. Example ------- I use the spam example from the official python doc, with one function PySpam_System which is exported in a C API, and the actual implementation is _pyspam_system. * spammodule.c: define the interface available from python interpreter: #include #include #define SPAM_MODULE #include "spammodule.h" #include "spammodule_imp.h" /* if we don't know how to deal with symbol visibility on the platform, just include everything in one file */ #ifdef SYMBOL_SCRIPT_UNSUPPORTED #include "spammodule_imp.c" #endif /* C API for spam module */ static int PySpam_System(const char *command) { _pyspam_implementation(command); return 0; } * spammodule_imp.h: declares the implementation, should only be included by spammodule.c and spammodule_imp.c which implements the actual function #ifndef _IMP_H_ #define _IMP_H_ #ifndef SPAM_MODULE #error this should not be included unless you really know what you are doing #endif #ifdef SYMBOL_SCRIPT_UNSUPPORTED #define SPAM_PRIVATE static #else #define SPAM_PRIVATE #endif SPAM_PRIVATE int _pyspam_implementation(const char *command); #endif For supported platforms (where SYMBOL_SCRIPT_UNSUPPORTED is not defined), _pyspam_implementation would not be visible because we would have a list of functions to export (only initspam in this case). Advantages ---------- This has several advantages on platforms where this is supported - code more amenable: source code which are thousand of lines are difficult to follow - faster compilation times: in my experience, compilation time doesn't scale linearly with the amount of code. - compilation can be better parallelized - changing one file does not force a whole multiarray/ufunc module recompilation (which can be pretty long when you chase bugs in it) Another advantage is related to namespace pollution. Since library extensions are static libraries for now, any symbol frome those libraries used by any extension is publicly available. For example, now that multiarray.so uses the npy_math library, every symbol in npy_math is in the public namespace. That's also true for every scipy extensions (for example, _fftpack.so exports the whole dfftpack public API). If we want to go further down the road of making core computational code publicly available, I think we should improve this first. Disadvantage ------------ We need to code it. There are two parts: - numpy.distutils support: I have already something working in for linux. Once we have one platform working, adding others should not be a problem - changing the C code: we could at first splitting things in .c files but still including everything, and then starting the conversion. From rsalvador.wk at gmail.com Mon Apr 20 09:44:37 2009 From: rsalvador.wk at gmail.com (Ruben Salvador) Date: Mon, 20 Apr 2009 15:44:37 +0200 Subject: [Numpy-discussion] Numpy float precision vs Python list float issue Message-ID: <4fe028e30904200644h73e7fcc6h3a5b6e1016477f8b@mail.gmail.com> Hi everybody! First of all I should say I am a newbie with Python/Scipy. Have been searching a little bit (google and lists) and haven't found a helpful answer...so I'm posting. I'm using Scipy/Numpy to do image wavelet transforms via the lifting scheme. I grabbed some code implementing the transforms with Python lists (float type). This code works perfectly, but slow for my needs (I'll be doing some genetic algorithms to evolve coefficients of the filters and the forward and inverse transform will be done many times). It's just implemented by looping in the lists and making computations this way. Reconstructed image after doing a forward and inverse transform is perfect, this is, original and reconstructed images difference is 0. With Scipy/Numpy float arrays slicing this code is much faster as you know. But the reconstructed image is not perfect. The image difference maximum and minimum values returns: maximum difference => 3.5527136788e-15 minimum difference => -3.5527136788e-15 Is this behavior expected? Because it seems sooo weird to me. If expected, anyway to override it? I include some test code for you to reproduce. It's part of a transform over a 8x8 2D signal (for simplicity). It's not complete (again for simplicity), but enough to reproduce. It does part of a forward and inverse transform both with lists and arrays, printing the differences (there is commented code showing some plots with the results as I used when transforming real images, but for the purpose, is enough with the return results I think). Code is simple (maybe long but it's always the same). Instead of using the faster array slicing as commented above, I am using here array looping, so that the math code is exactly the same as in the case list. This happens in the next three system/platforms. * System 1 (laptop): --------------------------- 64 bit processor running Kubuntu 8.04 32 bits Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52 Numpy version: 1:1.0.4-6ubuntu3 Scipy version: 0.6.0-8ubuntu1 * System 2 (PC): -------------------------- Windows Xp on 64 bit processor Enthought Python distribution (EPD Py25 v4.1.30101). This is a Python 2.5.2 with Numpy 1.1.1 and Scipy 0.6.0 * System 3 (same PC as 2): -------------------------------------- Debian Lenny 64 bit on 64 bit processor Not sure about versions here, but doesn't mind because behavior is prety much the same in the 3 systems Thanks everybody in advance for the help! Rub?n Salvador PhD Student Industrial Electronics Center Universidad Polit?cnica de Madrid -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: floattest.py Type: text/x-python Size: 7161 bytes Desc: not available URL: From david at ar.media.kyoto-u.ac.jp Mon Apr 20 09:47:09 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 20 Apr 2009 22:47:09 +0900 Subject: [Numpy-discussion] Numpy float precision vs Python list float issue In-Reply-To: <4fe028e30904200644h73e7fcc6h3a5b6e1016477f8b@mail.gmail.com> References: <4fe028e30904200644h73e7fcc6h3a5b6e1016477f8b@mail.gmail.com> Message-ID: <49EC7CDD.5000109@ar.media.kyoto-u.ac.jp> Hi Ruben, Ruben Salvador wrote: > Hi everybody! > > First of all I should say I am a newbie with Python/Scipy. Have been > searching a little bit (google and lists) and haven't found a helpful > answer...so I'm posting. > > I'm using Scipy/Numpy to do image wavelet transforms via the lifting > scheme. I grabbed some code implementing the transforms with Python > lists (float type). This code works perfectly, but slow for my needs > (I'll be doing some genetic algorithms to evolve coefficients of the > filters and the forward and inverse transform will be done many > times). It's just implemented by looping in the lists and making > computations this way. Reconstructed image after doing a forward and > inverse transform is perfect, this is, original and reconstructed > images difference is 0. > > With Scipy/Numpy float arrays slicing this code is much faster as you > know. But the reconstructed image is not perfect. The image difference > maximum and minimum values returns: > maximum difference => 3.5527136788e-15 > minimum difference => -3.5527136788e-15 > > Is this behavior expected? Yes, it is expected, it is inherent to how floating point works. By default, the precision for floating point array is double precision, for which, in normal settings, a = a + 1e-17. > Because it seems sooo weird to me. It shouldn't :) The usual answer is that you should read this: http://docs.sun.com/app/docs/doc/800-7895 Floating point is a useful abstraction, but has some strange properties, which do not matter much in most cases, but can catch you off guard. > If expected, anyway to override it? The only way to mitigate it is to use higher precision (depending on your OS/CPU combination, the long double type can help), or using a type with arbitrary precision. But in that later case, it will be much, much slower (as many computations are done in software), and you get a pretty limited subset of numpy features (enough to implement basic wavelet with Haar bases, though). Using extended precision: import numpy as np a = np.array([1, 2, 3], dtype=np.longdouble) # this gives you more precision You can also use the decimal module for (almost) arbitrary precision: from decimal import Decimal import numpy as np a = np.array([1, 2, 3], dtype=Decimal) But again, please keep in mind that many operations normally available cannot be done for arbitrary objects like Decimal instances. Generally, numerical computation are done with a limited precision, and it is rarely a problem if enough care s taken. If you need theoretically exact results, then you need a symbolic computation package, which numpy alone is not. cheers, David From david at ar.media.kyoto-u.ac.jp Mon Apr 20 09:50:04 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 20 Apr 2009 22:50:04 +0900 Subject: [Numpy-discussion] Numpy float precision vs Python list float issue In-Reply-To: <49EC7CDD.5000109@ar.media.kyoto-u.ac.jp> References: <4fe028e30904200644h73e7fcc6h3a5b6e1016477f8b@mail.gmail.com> <49EC7CDD.5000109@ar.media.kyoto-u.ac.jp> Message-ID: <49EC7D8C.7040108@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > Hi Ruben, > > Ruben Salvador wrote: > >> Hi everybody! >> >> First of all I should say I am a newbie with Python/Scipy. Have been >> searching a little bit (google and lists) and haven't found a helpful >> answer...so I'm posting. >> >> I'm using Scipy/Numpy to do image wavelet transforms via the lifting >> scheme. I grabbed some code implementing the transforms with Python >> lists (float type). This code works perfectly, but slow for my needs >> (I'll be doing some genetic algorithms to evolve coefficients of the >> filters and the forward and inverse transform will be done many >> times). It's just implemented by looping in the lists and making >> computations this way. Reconstructed image after doing a forward and >> inverse transform is perfect, this is, original and reconstructed >> images difference is 0. >> >> With Scipy/Numpy float arrays slicing this code is much faster as you >> know. But the reconstructed image is not perfect. The image difference >> maximum and minimum values returns: >> maximum difference => 3.5527136788e-15 >> minimum difference => -3.5527136788e-15 >> >> Is this behavior expected? >> > > Yes, it is expected, it is inherent to how floating point works. By > default, the precision for floating point array is double precision, for > which, in normal settings, a = a + 1e-17. > > >> Because it seems sooo weird to me. >> > > It shouldn't :) The usual answer is that you should read this: > > http://docs.sun.com/app/docs/doc/800-7895 > > Floating point is a useful abstraction, but has some strange properties, > which do not matter much in most cases, but can catch you off guard. > > >> If expected, anyway to override it? >> > > The only way to mitigate it is to use higher precision (depending on > your OS/CPU combination, the long double type can help), or using a type > with arbitrary precision. ^^^ this is inexact, I should have written the only way to avoid the problem it so use arbitrary precision, and you can mitigate with higher precision and/or better implementation. If double precision (the default) is not enough, it is often because there is something wrong in your implementation (compute exponential of big numbers instead of working in the log domain, etc...) David From rob.clewley at gmail.com Mon Apr 20 10:39:39 2009 From: rob.clewley at gmail.com (Rob Clewley) Date: Mon, 20 Apr 2009 10:39:39 -0400 Subject: [Numpy-discussion] Numpy float precision vs Python list float issue In-Reply-To: <49EC7CDD.5000109@ar.media.kyoto-u.ac.jp> References: <4fe028e30904200644h73e7fcc6h3a5b6e1016477f8b@mail.gmail.com> <49EC7CDD.5000109@ar.media.kyoto-u.ac.jp> Message-ID: David, I'm confused about your reply. I don't think Ruben was only asking why you'd ever get non-zero error after the forward and inverse transform, but why his implementation using lists gives zero error but using arrays he gets something of order 1e-15. On Mon, Apr 20, 2009 at 9:47 AM, David Cournapeau wrote: >> I'm using Scipy/Numpy to do image wavelet transforms via the lifting >> scheme. I grabbed some code implementing the transforms with Python >> lists (float type). This code works perfectly, but slow for my needs >> (I'll be doing some genetic algorithms to evolve coefficients of the >> filters and the forward and inverse transform will be done many >> times). It's just implemented by looping in the lists and making >> computations this way. Reconstructed image after doing a forward and >> inverse transform is perfect, this is, original and reconstructed >> images difference is 0. >> >> With Scipy/Numpy float arrays slicing this code is much faster as you >> know. But the reconstructed image is not perfect. The image difference >> maximum and minimum values returns: >> maximum difference => 3.5527136788e-15 >> minimum difference => -3.5527136788e-15 >> >> Is this behavior expected? > > Yes, it is expected, it is inherent to how floating point works. By > default, the precision for floating point array is double precision, for > which, in normal settings, a = a + 1e-17. I don't think it's expected in this sense. The question is why the exact same sequence of arithmetic ops on lists yields zero error but on arrays yields 3.6e-15 error. This doesn't seem to be about lists not showing full precision of the values, because the differences are even zero when extracted from the lists. >> Because it seems sooo weird to me. > > It shouldn't :) The usual answer is that you should read this: > > http://docs.sun.com/app/docs/doc/800-7895 This doesn't help! This is a python question, methinks. Best, Rob From wesmckinn at gmail.com Mon Apr 20 10:42:22 2009 From: wesmckinn at gmail.com (Wes McKinney) Date: Mon, 20 Apr 2009 10:42:22 -0400 Subject: [Numpy-discussion] NaN as dictionary key? Message-ID: <6c476c8a0904200742j331c4d32k4b877995e001a9ed@mail.gmail.com> I assume that, because NaN != NaN, even though both have the same hash value (hash(NaN) == -32768), that Python treats any NaN double as a distinct key in a dictionary. In [76]: a = np.repeat(nan, 10) In [77]: d = {} In [78]: for i, v in enumerate(a): ....: d[v] = i ....: ....: In [79]: d Out[79]: {nan: 0, nan: 1, nan: 6, nan: 4, nan: 3, nan: 9, nan: 7, nan: 2, nan: 8, nan: 5} I'm not sure if this ever worked in a past version of NumPy, however, I have code which does a "group by value" and occasionally in the real world those values are NaN. Any ideas or a way around this problem? Thanks, Wes -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Mon Apr 20 10:48:23 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 20 Apr 2009 23:48:23 +0900 Subject: [Numpy-discussion] Numpy float precision vs Python list float issue In-Reply-To: References: <4fe028e30904200644h73e7fcc6h3a5b6e1016477f8b@mail.gmail.com> <49EC7CDD.5000109@ar.media.kyoto-u.ac.jp> Message-ID: <49EC8B37.7040307@ar.media.kyoto-u.ac.jp> Rob Clewley wrote: > David, > > I'm confused about your reply. I don't think Ruben was only asking why > you'd ever get non-zero error after the forward and inverse transform, > but why his implementation using lists gives zero error but using > arrays he gets something of order 1e-15. > That's more likely just an accident. Forward + inverse = id is the surprising thing, actually. In any numerical package, if you do ifft(fft(a)), you will not recover a exactly for any non trivial size. For example, with floating point numbers, the order in which you do operations matters, so: a = 1e10 b = 1-20 c = a c -= a + b d = a d -= a d -= b Will give you different values for d and c, even if you "on paper", those are exactly the same. For those reasons, it is virtually impossible to have exactly the same values for two different implementations of the same algorithm. As long as the difference is small (if the reconstruction error falls in the 1e-15 range, it is mostly likely the case), it should not matter, David From charlesr.harris at gmail.com Mon Apr 20 11:48:37 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 20 Apr 2009 09:48:37 -0600 Subject: [Numpy-discussion] Linker script, smaller source files and symbol visibility In-Reply-To: <49EC6FC9.4090009@ar.media.kyoto-u.ac.jp> References: <49EC6FC9.4090009@ar.media.kyoto-u.ac.jp> Message-ID: Hi David On Mon, Apr 20, 2009 at 6:51 AM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Hi, > > For quite a long time I have been bothered by the very large files > needed for python extensions. In particular for numpy.core, which > consists in a few files which are ~ 1 Mb, I find this a pretty high > barrier of entry for newcomers, and it has quite a big impact on the > code organization. I think I have found a way to split things on common > platforms (this includes at least windows, mac os x, linux and solaris), > without impacting other potentially less capable platforms, or static > linking of numpy. > There was a discussion of this a couple of years ago. I was in favor of many small files maybe in subdirectories. Robert, IIRC, thought too many small files could become confusing, so there is a fine line in there somewhere. I am generally in favor of breaking the files up into their functional components and maybe rewriting some of the upper level interface files in cython. But it does need some agreement and we should probably start by just breaking up a few files. I don't have a problem with big files that are just collections of small routines all of the same type, umath_loops.inc.src for instance. > > Assuming my idea is technically sound and that I can demonstrate it > works on say Linux without impacting other platforms (see example > below), would that be considered useful ? > Definitly worth consideration. > > cheers, > > David > > Technical details > ================== > > The rationale for doing things as they are is a C limitation related > to symbol visibility being limited to file scope, i.e. if you want to > share a function into several files without making it public in the > binary, you have to tag the function static, and include all .c files > which use this function into one giant .c file. That's how we do it in > numpy. Many binary format (elf, coff and Mach-O) have a mechanism to > limit the symbol visibility, so that we can explicitly set the functions > we do want to export. With a couple of defines, we could either include > every files and tag the implementation functions as static, or link > every file together and limit symbol visibility with some linker magic. > Maybe just not worry about symbol visibility on other platforms. It is one of those warts that only becomes apparent when you go looking for it. For instance, the current *.so has some extraneous symbols but I don't hear folks complaining. > > Example > ------- > > I use the spam example from the official python doc, with one function > PySpam_System which is exported in a C API, and the actual > implementation is _pyspam_system. > > * spammodule.c: define the interface available from python interpreter: > > #include > > > #include > > > > > #define > SPAM_MODULE > > #include > "spammodule.h" > > #include > "spammodule_imp.h" > > > /* if we don't know how to deal with symbol visibility on the platform, > just include everything in one file */ > #ifdef > SYMBOL_SCRIPT_UNSUPPORTED > > #include > "spammodule_imp.c" > > #endif > > > /* C API for spam module */ > > > static int > PySpam_System(const char *command) > { > _pyspam_implementation(command); > return 0; > } > > * spammodule_imp.h: declares the implementation, should only be included > by spammodule.c and spammodule_imp.c which implements the actual function > > #ifndef _IMP_H_ > #define _IMP_H_ > > #ifndef SPAM_MODULE > #error this should not be included unless you really know what you are > doing > #endif > > #ifdef SYMBOL_SCRIPT_UNSUPPORTED > #define SPAM_PRIVATE static > #else > #define SPAM_PRIVATE > #endif > > SPAM_PRIVATE int > _pyspam_implementation(const char *command); > > #endif > > For supported platforms (where SYMBOL_SCRIPT_UNSUPPORTED is not > defined), _pyspam_implementation would not be visible because we would > have a list of functions to export (only initspam in this case). > > Advantages > ---------- > > This has several advantages on platforms where this is supported > - code more amenable: source code which are thousand of lines are > difficult to follow > - faster compilation times: in my experience, compilation time > doesn't scale linearly with the amount of code. > - compilation can be better parallelized > - changing one file does not force a whole multiarray/ufunc module > recompilation (which can be pretty long when you chase bugs in it) > > Another advantage is related to namespace pollution. Since library > extensions are static libraries for now, any symbol frome those > libraries used by any extension is publicly available. For example, now > that multiarray.so uses the npy_math library, every symbol in npy_math > is in the public namespace. That's also true for every scipy extensions > (for example, _fftpack.so exports the whole dfftpack public API). If we > want to go further down the road of making core computational code > publicly available, I think we should improve this first. > > Disadvantage > ------------ > > We need to code it. There are two parts: > - numpy.distutils support: I have already something working in for > linux. Once we have one platform working, adding others should not be a > problem > - changing the C code: we could at first splitting things in .c > files but still including everything, and then starting the conversion. > There's the rub. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.clewley at gmail.com Mon Apr 20 11:49:12 2009 From: rob.clewley at gmail.com (Rob Clewley) Date: Mon, 20 Apr 2009 11:49:12 -0400 Subject: [Numpy-discussion] Numpy float precision vs Python list float issue In-Reply-To: <49EC8B37.7040307@ar.media.kyoto-u.ac.jp> References: <4fe028e30904200644h73e7fcc6h3a5b6e1016477f8b@mail.gmail.com> <49EC7CDD.5000109@ar.media.kyoto-u.ac.jp> <49EC8B37.7040307@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Apr 20, 2009 at 10:48 AM, David Cournapeau wrote: > Rob Clewley wrote: >> David, >> >> I'm confused about your reply. I don't think Ruben was only asking why >> you'd ever get non-zero error after the forward and inverse transform, >> but why his implementation using lists gives zero error but using >> arrays he gets something of order 1e-15. >> > > That's more likely just an accident. Forward + inverse = id is the > surprising thing, actually. In any numerical package, if you do > ifft(fft(a)), you will not recover a exactly for any non trivial size. > For example, with floating point numbers, the order in which you do > operations matters, so: > Will give you different values for d and c, even if you "on paper", > those are exactly the same. For those reasons, it is virtually > impossible to have exactly the same values for two different > implementations of the same algorithm. As long as the difference is > small (if the reconstruction error falls in the 1e-15 range, it is > mostly likely the case), it should not matter, I understand the numerical mathematics behind this very well but my point is that his two algorithms appear to be identical (same operations, same order), he simply uses lists in one and arrays in the other. It's not like he used vectorization or other array-related operations - he uses for loops in both cases. Of course I agree that 1e-15 error should be acceptable, but that's not the point. I think there is legitimate curiosity in wondering why there is any difference between using the two data types in exactly the same algorithm. Maybe Ruben could re-code the example using a single function that accepts either a list or an array to really demonstrate that there is no mistake in any of the decimal literals or any subtle difference in the ordering of operations that I didn't spot. Would that convince you, David, that there is something interesting that remains to be explained in this example?! Best, Rob From charlesr.harris at gmail.com Mon Apr 20 12:03:39 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 20 Apr 2009 10:03:39 -0600 Subject: [Numpy-discussion] Numpy float precision vs Python list float issue In-Reply-To: References: <4fe028e30904200644h73e7fcc6h3a5b6e1016477f8b@mail.gmail.com> <49EC7CDD.5000109@ar.media.kyoto-u.ac.jp> <49EC8B37.7040307@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Apr 20, 2009 at 9:49 AM, Rob Clewley wrote: > On Mon, Apr 20, 2009 at 10:48 AM, David Cournapeau > wrote: > > Rob Clewley wrote: > >> David, > >> > >> I'm confused about your reply. I don't think Ruben was only asking why > >> you'd ever get non-zero error after the forward and inverse transform, > >> but why his implementation using lists gives zero error but using > >> arrays he gets something of order 1e-15. > >> > > > > That's more likely just an accident. Forward + inverse = id is the > > surprising thing, actually. In any numerical package, if you do > > ifft(fft(a)), you will not recover a exactly for any non trivial size. > > For example, with floating point numbers, the order in which you do > > operations matters, so: > > > Will give you different values for d and c, even if you "on paper", > > those are exactly the same. For those reasons, it is virtually > > impossible to have exactly the same values for two different > > implementations of the same algorithm. As long as the difference is > > small (if the reconstruction error falls in the 1e-15 range, it is > > mostly likely the case), it should not matter, > > I understand the numerical mathematics behind this very well but my > point is that his two algorithms appear to be identical (same > operations, same order), he simply uses lists in one and arrays in the > other. It's not like he used vectorization or other array-related > operations - he uses for loops in both cases. Of course I agree that > 1e-15 error should be acceptable, but that's not the point. I think > there is legitimate curiosity in wondering why there is any difference > between using the two data types in exactly the same algorithm. > Well, without an example it is hard to tell. Maybe the print formats are different precisions and the list values are just getting rounded. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Apr 20 12:04:53 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 21 Apr 2009 01:04:53 +0900 Subject: [Numpy-discussion] Numpy float precision vs Python list float issue In-Reply-To: References: <4fe028e30904200644h73e7fcc6h3a5b6e1016477f8b@mail.gmail.com> <49EC7CDD.5000109@ar.media.kyoto-u.ac.jp> <49EC8B37.7040307@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220904200904n24345a10u7b032f22fa2108af@mail.gmail.com> On Tue, Apr 21, 2009 at 12:49 AM, Rob Clewley wrote: > On Mon, Apr 20, 2009 at 10:48 AM, David Cournapeau > wrote: >> Rob Clewley wrote: >>> David, >>> >>> I'm confused about your reply. I don't think Ruben was only asking why >>> you'd ever get non-zero error after the forward and inverse transform, >>> but why his implementation using lists gives zero error but using >>> arrays he gets something of order 1e-15. >>> >> >> That's more likely just an accident. Forward + inverse = id is the >> surprising thing, actually. In any numerical package, if you do >> ifft(fft(a)), you will not recover a exactly for any non trivial size. >> For example, with floating point numbers, the order in which you do >> operations matters, so: > >> Will give you different values for d and c, even if you "on paper", >> those are exactly the same. For those reasons, it is virtually >> impossible to have exactly the same values for two different >> implementations of the same algorithm. As long as the difference is >> small (if the reconstruction error falls in the 1e-15 range, it is >> mostly likely the case), it should not matter, > > I understand the numerical mathematics behind this very well but my > point is that his two algorithms appear to be identical (same > operations, same order), he simply uses lists in one and arrays in the > other. It's not like he used vectorization or other array-related > operations - he uses for loops in both cases. Of course I agree that > 1e-15 error should be acceptable, but that's not the point. I think > there is legitimate curiosity in wondering why there is any difference > between using the two data types in exactly the same algorithm. Yes, it is legitimate and healthy to worry about the difference - but the surprising thing really is the list behavior when you are used to numerical computation :) And I maintain that the algorithms are not the same in both operations. For once, the operation of using arrays on the data do not give the same data in both cases, you can see right away that m and ml are not the same, e.g. print ml - morig shows that the internal representation is not exactly the same. David From cournape at gmail.com Mon Apr 20 13:27:25 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 21 Apr 2009 02:27:25 +0900 Subject: [Numpy-discussion] NaN as dictionary key? In-Reply-To: <6c476c8a0904200742j331c4d32k4b877995e001a9ed@mail.gmail.com> References: <6c476c8a0904200742j331c4d32k4b877995e001a9ed@mail.gmail.com> Message-ID: <5b8d13220904201027r188dbf05m9fe976650906f63d@mail.gmail.com> On Mon, Apr 20, 2009 at 11:42 PM, Wes McKinney wrote: > I assume that, because NaN != NaN, even though both have the same hash value > (hash(NaN) == -32768), that Python treats any NaN double as a distinct key > in a dictionary. I think that strictly speaking, nan should not be hashable because of nan != nan. But since that's not an error in python, I am not sure we should do something about it. David From josef.pktd at gmail.com Mon Apr 20 13:34:56 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 20 Apr 2009 13:34:56 -0400 Subject: [Numpy-discussion] Numpy float precision vs Python list float issue In-Reply-To: <5b8d13220904200904n24345a10u7b032f22fa2108af@mail.gmail.com> References: <4fe028e30904200644h73e7fcc6h3a5b6e1016477f8b@mail.gmail.com> <49EC7CDD.5000109@ar.media.kyoto-u.ac.jp> <49EC8B37.7040307@ar.media.kyoto-u.ac.jp> <5b8d13220904200904n24345a10u7b032f22fa2108af@mail.gmail.com> Message-ID: <1cd32cbb0904201034p4c9b0e78mcbba2e5517136c76@mail.gmail.com> On Mon, Apr 20, 2009 at 12:04 PM, David Cournapeau wrote: > On Tue, Apr 21, 2009 at 12:49 AM, Rob Clewley > wrote: > > On Mon, Apr 20, 2009 at 10:48 AM, David Cournapeau > > wrote: > >> Rob Clewley wrote: > >>> David, > >>> > >>> I'm confused about your reply. I don't think Ruben was only asking why > >>> you'd ever get non-zero error after the forward and inverse transform, > >>> but why his implementation using lists gives zero error but using > >>> arrays he gets something of order 1e-15. > >>> > >> > >> That's more likely just an accident. Forward + inverse = id is the > >> surprising thing, actually. In any numerical package, if you do > >> ifft(fft(a)), you will not recover a exactly for any non trivial size. > >> For example, with floating point numbers, the order in which you do > >> operations matters, so: > > > >> Will give you different values for d and c, even if you "on paper", > >> those are exactly the same. For those reasons, it is virtually > >> impossible to have exactly the same values for two different > >> implementations of the same algorithm. As long as the difference is > >> small (if the reconstruction error falls in the 1e-15 range, it is > >> mostly likely the case), it should not matter, > > > > I understand the numerical mathematics behind this very well but my > > point is that his two algorithms appear to be identical (same > > operations, same order), he simply uses lists in one and arrays in the > > other. It's not like he used vectorization or other array-related > > operations - he uses for loops in both cases. Of course I agree that > > 1e-15 error should be acceptable, but that's not the point. I think > > there is legitimate curiosity in wondering why there is any difference > > between using the two data types in exactly the same algorithm. > > Yes, it is legitimate and healthy to worry about the difference - but > the surprising thing really is the list behavior when you are used to > numerical computation :) And I maintain that the algorithms are not > the same in both operations. For once, the operation of using arrays > on the data do not give the same data in both cases, you can see right > away that m and ml are not the same, e.g. > > print ml - morig > > shows that the internal representation is not exactly the same. > > I think you are copying your result to your original list instead of morigl = ml[:] use: from copy import deepcopy morigl = deepcopy(ml) this is morigl after running your script >>> morigl [[1.0000000000000002, 5.0, 6.0, 1.9999999999999998, 9.0, 7.0, 1.0, 4.0], [3.0, 6.0, 2.0, 6.9999999999999982, 4.0, 8.0, 5.0, 9.0], [8.0, 2.0, 9.0, 0.99999999999999989, 3.0000000000000004, 4.0000000000000009, 1.0, 4.9999999999999991], [3.9999999999999964, 1.0, 5.0, 7.9999999999999991, 3.0, 1.0000000000000036, 4.0, 6.9999999999999991], [5.9999999999999982, 2.0, 0.99999999999999989, 2.9999999999999996, 8.0, 2.0, 4.0, 3.0], [7.9999999999999973, 5.0, 9.0, 5.0, 4.0, 2.0000000000000009, 1.0000000000000018, 5.0], [4.0, 8.0, 5.0, 9.0, 5.9999999999999991, 3.0, 2.0, 7.0000000000000009], [5.0, 6.0, 5.0, 1.0, 7.9999999999999964, 2.0, 9.0, 3.0000000000000036]] Josef -------------- next part -------------- An HTML attachment was scrubbed... URL: From Fadhley.Salim at uk.calyon.com Mon Apr 20 13:39:27 2009 From: Fadhley.Salim at uk.calyon.com (Fadhley Salim) Date: Mon, 20 Apr 2009 18:39:27 +0100 Subject: [Numpy-discussion] Getting the only object in a zero dimensional array References: <4fe028e30904200644h73e7fcc6h3a5b6e1016477f8b@mail.gmail.com><49EC7CDD.5000109@ar.media.kyoto-u.ac.jp><49EC8B37.7040307@ar.media.kyoto-u.ac.jp><5b8d13220904200904n24345a10u7b032f22fa2108af@mail.gmail.com> <1cd32cbb0904201034p4c9b0e78mcbba2e5517136c76@mail.gmail.com> Message-ID: <7F347D91614EBC48AA7540A2A76D3BB204484099@MXCU10MX1.MSX.CIB> I have a 0d array that looks like this: myarray = array( 0.1234 ) This generates a TypeError: myarray[0] I can get it's number using a hack like this... but it looks kind of wrong: myval = myarray + 0.0 There must be a better way to do it, right? All I want is the correct way to return the value of the only item in a 0d array. Thanks! Sal -------------- next part -------------- This email does not create a legal relationship between any member of the Cr=E9dit Agricole group and the recipient or constitute investment advice. The content of this email should not be copied or disclosed (in whole or part) to any other person. It may contain information which is confidential, privileged or otherwise protected from disclosure. If you are not the intended recipient, you should notify us and delete it from your system. Emails may be monitored, are not secure and may be amended, destroyed or contain viruses and in communicating with us such conditions are accepted. Any content which does not relate to business matters is not endorsed by us. Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et des Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. Calyon is incorporated in France with limited liability and registered in England & Wales. Registration number: FC008194. Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. This message and/or any attachments is intended for the sole use of its addressee. If you are not the addressee, please immediately notify the sender and then destroy the message. As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Cr?dit Agricole CIB. All rights reserved. Ce message et ses pi?ces jointes est destin? ? l'usage exclusif de son destinataire. Si vous recevez ce message par erreur, merci d'en aviser imm?diatement l'exp?diteur et de le d?truire ensuite. Le pr?sent message pouvant ?tre alt?r? ? notre insu, CALYON Cr?dit Agricole CIB ne peut pas ?tre engag? par son contenu. Tous droits r?serv?s. From robert.kern at gmail.com Mon Apr 20 13:40:55 2009 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 20 Apr 2009 12:40:55 -0500 Subject: [Numpy-discussion] Getting the only object in a zero dimensional array In-Reply-To: <7F347D91614EBC48AA7540A2A76D3BB204484099@MXCU10MX1.MSX.CIB> References: <4fe028e30904200644h73e7fcc6h3a5b6e1016477f8b@mail.gmail.com> <49EC7CDD.5000109@ar.media.kyoto-u.ac.jp> <49EC8B37.7040307@ar.media.kyoto-u.ac.jp> <5b8d13220904200904n24345a10u7b032f22fa2108af@mail.gmail.com> <1cd32cbb0904201034p4c9b0e78mcbba2e5517136c76@mail.gmail.com> <7F347D91614EBC48AA7540A2A76D3BB204484099@MXCU10MX1.MSX.CIB> Message-ID: <3d375d730904201040n3f4c1f5br5b51710787132b37@mail.gmail.com> On Mon, Apr 20, 2009 at 12:39, Fadhley Salim wrote: > I have a 0d array that looks like this: > > myarray = array( 0.1234 ) > > This generates a TypeError: > myarray[0] myarray[()] or myarray.item() -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From matthieu.brucher at gmail.com Mon Apr 20 13:47:57 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 20 Apr 2009 19:47:57 +0200 Subject: [Numpy-discussion] Numpy float precision vs Python list float issue In-Reply-To: References: <4fe028e30904200644h73e7fcc6h3a5b6e1016477f8b@mail.gmail.com> <49EC7CDD.5000109@ar.media.kyoto-u.ac.jp> <49EC8B37.7040307@ar.media.kyoto-u.ac.jp> Message-ID: > I understand the numerical mathematics behind this very well but my > point is that his two algorithms appear to be identical (same > operations, same order), he simply uses lists in one and arrays in the > other. It's not like he used vectorization or other array-related > operations - he uses for loops in both cases. Of course I agree that > 1e-15 error should be acceptable, but that's not the point. I think > there is legitimate curiosity in wondering why there is any difference > between using the two data types in exactly the same algorithm. Hi, There are cases where the same algorithm implementation outputs different results in optimized mode, depending on data alignment. As David said, this is something that is known or at least better known. Matthieu -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From josef.pktd at gmail.com Mon Apr 20 13:54:35 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 20 Apr 2009 13:54:35 -0400 Subject: [Numpy-discussion] Numpy float precision vs Python list float issue In-Reply-To: References: <4fe028e30904200644h73e7fcc6h3a5b6e1016477f8b@mail.gmail.com> <49EC7CDD.5000109@ar.media.kyoto-u.ac.jp> <49EC8B37.7040307@ar.media.kyoto-u.ac.jp> Message-ID: <1cd32cbb0904201054h4109347ewb892b2579c75d01@mail.gmail.com> On Mon, Apr 20, 2009 at 1:47 PM, Matthieu Brucher < matthieu.brucher at gmail.com> wrote: > > I understand the numerical mathematics behind this very well but my > > point is that his two algorithms appear to be identical (same > > operations, same order), he simply uses lists in one and arrays in the > > other. It's not like he used vectorization or other array-related > > operations - he uses for loops in both cases. Of course I agree that > > 1e-15 error should be acceptable, but that's not the point. I think > > there is legitimate curiosity in wondering why there is any difference > > between using the two data types in exactly the same algorithm. > > Hi, > There are cases where the same algorithm implementation outputs > different results in optimized mode, depending on data alignment. As > David said, this is something that is known or at least better known. > > in this example, there is no difference between numpy and python list if you deepcopy instead of mutating your original lists, see: diff_npl = [[0.0]*width for i in range(height)] for row in range(height): for col in range(width): diff_npl[row][col] = s[row][col] - sl[row][col] print "\n* Difference numpy list:" print diff_npl print "maximum difference =>", max(diff_npl) print "minimum diff =>", min(diff_npl) * Difference numpy list: [[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]] maximum difference => [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] minimum diff => [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdgleeson at mac.com Mon Apr 20 13:21:00 2009 From: jdgleeson at mac.com (John Gleeson) Date: Mon, 20 Apr 2009 11:21:00 -0600 Subject: [Numpy-discussion] Numpy float precision vs Python list float issue In-Reply-To: <5b8d13220904200904n24345a10u7b032f22fa2108af@mail.gmail.com> References: <4fe028e30904200644h73e7fcc6h3a5b6e1016477f8b@mail.gmail.com> <49EC7CDD.5000109@ar.media.kyoto-u.ac.jp> <49EC8B37.7040307@ar.media.kyoto-u.ac.jp> <5b8d13220904200904n24345a10u7b032f22fa2108af@mail.gmail.com> Message-ID: <0B502B9F-F07B-4204-891D-E1156DAD2C46@mac.com> On 2009-04-20, at 10:04 AM, David Cournapeau wrote: > > Yes, it is legitimate and healthy to worry about the difference - but > the surprising thing really is the list behavior when you are used to > numerical computation :) And I maintain that the algorithms are not > the same in both operations. For once, the operation of using arrays > on the data do not give the same data in both cases, you can see right > away that m and ml are not the same, e.g. > > print ml - morig > > shows that the internal representation is not exactly the same. > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion The discrepancy David found in ml - morig vanishes if you change line 118 morigl = ml[:] to import copy morigl = copy.deepcopy(ml) There is still an issue with disagreement in minimum diff, but I'd bet it is not a floating point precision problem. John From charlesr.harris at gmail.com Mon Apr 20 14:55:39 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 20 Apr 2009 12:55:39 -0600 Subject: [Numpy-discussion] Linker script, smaller source files and symbol visibility In-Reply-To: References: <49EC6FC9.4090009@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Apr 20, 2009 at 9:48 AM, Charles R Harris wrote: > Hi David > > On Mon, Apr 20, 2009 at 6:51 AM, David Cournapeau < > david at ar.media.kyoto-u.ac.jp> wrote: > >> Hi, >> >> For quite a long time I have been bothered by the very large files >> needed for python extensions. In particular for numpy.core, which >> consists in a few files which are ~ 1 Mb, I find this a pretty high >> barrier of entry for newcomers, and it has quite a big impact on the >> code organization. I think I have found a way to split things on common >> platforms (this includes at least windows, mac os x, linux and solaris), >> without impacting other potentially less capable platforms, or static >> linking of numpy. >> > > There was a discussion of this a couple of years ago. I was in favor of > many small files maybe in subdirectories. Robert, IIRC, thought too many > small files could become confusing, so there is a fine line in there > somewhere. I am generally in favor of breaking the files up into their > functional components and maybe rewriting some of the upper level interface > files in cython. But it does need some agreement and we should probably > start by just breaking up a few files. I don't have a problem with big files > that are just collections of small routines all of the same type, > umath_loops.inc.src for instance. > > Here is a link to the start of the old discussion. You took part in it also. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsalvador.wk at gmail.com Mon Apr 20 15:50:09 2009 From: rsalvador.wk at gmail.com (Ruben Salvador) Date: Mon, 20 Apr 2009 21:50:09 +0200 Subject: [Numpy-discussion] Numpy float precision vs Python list float issue In-Reply-To: <0B502B9F-F07B-4204-891D-E1156DAD2C46@mac.com> References: <4fe028e30904200644h73e7fcc6h3a5b6e1016477f8b@mail.gmail.com> <49EC7CDD.5000109@ar.media.kyoto-u.ac.jp> <49EC8B37.7040307@ar.media.kyoto-u.ac.jp> <5b8d13220904200904n24345a10u7b032f22fa2108af@mail.gmail.com> <0B502B9F-F07B-4204-891D-E1156DAD2C46@mac.com> Message-ID: <4fe028e30904201250s56881828xaf75a07407d60b54@mail.gmail.com> Well, thanks everybody for such a quick help!! I just couldn't imagine what could arise this difference. 1e-15 is enough precision for what I will be doing, but was just curious. Anyway, this 'deepcopy' really surprised me. I have now looked for it and I think I get an idea of it, though I wouldn't expect this difference in behavior to happen. From "my logical" point of view a copy is a copy, and if I find a method called copy() I can only expect it to...copy (whether I am copying compound or "simple" objects!!). This comment is only for the shake of curiosity. I guess maybe it is just my lack of knowledge in programming in general and this is such a needed difference in copy behavior. On Mon, Apr 20, 2009 at 7:21 PM, John Gleeson wrote: > > On 2009-04-20, at 10:04 AM, David Cournapeau wrote: > > > > > Yes, it is legitimate and healthy to worry about the difference - but > > the surprising thing really is the list behavior when you are used to > > numerical computation :) And I maintain that the algorithms are not > > the same in both operations. > It's not? Would you please mind commenting this a little bit? > For once, the operation of using arrays > > on the data do not give the same data in both cases, you can see right > > away that m and ml are not the same, e.g. > I don't get what you mean > > > > > print ml - morig > > > > shows that the internal representation is not exactly the same. > > > > David > > The discrepancy David found in ml - morig vanishes if you change line > 118 > > morigl = ml[:] Sorry for this mistake :S > to > > import copy > morigl = copy.deepcopy(ml) > > There is still an issue with disagreement in minimum diff, but I'd bet > it is not a floating point precision problem. > > John > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Mon Apr 20 17:38:26 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 20 Apr 2009 14:38:26 -0700 Subject: [Numpy-discussion] Numpy float precision vs Python list float issue In-Reply-To: <4fe028e30904201250s56881828xaf75a07407d60b54@mail.gmail.com> References: <4fe028e30904200644h73e7fcc6h3a5b6e1016477f8b@mail.gmail.com> <49EC7CDD.5000109@ar.media.kyoto-u.ac.jp> <49EC8B37.7040307@ar.media.kyoto-u.ac.jp> <5b8d13220904200904n24345a10u7b032f22fa2108af@mail.gmail.com> <0B502B9F-F07B-4204-891D-E1156DAD2C46@mac.com> <4fe028e30904201250s56881828xaf75a07407d60b54@mail.gmail.com> Message-ID: <49ECEB52.6020802@noaa.gov> Ruben Salvador wrote: > Anyway, this 'deepcopy' really surprised me. I have now looked for it > and I think I get an idea of it, though I wouldn't expect this > difference in behavior to happen. From "my logical" point of view a copy > is a copy, and if I find a method called copy() I can only expect it > to...copy (whether I am copying compound or "simple" objects!!). I think this also highlights an advantage of numpy - a 2-d array is a 2-d array, NOT a 1-d array of 1-d arrays. When you used lists, you were using a list-of-lists, so copying copied the main list, but only referenced the interval lists. One other note: if you do a complex computation with floats, then reverse it, and get back EXACTLY the same numbers, you can be pretty sure something is wrong! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Mon Apr 20 18:09:46 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 20 Apr 2009 15:09:46 -0700 Subject: [Numpy-discussion] numpy Mac binary for Python 2.6 In-Reply-To: References: <5b8d13220904161842k2f2f76c9v1dde62f4655c2e57@mail.gmail.com> Message-ID: <49ECF2AA.8080004@noaa.gov> Russell E. Owen wrote: >>> http://www.pymvpa.org/devguide.html > > The patch at the end of this document worked. Has anyone submitted these patches so they'll get into bdist_mpkg? I'm guessing Ronald Oussoren would be the person to accept them, but you can post to the MacPython list to be sure. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Mon Apr 20 19:44:10 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 20 Apr 2009 16:44:10 -0700 Subject: [Numpy-discussion] help getting started In-Reply-To: <5b8d13220904132314i3a0d4ae5n907e41233ce01f1@mail.gmail.com> References: <5b8d13220904101110n5f39f7cdwd5a5fb85c1d1c405@mail.gmail.com> <49DFAF23.5030100@noaa.gov> <5b8d13220904102050l4ee0e7fdhb9b3b23d9e21a133@mail.gmail.com> <49E01913.1030805@noaa.gov> <5b8d13220904102147h3620a9d8lcd7af04ed9269a04@mail.gmail.com> <5b8d13220904132314i3a0d4ae5n907e41233ce01f1@mail.gmail.com> Message-ID: <49ED08CA.2070109@noaa.gov> David Cournapeau wrote: > Christopher, would you mind trying the following binary ? > > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/scipy-0.7.0-py2.5-macosx10.5.mpkg.tar That binary seems to be working OK on my PPC 10.4 mac, with no gfortran installed. Is it up on the main site yet? Thanks for all your work on this, -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From david at ar.media.kyoto-u.ac.jp Mon Apr 20 22:43:24 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 21 Apr 2009 11:43:24 +0900 Subject: [Numpy-discussion] Numpy float precision vs Python list float issue In-Reply-To: <4fe028e30904201250s56881828xaf75a07407d60b54@mail.gmail.com> References: <4fe028e30904200644h73e7fcc6h3a5b6e1016477f8b@mail.gmail.com> <49EC7CDD.5000109@ar.media.kyoto-u.ac.jp> <49EC8B37.7040307@ar.media.kyoto-u.ac.jp> <5b8d13220904200904n24345a10u7b032f22fa2108af@mail.gmail.com> <0B502B9F-F07B-4204-891D-E1156DAD2C46@mac.com> <4fe028e30904201250s56881828xaf75a07407d60b54@mail.gmail.com> Message-ID: <49ED32CC.2040508@ar.media.kyoto-u.ac.jp> Ruben Salvador wrote: > Anyway, this 'deepcopy' really surprised me. I have now looked for it > and I think I get an idea of it, though I wouldn't expect this > difference in behavior to happen. From "my logical" point of view a > copy is a copy, and if I find a method called copy() I can only expect > it to...copy (whether I am copying compound or "simple" objects!!). > This comment is only for the shake of curiosity. I guess maybe it is > just my lack of knowledge in programming in general and this is such a > needed difference in copy behavior. That's something which is related to python. Python lists are mutable, meaning that when you modify one variable, you modify it in place. Simply said, copying a list means creating a new contained with the *same* items, and a deepcopy means creating a new container with a (potentially recursive) copy of every item. > > On Mon, Apr 20, 2009 at 7:21 PM, John Gleeson > wrote: > > > On 2009-04-20, at 10:04 AM, David Cournapeau wrote: > > > > > Yes, it is legitimate and healthy to worry about the difference > - but > > the surprising thing really is the list behavior when you are > used to > > numerical computation :) And I maintain that the algorithms are not > > the same in both operations. > > > It's not? Would you please mind commenting this a little bit? they are not because of the conversion to numpy array. And maybe other implementation details. The thing which matters is which operations are done in the hardware, and that's really difficult to control exactly once you have two implementations. The same code could give different results on different OS, different platforms, etc... The main point is to remember that you can't expect exact results with floating point units, and that for all practical purpose, you can't guarantee two implementations to be the same. For example, python float behavior and numpy array behavior may be different w.r.t FPU exceptions, depending on the environment and your settings.. They *could* be equal, but that would most likely be an accident. cheers, David From david at ar.media.kyoto-u.ac.jp Tue Apr 21 00:13:02 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 21 Apr 2009 13:13:02 +0900 Subject: [Numpy-discussion] Linker script, smaller source files and symbol visibility In-Reply-To: References: <49EC6FC9.4090009@ar.media.kyoto-u.ac.jp> Message-ID: <49ED47CE.7050100@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > Here is a link to the start of the old discussion > . > You took part in it also. Thanks, I remembered we had the discussion, but could not find it. The different is that I am much more familiar with the technical details and numpy codebase now :) I know how to control exported symbols on most platform which matter (I can't test for AIX or HP-UX unfortunately - but I am perfectly fine with ignoring namespace pollution on those anyway), and I would guess that the only platforms which do not support symbol visibility in one way or the other do not support shared library anyway (some CRAY stuff, for example). Concerning the file size, I don't think anyone would disagree that they are too big, but we don't need to go the "java-way" of one file/class-function either. One first split which I personally like is API/implementation. For example, for multiarray.c, we would only keep the public PyArray_* functions, and put everything else in another file. The other very big file is arrayobject.c, and this one is already mostly organized in independent parts (buffer protocol, number protocol, etc...) Another thing I would like to do it to make the global C API array pointer a 'true' global variable instead of a static one. It took me a while when I was working on the hashing protocol for dtype to understand why it was crashing (the array pointer being static, every file has its own copy, so it was never initialized in the hashdescr.c file). I think a true global variable, hidden through a symbol map, is easier to understand and more reliable. cheers, David From charlesr.harris at gmail.com Tue Apr 21 01:06:33 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 20 Apr 2009 23:06:33 -0600 Subject: [Numpy-discussion] Linker script, smaller source files and symbol visibility In-Reply-To: <49ED47CE.7050100@ar.media.kyoto-u.ac.jp> References: <49EC6FC9.4090009@ar.media.kyoto-u.ac.jp> <49ED47CE.7050100@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Apr 20, 2009 at 10:13 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > > Here is a link to the start of the old discussion > > < > http://article.gmane.org/gmane.comp.python.numeric.general/12974/match=exported+symbols+code+reorganization > >. > > You took part in it also. > > Thanks, I remembered we had the discussion, but could not find it. The > different is that I am much more familiar with the technical details and > numpy codebase now :) I know how to control exported symbols on most > platform which matter (I can't test for AIX or HP-UX unfortunately - but > I am perfectly fine with ignoring namespace pollution on those anyway), > and I would guess that the only platforms which do not support symbol > visibility in one way or the other do not support shared library anyway > (some CRAY stuff, for example). > > Concerning the file size, I don't think anyone would disagree that they > are too big, but we don't need to go the "java-way" of one > file/class-function either. One first split which I personally like is > API/implementation. For example, for multiarray.c, we would only keep > the public PyArray_* functions, and put everything else in another file. > The other very big file is arrayobject.c, and this one is already mostly > organized in independent parts (buffer protocol, number protocol, etc...) > > Another thing I would like to do it to make the global C API array > pointer a 'true' global variable instead of a static one. It took me a > while when I was working on the hashing protocol for dtype to understand > why it was crashing (the array pointer being static, every file has its > own copy, so it was never initialized in the hashdescr.c file). I think > a true global variable, hidden through a symbol map, is easier to > understand and more reliable. > I made an experiment along those lines a couple of years ago. There were compilation problems because the needed include files weren't available. No doubt that could be fixed in the build, but at some point I would like to have real include files, not the generated variety. Generated include files are kind of bogus IMHO, as they don't define an interface but rather reflect whatever the function definition happens to be. So as any part of a split I would also suggest writing the associated include files. That would also make separate compilation possible, which would make it easier to do test compilations while doing development. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From rowen at u.washington.edu Tue Apr 21 13:29:17 2009 From: rowen at u.washington.edu (Russell E. Owen) Date: Tue, 21 Apr 2009 10:29:17 -0700 Subject: [Numpy-discussion] numpy Mac binary for Python 2.6 References: <5b8d13220904161842k2f2f76c9v1dde62f4655c2e57@mail.gmail.com> <49ECF2AA.8080004@noaa.gov> Message-ID: In article <49ECF2AA.8080004 at noaa.gov>, Christopher Barker wrote: > Russell E. Owen wrote: > >>> http://www.pymvpa.org/devguide.html > > > > The patch at the end of this document worked. > > Has anyone submitted these patches so they'll get into bdist_mpkg? I'm > guessing Ronald Oussoren would be the person to accept them, but you can > post to the MacPython list to be sure. > > -Chris I have not, but would be happy to -- but where? The bdist_mpkg home pagem doesn't seem to list a bug tracker. -- Russell From emmanuelle.gouillart at normalesup.org Tue Apr 21 16:23:46 2009 From: emmanuelle.gouillart at normalesup.org (Emmanuelle Gouillart) Date: Tue, 21 Apr 2009 22:23:46 +0200 Subject: [Numpy-discussion] need some help with python scientific packages In-Reply-To: <49ED9A25.6030300@auf.org> References: <49ED9A25.6030300@auf.org> Message-ID: <20090421202346.GB16870@phare.normalesup.org> Hi Nacer, > I read your message on the african python tour mailing list. I am > looking for someone able to help me find the right scientific package > for parallel computing (data mining of frequent patterns). I am > involve in a DEA course and need such a package to illustrate the > topics I am developping in my "memoire". Could you please point me > some resources on the subject? first of all, I suggest you to ask such questions on the mailing-lists dedicated to scientific Python, among which numpy-discussion at scipy.org (Cced in this e-mail) or SciPy Users List scipy-user at scipy.org. These are very active mailing-lists, where you may get answers from a large pool of experts in scientific Python, and it is always faster and more efficient than asking a single person -- all the more than I'm not an expert of frequent pattern mining or parallel computing at all! You have to suscribe first to these lists in order to get the answers to your questions. Did you take a look at the page dedicated to parallel computing with Python http://wiki.python.org/moin/ParallelProcessing ? (You don't mention explicitely in your message that you're using Python for data processing, can you first confirm this?). You may see from this page that there is a great deal of possibilities. It is difficult for me to give you more information without knowing some further details. Could you please tell us: * which kind of parallel computing do you want to perform? And on which architecture? Do you want to run jobs on a multicore computer with shared memory or to perform distributed computing on a cluster? The solutions may be quite different depending on the answer. * do you wish to perform fine-grained parallel computing (e.g. do you need to separate your data objects into different parts that will be processed on different cores and "glue together" the output of the different tasks), or do you have independent high-level operations that could be easily parallelized (e.g. independent processing of different datasets) * and in general, can you tell us a little more about what you want to do? However, once again, I don't know much about such subjects, you might get far more interesting answers from the list. Cheers (and good luck with you Master thesis :D), Emmanuelle From Chris.Barker at noaa.gov Tue Apr 21 16:24:57 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 21 Apr 2009 13:24:57 -0700 Subject: [Numpy-discussion] numpy Mac binary for Python 2.6 In-Reply-To: References: <5b8d13220904161842k2f2f76c9v1dde62f4655c2e57@mail.gmail.com> <49ECF2AA.8080004@noaa.gov> Message-ID: <49EE2B99.4060607@noaa.gov> Russell E. Owen wrote: > Christopher Barker wrote: >> Has anyone submitted these patches so they'll get into bdist_mpkg? I'm >> guessing Ronald Oussoren would be the person to accept them, but you can >> post to the MacPython list to be sure. >> >> -Chris > > I have not, but would be happy to -- but where? The bdist_mpkg home > pagem doesn't seem to list a > bug tracker. bdist_mpkg is a part of py2app, which was written by Bob Ippolito, but is now kin dof orphaned. Ronald has access to the repository and has been applying small patches. He has talked about rolling it into the pyObjC project, which would give it a real home. In the meantime, posting it to PythonMac list will probably do it, and maybe cc Ronald just to make sure. Thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From myeates at jpl.nasa.gov Tue Apr 21 17:58:03 2009 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Tue, 21 Apr 2009 14:58:03 -0700 Subject: [Numpy-discussion] performance issue (again) Message-ID: <49EE416B.3010406@jpl.nasa.gov> Hi I posted something about this earlier Say I have 2 arrays X and Y with shapes (N,3) where N is large I am doing the following for row in range(N): result=polyfit(X[row,:],Y[row,:],1,full=True) # fit 3 points with a line This takes forever and I was hoping to find a way to speed things up. But now I'm starting to wonder if this pointless. If the routine "poly fit takes a long time, when compared with the time for a Python function call, then things can't be sped up. Any comments? Mathew From josef.pktd at gmail.com Tue Apr 21 18:10:43 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 21 Apr 2009 18:10:43 -0400 Subject: [Numpy-discussion] performance issue (again) In-Reply-To: <49EE416B.3010406@jpl.nasa.gov> References: <49EE416B.3010406@jpl.nasa.gov> Message-ID: <1cd32cbb0904211510v27caf40cqf2aac7176fb89b6@mail.gmail.com> On Tue, Apr 21, 2009 at 5:58 PM, Mathew Yeates wrote: > Hi > I posted something about this earlier > > Say I have 2 arrays X and Y with shapes (N,3) where N is large > I am doing the following > > for row in range(N): > result=polyfit(X[row,:],Y[row,:],1,full=True) # fit 3 points with a line > > This takes forever and I was hoping to find a way to speed things up. > But now I'm starting to wonder if this pointless. If the routine "poly > fit takes a long time, when compared with the time for a Python > function call, then things can't be sped up. > > ols with one regressor and 3 observations takes no time to calculate, I think you have (almost) only setup cost. Josef -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Tue Apr 21 18:23:11 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 21 Apr 2009 15:23:11 -0700 Subject: [Numpy-discussion] performance issue (again) In-Reply-To: <49EE416B.3010406@jpl.nasa.gov> References: <49EE416B.3010406@jpl.nasa.gov> Message-ID: On 4/21/09, Mathew Yeates wrote: > Hi > I posted something about this earlier > > Say I have 2 arrays X and Y with shapes (N,3) where N is large > I am doing the following > > for row in range(N): > result=polyfit(X[row,:],Y[row,:],1,full=True) # fit 3 points with a line > > This takes forever and I was hoping to find a way to speed things up. > But now I'm starting to wonder if this pointless. If the routine "poly > fit takes a long time, when compared with the time for a Python > function call, then things can't be sped up. > > Any comments? If you remove the mean from x and y (along axis = 1) then can't you just do something like (x*y).sum(1) / (x*x).sum(axis=1) From josef.pktd at gmail.com Tue Apr 21 18:41:12 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 21 Apr 2009 18:41:12 -0400 Subject: [Numpy-discussion] performance issue (again) In-Reply-To: References: <49EE416B.3010406@jpl.nasa.gov> Message-ID: <1cd32cbb0904211541k7cc79ee1g6978a752e7872529@mail.gmail.com> On Tue, Apr 21, 2009 at 6:23 PM, Keith Goodman wrote: > On 4/21/09, Mathew Yeates wrote: > > Hi > > I posted something about this earlier > > > > Say I have 2 arrays X and Y with shapes (N,3) where N is large > > I am doing the following > > > > for row in range(N): > > result=polyfit(X[row,:],Y[row,:],1,full=True) # fit 3 points with a > line > > > > This takes forever and I was hoping to find a way to speed things up. > > But now I'm starting to wonder if this pointless. If the routine "poly > > fit takes a long time, when compared with the time for a Python > > function call, then things can't be sped up. > > > > Any comments? > > If you remove the mean from x and y (along axis = 1) then can't you > just do something like > > (x*y).sum(1) / (x*x).sum(axis=1) > I think that's what I said 8 days ago. Josef -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Tue Apr 21 19:04:20 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 21 Apr 2009 16:04:20 -0700 Subject: [Numpy-discussion] performance issue (again) In-Reply-To: <1cd32cbb0904211541k7cc79ee1g6978a752e7872529@mail.gmail.com> References: <49EE416B.3010406@jpl.nasa.gov> <1cd32cbb0904211541k7cc79ee1g6978a752e7872529@mail.gmail.com> Message-ID: On 4/21/09, josef.pktd at gmail.com wrote: > On Tue, Apr 21, 2009 at 6:23 PM, Keith Goodman wrote: > >> On 4/21/09, Mathew Yeates wrote: >> > Hi >> > I posted something about this earlier >> > >> > Say I have 2 arrays X and Y with shapes (N,3) where N is large >> > I am doing the following >> > >> > for row in range(N): >> > result=polyfit(X[row,:],Y[row,:],1,full=True) # fit 3 points with a >> line >> > >> > This takes forever and I was hoping to find a way to speed things up. >> > But now I'm starting to wonder if this pointless. If the routine "poly >> > fit takes a long time, when compared with the time for a Python >> > function call, then things can't be sped up. >> > >> > Any comments? >> >> If you remove the mean from x and y (along axis = 1) then can't you >> just do something like >> >> (x*y).sum(1) / (x*x).sum(axis=1) >> > > I think that's what I said 8 days ago. Nice. You even calculated the intercepts. From myeates at jpl.nasa.gov Tue Apr 21 19:33:01 2009 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Tue, 21 Apr 2009 16:33:01 -0700 Subject: [Numpy-discussion] performance issue (again) In-Reply-To: References: <49EE416B.3010406@jpl.nasa.gov> Message-ID: <49EE57AD.4060209@jpl.nasa.gov> sheer genius. Done in the blink of an eye and my original was taking 20 minutes! Keith Goodman wrote: > On 4/21/09, Mathew Yeates wrote: > >> Hi >> I posted something about this earlier >> >> Say I have 2 arrays X and Y with shapes (N,3) where N is large >> I am doing the following >> >> for row in range(N): >> result=polyfit(X[row,:],Y[row,:],1,full=True) # fit 3 points with a line >> >> This takes forever and I was hoping to find a way to speed things up. >> But now I'm starting to wonder if this pointless. If the routine "poly >> fit takes a long time, when compared with the time for a Python >> function call, then things can't be sped up. >> >> Any comments? >> > > If you remove the mean from x and y (along axis = 1) then can't you > just do something like > > (x*y).sum(1) / (x*x).sum(axis=1) > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From myeates at jpl.nasa.gov Tue Apr 21 19:38:53 2009 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Tue, 21 Apr 2009 16:38:53 -0700 Subject: [Numpy-discussion] performance issue (again) In-Reply-To: References: <49EE416B.3010406@jpl.nasa.gov> Message-ID: <49EE590D.1050705@jpl.nasa.gov> sheer genius. Done in the blink of an eye and my original was taking 20 minutes! Keith Goodman wrote: > On 4/21/09, Mathew Yeates wrote: > >> Hi >> I posted something about this earlier >> >> Say I have 2 arrays X and Y with shapes (N,3) where N is large >> I am doing the following >> >> for row in range(N): >> result=polyfit(X[row,:],Y[row,:],1,full=True) # fit 3 points with a line >> >> This takes forever and I was hoping to find a way to speed things up. >> But now I'm starting to wonder if this pointless. If the routine "poly >> fit takes a long time, when compared with the time for a Python >> function call, then things can't be sped up. >> >> Any comments? >> > > If you remove the mean from x and y (along axis = 1) then can't you > just do something like > > (x*y).sum(1) / (x*x).sum(axis=1) > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From myeates at jpl.nasa.gov Tue Apr 21 19:41:25 2009 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Tue, 21 Apr 2009 16:41:25 -0700 Subject: [Numpy-discussion] performance issue (again) In-Reply-To: References: <49EE416B.3010406@jpl.nasa.gov> Message-ID: <49EE59A5.7020302@jpl.nasa.gov> sheer genius. Done in the blink of an eye and my original was taking 20 minutes! Keith Goodman wrote: > On 4/21/09, Mathew Yeates wrote: > >> Hi >> I posted something about this earlier >> >> Say I have 2 arrays X and Y with shapes (N,3) where N is large >> I am doing the following >> >> for row in range(N): >> result=polyfit(X[row,:],Y[row,:],1,full=True) # fit 3 points with a line >> >> This takes forever and I was hoping to find a way to speed things up. >> But now I'm starting to wonder if this pointless. If the routine "poly >> fit takes a long time, when compared with the time for a Python >> function call, then things can't be sped up. >> >> Any comments? >> > > If you remove the mean from x and y (along axis = 1) then can't you > just do something like > > (x*y).sum(1) / (x*x).sum(axis=1) > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Wed Apr 22 01:24:38 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 21 Apr 2009 23:24:38 -0600 Subject: [Numpy-discussion] Linker script, smaller source files and symbol visibility In-Reply-To: References: <49EC6FC9.4090009@ar.media.kyoto-u.ac.jp> <49ED47CE.7050100@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Apr 20, 2009 at 11:06 PM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Mon, Apr 20, 2009 at 10:13 PM, David Cournapeau < > david at ar.media.kyoto-u.ac.jp> wrote: > >> Charles R Harris wrote: >> >> > >> > Here is a link to the start of the old discussion >> > < >> http://article.gmane.org/gmane.comp.python.numeric.general/12974/match=exported+symbols+code+reorganization >> >. >> > You took part in it also. >> >> Thanks, I remembered we had the discussion, but could not find it. The >> different is that I am much more familiar with the technical details and >> numpy codebase now :) I know how to control exported symbols on most >> platform which matter (I can't test for AIX or HP-UX unfortunately - but >> I am perfectly fine with ignoring namespace pollution on those anyway), >> and I would guess that the only platforms which do not support symbol >> visibility in one way or the other do not support shared library anyway >> (some CRAY stuff, for example). >> >> Concerning the file size, I don't think anyone would disagree that they >> are too big, but we don't need to go the "java-way" of one >> file/class-function either. One first split which I personally like is >> API/implementation. For example, for multiarray.c, we would only keep >> the public PyArray_* functions, and put everything else in another file. >> The other very big file is arrayobject.c, and this one is already mostly >> organized in independent parts (buffer protocol, number protocol, etc...) >> >> Another thing I would like to do it to make the global C API array >> pointer a 'true' global variable instead of a static one. It took me a >> while when I was working on the hashing protocol for dtype to understand >> why it was crashing (the array pointer being static, every file has its >> own copy, so it was never initialized in the hashdescr.c file). I think >> a true global variable, hidden through a symbol map, is easier to >> understand and more reliable. >> > > I made an experiment along those lines a couple of years ago. There were > compilation problems because the needed include files weren't available. No > doubt that could be fixed in the build, but at some point I would like to > have real include files, not the generated variety. Generated include files > are kind of bogus IMHO, as they don't define an interface but rather reflect > whatever the function definition happens to be. So as any part of a split I > would also suggest writing the associated include files. That would also > make separate compilation possible, which would make it easier to do test > compilations while doing development. > The list of visible symbols has grown ;) ./multiarray.so 00039360 T PyArray_DescrHash 00073698 T _fini 0003e200 T _flat_copyinto 00007c58 T _init 00027d70 T initmultiarray 000728e0 T npy_acos 000723d0 T npy_acosf 00072880 T npy_acosh 00072370 T npy_acoshf 00072e50 T npy_acoshl 00072ee0 T npy_acosl 00072900 T npy_asin 000723f0 T npy_asinf 000728a0 T npy_asinh 00072390 T npy_asinhf 00072e80 T npy_asinhl 00072f10 T npy_asinl 000728c0 T npy_atan 000726b0 T npy_atan2 000721b0 T npy_atan2f 00072c10 T npy_atan2l 000723b0 T npy_atanf 00072860 T npy_atanh 00072350 T npy_atanhf 00072e20 T npy_atanhl 00072eb0 T npy_atanl 00071f70 T npy_ceil 00072040 T npy_ceilf 00071eb0 T npy_ceill 00072a90 T npy_cos 00072580 T npy_cosf 00072a30 T npy_cosh 00072520 T npy_coshf 000730b0 T npy_coshl 00073140 T npy_cosl 00071fd0 T npy_deg2rad 000720a0 T npy_deg2radf 00071f00 T npy_deg2radl 000726e0 T npy_exp 000727b0 T npy_exp2 00072720 T npy_exp2_1m 00072220 T npy_exp2_1mf 00072cc0 T npy_exp2_1ml 000722b0 T npy_exp2f 00072d50 T npy_exp2l 000721e0 T npy_expf 00072c60 T npy_expl 00072920 T npy_expm1 00072410 T npy_expm1f 00072f40 T npy_expm1l 00071f20 T npy_fabs 00071ff0 T npy_fabsf 00071e70 T npy_fabsl 00071f30 T npy_floor 00072000 T npy_floorf 00071e80 T npy_floorl 000725f0 T npy_fmod 000720f0 T npy_fmodf 00072b10 T npy_fmodl 00072680 T npy_hypot 00072180 T npy_hypotf 00072bc0 T npy_hypotl 00072940 T npy_log 00072960 T npy_log10 00072450 T npy_log10f 00072fa0 T npy_log10l 000727d0 T npy_log1p 000722d0 T npy_log1pf 00072d80 T npy_log1pl 00072700 T npy_log2 00072200 T npy_log2f 00072c90 T npy_log2l 000727f0 T npy_logaddexp 00073290 T npy_logaddexp2 00073390 T npy_logaddexp2f 000731a0 T npy_logaddexp2l 000722f0 T npy_logaddexpf 00072db0 T npy_logaddexpl 00072430 T npy_logf 00072f70 T npy_logl 000725c0 T npy_modf 000720c0 T npy_modff 00072ad0 T npy_modfl 00072650 T npy_pow 00072150 T npy_powf 00072b70 T npy_powl 00071fb0 T npy_rad2deg 00072080 T npy_rad2degf 00071ee0 T npy_rad2degl 000729f0 T npy_rint 000724e0 T npy_rintf 00073050 T npy_rintl 00072ab0 T npy_sin 000725a0 T npy_sinf 00072a50 T npy_sinh 00072540 T npy_sinhf 000730e0 T npy_sinhl 00073170 T npy_sinl 00072980 T npy_sqrt 00072470 T npy_sqrtf 00072fd0 T npy_sqrtl 00072a70 T npy_tan 00072560 T npy_tanf 00072a10 T npy_tanh 00072500 T npy_tanhf 00073080 T npy_tanhl 00073110 T npy_tanl 000729d0 T npy_trunc 000724c0 T npy_truncf 00073020 T npy_truncl ./umath_tests.so 00001a38 T _fini 000006b0 T _init 000014c0 T initumath_tests ./_sort.so 00019658 T _fini 000005a0 T _init 0000e700 T init_sort ./_dotblas.so 000049d8 T _fini 00000918 T _init 00004680 T init_dotblas ./umath.so 0002f368 T _fini 00004b64 T _init 0002ce10 T initumath 0002e120 T npy_acos 0002dc10 T npy_acosf 0002e0c0 T npy_acosh 0002dbb0 T npy_acoshf 0002e690 T npy_acoshl 0002e720 T npy_acosl 0002e140 T npy_asin 0002dc30 T npy_asinf 0002e0e0 T npy_asinh 0002dbd0 T npy_asinhf 0002e6c0 T npy_asinhl 0002e750 T npy_asinl 0002e100 T npy_atan 0002def0 T npy_atan2 0002d9f0 T npy_atan2f 0002e450 T npy_atan2l 0002dbf0 T npy_atanf 0002e0a0 T npy_atanh 0002db90 T npy_atanhf 0002e660 T npy_atanhl 0002e6f0 T npy_atanl 0002d7b0 T npy_ceil 0002d880 T npy_ceilf 0002d6f0 T npy_ceill 0002e2d0 T npy_cos 0002ddc0 T npy_cosf 0002e270 T npy_cosh 0002dd60 T npy_coshf 0002e8f0 T npy_coshl 0002e980 T npy_cosl 0002d810 T npy_deg2rad 0002d8e0 T npy_deg2radf 0002d740 T npy_deg2radl 0002df20 T npy_exp 0002dff0 T npy_exp2 0002df60 T npy_exp2_1m 0002da60 T npy_exp2_1mf 0002e500 T npy_exp2_1ml 0002daf0 T npy_exp2f 0002e590 T npy_exp2l 0002da20 T npy_expf 0002e4a0 T npy_expl 0002e160 T npy_expm1 0002dc50 T npy_expm1f 0002e780 T npy_expm1l 0002d760 T npy_fabs 0002d830 T npy_fabsf 0002d6b0 T npy_fabsl 0002d770 T npy_floor 0002d840 T npy_floorf 0002d6c0 T npy_floorl 0002de30 T npy_fmod 0002d930 T npy_fmodf 0002e350 T npy_fmodl 0002dec0 T npy_hypot 0002d9c0 T npy_hypotf 0002e400 T npy_hypotl 0002e180 T npy_log 0002e1a0 T npy_log10 0002dc90 T npy_log10f 0002e7e0 T npy_log10l 0002e010 T npy_log1p 0002db10 T npy_log1pf 0002e5c0 T npy_log1pl 0002df40 T npy_log2 0002da40 T npy_log2f 0002e4d0 T npy_log2l 0002e030 T npy_logaddexp 0002ead0 T npy_logaddexp2 0002ebd0 T npy_logaddexp2f 0002e9e0 T npy_logaddexp2l 0002db30 T npy_logaddexpf 0002e5f0 T npy_logaddexpl 0002dc70 T npy_logf 0002e7b0 T npy_logl 0002de00 T npy_modf 0002d900 T npy_modff 0002e310 T npy_modfl 0002de90 T npy_pow 0002d990 T npy_powf 0002e3b0 T npy_powl 0002d7f0 T npy_rad2deg 0002d8c0 T npy_rad2degf 0002d720 T npy_rad2degl 0002e230 T npy_rint 0002dd20 T npy_rintf 0002e890 T npy_rintl 0002e2f0 T npy_sin 0002dde0 T npy_sinf 0002e290 T npy_sinh 0002dd80 T npy_sinhf 0002e920 T npy_sinhl 0002e9b0 T npy_sinl 0002e1c0 T npy_sqrt 0002dcb0 T npy_sqrtf 0002e810 T npy_sqrtl 0002e2b0 T npy_tan 0002dda0 T npy_tanf 0002e250 T npy_tanh 0002dd40 T npy_tanhf 0002e8c0 T npy_tanhl 0002e950 T npy_tanl 0002e210 T npy_trunc 0002dd00 T npy_truncf 0002e860 T npy_truncl ./scalarmath.so 000217d8 T _fini 000013d0 T _init 00001b50 T initscalarmath $[charris at f9 core]$ Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Apr 22 01:38:08 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 21 Apr 2009 23:38:08 -0600 Subject: [Numpy-discussion] Linker script, smaller source files and symbol visibility In-Reply-To: References: <49EC6FC9.4090009@ar.media.kyoto-u.ac.jp> <49ED47CE.7050100@ar.media.kyoto-u.ac.jp> Message-ID: On Tue, Apr 21, 2009 at 11:24 PM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Mon, Apr 20, 2009 at 11:06 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Mon, Apr 20, 2009 at 10:13 PM, David Cournapeau < >> david at ar.media.kyoto-u.ac.jp> wrote: >> >>> Charles R Harris wrote: >>> >>> > >>> > Here is a link to the start of the old discussion >>> > < >>> http://article.gmane.org/gmane.comp.python.numeric.general/12974/match=exported+symbols+code+reorganization >>> >. >>> > You took part in it also. >>> >>> Thanks, I remembered we had the discussion, but could not find it. The >>> different is that I am much more familiar with the technical details and >>> numpy codebase now :) I know how to control exported symbols on most >>> platform which matter (I can't test for AIX or HP-UX unfortunately - but >>> I am perfectly fine with ignoring namespace pollution on those anyway), >>> and I would guess that the only platforms which do not support symbol >>> visibility in one way or the other do not support shared library anyway >>> (some CRAY stuff, for example). >>> >>> Concerning the file size, I don't think anyone would disagree that they >>> are too big, but we don't need to go the "java-way" of one >>> file/class-function either. One first split which I personally like is >>> API/implementation. For example, for multiarray.c, we would only keep >>> the public PyArray_* functions, and put everything else in another file. >>> The other very big file is arrayobject.c, and this one is already mostly >>> organized in independent parts (buffer protocol, number protocol, etc...) >>> >>> Another thing I would like to do it to make the global C API array >>> pointer a 'true' global variable instead of a static one. It took me a >>> while when I was working on the hashing protocol for dtype to understand >>> why it was crashing (the array pointer being static, every file has its >>> own copy, so it was never initialized in the hashdescr.c file). I think >>> a true global variable, hidden through a symbol map, is easier to >>> understand and more reliable. >>> >> >> I made an experiment along those lines a couple of years ago. There were >> compilation problems because the needed include files weren't available. No >> doubt that could be fixed in the build, but at some point I would like to >> have real include files, not the generated variety. Generated include files >> are kind of bogus IMHO, as they don't define an interface but rather reflect >> whatever the function definition happens to be. So as any part of a split I >> would also suggest writing the associated include files. That would also >> make separate compilation possible, which would make it easier to do test >> compilations while doing development. >> > > The list of visible symbols has grown ;) > > ./multiarray.so > 00039360 T PyArray_DescrHash > 00073698 T _fini > 0003e200 T _flat_copyinto > 00007c58 T _init > 00027d70 T initmultiarray > 000728e0 T npy_acos > 000723d0 T npy_acosf > 00072880 T npy_acosh > 00072370 T npy_acoshf > 00072e50 T npy_acoshl > 00072ee0 T npy_acosl > 00072900 T npy_asin > 000723f0 T npy_asinf > 000728a0 T npy_asinh > 00072390 T npy_asinhf > 00072e80 T npy_asinhl > 00072f10 T npy_asinl > 000728c0 T npy_atan > 000726b0 T npy_atan2 > 000721b0 T npy_atan2f > 00072c10 T npy_atan2l > 000723b0 T npy_atanf > 00072860 T npy_atanh > 00072350 T npy_atanhf > 00072e20 T npy_atanhl > 00072eb0 T npy_atanl > 00071f70 T npy_ceil > 00072040 T npy_ceilf > 00071eb0 T npy_ceill > 00072a90 T npy_cos > 00072580 T npy_cosf > 00072a30 T npy_cosh > 00072520 T npy_coshf > 000730b0 T npy_coshl > 00073140 T npy_cosl > 00071fd0 T npy_deg2rad > 000720a0 T npy_deg2radf > 00071f00 T npy_deg2radl > 000726e0 T npy_exp > 000727b0 T npy_exp2 > 00072720 T npy_exp2_1m > 00072220 T npy_exp2_1mf > 00072cc0 T npy_exp2_1ml > 000722b0 T npy_exp2f > 00072d50 T npy_exp2l > 000721e0 T npy_expf > 00072c60 T npy_expl > 00072920 T npy_expm1 > 00072410 T npy_expm1f > 00072f40 T npy_expm1l > 00071f20 T npy_fabs > 00071ff0 T npy_fabsf > 00071e70 T npy_fabsl > 00071f30 T npy_floor > 00072000 T npy_floorf > 00071e80 T npy_floorl > 000725f0 T npy_fmod > 000720f0 T npy_fmodf > 00072b10 T npy_fmodl > 00072680 T npy_hypot > 00072180 T npy_hypotf > 00072bc0 T npy_hypotl > 00072940 T npy_log > 00072960 T npy_log10 > 00072450 T npy_log10f > 00072fa0 T npy_log10l > 000727d0 T npy_log1p > 000722d0 T npy_log1pf > 00072d80 T npy_log1pl > 00072700 T npy_log2 > 00072200 T npy_log2f > 00072c90 T npy_log2l > 000727f0 T npy_logaddexp > 00073290 T npy_logaddexp2 > 00073390 T npy_logaddexp2f > 000731a0 T npy_logaddexp2l > 000722f0 T npy_logaddexpf > 00072db0 T npy_logaddexpl > 00072430 T npy_logf > 00072f70 T npy_logl > 000725c0 T npy_modf > 000720c0 T npy_modff > 00072ad0 T npy_modfl > 00072650 T npy_pow > 00072150 T npy_powf > 00072b70 T npy_powl > 00071fb0 T npy_rad2deg > 00072080 T npy_rad2degf > 00071ee0 T npy_rad2degl > 000729f0 T npy_rint > 000724e0 T npy_rintf > 00073050 T npy_rintl > 00072ab0 T npy_sin > 000725a0 T npy_sinf > 00072a50 T npy_sinh > 00072540 T npy_sinhf > 000730e0 T npy_sinhl > 00073170 T npy_sinl > 00072980 T npy_sqrt > 00072470 T npy_sqrtf > 00072fd0 T npy_sqrtl > 00072a70 T npy_tan > 00072560 T npy_tanf > 00072a10 T npy_tanh > 00072500 T npy_tanhf > 00073080 T npy_tanhl > 00073110 T npy_tanl > 000729d0 T npy_trunc > 000724c0 T npy_truncf > 00073020 T npy_truncl > ./umath_tests.so > 00001a38 T _fini > 000006b0 T _init > 000014c0 T initumath_tests > ./_sort.so > 00019658 T _fini > 000005a0 T _init > 0000e700 T init_sort > ./_dotblas.so > 000049d8 T _fini > 00000918 T _init > 00004680 T init_dotblas > ./umath.so > 0002f368 T _fini > 00004b64 T _init > 0002ce10 T initumath > 0002e120 T npy_acos > 0002dc10 T npy_acosf > 0002e0c0 T npy_acosh > 0002dbb0 T npy_acoshf > 0002e690 T npy_acoshl > 0002e720 T npy_acosl > 0002e140 T npy_asin > 0002dc30 T npy_asinf > 0002e0e0 T npy_asinh > 0002dbd0 T npy_asinhf > 0002e6c0 T npy_asinhl > 0002e750 T npy_asinl > 0002e100 T npy_atan > 0002def0 T npy_atan2 > 0002d9f0 T npy_atan2f > 0002e450 T npy_atan2l > 0002dbf0 T npy_atanf > 0002e0a0 T npy_atanh > 0002db90 T npy_atanhf > 0002e660 T npy_atanhl > 0002e6f0 T npy_atanl > 0002d7b0 T npy_ceil > 0002d880 T npy_ceilf > 0002d6f0 T npy_ceill > 0002e2d0 T npy_cos > 0002ddc0 T npy_cosf > 0002e270 T npy_cosh > 0002dd60 T npy_coshf > 0002e8f0 T npy_coshl > 0002e980 T npy_cosl > 0002d810 T npy_deg2rad > 0002d8e0 T npy_deg2radf > 0002d740 T npy_deg2radl > 0002df20 T npy_exp > 0002dff0 T npy_exp2 > 0002df60 T npy_exp2_1m > 0002da60 T npy_exp2_1mf > 0002e500 T npy_exp2_1ml > 0002daf0 T npy_exp2f > 0002e590 T npy_exp2l > 0002da20 T npy_expf > 0002e4a0 T npy_expl > 0002e160 T npy_expm1 > 0002dc50 T npy_expm1f > 0002e780 T npy_expm1l > 0002d760 T npy_fabs > 0002d830 T npy_fabsf > 0002d6b0 T npy_fabsl > 0002d770 T npy_floor > 0002d840 T npy_floorf > 0002d6c0 T npy_floorl > 0002de30 T npy_fmod > 0002d930 T npy_fmodf > 0002e350 T npy_fmodl > 0002dec0 T npy_hypot > 0002d9c0 T npy_hypotf > 0002e400 T npy_hypotl > 0002e180 T npy_log > 0002e1a0 T npy_log10 > 0002dc90 T npy_log10f > 0002e7e0 T npy_log10l > 0002e010 T npy_log1p > 0002db10 T npy_log1pf > 0002e5c0 T npy_log1pl > 0002df40 T npy_log2 > 0002da40 T npy_log2f > 0002e4d0 T npy_log2l > 0002e030 T npy_logaddexp > 0002ead0 T npy_logaddexp2 > 0002ebd0 T npy_logaddexp2f > 0002e9e0 T npy_logaddexp2l > 0002db30 T npy_logaddexpf > 0002e5f0 T npy_logaddexpl > 0002dc70 T npy_logf > 0002e7b0 T npy_logl > 0002de00 T npy_modf > 0002d900 T npy_modff > 0002e310 T npy_modfl > 0002de90 T npy_pow > 0002d990 T npy_powf > 0002e3b0 T npy_powl > 0002d7f0 T npy_rad2deg > 0002d8c0 T npy_rad2degf > 0002d720 T npy_rad2degl > 0002e230 T npy_rint > 0002dd20 T npy_rintf > 0002e890 T npy_rintl > 0002e2f0 T npy_sin > 0002dde0 T npy_sinf > 0002e290 T npy_sinh > 0002dd80 T npy_sinhf > 0002e920 T npy_sinhl > 0002e9b0 T npy_sinl > 0002e1c0 T npy_sqrt > 0002dcb0 T npy_sqrtf > 0002e810 T npy_sqrtl > 0002e2b0 T npy_tan > 0002dda0 T npy_tanf > 0002e250 T npy_tanh > 0002dd40 T npy_tanhf > 0002e8c0 T npy_tanhl > 0002e950 T npy_tanl > 0002e210 T npy_trunc > 0002dd00 T npy_truncf > 0002e860 T npy_truncl > ./scalarmath.so > 000217d8 T _fini > 000013d0 T _init > 00001b50 T initscalarmath > $[charris at f9 core]$ > And PyArray_API is a global symbol in multiarray.so ./multiarray.so ... 00087f60 D PyArray_API Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Wed Apr 22 02:47:08 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 22 Apr 2009 15:47:08 +0900 Subject: [Numpy-discussion] Linker script, smaller source files and symbol visibility In-Reply-To: References: <49EC6FC9.4090009@ar.media.kyoto-u.ac.jp> <49ED47CE.7050100@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220904212347r235d4a6dwde167eae25b68b39@mail.gmail.com> On Wed, Apr 22, 2009 at 2:24 PM, Charles R Harris wrote: > > > On Mon, Apr 20, 2009 at 11:06 PM, Charles R Harris > wrote: >> >> >> On Mon, Apr 20, 2009 at 10:13 PM, David Cournapeau >> wrote: >>> >>> Charles R Harris wrote: >>> >>> > >>> > Here is a link to the start of the old discussion >>> > >>> > . >>> > You took part in it also. >>> >>> Thanks, I remembered we had the discussion, but could not find it. The >>> different is that I am much more familiar with the technical details and >>> numpy codebase now :) I know how to control exported symbols on most >>> platform which matter (I can't test for AIX or HP-UX unfortunately - but >>> I am perfectly fine with ignoring namespace pollution on those anyway), >>> and I would guess that the only platforms which do not support symbol >>> visibility in one way or the other do not support shared library anyway >>> (some CRAY stuff, for example). >>> >>> Concerning the file size, I don't think anyone would disagree that they >>> are too big, but we don't need to go the "java-way" of one >>> file/class-function either. One first split which I personally like is >>> API/implementation. For example, for multiarray.c, we would only keep >>> the public PyArray_* functions, and put everything else in another file. >>> The other very big file is arrayobject.c, and this one is already mostly >>> organized in independent parts (buffer protocol, number protocol, etc...) >>> >>> Another thing I would like to do it to make the global C API array >>> pointer a 'true' global variable instead of a static one. It took me a >>> while when I was working on the hashing protocol for dtype to understand >>> why it was crashing (the array pointer being static, every file has its >>> own copy, so it was never initialized in the hashdescr.c file). I think >>> a true global variable, hidden through a symbol map, is easier to >>> understand and more reliable. >> >> I made an experiment along those lines a couple of years ago. There were >> compilation problems because the needed include files weren't available. No >> doubt that could be fixed in the build, but at some point I would like to >> have real include files, not the generated variety. Generated include files >> are kind of bogus IMHO, as they don't define an interface but rather reflect >> whatever the function definition happens to be. So as any part of a split I >> would also suggest writing the associated include files. That would also >> make separate compilation possible, which would make it easier to do test >> compilations while doing development. > > The list of visible symbols has grown ;) Yes. Except PyArray_DescrHash which is a mistake on my own, for all the npy_* symbols, there is nothing we can do ATM because they are from a pure C (static) library. That's one of the rationale in the original email :) David From myeates at jpl.nasa.gov Wed Apr 22 11:48:36 2009 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Wed, 22 Apr 2009 08:48:36 -0700 Subject: [Numpy-discussion] performance issue (again) In-Reply-To: <1cd32cbb0904211541k7cc79ee1g6978a752e7872529@mail.gmail.com> References: <49EE416B.3010406@jpl.nasa.gov> <1cd32cbb0904211541k7cc79ee1g6978a752e7872529@mail.gmail.com> Message-ID: <49EF3C54.9040407@jpl.nasa.gov> well, this isn't a perfect solution. polyfit is better because it determines rank based on condition values. Finds the eigenvalues ... etc. But, unless it can vectorized without Python looping, it's too slow for me to use Mathew josef.pktd at gmail.com wrote: > > > > If you remove the mean from x and y (along axis = 1) then can't you > just do something like > > (x*y).sum(1) / (x*x).sum(axis=1) > > > > I think that's what I said 8 days ago. > > Josef > From kwgoodman at gmail.com Wed Apr 22 11:58:23 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 22 Apr 2009 08:58:23 -0700 Subject: [Numpy-discussion] performance issue (again) In-Reply-To: <49EF3C54.9040407@jpl.nasa.gov> References: <49EE416B.3010406@jpl.nasa.gov> <1cd32cbb0904211541k7cc79ee1g6978a752e7872529@mail.gmail.com> <49EF3C54.9040407@jpl.nasa.gov> Message-ID: On Wed, Apr 22, 2009 at 8:48 AM, Mathew Yeates wrote: > well, this isn't a perfect solution. polyfit ?is better because it > determines rank based on condition values. Finds the eigenvalues ... > etc. But, unless it can vectorized without Python looping, it's too slow > for me to use I liked your "sheer genius" comment better. Yeah, maybe use polyfit only for those cases where abs((x*y).sum(1) / (x*x).sum(1)) is "large"? And ignore the slope calculation where (x*x).sum(1) is "small". From josef.pktd at gmail.com Wed Apr 22 12:38:32 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Apr 2009 12:38:32 -0400 Subject: [Numpy-discussion] performance issue (again) In-Reply-To: <49EF3C54.9040407@jpl.nasa.gov> References: <49EE416B.3010406@jpl.nasa.gov> <1cd32cbb0904211541k7cc79ee1g6978a752e7872529@mail.gmail.com> <49EF3C54.9040407@jpl.nasa.gov> Message-ID: <1cd32cbb0904220938y7ecc86a8ta5c74e0a85284184@mail.gmail.com> On Wed, Apr 22, 2009 at 11:48 AM, Mathew Yeates wrote: > well, this isn't a perfect solution. polyfit is better because it > determines rank based on condition values. Finds the eigenvalues ... > etc. But, unless it can vectorized without Python looping, it's too slow > for me to use > rank is a property of the design matrix. In your case the design matrix is a vector of ones and the x vector. So the only case, where you run into problems, is when your three observation of x are the same, then dot(x.T*x) is zero, you can only have one constant. If there is no slope in x then you don't have three different observations to estimate a slope coefficient. Just special case (x*x).sum(1)<1e-8 or something, in this case yestimate = y.mean eigen vectors with one regressor are pretty useless or trivial, same with rank. For higher order polynomials this will become more important, but not for a linear polynomial. Josef -------------- next part -------------- An HTML attachment was scrubbed... URL: From Fabrice.Pardo at Lpn.cnrs.fr Wed Apr 22 13:13:10 2009 From: Fabrice.Pardo at Lpn.cnrs.fr (Fabrice Pardo) Date: Wed, 22 Apr 2009 19:13:10 +0200 Subject: [Numpy-discussion] buggy fortran array reshape ? Message-ID: <49EF5026.7090907@Lpn.cnrs.fr> After reshaping a Fortran array, the new array doesn't share data with original array. I will be glad if someone can explain the strange behaviour of this program. Is it a numpy bug ? #vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv def check_bug(order): a = numpy.ndarray((3,2),order=order,dtype=int) a[0,0] = 1 b = a.reshape((6,)) a[0,0] = 2 print b[0] check_bug('C') # 2, good check_bug('F') # 1, wrong ??? print(numpy.version.version) # 1.2.1 #^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -- FP From josef.pktd at gmail.com Wed Apr 22 13:51:45 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Apr 2009 13:51:45 -0400 Subject: [Numpy-discussion] buggy fortran array reshape ? In-Reply-To: <49EF5026.7090907@Lpn.cnrs.fr> References: <49EF5026.7090907@Lpn.cnrs.fr> Message-ID: <1cd32cbb0904221051r3b99aba7odce9365ad9c900f8@mail.gmail.com> On Wed, Apr 22, 2009 at 1:13 PM, Fabrice Pardo wrote: > > After reshaping a Fortran array, the new array doesn't share data > with original array. > I will be glad if someone can explain the strange behaviour of this > program. Is it a numpy bug ? > > #vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv > def check_bug(order): > ? ?a = numpy.ndarray((3,2),order=order,dtype=int) > ? ?a[0,0] = 1 > ? ?b = a.reshape((6,)) > ? ?a[0,0] = 2 > ? ?print b[0] > > check_bug('C') # 2, good > check_bug('F') # 1, wrong ??? > print(numpy.version.version) # 1.2.1 > #^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ from help: """ Returns: reshaped_array : ndarray This will be a new view object if possible; otherwise, it will be a copy. """ " if possible" and "otherwise" are not very precise I guess reshape tries to return an array that is contiguous, if you do a reshape in the order of the array, i.e. change your line to b = a.reshape((6,), order=order) then the reshaped array is just a view. I still find view vs copy very confusing. Josef From Fabrice.Pardo at Lpn.cnrs.fr Wed Apr 22 14:18:14 2009 From: Fabrice.Pardo at Lpn.cnrs.fr (Fabrice Pardo) Date: Wed, 22 Apr 2009 20:18:14 +0200 Subject: [Numpy-discussion] buggy fortran array reshape ? In-Reply-To: <1cd32cbb0904221051r3b99aba7odce9365ad9c900f8@mail.gmail.com> References: <49EF5026.7090907@Lpn.cnrs.fr> <1cd32cbb0904221051r3b99aba7odce9365ad9c900f8@mail.gmail.com> Message-ID: <49EF5F66.8010609@Lpn.cnrs.fr> josef.pktd at gmail.com wrote: > > from help: > """ > Returns: > reshaped_array : ndarray > This will be a new view object if possible; otherwise, it will be a copy. > """ > > " if possible" and "otherwise" are not very precise > I guess reshape tries to return an array that is contiguous, if you do > a reshape in the order of the array, i.e. > > change your line to > b = a.reshape((6,), order=order) > > then the reshaped array is just a view. > > I still find view vs copy very confusing. > You are right, the documentation doesn't lies. The simplistic current version choice is a copy. In my example, b is 1D, contiguous, with no 'C' or 'F' difference. Then that's possible to do an other choice, making a view. The reshape function is unpredictable and its behaviour is not documented. It cannot be used safely. Another remark against reshape: OWNDATA flag is False, even if b is a copy ! -- FP From pav at iki.fi Wed Apr 22 14:37:47 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 22 Apr 2009 18:37:47 +0000 (UTC) Subject: [Numpy-discussion] buggy fortran array reshape ? References: <49EF5026.7090907@Lpn.cnrs.fr> <1cd32cbb0904221051r3b99aba7odce9365ad9c900f8@mail.gmail.com> <49EF5F66.8010609@Lpn.cnrs.fr> Message-ID: Wed, 22 Apr 2009 20:18:14 +0200, Fabrice Pardo wrote: [clip] > The reshape function is unpredictable and its behaviour is not > documented. > > It cannot be used safely. It is documented and it can be used safely. The manual, however, has currently no section on views that would explain these issues in depth. If you want to ensure no-copy, assign to shape: a.shape = (6,) > Another remark against reshape: > OWNDATA flag is False, even if b is a copy ! Apparently, reshape first copies to a contiguous array and then reshapes. This could be simplified. -- Pauli Virtanen From gael.varoquaux at normalesup.org Wed Apr 22 14:39:34 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 22 Apr 2009 20:39:34 +0200 Subject: [Numpy-discussion] buggy fortran array reshape ? In-Reply-To: <49EF5F66.8010609@Lpn.cnrs.fr> References: <49EF5026.7090907@Lpn.cnrs.fr> <1cd32cbb0904221051r3b99aba7odce9365ad9c900f8@mail.gmail.com> <49EF5F66.8010609@Lpn.cnrs.fr> Message-ID: <20090422183934.GA32103@phare.normalesup.org> On Wed, Apr 22, 2009 at 08:18:14PM +0200, Fabrice Pardo wrote: > It cannot be used safely. use: b = a.view() b.shape = (2, 3) This will return a view, or raise an exception. Ga?l From pav at iki.fi Wed Apr 22 14:39:16 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 22 Apr 2009 18:39:16 +0000 (UTC) Subject: [Numpy-discussion] buggy fortran array reshape ? References: <49EF5026.7090907@Lpn.cnrs.fr> <1cd32cbb0904221051r3b99aba7odce9365ad9c900f8@mail.gmail.com> Message-ID: Wed, 22 Apr 2009 13:51:45 -0400, josef.pktd wrote: [clip] > change your line to > b = a.reshape((6,), order=order) > > then the reshaped array is just a view. This has the effect that the unravelling is done in Fortran order (when order='F') rather than C-order, which can be confusing at times. -- Pauli Virtanen From josef.pktd at gmail.com Wed Apr 22 15:12:20 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Apr 2009 15:12:20 -0400 Subject: [Numpy-discussion] buggy fortran array reshape ? In-Reply-To: References: <49EF5026.7090907@Lpn.cnrs.fr> <1cd32cbb0904221051r3b99aba7odce9365ad9c900f8@mail.gmail.com> <49EF5F66.8010609@Lpn.cnrs.fr> Message-ID: <1cd32cbb0904221212l3044abc3y721b0ed72851a930@mail.gmail.com> On Wed, Apr 22, 2009 at 2:37 PM, Pauli Virtanen wrote: > Wed, 22 Apr 2009 20:18:14 +0200, Fabrice Pardo wrote: > [clip] >> The reshape function is unpredictable and its behaviour is not >> documented. >> >> It cannot be used safely. > > It is documented and it can be used safely. The manual, however, has > currently no section on views that would explain these issues in depth. > > If you want to ensure no-copy, assign to shape: > > a.shape = (6,) > >> Another remark against reshape: >> OWNDATA flag is False, even if b is a copy ! > > Apparently, reshape first copies to a contiguous array and then reshapes. > This could be simplified. > Is the difference between assigning to the attribute and using the method call explained somewhere? I had recently the puzzling case, where I wanted to create a structured array, and tried x.view(dtype=...) x.astype(..) x.dtype = ... I don't remember exactly, but view and astype didn't create the structured array that I wanted, while the assignment x.dtype = ... worked. >This has the effect that the unravelling is done in Fortran order (when >order='F') rather than C-order, which can be confusing at times. If he intentionally starts out in Fortran order, he might have a reason to stick to it. In stats, we are still focused by default on axis=0, and I usually think in terms of columns of random variables. But using a lot of transpose and newaxis, I never know what the memory layout is unless I check the flags, and I'm starting to realize that this requires more attention with numpy. Josef From pav at iki.fi Wed Apr 22 15:23:16 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 22 Apr 2009 19:23:16 +0000 (UTC) Subject: [Numpy-discussion] buggy fortran array reshape ? References: <49EF5026.7090907@Lpn.cnrs.fr> <1cd32cbb0904221051r3b99aba7odce9365ad9c900f8@mail.gmail.com> <49EF5F66.8010609@Lpn.cnrs.fr> <1cd32cbb0904221212l3044abc3y721b0ed72851a930@mail.gmail.com> Message-ID: Wed, 22 Apr 2009 15:12:20 -0400, josef.pktd wrote: > On Wed, Apr 22, 2009 at 2:37 PM, Pauli Virtanen wrote: [clip] >> If you want to ensure no-copy, assign to shape: >> >> a.shape = (6,) [clip] > Is the difference between assigning to the attribute and using the > method call explained > somewhere? No. But it certainly should be. The first place to fix is the attribute docstring which doesn't even mention it can be assigned to: http://docs.scipy.org/numpy/docs/numpy.ndarray.shape/ Similar review should be done to all attribute docstrings. Second, a separate more detailed discussion about memory layouts, views, etc. should be written. -- Pauli Virtanen From gokhansever at gmail.com Wed Apr 22 17:21:05 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_SEVER?=) Date: Wed, 22 Apr 2009 16:21:05 -0500 Subject: [Numpy-discussion] Masking an array with another array Message-ID: <49d6b3500904221421x2c32b8a8q136e48d0ce153dd6@mail.gmail.com> Hello, Could you please give me some hints about how to mask an array using another arrays like in the following example. In [14]: a = arange(5) In [15]: a Out[15]: array([0, 1, 2, 3, 4]) and my secondary array is "b" In [16]: b = array([2,3]) What I want to do is to mask a with b values and get an array of: array([False, False, True, True, False], dtype=bool) That is just an manually created array. I still don't know how to do this programmatically in Pythonic fashion or numpy's masked array functions. Thank you. G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Wed Apr 22 17:49:51 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 22 Apr 2009 17:49:51 -0400 Subject: [Numpy-discussion] Masking an array with another array In-Reply-To: <49d6b3500904221421x2c32b8a8q136e48d0ce153dd6@mail.gmail.com> References: <49d6b3500904221421x2c32b8a8q136e48d0ce153dd6@mail.gmail.com> Message-ID: On Apr 22, 2009, at 5:21 PM, G?khan SEVER wrote: > Hello, > > Could you please give me some hints about how to mask an array using > another arrays like in the following example. What about that ? numpy.logical_or.reduce([a==i for i in b]) From gokhansever at gmail.com Wed Apr 22 20:18:05 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_SEVER?=) Date: Wed, 22 Apr 2009 19:18:05 -0500 Subject: [Numpy-discussion] Masking an array with another array In-Reply-To: References: <49d6b3500904221421x2c32b8a8q136e48d0ce153dd6@mail.gmail.com> Message-ID: <49d6b3500904221718y443e9e10s8a7495bcebafbdec@mail.gmail.com> Yes Pierre, I like this one line of elegances in Python a lot. I was thinking that the answer lies in somewhere in masked array operations, but I proved wrong. Thanks for your input on this small riddle. Here is another way of doing that. (That's what I thought of initially and what Matthias Michler responded on matplotlib mailing list.) mask = zeros(len(a), dtype=bool) for index in xrange(len(a)): # run through array a if a[index] in b: mask[index] = True Ending with a quote about Pythonicness :) "...that something is Pythonic when it has a sense of quality, simplicity, clarity and elegance about it." G?khan On Wed, Apr 22, 2009 at 4:49 PM, Pierre GM wrote: > > On Apr 22, 2009, at 5:21 PM, G?khan SEVER wrote: > > > Hello, > > > > Could you please give me some hints about how to mask an array using > > another arrays like in the following example. > > What about that ? > numpy.logical_or.reduce([a==i for i in b]) > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Apr 22 21:03:57 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Apr 2009 21:03:57 -0400 Subject: [Numpy-discussion] Masking an array with another array In-Reply-To: <49d6b3500904221718y443e9e10s8a7495bcebafbdec@mail.gmail.com> References: <49d6b3500904221421x2c32b8a8q136e48d0ce153dd6@mail.gmail.com> <49d6b3500904221718y443e9e10s8a7495bcebafbdec@mail.gmail.com> Message-ID: <1cd32cbb0904221803h18c7e97k9f367ec924910eba@mail.gmail.com> On Wed, Apr 22, 2009 at 8:18 PM, G?khan SEVER wrote: > Yes Pierre, > > I like this one line of elegances in Python a lot. I was thinking that the > answer lies in somewhere in masked array operations, but I proved wrong. > > Thanks for your input on this small riddle. > > Here is another way of doing that. (That's what I thought of initially and > what Matthias Michler responded on matplotlib mailing list.) > > mask = zeros(len(a), dtype=bool) > for index in xrange(len(a)): ? ? ? ?# run through array a > ? ?if a[index] in b: > ? ? ? ?mask[index] = True > > > Ending with a quote about Pythonicness :) > > "...that something is Pythonic when it has a sense of quality, simplicity, > clarity and elegance about it." > > > G?khan > > > On Wed, Apr 22, 2009 at 4:49 PM, Pierre GM wrote: >> >> On Apr 22, 2009, at 5:21 PM, G?khan SEVER wrote: >> >> > Hello, >> > >> > Could you please give me some hints about how to mask an array using >> > another arrays like in the following example. >> >> What about that ? >> numpy.logical_or.reduce([a==i for i in b]) >> >> I prefer broad casting to list comprehension in numpy: >>> a = np.arange(5) >>> b = np.array([2,3]) >>> (a[:,np.newaxis]==b).any(1) array([False, False, True, True, False], dtype=bool) Josef From pgmdevlist at gmail.com Wed Apr 22 22:45:08 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 22 Apr 2009 22:45:08 -0400 Subject: [Numpy-discussion] Masking an array with another array In-Reply-To: <1cd32cbb0904221803h18c7e97k9f367ec924910eba@mail.gmail.com> References: <49d6b3500904221421x2c32b8a8q136e48d0ce153dd6@mail.gmail.com> <49d6b3500904221718y443e9e10s8a7495bcebafbdec@mail.gmail.com> <1cd32cbb0904221803h18c7e97k9f367ec924910eba@mail.gmail.com> Message-ID: On Apr 22, 2009, at 9:03 PM, josef.pktd at gmail.com wrote: > > I prefer broad casting to list comprehension in numpy: Pretty neat! I still dont have the broadcasting reflex. Now, any idea which one is more efficient in terms of speed? in terms of temporaries? From josef.pktd at gmail.com Wed Apr 22 23:43:28 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Apr 2009 23:43:28 -0400 Subject: [Numpy-discussion] Masking an array with another array In-Reply-To: References: <49d6b3500904221421x2c32b8a8q136e48d0ce153dd6@mail.gmail.com> <49d6b3500904221718y443e9e10s8a7495bcebafbdec@mail.gmail.com> <1cd32cbb0904221803h18c7e97k9f367ec924910eba@mail.gmail.com> Message-ID: <1cd32cbb0904222043w704c1ce2g72741e33971c967@mail.gmail.com> On Wed, Apr 22, 2009 at 10:45 PM, Pierre GM wrote: > > On Apr 22, 2009, at 9:03 PM, josef.pktd at gmail.com wrote: >> >> I prefer broad casting to list comprehension in numpy: > > Pretty neat! I still dont have the broadcasting reflex. Now, any idea > which one is more efficient in terms of speed? in terms of temporaries? > I used similar broadcasting for working with categorical data series and for creating dummy variables for regression. So I played already for some time with this. In this case, I would except that the memory consumption is essentially the same, you have a list of arrays and I have a 2d array, unless numpy needs an additional conversion to array in np.logical_or.reduce, which seems plausible but I don't know. The main point that Sturla convinced me in the discussion on kendalltau is that if b is large, 500 or 1000, then building the full intermediate boolean array is killing both memory and speed performance, compared to a python for loop, and very bad compared to a cython loop. In this example my version is at least twice as fast for len(b) = 4, your version does not scale very well at all to larger b, your takes 7 times as long as mine for len(b) = 400, which, I guess would mean that you have an extra copying step I added the for loop and it is always the fastest, even more for short b. I hope it's correct, I never used a inplace logical operator. Josef from time import time as time_ a = np.array(range(10)*1000) blen = 10#100 b = np.array([2,3,5,8]*blen) print "shape b", b.shape t = time_() for _ in range(100): (a[:,np.newaxis]==b).any(1) print time_() - t t = time_() for _ in range(100): np.logical_or.reduce([a==i for i in b]) print time_() - t t = time_() for _ in range(100): z = a == b[0] for ii in range(1,len(b)): z |= (a == b[ii]) print time_() - t #shape b (80,) #0.110000133514 #0.266000032425 #shape b (80,) #0.827999830246 #5.2650001049 #shape b (400,) #4.60899996758 #28.4370000362 #shape b (400,) #3.89100003242 #27.5 #shape b (400,) #3.89099979401 #27.3289999962 #3.51599979401 #for loop #shape b (40,) #0.453999996185 #2.54600000381 #0.359999895096 #for loop #shape b (4,) #0.108999967575 #0.28200006485 #0.0309998989105 #for loop From gael.varoquaux at normalesup.org Thu Apr 23 01:16:15 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 23 Apr 2009 07:16:15 +0200 Subject: [Numpy-discussion] Masking an array with another array In-Reply-To: <49d6b3500904221421x2c32b8a8q136e48d0ce153dd6@mail.gmail.com> References: <49d6b3500904221421x2c32b8a8q136e48d0ce153dd6@mail.gmail.com> Message-ID: <20090423051615.GB25215@phare.normalesup.org> On Wed, Apr 22, 2009 at 04:21:05PM -0500, G?khan SEVER wrote: > Could you please give me some hints about how to mask an array using > another arrays like in the following example. > In [14]: a = arange(5) > In [15]: a > Out[15]: array([0, 1, 2, 3, 4]) > and my secondary array is "b" > In [16]: b = array([2,3]) > What I want to do is to mask a with b values and get an array of: > array([False, False, True, True,? False], dtype=bool) This is an operation on 'sets': you are testing if members of a are 'in' b. Generally, set operations on arrays can be found in numpy.lib.arraysetops. I believe what you are interested in is setmember1d. HTH, Ga?l From gokhansever at gmail.com Thu Apr 23 01:24:35 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_SEVER?=) Date: Thu, 23 Apr 2009 00:24:35 -0500 Subject: [Numpy-discussion] Masking an array with another array In-Reply-To: <20090423051615.GB25215@phare.normalesup.org> References: <49d6b3500904221421x2c32b8a8q136e48d0ce153dd6@mail.gmail.com> <20090423051615.GB25215@phare.normalesup.org> Message-ID: <49d6b3500904222224u1451e693v98b812fcbad856dc@mail.gmail.com> Ahaa,, Thanks Ga?l. That method is more elegance than the previous inputs, and the simplest of all. Although one line of "import this" says: There should be one-- and preferably only one --obvious way to do it. I always find many different ways of implementing ideas in Python world. G?khan On Thu, Apr 23, 2009 at 12:16 AM, Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > On Wed, Apr 22, 2009 at 04:21:05PM -0500, G?khan SEVER wrote: > > Could you please give me some hints about how to mask an array using > > another arrays like in the following example. > > > In [14]: a = arange(5) > > > In [15]: a > > Out[15]: array([0, 1, 2, 3, 4]) > > > and my secondary array is "b" > > > In [16]: b = array([2,3]) > > > What I want to do is to mask a with b values and get an array of: > > > array([False, False, True, True, False], dtype=bool) > > This is an operation on 'sets': you are testing if members of a are 'in' > b. Generally, set operations on arrays can be found in > numpy.lib.arraysetops. I believe what you are interested in is > setmember1d. > > HTH, > > Ga?l > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nwagner at iam.uni-stuttgart.de Thu Apr 23 03:23:18 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 23 Apr 2009 09:23:18 +0200 Subject: [Numpy-discussion] String manipulation Message-ID: Hi all, How can I obtain the position of the minus sign within the following string ? >>> liste[1] '1.5-te' Nils From faltet at pytables.org Thu Apr 23 03:35:23 2009 From: faltet at pytables.org (Francesc Alted) Date: Thu, 23 Apr 2009 09:35:23 +0200 Subject: [Numpy-discussion] String manipulation In-Reply-To: References: Message-ID: <200904230935.23913.faltet@pytables.org> A Thursday 23 April 2009, Nils Wagner escrigu?: > Hi all, > > How can I obtain the position of the minus sign within the > following string ? > > >>> liste[1] > > '1.5-te' That's easy by using basic weaponery in Python: In [8]: liste[1].find('-') Out[8]: 3 In [9]: liste[1][3] Out[9]: '-' -- Francesc Alted "One would expect people to feel threatened by the 'giant brains or machines that think'. In fact, the frightening computer becomes less frightening if it is used only to simulate a familiar noncomputer." -- Edsger W. Dykstra "On the cruelty of really teaching computer science" From josef.pktd at gmail.com Thu Apr 23 09:08:11 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 23 Apr 2009 09:08:11 -0400 Subject: [Numpy-discussion] Masking an array with another array In-Reply-To: <49d6b3500904222224u1451e693v98b812fcbad856dc@mail.gmail.com> References: <49d6b3500904221421x2c32b8a8q136e48d0ce153dd6@mail.gmail.com> <20090423051615.GB25215@phare.normalesup.org> <49d6b3500904222224u1451e693v98b812fcbad856dc@mail.gmail.com> Message-ID: <1cd32cbb0904230608n6a172bcs8716a119bb98a9c2@mail.gmail.com> On Thu, Apr 23, 2009 at 1:24 AM, G?khan SEVER wrote: > Ahaa,, > > Thanks Ga?l. That method is more elegance than the previous inputs, and the > simplest of all. > > Although one line of "import this" says: > > There should be one-- and preferably only one --obvious way to do it. > > I always find many different ways of implementing ideas in Python world. > > G?khan > > > On Thu, Apr 23, 2009 at 12:16 AM, Gael Varoquaux > wrote: >> >> On Wed, Apr 22, 2009 at 04:21:05PM -0500, G?khan SEVER wrote: >> > ? ?Could you please give me some hints about how to mask an array using >> > ? ?another arrays like in the following example. >> >> > ? ?In [14]: a = arange(5) >> >> > ? ?In [15]: a >> > ? ?Out[15]: array([0, 1, 2, 3, 4]) >> >> > ? ?and my secondary array is "b" >> >> > ? ?In [16]: b = array([2,3]) >> >> > ? ?What I want to do is to mask a with b values and get an array of: >> >> > ? ?array([False, False, True, True,? False], dtype=bool) >> >> This is an operation on 'sets': you are testing if members of a are 'in' >> b. Generally, set operations on arrays can be found in >> numpy.lib.arraysetops. I believe what you are interested in is >> setmember1d. >> >> HTH, >> >> Ga?l setmember1d is very fast compared to the other solutions for large b. However, setmember1d requires that both arrays only have unique elements. So it doesn't work if, for example, your first array is a data vector with member ship in different groups (therefore not only uniques), and the second array is the sub group that you want to check. Josef >>> a = np.array([7, 5, 1, 3, 3, 0, 0, 8, 8, 2]) >>> b = np.array([0, 1]) >>> np.setmember1d(a,b) array([False, False, True, True, False, True, True, True, False, False], dtype=bool) >>> (a[:,np.newaxis]==b).any(1) array([False, False, True, False, False, True, True, False, False, False], dtype=bool) From fccoelho at gmail.com Thu Apr 23 09:27:49 2009 From: fccoelho at gmail.com (Flavio Coelho) Date: Thu, 23 Apr 2009 14:27:49 +0100 Subject: [Numpy-discussion] bug in stats.randint Message-ID: Hi, I stumbled upon something I think is a bug in scipy: In [4]: stats.randint(1.,15.).ppf([.1,.2,.3,.4,.5]) Out[4]: array([ 2., 3., 5., 6., 7.]) When you pass float arguments to stats.randint and then call the ppf method, you get an array of floats, which clearly wrong. The rvs method doesn't display this bug so I think is a matter of poor type checking in the ppf implementation... -- --- Fl?vio Code?o Coelho -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Thu Apr 23 09:38:21 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 23 Apr 2009 22:38:21 +0900 Subject: [Numpy-discussion] Porting strategy for py3k Message-ID: <5b8d13220904230638i553d0e7bx2f3d93572861940c@mail.gmail.com> Hi, I looked more in detail on what would be needed to port numpy to py3k. In particular, I was interested at the possible strategies to keep one single codebase for both python 2.x and python 3.x. The first step is to remove all py3k warnings reported by python 2.6. A couple of recurrent problems - reduce is removed in py3k - print is removed - some exceptions constructions are not allowed I think a first step would be to add a compatibility module to remove any use of print/reduce in the numpy codebase (should not be too difficult). For reduce, we could import functools from python code (functools is not available in python 2.4). Unless someone has an objection, I will import functools code, make it compile for python 2.4 and above, and remove all builtin reduce calls in numpy. Another problem is related to nose: there is an experimental branch for nose which supports py3k, but nose <= 1.0 will not support py3k. The nose author intend to support py3k in a version > 1.0, at which point he will only support python 2.6 and above. I don't know what to do with this (include our own nose in numpy for version < 2.6 - giving up support for python < 2.6 does not sound like a realistic option in the next few years). cheers, David From josef.pktd at gmail.com Thu Apr 23 09:56:45 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 23 Apr 2009 09:56:45 -0400 Subject: [Numpy-discussion] bug in stats.randint In-Reply-To: References: Message-ID: <1cd32cbb0904230656i7127e91ek80cf0574663a7b85@mail.gmail.com> On Thu, Apr 23, 2009 at 9:27 AM, Flavio Coelho wrote: > > Hi, > > I stumbled upon something I think is a bug in scipy: > > In [4]: stats.randint(1.,15.).ppf([.1, > .2,.3,.4,.5]) > Out[4]: array([ 2.,? 3.,? 5.,? 6.,? 7.]) > > When you pass float arguments to stats.randint and then call the ppf method, > you get an array of floats, which clearly wrong. The rvs method doesn't > display this bug so I think is a matter of poor type checking in the ppf > implementation... > I switched to using floats intentionally, to have correct handling of inf and nans. and the argument checking is generic for all discrete distributions and not special cased. Nans are converted to zero when casting to integers, which is wrong and very confusing. inf raise an exception. I prefer correct numbers to correct types. see examples below If you don't have inf and nans you can cast them to int yourself. Josef >>> aint = np.zeros(5,dtype=int) >>> aint[0]= np.nan >>> aint array([0, 0, 0, 0, 0]) >>> aint[1]= np.inf Traceback (most recent call last): File "", line 1, in OverflowError: cannot convert float infinity to long >>> from scipy import stats >>> stats.poisson.ppf(1,1) inf >>> stats.poisson.ppf(2,1) nan >>> stats.poisson.ppf(1,1).astype(int) -2147483648 >>> aint[2] = stats.poisson.ppf(1,1) Traceback (most recent call last): File "", line 1, in OverflowError: cannot convert float infinity to long From josef.pktd at gmail.com Thu Apr 23 09:59:39 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 23 Apr 2009 09:59:39 -0400 Subject: [Numpy-discussion] NaN as dictionary key? In-Reply-To: <6c476c8a0904200742j331c4d32k4b877995e001a9ed@mail.gmail.com> References: <6c476c8a0904200742j331c4d32k4b877995e001a9ed@mail.gmail.com> Message-ID: <1cd32cbb0904230659u4105ef8dlbed7d3c53387083e@mail.gmail.com> 2009/4/20 Wes McKinney : > I assume that, because NaN != NaN, even though both have the same hash value > (hash(NaN) == -32768), that Python treats any NaN double as a distinct key > in a dictionary. > > In [76]: a = np.repeat(nan, 10) > > In [77]: d = {} > > In [78]: for i, v in enumerate(a): > ?? ....:???? d[v] = i > ?? ....: > ?? ....: > > In [79]: d > Out[79]: > {nan: 0, > ?nan: 1, > ?nan: 6, > ?nan: 4, > ?nan: 3, > ?nan: 9, > ?nan: 7, > ?nan: 2, > ?nan: 8, > ?nan: 5} > > I'm not sure if this ever worked in a past version of NumPy, however, I have > code which does a "group by value" and occasionally in the real world those > values are NaN. Any ideas or a way around this problem? For non hashable keys, I convert them to string, eg with repr or str or some other string representation for floating point. I use it for example to feed it to unique1d. Josef >>> a array([ NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN]) >>> np.unique1d(a) array([ NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN]) using type string is not good with nan (automatic conversion of nans in casting) >>> np.unique1d(a.astype(str)) array(['1'], dtype='|S1') >>> a.astype(str) array(['1', '1', '1', '1', '1', '1', '1', '1', '1', '1'], dtype='|S1') >>> np.unique1d([repr(ii) for ii in a]) array(['nan'], dtype='|S3') but nans don't round trip, is this intended (at least not on windows >>> np.unique1d(np.arange(10).astype(str)).astype(float) array([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.]) >>> np.all(np.array([repr(ii) for ii in np.pi*np.arange(10)]).astype(float) == np.pi*np.arange(10)) True >>> np.unique1d([repr(ii) for ii in a]).astype(float) Traceback (most recent call last): File "", line 1, in ValueError: invalid literal for float(): nan From pav at iki.fi Thu Apr 23 10:20:19 2009 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 23 Apr 2009 14:20:19 +0000 (UTC) Subject: [Numpy-discussion] Porting strategy for py3k References: <5b8d13220904230638i553d0e7bx2f3d93572861940c@mail.gmail.com> Message-ID: Thu, 23 Apr 2009 22:38:21 +0900, David Cournapeau kirjoitti: [clip] > I looked more in detail on what would be needed to port numpy to > py3k. In particular, I was interested at the possible strategies to keep > one single codebase for both python 2.x and python 3.x. The first step > is to remove all py3k warnings reported by python 2.6. A couple of > recurrent problems > - reduce is removed in py3k > - print is removed Print is not removed, just changed to a function. So, print("foo") is valid code in 2.X and 3.0. OTOH, I don't think there are any valid use cases for "print" in Numpy code anyway (maybe apart from tests). > - some exceptions constructions are not allowed Some more: - xrange is no more - dict.iteritems / dict.iterkeys / etc are no more etc. > Another problem is related to nose: there is an experimental branch for > nose which supports py3k, but nose <= 1.0 will not support py3k. The > nose author intend to support py3k in a version > 1.0, at which point he > will only support python 2.6 and above. I don't know what to do with > this (include our own nose in numpy for version < 2.6 - giving up > support for python < 2.6 does not sound like a realistic option in the > next few years). Can't we just assume Python 2.6 and 3.0 users have the appropriate versions of Nose installed? Or is the old Nose version going to disappear? This would limit our responsibility to just ensuring that our Nose extensions work with all relevant Nose versions. -- Pauli Virtanen From bsouthey at gmail.com Thu Apr 23 10:21:03 2009 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 23 Apr 2009 09:21:03 -0500 Subject: [Numpy-discussion] Porting strategy for py3k In-Reply-To: <5b8d13220904230638i553d0e7bx2f3d93572861940c@mail.gmail.com> References: <5b8d13220904230638i553d0e7bx2f3d93572861940c@mail.gmail.com> Message-ID: <49F0794F.4060000@gmail.com> David Cournapeau wrote: > Hi, > > I looked more in detail on what would be needed to port numpy to > py3k. In particular, I was interested at the possible strategies to > keep one single codebase for both python 2.x and python 3.x. The first > step is to remove all py3k warnings reported by python 2.6. A couple > of recurrent problems > - reduce is removed in py3k > - print is removed > - some exceptions constructions are not allowed > > I think a first step would be to add a compatibility module to remove > any use of print/reduce in the numpy codebase (should not be too > difficult). For reduce, we could import functools from python code > (functools is not available in python 2.4). Unless someone has an > objection, I will import functools code, make it compile for python > 2.4 and above, and remove all builtin reduce calls in numpy. > > Another problem is related to nose: there is an experimental branch > for nose which supports py3k, but nose <= 1.0 will not support py3k. > The nose author intend to support py3k in a version > 1.0, at which > point he will only support python 2.6 and above. I don't know what to > do with this (include our own nose in numpy for version < 2.6 - giving > up support for python < 2.6 does not sound like a realistic option in > the next few years). > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Hi, It is good news that nose has some Py3K support. Hopefully all NumPy tests will work with all versions of nose as Python 2.6 is meant to be backwards compatible. I agree that a single code base should be used if possible however there are a lot of C code changes required as well as Python code changes: http://www.scipy.org/Python3k http://jarrodmillman.blogspot.com/2009/01/when-will-numpy-and-scipy-migrate-to.html Are the C code changes independent of the warnings provided by Python 2.6 -3 option and eventually the 2to3 tool? Bruce From neilcrighton at gmail.com Thu Apr 23 10:23:57 2009 From: neilcrighton at gmail.com (Neil Crighton) Date: Thu, 23 Apr 2009 14:23:57 +0000 (UTC) Subject: [Numpy-discussion] Masking an array with another array References: <49d6b3500904221421x2c32b8a8q136e48d0ce153dd6@mail.gmail.com> <20090423051615.GB25215@phare.normalesup.org> <49d6b3500904222224u1451e693v98b812fcbad856dc@mail.gmail.com> <1cd32cbb0904230608n6a172bcs8716a119bb98a9c2@mail.gmail.com> Message-ID: gmail.com> writes: > setmember1d is very fast compared to the other solutions for large b. > > However, setmember1d requires that both arrays only have unique elements. > > So it doesn't work if, for example, your first array is a data vector > with member ship in different groups (therefore not only uniques), and > the second array is the sub group that you want to check. > Note there's a patch waiting to be reviewed that adds another version of setmember_1d for non-unique inputs. http://projects.scipy.org/numpy/ticket/1036 Neil From josef.pktd at gmail.com Thu Apr 23 10:28:21 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 23 Apr 2009 10:28:21 -0400 Subject: [Numpy-discussion] bug in stats.randint In-Reply-To: <1cd32cbb0904230656i7127e91ek80cf0574663a7b85@mail.gmail.com> References: <1cd32cbb0904230656i7127e91ek80cf0574663a7b85@mail.gmail.com> Message-ID: <1cd32cbb0904230728p5ae5ba8bp166d32d12d2809a3@mail.gmail.com> On Thu, Apr 23, 2009 at 9:56 AM, wrote: > On Thu, Apr 23, 2009 at 9:27 AM, Flavio Coelho wrote: >> >> Hi, >> >> I stumbled upon something I think is a bug in scipy: >> >> In [4]: stats.randint(1.,15.).ppf([.1, >> .2,.3,.4,.5]) >> Out[4]: array([ 2.,? 3.,? 5.,? 6.,? 7.]) >> >> When you pass float arguments to stats.randint and then call the ppf method, >> you get an array of floats, which clearly wrong. The rvs method doesn't >> display this bug so I think is a matter of poor type checking in the ppf >> implementation... >> > > I switched to using floats intentionally, to have correct handling of > inf and nans. and the argument checking is generic for all discrete > distributions and not special cased. Nans are converted to zero when > casting to integers, which is wrong and very confusing. inf raise an > exception. I prefer correct numbers to correct types. see examples > below > > If you don't have inf and nans you can cast them to int yourself. > > Josef > >>>> aint = np.zeros(5,dtype=int) >>>> aint[0]= np.nan >>>> aint > array([0, 0, 0, 0, 0]) >>>> aint[1]= np.inf > Traceback (most recent call last): > ?File "", line 1, in > OverflowError: cannot convert float infinity to long > >>>> from scipy import stats >>>> stats.poisson.ppf(1,1) > inf >>>> stats.poisson.ppf(2,1) > nan > >>>> stats.poisson.ppf(1,1).astype(int) > -2147483648 >>>> aint[2] = stats.poisson.ppf(1,1) > Traceback (most recent call last): > ?File "", line 1, in > OverflowError: cannot convert float infinity to long > There are still some corner cases in the distributions that might not or do not work correctly. So please report anything that looks suspicious. I wasn't sure what the limiting behavior of ppf is, but in this example ppf still looks good, but cdf is restricted to long integers. Josef >>> stats.poisson.ppf(1-1e-15,1e10) 10000794155.0 >>> stats.poisson.ppf(1-1e-15,1e10).astype(int) -2147483648 this also works: >>> stats.poisson.ppf(0.5,1e8).astype(int) 100000000 >>> stats.poisson.cdf(100000000,1e8) 0.50002670277569705 but this doesn't >>> stats.poisson.ppf(0.5,1e10) 9999999996.0 >>> stats.poisson.cdf(9999999996,1e10) nan From bsouthey at gmail.com Thu Apr 23 10:38:10 2009 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 23 Apr 2009 09:38:10 -0500 Subject: [Numpy-discussion] NaN as dictionary key? In-Reply-To: <1cd32cbb0904230659u4105ef8dlbed7d3c53387083e@mail.gmail.com> References: <6c476c8a0904200742j331c4d32k4b877995e001a9ed@mail.gmail.com> <1cd32cbb0904230659u4105ef8dlbed7d3c53387083e@mail.gmail.com> Message-ID: <49F07D52.4060701@gmail.com> josef.pktd at gmail.com wrote: > 2009/4/20 Wes McKinney : > >> I assume that, because NaN != NaN, even though both have the same hash value >> (hash(NaN) == -32768), that Python treats any NaN double as a distinct key >> in a dictionary. >> >> In [76]: a = np.repeat(nan, 10) >> >> In [77]: d = {} >> >> In [78]: for i, v in enumerate(a): >> ....: d[v] = i >> ....: >> ....: >> >> In [79]: d >> Out[79]: >> {nan: 0, >> nan: 1, >> nan: 6, >> nan: 4, >> nan: 3, >> nan: 9, >> nan: 7, >> nan: 2, >> nan: 8, >> nan: 5} >> >> I'm not sure if this ever worked in a past version of NumPy, however, I have >> code which does a "group by value" and occasionally in the real world those >> values are NaN. Any ideas or a way around this problem? >> > > For non hashable keys, I convert them to string, eg with repr or str > or some other string representation for floating point. > > I use it for example to feed it to unique1d. > > Josef > > > >>>> a >>>> > array([ NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN]) > >>>> np.unique1d(a) >>>> > array([ NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN]) > > using type string is not good with nan (automatic conversion of nans in casting) > >>>> np.unique1d(a.astype(str)) >>>> > array(['1'], > dtype='|S1') > >>>> a.astype(str) >>>> > array(['1', '1', '1', '1', '1', '1', '1', '1', '1', '1'], > dtype='|S1') > > >>>> np.unique1d([repr(ii) for ii in a]) >>>> > array(['nan'], > dtype='|S3') > > > but nans don't round trip, is this intended (at least not on windows > > >>>> np.unique1d(np.arange(10).astype(str)).astype(float) >>>> > array([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.]) > >>>> np.all(np.array([repr(ii) for ii in np.pi*np.arange(10)]).astype(float) == np.pi*np.arange(10)) >>>> > True > > >>>> np.unique1d([repr(ii) for ii in a]).astype(float) >>>> > Traceback (most recent call last): > File "", line 1, in > ValueError: invalid literal for float(): nan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Hi, Perhaps you want something use isfinite and friends such as: import numpy as np a = np.array([1,2,3, np.inf, np.nan, 10]) e = {} for i, v in enumerate(a): if np.isfinite(v): e[v] = i else: e[repr(v)]=i You probably should use isfinite outside of the loop. If you really do not care about NaN and infinity, then you could use a masked array where NaN and infinity are masked. Bruce From cournape at gmail.com Thu Apr 23 10:52:24 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 23 Apr 2009 23:52:24 +0900 Subject: [Numpy-discussion] Porting strategy for py3k In-Reply-To: References: <5b8d13220904230638i553d0e7bx2f3d93572861940c@mail.gmail.com> Message-ID: <5b8d13220904230752u3c8dd7fyb14f937d358472fd@mail.gmail.com> On Thu, Apr 23, 2009 at 11:20 PM, Pauli Virtanen wrote: > Thu, 23 Apr 2009 22:38:21 +0900, David Cournapeau kirjoitti: > [clip] >> ? ? I looked more in detail on what would be needed to port numpy to >> py3k. In particular, I was interested at the possible strategies to keep >> one single codebase for both python 2.x and python 3.x. The first step >> is to remove all py3k warnings reported by python 2.6. A couple of >> recurrent problems >> ? ? - reduce is removed in py3k >> ? ? - print is removed > > Print is not removed, just changed to a function. So, > > ? ? ? ?print("foo") Yes, as reduce, they are still available, but not as builtins anymore. But replacing print is not as easy as reduce. Things like print "yoyo", a do not work, for example. > is valid code in 2.X and 3.0. OTOH, I don't think there are any valid use > cases for "print" in Numpy code anyway (maybe apart from tests). Yes, maybe. I will look at removing them completely. >> Another problem is related to nose: there is an experimental branch for >> nose which supports py3k, but nose <= 1.0 will not support py3k. The >> nose author intend to support py3k in a version > 1.0, at which point he >> will only support python 2.6 and above. I don't know what to do with >> this (include our own nose in numpy for version < 2.6 - giving up >> support for python < 2.6 does not sound like a realistic option in the >> next few years). > > Can't we just assume Python 2.6 and 3.0 users have the appropriate > versions of Nose installed? Or is the old Nose version going to disappear? The problem is that nose 1.0 and py3k nose won't be compatible, and I don't really know how much different they will be. It is still not clear whether we will have to change our unit tests, for example. David From cournape at gmail.com Thu Apr 23 11:01:46 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 24 Apr 2009 00:01:46 +0900 Subject: [Numpy-discussion] Porting strategy for py3k In-Reply-To: <49F0794F.4060000@gmail.com> References: <5b8d13220904230638i553d0e7bx2f3d93572861940c@mail.gmail.com> <49F0794F.4060000@gmail.com> Message-ID: <5b8d13220904230801y517a89d6nc06064bb09d4416f@mail.gmail.com> On Thu, Apr 23, 2009 at 11:21 PM, Bruce Southey wrote: > I agree that a single code base should be used if possible however there > are a lot of C code changes required as well as Python code changes: > http://www.scipy.org/Python3k > http://jarrodmillman.blogspot.com/2009/01/when-will-numpy-and-scipy-migrate-to.html Yes, the C code is most likely the big work. That's another reason why I have started to split the C code into smaller files which don't include each other - we can port one file after the other. And even though porting C code is obviously more difficult, handling thing with one codebase should be easier in C with conditional compilation, because it is "just" API changes, not language changes. > > Are the C code changes independent of the warnings provided by Python > 2.6 -3 option and eventually the 2to3 tool? I don't think there is any support for C code with the 2to3 tool. The following website has interesting links: http://pvergain.wordpress.com/py3k/ In particular, this one experience from Martin Loewis on porting some codebase with C: http://mail.python.org/pipermail/python-porting/2008-December/000010.html David From fccoelho at gmail.com Thu Apr 23 11:18:18 2009 From: fccoelho at gmail.com (Flavio Coelho) Date: Thu, 23 Apr 2009 16:18:18 +0100 Subject: [Numpy-discussion] bug in stats.randint In-Reply-To: <1cd32cbb0904230656i7127e91ek80cf0574663a7b85@mail.gmail.com> References: <1cd32cbb0904230656i7127e91ek80cf0574663a7b85@mail.gmail.com> Message-ID: On Thu, Apr 23, 2009 at 2:56 PM, wrote: > On Thu, Apr 23, 2009 at 9:27 AM, Flavio Coelho wrote: > > > > Hi, > > > > I stumbled upon something I think is a bug in scipy: > > > > In [4]: stats.randint(1.,15.).ppf([.1, > > .2,.3,.4,.5]) > > Out[4]: array([ 2., 3., 5., 6., 7.]) > > > > When you pass float arguments to stats.randint and then call the ppf > method, > > you get an array of floats, which clearly wrong. The rvs method doesn't > > display this bug so I think is a matter of poor type checking in the ppf > > implementation... > > > > I switched to using floats intentionally, to have correct handling of > inf and nans. and the argument checking is generic for all discrete > distributions and not special cased. Nans are converted to zero when > casting to integers, which is wrong and very confusing. inf raise an > exception. I prefer correct numbers to correct types. see examples > below I understand. I couldn't find, however, any parameterizations which makes randint return INF or NAN. So it should be safe to make randint return integers. I tested the equivalent function (for Poisson) in R (qpois) and it also returns INF when quantile is 1. so no problem there. I found some weird behaviors with stats.poisson.ppf: In [34]: stats.poisson.ppf([0,1]) Out[34]: array([ -1., Inf]) #R returns [0,inf] In [35]: stats.poisson.ppf([0.,.1]) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /home/fccoelho/ in () /usr/lib/python2.6/dist-packages/scipy/stats/distributions.pyc in ppf(self, q, *args, **kwds) 4002 goodargs = argsreduce(cond, *((q,)+args+(loc,))) 4003 loc, goodargs = goodargs[-1], goodargs[:-1] -> 4004 place(output,cond,self._ppf(*goodargs) + loc) 4005 4006 if output.ndim == 0: TypeError: _ppf() takes exactly 3 arguments (2 given) I hope these bug reports help. I think handling infinity and zero is almost problematic, maybe the best we can do is to aim for predictable behavior (possibly matching the behavior of equivalent R functions) I think that if randint would return ints no matter what the type of the parameters, it would make the behavior of my code a lot more predictable. > > > If you don't have inf and nans you can cast them to int yourself. > > Josef > > >>> aint = np.zeros(5,dtype=int) > >>> aint[0]= np.nan > >>> aint > array([0, 0, 0, 0, 0]) > >>> aint[1]= np.inf > Traceback (most recent call last): > File "", line 1, in > OverflowError: cannot convert float infinity to long > > >>> from scipy import stats > >>> stats.poisson.ppf(1,1) > inf > >>> stats.poisson.ppf(2,1) > nan > > >>> stats.poisson.ppf(1,1).astype(int) > -2147483648 > >>> aint[2] = stats.poisson.ppf(1,1) > Traceback (most recent call last): > File "", line 1, in > OverflowError: cannot convert float infinity to long > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- --- Fl?vio Code?o Coelho -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmay31 at gmail.com Thu Apr 23 11:30:19 2009 From: rmay31 at gmail.com (Ryan May) Date: Thu, 23 Apr 2009 10:30:19 -0500 Subject: [Numpy-discussion] Porting strategy for py3k In-Reply-To: <5b8d13220904230752u3c8dd7fyb14f937d358472fd@mail.gmail.com> References: <5b8d13220904230638i553d0e7bx2f3d93572861940c@mail.gmail.com> <5b8d13220904230752u3c8dd7fyb14f937d358472fd@mail.gmail.com> Message-ID: On Thu, Apr 23, 2009 at 9:52 AM, David Cournapeau wrote: > On Thu, Apr 23, 2009 at 11:20 PM, Pauli Virtanen wrote: > > Thu, 23 Apr 2009 22:38:21 +0900, David Cournapeau kirjoitti: > > [clip] > >> I looked more in detail on what would be needed to port numpy to > >> py3k. In particular, I was interested at the possible strategies to keep > >> one single codebase for both python 2.x and python 3.x. The first step > >> is to remove all py3k warnings reported by python 2.6. A couple of > >> recurrent problems > >> - reduce is removed in py3k > >> - print is removed > > > > Print is not removed, just changed to a function. So, > > > > print("foo") > > Yes, as reduce, they are still available, but not as builtins anymore. > But replacing print is not as easy as reduce. Things like print > "yoyo", a do not work, for example. > I think the point is that you can just change it to print("yoyo") which will work in both python 2.x and 3.x. The parenthesis are just extraneous in python 2.x. Now, the more complicated uses of print won't be as easy to change, but I'm not sure how prevalent their use is. Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma Sent from Norman, Oklahoma, United States -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Thu Apr 23 12:23:12 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 23 Apr 2009 09:23:12 -0700 Subject: [Numpy-discussion] Porting strategy for py3k In-Reply-To: References: <5b8d13220904230638i553d0e7bx2f3d93572861940c@mail.gmail.com> <5b8d13220904230752u3c8dd7fyb14f937d358472fd@mail.gmail.com> Message-ID: <49F095F0.90604@noaa.gov> Ryan May wrote: > On Thu, Apr 23, 2009 at 9:52 AM, David Cournapeau But replacing print is not as easy as reduce. Things like print > "yoyo", a do not work, for example. > > I think the point is that you can just change it to print("yoyo") which > will work in both python 2.x and 3.x. I think he meant: print "yoyo", a which can not be translated by adding parens: >>> print "yoyo", a yoyo [ 1. 1. 1.] >>> print ("yoyo", a) ('yoyo', array([ 1., 1., 1.])) I suppose we can write something like: def new_print(*args): print (" ".join([str(s) for s in args])) >>> new_print("yoyo", a) yoyo [ 1. 1. 1.] Though I'm a bit surprised that that's not how the print function is written in the first place (maybe it is in py3k -- I'm testing on 2.5) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From rmay31 at gmail.com Thu Apr 23 12:26:49 2009 From: rmay31 at gmail.com (Ryan May) Date: Thu, 23 Apr 2009 11:26:49 -0500 Subject: [Numpy-discussion] Porting strategy for py3k In-Reply-To: <49F095F0.90604@noaa.gov> References: <5b8d13220904230638i553d0e7bx2f3d93572861940c@mail.gmail.com> <5b8d13220904230752u3c8dd7fyb14f937d358472fd@mail.gmail.com> <49F095F0.90604@noaa.gov> Message-ID: On Thu, Apr 23, 2009 at 11:23 AM, Christopher Barker wrote: > Ryan May wrote: > > On Thu, Apr 23, 2009 at 9:52 AM, David Cournapeau > But replacing print is not as easy as reduce. Things like print > > "yoyo", a do not work, for example. > > > > I think the point is that you can just change it to print("yoyo") which > > will work in both python 2.x and 3.x. > > I think he meant: > > print "yoyo", a > > which can not be translated by adding parens: > > >>> print "yoyo", a > yoyo [ 1. 1. 1.] > >>> print ("yoyo", a) > ('yoyo', array([ 1., 1., 1.])) > > > I suppose we can write something like: > > def new_print(*args): > print (" ".join([str(s) for s in args])) > > >>> new_print("yoyo", a) > yoyo [ 1. 1. 1.] > > > Though I'm a bit surprised that that's not how the print function is > written in the first place (maybe it is in py3k -- I'm testing on 2.5) > > -Chris > Good point. We could just borrow the implementation from 2.6 and in fact just import print from future on 2.6. Just a thought... Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma Sent from Norman, Oklahoma, United States -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Apr 23 12:56:20 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 23 Apr 2009 12:56:20 -0400 Subject: [Numpy-discussion] bug in stats.randint In-Reply-To: References: <1cd32cbb0904230656i7127e91ek80cf0574663a7b85@mail.gmail.com> Message-ID: <1cd32cbb0904230956s2dfd54d5u46fb4db79ea8f99c@mail.gmail.com> On Thu, Apr 23, 2009 at 11:18 AM, Flavio Coelho wrote: > > > On Thu, Apr 23, 2009 at 2:56 PM, wrote: >> >> On Thu, Apr 23, 2009 at 9:27 AM, Flavio Coelho wrote: >> > >> > Hi, >> > >> > I stumbled upon something I think is a bug in scipy: >> > >> > In [4]: stats.randint(1.,15.).ppf([.1, >> > .2,.3,.4,.5]) >> > Out[4]: array([ 2.,? 3.,? 5.,? 6.,? 7.]) >> > >> > When you pass float arguments to stats.randint and then call the ppf >> > method, >> > you get an array of floats, which clearly wrong. The rvs method doesn't >> > display this bug so I think is a matter of poor type checking in the ppf >> > implementation... >> > >> >> I switched to using floats intentionally, to have correct handling of >> inf and nans. and the argument checking is generic for all discrete >> distributions and not special cased. Nans are converted to zero when >> casting to integers, which is wrong and very confusing. inf raise an >> exception. I prefer correct numbers to correct types. see examples >> below > > I understand. I couldn't find, however, any parameterizations which makes > randint return INF or NAN. So it should be safe to make randint return > integers. > > I tested the equivalent function (for Poisson) in R (qpois) and it also > returns INF when quantile is 1. so no problem there. > > I found some weird behaviors with stats.poisson.ppf: > > In [34]: stats.poisson.ppf([0,1]) > Out[34]: array([ -1.,? Inf]) #R returns [0,inf] > In [35]: stats.poisson.ppf([0.,.1]) > --------------------------------------------------------------------------- > TypeError???????????????????????????????? Traceback (most recent call last) > > /home/fccoelho/ in () > > /usr/lib/python2.6/dist-packages/scipy/stats/distributions.pyc in ppf(self, > q, *args, **kwds) > ?? 4002???????????? goodargs = argsreduce(cond, *((q,)+args+(loc,))) > ?? 4003???????????? loc, goodargs = goodargs[-1], goodargs[:-1] > -> 4004???????????? place(output,cond,self._ppf(*goodargs) + loc) > ?? 4005 > ?? 4006???????? if output.ndim == 0: > > TypeError: _ppf() takes exactly 3 arguments (2 given) poisson has one required paramater for the distribution >>> print stats.poisson.extradoc Poisson distribution poisson.pmf(k, mu) = exp(-mu) * mu**k / k! for k >= 0 >>> stats.poisson.numargs # number of required distribution parameters 1 so raising an exception in stats.poisson.ppf([0.,.1]) is correct since you didn't specify a parameter stats.poisson.ppf([0.,.1],1) works stats.poisson.ppf([0,1]) is getting the boundary treatment and is never checked for valid args However, the difference for the two cases is strange, and I need to look closer at this. stats.poisson.ppf([0,0.5,1]) raises an exception as expected. The ppf at 0 was defined as "lower bound minus one" in Travis' pdf files, which is consistent with the definition of the ppf >>> stats.poisson.ppf(0,1) -1.0 It always takes me a while to figure out how the inequality in ppf are supposed to go, the best is to draw a step function and check the inverse mapping. >>> k=[-1,0,1,2]; stats.poisson.ppf(stats.poisson.cdf(k,1),1) array([-1., 0., 1., 2.]) >>> stats.poisson.ppf(stats.poisson.cdf(k,1)-1e-5,1) array([ NaN, 0., 1., 2.]) >>> stats.poisson.ppf(stats.poisson.cdf(k,1)+1e-5,1) array([ 0., 1., 2., 3.]) If q are quantiles, then a half open interval, of q, open left, closed right, maps to the same integer k. so q=0 does not belong to the interval that maps to k=0. This might be a convention, but is required for consistency in some cases, which I don't remember. I think defining stats.poisson.ppf(0,1)=0 violates some inequalities and would need special casing. > > I hope these bug reports help. > I think handling infinity and zero is almost problematic, maybe the best we > can do is to aim for predictable behavior (possibly matching the behavior of > equivalent R functions) > > > > > > I think that if randint would return ints no matter what the type of the > parameters, it would make the behavior of my code a lot more predictable. >> >> >> If you don't have inf and nans you can cast them to int yourself. you just need to add "int(result)" for scalar or "result.astype(int)" for array returns if you want to have ints. But it is your responsibility to check that your code doesn't break with nan, if you have an invalid input. e.g. >>> stats.poisson.ppf(0.5,-1) nan I had checked with R, when I made the type change, and what I remember is that, for most distributions, R return floats for ppf, but not in all packages or for all distributions. >> >> Josef >> >> >>> aint = np.zeros(5,dtype=int) >> >>> aint[0]= np.nan >> >>> aint >> array([0, 0, 0, 0, 0]) >> >>> aint[1]= np.inf >> Traceback (most recent call last): >> ?File "", line 1, in >> OverflowError: cannot convert float infinity to long >> >> >>> from scipy import stats >> >>> stats.poisson.ppf(1,1) >> inf >> >>> stats.poisson.ppf(2,1) >> nan >> >> >>> stats.poisson.ppf(1,1).astype(int) >> -2147483648 >> >>> aint[2] = stats.poisson.ppf(1,1) >> Traceback (most recent call last): >> ?File "", line 1, in >> OverflowError: cannot convert float infinity to long From sccolbert at gmail.com Thu Apr 23 14:54:10 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Thu, 23 Apr 2009 14:54:10 -0400 Subject: [Numpy-discussion] numpy 1.3.0 test() failures on ubuntu 9.04 Message-ID: <7f014ea60904231154s5b3afa0aueed5a0243fcccf18@mail.gmail.com> Hi all, I'm building numpy 1.3.0 from source with libatlas-sse2-dev from the jaunty repos. I'm running into 16 failures when running the nose test. This is a fresh install of 9.04 and i've repod the following packages: build-essential swig gfortran python-dev libatlas-sse2-dev libatlas-base-dev (the atlas packages include lapack and blas libraries) The pastebin links to site.cfg build.log and test.log are at the end of this email. If anyone could help me out here as to what the problems may be, i would appreciate it! Thanks! Chris site.cfg http://ubuntu.pastebin.com/m671e9454 build.log http://ubuntu.pastebin.com/m581eee89 test.log http://ubuntu.pastebin.com/m7b63f6a0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Thu Apr 23 14:57:28 2009 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 23 Apr 2009 18:57:28 +0000 (UTC) Subject: [Numpy-discussion] numpy 1.3.0 test() failures on ubuntu 9.04 References: <7f014ea60904231154s5b3afa0aueed5a0243fcccf18@mail.gmail.com> Message-ID: Thu, 23 Apr 2009 14:54:10 -0400, Chris Colbert wrote: [clip] > libatlas-sse2-dev [clip] > The pastebin links to site.cfg build.log and test.log are at the end of > this email. If anyone could help me out here as to what the problems may > be, i would appreciate it! The SSE-optimized Atlas libraries shipped with Ubuntu 9.04 are broken: https://bugs.launchpad.net/bugs/363510 -- Pauli Virtanen From sccolbert at gmail.com Thu Apr 23 15:00:40 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Thu, 23 Apr 2009 15:00:40 -0400 Subject: [Numpy-discussion] numpy 1.3.0 test() failures on ubuntu 9.04 In-Reply-To: References: <7f014ea60904231154s5b3afa0aueed5a0243fcccf18@mail.gmail.com> Message-ID: <7f014ea60904231200x267d0a49g9553eaca831f42ca@mail.gmail.com> ha! gotta love the one line answers. Thanks Pauli! On Thu, Apr 23, 2009 at 2:57 PM, Pauli Virtanen wrote: > Thu, 23 Apr 2009 14:54:10 -0400, Chris Colbert wrote: > [clip] > > libatlas-sse2-dev > [clip] > > The pastebin links to site.cfg build.log and test.log are at the end of > > this email. If anyone could help me out here as to what the problems may > > be, i would appreciate it! > > The SSE-optimized Atlas libraries shipped with Ubuntu 9.04 are broken: > > https://bugs.launchpad.net/bugs/363510 > > -- > Pauli Virtanen > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Apr 23 15:39:10 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 23 Apr 2009 14:39:10 -0500 Subject: [Numpy-discussion] Porting strategy for py3k In-Reply-To: <5b8d13220904230752u3c8dd7fyb14f937d358472fd@mail.gmail.com> References: <5b8d13220904230638i553d0e7bx2f3d93572861940c@mail.gmail.com> <5b8d13220904230752u3c8dd7fyb14f937d358472fd@mail.gmail.com> Message-ID: <3d375d730904231239o51b7f80av22d0c0f76caaf952@mail.gmail.com> On Thu, Apr 23, 2009 at 09:52, David Cournapeau wrote: > On Thu, Apr 23, 2009 at 11:20 PM, Pauli Virtanen wrote: >> Thu, 23 Apr 2009 22:38:21 +0900, David Cournapeau kirjoitti: >> [clip] >>> ? ? I looked more in detail on what would be needed to port numpy to >>> py3k. In particular, I was interested at the possible strategies to keep >>> one single codebase for both python 2.x and python 3.x. The first step >>> is to remove all py3k warnings reported by python 2.6. A couple of >>> recurrent problems >>> ? ? - reduce is removed in py3k >>> ? ? - print is removed >> >> Print is not removed, just changed to a function. So, >> >> ? ? ? ?print("foo") > > Yes, as reduce, they are still available, but not as builtins anymore. > But replacing print is not as easy as reduce. Things like print > "yoyo", a do not work, for example. Doesn't the 2to3 tool do this conversion for you? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From michael.abshoff at googlemail.com Thu Apr 23 18:59:08 2009 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Thu, 23 Apr 2009 15:59:08 -0700 Subject: [Numpy-discussion] numpy 1.3.0 test() failures on ubuntu 9.04 In-Reply-To: References: <7f014ea60904231154s5b3afa0aueed5a0243fcccf18@mail.gmail.com> Message-ID: <49F0F2BC.9060804@gmail.com> Pauli Virtanen wrote: > Thu, 23 Apr 2009 14:54:10 -0400, Chris Colbert wrote: > [clip] >> libatlas-sse2-dev > [clip] >> The pastebin links to site.cfg build.log and test.log are at the end of >> this email. If anyone could help me out here as to what the problems may >> be, i would appreciate it! > > The SSE-optimized Atlas libraries shipped with Ubuntu 9.04 are broken: > > https://bugs.launchpad.net/bugs/363510 > Slightly OT: Is Ubuntu still shipping ATLAS 3.6.0 or am I misinterpreting the version number somehow? Cheers, Michael From gaurdro at gmail.com Thu Apr 23 19:29:30 2009 From: gaurdro at gmail.com (ross smith) Date: Thu, 23 Apr 2009 19:29:30 -0400 Subject: [Numpy-discussion] Porting strategy for py3k In-Reply-To: <3d375d730904231239o51b7f80av22d0c0f76caaf952@mail.gmail.com> References: <5b8d13220904230638i553d0e7bx2f3d93572861940c@mail.gmail.com> <5b8d13220904230752u3c8dd7fyb14f937d358472fd@mail.gmail.com> <3d375d730904231239o51b7f80av22d0c0f76caaf952@mail.gmail.com> Message-ID: <73531abb0904231629y13353867v9c5c844488fba0a@mail.gmail.com> On Thu, Apr 23, 2009 at 15:39, Robert Kern wrote: > On Thu, Apr 23, 2009 at 09:52, David Cournapeau > wrote: > > On Thu, Apr 23, 2009 at 11:20 PM, Pauli Virtanen wrote: > >> Thu, 23 Apr 2009 22:38:21 +0900, David Cournapeau kirjoitti: > >> [clip] > >>> I looked more in detail on what would be needed to port numpy to > >>> py3k. In particular, I was interested at the possible strategies to > keep > >>> one single codebase for both python 2.x and python 3.x. The first step > >>> is to remove all py3k warnings reported by python 2.6. A couple of > >>> recurrent problems > >>> - reduce is removed in py3k > >>> - print is removed > >> > >> Print is not removed, just changed to a function. So, > >> > >> print("foo") > > > > Yes, as reduce, they are still available, but not as builtins anymore. > > But replacing print is not as easy as reduce. Things like print > > "yoyo", a do not work, for example. > > Doesn't the 2to3 tool do this conversion for you? It does for Print at least. conditional compiling for C would work. I could see extending it to the python code as well. I don't know how feasible this is, but have a script that runs at installation and cuts out blocks of code that aren't compatible with the version it's being installed on. for example, run the script on a file like below, and keep only what is useable by the version installed. import foo ###py2x.begin import py2xbar.py ###py2x.end ###py3k.begin import py3kbar.py ###py3k.end -just my .02 -Ross > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Thu Apr 23 20:05:25 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 24 Apr 2009 09:05:25 +0900 Subject: [Numpy-discussion] numpy 1.3.0 test() failures on ubuntu 9.04 In-Reply-To: <49F0F2BC.9060804@gmail.com> References: <7f014ea60904231154s5b3afa0aueed5a0243fcccf18@mail.gmail.com> <49F0F2BC.9060804@gmail.com> Message-ID: <5b8d13220904231705m27f95c4fjac78f663511599c2@mail.gmail.com> On Fri, Apr 24, 2009 at 7:59 AM, Michael Abshoff wrote: > Pauli Virtanen wrote: >> Thu, 23 Apr 2009 14:54:10 -0400, Chris Colbert wrote: >> [clip] >>> libatlas-sse2-dev >> [clip] >>> The pastebin links to site.cfg build.log and test.log are at the end of >>> this email. If anyone could help me out here as to what the problems may >>> be, i would appreciate it! >> >> The SSE-optimized Atlas libraries shipped with Ubuntu 9.04 are broken: >> >> ? ? ? https://bugs.launchpad.net/bugs/363510 >> > > Slightly OT: Is Ubuntu still shipping ATLAS 3.6.0 or am I > misinterpreting the version number somehow? Yes they do. Updating the atlas package is a lot of work because of the way they are generated (the build commands are recorded and replayed to regenerate the package)> David From william.ratcliff at gmail.com Fri Apr 24 00:12:37 2009 From: william.ratcliff at gmail.com (william ratcliff) Date: Fri, 24 Apr 2009 00:12:37 -0400 Subject: [Numpy-discussion] patching docs Message-ID: <827183970904232112o1ae7a507v4cfd632fa36dd208@mail.gmail.com> Hi! I'd like to suggest a patch for: numpy .core .fromnumeric .put The docstring contains: for i, val in zip(ind,v): x.flat[i]=val It should be: for i, val in zip(ind,v): a.flat[i]=val This would be more consistent with the rest of the docstring. For the future, is there an easy way for us to patch docstrings? Thanks, William -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Fri Apr 24 00:19:06 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 24 Apr 2009 13:19:06 +0900 Subject: [Numpy-discussion] Porting strategy for py3k In-Reply-To: <3d375d730904231239o51b7f80av22d0c0f76caaf952@mail.gmail.com> References: <5b8d13220904230638i553d0e7bx2f3d93572861940c@mail.gmail.com> <5b8d13220904230752u3c8dd7fyb14f937d358472fd@mail.gmail.com> <3d375d730904231239o51b7f80av22d0c0f76caaf952@mail.gmail.com> Message-ID: <49F13DBA.5040904@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > > Doesn't the 2to3 tool do this conversion for you? > I have not seen it handling everything. It does not handle reduce, for example: print reduce(lambda x, y: x+y, [1, 2, 3]) is translated to print(reduce(lambda x, y: x + y, [1, 1, 1])) I guess an alternative to a compat module would be to extend 2to3, or maybe mixing the two approaches. My understanding is that 2to3 is "optimized" to get a single codebase for python 2.6 and 3.x, but we want to be compatible below (>=2.4). David From david at ar.media.kyoto-u.ac.jp Fri Apr 24 00:24:15 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 24 Apr 2009 13:24:15 +0900 Subject: [Numpy-discussion] Porting strategy for py3k In-Reply-To: <49F095F0.90604@noaa.gov> References: <5b8d13220904230638i553d0e7bx2f3d93572861940c@mail.gmail.com> <5b8d13220904230752u3c8dd7fyb14f937d358472fd@mail.gmail.com> <49F095F0.90604@noaa.gov> Message-ID: <49F13EEF.6060205@ar.media.kyoto-u.ac.jp> Christopher Barker wrote: > Though I'm a bit surprised that that's not how the print function is > written in the first place (maybe it is in py3k -- I'm testing on 2.5) > That's actually how it works as far as I can tell. The thing with removing those print is that we can do it without too much trouble. As long as we cannot actually test any py3k code, warnings from python 2.6 is all we can get. I think we should aim at getting "something" which builds and runs (even if does not go further than import stage), so we can gradually port. For now, porting py3k is this huge thing that nobody can work on for say one hour. I would like to make sure we get at that stage, so that many people can take part of it, instead of the currently quite few people who are deeply intimate with numpy. David From stefan at sun.ac.za Fri Apr 24 01:03:14 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 24 Apr 2009 07:03:14 +0200 Subject: [Numpy-discussion] patching docs In-Reply-To: <827183970904232112o1ae7a507v4cfd632fa36dd208@mail.gmail.com> References: <827183970904232112o1ae7a507v4cfd632fa36dd208@mail.gmail.com> Message-ID: <9457e7c80904232203o16c07253o5c05dbd2d2d01c74@mail.gmail.com> Hi William Please sign up for an account on the docs editor at http://docs.scipy.org Regards St?fan 2009/4/24 william ratcliff : > Hi!? I'd like to suggest a patch for: > > numpy.core.fromnumeric.put [...] > This would be more consistent with the rest of the docstring.? For the > future, is there an easy way for us to patch docstrings? From william.ratcliff at gmail.com Fri Apr 24 01:44:41 2009 From: william.ratcliff at gmail.com (william ratcliff) Date: Fri, 24 Apr 2009 01:44:41 -0400 Subject: [Numpy-discussion] patching docs In-Reply-To: <9457e7c80904232203o16c07253o5c05dbd2d2d01c74@mail.gmail.com> References: <827183970904232112o1ae7a507v4cfd632fa36dd208@mail.gmail.com> <9457e7c80904232203o16c07253o5c05dbd2d2d01c74@mail.gmail.com> Message-ID: <827183970904232244u20c87915x9fcf5992d2b0df6e@mail.gmail.com> I signed up for an account as williamratcliff May I have edit rights? Thanks, William 2009/4/24 St?fan van der Walt > Hi William > > Please sign up for an account on the docs editor at > > http://docs.scipy.org > > Regards > St?fan > > 2009/4/24 william ratcliff : > > Hi! I'd like to suggest a patch for: > > > > numpy.core.fromnumeric.put > > [...] > > > This would be more consistent with the rest of the docstring. For the > > future, is there an easy way for us to patch docstrings? > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Fri Apr 24 02:53:16 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 24 Apr 2009 08:53:16 +0200 Subject: [Numpy-discussion] patching docs In-Reply-To: <827183970904232244u20c87915x9fcf5992d2b0df6e@mail.gmail.com> References: <827183970904232112o1ae7a507v4cfd632fa36dd208@mail.gmail.com> <9457e7c80904232203o16c07253o5c05dbd2d2d01c74@mail.gmail.com> <827183970904232244u20c87915x9fcf5992d2b0df6e@mail.gmail.com> Message-ID: <9457e7c80904232353y25f159cq1176407b7391ac8c@mail.gmail.com> 2009/4/24 william ratcliff : > I signed up for an account as williamratcliff > > May I have edit rights? Done! Cheers St?fan From dagss at student.matnat.uio.no Fri Apr 24 03:49:43 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 24 Apr 2009 09:49:43 +0200 Subject: [Numpy-discussion] Porting strategy for py3k In-Reply-To: <49F13EEF.6060205@ar.media.kyoto-u.ac.jp> References: <5b8d13220904230638i553d0e7bx2f3d93572861940c@mail.gmail.com> <5b8d13220904230752u3c8dd7fyb14f937d358472fd@mail.gmail.com> <49F095F0.90604@noaa.gov> <49F13EEF.6060205@ar.media.kyoto-u.ac.jp> Message-ID: <49F16F17.9000303@student.matnat.uio.no> David Cournapeau wrote: > Christopher Barker wrote: >> Though I'm a bit surprised that that's not how the print function is >> written in the first place (maybe it is in py3k -- I'm testing on 2.5) >> > > That's actually how it works as far as I can tell. The thing with > removing those print is that we can do it without too much trouble. As > long as we cannot actually test any py3k code, warnings from python 2.6 > is all we can get. > > I think we should aim at getting "something" which builds and runs (even > if does not go further than import stage), so we can gradually port. For > now, porting py3k is this huge thing that nobody can work on for say one > hour. I would like to make sure we get at that stage, so that many > people can take part of it, instead of the currently quite few people > who are deeply intimate with numpy. One thing somebody *could* work on rather independently for some hours is proper PEP 3118 support, as that is available in Python 2.6+ as well and could be conditionally used on those systems. Cython already contains the start of an implementation of this on NumPy's behalf (though it would need to be improved as one does no longer know the characteristics of the client but must implement the whole API). I think it would be better if I spend my time on Cython, but I'm willing to mentor anybody who'd like to do this in NumPy as I've worked a lot with PEP 3118. (OTOH Travis wrote that PEP :-) ). -- Dag Sverre From pav at iki.fi Fri Apr 24 04:00:18 2009 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 24 Apr 2009 08:00:18 +0000 (UTC) Subject: [Numpy-discussion] Keeping track of backports in Trac Message-ID: Hi, Is it possible to assign a bug to multiple versions in Trac? Now that we have active stable branches eg. 0.7.x in Scipy, it would be useful if it was easy to track what to backport. For example, having a Trac custom field "Pending backports", either a boolean or a string listing the branches, would help this. -- Pauli Virtanen From jh at physics.ucf.edu Fri Apr 24 09:43:56 2009 From: jh at physics.ucf.edu (Joe Harrington) Date: Fri, 24 Apr 2009 09:43:56 -0400 Subject: [Numpy-discussion] patching docs In-Reply-To: (numpy-discussion-request@scipy.org) References: Message-ID: william ratcliff writes: > Hi! I'd like to suggest a patch for: > numpy .core > .fromnumeric .put > The docstring contains: > for i, val in zip(ind,v): > x.flat[i]=val > It should be: > for i, val in zip(ind,v): > a.flat[i]=val Thanks! It appears you are basing your patch on a numpy release rather than the wiki you have actually linked to, as this problem was fixed on 2009-03-16 by Ralf Gommers. > This would be more consistent with the rest of the docstring. For the > future, is there an easy way for us to patch docstrings? Sure! Follow the instructions on docs.scipy.org/numpy/ and make yourself a doc wiki account. Post the account name to the scipy-dev mailing list and ask to be made an editor. Then please edit! But, please be careful that you develop any changes based on the current docstrings and not something from a release. Because the releases are snapshots of a fast-moving project, they are often out of date (In fact, I tell my students to use the doc wiki as their best source of docs, and to ignore help()). Thanks again, --jh-- From william.ratcliff at gmail.com Fri Apr 24 12:12:07 2009 From: william.ratcliff at gmail.com (william ratcliff) Date: Fri, 24 Apr 2009 12:12:07 -0400 Subject: [Numpy-discussion] patching docs In-Reply-To: References: Message-ID: <827183970904240912v78b019f9kbd2c9e5bac9620e0@mail.gmail.com> Actually, if I look here: http://docs.scipy.org/numpy/docs/numpy.core.fromnumeric.put/ The text that appears in by browser is: *put(a, ind, v, mode='raise')* Changes specific elements of one array by replacing from another array. The indexing works on the flattened target array, putis roughly equivalent to: for i, val in zip(ind, v): x.flat[i] = val Am I missing something obvious???? cheers, William On Fri, Apr 24, 2009 at 9:43 AM, Joe Harrington wrote: > william ratcliff writes: > > > Hi! I'd like to suggest a patch for: > > numpy .core< > http://docs.scipy.org/numpy/docs/numpy.core/> > > .fromnumeric >.put > > The docstring contains: > > > for i, val in zip(ind,v): > > x.flat[i]=val > > > It should be: > > > for i, val in zip(ind,v): > > a.flat[i]=val > > Thanks! > > It appears you are basing your patch on a numpy release rather than > the wiki you have actually linked to, as this problem was fixed on > 2009-03-16 by Ralf Gommers. > > > This would be more consistent with the rest of the docstring. For the > > future, is there an easy way for us to patch docstrings? > > Sure! Follow the instructions on docs.scipy.org/numpy/ and make > yourself a doc wiki account. Post the account name to the scipy-dev > mailing list and ask to be made an editor. Then please edit! But, > please be careful that you develop any changes based on the current > docstrings and not something from a release. Because the releases are > snapshots of a fast-moving project, they are often out of date (In > fact, I tell my students to use the doc wiki as their best source of > docs, and to ignore help()). > > Thanks again, > > --jh-- > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Fri Apr 24 12:40:05 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 24 Apr 2009 18:40:05 +0200 Subject: [Numpy-discussion] patching docs In-Reply-To: <827183970904240912v78b019f9kbd2c9e5bac9620e0@mail.gmail.com> References: <827183970904240912v78b019f9kbd2c9e5bac9620e0@mail.gmail.com> Message-ID: <9457e7c80904240940y601e64d6y7765428222ec52ca@mail.gmail.com> 2009/4/24 william ratcliff : > Actually, if I look here: > http://docs.scipy.org/numpy/docs/numpy.core.fromnumeric.put/ > > The text that appears in by browser is: > > put(a, ind, v, mode='raise') > > Changes specific elements of one array by replacing from another array. This is pretty cryptic. I've changed it to """ put(a, ind, v, mode='raise') Replaces specified elements of an array with given values. """ Your error report was correct. I've changed the x to an a. Cheers St?fan From michael.s.gilbert at gmail.com Fri Apr 24 12:52:33 2009 From: michael.s.gilbert at gmail.com (Michael S. Gilbert) Date: Fri, 24 Apr 2009 12:52:33 -0400 Subject: [Numpy-discussion] Idea: fractional polynomial class Message-ID: <20090424125233.b067f24d.michael.s.gilbert@gmail.com> Hello, I've been working with numpy's poly1d class recently, and it would be very useful to me if the class worked with fractions instead of floats (since I'm encountering quantities that often mostly cancel out, which lead to instabilities in my algorithms; hence it would be useful to use fractions that don't lose precision). Since python 2.6 introduced a "fractions" class, it should be fairly straightforward to implement/reimplement the polynomial class with support for fractions. I'm curious as to whether there would be any interest from the project on this and whether you all would be open to patches/code contributions. Best Regards, Mike From pav at iki.fi Fri Apr 24 13:09:06 2009 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 24 Apr 2009 17:09:06 +0000 (UTC) Subject: [Numpy-discussion] Idea: fractional polynomial class References: <20090424125233.b067f24d.michael.s.gilbert@gmail.com> Message-ID: Hi, Fri, 24 Apr 2009 12:52:33 -0400, Michael S. Gilbert kirjoitti: > I've been working with numpy's poly1d class recently, and it would be > very useful to me if the class worked with fractions instead of floats > (since I'm encountering quantities that often mostly cancel out, which > lead to instabilities in my algorithms; hence it would be useful to use > fractions that don't lose precision). Since python 2.6 introduced a > "fractions" class, it should be fairly straightforward to > implement/reimplement the polynomial class with support for fractions. > > I'm curious as to whether there would be any interest from the project > on this and whether you all would be open to patches/code contributions. Definitely, provided fractions support the usual Python operators. (I object to having a specialized 'fractional' polynomial class, though.) The changes required are probably very small: only `polyint` needs some fixes. So, thanks to duck typing, it mostly works out of the box already. For example, I can use mpmath's arbitrary precision numbers in poly1d: >>> import numpy as np >>> from mpmath import mp, mpf >>> p = np.poly1d([mpf('1.0'), mpf('2.0'), mpf('3.0')]) >>> p(1/mpf('3')) mpf('3.777777777777777777777777777777777777777773') >>> p * mpf('1.9999999999999999999999999') poly1d([1.9999999999999999999999999, 3.9999999999999999999999998, 5.9999999999999999999999997], dtype=object) >>> p*p poly1d([1.0, 4.0, 10.0, 12.0, 9.0], dtype=object) >>> np.polyder(p*p, 3) poly1d([24.0, 24.0], dtype=object) Polyder uses only exact integers, so it does not lose precision. The only thing that does not work is integration, since it seems to force-cast to float: >>> np.polyint(p) poly1d([ 0.33333333, 1. , 3. , 0. ]) This is a bug in itself, since poly1d could well have complex coefficients. -- Pauli Virtanen From dwf at cs.toronto.edu Fri Apr 24 13:45:33 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Fri, 24 Apr 2009 13:45:33 -0400 Subject: [Numpy-discussion] inplace matrix multiplication In-Reply-To: <3d375d730901091331u26a0d6dcw8dffca7349fec96@mail.gmail.com> References: <2d1d7fe70901090625k53fe9415uf177e2d80645f12c@mail.gmail.com> <3d375d730901091331u26a0d6dcw8dffca7349fec96@mail.gmail.com> Message-ID: On 9-Jan-09, at 4:31 PM, Robert Kern wrote: > You can't in numpy. With scipy.linalg.fblas.dgemm() and the right > arguments, you can. Make sure your output array is Fortran-ordered, however, otherwise copies will be made. David From william.ratcliff at gmail.com Fri Apr 24 13:58:23 2009 From: william.ratcliff at gmail.com (william ratcliff) Date: Fri, 24 Apr 2009 13:58:23 -0400 Subject: [Numpy-discussion] patching docs In-Reply-To: <9457e7c80904240940y601e64d6y7765428222ec52ca@mail.gmail.com> References: <827183970904240912v78b019f9kbd2c9e5bac9620e0@mail.gmail.com> <9457e7c80904240940y601e64d6y7765428222ec52ca@mail.gmail.com> Message-ID: <827183970904241058t28eceeffmf950eac3ff30b1c3@mail.gmail.com> Thanks! I was up pretty late last night and wanted to make sure I hadn't missed something while sleep deprived! Cheers, William 2009/4/24 St?fan van der Walt > 2009/4/24 william ratcliff : > > Actually, if I look here: > > http://docs.scipy.org/numpy/docs/numpy.core.fromnumeric.put/ > > > > The text that appears in by browser is: > > > > put(a, ind, v, mode='raise') > > > > Changes specific elements of one array by replacing from another array. > > This is pretty cryptic. I've changed it to > > """ > put(a, ind, v, mode='raise') > > Replaces specified elements of an array with given values. > """ > > Your error report was correct. I've changed the x to an a. > > Cheers > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Apr 24 14:49:36 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 24 Apr 2009 12:49:36 -0600 Subject: [Numpy-discussion] Idea: fractional polynomial class In-Reply-To: References: <20090424125233.b067f24d.michael.s.gilbert@gmail.com> Message-ID: On Fri, Apr 24, 2009 at 11:09 AM, Pauli Virtanen wrote: > Hi, > > Fri, 24 Apr 2009 12:52:33 -0400, Michael S. Gilbert kirjoitti: > > I've been working with numpy's poly1d class recently, and it would be > > very useful to me if the class worked with fractions instead of floats > > (since I'm encountering quantities that often mostly cancel out, which > > lead to instabilities in my algorithms; hence it would be useful to use > > fractions that don't lose precision). Since python 2.6 introduced a > > "fractions" class, it should be fairly straightforward to > > implement/reimplement the polynomial class with support for fractions. > > > > I'm curious as to whether there would be any interest from the project > > on this and whether you all would be open to patches/code contributions. > > Definitely, provided fractions support the usual Python operators. > (I object to having a specialized 'fractional' polynomial class, though.) > I tend to think that this sort of thing belongs to a specialized package like sympy or sage. The polynomial class in numpy is a quick and dirty convenience class for numerical computation. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Apr 24 15:04:50 2009 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 24 Apr 2009 19:04:50 +0000 (UTC) Subject: [Numpy-discussion] Idea: fractional polynomial class References: <20090424125233.b067f24d.michael.s.gilbert@gmail.com> Message-ID: Fri, 24 Apr 2009 12:49:36 -0600, Charles R Harris wrote: [clip] > I tend to think that this sort of thing belongs to a specialized package > like sympy or sage. The polynomial class in numpy is a quick and dirty > convenience class for numerical computation. Well, Numpy's poly1d will work fine with minor changes, so I see no reason not to make sure we're consistent with duck typing. -- Pauli Virtanen From jh at physics.ucf.edu Fri Apr 24 16:41:36 2009 From: jh at physics.ucf.edu (Joe Harrington) Date: Fri, 24 Apr 2009 16:41:36 -0400 Subject: [Numpy-discussion] patching docs In-Reply-To: (numpy-discussion-request@scipy.org) References: Message-ID: > 2009/4/24 william ratcliff : > > Actually, if I look here: > > http://docs.scipy.org/numpy/docs/numpy.core.fromnumeric.put/ > > > > The text that appears in by browser is: > > > > put(a, ind, v, mode='raise') > > > > Changes specific elements of one array by replacing from another array. > This is pretty cryptic. I've changed it to > """ > put(a, ind, v, mode='raise') > > Replaces specified elements of an array with given values. > """ > Your error report was correct. I've changed the x to an a. Ok, I thought you were complaining that the a and x in the examples were not consistently using the same variable (which is not super-important but Ralf fixed it anyway). Now I see that you were saying that the variable name in parameters was not the same as that in the example. Good catch. I've now made the examples all use a. Also, the example in the extended summary was very unpythonic, so I removed the loop. I think that's ok but someone tell me if it's not, or just go fix it. I did test it on a case like one of the examples and it worked. Thanks, --jh-- From pav at iki.fi Fri Apr 24 17:30:56 2009 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 24 Apr 2009 21:30:56 +0000 (UTC) Subject: [Numpy-discussion] Idea: fractional polynomial class References: <20090424125233.b067f24d.michael.s.gilbert@gmail.com> Message-ID: Fri, 24 Apr 2009 19:04:50 +0000, Pauli Virtanen wrote: > Well, Numpy's poly1d will work fine with minor changes, so I see no > reason not to make sure we're consistent with duck typing. polyint fixed in r6883, there doesn't seem anything else to fix: >>> import numpy as np >>> from fractions import Fraction >>> a = np.poly1d([Fraction(4,19), Fraction(3,19), Fraction(2,19), 0]) >>> a poly1d([4/19, 3/19, 2/19, 0], dtype=object) >>> a.integ() poly1d([1/19, 1/19, 1/19, 0.0, 0], dtype=object) >>> a.deriv() poly1d([12/19, 6/19, 2/19], dtype=object) >>> a / 19 poly1d([4/361, 3/361, 2/361, 0], dtype=object) >>> a**3 poly1d([64/6859, 144/6859, 204/6859, 9/361, 102/6859, 36/6859, 8/6859, 0, 0, 0], dtype=object) -- Pauli Virtanen From stefan at sun.ac.za Fri Apr 24 21:42:46 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sat, 25 Apr 2009 03:42:46 +0200 Subject: [Numpy-discussion] Idea: fractional polynomial class In-Reply-To: References: <20090424125233.b067f24d.michael.s.gilbert@gmail.com> Message-ID: <9457e7c80904241842l3e1df274u80f5f5febdf0381@mail.gmail.com> 2009/4/24 Pauli Virtanen : > polyint fixed in r6883, there doesn't seem anything else to fix: > >>>> import numpy as np >>>> from fractions import Fraction >>>> a = np.poly1d([Fraction(4,19), Fraction(3,19), Fraction(2,19), 0]) Very cool. I think we should mention thi example in the docs as well? Cheers St?fan From sccolbert at gmail.com Fri Apr 24 22:11:07 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Fri, 24 Apr 2009 22:11:07 -0400 Subject: [Numpy-discussion] after building from source, how to register numpy with synaptic? Message-ID: <7f014ea60904241911t77c02fe7w4069bb92434e6421@mail.gmail.com> Like the subject says, is there a way to register numpy with synaptic after building numpy from source? I would like to snag matplotlib from the ubuntu repos, but it won't let me without also getting numpy and its dependency, which would ruin all the work I did building numpy on my machine. This is probably a linux n00b question, but well, that's what I am :) Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Apr 24 22:12:12 2009 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 24 Apr 2009 21:12:12 -0500 Subject: [Numpy-discussion] after building from source, how to register numpy with synaptic? In-Reply-To: <7f014ea60904241911t77c02fe7w4069bb92434e6421@mail.gmail.com> References: <7f014ea60904241911t77c02fe7w4069bb92434e6421@mail.gmail.com> Message-ID: <3d375d730904241912l5b5855d9kca6619ac010d341c@mail.gmail.com> On Fri, Apr 24, 2009 at 21:11, Chris Colbert wrote: > Like the subject says, is there a way to register numpy with synaptic after > building numpy from source? > > I would like to snag matplotlib from the ubuntu repos, but it won't let me > without also getting numpy and its dependency, which would ruin all the work > I did building numpy on my machine. > > This is probably a linux n00b question, but well, that's what I am :) You'll want to ask on an Ubuntu or Debian forum. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Fri Apr 24 22:38:23 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 24 Apr 2009 20:38:23 -0600 Subject: [Numpy-discussion] after building from source, how to register numpy with synaptic? In-Reply-To: <7f014ea60904241911t77c02fe7w4069bb92434e6421@mail.gmail.com> References: <7f014ea60904241911t77c02fe7w4069bb92434e6421@mail.gmail.com> Message-ID: On Fri, Apr 24, 2009 at 8:11 PM, Chris Colbert wrote: > Like the subject says, is there a way to register numpy with synaptic after > building numpy from source? > > I would like to snag matplotlib from the ubuntu repos, but it won't let me > without also getting numpy and its dependency, which would ruin all the work > I did building numpy on my machine. > Why not build matplotlib from source? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Apr 24 22:42:14 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 24 Apr 2009 22:42:14 -0400 Subject: [Numpy-discussion] Idea: fractional polynomial class In-Reply-To: <9457e7c80904241842l3e1df274u80f5f5febdf0381@mail.gmail.com> References: <20090424125233.b067f24d.michael.s.gilbert@gmail.com> <9457e7c80904241842l3e1df274u80f5f5febdf0381@mail.gmail.com> Message-ID: <1cd32cbb0904241942m76858606v27f67e13b0625dbe@mail.gmail.com> 2009/4/24 St?fan van der Walt : > 2009/4/24 Pauli Virtanen : >> polyint fixed in r6883, there doesn't seem anything else to fix: >> >>>>> import numpy as np >>>>> from fractions import Fraction >>>>> a = np.poly1d([Fraction(4,19), Fraction(3,19), Fraction(2,19), 0]) > > Very cool. ?I think we should mention thi example in the docs as well? > >>> from fractions import Fraction Traceback (most recent call last): File "", line 1, in from fractions import Fraction ImportError: No module named fractions where does fractions come from? Josef From robert.kern at gmail.com Fri Apr 24 22:43:31 2009 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 24 Apr 2009 21:43:31 -0500 Subject: [Numpy-discussion] Idea: fractional polynomial class In-Reply-To: <1cd32cbb0904241942m76858606v27f67e13b0625dbe@mail.gmail.com> References: <20090424125233.b067f24d.michael.s.gilbert@gmail.com> <9457e7c80904241842l3e1df274u80f5f5febdf0381@mail.gmail.com> <1cd32cbb0904241942m76858606v27f67e13b0625dbe@mail.gmail.com> Message-ID: <3d375d730904241943i3573feb4rb76fcca3eb474523@mail.gmail.com> On Fri, Apr 24, 2009 at 21:42, wrote: > 2009/4/24 St?fan van der Walt : >> 2009/4/24 Pauli Virtanen : >>> polyint fixed in r6883, there doesn't seem anything else to fix: >>> >>>>>> import numpy as np >>>>>> from fractions import Fraction >>>>>> a = np.poly1d([Fraction(4,19), Fraction(3,19), Fraction(2,19), 0]) >> >> Very cool. ?I think we should mention thi example in the docs as well? >> > >>>> from fractions import Fraction > Traceback (most recent call last): > ?File "", line 1, in > ? ?from fractions import Fraction > ImportError: No module named fractions > > where does fractions come from? Python 2.6. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From josef.pktd at gmail.com Fri Apr 24 22:46:29 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 24 Apr 2009 22:46:29 -0400 Subject: [Numpy-discussion] Idea: fractional polynomial class In-Reply-To: <3d375d730904241943i3573feb4rb76fcca3eb474523@mail.gmail.com> References: <20090424125233.b067f24d.michael.s.gilbert@gmail.com> <9457e7c80904241842l3e1df274u80f5f5febdf0381@mail.gmail.com> <1cd32cbb0904241942m76858606v27f67e13b0625dbe@mail.gmail.com> <3d375d730904241943i3573feb4rb76fcca3eb474523@mail.gmail.com> Message-ID: <1cd32cbb0904241946r3ee71100n63bf422f960d4be5@mail.gmail.com> On Fri, Apr 24, 2009 at 10:43 PM, Robert Kern wrote: > On Fri, Apr 24, 2009 at 21:42, ? wrote: >> 2009/4/24 St?fan van der Walt : >>> 2009/4/24 Pauli Virtanen : >>>> polyint fixed in r6883, there doesn't seem anything else to fix: >>>> >>>>>>> import numpy as np >>>>>>> from fractions import Fraction >>>>>>> a = np.poly1d([Fraction(4,19), Fraction(3,19), Fraction(2,19), 0]) >>> >>> Very cool. ?I think we should mention thi example in the docs as well? >>> >> >>>>> from fractions import Fraction >> Traceback (most recent call last): >> ?File "", line 1, in >> ? ?from fractions import Fraction >> ImportError: No module named fractions >> >> where does fractions come from? > > Python 2.6. > I should have read the details of the original message. "Since python 2.6 introduced a "fractions" class" I'm not so far yet. Josef From dwf at cs.toronto.edu Fri Apr 24 23:51:06 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Fri, 24 Apr 2009 23:51:06 -0400 Subject: [Numpy-discussion] after building from source, how to register numpy with synaptic? In-Reply-To: <7f014ea60904241911t77c02fe7w4069bb92434e6421@mail.gmail.com> References: <7f014ea60904241911t77c02fe7w4069bb92434e6421@mail.gmail.com> Message-ID: <4BCB226E-F611-40F7-A8C5-2E62731B03E6@cs.toronto.edu> On 24-Apr-09, at 10:11 PM, Chris Colbert wrote: > Like the subject says, is there a way to register numpy with > synaptic after building numpy from source? > > I would like to snag matplotlib from the ubuntu repos, but it won't > let me without also getting numpy and its dependency, which would > ruin all the work I did building numpy on my machine. I'd recommend building MPL from source too, as every MPL release makes leaps and bounds and the packages are likely a bit stale. But most package managers allow you to force installs without checking dependencies, as in "I know what I'm doing, just install it". With apt-get I believe you'd have to download the .deb package and do something like sudo dpkg -i --force-depends matplotlib.deb David From gael.varoquaux at normalesup.org Sat Apr 25 05:36:52 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 25 Apr 2009 11:36:52 +0200 Subject: [Numpy-discussion] after building from source, how to register numpy with synaptic? In-Reply-To: <7f014ea60904241911t77c02fe7w4069bb92434e6421@mail.gmail.com> References: <7f014ea60904241911t77c02fe7w4069bb92434e6421@mail.gmail.com> Message-ID: <20090425093652.GD25700@phare.normalesup.org> On Fri, Apr 24, 2009 at 10:11:07PM -0400, Chris Colbert wrote: > Like the subject says, is there a way to register numpy with synaptic > after building numpy from source? Don't play with the system's packaging system unless you really know what you are doing. Just install the numpy you are building outside of /usr/lib/... (you should never be installing home-build stuff in there). For instance install it in /usr/local: sudo python setup.py install --prefix /usr/local Now it will override the system's numpy. So you can install matplotlib, which will drag along the system's numpy, but you won't see it. On a side note, I tend to install home-built packages that overide system packages only in my home. I have a $HOME/usr directory, with a small directory hierarchy (usr/lib, usr/bin, ...), it is added in my PATH and PYTHONPATH, and I install there. Ga?l From pgmdevlist at gmail.com Sat Apr 25 12:33:33 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Sat, 25 Apr 2009 12:33:33 -0400 Subject: [Numpy-discussion] after building from source, how to register numpy with synaptic? In-Reply-To: <20090425093652.GD25700@phare.normalesup.org> References: <7f014ea60904241911t77c02fe7w4069bb92434e6421@mail.gmail.com> <20090425093652.GD25700@phare.normalesup.org> Message-ID: On Apr 25, 2009, at 5:36 AM, Gael Varoquaux wrote: > On Fri, Apr 24, 2009 at 10:11:07PM -0400, Chris Colbert wrote: >> Like the subject says, is there a way to register numpy with >> synaptic >> after building numpy from source? > > Don't play with the system's packaging system unless you really know > what > you are doing. > > Just install the numpy you are building outside of /usr/lib/... (you > should never be installing home-build stuff in there). One link: http://www.doughellmann.com/projects/virtualenvwrapper/ I became a fan of virtualenvs, which lets you install different packages (not always compatible) without messing up the system's Python. Quite useful for tests and/or having multiple numpy versions in parallel. > For instance > install it in /usr/local: > > sudo python setup.py install --prefix /usr/local > > Now it will override the system's numpy. So you can install > matplotlib, > which will drag along the system's numpy, but you won't see it. > > On a side note, I tend to install home-built packages that overide > system > packages only in my home. I have a $HOME/usr directory, with a small > directory hierarchy (usr/lib, usr/bin, ...), it is added in my PATH > and > PYTHONPATH, and I install there. > > Ga?l > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From geometrian at gmail.com Sat Apr 25 14:50:11 2009 From: geometrian at gmail.com (Ian Mallett) Date: Sat, 25 Apr 2009 11:50:11 -0700 Subject: [Numpy-discussion] Distance Formula on an Array Message-ID: Hi, I have an array sized n*3. Each three-component is a 3D position. Given another 3D position, how is the distance between it and every three-component in the array found with NumPy? So, for example, if the array is: [[0,0,0],[0,1,0],[0,0,3]] And the position is: [0,4,0] I need this array out: [4,3,5] (Just a simple Pythagorean Distance Formula) Ideas? Thanks, Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Apr 25 15:57:13 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 25 Apr 2009 13:57:13 -0600 Subject: [Numpy-discussion] Distance Formula on an Array In-Reply-To: References: Message-ID: On Sat, Apr 25, 2009 at 12:50 PM, Ian Mallett wrote: > Hi, > > I have an array sized n*3. Each three-component is a 3D position. Given > another 3D position, how is the distance between it and every > three-component in the array found with NumPy? > > So, for example, if the array is: > [[0,0,0],[0,1,0],[0,0,3]] > And the position is: > [0,4,0] > I need this array out: > [4,3,5] > (Just a simple Pythagorean Distance Formula) > In [3]: vec = array([[0,0,0],[0,1,0],[0,0,3]]) In [4]: pos = array([0,4,0]) In [5]: sqrt(((vec - pos)**2).sum(1)) Out[5]: array([ 4., 3., 5.]) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Apr 25 16:00:41 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 25 Apr 2009 16:00:41 -0400 Subject: [Numpy-discussion] Distance Formula on an Array In-Reply-To: References: Message-ID: <1cd32cbb0904251300r7c83a0ear380a056a327f823f@mail.gmail.com> On Sat, Apr 25, 2009 at 3:57 PM, Charles R Harris wrote: > > > On Sat, Apr 25, 2009 at 12:50 PM, Ian Mallett wrote: >> >> Hi, >> >> I have an array sized n*3.? Each three-component is a 3D position.? Given >> another 3D position, how is the distance between it and every >> three-component in the array found with NumPy? >> >> So, for example, if the array is: >> [[0,0,0],[0,1,0],[0,0,3]] >> And the position is: >> [0,4,0] >> I need this array out: >> [4,3,5] >> (Just a simple Pythagorean Distance Formula) > > In [3]: vec = array([[0,0,0],[0,1,0],[0,0,3]]) > > In [4]: pos = array([0,4,0]) > > In [5]: sqrt(((vec - pos)**2).sum(1)) > Out[5]: array([ 4.,? 3.,? 5.]) > if scipy is permitted: >>> a = np.array([[0,0,0],[0,1,0],[0,0,3]]) >>> scipy.spatial.distance_matrix(a, [[0,4,0]]) array([[ 4.], [ 3.], [ 5.]]) >>> scipy.spatial.minkowski_distance(a, [0,4,0]) array([ 4., 3., 5.]) Josef From geometrian at gmail.com Sat Apr 25 16:26:25 2009 From: geometrian at gmail.com (Ian Mallett) Date: Sat, 25 Apr 2009 13:26:25 -0700 Subject: [Numpy-discussion] Distance Formula on an Array In-Reply-To: <1cd32cbb0904251300r7c83a0ear380a056a327f823f@mail.gmail.com> References: <1cd32cbb0904251300r7c83a0ear380a056a327f823f@mail.gmail.com> Message-ID: On Sat, Apr 25, 2009 at 12:57 PM, Charles R Harris < charlesr.harris at gmail.com> wrote: > In [3]: vec = array([[0,0,0],[0,1,0],[0,0,3]]) > > In [4]: pos = array([0,4,0]) > > In [5]: sqrt(((vec - pos)**2).sum(1)) > Out[5]: array([ 4., 3., 5.]) > > Chuck > On Sat, Apr 25, 2009 at 1:00 PM, wrote: > if scipy is permitted: > > >>> a = np.array([[0,0,0],[0,1,0],[0,0,3]]) > >>> scipy.spatial.distance_matrix(a, [[0,4,0]]) > array([[ 4.], > [ 3.], > [ 5.]]) > >>> scipy.spatial.minkowski_distance(a, [0,4,0]) > array([ 4., 3., 5.]) > > Josef Thanks you two. I'm going to guess SciPy might be faster (?), but unfortunately it's not going to be available. Thanks, though. Problem solved. Thanks again, Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From geometrian at gmail.com Sat Apr 25 16:38:13 2009 From: geometrian at gmail.com (Ian Mallett) Date: Sat, 25 Apr 2009 13:38:13 -0700 Subject: [Numpy-discussion] Distance Formula on an Array In-Reply-To: References: <1cd32cbb0904251300r7c83a0ear380a056a327f823f@mail.gmail.com> Message-ID: Oops, one more thing. In reference to: vec = array([[0,0,0],[0,1,0],[0,0,3]]) pos = array([0,4,0]) sqrt(((vec - pos)**2).sum(1)) -> array([ 4., 3., 5.]) Can I make "vec" an array of class instances? I tried: class c: def __init__(self): self.position = [0,0,0] vec = array([c(),c(),c()]) pos = array([0,4,0]) sqrt(((vec.position - pos)**2).sum(1)) Which doesn't work. I'm not familiar with class objects in arrays--how should they be referenced? Thanks again, Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Apr 25 18:01:21 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 25 Apr 2009 18:01:21 -0400 Subject: [Numpy-discussion] Distance Formula on an Array In-Reply-To: References: <1cd32cbb0904251300r7c83a0ear380a056a327f823f@mail.gmail.com> Message-ID: <1cd32cbb0904251501w2192b1e3pe1d7b192593dfdd5@mail.gmail.com> On Sat, Apr 25, 2009 at 4:38 PM, Ian Mallett wrote: > Oops, one more thing.? In reference to: > vec = array([[0,0,0],[0,1,0],[0,0,3]]) > pos = array([0,4,0]) > sqrt(((vec - pos)**2).sum(1)) -> array([ 4.,? 3.,? 5.]) > > Can I make "vec" an array of class instances?? I tried: > class c: > ??? def __init__(self): > ??????? self.position = [0,0,0] > vec = array([c(),c(),c()]) > pos = array([0,4,0]) > sqrt(((vec.position - pos)**2).sum(1)) > > Which doesn't work.? I'm not familiar with class objects in arrays--how > should they be referenced? I never work with object arrays. I think they will kill your performance if you need fast calculation for many positions. list comprehension works: print [sqrt(((v.position - pos)**2).sum()) for v in vec] I didn't find any other way either. Josef From geometrian at gmail.com Sun Apr 26 16:20:58 2009 From: geometrian at gmail.com (Ian Mallett) Date: Sun, 26 Apr 2009 13:20:58 -0700 Subject: [Numpy-discussion] Distance Formula on an Array In-Reply-To: <1cd32cbb0904251501w2192b1e3pe1d7b192593dfdd5@mail.gmail.com> References: <1cd32cbb0904251300r7c83a0ear380a056a327f823f@mail.gmail.com> <1cd32cbb0904251501w2192b1e3pe1d7b192593dfdd5@mail.gmail.com> Message-ID: Well, if it will kill performance, I'm afraid I can't do that. Thanks though. I think it's working now. Now that I have the 1D array of distances, I need the indices of those distances that are less than a number "d". what should I do to do that? Thanks, Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From geometrian at gmail.com Sun Apr 26 16:27:50 2009 From: geometrian at gmail.com (Ian Mallett) Date: Sun, 26 Apr 2009 13:27:50 -0700 Subject: [Numpy-discussion] Distance Formula on an Array In-Reply-To: References: <1cd32cbb0904251300r7c83a0ear380a056a327f823f@mail.gmail.com> <1cd32cbb0904251501w2192b1e3pe1d7b192593dfdd5@mail.gmail.com> Message-ID: It would be: numpy.where(array From efiring at hawaii.edu Sun Apr 26 16:49:24 2009 From: efiring at hawaii.edu (Eric Firing) Date: Sun, 26 Apr 2009 10:49:24 -1000 Subject: [Numpy-discussion] Distance Formula on an Array In-Reply-To: References: <1cd32cbb0904251300r7c83a0ear380a056a327f823f@mail.gmail.com> <1cd32cbb0904251501w2192b1e3pe1d7b192593dfdd5@mail.gmail.com> Message-ID: <49F4C8D4.2010000@hawaii.edu> Ian Mallett wrote: > It would be: > numpy.where(array right? Almost. where(cond) is equivalent to nonzero(cond), and both return tuples. Assuming your array is 1-D, you can use: numpy.where(array References: <1cd32cbb0904251300r7c83a0ear380a056a327f823f@mail.gmail.com> <1cd32cbb0904251501w2192b1e3pe1d7b192593dfdd5@mail.gmail.com> <49F4C8D4.2010000@hawaii.edu> Message-ID: Yeah, I ended up finding the [0] bit at the end through trial and error. I actually do need the indices, though. I'm having a strange new problem though. numpy.array([1,2,3]) is returning a sequence??? I'm really confused. Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From geometrian at gmail.com Sun Apr 26 17:04:10 2009 From: geometrian at gmail.com (Ian Mallett) Date: Sun, 26 Apr 2009 14:04:10 -0700 Subject: [Numpy-discussion] Distance Formula on an Array In-Reply-To: References: <1cd32cbb0904251300r7c83a0ear380a056a327f823f@mail.gmail.com> <1cd32cbb0904251501w2192b1e3pe1d7b192593dfdd5@mail.gmail.com> <49F4C8D4.2010000@hawaii.edu> Message-ID: Hmmm, I played around with some other code, and it's working right now--not sure what I did... -------------- next part -------------- An HTML attachment was scrubbed... URL: From eike.welk at gmx.net Sun Apr 26 16:56:08 2009 From: eike.welk at gmx.net (Eike Welk) Date: Sun, 26 Apr 2009 22:56:08 +0200 Subject: [Numpy-discussion] Distance Formula on an Array In-Reply-To: References: Message-ID: <200904262256.09004.eike.welk@gmx.net> Hello Ian! On Saturday 25 April 2009, Ian Mallett wrote: > Can I make "vec" an array of class instances? I tried: > class c: > def __init__(self): > self.position = [0,0,0] > vec = array([c(),c(),c()]) > pos = array([0,4,0]) > sqrt(((vec.position - pos)**2).sum(1)) > > Which doesn't work. I'm not familiar with class objects in > arrays--how should they be referenced? You could use objects that don't store the positions but who know how to access them. The positions are stored in a Numpy array, the objects store only the index to the array. This is similar to the flyweight design pattern. Here is an example class: from numpy import array class MyThing(object): position_array = array([], 'd') def __init__(self, index): self.index = index def get_position(self): return MyThing.position_array[self.index] def set_position(self, new_val): MyThing.position_array[self.index] = new_val position = property(get_position, set_position, None, 'Position of MyThing, stored collectively for all objects in ' 'MyThing.position_array.') #--------- Use the MyThig class ------------------------------ #fill the array of positions with data MyThing.position_array = array([[0,0,0], [0,1,0], [0,0,3]], 'd') #create instance at index 1 thing1 = MyThing(1) print thing1.get_position() print thing1.position assert (thing1.position == array([0,1,0])).all() #create instance at index 2 thing2 = MyThing(2) print thing2.position assert (thing2.position == array([0,0,3])).all() thing2.position = array([1,2,3]) print thing2.position assert (thing2.position == array([1,2,3])).all() Kind regards, Eike. From geometrian at gmail.com Sun Apr 26 17:32:55 2009 From: geometrian at gmail.com (Ian Mallett) Date: Sun, 26 Apr 2009 14:32:55 -0700 Subject: [Numpy-discussion] Distance Formula on an Array In-Reply-To: <200904262256.09004.eike.welk@gmx.net> References: <200904262256.09004.eike.welk@gmx.net> Message-ID: Yes, this is pretty much what I'm doing. Right now, I'm having weird troubles with the objects themselves; the objects should and do terminate after a certain time, yet for some reason they're still being drawn. User error, I'm sure. Thanks, Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Apr 26 17:39:06 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 26 Apr 2009 15:39:06 -0600 Subject: [Numpy-discussion] Distance Formula on an Array In-Reply-To: References: <200904262256.09004.eike.welk@gmx.net> Message-ID: On Sun, Apr 26, 2009 at 3:32 PM, Ian Mallett wrote: > Yes, this is pretty much what I'm doing. Right now, I'm having weird > troubles with the objects themselves; the objects should and do terminate > after a certain time, yet for some reason they're still being drawn. User > error, I'm sure. > Thanks, It sounds like you might want to look into the spatial index in scipy. It can be used for finding neighbors of a given point. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From geometrian at gmail.com Sun Apr 26 17:41:47 2009 From: geometrian at gmail.com (Ian Mallett) Date: Sun, 26 Apr 2009 14:41:47 -0700 Subject: [Numpy-discussion] Distance Formula on an Array In-Reply-To: References: <1cd32cbb0904251300r7c83a0ear380a056a327f823f@mail.gmail.com> Message-ID: On Sat, Apr 25, 2009 at 1:26 PM, Ian Mallett wrote: > I'm going to guess SciPy might be faster (?), but unfortunately it's not > going to be available. Thanks, though. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From geometrian at gmail.com Sun Apr 26 17:54:32 2009 From: geometrian at gmail.com (Ian Mallett) Date: Sun, 26 Apr 2009 14:54:32 -0700 Subject: [Numpy-discussion] Distance Formula on an Array In-Reply-To: References: <1cd32cbb0904251300r7c83a0ear380a056a327f823f@mail.gmail.com> Message-ID: The problem is that the object moves too much between frames. A reasonable bounding sphere is 1 for this purpose, but the objects move at least 3. So, given the two arrays, one holding the objects' positions and the other their previous positions, how can I find if, at some point between, the objects collided with the bounding sphere? Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From efiring at hawaii.edu Sun Apr 26 18:48:53 2009 From: efiring at hawaii.edu (Eric Firing) Date: Sun, 26 Apr 2009 12:48:53 -1000 Subject: [Numpy-discussion] Distance Formula on an Array In-Reply-To: References: <1cd32cbb0904251300r7c83a0ear380a056a327f823f@mail.gmail.com> <1cd32cbb0904251501w2192b1e3pe1d7b192593dfdd5@mail.gmail.com> <49F4C8D4.2010000@hawaii.edu> Message-ID: <49F4E4D5.30807@hawaii.edu> Ian Mallett wrote: > Yeah, I ended up finding the [0] bit at the end through trial and > error. I actually do need the indices, though. If you are not already doing so, I strongly recommend using ipython. It is enormously useful in accessing docstrings (type a function name followed by a question mark), source code (same, but with two question marks), listing namespaces, finding functions, testing, running the pdb debugger, etc. > > I'm having a strange new problem though. > numpy.array([1,2,3]) > is returning a sequence??? I'm really confused. In ipython: In [1]:import numpy as np In [2]:x = np.array([1,2,3]) In [3]:type(x) Out[3]: In [4]:x Out[4]:array([1, 2, 3]) The term "sequence" is a generality to include tuples, lists, ndarrays, and potentially other such ordered containers. But where are you seeing the ndarray instance created by numpy.array() referred to as a "sequence"? It's not wrong, but I can see why you might not expect it. Eric From geometrian at gmail.com Sun Apr 26 18:53:18 2009 From: geometrian at gmail.com (Ian Mallett) Date: Sun, 26 Apr 2009 15:53:18 -0700 Subject: [Numpy-discussion] Distance Formula on an Array In-Reply-To: <49F4E4D5.30807@hawaii.edu> References: <1cd32cbb0904251300r7c83a0ear380a056a327f823f@mail.gmail.com> <1cd32cbb0904251501w2192b1e3pe1d7b192593dfdd5@mail.gmail.com> <49F4C8D4.2010000@hawaii.edu> <49F4E4D5.30807@hawaii.edu> Message-ID: It was in an error code somewhere. I fixed the problem by messing around with it. I tried the following: a = numpy.array([1, 2, 3]) print a and it gave: [1, 2, 3] instead of: array([1, 2, 3]) Then there were errors about it being a sequence instead of an array somewhere else. Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From geometrian at gmail.com Sun Apr 26 19:17:06 2009 From: geometrian at gmail.com (Ian Mallett) Date: Sun, 26 Apr 2009 16:17:06 -0700 Subject: [Numpy-discussion] Distance Formula on an Array In-Reply-To: References: <1cd32cbb0904251501w2192b1e3pe1d7b192593dfdd5@mail.gmail.com> <49F4C8D4.2010000@hawaii.edu> <49F4E4D5.30807@hawaii.edu> Message-ID: Using http://mathworld.wolfram.com/Point-LineDistance3-Dimensional.html, I came up with: x0 = numpy.array(#point to collide with) x1 = #objects' positions x2 = #objects' previous positions numerator = numpy.sqrt((numpy.cross((x0-x1),(x0-x2))**2).sum(1)) denominator = numpy.sqrt(((x2-x1)**2).sum(1)) dist = numerator/denominator collided_indices = numpy.where(dist<1.0)[0] #1.0==radius Unfortunately, in some cases, it returns indices when it shouldn't--when the objects clearly don't penetrate the sphere. I also get the feeling this will give false positives if the line containing the vector intersects the sphere, but the vector itself does not, though I don't know. Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Mon Apr 27 04:53:54 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 27 Apr 2009 17:53:54 +0900 Subject: [Numpy-discussion] Splitting multiarray and umath into smaller files: ready for review Message-ID: <49F572A2.90406@ar.media.kyoto-u.ac.jp> Hi, Following the "linker script, smaller source files and symbols visibility" discussion, I have implemented something which seems to work reasonably well. The code is available on github for review: http://github.com/cournape/numpy/tree/split_comp This branch implements the following: - Both umath and multiarray sources are split into smaller files, in separate directories. - By default, they are still all included into one big .c file, in which case nothing much is changed. - The more interesting stuff happens when ENABLE_SEPARATE_COMPILATION is set to True in setup.py/setupscons.py file, in which case the files are actually compiled separately and link together at the end. This separate compilation mode uses some toolchain specific feature to avoid exporting symbols; I have tested it on mac os x, linux and windows (windows needs nothing as by default, no symbols are exported in dynamically loaded libraries). At some point, it should be enabled on platforms where it is known to work, I guess. One big advantage of separate compilation is the compilation/debug cycle which is much shorter (compilation time after modifying one file of multiarray goes from half a minute to a couple of seconds on my configuration). I split the file in multiarray following the C reference guide, as well as some logical units (one file for the mapping protocol, one file for iterators, one file for 'array constructors', etc....). There are still some things which do not belong where they should, but this can be improved later. There is a second branch which implements symbol export maps on a few platforms (darwin and linux for now), to specify exactly the symbols we want to export: http://github.com/cournape/numpy/tree/version_script This branch is much smaller, as it just implements a couple of functions for distutils. With this work, multiarray and umath now only export one function, and this can be easily extended to other extensions as well, cheers, David From stefan at sun.ac.za Mon Apr 27 06:59:44 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 27 Apr 2009 12:59:44 +0200 Subject: [Numpy-discussion] Splitting multiarray and umath into smaller files: ready for review In-Reply-To: <49F572A2.90406@ar.media.kyoto-u.ac.jp> References: <49F572A2.90406@ar.media.kyoto-u.ac.jp> Message-ID: <9457e7c80904270359q7bb63405i942caf41c8851ca9@mail.gmail.com> Hi David 2009/4/27 David Cournapeau : > ? ?Following the "linker script, smaller source files and symbols > visibility" discussion, I have implemented something which seems to work > reasonably well. The code is available on github for review: > > http://github.com/cournape/numpy/tree/split_comp I really like the new structure. Thank you for taking the time to reorganise the core. Regards St?fan From tim.whitcomb at nrlmry.navy.mil Mon Apr 27 15:12:05 2009 From: tim.whitcomb at nrlmry.navy.mil (Whitcomb, Mr. Tim) Date: Mon, 27 Apr 2009 12:12:05 -0700 Subject: [Numpy-discussion] MemoryError in numpy.test() on AIX installation In-Reply-To: <827183970904232244u20c87915x9fcf5992d2b0df6e@mail.gmail.com> References: <9457e7c80904232203o16c07253o5c05dbd2d2d01c74@mail.gmail.com> <827183970904232244u20c87915x9fcf5992d2b0df6e@mail.gmail.com> Message-ID: I'm trying to install NumPy 1.3.0 on an IBM Power6. I can build and install successfully, but when I try to execute the unit test suite it crashes with a MemoryError (with no additional information on the error - just the traceback). The failure occurs in test_roundtrip in numpy/lib/tests/test_format.py and the following snippet reproduces the error: ---------------------------------- import numpy as np from cStringIO import StringIO from numpy.lib import format f = StringIO() a = np.array([], dtype='object') format.write_array(f, a) f2 = StringIO(f.getvalue()) a2 = format.read_array(f2) print('%s %s' % (a, a2)) ----------------------------------- On the AIX/PPC install, this crashes with: Traceback (most recent call last): File "", line 1, in File "crash_numpy.py", line 10, in a2 = format.read_array(f2) File "$HOME/.local/python/lib/python2.5/site-packages/numpy/lib/format.py", line 362, in read_array array = cPickle.load(fp) MemoryError Trying this same bit of code on my Linux and Windows install seems to work just fine, and other tests using empty arrays but different dtypes work on all platforms. To further complicate issues, I do not have sysadmin rights on the machine in question, and I'm not entirely confident that Python itself was built properly (I've had to sort out some other issues as well). The version in use is: Python 2.5.2 (r252:60911, Sep 17 2008, 13:24:29) [C] on aix5 I'm new to AIX, Python, and Numpy so I'm trying to learn a lot very quickly. Any ideas for what could be causing this error? Any pointers on where I can look to get some more information to diagnose the problem? I'd be happy to provide any more data I can. Thanks, Tim [w] From myeates at jpl.nasa.gov Mon Apr 27 15:19:17 2009 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Mon, 27 Apr 2009 12:19:17 -0700 Subject: [Numpy-discussion] help with vectorization Message-ID: <49F60535.1040000@jpl.nasa.gov> I know this must be trivial but I can't seem to get it right I have N 2x2 arrays which perform a rotation. I also have N xy pairs to transpose. What is the simplest way to perform the transformation without looping? Thanks from someone about to punch their screen. From nouiz at nouiz.org Mon Apr 27 15:21:16 2009 From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=) Date: Mon, 27 Apr 2009 15:21:16 -0400 Subject: [Numpy-discussion] inplace matrix multiplication In-Reply-To: References: <2d1d7fe70901090625k53fe9415uf177e2d80645f12c@mail.gmail.com> <3d375d730901091331u26a0d6dcw8dffca7349fec96@mail.gmail.com> Message-ID: <2d1d7fe70904271221s12ff78b6n2d4bc9c59006bf7f@mail.gmail.com> You are right, some here told me about this. I should have posted it here for reference. thanks Fred On Fri, Apr 24, 2009 at 1:45 PM, David Warde-Farley wrote: > On 9-Jan-09, at 4:31 PM, Robert Kern wrote: > > > You can't in numpy. With scipy.linalg.fblas.dgemm() and the right > > arguments, you can. > > Make sure your output array is Fortran-ordered, however, otherwise > copies will be made. > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From myeates at jpl.nasa.gov Mon Apr 27 15:24:00 2009 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Mon, 27 Apr 2009 12:24:00 -0700 Subject: [Numpy-discussion] help with vectorization In-Reply-To: <49F60535.1040000@jpl.nasa.gov> References: <49F60535.1040000@jpl.nasa.gov> Message-ID: <49F60650.9080606@jpl.nasa.gov> I should add, I'm starting with N rotation angles. So I should rephrase and say I'm starting with N angles and N xy pairs. Mathew Yeates wrote: > I know this must be trivial but I can't seem to get it right > > I have N 2x2 arrays which perform a rotation. I also have N xy pairs to > transpose. What is the simplest way to perform the transformation > without looping? > > Thanks from someone about to punch their screen. > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Mon Apr 27 15:39:54 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 27 Apr 2009 13:39:54 -0600 Subject: [Numpy-discussion] MemoryError in numpy.test() on AIX installation In-Reply-To: References: <9457e7c80904232203o16c07253o5c05dbd2d2d01c74@mail.gmail.com> <827183970904232244u20c87915x9fcf5992d2b0df6e@mail.gmail.com> Message-ID: On Mon, Apr 27, 2009 at 1:12 PM, Whitcomb, Mr. Tim < tim.whitcomb at nrlmry.navy.mil> wrote: > I'm trying to install NumPy 1.3.0 on an IBM Power6. I can build and > install successfully, but when I try to execute the unit test suite it > crashes with a MemoryError (with no additional information on the error > - just the traceback). The failure occurs in test_roundtrip in > numpy/lib/tests/test_format.py and the following snippet reproduces the > error: > > ---------------------------------- > import numpy as np > from cStringIO import StringIO > from numpy.lib import format > > f = StringIO() > a = np.array([], dtype='object') > format.write_array(f, a) > f2 = StringIO(f.getvalue()) > a2 = format.read_array(f2) > > print('%s %s' % (a, a2)) > ----------------------------------- > > On the AIX/PPC install, this crashes with: > Traceback (most recent call last): > File "", line 1, in > File "crash_numpy.py", line 10, in > a2 = format.read_array(f2) > File > "$HOME/.local/python/lib/python2.5/site-packages/numpy/lib/format.py", > line 362, in read_array > array = cPickle.load(fp) > MemoryError > I think this means out of memory, which is probably not correct. What happens if you simply pickle and object array? Something like In [1]: import cPickle In [2]: a = np.array([], dtype='object') In [3]: cPickle.dump(a,open('foo.pkl','w')) In [4]: b = cPickle.load(open('foo.pkl','r')) In [5]: b Out[5]: array([], dtype=object) If that fails, try it using pickle instead of cPickle, then with a non-empty object array like a = np.array([None,None], dtype='object') Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.whitcomb at nrlmry.navy.mil Mon Apr 27 15:59:10 2009 From: tim.whitcomb at nrlmry.navy.mil (Whitcomb, Mr. Tim) Date: Mon, 27 Apr 2009 12:59:10 -0700 Subject: [Numpy-discussion] MemoryError in numpy.test() on AIXinstallation In-Reply-To: References: <9457e7c80904232203o16c07253o5c05dbd2d2d01c74@mail.gmail.com><827183970904232244u20c87915x9fcf5992d2b0df6e@mail.gmail.com> Message-ID: >> ---------------------------------- >> import numpy as np >> from cStringIO import StringIO >> from numpy.lib import format >> >> f = StringIO() >> a = np.array([], dtype='object') >> format.write_array(f, a) >> f2 = StringIO(f.getvalue()) >> a2 = format.read_array(f2) >> >> print('%s %s' % (a, a2)) >> ----------------------------------- >> On the AIX/PPC install, this crashes with: >> Traceback (most recent call last): >> File "", line 1, in >> File "crash_numpy.py", line 10, in >> a2 = format.read_array(f2) >> File >> "$HOME/.local/python/lib/python2.5/site-packages/numpy/lib/format.py", >> line 362, in read_array >> array = cPickle.load(fp) >> MemoryError > I think this means out of memory, which is probably not correct. What happens if you simply pickle and object array? Something like > > If that fails, try it using pickle instead of cPickle, then with a non-empty object array like a = np.array([None,None], dtype='object') Just trying to pickle an empty object array: ------- pickle_file = 'foo.pkl' a = np.array([], dtype='object') cPickle.dump(a, open(pickle_file, 'w')) b = cPickle.load(open(pickle_file, 'r')) print(b) ------- Traceback (most recent call last): File "c_harris_test.py", line 7, in b = cPickle.load(open(pickle_file, 'r')) MemoryError Changing 'cPickle' to 'pickle' crashes again, with: Traceback (most recent call last): File "c_harris_test.py", line 7, in b = pickle.load(open(pickle_file, 'r')) File "/site/python-2.5.2/lib/python2.5/pickle.py", line 1370, in load return Unpickler(file).load() File "/site/python-2.5.2/lib/python2.5/pickle.py", line 858, in load dispatch[key](self) File "/site/python-2.5.2/lib/python2.5/pickle.py", line 1217, in load_build setstate(state) MemoryError If I try either pickle or cPickle with a non-empty object (e.g. np.array([None])), it works fine. Using np.array([ [] ]) crashes as well. If I take Numpy out of the loop, I can successfully pickle an empty list with no issues. Would platform endian-ness cause a problem? Tim From charlesr.harris at gmail.com Mon Apr 27 16:01:30 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 27 Apr 2009 14:01:30 -0600 Subject: [Numpy-discussion] help with vectorization In-Reply-To: <49F60650.9080606@jpl.nasa.gov> References: <49F60535.1040000@jpl.nasa.gov> <49F60650.9080606@jpl.nasa.gov> Message-ID: On Mon, Apr 27, 2009 at 1:24 PM, Mathew Yeates wrote: > I should add, I'm starting with N rotation angles. So I should rephrase > and say I'm starting with N angles and N xy pairs. > > > > Mathew Yeates wrote: > > I know this must be trivial but I can't seem to get it right > > > > I have N 2x2 arrays which perform a rotation. I also have N xy pairs to > > transpose. What is the simplest way to perform the transformation > > without looping? > > > Well, in two dimensions complex multiplication should do the trick. In [18]: v = array([1]*4, dtype=complex) In [19]: a = arange(4)*pi/2 In [20]: around(exp(1j*a)*v,5) Out[20]: array([ 1.+0.j, 0.+1.j, -1.+0.j, -0.-1.j]) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Apr 27 16:10:59 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 27 Apr 2009 14:10:59 -0600 Subject: [Numpy-discussion] MemoryError in numpy.test() on AIXinstallation In-Reply-To: References: <9457e7c80904232203o16c07253o5c05dbd2d2d01c74@mail.gmail.com> <827183970904232244u20c87915x9fcf5992d2b0df6e@mail.gmail.com> Message-ID: On Mon, Apr 27, 2009 at 1:59 PM, Whitcomb, Mr. Tim < tim.whitcomb at nrlmry.navy.mil> wrote: > >> ---------------------------------- > >> import numpy as np > >> from cStringIO import StringIO > >> from numpy.lib import format > >> > >> f = StringIO() > >> a = np.array([], dtype='object') > >> format.write_array(f, a) > >> f2 = StringIO(f.getvalue()) > >> a2 = format.read_array(f2) > >> > >> print('%s %s' % (a, a2)) > >> ----------------------------------- > > >> On the AIX/PPC install, this crashes with: > >> Traceback (most recent call last): > >> File "", line 1, in > >> File "crash_numpy.py", line 10, in > >> a2 = format.read_array(f2) > >> File > >> > "$HOME/.local/python/lib/python2.5/site-packages/numpy/lib/format.py", > >> line 362, in read_array > >> array = cPickle.load(fp) > >> MemoryError > > > I think this means out of memory, which is probably not correct. > What happens if you simply pickle and object array? Something like > > > > If that fails, try it using pickle instead of cPickle, then with > a non-empty object array like a = np.array([None,None], dtype='object') > > Just trying to pickle an empty object array: > ------- > pickle_file = 'foo.pkl' > a = np.array([], dtype='object') > cPickle.dump(a, open(pickle_file, 'w')) > b = cPickle.load(open(pickle_file, 'r')) > print(b) > ------- > Traceback (most recent call last): > File "c_harris_test.py", line 7, in > b = cPickle.load(open(pickle_file, 'r')) > MemoryError > > Changing 'cPickle' to 'pickle' crashes again, with: > Traceback (most recent call last): > File "c_harris_test.py", line 7, in > b = pickle.load(open(pickle_file, 'r')) > File "/site/python-2.5.2/lib/python2.5/pickle.py", line 1370, in load > return Unpickler(file).load() > File "/site/python-2.5.2/lib/python2.5/pickle.py", line 858, in load > dispatch[key](self) > File "/site/python-2.5.2/lib/python2.5/pickle.py", line 1217, in > load_build > setstate(state) > MemoryError > > If I try either pickle or cPickle with a non-empty object (e.g. > np.array([None])), it works fine. Using np.array([ [] ]) crashes as > well. If I take Numpy out of the loop, I can successfully pickle an > empty list with no issues. Would platform endian-ness cause a problem? > It could, but I think AIX on the PPC is big endian, as is OSX, and I would expect endian problems to show up in many other places, not just for empty arrays. Hmm... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.s.gilbert at gmail.com Mon Apr 27 17:04:17 2009 From: michael.s.gilbert at gmail.com (Michael S. Gilbert) Date: Mon, 27 Apr 2009 17:04:17 -0400 Subject: [Numpy-discussion] Some minor issues with numpy and fractions Message-ID: <20090427170417.301de0b0.michael.s.gilbert@gmail.com> I had mentioned recently some interest in using fractions in the numpy polynomial class. Suprisingly, it actually works for the most part out of the box, which is great. However, there are some minor issues. For example: >>>> numpy.poly1d( [ fractions.Fraction(1,2) , fractions.Fraction(1,8) ] )/fractions.Fraction(3,4) fails, so I have to move the division into the inner list, which leads to a lot of duplicate typing and general messiness: >>>> numpy.poly1d( [ fractions.Fraction(1,2)/fractions.Fraction(3,4) , fractions.Fraction(1,8)/fractions.Fraction(3,4) ] ) Another item of general interest for numpy is that there is no simple way to allocate memory for fraction array. >>>> def zeros( nsamples ): >>>> list = [] >>>> for n in range( nsamples ): >>>> list.append( fractions.Fraction( 0 ) ) >>>> return numpy.array( list ) is a solution, but it would be nice to have this built-in, e.g.: >>>> numpy.zeros( n , dtype=numpy.fraction ) Perhaps developing a fraction dtype would solve the above poly1d issue as well? Anyway, just some thoughts to ponder. Best wishes, Mike From wesmckinn at gmail.com Mon Apr 27 17:05:21 2009 From: wesmckinn at gmail.com (Wes McKinney) Date: Mon, 27 Apr 2009 17:05:21 -0400 Subject: [Numpy-discussion] Managing Python with NumPy and many external libraries on multiple Windows machines Message-ID: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> Hello, I am wondering if anyone can offer some suggestions on this problem. Over the last year or so I have been building a number of libraries on top of NumPy + SciPy + matplotlib and other libraries which are being used for investigative research for my company's problem domain in place of, say, Matlab and R (which are more "ready out of the box" systems). I have approximately 20 users, all of whom are running Windows on a very Microsoft-centric network with databases, etc. Has anyone had any luck managing a standardized Python environment on lots of Windows machines with a large number of 3rd-party Python libraries? Upgrading packages right now involves getting 20 people to click through an .exe installer, which is hardly a good solution. For example, I was recently forced to upgrade everyone's NumPy to 1.3 after I discovered that a DLL I had built against 1.3 was incompatible with 1.2.x. It seems like on a Unix / OS X environment with root access this problem would be fairly easy to script away, but on Windows it's rather painful. Any advice would be much appreciated. Thanks, Wes -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Apr 27 17:24:02 2009 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 27 Apr 2009 21:24:02 +0000 (UTC) Subject: [Numpy-discussion] Some minor issues with numpy and fractions References: <20090427170417.301de0b0.michael.s.gilbert@gmail.com> Message-ID: Mon, 27 Apr 2009 17:04:17 -0400, Michael S. Gilbert wrote: > I had mentioned recently some interest in using fractions in the numpy > polynomial class. Suprisingly, it actually works for the most part out > of the box, which is great. However, there are some minor issues. For > example: > > >>> numpy.poly1d( [ fractions.Fraction(1,2) , fractions.Fraction(1, 8) ] > ... )/fractions.Fraction(3,4) Polydiv can probably be fixed to handle this, it doesn't appear to be duck-typing clean in this case. > > >>> numpy.poly1d( [ fractions.Fraction(1,2)/fractions.Fraction(3,4) , > > >>> fractions.Fraction(1,8)/fractions.Fraction(3,4) ] ) > > Another item of general interest for numpy is that there is no simple > way to allocate memory for fraction array. Something similar came up a while ago. You can do this: >>> a = np.frompyfunc(Fraction, 1, 1)(np.zeros((50, 50), int)) and so >>> a[0,0] += Fraction(1, 3) >>> a /= 7 >>> a array([[1/21, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 0, 0, 0], ..., [0, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 0, 0, 0]], dtype=object) The trick is to create an universal function out of the object constructor, and feed it initializer data. Works for any classes. I'm undecided whether it'd be a good idea to add a specialized routine for doing this... -- Pauli Virtanen From michael.s.gilbert at gmail.com Mon Apr 27 17:25:19 2009 From: michael.s.gilbert at gmail.com (Michael S. Gilbert) Date: Mon, 27 Apr 2009 17:25:19 -0400 Subject: [Numpy-discussion] Some minor issues with numpy and fractions In-Reply-To: <20090427170417.301de0b0.michael.s.gilbert@gmail.com> References: <20090427170417.301de0b0.michael.s.gilbert@gmail.com> Message-ID: <20090427172519.a7732800.michael.s.gilbert@gmail.com> On Mon, 27 Apr 2009 17:04:17 -0400, Michael S. Gilbert wrote: > I had mentioned recently some interest in using fractions in the numpy > polynomial class. Suprisingly, it actually works for the most part out > of the box, which is great. However, there are some minor issues. For > example: > > >>>> numpy.poly1d( [ fractions.Fraction(1,2) , fractions.Fraction(1,8) ] )/fractions.Fraction(3,4) > > fails, so I have to move the division into the inner list, which leads > to a lot of duplicate typing and general messiness: > > >>>> numpy.poly1d( [ fractions.Fraction(1,2)/fractions.Fraction(3,4) , fractions.Fraction(1,8)/fractions.Fraction(3,4) ] ) > > Another item of general interest for numpy is that there is no simple > way to allocate memory for fraction array. > > >>>> def zeros( nsamples ): > >>>> list = [] > >>>> for n in range( nsamples ): > >>>> list.append( fractions.Fraction( 0 ) ) > >>>> return numpy.array( list ) > > is a solution, but it would be nice to have this built-in, e.g.: > > >>>> numpy.zeros( n , dtype=numpy.fraction ) > > Perhaps developing a fraction dtype would solve the above poly1d issue > as well? Anyway, just some thoughts to ponder. It also seems that numpy's basic math operations do not work with fractions: >>> math.sin(fractions.Fraction(1,2)) 0.47942553860420301 >>> numpy.sin(fractions.Fraction(1,2)) Traceback (most recent call last): File "", line 1, in AttributeError: sin >>> numpy.exp(fractions.Fraction(1,2)) Traceback (most recent call last): File "", line 1, in AttributeError: exp Mike From charlesr.harris at gmail.com Mon Apr 27 17:59:21 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 27 Apr 2009 15:59:21 -0600 Subject: [Numpy-discussion] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> Message-ID: On Mon, Apr 27, 2009 at 3:05 PM, Wes McKinney wrote: > Hello, > > I am wondering if anyone can offer some suggestions on this problem. Over > the last year or so I have been building a number of libraries on top of > NumPy + SciPy + matplotlib and other libraries which are being used for > investigative research for my company's problem domain in place of, say, > Matlab and R (which are more "ready out of the box" systems). I have > approximately 20 users, all of whom are running Windows on a very > Microsoft-centric network with databases, etc. Has anyone had any luck > managing a standardized Python environment on lots of Windows machines with > a large number of 3rd-party Python libraries? Upgrading packages right now > involves getting 20 people to click through an .exe installer, which is > hardly a good solution. For example, I was recently forced to upgrade > everyone's NumPy to 1.3 after I discovered that a DLL I had built against > 1.3 was incompatible with 1.2.x. > Out of curiosity, what was the nature of the incompatibility? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Apr 27 18:03:45 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 27 Apr 2009 16:03:45 -0600 Subject: [Numpy-discussion] Some minor issues with numpy and fractions In-Reply-To: <20090427172519.a7732800.michael.s.gilbert@gmail.com> References: <20090427170417.301de0b0.michael.s.gilbert@gmail.com> <20090427172519.a7732800.michael.s.gilbert@gmail.com> Message-ID: On Mon, Apr 27, 2009 at 3:25 PM, Michael S. Gilbert < michael.s.gilbert at gmail.com> wrote: > On Mon, 27 Apr 2009 17:04:17 -0400, Michael S. Gilbert wrote: > > I had mentioned recently some interest in using fractions in the numpy > > polynomial class. Suprisingly, it actually works for the most part out > > of the box, which is great. However, there are some minor issues. For > > example: > > > > >>>> numpy.poly1d( [ fractions.Fraction(1,2) , fractions.Fraction(1,8) ] > )/fractions.Fraction(3,4) > > > > fails, so I have to move the division into the inner list, which leads > > to a lot of duplicate typing and general messiness: > > > > >>>> numpy.poly1d( [ fractions.Fraction(1,2)/fractions.Fraction(3,4) , > fractions.Fraction(1,8)/fractions.Fraction(3,4) ] ) > > > > Another item of general interest for numpy is that there is no simple > > way to allocate memory for fraction array. > > > > >>>> def zeros( nsamples ): > > >>>> list = [] > > >>>> for n in range( nsamples ): > > >>>> list.append( fractions.Fraction( 0 ) ) > > >>>> return numpy.array( list ) > > > > is a solution, but it would be nice to have this built-in, e.g.: > > > > >>>> numpy.zeros( n , dtype=numpy.fraction ) > > > > Perhaps developing a fraction dtype would solve the above poly1d issue > > as well? Anyway, just some thoughts to ponder. > > It also seems that numpy's basic math operations do not work with > fractions: > > >>> math.sin(fractions.Fraction(1,2)) > 0.47942553860420301 > >>> numpy.sin(fractions.Fraction(1,2)) > Traceback (most recent call last): > File "", line 1, in > AttributeError: sin > >>> numpy.exp(fractions.Fraction(1,2)) > Traceback (most recent call last): > File "", line 1, in > AttributeError: exp > Yep. When objects are involved the ufuncs expect the functions to be object specific and defined as methods. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wesmckinn at gmail.com Mon Apr 27 19:36:27 2009 From: wesmckinn at gmail.com (Wes McKinney) Date: Mon, 27 Apr 2009 19:36:27 -0400 Subject: [Numpy-discussion] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> Message-ID: <6c476c8a0904271636r2f632ac2r46c3fc57a4e1d85d@mail.gmail.com> On Mon, Apr 27, 2009 at 5:59 PM, Charles R Harris wrote: > > > On Mon, Apr 27, 2009 at 3:05 PM, Wes McKinney wrote: > >> Hello, >> >> I am wondering if anyone can offer some suggestions on this problem. Over >> the last year or so I have been building a number of libraries on top of >> NumPy + SciPy + matplotlib and other libraries which are being used for >> investigative research for my company's problem domain in place of, say, >> Matlab and R (which are more "ready out of the box" systems). I have >> approximately 20 users, all of whom are running Windows on a very >> Microsoft-centric network with databases, etc. Has anyone had any luck >> managing a standardized Python environment on lots of Windows machines with >> a large number of 3rd-party Python libraries? Upgrading packages right now >> involves getting 20 people to click through an .exe installer, which is >> hardly a good solution. For example, I was recently forced to upgrade >> everyone's NumPy to 1.3 after I discovered that a DLL I had built against >> 1.3 was incompatible with 1.2.x. >> > > Out of curiosity, what was the nature of the incompatibility? > > Chuck > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > A Cython DLL using the NumPy include and buffer interface (which worked fine in 1.2.x, too) caused a hard crash on import, I wasn't able to diagnose further. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Apr 27 19:40:39 2009 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 27 Apr 2009 18:40:39 -0500 Subject: [Numpy-discussion] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: <6c476c8a0904271636r2f632ac2r46c3fc57a4e1d85d@mail.gmail.com> References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> <6c476c8a0904271636r2f632ac2r46c3fc57a4e1d85d@mail.gmail.com> Message-ID: <3d375d730904271640u512ed62exc527280ca6d158c5@mail.gmail.com> On Mon, Apr 27, 2009 at 18:36, Wes McKinney wrote: > On Mon, Apr 27, 2009 at 5:59 PM, Charles R Harris > wrote: >> >> On Mon, Apr 27, 2009 at 3:05 PM, Wes McKinney wrote: >>> >>> Hello, >>> >>> I am wondering if anyone can offer some suggestions on this problem. Over >>> the last year or so I have been building a number of libraries on top of >>> NumPy + SciPy + matplotlib and other libraries which are being used for >>> investigative research for my company's problem domain in place of, say, >>> Matlab and R (which are more "ready out of the box" systems). I have >>> approximately 20 users, all of whom are running Windows on a very >>> Microsoft-centric network with databases, etc. Has anyone had any luck >>> managing a standardized Python environment on lots of Windows machines with >>> a large number of 3rd-party Python libraries? Upgrading packages right now >>> involves getting 20 people to click through an .exe installer, which is >>> hardly a good solution. For example, I was recently forced to upgrade >>> everyone's NumPy to 1.3 after I discovered that a DLL I had built against >>> 1.3 was incompatible with 1.2.x. >> >> Out of curiosity, what was the nature of the incompatibility? >> >> Chuck > > A Cython DLL using the NumPy include and buffer interface (which worked fine > in 1.2.x, too) caused a hard crash on import, I wasn't able to diagnose > further. We have also encountered a number of segfaults upon import_array() on multiple platforms due to an extension module being built against numpy 1.3 but being imported after numpy 1.2. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Mon Apr 27 20:56:45 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 27 Apr 2009 18:56:45 -0600 Subject: [Numpy-discussion] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: <3d375d730904271640u512ed62exc527280ca6d158c5@mail.gmail.com> References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> <6c476c8a0904271636r2f632ac2r46c3fc57a4e1d85d@mail.gmail.com> <3d375d730904271640u512ed62exc527280ca6d158c5@mail.gmail.com> Message-ID: On Mon, Apr 27, 2009 at 5:40 PM, Robert Kern wrote: > On Mon, Apr 27, 2009 at 18:36, Wes McKinney wrote: > > On Mon, Apr 27, 2009 at 5:59 PM, Charles R Harris > > wrote: > >> > >> On Mon, Apr 27, 2009 at 3:05 PM, Wes McKinney > wrote: > >>> > >>> Hello, > >>> > >>> I am wondering if anyone can offer some suggestions on this problem. > Over > >>> the last year or so I have been building a number of libraries on top > of > >>> NumPy + SciPy + matplotlib and other libraries which are being used for > >>> investigative research for my company's problem domain in place of, > say, > >>> Matlab and R (which are more "ready out of the box" systems). I have > >>> approximately 20 users, all of whom are running Windows on a very > >>> Microsoft-centric network with databases, etc. Has anyone had any luck > >>> managing a standardized Python environment on lots of Windows machines > with > >>> a large number of 3rd-party Python libraries? Upgrading packages right > now > >>> involves getting 20 people to click through an .exe installer, which is > >>> hardly a good solution. For example, I was recently forced to upgrade > >>> everyone's NumPy to 1.3 after I discovered that a DLL I had built > against > >>> 1.3 was incompatible with 1.2.x. > >> > >> Out of curiosity, what was the nature of the incompatibility? > >> > >> Chuck > > > > A Cython DLL using the NumPy include and buffer interface (which worked > fine > > in 1.2.x, too) caused a hard crash on import, I wasn't able to diagnose > > further. > > We have also encountered a number of segfaults upon import_array() on > multiple platforms due to an extension module being built against > numpy 1.3 but being imported after numpy 1.2. > Hmm... OK, I think that is because of the endianess function added to the API. It is called during module load and since it isn't there for 1.2, crash. There is also a check for the API version that should have raised an error, but evidently not. I think the endianess check should probably be moved to testing and not be done during the load. Or it can simply be removed from the interface and/or be implemented inline. I don't think having it in the interface is that important. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Apr 27 21:28:08 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 28 Apr 2009 10:28:08 +0900 Subject: [Numpy-discussion] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> <6c476c8a0904271636r2f632ac2r46c3fc57a4e1d85d@mail.gmail.com> <3d375d730904271640u512ed62exc527280ca6d158c5@mail.gmail.com> Message-ID: <5b8d13220904271828j7be07d3aw14b7653a974e1e02@mail.gmail.com> On Tue, Apr 28, 2009 at 9:56 AM, Charles R Harris wrote: > Hmm... OK, I think that is because of the endianess function added to the > API. It is called during module load and since it isn't there for 1.2, > crash. There is also a check for the API version that should have raised an > error, but evidently not. Yes, this should have been detected. > I think the endianess check should probably be > moved to testing and not be done during the load. Or it can simply be > removed from the interface and/or be implemented inline. I don't think > having it in the interface is that important. I am not sure I understand: it is expected that building software A against B >= 1.3 cannot run with B < 1.3 (forward compatibility). It is the same as a runtime error when running something with libfoo 1.2 but linked against 1.3, using a function 1.3 specific. We should fix the undetected incompatibility, but I don't think we should care about the incompatibility itself. David From charlesr.harris at gmail.com Mon Apr 27 21:43:57 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 27 Apr 2009 19:43:57 -0600 Subject: [Numpy-discussion] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: <5b8d13220904271828j7be07d3aw14b7653a974e1e02@mail.gmail.com> References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> <6c476c8a0904271636r2f632ac2r46c3fc57a4e1d85d@mail.gmail.com> <3d375d730904271640u512ed62exc527280ca6d158c5@mail.gmail.com> <5b8d13220904271828j7be07d3aw14b7653a974e1e02@mail.gmail.com> Message-ID: On Mon, Apr 27, 2009 at 7:28 PM, David Cournapeau wrote: > On Tue, Apr 28, 2009 at 9:56 AM, Charles R Harris > wrote: > > > Hmm... OK, I think that is because of the endianess function added to the > > API. It is called during module load and since it isn't there for 1.2, > > crash. There is also a check for the API version that should have raised > an > > error, but evidently not. > > Yes, this should have been detected. > > > I think the endianess check should probably be > > moved to testing and not be done during the load. Or it can simply be > > removed from the interface and/or be implemented inline. I don't think > > having it in the interface is that important. > > I am not sure I understand: it is expected that building software A > against B >= 1.3 cannot run with B < 1.3 (forward compatibility). It > is the same as a runtime error when running something with libfoo 1.2 > but linked against 1.3, using a function 1.3 specific. > > We should fix the undetected incompatibility, but I don't think we > should care about the incompatibility itself. > I think you are right. So the API version check needs to be fixed. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Apr 27 21:57:36 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 27 Apr 2009 19:57:36 -0600 Subject: [Numpy-discussion] Splitting multiarray and umath into smaller files: ready for review In-Reply-To: <49F572A2.90406@ar.media.kyoto-u.ac.jp> References: <49F572A2.90406@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Apr 27, 2009 at 2:53 AM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Hi, > > Following the "linker script, smaller source files and symbols > visibility" discussion, I have implemented something which seems to work > reasonably well. The code is available on github for review: > > http://github.com/cournape/numpy/tree/split_comp > > This branch implements the following: > - Both umath and multiarray sources are split into smaller files, in > separate directories. > - By default, they are still all included into one big .c file, in > which case nothing much is changed. > - The more interesting stuff happens when > ENABLE_SEPARATE_COMPILATION is set to True in setup.py/setupscons.py > file, in which case the files are actually compiled separately and link > together at the end. > > This separate compilation mode uses some toolchain specific feature to > avoid exporting symbols; I have tested it on mac os x, linux and windows > (windows needs nothing as by default, no symbols are exported in > dynamically loaded libraries). At some point, it should be enabled on > platforms where it is known to work, I guess. One big advantage of > separate compilation is the compilation/debug cycle which is much > shorter (compilation time after modifying one file of multiarray goes > from half a minute to a couple of seconds on my configuration). I split > the file in multiarray following the C reference guide, as well as some > logical units (one file for the mapping protocol, one file for > iterators, one file for 'array constructors', etc....). There are still > some things which do not belong where they should, but this can be > improved later. > > There is a second branch which implements symbol export maps on a few > platforms (darwin and linux for now), to specify exactly the symbols we > want to export: > > http://github.com/cournape/numpy/tree/version_script > > This branch is much smaller, as it just implements a couple of functions > for distutils. With this work, multiarray and umath now only export one > function, and this can be easily extended to other extensions as well, > I think some of the fixes for too big arrays should be backported to 1.3.x before this is merged. That's r6851 and r6853. I'll do that. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Apr 27 22:17:01 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 27 Apr 2009 20:17:01 -0600 Subject: [Numpy-discussion] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> <6c476c8a0904271636r2f632ac2r46c3fc57a4e1d85d@mail.gmail.com> <3d375d730904271640u512ed62exc527280ca6d158c5@mail.gmail.com> <5b8d13220904271828j7be07d3aw14b7653a974e1e02@mail.gmail.com> Message-ID: On Mon, Apr 27, 2009 at 7:43 PM, Charles R Harris wrote: > > > On Mon, Apr 27, 2009 at 7:28 PM, David Cournapeau wrote: > >> On Tue, Apr 28, 2009 at 9:56 AM, Charles R Harris >> wrote: >> >> > Hmm... OK, I think that is because of the endianess function added to >> the >> > API. It is called during module load and since it isn't there for 1.2, >> > crash. There is also a check for the API version that should have raised >> an >> > error, but evidently not. >> >> Yes, this should have been detected. >> >> > I think the endianess check should probably be >> > moved to testing and not be done during the load. Or it can simply be >> > removed from the interface and/or be implemented inline. I don't think >> > having it in the interface is that important. >> >> I am not sure I understand: it is expected that building software A >> against B >= 1.3 cannot run with B < 1.3 (forward compatibility). It >> is the same as a runtime error when running something with libfoo 1.2 >> but linked against 1.3, using a function 1.3 specific. >> >> We should fix the undetected incompatibility, but I don't think we >> should care about the incompatibility itself. >> > > I think you are right. So the API version check needs to be fixed. > Looks like NPY_FEATURE_VERSION needs to be checked *and* updated. Unfortunately, it isn't part of the API, so can't be checked. That is unfortunate, but I think there is an ugly way to merge it with NPY_VERSION and fix the check, i.e., we can version the version number by setting the top bit ;) /* Binary compatibility version number. This number is increased whenever the C-API is changed such that binary compatibility is broken, i.e. whenever a recompile of extension modules is needed. */ #define NPY_VERSION 0x01000009 /* Minor API version. This number is increased whenever a change is made to the C-API -- whether it breaks binary compatibility or not. Some changes, such as adding a function pointer to the end of the function table, can be made without breaking binary compatibility. In this case, only the NPY_FEATURE_VERSION (*not* NPY_VERSION) would be increased. Whenever binary compatibility is broken, both NPY_VERSION and NPY_FEATURE_VERSION should be increased. */ #define NPY_FEATURE_VERSION 0x00000001 Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Apr 27 22:24:27 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 27 Apr 2009 20:24:27 -0600 Subject: [Numpy-discussion] MemoryError in numpy.test() on AIXinstallation In-Reply-To: References: <9457e7c80904232203o16c07253o5c05dbd2d2d01c74@mail.gmail.com> <827183970904232244u20c87915x9fcf5992d2b0df6e@mail.gmail.com> Message-ID: On Mon, Apr 27, 2009 at 2:10 PM, Charles R Harris wrote: > > > On Mon, Apr 27, 2009 at 1:59 PM, Whitcomb, Mr. Tim < > tim.whitcomb at nrlmry.navy.mil> wrote: > >> >> ---------------------------------- >> >> import numpy as np >> >> from cStringIO import StringIO >> >> from numpy.lib import format >> >> >> >> f = StringIO() >> >> a = np.array([], dtype='object') >> >> format.write_array(f, a) >> >> f2 = StringIO(f.getvalue()) >> >> a2 = format.read_array(f2) >> >> >> >> print('%s %s' % (a, a2)) >> >> ----------------------------------- >> >> >> On the AIX/PPC install, this crashes with: >> >> Traceback (most recent call last): >> >> File "", line 1, in >> >> File "crash_numpy.py", line 10, in >> >> a2 = format.read_array(f2) >> >> File >> >> >> "$HOME/.local/python/lib/python2.5/site-packages/numpy/lib/format.py", >> >> line 362, in read_array >> >> array = cPickle.load(fp) >> >> MemoryError >> >> > I think this means out of memory, which is probably not correct. >> What happens if you simply pickle and object array? Something like >> > >> > If that fails, try it using pickle instead of cPickle, then with >> a non-empty object array like a = np.array([None,None], dtype='object') >> >> Just trying to pickle an empty object array: >> ------- >> pickle_file = 'foo.pkl' >> a = np.array([], dtype='object') >> cPickle.dump(a, open(pickle_file, 'w')) >> b = cPickle.load(open(pickle_file, 'r')) >> print(b) >> ------- >> Traceback (most recent call last): >> File "c_harris_test.py", line 7, in >> b = cPickle.load(open(pickle_file, 'r')) >> MemoryError >> >> Changing 'cPickle' to 'pickle' crashes again, with: >> Traceback (most recent call last): >> File "c_harris_test.py", line 7, in >> b = pickle.load(open(pickle_file, 'r')) >> File "/site/python-2.5.2/lib/python2.5/pickle.py", line 1370, in load >> return Unpickler(file).load() >> File "/site/python-2.5.2/lib/python2.5/pickle.py", line 858, in load >> dispatch[key](self) >> File "/site/python-2.5.2/lib/python2.5/pickle.py", line 1217, in >> load_build >> setstate(state) >> MemoryError >> >> If I try either pickle or cPickle with a non-empty object (e.g. >> np.array([None])), it works fine. Using np.array([ [] ]) crashes as >> well. If I take Numpy out of the loop, I can successfully pickle an >> empty list with no issues. Would platform endian-ness cause a problem? >> > > It could, but I think AIX on the PPC is big endian, as is OSX, and I would > expect endian problems to show up in many other places, not just for empty > arrays. Hmm... > What compiler are you using? Any optimization flags? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Mon Apr 27 23:20:06 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 28 Apr 2009 12:20:06 +0900 Subject: [Numpy-discussion] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> Message-ID: <49F675E6.3030108@ar.media.kyoto-u.ac.jp> Wes McKinney wrote: > Hello, > > I am wondering if anyone can offer some suggestions on this problem. > Over the last year or so I have been building a number of libraries on > top of NumPy + SciPy + matplotlib and other libraries which are being > used for investigative research for my company's problem domain in > place of, say, Matlab and R (which are more "ready out of the box" > systems). I have approximately 20 users, all of whom are running > Windows on a very Microsoft-centric network with databases, etc. Has > anyone had any luck managing a standardized Python environment on lots > of Windows machines with a large number of 3rd-party Python libraries? > Upgrading packages right now involves getting 20 people to click > through an .exe installer, which is hardly a good solution. There is no good solution that I know of for python for this problem. Upgrades on windows and mac os x are usually handled on a per-application basis, but that requires that you control everything. Existing solutions (e.g. based on eggs) will require you to do something in any case. > For example, I was recently forced to upgrade everyone's NumPy to 1.3 > after I discovered that a DLL I had built against 1.3 was incompatible > with 1.2.x. The solution is to build against numpy 1.2.x. You can't expect to link against a library v2 and runs the application with v1 is loaded when v1 < v2. Very few, if any library handle forward compatibility. cheers, David From david at ar.media.kyoto-u.ac.jp Mon Apr 27 23:21:18 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 28 Apr 2009 12:21:18 +0900 Subject: [Numpy-discussion] MemoryError in numpy.test() on AIX installation In-Reply-To: References: <9457e7c80904232203o16c07253o5c05dbd2d2d01c74@mail.gmail.com> <827183970904232244u20c87915x9fcf5992d2b0df6e@mail.gmail.com> Message-ID: <49F6762E.6050301@ar.media.kyoto-u.ac.jp> Whitcomb, Mr. Tim wrote: > To further complicate issues, I do not have sysadmin rights on the > machine in question, and I'm not entirely confident that Python itself > was built properly (I've had to sort out some other issues as well). > The version in use is: > Python 2.5.2 (r252:60911, Sep 17 2008, 13:24:29) [C] on aix5 > Did you run the python test suite ? I would first check this, to make sure the problem is related to numpy and not python. David From dfnsonfsduifb at gmx.de Tue Apr 28 02:36:57 2009 From: dfnsonfsduifb at gmx.de (Johannes Bauer) Date: Tue, 28 Apr 2009 08:36:57 +0200 Subject: [Numpy-discussion] Creating a RGB-image from BW Message-ID: <49F6A409.6060306@gmx.de> Hello group, I've been redicted from usenet ("Convert numpy.ndarray into "normal" array", <75dgm1F16hqntU1 at mid.dfncis.de>) here and hope this is the right place. Basically, what I have is a numpy-Array which I got from a FITS-file (it's black/white). I want to display that using GTK. Therefore every color needs to appear three times (to make it look gray R = G = B). The basic framework looks like [...] pb = gtk.gdk.Pixbuf(gtk.gdk.COLORSPACE_RGB, False, 8, width, height) pb_pixels = pb.get_pixels_array() print(type(pb_pixels), pb_pixels.shape, pb_pixels.typecode()) print(type(fits_pixels), fits_pixels.shape, fits_pixels.dtype) which gives (, (480, 640, 3), 'b') (, (480, 640), dtype('uint8')) so now I need to assign values. Naively I started out with for x in range(width): for y in range(height): pb_pixels[y, x] = fits_pixels[y, x] which was horribly slow (around 3 seconds). Thanks to the usenet help, I now got somewhat better: fits_colors = numpy.zeros((height, width, 3), dtype="uint8") for y in range(height): for x in range(width): fits_colors[height - y - 1, x] = fits_pixels[y, x] pb_pixels[:, :] = fits_colors This also works, and is a lot faster (around 0.7 seconds). However, there seems to be a better way to do it. I played around with fits_colors = numpy.fromfunction(lambda y, x, z: fits_pixels[y, x], (height, width, 3), dtype="uint8") pb_pixels[:, :] = fits_colors Which worked somewhat - but gives weird results: The picture is rotatated 90? to the right and the lower left part is displayed repeatedly after 256 pixels... (I can make a screenshot if that's easier). The fromfunction Function is quite fast in my context (around 0.2 second). How should I solve this problem the right way? Kind regards, Johannes From zachary.pincus at yale.edu Tue Apr 28 07:47:38 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Tue, 28 Apr 2009 07:47:38 -0400 Subject: [Numpy-discussion] Creating a RGB-image from BW In-Reply-To: <49F6A409.6060306@gmx.de> References: <49F6A409.6060306@gmx.de> Message-ID: <01E7A471-B3B5-4CDC-9020-4C7755FE9A08@yale.edu> Hi Johannes, According to http://www.pygtk.org/pygtk2reference/class- gdkpixbuf.html , the pixels_array is a numeric python array (a predecessor to numpy). The upshot is that perhaps the nice broadcasting machinery will work fine: pb_pixels[...] = fits_pixels[..., numpy.newaxis] This might not work, though. But perhaps it would be possible to make a numpy array that's really just a view onto the memory of pb_pixels, or perhaps one could convert fits_pixels into a numeric array... Hopefully someone on the list can make a suggestion about dealing with numeric arrays. Alternately, there are pixbuf methods for reading image data from strings. You'd just need to get fits_pixels set up properly, then call tostring() on it, and pass that to the pixbuf. The trick is in setting up fits_pixels so that its memory layout corresponds to what gtk wants. Usually, images are stored in memory as (r,g,b) tuples packed by rows and then columns; this is I assume what GTK wants. So you'd do something like: fits_color = numpy.empty((height, width, 3), dtype=numpy.uint8) fits_color[...] = fits_pixels[..., numpy.newaxis] fits_string = fits_color.tostring() pb = gtk.gdk.pixbuf_new_from_data(fits_string, gtk.gdk.COLORSPACE_RGB, False, 8, 640, 480, 3*640) Zach On Apr 28, 2009, at 2:36 AM, Johannes Bauer wrote: > Hello group, > > I've been redicted from usenet ("Convert numpy.ndarray into "normal" > array", <75dgm1F16hqntU1 at mid.dfncis.de>) here and hope this is the > right > place. > > Basically, what I have is a numpy-Array which I got from a FITS-file > (it's black/white). I want to display that using GTK. Therefore every > color needs to appear three times (to make it look gray R = G = B). > > The basic framework looks like > > [...] > pb = gtk.gdk.Pixbuf(gtk.gdk.COLORSPACE_RGB, False, 8, width, height) > pb_pixels = pb.get_pixels_array() > > > print(type(pb_pixels), pb_pixels.shape, pb_pixels.typecode()) > print(type(fits_pixels), fits_pixels.shape, fits_pixels.dtype) > > which gives > > (, (480, 640, 3), 'b') > (, (480, 640), dtype('uint8')) > > so now I need to assign values. Naively I started out with > > for x in range(width): > for y in range(height): > pb_pixels[y, x] = fits_pixels[y, x] > > which was horribly slow (around 3 seconds). Thanks to the usenet > help, I > now got somewhat better: > > fits_colors = numpy.zeros((height, width, 3), dtype="uint8") > for y in range(height): > for x in range(width): > fits_colors[height - y - 1, x] = fits_pixels[y, x] > pb_pixels[:, :] = fits_colors > > This also works, and is a lot faster (around 0.7 seconds). However, > there seems to be a better way to do it. I played around with > > fits_colors = numpy.fromfunction(lambda y, x, z: fits_pixels[y, x], > (height, width, 3), dtype="uint8") > pb_pixels[:, :] = fits_colors > > Which worked somewhat - but gives weird results: The picture is > rotatated 90? to the right and the lower left part is displayed > repeatedly after 256 pixels... (I can make a screenshot if that's > easier). The fromfunction Function is quite fast in my context (around > 0.2 second). > > How should I solve this problem the right way? > > Kind regards, > Johannes > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From oliphant at enthought.com Tue Apr 28 10:58:29 2009 From: oliphant at enthought.com (Travis Oliphant) Date: Tue, 28 Apr 2009 09:58:29 -0500 Subject: [Numpy-discussion] [SciPy-user] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> Message-ID: On Apr 27, 2009, at 4:05 PM, Wes McKinney wrote: > Hello, > > I am wondering if anyone can offer some suggestions on this problem. > Over the last year or so I have been building a number of libraries > on top of NumPy + SciPy + matplotlib and other libraries which are > being used for investigative research for my company's problem > domain in place of, say, Matlab and R (which are more "ready out of > the box" systems). I have approximately 20 users, all of whom are > running Windows on a very Microsoft-centric network with databases, > etc. Has anyone had any luck managing a standardized Python > environment on lots of Windows machines with a large number of 3rd- > party Python libraries? Upgrading packages right now involves > getting 20 people to click through an .exe installer, which is > hardly a good solution. For example, I was recently forced to > upgrade everyone's NumPy to 1.3 after I discovered that a DLL I had > built against 1.3 was incompatible with 1.2.x. There has been a good discussion of this with others. I'm pretty sure we had fixed this in NumPy so at least you would get a sane error instead of a segfault. The solution may not be ideal and it can be hard to remember to edit the right place in the headers when a change is made so it may not have been done for 1.3 > > > It seems like on a Unix / OS X environment with root access this > problem would be fairly easy to script away, but on Windows it's > rather painful. Any advice would be much appreciated. This is a real problem we have encountered at Enthought as well. The Enthought Python Distribution solves this problem using a repository of eggs that the users have access to and a system that can update their machines on a regular schedule (i.e. nightly). I think others have done similar things (copy eggs into a directory and update the search path). This kind of large scale use and maintenance problem is something we are trying to solve with the Enthought Python Distribution. If anybody is interested in it, feel free to email or call me off line. EPD is more than a one-click installer. It's also a repo that can be locally mirrored and added to for your own distribution needs. Best regards, -Travis From dg.gmane at thesamovar.net Tue Apr 28 10:56:20 2009 From: dg.gmane at thesamovar.net (Dan Goodman) Date: Tue, 28 Apr 2009 16:56:20 +0200 Subject: [Numpy-discussion] inplace matrix multiplication In-Reply-To: <2d1d7fe70904271221s12ff78b6n2d4bc9c59006bf7f@mail.gmail.com> References: <2d1d7fe70901090625k53fe9415uf177e2d80645f12c@mail.gmail.com> <3d375d730901091331u26a0d6dcw8dffca7349fec96@mail.gmail.com> <2d1d7fe70904271221s12ff78b6n2d4bc9c59006bf7f@mail.gmail.com> Message-ID: Can anyone explain the results below? It seems that for small matrices dot(x,y) is outperforming dgemm(1,x,y,0,y,overwrite_c=1), but for larger matrices the latter is winning. In principle it seems like I ought to be able to always do better with inplace rather than making copies? From looking at the RAM used by python.exe it seems that indeed dot(x,y) is allocating lots of memory and making copies, and dgemm is not. My system is a Windows PC with a Pentium M processor 1.86GHz family 6 model 13 (which I think supports SSE2 but not SSE3), Python version 2.5, scipy version 0.7.0.dev5410 and numpy version 1.2.1. In [71]: x=array(randn(3,3), order='F') In [73]: n=1000 In [74]: y=array(randn(3,n), order='F') In [75]: y0=copy(y) In [76]: %timeit y[:]=y0[:]; dot(x,y) 10000 loops, best of 3: 48.5 ??s per loop In [77]: %timeit y[:]=y0[:]; dgemm(1,x,y,0,y,overwrite_c=1) 10000 loops, best of 3: 61.6 ??s per loop In [79]: n=100000 In [80]: y=array(randn(3,n), order='F') In [81]: y0=copy(y) In [82]: %timeit y[:]=y0[:]; dot(x,y) 10 loops, best of 3: 22.9 ms per loop In [83]: %timeit y[:]=y0[:]; dgemm(1,x,y,0,y,overwrite_c=1) 100 loops, best of 3: 8.37 ms per loop Dan Fr?d?ric Bastien wrote: > You are right, some here told me about this. I should have posted it > here for reference. > > thanks > > Fred > > On Fri, Apr 24, 2009 at 1:45 PM, David Warde-Farley > wrote: > > On 9-Jan-09, at 4:31 PM, Robert Kern wrote: > > > You can't in numpy. With scipy.linalg.fblas.dgemm() and the right > > arguments, you can. > > Make sure your output array is Fortran-ordered, however, otherwise > copies will be made. > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From cournape at gmail.com Tue Apr 28 11:41:43 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 29 Apr 2009 00:41:43 +0900 Subject: [Numpy-discussion] [SciPy-user] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> Message-ID: <5b8d13220904280841m5ddafe02nf766f02607ba7011@mail.gmail.com> On Tue, Apr 28, 2009 at 11:58 PM, Travis Oliphant wrote: > > On Apr 27, 2009, at 4:05 PM, Wes McKinney wrote: > >> Hello, >> >> I am wondering if anyone can offer some suggestions on this problem. >> Over the last year or so I have been building a number of libraries >> on top of NumPy + SciPy + matplotlib and other libraries which are >> being used for investigative research for my company's problem >> domain in place of, say, Matlab and R (which are more "ready out of >> the box" systems). I have approximately 20 users, all of whom are >> running Windows on a very Microsoft-centric network with databases, >> etc. Has anyone had any luck managing a standardized Python >> environment on lots of Windows machines with a large number of 3rd- >> party Python libraries? Upgrading packages right now involves >> getting 20 people to click through an .exe installer, which is >> hardly a good solution. For example, I was recently forced to >> upgrade everyone's NumPy to 1.3 after I discovered that a DLL I had >> built against 1.3 was incompatible with 1.2.x. > > There has been a good discussion of this with others. ? I'm pretty > sure we had fixed this in NumPy so at least you would get a sane error > instead of a segfault. ? The solution may not be ideal and it can be > hard to remember to edit the right place in the headers when a change > is made so it may not have been done for 1.3 Maybe we could supplement the manual editing: the code generators should guarantee the order of each item of the API, so the number of items in the array is enough to compare to API. So if we implement a guard in the PyArray_API, we could dynamically check the number of items in the array, and check it against the pre-computed number. A different number means incompatible API. David From charlesr.harris at gmail.com Tue Apr 28 11:45:11 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 28 Apr 2009 09:45:11 -0600 Subject: [Numpy-discussion] [SciPy-user] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: <5b8d13220904280841m5ddafe02nf766f02607ba7011@mail.gmail.com> References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> <5b8d13220904280841m5ddafe02nf766f02607ba7011@mail.gmail.com> Message-ID: On Tue, Apr 28, 2009 at 9:41 AM, David Cournapeau wrote: > On Tue, Apr 28, 2009 at 11:58 PM, Travis Oliphant > wrote: > > > > On Apr 27, 2009, at 4:05 PM, Wes McKinney wrote: > > > >> Hello, > >> > >> I am wondering if anyone can offer some suggestions on this problem. > >> Over the last year or so I have been building a number of libraries > >> on top of NumPy + SciPy + matplotlib and other libraries which are > >> being used for investigative research for my company's problem > >> domain in place of, say, Matlab and R (which are more "ready out of > >> the box" systems). I have approximately 20 users, all of whom are > >> running Windows on a very Microsoft-centric network with databases, > >> etc. Has anyone had any luck managing a standardized Python > >> environment on lots of Windows machines with a large number of 3rd- > >> party Python libraries? Upgrading packages right now involves > >> getting 20 people to click through an .exe installer, which is > >> hardly a good solution. For example, I was recently forced to > >> upgrade everyone's NumPy to 1.3 after I discovered that a DLL I had > >> built against 1.3 was incompatible with 1.2.x. > > > > There has been a good discussion of this with others. I'm pretty > > sure we had fixed this in NumPy so at least you would get a sane error > > instead of a segfault. The solution may not be ideal and it can be > > hard to remember to edit the right place in the headers when a change > > is made so it may not have been done for 1.3 > > Maybe we could supplement the manual editing: the code generators > should guarantee the order of each item of the API, so the number of > items in the array is enough to compare to API. So if we implement a > guard in the PyArray_API, we could dynamically check the number of > items in the array, and check it against the pre-computed number. A > different number means incompatible API. > That will work as long as we maintain backward compatibility. If we change the ABI at some point we will need something stronger. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Tue Apr 28 11:57:22 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 29 Apr 2009 00:57:22 +0900 Subject: [Numpy-discussion] [SciPy-user] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> <5b8d13220904280841m5ddafe02nf766f02607ba7011@mail.gmail.com> Message-ID: <5b8d13220904280857r2ea35d07yb14c8cc6052e92e@mail.gmail.com> On Wed, Apr 29, 2009 at 12:45 AM, Charles R Harris wrote: > > > On Tue, Apr 28, 2009 at 9:41 AM, David Cournapeau > wrote: >> >> On Tue, Apr 28, 2009 at 11:58 PM, Travis Oliphant >> wrote: >> > >> > On Apr 27, 2009, at 4:05 PM, Wes McKinney wrote: >> > >> >> Hello, >> >> >> >> I am wondering if anyone can offer some suggestions on this problem. >> >> Over the last year or so I have been building a number of libraries >> >> on top of NumPy + SciPy + matplotlib and other libraries which are >> >> being used for investigative research for my company's problem >> >> domain in place of, say, Matlab and R (which are more "ready out of >> >> the box" systems). I have approximately 20 users, all of whom are >> >> running Windows on a very Microsoft-centric network with databases, >> >> etc. Has anyone had any luck managing a standardized Python >> >> environment on lots of Windows machines with a large number of 3rd- >> >> party Python libraries? Upgrading packages right now involves >> >> getting 20 people to click through an .exe installer, which is >> >> hardly a good solution. For example, I was recently forced to >> >> upgrade everyone's NumPy to 1.3 after I discovered that a DLL I had >> >> built against 1.3 was incompatible with 1.2.x. >> > >> > There has been a good discussion of this with others. ? I'm pretty >> > sure we had fixed this in NumPy so at least you would get a sane error >> > instead of a segfault. ? The solution may not be ideal and it can be >> > hard to remember to edit the right place in the headers when a change >> > is made so it may not have been done for 1.3 >> >> Maybe we could supplement the manual editing: the code generators >> should guarantee the order of each item of the API, so the number of >> items in the array is enough to compare to API. So if we implement a >> guard in the PyArray_API, we could dynamically check the number of >> items in the array, and check it against the pre-computed number. A >> different number means incompatible API. > > That will work as long as we maintain backward compatibility. If we change > the ABI at some point we will need something stronger. I was only suggesting to supplement the current scheme. The current scheme is necessary in any case. But manually editing some macro in the header file is not very reliable. More generally, I think we should have a way to track C API and ABI relatively to the number version, as well as automatically generate the related documentation. While working on the split_comp branch, I notice several undocumented functions, several functions wrongly documented, etc... All this should be handled as automatically as possible. cheers, David From charlesr.harris at gmail.com Tue Apr 28 12:13:08 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 28 Apr 2009 10:13:08 -0600 Subject: [Numpy-discussion] [SciPy-user] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: <5b8d13220904280857r2ea35d07yb14c8cc6052e92e@mail.gmail.com> References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> <5b8d13220904280841m5ddafe02nf766f02607ba7011@mail.gmail.com> <5b8d13220904280857r2ea35d07yb14c8cc6052e92e@mail.gmail.com> Message-ID: On Tue, Apr 28, 2009 at 9:57 AM, David Cournapeau wrote: > On Wed, Apr 29, 2009 at 12:45 AM, Charles R Harris > wrote: > > > > > > On Tue, Apr 28, 2009 at 9:41 AM, David Cournapeau > > wrote: > >> > >> On Tue, Apr 28, 2009 at 11:58 PM, Travis Oliphant > >> wrote: > >> > > >> > On Apr 27, 2009, at 4:05 PM, Wes McKinney wrote: > >> > > >> >> Hello, > >> >> > >> >> I am wondering if anyone can offer some suggestions on this problem. > >> >> Over the last year or so I have been building a number of libraries > >> >> on top of NumPy + SciPy + matplotlib and other libraries which are > >> >> being used for investigative research for my company's problem > >> >> domain in place of, say, Matlab and R (which are more "ready out of > >> >> the box" systems). I have approximately 20 users, all of whom are > >> >> running Windows on a very Microsoft-centric network with databases, > >> >> etc. Has anyone had any luck managing a standardized Python > >> >> environment on lots of Windows machines with a large number of 3rd- > >> >> party Python libraries? Upgrading packages right now involves > >> >> getting 20 people to click through an .exe installer, which is > >> >> hardly a good solution. For example, I was recently forced to > >> >> upgrade everyone's NumPy to 1.3 after I discovered that a DLL I had > >> >> built against 1.3 was incompatible with 1.2.x. > >> > > >> > There has been a good discussion of this with others. I'm pretty > >> > sure we had fixed this in NumPy so at least you would get a sane error > >> > instead of a segfault. The solution may not be ideal and it can be > >> > hard to remember to edit the right place in the headers when a change > >> > is made so it may not have been done for 1.3 > >> > >> Maybe we could supplement the manual editing: the code generators > >> should guarantee the order of each item of the API, so the number of > >> items in the array is enough to compare to API. So if we implement a > >> guard in the PyArray_API, we could dynamically check the number of > >> items in the array, and check it against the pre-computed number. A > >> different number means incompatible API. > > > > That will work as long as we maintain backward compatibility. If we > change > > the ABI at some point we will need something stronger. > > I was only suggesting to supplement the current scheme. The current > scheme is necessary in any case. But manually editing some macro in > the header file is not very reliable. More generally, I think we > should have a way to track C API and ABI relatively to the number > version, as well as automatically generate the related documentation. > While working on the split_comp branch, I notice several undocumented > functions, several functions wrongly documented, etc... All this > should be handled as automatically as possible. > I don't disagree with that, something in a release script perhaps? The current NPY_VERSION number only tracks the ABI, so we need some fixups in any case. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Tue Apr 28 12:19:21 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 28 Apr 2009 18:19:21 +0200 Subject: [Numpy-discussion] [SciPy-user] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: <5b8d13220904280857r2ea35d07yb14c8cc6052e92e@mail.gmail.com> References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> <5b8d13220904280841m5ddafe02nf766f02607ba7011@mail.gmail.com> <5b8d13220904280857r2ea35d07yb14c8cc6052e92e@mail.gmail.com> Message-ID: <9457e7c80904280919i20e52f72n5b9c2e4ad72f55b@mail.gmail.com> 2009/4/28 David Cournapeau : > the header file is not very reliable. More generally, I think we > should have a way to track C API and ABI relatively to the number > version, as well as automatically generate the related documentation. We can already do this: simply choose a convention for NPY_VERSION and NPY_FEATURE_VERSION so that they are related. St?fan From tim.whitcomb at nrlmry.navy.mil Tue Apr 28 13:26:04 2009 From: tim.whitcomb at nrlmry.navy.mil (Whitcomb, Mr. Tim) Date: Tue, 28 Apr 2009 10:26:04 -0700 Subject: [Numpy-discussion] MemoryError in numpy.test() on AIXinstallation In-Reply-To: <49F6762E.6050301@ar.media.kyoto-u.ac.jp> References: <9457e7c80904232203o16c07253o5c05dbd2d2d01c74@mail.gmail.com> <827183970904232244u20c87915x9fcf5992d2b0df6e@mail.gmail.com> <49F6762E.6050301@ar.media.kyoto-u.ac.jp> Message-ID: > > To further complicate issues, I do not have sysadmin rights on the > > machine in question, and I'm not entirely confident that > Python itself > > was built properly (I've had to sort out some other issues as well). > > The version in use is: > > Python 2.5.2 (r252:60911, Sep 17 2008, 13:24:29) [C] on aix5 > > > > Did you run the python test suite ? I would first check this, > to make sure the problem is related to numpy and not python. I was trying to find out how to do this at the same time I sent the original message. If I run $ python /site/python-2.5.2/lib/python2.5/test/regrtest.py I get: 5 tests failed: test_cookielib test_mmap test_uuid test_wait4 test_zipfile 53 tests skipped: test_aepack test_al test_applesingle test_bsddb test_bsddb185 test_bsddb3 test_bz2 test_cd test_cl test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_ctypes test_curses test_dl test_gdbm test_gl test_gzip test_hashlib test_hmac test_imgfile test_largefile test_linuxaudiodev test_macfs test_macostools test_md5 test_nis test_normalization test_ossaudiodev test_pep247 test_pep277 test_plistlib test_scriptpackages test_sha test_socket_ssl test_socketserver test_sqlite test_startfile test_sunaudiodev test_tcl test_timeout test_unicodedata test_urllib2 test_urllib2_localnet test_urllib2net test_urllibnet test_winreg test_winsound test_zipfile64 test_zipimport test_zlib 12 skips unexpected on aix5: test_hashlib test_curses test_hmac test_urllib2net test_largefile test_ctypes test_urllib2 test_urllib2_localnet test_sha test_pep247 test_unicodedata test_md5 Most of the unexpected skips happen because of not being able to load the _md5 module, but the ctypes failure has me worried ("test_ctypes skipped -- No module named _ctypes"). I can see in the numpy source where it tries to import this library. Would missing this and/or the _md5 module cause the issue that I'm seeing? I've sent a note to the system administrators to try and get this fixed in the installation. Thanks, Tim [w] From bruce.sherwood at gmail.com Tue Apr 28 13:32:31 2009 From: bruce.sherwood at gmail.com (Bruce Sherwood) Date: Tue, 28 Apr 2009 13:32:31 -0400 Subject: [Numpy-discussion] numpy.int32 -> double? Message-ID: I'm not sure this is the right forum for my question; it's quite possible that this is a Boost question. VPython (vpython.org) is the name of Python plus the 3D Visual module, which is mostly written in C++ and connected to Python using the Boost libraries. Visual imports numpy. When I build Visual for Python 2.6 and numpy 1.3.0, I get a failure on something that works on Python 2.5 with numpy 1.2.1. There is a vector(double,double,double) class in Visual which now fails if handed numpy.int32 arguments (works fine with numpy.float64 arguments). I was using Boost 1.35 with Python 2.5 and am using Boost 1.38 with Python 2.6. Any ideas or suggestions? Thanks. Bruce Sherwood -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Apr 28 14:43:19 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 28 Apr 2009 12:43:19 -0600 Subject: [Numpy-discussion] [SciPy-user] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: <9457e7c80904280919i20e52f72n5b9c2e4ad72f55b@mail.gmail.com> References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> <5b8d13220904280841m5ddafe02nf766f02607ba7011@mail.gmail.com> <5b8d13220904280857r2ea35d07yb14c8cc6052e92e@mail.gmail.com> <9457e7c80904280919i20e52f72n5b9c2e4ad72f55b@mail.gmail.com> Message-ID: 2009/4/28 St?fan van der Walt > 2009/4/28 David Cournapeau : > > the header file is not very reliable. More generally, I think we > > should have a way to track C API and ABI relatively to the number > > version, as well as automatically generate the related documentation. > > We can already do this: simply choose a convention for NPY_VERSION and > NPY_FEATURE_VERSION so that they are related. > NPY_VERSION has been 0x01000009 since 1.0. NPY_FEATURE_VERSION was added for 1.2, but is useless since it can't be obtained through the API. I suggest we keep only NPY_VERSION but split it into two parts, the upper 16 bits for ABI changes that require extensions to be recompiled, and the lower 16 bits for API changes that are backward compatible, so that the next release should have version 0x0100000A. What to do with the current release is another question. If we change the version for a bugfix release it won't be compatible with 1.3.0. Hmm... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Tue Apr 28 15:44:19 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 28 Apr 2009 21:44:19 +0200 Subject: [Numpy-discussion] [SciPy-user] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> <5b8d13220904280841m5ddafe02nf766f02607ba7011@mail.gmail.com> <5b8d13220904280857r2ea35d07yb14c8cc6052e92e@mail.gmail.com> <9457e7c80904280919i20e52f72n5b9c2e4ad72f55b@mail.gmail.com> Message-ID: <9457e7c80904281244xc0313c3wdef5aca06dcc214@mail.gmail.com> 2009/4/28 Charles R Harris : > 2009/4/28 St?fan van der Walt >> We can already do this: simply choose a convention for NPY_VERSION and >> NPY_FEATURE_VERSION so that they are related. > > NPY_VERSION has been 0x01000009 since 1.0. NPY_FEATURE_VERSION was added for > 1.2, but is useless since it can't be obtained through the API. I suggest we > keep only NPY_VERSION but split it into two parts, the upper 16 bits for ABI > changes that require extensions to be recompiled, and the lower 16 bits for > API changes that are backward compatible, so that the next release should > have version 0x0100000A. What to do with the current release is another > question. If we change the version for a bugfix release it won't be > compatible with 1.3.0. Hmm... The current system does exactly this. The ABI compatibility is tracked in NPY_VERSION and the C API version is tracked in NPY_FEATURE_VERSION. NPY_VERSION is only visible where it is needed -- during compilation of extensions. Regards St?fan From charlesr.harris at gmail.com Tue Apr 28 16:00:34 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 28 Apr 2009 14:00:34 -0600 Subject: [Numpy-discussion] [SciPy-user] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: <9457e7c80904281244xc0313c3wdef5aca06dcc214@mail.gmail.com> References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> <5b8d13220904280841m5ddafe02nf766f02607ba7011@mail.gmail.com> <5b8d13220904280857r2ea35d07yb14c8cc6052e92e@mail.gmail.com> <9457e7c80904280919i20e52f72n5b9c2e4ad72f55b@mail.gmail.com> <9457e7c80904281244xc0313c3wdef5aca06dcc214@mail.gmail.com> Message-ID: 2009/4/28 St?fan van der Walt > 2009/4/28 Charles R Harris : > > 2009/4/28 St?fan van der Walt > >> We can already do this: simply choose a convention for NPY_VERSION and > >> NPY_FEATURE_VERSION so that they are related. > > > > NPY_VERSION has been 0x01000009 since 1.0. NPY_FEATURE_VERSION was added > for > > 1.2, but is useless since it can't be obtained through the API. I suggest > we > > keep only NPY_VERSION but split it into two parts, the upper 16 bits for > ABI > > changes that require extensions to be recompiled, and the lower 16 bits > for > > API changes that are backward compatible, so that the next release should > > have version 0x0100000A. What to do with the current release is another > > question. If we change the version for a bugfix release it won't be > > compatible with 1.3.0. Hmm... > > The current system does exactly this. The ABI compatibility is tracked > in NPY_VERSION and the C API version is tracked in > NPY_FEATURE_VERSION. NPY_VERSION is only visible where it is needed > -- during compilation of extensions. It is needed when the extension is loaded, that is why the very first function in the API returns it. Otherwise it is impossible to check if extensions compiled against one version of numpy can be loaded when another version of numpy is present. You need to know what API numpy presents and that can be different than the API the extension was compiled against. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Tue Apr 28 16:26:25 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 28 Apr 2009 22:26:25 +0200 Subject: [Numpy-discussion] [SciPy-user] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> <5b8d13220904280841m5ddafe02nf766f02607ba7011@mail.gmail.com> <5b8d13220904280857r2ea35d07yb14c8cc6052e92e@mail.gmail.com> <9457e7c80904280919i20e52f72n5b9c2e4ad72f55b@mail.gmail.com> <9457e7c80904281244xc0313c3wdef5aca06dcc214@mail.gmail.com> Message-ID: <9457e7c80904281326p18616b78j49894aaf51bcab07@mail.gmail.com> 2009/4/28 Charles R Harris : > It is needed when the extension is loaded, that is why the very first > function in the API returns it. Otherwise it is impossible to check if > extensions compiled against one version of numpy can be loaded when another > version of numpy is present. You need to know what API numpy presents and > that can be different than the API the extension was compiled against. Right, I phrased myself incorrectly. NPY_VERSION is checked at runtime to make sure modules compiled against a different ABI are not imported. These defines used to be closer to the top of the header file, but now a lot of other stuff has been added. Maybe it would be good to include this as a separate header file, npy_version.h or npy_api.h. IIRC, we can expose NPY_FEATURE_VERSION as part of the API without breaking ABI compatibility, as long as we add it at the end of the API functions list. As a hack, we can then check the length of the API functions list to make sure it is available before we call it. Cheers St?fan From charlesr.harris at gmail.com Tue Apr 28 17:16:25 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 28 Apr 2009 15:16:25 -0600 Subject: [Numpy-discussion] [SciPy-user] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: <9457e7c80904281326p18616b78j49894aaf51bcab07@mail.gmail.com> References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> <5b8d13220904280841m5ddafe02nf766f02607ba7011@mail.gmail.com> <5b8d13220904280857r2ea35d07yb14c8cc6052e92e@mail.gmail.com> <9457e7c80904280919i20e52f72n5b9c2e4ad72f55b@mail.gmail.com> <9457e7c80904281244xc0313c3wdef5aca06dcc214@mail.gmail.com> <9457e7c80904281326p18616b78j49894aaf51bcab07@mail.gmail.com> Message-ID: 2009/4/28 St?fan van der Walt > 2009/4/28 Charles R Harris : > > It is needed when the extension is loaded, that is why the very first > > function in the API returns it. Otherwise it is impossible to check if > > extensions compiled against one version of numpy can be loaded when > another > > version of numpy is present. You need to know what API numpy presents and > > that can be different than the API the extension was compiled against. > > Right, I phrased myself incorrectly. NPY_VERSION is checked at > runtime to make sure modules compiled against a different ABI are not > imported. These defines used to be closer to the top of the header > file, but now a lot of other stuff has been added. Maybe it would be > good to include this as a separate header file, npy_version.h or > npy_api.h. > > IIRC, we can expose NPY_FEATURE_VERSION as part of the API without > breaking ABI compatibility, as long as we add it at the end of the API > functions list. As a hack, we can then check the length of the API > functions list to make sure it is available before we call it. > Why bother when you can combine it with NPY_VERSION and use the existing API? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Tue Apr 28 17:52:03 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 28 Apr 2009 23:52:03 +0200 Subject: [Numpy-discussion] [SciPy-user] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> <5b8d13220904280841m5ddafe02nf766f02607ba7011@mail.gmail.com> <5b8d13220904280857r2ea35d07yb14c8cc6052e92e@mail.gmail.com> <9457e7c80904280919i20e52f72n5b9c2e4ad72f55b@mail.gmail.com> <9457e7c80904281244xc0313c3wdef5aca06dcc214@mail.gmail.com> <9457e7c80904281326p18616b78j49894aaf51bcab07@mail.gmail.com> Message-ID: <9457e7c80904281452gd97c21ci579bc66fcd69b864@mail.gmail.com> 2009/4/28 Charles R Harris : >> IIRC, we can expose NPY_FEATURE_VERSION as part of the API without >> breaking ABI compatibility, as long as we add it at the end of the API >> functions list. ?As a hack, we can then check the length of the API >> functions list to make sure it is available before we call it. > > Why bother when you can combine it with NPY_VERSION and use the existing > API? We could, as long as NPY_VERSION always remains greater than its current value. We considered this option at SciPy'08 -- can you remember why we decided to go for the "split" route instead? When I implemented the change originally, I combined the two into one vaue, as can be seen in the reviewed patches: http://codereview.appspot.com/2946/show St?fan From charlesr.harris at gmail.com Tue Apr 28 18:51:09 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 28 Apr 2009 16:51:09 -0600 Subject: [Numpy-discussion] [SciPy-user] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: <9457e7c80904281452gd97c21ci579bc66fcd69b864@mail.gmail.com> References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> <5b8d13220904280857r2ea35d07yb14c8cc6052e92e@mail.gmail.com> <9457e7c80904280919i20e52f72n5b9c2e4ad72f55b@mail.gmail.com> <9457e7c80904281244xc0313c3wdef5aca06dcc214@mail.gmail.com> <9457e7c80904281326p18616b78j49894aaf51bcab07@mail.gmail.com> <9457e7c80904281452gd97c21ci579bc66fcd69b864@mail.gmail.com> Message-ID: 2009/4/28 St?fan van der Walt > 2009/4/28 Charles R Harris : > >> IIRC, we can expose NPY_FEATURE_VERSION as part of the API without > >> breaking ABI compatibility, as long as we add it at the end of the API > >> functions list. As a hack, we can then check the length of the API > >> functions list to make sure it is available before we call it. > > > > Why bother when you can combine it with NPY_VERSION and use the existing > > API? > > We could, as long as NPY_VERSION always remains greater than its > current value. We considered this option at SciPy'08 -- can you > remember why we decided to go for the "split" route instead? > No, I just vaguely remember it. > > When I implemented the change originally, I combined the two into one > vaue, as can be seen in the reviewed patches: > > http://codereview.appspot.com/2946/show > We could do #define NPY_ABI_VERSION xxxx #define NPY_API_VERSION yyyy #define NPY_VERSION ((xxxx << 16) || yyyy) Since the current value is 0x01000009 we could start with xxxx == 0x0100 and yyyy == 0x000A. I'm not sure how to handle the current 1.3.0 problem. Maybe the best thing to do there is check the numpy release tag from the loaded module and make a exception in the check. After all, things compiled against 1.4 should probably run under with 1.3. Or we could just bite the bullet and make folks upgrade their numpy installations again. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Tue Apr 28 19:24:58 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 29 Apr 2009 01:24:58 +0200 Subject: [Numpy-discussion] [SciPy-user] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> <5b8d13220904280857r2ea35d07yb14c8cc6052e92e@mail.gmail.com> <9457e7c80904280919i20e52f72n5b9c2e4ad72f55b@mail.gmail.com> <9457e7c80904281244xc0313c3wdef5aca06dcc214@mail.gmail.com> <9457e7c80904281326p18616b78j49894aaf51bcab07@mail.gmail.com> <9457e7c80904281452gd97c21ci579bc66fcd69b864@mail.gmail.com> Message-ID: <9457e7c80904281624w60dbb834w9d7580942bacbfcc@mail.gmail.com> 2009/4/29 Charles R Harris : > 1.4 should probably run under with 1.3. Or we could just bite the bullet and > make folks upgrade their numpy installations again. I don't want to bite any bullets :) I'd opt for a workaround on our side, as not to inconvenience users. St?fan From dg.gmane at thesamovar.net Tue Apr 28 22:05:20 2009 From: dg.gmane at thesamovar.net (Dan Goodman) Date: Wed, 29 Apr 2009 04:05:20 +0200 Subject: [Numpy-discussion] Arithmetic on arrays of pointers Message-ID: Hi all, I have a slightly strange idea for something I would like to do with numpy which I guess probably doesn't exist, but I may be wrong. What I want to do, essentially, is to have two arrays of equal size, say X and Y, of pointers to doubles say. Then I want to do (in C notation) *x += *y for each x, y in X, Y. I'd like to write this in Python as X+=Y where X and Y understand that they consist of arrays of pointers to doubles and that it should do the arithmetic on the values pointed to. Now I can do this in C easily enough, and integrate it with numpy via weave or whatever (it only took a few lines using scipy.weave.inline), but it would be really nice if something along these lines were already in numpy or an extension of it already. Anyone know of anything like that or had any experience with similar situations? I'd also like to be able to take pointers and do pointer arithmetic too (because I want to construct the 1D arrays X, Y above from values picked from a variety of possible data structures, including dense and sparse 2D arrays, but in such a way that the code that uses these values doesn't have to know where the values came from in these data structures). Why? you might ask. Well, because I'm writing simulation code which involves scheduling events in the future (simple events like add a value to another value), but the library is supposed to be easily extensible for people who don't know C so I want to make it as easy to use as possible. Well, that's a slightly underdefined problem, basically I just have a vague idea that something like the above would be useful to me and I'm looking for any ideas that might seem relevant. Thanks in advance! Dan From andyjian430074 at gmail.com Wed Apr 29 01:52:37 2009 From: andyjian430074 at gmail.com (Zhenxin Zhan) Date: Tue, 28 Apr 2009 23:52:37 -0600 Subject: [Numpy-discussion] MemoryError for computing eigen-vector on 10, 000*10, 000 matrix Message-ID: <200904282352365469557@gmail.com> Hello, I am a new learner of Numpy. From 'numpybook', I use numpy.linalg.eig(A) to calculate a matrix 2,000*2,000 and it works well. But, when I calculate eigen vector for 10,000*10,000 matrix, there is 'MemoryError' error message in statement numpy.array(...). My laptop has 4GB memory. How to solve this problem? Thanks so much. Jankins -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Wed Apr 29 00:46:22 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 29 Apr 2009 13:46:22 +0900 Subject: [Numpy-discussion] MemoryError for computing eigen-vector on 10, 000*10, 000 matrix In-Reply-To: <200904282352365469557@gmail.com> References: <200904282352365469557@gmail.com> Message-ID: <49F7DB9E.8090003@ar.media.kyoto-u.ac.jp> Zhenxin Zhan wrote: > Hello, > > I am a new learner of Numpy. From 'numpybook', I use > numpy.linalg.eig(A) to calculate a matrix 2,000*2,000 and it works well. > > But, when I calculate eigen vector for 10,000*10,000 matrix, there is > 'MemoryError' error message in statement numpy.array(...). My laptop > has 4GB memory. Which OS are you using ? If your OS is 32 bits, you won't be able to use 4 Gb for you python process, and a 10000x10000 matrix is big (the matrix alone takes ~750 Mb of memory). If that's an option, you could also try single precision instead of the default double precision: import numpy as np A = np.array(...., dtype=np.float32) np.linalg.eig(A) cheers, David From andyjian430074 at gmail.com Wed Apr 29 01:19:38 2009 From: andyjian430074 at gmail.com (Zhenxin Zhan) Date: Wed, 29 Apr 2009 00:19:38 -0500 Subject: [Numpy-discussion] MemoryError for computing eigen-vector on 10, 000*10, 000 matrix References: <200904282352365469557@gmail.com>, <49F7DB9E.8090003@ar.media.kyoto-u.ac.jp> Message-ID: <200904290019375006457@gmail.com> Thanks for your reply. My os is Windows XP SP3. I tried to use array(ojb, dtype=float), but it didn't work. And I tried 'float32' as you told me. And here is the error message: File "C:\Python26\Lib\site-packages\numpy\linalg\linalg.py", line 791, in eig a, t, result_t = _convertarray(a) # convert to double or cdouble type File "C:\Python26\Lib\site-packages\numpy\linalg\linalg.py", line 727, in _con vertarray a = _fastCT(a.astype(t)) MemoryError Thanks. 2009-04-29 Zhenxin Zhan ???? David Cournapeau ????? 2009-04-29 00:04:28 ???? Discussion of Numerical Python ??? ??? Re: [Numpy-discussion] MemoryError for computing eigen-vector on 10,000*10, 000 matrix Zhenxin Zhan wrote: > Hello, > > I am a new learner of Numpy. From 'numpybook', I use > numpy.linalg.eig(A) to calculate a matrix 2,000*2,000 and it works well. > > But, when I calculate eigen vector for 10,000*10,000 matrix, there is > 'MemoryError' error message in statement numpy.array(...). My laptop > has 4GB memory. Which OS are you using ? If your OS is 32 bits, you won't be able to use 4 Gb for you python process, and a 10000x10000 matrix is big (the matrix alone takes ~750 Mb of memory). If that's an option, you could also try single precision instead of the default double precision: import numpy as np A = np.array(...., dtype=np.float32) np.linalg.eig(A) cheers, David _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Apr 29 01:35:51 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 28 Apr 2009 23:35:51 -0600 Subject: [Numpy-discussion] MemoryError for computing eigen-vector on 10, 000*10, 000 matrix In-Reply-To: <200904290019375006457@gmail.com> References: <200904282352365469557@gmail.com> <49F7DB9E.8090003@ar.media.kyoto-u.ac.jp> <200904290019375006457@gmail.com> Message-ID: 2009/4/28 Zhenxin Zhan > Thanks for your reply. > > My os is Windows XP SP3. I tried to use array(ojb, dtype=float), but it > didn't work. And I tried 'float32' as you told me. And here is the error > message: > > > File "C:\Python26\Lib\site-packages\numpy\linalg\linalg.py", line 791, in eig > a, t, result_t = _convertarray(a) # convert to double or cdouble type > > File "C:\Python26\Lib\site-packages\numpy\linalg\linalg.py", line 727, in _con > vertarray > a = _fastCT(a.astype(t)) > MemoryError > > Looks like only a double routine is available for eig. Eigh is better for symmetric routines and if you only want the eigenvalues and not the eigenvectors then you should use eigvals or eigvalsh and save the space devoted to the eigenvectors, which in themselves will put you over the memory limit. The os question is whether or not you are running a 64 bit or 32 bit os. A 64 bit os could use swap, although the routine would take forever to finish. Really, you don't have enough memory for a problem that size. Perhaps if you tell us what you want to achieve we can suggest a better approach. Also, if your matrix is sparse other algorithms might be more appropriate. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Wed Apr 29 01:21:51 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 29 Apr 2009 14:21:51 +0900 Subject: [Numpy-discussion] MemoryError for computing eigen-vector on 10, 000*10, 000 matrix In-Reply-To: <200904290019375006457@gmail.com> References: <200904282352365469557@gmail.com>, <49F7DB9E.8090003@ar.media.kyoto-u.ac.jp> <200904290019375006457@gmail.com> Message-ID: <49F7E3EF.4030004@ar.media.kyoto-u.ac.jp> Zhenxin Zhan wrote: > Thanks for your reply. > My os is Windows XP SP3. I tried to use array(ojb, dtype=float), but > it didn't work. And I tried 'float32' as you told me. And here is the > error message: > File "C:\Python26\Lib\site-packages\numpy\linalg\linalg.py", line 791, > in eig > a, t, result_t = _convertarray(a) # convert to double or cdouble type > File "C:\Python26\Lib\site-packages\numpy\linalg\linalg.py", line 727, > in _con > vertarray > a = _fastCT(a.astype(t)) > MemoryError Ah, sorry, it seems that numpy.linalg.eig only handles double precision, so using single precision input will only make it worse, because of temporaries. If you can use scipy, I would try scipy.linalg, which has a more complete lapack implementation, and should handle single precision correctly. Otherwise, I am afraid there is no solution outside going 64 bits: - your matrix takes 750 Mb, and if it needs to be done in complex, it will take 1.5Gb just by itself - any temporary will likely make the process address space grows beyond 2 Gb (on windows 32 bits, by default, the address space cannot be bigger than 2Gb). With numpy at least, you won't be able to use more than 2 Gb - you can allocate more than that, but you won't be able to use it all at once, you would need to code specifically for it. cheers, David From andyjian430074 at gmail.com Wed Apr 29 01:49:58 2009 From: andyjian430074 at gmail.com (=?utf-8?B?WmhlbnhpbiBaaGFu?=) Date: Wed, 29 Apr 2009 00:49:58 -0500 Subject: [Numpy-discussion] =?utf-8?q?MemoryError_for_computing_eigen-vect?= =?utf-8?q?or_on_10=2C000*10=2C_000_matrix?= References: <200904282352365469557@gmail.com>, <49F7DB9E.8090003@ar.media.kyoto-u.ac.jp>, <200904290019375006457@gmail.com> Message-ID: <200904290049580465625@gmail.com> Thanks. My mistake. The os is 32-bit. I am doing a network-simulation for my teacher. The average degree of the network topology is about 6.0. So I think it is sparse. The paper needs the eigen values and the eigen vectors which are necessary for the further simulation. I use the following procedure: 1. read the network vertices information from a txt file to a 10,000*10,000 list 'lists'. 2. And then use numpy.array(lits, dtype=float) to get a array object 'A' 3. Finally, use numpy.linalg.eig(A) to get the eigen values and eigen vectors. 4. Using 'tofile' function to write them to local file. I will refer to scipy. Thanks so much. 2009-04-29 Zhenxin Zhan ???? Charles R Harris ????? 2009-04-29 00:36:03 ???? Discussion of Numerical Python ??? ??? Re: [Numpy-discussion] MemoryError for computing eigen-vector on 10,000*10, 000 matrix 2009/4/28 Zhenxin Zhan Thanks for your reply. My os is Windows XP SP3. I tried to use array(ojb, dtype=float), but it didn't work. And I tried 'float32' as you told me. And here is the error message: File "C:\Python26\Lib\site-packages\numpy\linalg\linalg.py", line 791, in eig a, t, result_t = _convertarray(a) # convert to double or cdouble type File "C:\Python26\Lib\site-packages\numpy\linalg\linalg.py", line 727, in _con vertarray a = _fastCT(a.astype(t)) MemoryError Looks like only a double routine is available for eig. Eigh is better for symmetric routines and if you only want the eigenvalues and not the eigenvectors then you should use eigvals or eigvalsh and save the space devoted to the eigenvectors, which in themselves will put you over the memory limit. The os question is whether or not you are running a 64 bit or 32 bit os. A 64 bit os could use swap, although the routine would take forever to finish. Really, you don't have enough memory for a problem that size. Perhaps if you tell us what you want to achieve we can suggest a better approach. Also, if your matrix is sparse other algorithms might be more appropriate. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Wed Apr 29 01:46:20 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 29 Apr 2009 14:46:20 +0900 Subject: [Numpy-discussion] [SciPy-user] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: <9457e7c80904280919i20e52f72n5b9c2e4ad72f55b@mail.gmail.com> References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> <5b8d13220904280841m5ddafe02nf766f02607ba7011@mail.gmail.com> <5b8d13220904280857r2ea35d07yb14c8cc6052e92e@mail.gmail.com> <9457e7c80904280919i20e52f72n5b9c2e4ad72f55b@mail.gmail.com> Message-ID: <49F7E9AC.7090909@ar.media.kyoto-u.ac.jp> St?fan van der Walt wrote: > 2009/4/28 David Cournapeau : > >> the header file is not very reliable. More generally, I think we >> should have a way to track C API and ABI relatively to the number >> version, as well as automatically generate the related documentation. >> > > We can already do this: simply choose a convention for NPY_VERSION and > NPY_FEATURE_VERSION so that they are related. > By tracking, I meant more than just the actual number in the source code, like something in the documentation. So that if someone wants to know which function is present in which version of numpy, he can quickly find it in the documentation. Doxygen has a version field for documentation, I wonder whether we could use something like this for the C code. For making sure we don't forget to increase those numbers when necessary, here are two possibilities: - always append NULL as the last item in PyArray_API, so that we can easily get the size of the API, and detect mismatch (a mismatch guarantees incompatibilities, but a match does not guarantee compatibility) - the API is mostly changed when we add functions to the API, that is when either multiarray.txt or ufunc.txt is changed: we could track the checksum of both multiarray.txt and ufunc.txt, and make a relationship between the checksum and the C-API versions. Every time the code generator, it would check the checksum against this list, and barf in case of mismatch; this would remind us to update the version numbers accordingly. cheers, David From dwf at cs.toronto.edu Wed Apr 29 02:19:17 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Wed, 29 Apr 2009 02:19:17 -0400 Subject: [Numpy-discussion] inplace matrix multiplication In-Reply-To: References: <2d1d7fe70901090625k53fe9415uf177e2d80645f12c@mail.gmail.com> <3d375d730901091331u26a0d6dcw8dffca7349fec96@mail.gmail.com> <2d1d7fe70904271221s12ff78b6n2d4bc9c59006bf7f@mail.gmail.com> Message-ID: On 28-Apr-09, at 10:56 AM, Dan Goodman wrote: > Can anyone explain the results below? It seems that for small matrices > dot(x,y) is outperforming dgemm(1,x,y,0,y,overwrite_c=1), but for > larger > matrices the latter is winning. In principle it seems like I ought > to be > able to always do better with inplace rather than making copies? It sounds as though there is some overhead associated with calling dgemm directly that exceeds the time cost of copying for small matrices but is eaten on large matrices by the time saved by not allocating an output array. I'm not all that familiar with the core numpy/scipy linear algebra systems, it could be some cost associated with making a Fortran call if numpy.dot is using CBLAS and dgemm is from FBLAS, I really have no idea. You could try stepping through the code with a debugger and trying to find out. ;) David From charlesr.harris at gmail.com Wed Apr 29 03:09:41 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 29 Apr 2009 01:09:41 -0600 Subject: [Numpy-discussion] MemoryError for computing eigen-vector on 10, 000*10, 000 matrix In-Reply-To: <200904290049580465625@gmail.com> References: <200904282352365469557@gmail.com> <49F7DB9E.8090003@ar.media.kyoto-u.ac.jp> <200904290019375006457@gmail.com> <200904290049580465625@gmail.com> Message-ID: On Tue, Apr 28, 2009 at 11:49 PM, Zhenxin Zhan wrote: > Thanks. My mistake. > > The os is 32-bit. I am doing a network-simulation for my teacher. The > average degree of the network topology is about 6.0. So I think it is > sparse. > > The paper needs the eigen values and the eigen vectors which are necessary > for the further simulation. I use the following procedure: > > 1. read the network vertices information from a txt file to a > 10,000*10,000 list 'lists'. > 2. And then use numpy.array(lits, dtype=float) to get a array object 'A' > 3. Finally, use numpy.linalg.eig(A) to get the eigen values and eigen > vectors. > 4. Using 'tofile' function to write them to local file. > > I will refer to scipy. > > So is this some sort of Markov simulation with the messages passing between nodes with a given probability? Chuck > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Wed Apr 29 03:10:17 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 29 Apr 2009 09:10:17 +0200 Subject: [Numpy-discussion] [SciPy-user] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: <49F7E9AC.7090909@ar.media.kyoto-u.ac.jp> References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> <5b8d13220904280841m5ddafe02nf766f02607ba7011@mail.gmail.com> <5b8d13220904280857r2ea35d07yb14c8cc6052e92e@mail.gmail.com> <9457e7c80904280919i20e52f72n5b9c2e4ad72f55b@mail.gmail.com> <49F7E9AC.7090909@ar.media.kyoto-u.ac.jp> Message-ID: <9457e7c80904290010g27ff4bf1i5874388a64a073f3@mail.gmail.com> 2009/4/29 David Cournapeau : > St?fan van der Walt wrote: >> We can already do this: simply choose a convention for NPY_VERSION and >> NPY_FEATURE_VERSION so that they are related. > > By tracking, I meant more than just the actual number in the source > code, like something in the documentation. So that if someone wants to > know which function is present in which version of numpy, he can quickly > find it in the documentation. Doxygen has a version field for > documentation, I wonder whether we could use something like this for the > C code. I see a number of docstrings already have .. versionadded:: 1.3.0 > ? ?- always append NULL as the last item in PyArray_API, so that we can > easily get the size of the API, and detect mismatch (a mismatch > guarantees incompatibilities, but a match does not guarantee compatibility) Good idea. > ? ? - the API is mostly changed when we add functions to the API, that > is when either multiarray.txt or ufunc.txt is changed: we could track > the checksum of both multiarray.txt and ufunc.txt, and make a > relationship between the checksum and the C-API versions. Every time the > code generator, it would check the checksum against this list, and barf > in case of mismatch; this would remind us to update the version numbers > accordingly. I assume you are referring to ufunc_api_order.txt and numpy_api_order.txt. Fixing them by hashing should be fine; we must just be very careful not to change those files by accident. Also, are we guaranteed that endlines and such will check out the same on the different platforms? Cheers St?fan From sebastian.walter at gmail.com Wed Apr 29 03:17:14 2009 From: sebastian.walter at gmail.com (Sebastian Walter) Date: Wed, 29 Apr 2009 09:17:14 +0200 Subject: [Numpy-discussion] MemoryError for computing eigen-vector on 10, 000*10, 000 matrix In-Reply-To: <200904290049580465625@gmail.com> References: <200904282352365469557@gmail.com> <49F7DB9E.8090003@ar.media.kyoto-u.ac.jp> <200904290019375006457@gmail.com> <200904290049580465625@gmail.com> Message-ID: +1 to that Often, one is only interested in the largest or smallest eigenvalues/vectors of a problem. Then the method of choice are iterative solvers, e.g. Lanczos algorithm. If only the largest eigenvalue/vector is needed, you could try the power iteration. On Wed, Apr 29, 2009 at 7:49 AM, Zhenxin Zhan wrote: > Thanks. My mistake. > > The os is 32-bit. I am doing a network-simulation for my teacher. The > average degree of the network topology is about 6.0. So I think it is > sparse. > > The paper needs the eigen values and the eigen vectors which are necessary > for the further simulation. I use the following procedure: > > 1. read the network vertices information from a txt file to a 10,000*10,000 > list 'lists'. > 2. And then use numpy.array(lits, dtype=float) to get a array object 'A' > 3. Finally, use numpy.linalg.eig(A) to get the eigen values and eigen > vectors. > 4. Using 'tofile' function to write them to local file. > > I will refer to scipy. > > Thanks so much. > > > > > 2009-04-29 > ________________________________ > Zhenxin Zhan > ________________________________ > ???? Charles R Harris > ????? 2009-04-29 00:36:03 > ???? Discussion of Numerical Python > ??? > ??? Re: [Numpy-discussion] MemoryError for computing eigen-vector on > 10,000*10, 000 matrix > > 2009/4/28 Zhenxin Zhan >> >> Thanks for your reply. >> >> My os is Windows XP SP3. I tried to use array(ojb, dtype=float), but it >> didn't work. And I tried 'float32' as you told me. And here is the error >> message: >> >> >> File "C:\Python26\Lib\site-packages\numpy\linalg\linalg.py", line 791, in eig >> a, t, result_t = _convertarray(a) # convert to double or cdouble type >> >> File "C:\Python26\Lib\site-packages\numpy\linalg\linalg.py", line 727, in _con >> vertarray >> a = _fastCT(a.astype(t)) >> MemoryError >> > > Looks like only a double routine is available for eig. Eigh is better for > symmetric routines and if you only want the eigenvalues and not the > eigenvectors then you should use eigvals or eigvalsh and save the space > devoted to the eigenvectors, which in themselves will put you over the > memory limit. > > The os question is whether or not you are running a 64 bit or 32 bit os. A > 64 bit os could use swap, although the routine would take forever to finish. > Really, you don't have enough memory for a problem that size. Perhaps if you > tell us what you want to achieve we can suggest a better approach. Also, if > your matrix is sparse other algorithms might be more appropriate. > > Chuck > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From stefan at sun.ac.za Wed Apr 29 03:26:46 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 29 Apr 2009 09:26:46 +0200 Subject: [Numpy-discussion] Arithmetic on arrays of pointers In-Reply-To: References: Message-ID: <9457e7c80904290026p3f625b8dv420cf02bae0bf48b@mail.gmail.com> Hi Dan 2009/4/29 Dan Goodman : > I have a slightly strange idea for something I would like to do with > numpy which I guess probably doesn't exist, but I may be wrong. What I > want to do, essentially, is to have two arrays of equal size, say X and > Y, of pointers to doubles say. Then I want to do (in C notation) *x += > *y for each x, y in X, Y. I'd like to write this in Python as X+=Y where > X and Y understand that they consist of arrays of pointers to doubles > and that it should do the arithmetic on the values pointed to. Now I can > do this in C easily enough, and integrate it with numpy via weave or > whatever (it only took a few lines using scipy.weave.inline), but it > would be really nice if something along these lines were already in > numpy or an extension of it already. Anyone know of anything like that > or had any experience with similar situations? That's an interesting idea; I'm just glad I don't have to deal with all those segfaults :-) If you are not concerned about speed, you can define an array of objects, and tell those objects how to handle the different operations. But, since you are doing simulations, I guess you want it to be fast, so I would look at defining a "pointer" data-type. All this is much more work than simply writing the loop in Cython or ctypes, though! I wonder if it wouldn't be safer to abstract your "pointers" to an object that is less hazardous to use, say a list. Then you have slots = [obj1, obj2, obj3, ..., obj10] links = array([slots[0], slots[6], slots[20]], dtype=object) # at least this won't segfault links = array([slots[0], slots[1], slots[3]], dtype=object) # and this should work fine Regards St?fan From david at ar.media.kyoto-u.ac.jp Wed Apr 29 04:18:07 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 29 Apr 2009 17:18:07 +0900 Subject: [Numpy-discussion] [SciPy-user] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: <9457e7c80904290010g27ff4bf1i5874388a64a073f3@mail.gmail.com> References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> <5b8d13220904280841m5ddafe02nf766f02607ba7011@mail.gmail.com> <5b8d13220904280857r2ea35d07yb14c8cc6052e92e@mail.gmail.com> <9457e7c80904280919i20e52f72n5b9c2e4ad72f55b@mail.gmail.com> <49F7E9AC.7090909@ar.media.kyoto-u.ac.jp> <9457e7c80904290010g27ff4bf1i5874388a64a073f3@mail.gmail.com> Message-ID: <49F80D3F.1050708@ar.media.kyoto-u.ac.jp> St?fan van der Walt wrote: > > I see a number of docstrings already have > > .. versionadded:: 1.3.0 > Yes, but I don't think it is enough. In particular, I don't like so much the fact that the C API documentation for every function is separate from the code. There are several mismatches between the refguide and the actual code ATM. Maybe we could use sphinx to get the doc from the .c code to document the C api functions, so everything is at one place ? > I assume you are referring to ufunc_api_order.txt and > numpy_api_order.txt. yes. > Fixing them by hashing should be fine; we must > just be very careful not to change those files by accident. I think as a rule, we should not change those by accident :) For pure C libraries, the public headers should always be treated very carefully - unfortunately, we don't have anything like that for python C extensions, but those .txt files are the closest thing we have. I put some code in the detect_capi_mismatch svn branch: - the checksum is computed on a string representation of the ordered list of both numpy and ufunc C API - the ordered list is the one returned by the parsing function used to generate those headers - I keep a checksum of each new version in a separate file. A warning is raised if there is a mismatch cheers, David From daniel.yarlett at gmail.com Wed Apr 29 09:03:00 2009 From: daniel.yarlett at gmail.com (Daniel Yarlett) Date: Wed, 29 Apr 2009 14:03:00 +0100 Subject: [Numpy-discussion] Vectorizing array updates Message-ID: Hi All, I'm coding an iterative algorithm which needs to update a results array on each iteration. I have several (one-dimensional) arrays, and currently I have the following non-vectorized code which gets called at the end of each iteration (Inds, Current, and Update are all one- dimensional and the same size): for i in xrange(Indices.size): Current[Indices[i]] += Update[i] This is running too slowly, and so I am going through the usual process of attempting to vectorize this code (Indices.size > 5000). My attempt at vectorizing this was: Current[Indices] += Update But this does not yield the same result. Here's a simple example. First the iterative version: >>> Current = zeros(2,'d') >>> Update = ones_like(Current) >>> Indices = zeros(2,'i') >>> for i in xrange(Indices.size): ... Current[Indices[i]] += Update[i] ... >>> Current array([ 2., 0.]) And then my vectorized attempt: >>> Current = zeros(2,'d') >>> Update = ones_like(Current) >>> Indices = zeros(2,'i') >>> Current[Indices] += Update >>> Current array([ 1., 0.]) As you can see, Current is different in the two cases. Any ideas how I can recreate the behavior of the iterative process in a more numpy- friendly, vectorized (and hopefully quicker) way? And possibly also about why my intuitions concerning the semantics of the vectorized code are wrong? Many thanks, Dan. From robince at gmail.com Wed Apr 29 09:25:21 2009 From: robince at gmail.com (Robin) Date: Wed, 29 Apr 2009 14:25:21 +0100 Subject: [Numpy-discussion] setterr question Message-ID: Hi, I have been using seterr to try to catch where Nans are appearing in my analysis. I used all: 'warn' which worked the first time I ran the function, but as specified in the documentation it only warns 'once only'. Is there a way I can reset the count so it will warn again, without loosing my session? Thanks Robin From andyjian430074 at gmail.com Wed Apr 29 10:11:22 2009 From: andyjian430074 at gmail.com (=?utf-8?B?WmhlbnhpbiBaaGFu?=) Date: Wed, 29 Apr 2009 09:11:22 -0500 Subject: [Numpy-discussion] =?utf-8?q?MemoryError_for_computing_eigen-vect?= =?utf-8?q?or_on_10=2C000*10=2C_000_matrix?= References: <200904282352365469557@gmail.com>, <49F7DB9E.8090003@ar.media.kyoto-u.ac.jp>, <200904290019375006457@gmail.com>, <200904290049580465625@gmail.com> Message-ID: <200904290911211566206@gmail.com> It's kind of characterizing the dynamics of virus spreading in arbitrary networks. It uses Markov knowledge. There are a lot of complicated mathmatics equations. 2009-04-29 Zhenxin Zhan ???? Charles R Harris ????? 2009-04-29 02:09:57 ???? Discussion of Numerical Python ??? ??? Re: [Numpy-discussion] MemoryError for computing eigen-vector on 10,000*10, 000 matrix On Tue, Apr 28, 2009 at 11:49 PM, Zhenxin Zhan wrote: Thanks. My mistake. The os is 32-bit. I am doing a network-simulation for my teacher. The average degree of the network topology is about 6.0. So I think it is sparse. The paper needs the eigen values and the eigen vectors which are necessary for the further simulation. I use the following procedure: 1. read the network vertices information from a txt file to a 10,000*10,000 list 'lists'. 2. And then use numpy.array(lits, dtype=float) to get a array object 'A' 3. Finally, use numpy.linalg.eig(A) to get the eigen values and eigen vectors. 4. Using 'tofile' function to write them to local file. I will refer to scipy. So is this some sort of Markov simulation with the messages passing between nodes with a given probability? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Wed Apr 29 10:11:36 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 29 Apr 2009 16:11:36 +0200 Subject: [Numpy-discussion] setterr question In-Reply-To: References: Message-ID: <9457e7c80904290711t54d7da7bh8e7559b7584a953@mail.gmail.com> 2009/4/29 Robin : > I have been using seterr to try to catch where Nans are appearing in > my analysis. > > I used all: 'warn' which worked the first time I ran the function, but > as specified in the documentation it only warns 'once only'. Is there > a way I can reset the count so it will warn again, without loosing my > session? Try import warnings warnings.simplefilter('always') Cheers St?fan From andyjian430074 at gmail.com Wed Apr 29 10:14:36 2009 From: andyjian430074 at gmail.com (Zhenxin Zhan) Date: Wed, 29 Apr 2009 09:14:36 -0500 Subject: [Numpy-discussion] MemoryError for computing eigen-vector on 10, 000*10, 000 matrix References: <200904282352365469557@gmail.com>, <49F7DB9E.8090003@ar.media.kyoto-u.ac.jp>, <200904290019375006457@gmail.com>, <200904290049580465625@gmail.com>, Message-ID: <200904290914355627896@gmail.com> Most of the time, it just needs the largest or smallest eigenvalues/vectors. But unfortunately, it needs more. Thanks. 2009-04-29 Zhenxin Zhan ???? Sebastian Walter ????? 2009-04-29 02:17:38 ???? Discussion of Numerical Python ??? ??? Re: [Numpy-discussion] MemoryError for computing eigen-vector on 10,000*10, 000 matrix +1 to that Often, one is only interested in the largest or smallest eigenvalues/vectors of a problem. Then the method of choice are iterative solvers, e.g. Lanczos algorithm. If only the largest eigenvalue/vector is needed, you could try the power iteration. On Wed, Apr 29, 2009 at 7:49 AM, Zhenxin Zhan wrote: > Thanks. My mistake. > > The os is 32-bit. I am doing a network-simulation for my teacher. The > average degree of the network topology is about 6.0. So I think it is > sparse. > > The paper needs the eigen values and the eigen vectors which are necessary > for the further simulation. I use the following procedure: > > 1. read the network vertices information from a txt file to a 10,000*10,000 > list 'lists'. > 2. And then use numpy.array(lits, dtype=float) to get a array object 'A' > 3. Finally, use numpy.linalg.eig(A) to get the eigen values and eigen > vectors. > 4. Using 'tofile' function to write them to local file. > > I will refer to scipy. > > Thanks so much. > > > > > 2009-04-29 > ________________________________ > Zhenxin Zhan > ________________________________ > ???? Charles R Harris > ????? 2009-04-29 00:36:03 > ???? Discussion of Numerical Python > ??? > ??? Re: [Numpy-discussion] MemoryError for computing eigen-vector on > 10,000*10, 000 matrix > > 2009/4/28 Zhenxin Zhan >> >> Thanks for your reply. >> >> My os is Windows XP SP3. I tried to use array(ojb, dtype=float), but it >> didn't work. And I tried 'float32' as you told me. And here is the error >> message: >> >> >> File "C:\Python26\Lib\site-packages\numpy\linalg\linalg.py", line 791, in eig >> a, t, result_t = _convertarray(a) # convert to double or cdouble type >> >> File "C:\Python26\Lib\site-packages\numpy\linalg\linalg.py", line 727, in _con >> vertarray >> a = _fastCT(a.astype(t)) >> MemoryError >> > > Looks like only a double routine is available for eig. Eigh is better for > symmetric routines and if you only want the eigenvalues and not the > eigenvectors then you should use eigvals or eigvalsh and save the space > devoted to the eigenvectors, which in themselves will put you over the > memory limit. > > The os question is whether or not you are running a 64 bit or 32 bit os. A > 64 bit os could use swap, although the routine would take forever to finish. > Really, you don't have enough memory for a problem that size. Perhaps if you > tell us what you want to achieve we can suggest a better approach. Also, if > your matrix is sparse other algorithms might be more appropriate. > > Chuck > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From robince at gmail.com Wed Apr 29 10:35:22 2009 From: robince at gmail.com (Robin) Date: Wed, 29 Apr 2009 15:35:22 +0100 Subject: [Numpy-discussion] setterr question In-Reply-To: <9457e7c80904290711t54d7da7bh8e7559b7584a953@mail.gmail.com> References: <9457e7c80904290711t54d7da7bh8e7559b7584a953@mail.gmail.com> Message-ID: 2009/4/29 St?fan van der Walt : > 2009/4/29 Robin : >> I have been using seterr to try to catch where Nans are appearing in >> my analysis. >> >> I used all: 'warn' which worked the first time I ran the function, but >> as specified in the documentation it only warns 'once only'. Is there >> a way I can reset the count so it will warn again, without loosing my >> session? > > Try > > import warnings > warnings.simplefilter('always') Thanks very much, I thought it would be an easy one! Robin From dfnsonfsduifb at gmx.de Wed Apr 29 11:55:41 2009 From: dfnsonfsduifb at gmx.de (Johannes Bauer) Date: Wed, 29 Apr 2009 17:55:41 +0200 Subject: [Numpy-discussion] Creating a RGB-image from BW In-Reply-To: <01E7A471-B3B5-4CDC-9020-4C7755FE9A08@yale.edu> References: <49F6A409.6060306@gmx.de> <01E7A471-B3B5-4CDC-9020-4C7755FE9A08@yale.edu> Message-ID: <49F8787D.8080101@gmx.de> Hi Zach, Zachary Pincus schrieb: > According to http://www.pygtk.org/pygtk2reference/class- > gdkpixbuf.html , the pixels_array is a numeric python array (a > predecessor to numpy). The upshot is that perhaps the nice > broadcasting machinery will work fine: > > pb_pixels[...] = fits_pixels[..., numpy.newaxis] Ayyyyyyyyyyyyyyyy! *jump* That does what I want so fast that my current implementation can't even measure it. You're a genius! Thanks so much! Kind regards, Johannes From Chris.Barker at noaa.gov Wed Apr 29 13:31:22 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 29 Apr 2009 10:31:22 -0700 Subject: [Numpy-discussion] Arithmetic on arrays of pointers In-Reply-To: References: Message-ID: <49F88EEA.2020503@noaa.gov> Dan Goodman wrote: > (because I want to construct the 1D arrays X, Y above from values picked > from a variety of possible data structures, including dense and sparse > 2D arrays, but in such a way that the code that uses these values > doesn't have to know where the values came from in these data > structures). I wonder if you could define a new datatype (or set of datatypes) for numpy: pointer_to_double, etc. What I'm wondering is how you would construct an array of such types in python, though I suppose you could do something like: a = np.array(data, dtype=np.float) pa = np.empty(size, dytpe=np.p_float) pa[i] = a[i] and have it do "the right thing". You might also be able to grab pointers with ctypes or cython. > code which involves scheduling events in the future (simple events like > add a value to another value), but the library is supposed to be easily > extensible for people who don't know C so I want to make it as easy to > use as possible. > > Well, that's a slightly underdefined problem, basically I just have a > vague idea that something like the above would be useful to me and I'm > looking for any ideas that might seem relevant. Overall, if you want easy to use, and not a lot of segfalts, I suspect you are better off getting away from pointers in user code, and instead creating objects for your users to manipulate that grab copies of data from where they need it, and update the original data when it changes. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From dg.gmane at thesamovar.net Wed Apr 29 16:20:56 2009 From: dg.gmane at thesamovar.net (Dan Goodman) Date: Wed, 29 Apr 2009 22:20:56 +0200 Subject: [Numpy-discussion] Arithmetic on arrays of pointers In-Reply-To: <49F88EEA.2020503@noaa.gov> References: <49F88EEA.2020503@noaa.gov> Message-ID: Thanks Chris and Stefan, In the end, I'm going to take both your advice and not do this after all. Actually I worked out another way to do the same thing. Essentially, rather than store a pointer I store a reference to the array, and an array of indices in that array. Much safer and the things I wanted to do can be done in pure Python/numpy rather than needing C code. It has raised another question though, but I'll start a new thread for that. Thanks again for the suggestions! Dan From dg.gmane at thesamovar.net Wed Apr 29 16:25:53 2009 From: dg.gmane at thesamovar.net (Dan Goodman) Date: Wed, 29 Apr 2009 22:25:53 +0200 Subject: [Numpy-discussion] Vectorizing array updates In-Reply-To: References: Message-ID: > As you can see, Current is different in the two cases. Any ideas how I > can recreate the behavior of the iterative process in a more numpy- > friendly, vectorized (and hopefully quicker) way? And possibly also > about why my intuitions concerning the semantics of the vectorized > code are wrong? I'm also interested to know how to do this, I know it can be done with a C/Cython loop, but I'd like to do it just using numpy if possible. Dan G. From robert.kern at gmail.com Wed Apr 29 16:52:09 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 29 Apr 2009 15:52:09 -0500 Subject: [Numpy-discussion] Vectorizing array updates In-Reply-To: References: Message-ID: <3d375d730904291352u198c21cdw7c931fa5d4557799@mail.gmail.com> On Wed, Apr 29, 2009 at 08:03, Daniel Yarlett wrote: > As you can see, Current is different in the two cases. Any ideas how I > can recreate the behavior of the iterative process in a more numpy- > friendly, vectorized (and hopefully quicker) way? Use bincount(). > And possibly also > about why my intuitions concerning the semantics of the vectorized > code are wrong? In Python, the statement x[indices] += y turns into xi = x.__getitem__(indices) tmp = xi.__iadd__(y) x.__setitem__(indices, tmp) The first statement necessarily copies the data. Then the __iadd__() method modifies the copy in-place (tmp is xi after the operation for numpy arrays, but not necessarily for other objects). Then the final statement assigns the result back into the original array using fancy indexing. Since there are duplicate indices, the last entry in tmp for each duplicate index wins. Because Python breaks up the syntax "x[indices] += y" into those three discrete steps, information is lost on the way, and we cannot use that syntax with the semantics of the loop. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dg.gmane at thesamovar.net Wed Apr 29 17:06:39 2009 From: dg.gmane at thesamovar.net (Dan Goodman) Date: Wed, 29 Apr 2009 23:06:39 +0200 Subject: [Numpy-discussion] Funky vectorisation question Message-ID: Hi all, Here's the problem I want to write vectorised code for. I start with an array of indices, say I=array([0,1,0,2,0,1,4]), and I want to come up with an array C that counts how many times each index has been seen so far if you were counting through the array from the beginning to the end, e.g. for that I, C=array([0,0,1,0,2,1,0]). This is easy enough to do with a C loop, but trickier to do just with numpy. Any ideas? I came up with a solution, but it's not particularly elegant. I bet it can be done better. But here's my solution FWIW: First, sort I (we can always undo this at the end so no loss of generality there). Now we have I=[0,0,0,1,1,2,4]. Next we construct the array J=I[1:]!=I[:-1], i.e. marking the points where the index changes, and the array K=-I. Write A, B for the cumsum of J and K (but with an extra 0 on the beginning of each), so: A=[0,0,0,1,1,2,3] B=[0,1,2,2,3,3,3] Now A is a sort of index of the indices, and B is sort of close to the C we're looking for - it counts upwards at the right times but doesn't reset to 0 when the index changes. So we fix this by subtracting the value at the point where it should reset. The array of these values that need to be subtracted is B[J], although this misses the first one, so we write BJ for the array B[J] with 0 prepended. Now A indexes the indices, so BJ[A] is an array of the same length as the initial array with value the initial value for each segment: BJ=[0,2,3,3] BJ[A]=[0,0,0,2,2,3,3] Finally, C is then B-BJ[A]: C=[0,1,2,0,1,0,0] Complete code: J = I[1:]!=I[:-1] K = I[1:]==I[:-1] A = hstack((0, cumsum(array(J, dtype=int)))) B = hstack((0, cumsum(array(K, dtype=int)))) BJ = hstack((0, B[J])) C = B-BJ[A] OK, so that's my version - can you improve on it? I don't mind so much if the code is unreadable as long as it's fast! :-) Dan From dg.gmane at thesamovar.net Wed Apr 29 17:19:32 2009 From: dg.gmane at thesamovar.net (Dan Goodman) Date: Wed, 29 Apr 2009 23:19:32 +0200 Subject: [Numpy-discussion] Vectorizing array updates In-Reply-To: <3d375d730904291352u198c21cdw7c931fa5d4557799@mail.gmail.com> References: <3d375d730904291352u198c21cdw7c931fa5d4557799@mail.gmail.com> Message-ID: Robert Kern wrote: > On Wed, Apr 29, 2009 at 08:03, Daniel Yarlett wrote: > >> As you can see, Current is different in the two cases. Any ideas how I >> can recreate the behavior of the iterative process in a more numpy- >> friendly, vectorized (and hopefully quicker) way? > > Use bincount(). Neat. Is there a memory efficient way of doing it if the indices are very large but there aren't many of them? e.g. if the indices were I=[10000, 20000] then bincount would create a gigantic array of 20000 elements for just two addition operations! Dan G. From dwf at cs.toronto.edu Wed Apr 29 17:28:45 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Wed, 29 Apr 2009 17:28:45 -0400 Subject: [Numpy-discussion] Funky vectorisation question In-Reply-To: References: Message-ID: <294E60A5-3CF6-4FA9-8DE2-A2EF5C5543F8@cs.toronto.edu> On 29-Apr-09, at 5:06 PM, Dan Goodman wrote: > Here's the problem I want to write vectorised code for. I start with > an > array of indices, say I=array([0,1,0,2,0,1,4]), and I want to come up > with an array C that counts how many times each index has been seen so > far if you were counting through the array from the beginning to the > end, e.g. for that I, C=array([0,0,1,0,2,1,0]). This is easy enough to > do with a C loop, but trickier to do just with numpy. Any ideas? How about this: In [28]: I = array([0,1,0,2,0,1,4]) In [29]: C = empty(len(I),dtype=int) In [30]: for idx in unique(I): ....: V = I == idx ....: C[V] = arange(len(V)) ....: In [31]: C Out[31]: array([0, 0, 1, 0, 2, 1, 0]) David From stefan at sun.ac.za Wed Apr 29 17:48:11 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 29 Apr 2009 23:48:11 +0200 Subject: [Numpy-discussion] Funky vectorisation question In-Reply-To: References: Message-ID: <9457e7c80904291448q163bc7dbo2a7aa694ea0f30de@mail.gmail.com> 2009/4/29 Dan Goodman : > Here's the problem I want to write vectorised code for. I start with an > array of indices, say I=array([0,1,0,2,0,1,4]), and I want to come up > with an array C that counts how many times each index has been seen so > far if you were counting through the array from the beginning to the > end, e.g. for that I, C=array([0,0,1,0,2,1,0]). This is easy enough to > do with a C loop, but trickier to do just with numpy. Any ideas? This is not effective memory-wise, but it avoids the for-loop: np.diag(np.cumsum(x == x[:, None], axis=1)) - 1 Cheers St?fan From dg.gmane at thesamovar.net Wed Apr 29 17:49:46 2009 From: dg.gmane at thesamovar.net (Dan Goodman) Date: Wed, 29 Apr 2009 23:49:46 +0200 Subject: [Numpy-discussion] Funky vectorisation question In-Reply-To: <294E60A5-3CF6-4FA9-8DE2-A2EF5C5543F8@cs.toronto.edu> References: <294E60A5-3CF6-4FA9-8DE2-A2EF5C5543F8@cs.toronto.edu> Message-ID: David Warde-Farley wrote: > On 29-Apr-09, at 5:06 PM, Dan Goodman wrote: > >> Here's the problem I want to write vectorised code for. I start with >> an >> array of indices, say I=array([0,1,0,2,0,1,4]), and I want to come up >> with an array C that counts how many times each index has been seen so >> far if you were counting through the array from the beginning to the >> end, e.g. for that I, C=array([0,0,1,0,2,1,0]). This is easy enough to >> do with a C loop, but trickier to do just with numpy. Any ideas? > > How about this: > > In [28]: I = array([0,1,0,2,0,1,4]) > > In [29]: C = empty(len(I),dtype=int) > > In [30]: for idx in unique(I): > ....: V = I == idx > ....: C[V] = arange(len(V)) > ....: > > In [31]: C > Out[31]: array([0, 0, 1, 0, 2, 1, 0]) > > David Thanks David, that's nice but unfortunately that Python loop will kill me. I'm thinking about some simulation code I'm writing where this operation will be carried out many, many times, with large arrays I. I figure I need to keep the Python overheads to a fixed cost to get good performance. Dan From robert.kern at gmail.com Wed Apr 29 17:58:21 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 29 Apr 2009 16:58:21 -0500 Subject: [Numpy-discussion] Vectorizing array updates In-Reply-To: References: <3d375d730904291352u198c21cdw7c931fa5d4557799@mail.gmail.com> Message-ID: <3d375d730904291458w73675641vc858603a1886e13b@mail.gmail.com> On Wed, Apr 29, 2009 at 16:19, Dan Goodman wrote: > Robert Kern wrote: >> On Wed, Apr 29, 2009 at 08:03, Daniel Yarlett wrote: >> >>> As you can see, Current is different in the two cases. Any ideas how I >>> can recreate the behavior of the iterative process in a more numpy- >>> friendly, vectorized (and hopefully quicker) way? >> >> Use bincount(). > > Neat. Is there a memory efficient way of doing it if the indices are > very large but there aren't many of them? e.g. if the indices were > I=[10000, 20000] then bincount would create a gigantic array of 20000 > elements for just two addition operations! indices -= indices.min() -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dg.gmane at thesamovar.net Wed Apr 29 18:09:27 2009 From: dg.gmane at thesamovar.net (Dan Goodman) Date: Thu, 30 Apr 2009 00:09:27 +0200 Subject: [Numpy-discussion] Vectorizing array updates In-Reply-To: <3d375d730904291458w73675641vc858603a1886e13b@mail.gmail.com> References: <3d375d730904291352u198c21cdw7c931fa5d4557799@mail.gmail.com> <3d375d730904291458w73675641vc858603a1886e13b@mail.gmail.com> Message-ID: <49F8D017.3070201@thesamovar.net> Robert Kern wrote: > On Wed, Apr 29, 2009 at 16:19, Dan Goodman wrote: >> Robert Kern wrote: >>> On Wed, Apr 29, 2009 at 08:03, Daniel Yarlett wrote: >>> >>>> As you can see, Current is different in the two cases. Any ideas how I >>>> can recreate the behavior of the iterative process in a more numpy- >>>> friendly, vectorized (and hopefully quicker) way? >>> Use bincount(). >> Neat. Is there a memory efficient way of doing it if the indices are >> very large but there aren't many of them? e.g. if the indices were >> I=[10000, 20000] then bincount would create a gigantic array of 20000 >> elements for just two addition operations! > > indices -= indices.min() > Ah OK, but bincount is still going to make an array of 10000 elements in that case... I came up with this trick, but I wonder whether it's overkill: Supposing you want to do y+=x[I] where x is big and indices in I are large but sparse (although potentially including repeats). J=unique(I) K=digitize(I,J)-1 b=bincount(K) y+=b*x[J] Well it seems to work, but surely there must be a nicer way? Dan From dwf at cs.toronto.edu Wed Apr 29 18:14:09 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Wed, 29 Apr 2009 18:14:09 -0400 Subject: [Numpy-discussion] Funky vectorisation question In-Reply-To: References: <294E60A5-3CF6-4FA9-8DE2-A2EF5C5543F8@cs.toronto.edu> Message-ID: <03AD1BD0-D4E4-4C97-B506-BC12F2E20863@cs.toronto.edu> On 29-Apr-09, at 5:49 PM, Dan Goodman wrote: > Thanks David, that's nice but unfortunately that Python loop will kill > me. I'm thinking about some simulation code I'm writing where this > operation will be carried out many, many times, with large arrays I. I > figure I need to keep the Python overheads to a fixed cost to get good > performance. I see. Well, keep in mind that the loop only scales in the number of unique elements in I, rather than the total number of elements. This might make it much less costly than you might think depending on the typical distribution of elements in I. Have you considered coding up a looped version in Cython? If this is going to be a bottleneck then it would be very worthwhile. St?fan's code is clever, although as he points out, it will create an intermediate array of size (len(I))**2, which may end up being as much of a problem as a Python loop if you're allocating and garbage collecting an N**2 array every time. David From dg.gmane at thesamovar.net Wed Apr 29 18:21:16 2009 From: dg.gmane at thesamovar.net (Dan Goodman) Date: Thu, 30 Apr 2009 00:21:16 +0200 Subject: [Numpy-discussion] Funky vectorisation question In-Reply-To: <9457e7c80904291448q163bc7dbo2a7aa694ea0f30de@mail.gmail.com> References: <9457e7c80904291448q163bc7dbo2a7aa694ea0f30de@mail.gmail.com> Message-ID: St?fan van der Walt wrote: > 2009/4/29 Dan Goodman : >> Here's the problem I want to write vectorised code for. I start with an >> array of indices, say I=array([0,1,0,2,0,1,4]), and I want to come up >> with an array C that counts how many times each index has been seen so >> far if you were counting through the array from the beginning to the >> end, e.g. for that I, C=array([0,0,1,0,2,1,0]). This is easy enough to >> do with a C loop, but trickier to do just with numpy. Any ideas? > > This is not effective memory-wise, but it avoids the for-loop: > > np.diag(np.cumsum(x == x[:, None], axis=1)) - 1 > > Cheers > St?fan That's incredibly ingenious! Kudos! But, as you say, I think the memory issue will cause me problems here... Dan From dg.gmane at thesamovar.net Wed Apr 29 18:30:20 2009 From: dg.gmane at thesamovar.net (Dan Goodman) Date: Thu, 30 Apr 2009 00:30:20 +0200 Subject: [Numpy-discussion] Funky vectorisation question In-Reply-To: <03AD1BD0-D4E4-4C97-B506-BC12F2E20863@cs.toronto.edu> References: <294E60A5-3CF6-4FA9-8DE2-A2EF5C5543F8@cs.toronto.edu> <03AD1BD0-D4E4-4C97-B506-BC12F2E20863@cs.toronto.edu> Message-ID: David Warde-Farley wrote: > On 29-Apr-09, at 5:49 PM, Dan Goodman wrote: > >> Thanks David, that's nice but unfortunately that Python loop will kill >> me. I'm thinking about some simulation code I'm writing where this >> operation will be carried out many, many times, with large arrays I. I >> figure I need to keep the Python overheads to a fixed cost to get good >> performance. > > > I see. Well, keep in mind that the loop only scales in the number of > unique elements in I, rather than the total number of elements. This > might make it much less costly than you might think depending on the > typical distribution of elements in I. Indeed, I'll try various things out, but I think I'll probably have a reasonably large number of unique elements (although there'll definitely be overlap too). > Have you considered coding up a looped version in Cython? If this is > going to be a bottleneck then it would be very worthwhile. St?fan's > code is clever, although as he points out, it will create an > intermediate array of size (len(I))**2, which may end up being as much > of a problem as a Python loop if you're allocating and garbage > collecting an N**2 array every time. > > David I have a looped version using scipy.weave at the moment (although Cython/Swig etc. are other possibilities for this), but the goal is to have this included in a package designed for people who don't necessarily have access to anything other than numpy and scipy. What we're doing at the moment is we have various bottleneck algorithms which we have C code for, but there's always a pure numpy fallback option for people who, for whatever reason, can't use the C stuff. Actually I'm not entirely sure this is necessary given that for Windows we can provide a built distribution, and everyone who is on linux has gcc and so forth already installed. But that's the policy (at least at the moment, it might change). Dan From peridot.faceted at gmail.com Wed Apr 29 18:45:21 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 29 Apr 2009 18:45:21 -0400 Subject: [Numpy-discussion] Vectorizing array updates In-Reply-To: <49F8D017.3070201@thesamovar.net> References: <3d375d730904291352u198c21cdw7c931fa5d4557799@mail.gmail.com> <3d375d730904291458w73675641vc858603a1886e13b@mail.gmail.com> <49F8D017.3070201@thesamovar.net> Message-ID: 2009/4/29 Dan Goodman : > Robert Kern wrote: >> On Wed, Apr 29, 2009 at 16:19, Dan Goodman wrote: >>> Robert Kern wrote: >>>> On Wed, Apr 29, 2009 at 08:03, Daniel Yarlett wrote: >>>> >>>>> As you can see, Current is different in the two cases. Any ideas how I >>>>> can recreate the behavior of the iterative process in a more numpy- >>>>> friendly, vectorized (and hopefully quicker) way? >>>> Use bincount(). >>> Neat. Is there a memory efficient way of doing it if the indices are >>> very large but there aren't many of them? e.g. if the indices were >>> I=[10000, 20000] then bincount would create a gigantic array of 20000 >>> elements for just two addition operations! >> >> indices -= indices.min() >> > > Ah OK, but bincount is still going to make an array of 10000 elements in > that case... > > I came up with this trick, but I wonder whether it's overkill: > > Supposing you want to do y+=x[I] where x is big and indices in I are > large but sparse (although potentially including repeats). > > J=unique(I) > K=digitize(I,J)-1 > b=bincount(K) > y+=b*x[J] > > Well it seems to work, but surely there must be a nicer way? Well, a few lines serve to rearrange the indices array so it contains each number only once, and then to add together (with bincount) all values with the same index: ix = np.unique1d(indices) reverse_index = np.searchsorted(ix, indices) r = np.bincount(reverse_index, weights=values) You can then do: y[ix] += r All this could be done away with if bincount supported input and output arguments. Then y[indices] += values could become np.bincount(indices, weights=values, input=y, output=y) implemented y the same nice tight loop one would use in C. (I'm not sure bincount is the right name for such a general function, though.) This question comes up fairly often on the mailing list, so it would be nice to have a single function to point to. Anne From stefan at sun.ac.za Wed Apr 29 20:05:37 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 30 Apr 2009 02:05:37 +0200 Subject: [Numpy-discussion] Funky vectorisation question In-Reply-To: <03AD1BD0-D4E4-4C97-B506-BC12F2E20863@cs.toronto.edu> References: <294E60A5-3CF6-4FA9-8DE2-A2EF5C5543F8@cs.toronto.edu> <03AD1BD0-D4E4-4C97-B506-BC12F2E20863@cs.toronto.edu> Message-ID: <9457e7c80904291705x73dbd82cu90ee1ea0cc0280eb@mail.gmail.com> 2009/4/30 David Warde-Farley : > Have you considered coding up a looped version in Cython? If this is > going to be a bottleneck then it would be very worthwhile. St?fan's > code is clever, although as he points out, it will create an > intermediate array of size (len(I))**2, which may end up being as much > of a problem as a Python loop if you're allocating and garbage > collecting an N**2 array every time. Here is a version that does not create such large intermediate arrays. It's 0200 here, so I'd be surprised if this works for all the corner cases :) [I attach a file with the 3 different approaches given so far, so that you can benchmark them. My second attempt is about 30 times faster than my first attempt, and about 7 times faster than David's for-loop]. import numpy as np def count1(x): """stefan-2: Improved memory footprint. """ sort_idx = np.argsort(x) x_sorted = x[sort_idx] jumps = np.hstack([[0], np.where(np.diff(x_sorted) > 0)[0] + 1]) count = np.ones(len(x)) count[jumps] -= np.hstack([jumps[0] + 1, np.diff(jumps)]) out = np.empty(len(x)) out[sort_idx] = np.cumsum(count) return out Cheers St?fan -------------- next part -------------- A non-text attachment was scrubbed... Name: crazysort.py Type: application/octet-stream Size: 1005 bytes Desc: not available URL: From mjanikas at esri.com Wed Apr 29 21:15:54 2009 From: mjanikas at esri.com (Mark Janikas) Date: Wed, 29 Apr 2009 18:15:54 -0700 Subject: [Numpy-discussion] Timing array construction Message-ID: <6DF3F8F869B22C4393D67CA19A35AA0E01B49B15C4@redmx1.esri.com> Hello All, I was exploring some different ways to concatenate arrays, and using "c_" is the fastest by far. Is there a difference I am missing that can account for the huge disparity? Obviously the "zip" function makes the "as array" and "array" calls slower, but the same arguments (xCoords, yCoords) are being passed to the methods... so if there is no difference in the outputs (there doesn't appear to be) then what reason would I have to use "array" or "as array" in this context? Thanks so much ahead of time.. MJ ############## Snippet ################### import numpy as NUM def useAsArray(xCoords, yCoords): return NUM.asarray(zip(xCoords, yCoords)) def useArray(xCoords, yCoords): return NUM.array(zip(xCoords, yCoords)) def useC(xCoords, yCoords): return NUM.c_[xCoords, yCoords] if __name__ == "__main__": from timeit import Timer import numpy.random as RAND import collections as COLL resAsArray = COLL.defaultdict(float) resArray = COLL.defaultdict(float) resMat = COLL.defaultdict(float) numTests = 0.0 sameTests = 0.0 N = [100, 200, 400, 800, 1600, 3200, 6400, 12800] for i in N: print "Time Join List into Array for N = " + str(i) xCoords = RAND.normal(10, 1, i) yCoords = RAND.normal(10, 1, i) statement = 'from __main__ import xCoords, yCoords, useAsArray' t1 = Timer('useAsArray(xCoords, yCoords)', statement) resAsArray[i] = t1.timeit(10) statement = 'from __main__ import xCoords, yCoords, useArray' t2 = Timer('useArray(xCoords, yCoords)', statement) resArray[i] = t2.timeit(10) statement = 'from __main__ import xCoords, yCoords, useC' t3 = Timer('useC(xCoords, yCoords)', statement) resMat[i] = t3.timeit(10) for n in N: print "%i, %0.4f, %0.4f, %0.4f" % (n, resAsArray[n], resArray[n], resMat[n]) ############################################################### RESULT N, useAsArray, useArray, useC 100, 0.0066, 0.0065, 0.0007 200, 0.0137, 0.0140, 0.0008 400, 0.0277, 0.0288, 0.0007 800, 0.0579, 0.0577, 0.0008 1600, 0.1175, 0.1289, 0.0009 3200, 0.2291, 0.2309, 0.0012 6400, 0.4561, 0.4564, 0.0013 12800, 0.9218, 0.9122, 0.0019 Mark Janikas Product Engineer ESRI, Geoprocessing 380 New York St. Redlands, CA 92373 909-793-2853 (2563) mjanikas at esri.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From efiring at hawaii.edu Thu Apr 30 02:49:20 2009 From: efiring at hawaii.edu (Eric Firing) Date: Wed, 29 Apr 2009 20:49:20 -1000 Subject: [Numpy-discussion] Timing array construction In-Reply-To: <6DF3F8F869B22C4393D67CA19A35AA0E01B49B15C4@redmx1.esri.com> References: <6DF3F8F869B22C4393D67CA19A35AA0E01B49B15C4@redmx1.esri.com> Message-ID: <49F949F0.4080609@hawaii.edu> Mark Janikas wrote: > Hello All, > > > > I was exploring some different ways to concatenate arrays, and using > ?c_? is the fastest by far. Is there a difference I am missing that can > account for the huge disparity? Obviously the ?zip? function makes the > ?as array? and ?array? calls slower, but the same arguments (xCoords, > yCoords) are being passed to the methods? so if there is no difference > in the outputs (there doesn?t appear to be) then what reason would I > have to use ?array? or ?as array? in this context? Thanks so much ahead > of time.. If you really want speed, use something like this: import numpy as np def useEmpty(xCoords, yCoords): out = np.empty((len(xCoords), 2), dtype=xCoords.dtype) out[:,0] = xCoords out[:,1] = yCoords return out It is quite a bit faster than using c_; more than a factor of two on my machine for all your test cases. All your methods using zip and array are doing a lot of unpacking, repacking, checking, iterating... Even the c_ method is slower than it needs to be for this case because it is more general and flexible. Eric > > > > MJ > > > > ############## Snippet ################### > > import numpy as NUM > > > > def useAsArray(xCoords, yCoords): > > return NUM.asarray(zip(xCoords, yCoords)) > > > > def useArray(xCoords, yCoords): > > return NUM.array(zip(xCoords, yCoords)) > > > > def useC(xCoords, yCoords): > > return NUM.c_[xCoords, yCoords] > > > > > > if __name__ == "__main__": > > from timeit import Timer > > import numpy.random as RAND > > import collections as COLL > > > > resAsArray = COLL.defaultdict(float) > > resArray = COLL.defaultdict(float) > > resMat = COLL.defaultdict(float) > > numTests = 0.0 > > sameTests = 0.0 > > N = [100, 200, 400, 800, 1600, 3200, 6400, 12800] > > for i in N: > > print "Time Join List into Array for N = " + str(i) > > xCoords = RAND.normal(10, 1, i) > > yCoords = RAND.normal(10, 1, i) > > > > statement = 'from __main__ import xCoords, yCoords, useAsArray' > > t1 = Timer('useAsArray(xCoords, yCoords)', statement) > > resAsArray[i] = t1.timeit(10) > > > > statement = 'from __main__ import xCoords, yCoords, useArray' > > t2 = Timer('useArray(xCoords, yCoords)', statement) > > resArray[i] = t2.timeit(10) > > > > statement = 'from __main__ import xCoords, yCoords, useC' > > t3 = Timer('useC(xCoords, yCoords)', statement) > > resMat[i] = t3.timeit(10) > > > > for n in N: > > print "%i, %0.4f, %0.4f, %0.4f" % (n, resAsArray[n], > resArray[n], resMat[n]) > > ############################################################### > > > > RESULT > > > > N, useAsArray, useArray, useC > > 100, 0.0066, 0.0065, 0.0007 > > 200, 0.0137, 0.0140, 0.0008 > > 400, 0.0277, 0.0288, 0.0007 > > 800, 0.0579, 0.0577, 0.0008 > > 1600, 0.1175, 0.1289, 0.0009 > > 3200, 0.2291, 0.2309, 0.0012 > > 6400, 0.4561, 0.4564, 0.0013 > > 12800, 0.9218, 0.9122, 0.0019 > > > > > > Mark Janikas > > Product Engineer > > ESRI, Geoprocessing > > 380 New York St. > > Redlands, CA 92373 > > 909-793-2853 (2563) > > mjanikas at esri.com > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From cournape at gmail.com Thu Apr 30 05:08:25 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 30 Apr 2009 18:08:25 +0900 Subject: [Numpy-discussion] Splitting multiarray and umath into smaller files: ready for review In-Reply-To: References: <49F572A2.90406@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220904300208m66580455kb84cbafdcebdb4df@mail.gmail.com> 2009/4/28 Charles R Harris : > > > I think some of the fixes for too big arrays should be backported to 1.3.x > before this is merged. > That's r6851 and r6853. I'll do that. Ok, I put the changes in the trunk. I will add some documentation as well, cheers, David From mjanikas at esri.com Thu Apr 30 12:51:55 2009 From: mjanikas at esri.com (Mark Janikas) Date: Thu, 30 Apr 2009 09:51:55 -0700 Subject: [Numpy-discussion] Timing array construction In-Reply-To: <49F949F0.4080609@hawaii.edu> References: <6DF3F8F869B22C4393D67CA19A35AA0E01B49B15C4@redmx1.esri.com> <49F949F0.4080609@hawaii.edu> Message-ID: <6DF3F8F869B22C4393D67CA19A35AA0E01B49B15C6@redmx1.esri.com> Thanks Eric! I have a lot of array constructions in my code that use NUM.array([list of values])... I am going to replace it with the empty allocation and insertion. It is indeed twice as fast as "c_" (when it matters, I.e. N is relatively large): "c_", "empty" 100 0.0007, 0.0230 200 0.0007, 0.0002 400 0.0007, 0.0002 800 0.0020, 0.0002 1600 0.0009, 0.0003 3200 0.0010, 0.0003 6400 0.0013, 0.0005 12800 0.0058, 0.0032 -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Eric Firing Sent: Wednesday, April 29, 2009 11:49 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Timing array construction Mark Janikas wrote: > Hello All, > > > > I was exploring some different ways to concatenate arrays, and using > "c_" is the fastest by far. Is there a difference I am missing that can > account for the huge disparity? Obviously the "zip" function makes the > "as array" and "array" calls slower, but the same arguments (xCoords, > yCoords) are being passed to the methods... so if there is no difference > in the outputs (there doesn't appear to be) then what reason would I > have to use "array" or "as array" in this context? Thanks so much ahead > of time.. If you really want speed, use something like this: import numpy as np def useEmpty(xCoords, yCoords): out = np.empty((len(xCoords), 2), dtype=xCoords.dtype) out[:,0] = xCoords out[:,1] = yCoords return out It is quite a bit faster than using c_; more than a factor of two on my machine for all your test cases. All your methods using zip and array are doing a lot of unpacking, repacking, checking, iterating... Even the c_ method is slower than it needs to be for this case because it is more general and flexible. Eric > > > > MJ > > > > ############## Snippet ################### > > import numpy as NUM > > > > def useAsArray(xCoords, yCoords): > > return NUM.asarray(zip(xCoords, yCoords)) > > > > def useArray(xCoords, yCoords): > > return NUM.array(zip(xCoords, yCoords)) > > > > def useC(xCoords, yCoords): > > return NUM.c_[xCoords, yCoords] > > > > > > if __name__ == "__main__": > > from timeit import Timer > > import numpy.random as RAND > > import collections as COLL > > > > resAsArray = COLL.defaultdict(float) > > resArray = COLL.defaultdict(float) > > resMat = COLL.defaultdict(float) > > numTests = 0.0 > > sameTests = 0.0 > > N = [100, 200, 400, 800, 1600, 3200, 6400, 12800] > > for i in N: > > print "Time Join List into Array for N = " + str(i) > > xCoords = RAND.normal(10, 1, i) > > yCoords = RAND.normal(10, 1, i) > > > > statement = 'from __main__ import xCoords, yCoords, useAsArray' > > t1 = Timer('useAsArray(xCoords, yCoords)', statement) > > resAsArray[i] = t1.timeit(10) > > > > statement = 'from __main__ import xCoords, yCoords, useArray' > > t2 = Timer('useArray(xCoords, yCoords)', statement) > > resArray[i] = t2.timeit(10) > > > > statement = 'from __main__ import xCoords, yCoords, useC' > > t3 = Timer('useC(xCoords, yCoords)', statement) > > resMat[i] = t3.timeit(10) > > > > for n in N: > > print "%i, %0.4f, %0.4f, %0.4f" % (n, resAsArray[n], > resArray[n], resMat[n]) > > ############################################################### > > > > RESULT > > > > N, useAsArray, useArray, useC > > 100, 0.0066, 0.0065, 0.0007 > > 200, 0.0137, 0.0140, 0.0008 > > 400, 0.0277, 0.0288, 0.0007 > > 800, 0.0579, 0.0577, 0.0008 > > 1600, 0.1175, 0.1289, 0.0009 > > 3200, 0.2291, 0.2309, 0.0012 > > 6400, 0.4561, 0.4564, 0.0013 > > 12800, 0.9218, 0.9122, 0.0019 > > > > > > Mark Janikas > > Product Engineer > > ESRI, Geoprocessing > > 380 New York St. > > Redlands, CA 92373 > > 909-793-2853 (2563) > > mjanikas at esri.com > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From Chris.Barker at noaa.gov Thu Apr 30 15:16:05 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 30 Apr 2009 12:16:05 -0700 Subject: [Numpy-discussion] Timing array construction In-Reply-To: <6DF3F8F869B22C4393D67CA19A35AA0E01B49B15C6@redmx1.esri.com> References: <6DF3F8F869B22C4393D67CA19A35AA0E01B49B15C4@redmx1.esri.com> <49F949F0.4080609@hawaii.edu> <6DF3F8F869B22C4393D67CA19A35AA0E01B49B15C6@redmx1.esri.com> Message-ID: <49F9F8F5.2010307@noaa.gov> Mark Janikas wrote: > I have a lot of array constructions in my code that use > NUM.array([list of values])... I am going to replace it with the > empty allocation and insertion. It may not be worth it, depending on where list_of_values comes from/is. A rule of thumb may be: it's going to be slow going from a numpy array to a regular old python list or tuple, back to a numpy array. If your data is a python list already, than np.array(list) is a fine choice. >> def useAsArray(xCoords, yCoords): >> >> return NUM.asarray(zip(xCoords, yCoords)) Here are some of the issues with this one: zip unpacks two generic python sequences and then put the items into tuple, then puts them in a list. Essentially this: new_list = [] for i in range(len(xCoords)): new_list.append((xCoords[i], yCoords[i])) In each iteration of that loop, it's indexing into the numpy arrays, making a python object out of them, putting them into a tuple, and appending that tuple to the list, which may have to re-allocate memory a few times. Then the np.array() call loops through that list, unpacks each tuple, examines the python object, decides what it is, and turn it into a raw c-type to put into the array. whereas: def useEmpty(xCoords, yCoords): out = np.empty((len(xCoords), 2), dtype=xCoords.dtype) out[:,0] = xCoords out[:,1] = yCoords return out allocates an array the right size. directly copies the data from xCoords and yCoords to it. that's it. You can see why it's so much faster! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From bsouthey at gmail.com Thu Apr 30 15:32:47 2009 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 30 Apr 2009 14:32:47 -0500 Subject: [Numpy-discussion] Timing array construction In-Reply-To: <6DF3F8F869B22C4393D67CA19A35AA0E01B49B15C6@redmx1.esri.com> References: <6DF3F8F869B22C4393D67CA19A35AA0E01B49B15C4@redmx1.esri.com> <49F949F0.4080609@hawaii.edu> <6DF3F8F869B22C4393D67CA19A35AA0E01B49B15C6@redmx1.esri.com> Message-ID: <49F9FCDF.7070700@gmail.com> Mark Janikas wrote: > Thanks Eric! > > I have a lot of array constructions in my code that use NUM.array([list of values])... I am going to replace it with the empty allocation and insertion. It is indeed twice as fast as "c_" (when it matters, I.e. N is relatively large): > > "c_", "empty" > 100 0.0007, 0.0230 > 200 0.0007, 0.0002 > 400 0.0007, 0.0002 > 800 0.0020, 0.0002 > 1600 0.0009, 0.0003 > 3200 0.0010, 0.0003 > 6400 0.0013, 0.0005 > 12800 0.0058, 0.0032 > > -----Original Message----- > From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Eric Firing > Sent: Wednesday, April 29, 2009 11:49 PM > To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion] Timing array construction > > Mark Janikas wrote: > >> Hello All, >> >> >> >> I was exploring some different ways to concatenate arrays, and using >> "c_" is the fastest by far. Is there a difference I am missing that can >> account for the huge disparity? Obviously the "zip" function makes the >> "as array" and "array" calls slower, but the same arguments (xCoords, >> yCoords) are being passed to the methods... so if there is no difference >> in the outputs (there doesn't appear to be) then what reason would I >> have to use "array" or "as array" in this context? Thanks so much ahead >> of time.. >> > > If you really want speed, use something like this: > > import numpy as np > def useEmpty(xCoords, yCoords): > out = np.empty((len(xCoords), 2), dtype=xCoords.dtype) > out[:,0] = xCoords > out[:,1] = yCoords > return out > > It is quite a bit faster than using c_; more than a factor of two on my > machine for all your test cases. > > All your methods using zip and array are doing a lot of unpacking, > repacking, checking, iterating... Even the c_ method is slower than it > needs to be for this case because it is more general and flexible. > > Eric > >> >> >> MJ >> >> >> >> ############## Snippet ################### >> >> import numpy as NUM >> >> >> >> def useAsArray(xCoords, yCoords): >> >> return NUM.asarray(zip(xCoords, yCoords)) >> >> >> >> def useArray(xCoords, yCoords): >> >> return NUM.array(zip(xCoords, yCoords)) >> >> >> >> def useC(xCoords, yCoords): >> >> return NUM.c_[xCoords, yCoords] >> >> >> >> >> >> if __name__ == "__main__": >> >> from timeit import Timer >> >> import numpy.random as RAND >> >> import collections as COLL >> >> >> >> resAsArray = COLL.defaultdict(float) >> >> resArray = COLL.defaultdict(float) >> >> resMat = COLL.defaultdict(float) >> >> numTests = 0.0 >> >> sameTests = 0.0 >> >> N = [100, 200, 400, 800, 1600, 3200, 6400, 12800] >> >> for i in N: >> >> print "Time Join List into Array for N = " + str(i) >> >> xCoords = RAND.normal(10, 1, i) >> >> yCoords = RAND.normal(10, 1, i) >> >> >> >> statement = 'from __main__ import xCoords, yCoords, useAsArray' >> >> t1 = Timer('useAsArray(xCoords, yCoords)', statement) >> >> resAsArray[i] = t1.timeit(10) >> >> >> >> statement = 'from __main__ import xCoords, yCoords, useArray' >> >> t2 = Timer('useArray(xCoords, yCoords)', statement) >> >> resArray[i] = t2.timeit(10) >> >> >> >> statement = 'from __main__ import xCoords, yCoords, useC' >> >> t3 = Timer('useC(xCoords, yCoords)', statement) >> >> resMat[i] = t3.timeit(10) >> >> >> >> for n in N: >> >> print "%i, %0.4f, %0.4f, %0.4f" % (n, resAsArray[n], >> resArray[n], resMat[n]) >> >> ############################################################### >> >> >> >> RESULT >> >> >> >> N, useAsArray, useArray, useC >> >> 100, 0.0066, 0.0065, 0.0007 >> >> 200, 0.0137, 0.0140, 0.0008 >> >> 400, 0.0277, 0.0288, 0.0007 >> >> 800, 0.0579, 0.0577, 0.0008 >> >> 1600, 0.1175, 0.1289, 0.0009 >> >> 3200, 0.2291, 0.2309, 0.0012 >> >> 6400, 0.4561, 0.4564, 0.0013 >> >> 12800, 0.9218, 0.9122, 0.0019 >> >> >> >> >> >> Mark Janikas >> >> Product Engineer >> >> ESRI, Geoprocessing >> >> 380 New York St. >> >> Redlands, CA 92373 >> >> 909-793-2853 (2563) >> >> mjanikas at esri.com >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> Hi, You can also use column_stack (due to the desired result) as in: numpy.column_stack((xCoords, yCoords)) numpy.concatenate() is more general. While not as fast as using numpy.empty(), it does provide a more readable and flexible syntax (for example, you do not have to know in advance how many columns). Bruce From mjanikas at esri.com Thu Apr 30 18:36:12 2009 From: mjanikas at esri.com (Mark Janikas) Date: Thu, 30 Apr 2009 15:36:12 -0700 Subject: [Numpy-discussion] Timing array construction In-Reply-To: <49F9F8F5.2010307@noaa.gov> References: <6DF3F8F869B22C4393D67CA19A35AA0E01B49B15C4@redmx1.esri.com> <49F949F0.4080609@hawaii.edu> <6DF3F8F869B22C4393D67CA19A35AA0E01B49B15C6@redmx1.esri.com> <49F9F8F5.2010307@noaa.gov> Message-ID: <6DF3F8F869B22C4393D67CA19A35AA0E01B49B15CB@redmx1.esri.com> Thanks Chris and Bruce for the further input. I kindof like the "c_" method because it is still relatively speedy and easy to implement. But, the empty method seems to be closest to what is actually done no matter which direction you go in... I.e. preallocate space and insert. I am in the process of ripping all of my zip calls out. The profile of my first set of techniques is already significantly better. This whole exercise has been very enlightening, as I spend so much time working on speeding up my algorithms and simple things like this should be tackled first. Thanks again! MJ -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Christopher Barker Sent: Thursday, April 30, 2009 12:16 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Timing array construction Mark Janikas wrote: > I have a lot of array constructions in my code that use > NUM.array([list of values])... I am going to replace it with the > empty allocation and insertion. It may not be worth it, depending on where list_of_values comes from/is. A rule of thumb may be: it's going to be slow going from a numpy array to a regular old python list or tuple, back to a numpy array. If your data is a python list already, than np.array(list) is a fine choice. >> def useAsArray(xCoords, yCoords): >> >> return NUM.asarray(zip(xCoords, yCoords)) Here are some of the issues with this one: zip unpacks two generic python sequences and then put the items into tuple, then puts them in a list. Essentially this: new_list = [] for i in range(len(xCoords)): new_list.append((xCoords[i], yCoords[i])) In each iteration of that loop, it's indexing into the numpy arrays, making a python object out of them, putting them into a tuple, and appending that tuple to the list, which may have to re-allocate memory a few times. Then the np.array() call loops through that list, unpacks each tuple, examines the python object, decides what it is, and turn it into a raw c-type to put into the array. whereas: def useEmpty(xCoords, yCoords): out = np.empty((len(xCoords), 2), dtype=xCoords.dtype) out[:,0] = xCoords out[:,1] = yCoords return out allocates an array the right size. directly copies the data from xCoords and yCoords to it. that's it. You can see why it's so much faster! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion