From jh at oobleck.astro.cornell.edu Wed Jan 2 11:28:14 2002 From: jh at oobleck.astro.cornell.edu (Joe Harrington) Date: Wed Jan 2 11:28:14 2002 Subject: [Numpy-discussion] RPMs out of date, have problems Message-ID: <200201021929.g02JTKF30031@oobleck.astro.cornell.edu> Hi folks, Problem: The latest Numeric release on the web site is 20.3. The latest with an RPM is 20.1, and that RPM has a problem: it creates a directory in the system root directory. Paul D. says he will implement a solution but doesn't have the experience with RPMs (or the time) to find the problem quickly. I haven't dealt with building Python packages or distutils (is distutils a separate thing or part of Python?) at all. Can someone with the relevant experience fix the current problem and help Paul implement the solution so he can post current RPMs that install right? Ditto anyone who knows how to make packages for Debian, Solaris, and other popular package managers. Rationale: As I've mentionned previously, I'm getting an increasing number of queries from astronomers who want to play with Numeric. At this stage many of the converts will be application code contributors who will help build a library of discipline-specific routines. In talking to these people, I am finding them less than patient with the good 'ol tarball (a position I take myself, following the experience of maintaining the Clue Files, see ftp://oobleck.astro.cornell.edu/pub/clues.tar.gz). To them, it's not serious software if it isn't prepared under their system's installation manager. We need these (very) early adopters, so I think that having a current Numeric RPM for i386 Linux (and the equivalent for i386 Debian GNU/Linux and Solaris Sparc architectures, if someone knows how to build them) would be a Good Thing. Trivial install -> more users, more users -> more volunteers and more contributed code. Also, it would be more consistent with the RPM naming scheme to call the RPM "python-Numeric" (or "python-numeric", or even "numpy") rather than just "Numeric". If that's hard or philosophically undesirable, don't bother, but the name has changed a few times, so I hope it isn't a big deal. Sysadmins have to deal with more than 1000 packages now, and knowing what a package is just by looking at the name is a big help. Also, you can do things like 'rpm -qa | grep python' and get a list of all the python-related packages on your system. "Numeric" is too general outside the context of Python. All of the above goes for Numarray, when its developers are ready for the community at large to start writing code that uses it. Thanks, --jh-- From ransom at physics.mcgill.ca Wed Jan 2 11:37:08 2002 From: ransom at physics.mcgill.ca (Scott Ransom) Date: Wed Jan 2 11:37:08 2002 Subject: [Numpy-discussion] RPMs out of date, have problems In-Reply-To: <200201021929.g02JTKF30031@oobleck.astro.cornell.edu> References: <200201021929.g02JTKF30031@oobleck.astro.cornell.edu> Message-ID: Hello All, Debian (unstable -- which is actually quite stable) provides slightly more up-to-date packages of numeric (20.2) and its extensions. You can find them here: http://packages.debian.org/unstable/math/python-numeric.html http://packages.debian.org/unstable/interpreters/python-numeric-ext.html For a quick hack at a more up-to-date RPM, it _might_ be possible to use alien to convert the *.debs to *.rpms... Scott PS: As an astronomer myself, I am also seeing an increasing interest in my collegues towards python and numeric (especially since I keep preaching the Python gospel to them ;) On January 2, 2002 02:29 pm, Joe Harrington wrote: > Hi folks, > > Problem: > > The latest Numeric release on the web site is 20.3. The latest with > an RPM is 20.1, and that RPM has a problem: it creates a directory in > the system root directory. Paul D. says he will implement a solution > but doesn't have the experience with RPMs (or the time) to find the > problem quickly. I haven't dealt with building Python packages or > distutils (is distutils a separate thing or part of Python?) at all. > Can someone with the relevant experience fix the current problem and > help Paul implement the solution so he can post current RPMs that > install right? Ditto anyone who knows how to make packages for Debian, > Solaris, and other popular package managers. > > Rationale: > > As I've mentionned previously, I'm getting an increasing number of > queries from astronomers who want to play with Numeric. At this stage > many of the converts will be application code contributors who will > help build a library of discipline-specific routines. In talking to > these people, I am finding them less than patient with the good 'ol > tarball (a position I take myself, following the experience of > maintaining the Clue Files, see > ftp://oobleck.astro.cornell.edu/pub/clues.tar.gz). To them, it's not > serious software if it isn't prepared under their system's > installation manager. We need these (very) early adopters, so I think > that having a current Numeric RPM for i386 Linux (and the equivalent > for i386 Debian GNU/Linux and Solaris Sparc architectures, if someone > knows how to build them) would be a Good Thing. Trivial install -> > more users, more users -> more volunteers and more contributed code. > > Also, it would be more consistent with the RPM naming scheme to call > the RPM "python-Numeric" (or "python-numeric", or even "numpy") rather > than just "Numeric". If that's hard or philosophically undesirable, > don't bother, but the name has changed a few times, so I hope it isn't > a big deal. Sysadmins have to deal with more than 1000 packages now, > and knowing what a package is just by looking at the name is a big > help. Also, you can do things like 'rpm -qa | grep python' and get a > list of all the python-related packages on your system. "Numeric" is > too general outside the context of Python. > > All of the above goes for Numarray, when its developers are ready for > the community at large to start writing code that uses it. > > Thanks, > > --jh-- > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Scott M. Ransom Address: McGill Univ. Physics Dept. Phone: (514) 398-6492 3600 University St., Rm 338 email: ransom at physics.mcgill.ca Montreal, QC Canada H3A 2T8 GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From gvermeul at labs.polycnrs-gre.fr Thu Jan 3 00:20:03 2002 From: gvermeul at labs.polycnrs-gre.fr (Gerard Vermeulen) Date: Thu Jan 3 00:20:03 2002 Subject: [Numpy-discussion] RPMs out of date, have problems In-Reply-To: <200201021929.g02JTKF30031@oobleck.astro.cornell.edu> References: <200201021929.g02JTKF30031@oobleck.astro.cornell.edu> Message-ID: <02010309191300.12231@taco.polycnrs-gre.fr> Hi, Try python setup.py bdist_rpm Works like a charm. The same for the subdirectories/subpackages (maybe after installation of the main RPMs). Gerard PS: it is impossible to provide binary RPMs, because there are too many different Linux distributions. On Wednesday 02 January 2002 20:29, Joe Harrington wrote: > Hi folks, > > Problem: > > The latest Numeric release on the web site is 20.3. The latest with > an RPM is 20.1, and that RPM has a problem: it creates a directory in > the system root directory. Paul D. says he will implement a solution > but doesn't have the experience with RPMs (or the time) to find the > problem quickly. I haven't dealt with building Python packages or > distutils (is distutils a separate thing or part of Python?) at all. > Can someone with the relevant experience fix the current problem and > help Paul implement the solution so he can post current RPMs that > install right? Ditto anyone who knows how to make packages for Debian, > Solaris, and other popular package managers. > > Rationale: > > As I've mentionned previously, I'm getting an increasing number of > queries from astronomers who want to play with Numeric. At this stage > many of the converts will be application code contributors who will > help build a library of discipline-specific routines. In talking to > these people, I am finding them less than patient with the good 'ol > tarball (a position I take myself, following the experience of > maintaining the Clue Files, see > ftp://oobleck.astro.cornell.edu/pub/clues.tar.gz). To them, it's not > serious software if it isn't prepared under their system's > installation manager. We need these (very) early adopters, so I think > that having a current Numeric RPM for i386 Linux (and the equivalent > for i386 Debian GNU/Linux and Solaris Sparc architectures, if someone > knows how to build them) would be a Good Thing. Trivial install -> > more users, more users -> more volunteers and more contributed code. > > Also, it would be more consistent with the RPM naming scheme to call > the RPM "python-Numeric" (or "python-numeric", or even "numpy") rather > than just "Numeric". If that's hard or philosophically undesirable, > don't bother, but the name has changed a few times, so I hope it isn't > a big deal. Sysadmins have to deal with more than 1000 packages now, > and knowing what a package is just by looking at the name is a big > help. Also, you can do things like 'rpm -qa | grep python' and get a > list of all the python-related packages on your system. "Numeric" is > too general outside the context of Python. > > All of the above goes for Numarray, when its developers are ready for > the community at large to start writing code that uses it. > > Thanks, > > --jh-- > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From hinsen at cnrs-orleans.fr Fri Jan 4 02:32:02 2002 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Fri Jan 4 02:32:02 2002 Subject: [Numpy-discussion] RPMs out of date, have problems References: <200201021929.g02JTKF30031@oobleck.astro.cornell.edu> Message-ID: <200201041032.g04AWQg08269@localhost.localdomain> Joe Harrington writes: > The latest Numeric release on the web site is 20.3. The latest with > an RPM is 20.1, and that RPM has a problem: it creates a directory in > the system root directory. Paul D. says he will implement a solution > but doesn't have the experience with RPMs (or the time) to find the > problem quickly. I haven't dealt with building Python packages or If someone can give a precise description of the problem, I'll look at it, I think I have done enough RPMs to find my way... The RPMs generated by Distutils are often good enough, but sometimes need some manual cleanup. > distutils (is distutils a separate thing or part of Python?) at all. It is a part of Python since Python 1.6. > installation manager. We need these (very) early adopters, so I think > that having a current Numeric RPM for i386 Linux (and the equivalent I agree in principle, but providing Linux binary RPMs is becoming more and more messy: it depends on the Python version and on the distribution, sometimes even the distribution version number. That's a lot of RPMs. Source RPMs are easier, with some care they should work everywhere, but installation requires a compiler plus some of the "-devel" packages that not everyone has. > Also, it would be more consistent with the RPM naming scheme to call > the RPM "python-Numeric" (or "python-numeric", or even "numpy") rather > than just "Numeric". If that's hard or philosophically undesirable, Agreed. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From jh at oobleck.astro.cornell.edu Fri Jan 4 10:25:04 2002 From: jh at oobleck.astro.cornell.edu (Joe Harrington) Date: Fri Jan 4 10:25:04 2002 Subject: [Numpy-discussion] RPMs out of date, have problems In-Reply-To: <200201041032.g04AWQg08269@localhost.localdomain> (message from Konrad Hinsen on Fri, 4 Jan 2002 11:32:26 +0100) References: <200201021929.g02JTKF30031@oobleck.astro.cornell.edu> <200201041032.g04AWQg08269@localhost.localdomain> Message-ID: <200201041824.g04IODa15381@oobleck.astro.cornell.edu> In the exchange below, my query is >>, Gerard's response is >, and my present response to that is unquoted. >>From: Gerard Vermeulen >>Subject: Re: [Numpy-discussion] RPMs out of date, have problems >>Date: Fri, 4 Jan 2002 09:44:21 +0100 >> Could you check that it produces an RPM with files that go in >> /usr/lib/python2.1 rather than /pcmdi/dubois/linux/lib/python2.1 and >> /usr/include/python2.1 rather than /pcmdi/dubois/linux/include/python2.1? >Yes, mine go into /usr/lib/python2.1 >> Paul D. isn't sure why /pcmdi is being created. It's probably that >> way on his machine and he doesn't know how to tell distutils to use a >> different default prefix. Neither do I. That is the issue. Do you? >Well, the paths are figured out in /usr/lib/python2.1/distutils/sysconfig.py, >using sys.prefix or sys.exec_prefix. >So, I think that Paul's python has been build with >./configure --prefix=/pcmdi/dubois/linux >and that it still lives there, or has been moved to /usr afterwards. >Or that it resides on an NFS mounted volume??? >Or some python startup file that messes sys.(exec_)prefix. Ok, Paul, Konrad, is this what you need? >Anyhow, I think that with respect to binary packages >Windows has the edge over Linux (I know from experience, >because I am author of a Python wrapper to Qwt - a Qt library >allowing to do interactive plotting and it is SO easy to >produce a Windows). >Every combination of Distro-Version-Python-Version would >require its own binary RPM. If you consider Microsoft Windows one OS, Red Hat Linux another, Debian GNU/Linux a third, and so on, the distinction goes away. The fact that Microsoft has a monopoly is definitely convenient to producers. However, as users, we lose and shouldn't support it. >(A) IMHO, it is better to add a README.DIY.RPM: >(I am going through the steps based on Numeric-20.2.1) >(1) unpack Numeric-20.2.1.tar.gz (or whatever) >(2) cd Numeric-20.2.1 >(2) python setup.py bdist_rpm >(3) cd dist >(4) rpm -Uvh Numeric-20.2.1-1.i?86.rpm (as root, i?86 could be another >architecture) >But it is not so easy to build the extension packages (FFT & friends) like >this, without knowledge of RPM. I could try to hack setup.py to make it >compatible with RPM (means making 1 big package without subpackages) >(B) The other possibility is to include a generic python-numeric.spec file >(you build an RPM with rpm -ba python-numeric.spec). I can adapt mine. >Gerard >PS: do you have a RPM based Linux, yourself? Yes, I run Red Hat. So do the vast majority of US astronomers I have spoken to. I understand that SuSE is equally popular in Europe. A few other dists just copy one of them and add a few packages, and so don't need separate RPMs of their own. If we configure RPMs with the version of python included in these dists, we're not talking about a large number of packages, after all, to get 98% of the users or better. I think you are onto a solution: We build RPMs (and .deb files for Debian, if we think there are enough Debian users to make it worthwhile, and the same for Solaris if it uses a different package manager) for the current release of each popular distribution, with that release's python. That's maybe 4 binary packages per Numeric release (Red Hat, SuSE, Debian, Solaris). We also make a source RPM and distribute with it instructions for building a new binary RPM with appropriately-named version ID (e.g., python-numeric-20.4py2.2glibc6-1). We solicit users of unsupported combinations of OS and Python version to contribute those RPMs along with a simple ASCII form specifying what kind of system made it, contact info, etc. I predict we will not get a huge number of such submissions from outside this list, as the vast majority of Numeric users who upgrade their Python when a new Python release comes out (as opposed to a new OS release) are developers who don't care about the RPM anyway. The ultimate solution is to get this thing incorporated into the python core. --jh-- From paul at pfdubois.com Fri Jan 4 10:58:03 2002 From: paul at pfdubois.com (Paul F. Dubois) Date: Fri Jan 4 10:58:03 2002 Subject: [Numpy-discussion] RPMs out of date, have problems In-Reply-To: <200201041824.g04IODa15381@oobleck.astro.cornell.edu> Message-ID: <001101c19551$37d3c6c0$0c01a8c0@freedom> I can confirm that the Python I use is in /pcmdi/dubois/python. I *never* build into /usr or /usr/local. There are lots of people like me that have multiple pythons around with other stuff added in. I suppose what was bothering people was having to use --prefix because the default will not work because my Python is not in some standard place (said standard place seeming to me to be a mythical location given how I see people actually working). I continue to be less than impressed by RPMs. Every time I try one I get version conflicts with other software. From rossini at blindglobe.net Fri Jan 4 11:30:01 2002 From: rossini at blindglobe.net (A.J. Rossini) Date: Fri Jan 4 11:30:01 2002 Subject: [Numpy-discussion] RPMs out of date, have problems In-Reply-To: References: <200201021929.g02JTKF30031@oobleck.astro.cornell.edu> Message-ID: <87itahapbo.fsf@jeeves.blindglobe.net> >>>>> "SR" == Scott Ransom writes: SR> Hello All, Debian (unstable -- which is actually quite stable) SR> provides slightly more up-to-date packages of numeric (20.2) SR> and its extensions. You can find them here: SR> http://packages.debian.org/unstable/math/python-numeric.html SR> http://packages.debian.org/unstable/interpreters/python-numeric-ext.html Nothing like 2 days to make it now 20.3 in Debian unstable/Sid. best, -tony -- A.J. Rossini Rsrch. Asst. Prof. of Biostatistics U. of Washington Biostatistics rossini at u.washington.edu FHCRC/SCHARP/HIV Vaccine Trials Net rossini at scharp.org -------------- http://software.biostat.washington.edu/ -------------- FHCRC: M-W: 206-667-7025 (fax=4812)|Voicemail is pretty sketchy/use Email UW: T-Th: 206-543-1044 (fax=3286)|Change last 4 digits of phone to FAX Rosen: (Mullins' Lab) Fridays, and I'm unreachable except by email. From jh at oobleck.astro.cornell.edu Fri Jan 4 12:14:30 2002 From: jh at oobleck.astro.cornell.edu (Joe Harrington) Date: Fri Jan 4 12:14:30 2002 Subject: [Numpy-discussion] RPMs out of date, have problems In-Reply-To: <001101c19551$37d3c6c0$0c01a8c0@freedom> (paul@pfdubois.com) References: <001101c19551$37d3c6c0$0c01a8c0@freedom> Message-ID: <200201042009.g04K9ha15764@oobleck.astro.cornell.edu> > I *never* build into /usr or /usr/local. Good! You should leave /usr for your package manager to deal with. However, you should *release* software to install there, which you can with RPM without having to build it in /usr. /usr/local is yours to play with. The only thing in mine is IDL. (...which is why I'd like to delete it!) > I suppose > what was bothering people was having to use --prefix because the default > will not work because my Python is not in some standard place (said > standard place seeming to me to be a mythical location given how I see > people actually working). There was no documentation anywhere in the download area suggesting the use of --prefix. Even if there were, RPMs get collected and put in archives (rpmfind.net) without any other associated docs. They should always install into the system following the Linux file system standards. Nobody expects to have to configure an RPM. They're just supposed to work, and the vast majority just do. > There are lots of people like me > that have multiple pythons around with other stuff added in. There are lots if *developers* like you. The current group of Numpy users is demographically distinct from the future user base, if Numpy becomes a player for numerical work in science and engineering. We will have many users who have never written a line of C or C++ or FORTRAN and who will run screaming at the mention of the word "tar". We will have high school teachers and students. We will have lots of people who have zero time and zero tolerance for learning unrelated topics like system administration or software tweaking. If we don't care about them, they won't use our software. > I continue to be less than impressed by RPMs. Every time I try one I get > version conflicts with other software. I understand your frustration with the bad RPMs, but it's not the fault of RPM, it's the fault of either the package developer or the person installing the package. If the package developer links against a library that is itself under rapid development, there is a problem waiting to happen. It's worse if that library is included in some or all distributions. The simple solution is to statically link that library, or rename it and include it in the package. If the user has an out-of-date system, it's also trouble. If we build against the C library and python that are in the OS releases, we're in good shape. I don't think we need to bend over backwards to provide RPMs for people who install their own Python, unless a new version of Numeric requires use of an updated Python RPM. Very few (maybe 0.5%?) "mass-market" users will install their own Python. The experts who do can handle a tarball install, or one will build an RPM and distribute it if it's a particularly important release. The conflict problems pale in comparison to the problems with manually building all packages you want to install, particularly if you don't know how to fix makefile problems or don't have a compiler. If you're only installing a few packages, it's not that big a deal. If you're installing 90, it's a month's work by hand or a day with RPM, if you don't know the install procedure for the software beforehand. See ftp://oobleck.astro.cornell.edu/pub/clues.tar.gz for my attempt to tame this problem. As you can see from the dates of the files therein, I abandonned maintaining each package page as it came out under RPM. It was just impossible to keep up. It's trivial under RPM. Plus, I can uninstall! What matters is what the general users want, and they have spoken loudly in favor of package managers. Konrad writes: > RedHat > is rather conservative among the Linux distributors, but that is also > the reason for still using Python 1.5 in their latest versions. /usr/bin/python2.1 exists under 7.2, and claims in its startup that it existed in 7.1. You get 1.5.2 if you just type "python", but you get 2.1.1 if you type "python2.1". alias python python2.1... --jh-- From hinsen at cnrs-orleans.fr Fri Jan 4 13:45:02 2002 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Fri Jan 4 13:45:02 2002 Subject: [Numpy-discussion] RPMs out of date, have problems References: <001101c19551$37d3c6c0$0c01a8c0@freedom> <200201042009.g04K9ha15764@oobleck.astro.cornell.edu> Message-ID: <200201042144.g04LiqR01772@localhost.localdomain> Joe Harrington writes: > RPM. Plus, I can uninstall! What matters is what the general users > want, and they have spoken loudly in favor of package managers. I couldn't agree more. I tend to make many of my own RPMs, and still I prefer RPMs to straightforward installation in /usr/local. Clean updating and uninstalling is a big advantage. > /usr/bin/python2.1 exists under 7.2, and claims in its startup that it > existed in 7.1. You get 1.5.2 if you just type "python", but you get > 2.1.1 if you type "python2.1". alias python python2.1... Definitely not in 7.1, there was not Python 2.1 at all (before I installed it by hand). Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From tneidt at fidnet.com Sat Jan 5 02:40:04 2002 From: tneidt at fidnet.com (Tod M. Neidt) Date: Sat Jan 5 02:40:04 2002 Subject: [Numpy-discussion] Obscure setup question Message-ID: <1010227506.1141.6.camel@Q.neidt.net> Hi! I have noticed an interesting difference when installing Numeric-20.3 against python-2.1 vs. python-2.2. When installing against python-2.1, setup trys to write to /usr/lib/python2.1/distutils/command/install_headers.pyc But when installing against python-2.2, this doesn't happen. Note: this question came up because I am making an ebuild (basically a compile and install script) for Gentoo linux, http://www.gentoo.org. The attempted write to install_headers.pyc with python-2.1 violates Gentoo's sandboxed build environment causing the merge to fail. If you are interested see http://bugs.gentoo.org/show_bug.cgi?id=21 Just curious if there is a known explanation for this. The simple fix on my end, is to set the dependency >=python-2.2. Thanks for the great program! tod From gvermeul at labs.polycnrs-gre.fr Sat Jan 5 06:25:02 2002 From: gvermeul at labs.polycnrs-gre.fr (gvermeul at labs.polycnrs-gre.fr) Date: Sat Jan 5 06:25:02 2002 Subject: [Numpy-discussion] Obscure setup question Message-ID: <200201051424.g05EOCS26584@labs.polycnrs-gre.fr> Hi, This is very strange. I am also making packages (RPMs in my case). I do this as a normal user without write access to /usr/..... If running setup.py would have as a consequence that /usr/lib/....../install_headers.pyc would get overwritten, then setup.py would die. Can you tell if your python-2.1 is really 2.1 or 2.1.1? It happens that I am tweaking the setup.py script of Numeric and testing with python-2.1.1 and I am sure that distutils works. I suggest the following test: login as a normal user, unpack Numeric, cd Numeric and do python2.1 setup.py install --root=~/tmp This will install Numeric in ~/tmp/usr/lib/python2.1 and you will see if setup.py still tries to overwrite install_headers.pyc If so, your python2.1 is broken for some reason, if not your build system and/or sandbox are broken. Gerard > Hi! > > I have noticed an interesting difference when installing Numeric-20.3 > against python-2.1 vs. python-2.2. > > When installing against python-2.1, setup trys to write to > /usr/lib/python2.1/distutils/command/install_headers.pyc > > But when installing against python-2.2, this doesn't happen. > > Note: this question came up because I am making an ebuild (basically a > compile and install script) for Gentoo linux, http://www.gentoo.org. > The attempted write to install_headers.pyc with python-2.1 violates > Gentoo's sandboxed build environment causing the merge to fail. If you > are interested see > > http://bugs.gentoo.org/show_bug.cgi?id=21 > > Just curious if there is a known explanation for this. The simple fix > on my end, is to set the dependency >=python-2.2. > > Thanks for the great program! > > tod > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > --------------------------------------------- This message was sent using Endymion MailMan. http://www.endymion.com/products/mailman/ From gvermeul at labs.polycnrs-gre.fr Sat Jan 5 07:38:02 2002 From: gvermeul at labs.polycnrs-gre.fr (gvermeul at labs.polycnrs-gre.fr) Date: Sat Jan 5 07:38:02 2002 Subject: [Numpy-discussion] RPMs out of date, have problems Message-ID: <200201051536.g05FawS31545@labs.polycnrs-gre.fr> > > > I continue to be less than impressed by RPMs. Every time I try one I get > > version conflicts with other software. > Agreed, I always use RPMs because they help me to keep my systems in a reversible state. BUT, I never install binary RPMs that are not taylored to the Distro-Version that I am running (Try to install RH-7.2 packages on a RH-7.0). I may take source RPMs from other distributions as a starting point for my own tweaks, but I NEVER install 'foreign' RPMs. Attached you'll find a replacement for the setup.py script allowing to build a single binary and source RPM for Numeric + subpackages. The README.RPM for a normal user would be something like: (1) unpack python-numeric-XXX.YY.Z.tar.gz (2) cd python-numeric-XXX.YY.Z (3) python setup.py bdist_rpm This will build a binary and source RPM adapted to YOUR system (even yours, Paul, with your non-standard Python location) in the subdirectory dist (to be created automatically) (4) install with rpm -Uvh python-numeric-XXX.YY.Z-1.i?86.rpm You can always uninstall with rpm -e python-numeric. IMHO, this is a better way than trying to distibute binary RPMs. It happens that my favorite distribution is not on JH's list :-( But also, http://www.mandrakesoft.com/products/81/gaming-edition , the favorite toy of 11 years old whizkids, doing pygame-NumPy programming and who are certainly future students of astronomy. If you want to test the script: (1) replace the old setup.py with mine (2) rename Lib/Numeric.py to Lib/__init__.py (Numeric will become a real Python package). (3) python setup.py bdist (4) and install the RPM Caveats: (1) renaming Numeric to python-numeric has as consequence that the header files go into .../include/python2.1/python-numeric instead of .../include/python2.1/Numeric. Personally I don't like it (being author of a package that builds on Numeric). (I think this can be fixed, if needed). (2) I tried to acknowledge Paul Dubois' contributions in a long_description (the subpackages do not build to separate packages anymore). (3) The setup.py script shows how to add documentation to an RPM (requires to work around a bug in distutils). Personnally, I think that HTML and PDF documentation should also be added. In the current state of distutils this requires inclusion of the documentation in the python-numeric-XXX.YY.Z.tar.gz. Gerard --------------------------------------------- This message was sent using Endymion MailMan. http://www.endymion.com/products/mailman/ -------------- next part -------------- A non-text attachment was scrubbed... Name: setup.py.gz Type: application/x-gzip Size: 2553 bytes Desc: not available URL: From hinsen at cnrs-orleans.fr Sat Jan 5 09:00:02 2002 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Sat Jan 5 09:00:02 2002 Subject: [Numpy-discussion] RPMs out of date, have problems References: <200201051536.g05FawS31545@labs.polycnrs-gre.fr> Message-ID: <200201051659.g05Gxvf01911@localhost.localdomain> gvermeul at labs.polycnrs-gre.fr writes: > Caveats: > (1) renaming Numeric to python-numeric has as consequence that > the header files go into .../include/python2.1/python-numeric > instead of .../include/python2.1/Numeric. Personally I don't > like it (being author of a package that builds on Numeric). Me neither. But the fix is simple: take the distutils-generated spec file, edit the package name, and put it into the tar ball. The added advantage is that the binary RPM can be generated by rpm --rebuild numpy_xxx.tar.gz. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From gvermeul at labs.polycnrs-gre.fr Sat Jan 5 15:33:09 2002 From: gvermeul at labs.polycnrs-gre.fr (gvermeul at labs.polycnrs-gre.fr) Date: Sat Jan 5 15:33:09 2002 Subject: [Numpy-discussion] RPMs out of date, have problems Message-ID: <200201052332.g05NWnS28660@labs.polycnrs-gre.fr> > gvermeul at labs.polycnrs-gre.fr writes: > > > Caveats: > > (1) renaming Numeric to python-numeric has as consequence that > > the header files go into .../include/python2.1/python-numeric > > instead of .../include/python2.1/Numeric. Personally I don't > > like it (being author of a package that builds on Numeric). > > Me neither. But the fix is simple: take the distutils-generated spec > file, edit the package name, and put it into the tar ball. The added > advantage is that the binary RPM can be generated by rpm --rebuild > numpy_xxx.tar.gz. > You mean rpm -ta numpy_xxx.tar.gz Disadvantage: by default, the package will go into a distro dependent place (Mandrake, Red Hat and Suse are all different) and the package builder has to have write access to that place (Suse-7.3 grants it, Mandrake not, Red Hat don't know). This invites to building packages as root (DON'T). Building packages as a normal user protects you from erratic writes in /usr, /, ... Remember, we want to satisfy the needs of astronomer newbies. Users like you will allways find a solution. python setup.py bdist_rpm needs only write access to the user's home directory. Of course, adding a generic spec file is the easy 2 minutes solution, but I have choosen not to do that because it will invite bad policy (providing binary RPMs is asking for trouble). IMHO, the current setup.py/setup_all.py are not permitting distutils's way of 'RPM building'. It is better to fix that. I will try to see renaming the package from Numeric to python-numeric allows to install the headers in ../python2.1/Numeric instead of ./python2.1/python-numeric by changing my setup.py file. Gerard --------------------------------------------- This message was sent using Endymion MailMan. http://www.endymion.com/products/mailman/ From hinsen at cnrs-orleans.fr Sun Jan 6 02:25:02 2002 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Sun Jan 6 02:25:02 2002 Subject: [Numpy-discussion] RPMs out of date, have problems In-Reply-To: <200201052332.g05NWnS28660@labs.polycnrs-gre.fr> (gvermeul@labs.polycnrs-gre.fr) References: <200201052332.g05NWnS28660@labs.polycnrs-gre.fr> Message-ID: <200201061024.g06AOsY01387@localhost.localdomain> > Disadvantage: by default, the package will go into a distro dependent place > (Mandrake, Red Hat and Suse are all different) and the package builder > has to have write access to that place (Suse-7.3 grants it, Mandrake not, > Red Hat don't know). This invites to building packages as root (DON'T). RedHat doesn't by default (I changed that on my system). > python setup.py bdist_rpm needs only write access to the user's home > directory. Everywhere? It didn't when I first tried (that was an early distutils release under RedHat 6.2). > Of course, adding a generic spec file is the easy 2 minutes solution, > but I have choosen not to do that because it will invite bad policy > (providing binary RPMs is asking for trouble). And yet many people will not accept anything but binary RPMs, out of ignorance or fear. We won't make everyone happy... I admit that I never looked into fully automatic RPM generation by distutils as I usually need to add some manual steps to the build/install process. For example, distutils doesn't let me specify compiler options, but I want time-critical code to be compiled with the highest optimization level. Hopefully this will be addressed in distutils one day. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From gvermeul at labs.polycnrs-gre.fr Sun Jan 6 06:48:05 2002 From: gvermeul at labs.polycnrs-gre.fr (gvermeul at labs.polycnrs-gre.fr) Date: Sun Jan 6 06:48:05 2002 Subject: [Numpy-discussion] RPMs out of date, have problems Message-ID: <200201061447.g06ElbS26541@labs.polycnrs-gre.fr> > > > python setup.py bdist_rpm needs only write access to the user's home > > directory. > > Everywhere? It didn't when I first tried (that was an early distutils > release under RedHat 6.2). > Can you check which version of Python/distutils. If it is a version python that does not come with distutils, we can count it out. > > > Of course, adding a generic spec file is the easy 2 minutes solution, > > but I have choosen not to do that because it will invite bad policy > > (providing binary RPMs is asking for trouble). > > And yet many people will not accept anything but binary RPMs, out of > ignorance or fear. We won't make everyone happy... > Yes, and they will complain to JH if they are failing to install JH's binary RPMs. The best we can do is explain the motivation of NOT providing binary RPMs. If that is not sufficient for some, we can only advise that they switch to Windows or pester their distro providers for RPMs of the latest NumPy. > > I admit that I never looked into fully automatic RPM generation by > distutils as I usually need to add some manual steps to the > build/install process. For example, distutils doesn't let me specify > compiler options, but I want time-critical code to be compiled with > the highest optimization level. Hopefully this will be addressed in > distutils one day. > There are two ways to do this: (1) Extension( ..., extra_compile_args = [ "-O17" ], ...) This will append the optimization level 17 to the compiler flag (the last -O option counts). (2) export CFLAGS=-O3; python setup.py build This will append -O3 The result of (1) and (2) is gcc .... -017 -03 Gerard PS: distutils may look intimidating at first, but it is really simple compared to automake/autoconf. The disadvantage is that it is not very well tested. --------------------------------------------- This message was sent using Endymion MailMan. http://www.endymion.com/products/mailman/ From hinsen at cnrs-orleans.fr Sun Jan 6 13:34:03 2002 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Sun Jan 6 13:34:03 2002 Subject: [Numpy-discussion] RPMs out of date, have problems In-Reply-To: <200201061447.g06ElbS26541@labs.polycnrs-gre.fr> (gvermeul@labs.polycnrs-gre.fr) References: <200201061447.g06ElbS26541@labs.polycnrs-gre.fr> Message-ID: <200201062134.g06LY1G02583@localhost.localdomain> > Can you check which version of Python/distutils. If it is a version python > that does not come with distutils, we can count it out. I don't remember, and I don't have any of that stuff left. > Yes, and they will complain to JH if they are failing to install JH's > binary RPMs. The best we can do is explain the motivation of NOT > providing binary RPMs. If that is not sufficient for some, we can Somebody else will provide them, and you will still get complaints. I also get them. Fortunately there is a standard answer to send out. > There are two ways to do this: > > (1) Extension( ..., extra_compile_args = [ "-O17" ], ...) > This will append the optimization level 17 to the compiler flag > (the last -O option counts). But that will add the argument for all platforms. I need to specify compilation flags separately for each platform/compiler. I might even want to compile some modules with the new Intel compiler for Linux. > (2) export CFLAGS=-O3; python setup.py build > This will append -O3 For all extension modules - I need to specify different flags for each extension module, in one case even for a single source code file. My current solution is to build everything with distutils and then recompile the critical modules manually in the spec file. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From eric at enthought.com Sun Jan 6 20:29:03 2002 From: eric at enthought.com (eric) Date: Sun Jan 6 20:29:03 2002 Subject: [Numpy-discussion] RPMs out of date, have problems References: <200201061447.g06ElbS26541@labs.polycnrs-gre.fr> <200201062134.g06LY1G02583@localhost.localdomain> Message-ID: <08f601c1972b$3af04fa0$777ba8c0@ericlaptop> > > (1) Extension( ..., extra_compile_args = [ "-O17" ], ...) > > This will append the optimization level 17 to the compiler flag > > (the last -O option counts). > > But that will add the argument for all platforms. I need to specify > compilation flags separately for each platform/compiler. I might even > want to compile some modules with the new Intel compiler for Linux. > > > (2) export CFLAGS=-O3; python setup.py build > > This will append -O3 > > For all extension modules - I need to specify different flags for each > extension module, in one case even for a single source code file. This seems to be one of the big complaints about distutils for scientist (and maybe others). I think it is also one of Paul Dubois' major reasons for not liking distutils for Fortran files. I've run into this occasionally too and wished for a dictionary argument, maybe file_special_instructions, that had file names as keys and a dictionary of options for that file as the value. I think this might go some distance to solving the problem. file_special_instructions = { 'really_fast.c': { compiler = 'icc', override_compiler_flags = ['O2', 'tpp7'] } 'sorta_fast.c': { extra_compile_args = ['-O2']} } So an extension module that used this and had 'slow.c', 'really_fast.c', and 'kinda_fast.c' would use the standard compile tools for 'slow.c', use the specified compiler with the specified flags for 'really_fast.c', and add the specified compiler flags to the standard flags for 'sorta_fast.c'. This approach would provide quite a bit of flexiblity. To handle platform specific versions of file_special_instructions, I think you'd have to use if/thens within the setup.py file to build a separate set of instructions for each platform. There is one other major issue I've run into with distutils that I haven't found a way around. Sometimes you need compiler/linker flags inter-mingled with source or library files in a certain way. This can occur on SunOS when you want to link statically to some libraries and dynamically to others. distutils just doesn't provide a way of doing this that I have found. SciPy now has a package of extensions/changes to distutils called scipy_distutils helpful for building fortran based extension modules and other things needed by SciPy. We could experiment with adding something like the file_special_instructions flag if others thought it useful. Later, it could be folded back into distutils if the rest of the community wanted it. Thoughts? eric From paul at pfdubois.com Sun Jan 6 22:09:06 2002 From: paul at pfdubois.com (Paul F. Dubois) Date: Sun Jan 6 22:09:06 2002 Subject: distutils for scientific stuff, was RE: [Numpy-discussion] RPMs out of date, have problems In-Reply-To: <200201062134.g06LY1G02583@localhost.localdomain> Message-ID: <000901c19741$4c90a1e0$0c01a8c0@freedom> About doing different things in distutils for different platforms, that is easy. It is a Python script so you can set things depending on sys.platform. E.g. from Numerical, # You might need to add a case here for your system if sys.platform == 'win32': mathlibs = [] define_macros = [] undef_macros = ['HAVE_INVERSE_HYPERBOLIC'] elif sys.platform == 'beos5': mathlibs = [] Later the setup call uses these variables in its argument list. Adding Fortran support is quite a bit more difficult because the whole idea of distutils is to piggy-back off the Python configure, which doesn't configure Fortran compiler options or paths. I don't think distutils ought to try, really. You just compile the Fortran into a library and then use that in your setup.py. I think a possible way out is scons, but that is just a preliminary impression. It bears a strong resemblance to the system I was working on in 1998 and abandoned when I changed jobs. My theory was that the build should be rule-based, with finer and finer rules for special cases or platforms. The highest priority rule that governs a particular file, wins. E.g., Rule 1: To compile a .c file, do cc -O ... Rule 2: To compile foo.c, do cc -O3 ... Rule 3: To compile foo.c on platform win32, do cc -O1 ... Rule 4: compile bar.c but only on platform win32 There was more to it, because one of the tricky points is that nowadays files produce multiple outputs per execution and may need to be processed in more than one way. Note that if you add a new .c file or a new platform, you're covered unless it needs special treatment. Anyway, this is actually not a design area the numerical community ought to try to get into; there are people who have spent a lot of time on this already. Given that sentence, I can't explain why I was doing it, other than that I hate make so badly I was driven to it. The project was called MMD: Make Must Die. From pearu at cens.ioc.ee Sun Jan 6 23:02:04 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Sun Jan 6 23:02:04 2002 Subject: distutils for scientific stuff, was RE: [Numpy-discussion] RPMs out of date, have problems In-Reply-To: <000901c19741$4c90a1e0$0c01a8c0@freedom> Message-ID: On Sun, 6 Jan 2002, Paul F. Dubois wrote: > Adding Fortran support is quite a bit more difficult because the whole > idea of distutils is to piggy-back off the Python configure, which > doesn't configure Fortran compiler options or paths. I don't think > distutils ought to try, really. You just compile the Fortran into a > library and then use that in your setup.py. Saying that "just compile the Fortran into .." sounds like a common step for using Fortran, then I don't see why distutils shouldn't try to support this. As Eric mentioned in the previous message, scipy_distutils already does this step for an intermediate user. We have tested this to work for different compilers like Gnu,Intel F90,VAST,NAG F95,Absoft,etc and it works like a charm. Regards, Pearu From rob at pythonemproject.com Mon Jan 7 09:31:08 2002 From: rob at pythonemproject.com (Rob) Date: Mon Jan 7 09:31:08 2002 Subject: distutils for scientific stuff, was RE: [Numpy-discussion] RPMsout of date, have problems References: Message-ID: <3C39DA94.827E90EA@pythonemproject.com> Please when writing these utilities keep us BSD users in mind. I had to severely hack SciPy to get it to compile on FreeBSD, part of which was the Fortran utility. Rob. Pearu Peterson wrote: > > On Sun, 6 Jan 2002, Paul F. Dubois wrote: > > > Adding Fortran support is quite a bit more difficult because the whole > > idea of distutils is to piggy-back off the Python configure, which > > doesn't configure Fortran compiler options or paths. I don't think > > distutils ought to try, really. You just compile the Fortran into a > > library and then use that in your setup.py. > > Saying that "just compile the Fortran into .." sounds like a common step > for using Fortran, then I don't see why distutils shouldn't try to support > this. As Eric mentioned in the previous message, scipy_distutils already > does this step for an intermediate user. We have tested this to work for > different compilers like Gnu,Intel F90,VAST,NAG F95,Absoft,etc and it > works like a charm. > > Regards, > Pearu > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- The Numeric Python EM Project www.pythonemproject.com From bsder at allcaps.org Mon Jan 7 22:19:05 2002 From: bsder at allcaps.org (Andrew P. Lentvorski) Date: Mon Jan 7 22:19:05 2002 Subject: [Numpy-discussion] Sparse matricies Message-ID: <20020107221215.R65587-100000@mail.allcaps.org> I just looked through the archives and the web searches and didn't really find anything current on sparse matricies. Is there anything current for Python which handles sparse matrix codes? If someone is looking for a C implementation of sparse matrix handlers, I would refer to the sparse matrix package developed at Berekeley which is part of the Spice 3f4 circuit simulator (released under a modified old-style BSD license). I can provide pointers if they are desired. My goal is to actually create a useful circuit simulator wholly in Python so that it would be useful for both teaching and modification in special cases. Andy L. From oliphant at ee.byu.edu Tue Jan 8 08:01:12 2002 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Jan 8 08:01:12 2002 Subject: [Numpy-discussion] Sparse matricies In-Reply-To: <20020107221215.R65587-100000@mail.allcaps.org> Message-ID: > > Is there anything current for Python which handles sparse matrix codes? > > If someone is looking for a C implementation of sparse matrix handlers, I > would refer to the sparse matrix package developed at Berekeley which > is part of the Spice 3f4 circuit simulator (released under a modified > old-style BSD license). I can provide pointers if they are desired. I've done about three Sparse matrix packages built on SPARSEKIT, UMFPACK, and some of my own rolled stuff. SciPy has a subpackage for Sparse matrices, but it is not ready yet. I'd be interested in the package you are talking about. -Travis Oliphant From bsder at allcaps.org Tue Jan 8 18:45:02 2002 From: bsder at allcaps.org (Andrew P. Lentvorski) Date: Tue Jan 8 18:45:02 2002 Subject: [Numpy-discussion] Sparse matricies In-Reply-To: Message-ID: <20020108183038.O67212-100000@mail.allcaps.org> On Tue, 8 Jan 2002, Travis Oliphant wrote: > SciPy has a subpackage for Sparse matrices, but it is not ready yet. Well, I'm in the process of setting up the simulator now. I don't need the sparse matrix package immediately, but if there isn't anything useful in a week or so, I'll have to figure out a workaround. I'd *really* rather not. Even something which isn't quite ready would be better than what I would write (and then have to throw away). > I'd be interested in the package you are talking about. http://www-cad.eecs.berkeley.edu:80/Software/software.html The packge is SPICE, which is an electronic circuit simulator. Spice3f4 in particular. Download the Spice3f4 kit. Unpack it. The entire package is in src/lib/sparse. Andy L. From bsder at allcaps.org Tue Jan 8 18:52:01 2002 From: bsder at allcaps.org (Andrew P. Lentvorski) Date: Tue Jan 8 18:52:01 2002 Subject: [Numpy-discussion] Sparse matricies In-Reply-To: Message-ID: <20020108185024.V67212-100000@mail.allcaps.org> Perhaps next time I should dig on the web a little more ... On Tue, 8 Jan 2002, Travis Oliphant wrote: > I'd be interested in the package you are talking about. Also available as a separate package, http://buffy.eecs.berkeley.edu/IRO/Software/Catalog/Description/sparse1.3.html -a From bsder at allcaps.org Tue Jan 8 19:02:02 2002 From: bsder at allcaps.org (Andrew P. Lentvorski) Date: Tue Jan 8 19:02:02 2002 Subject: [Numpy-discussion] Sparse matricies In-Reply-To: Message-ID: <20020108190019.G67264-100000@mail.allcaps.org> Travis, Here is some of the most recent stuff I've found on sparse matricies. http://buffy.EECS.Berkeley.EDU/IRO/Summary/01abstracts/chapter18index.html http://buffy.EECS.Berkeley.EDU/IRO/Summary/01abstracts/ejr.1.html Andy L. From rob at pythonemproject.com Thu Jan 10 06:49:16 2002 From: rob at pythonemproject.com (Rob) Date: Thu Jan 10 06:49:16 2002 Subject: [Numpy-discussion] taking on new EM simulator project- ASAP Message-ID: <3C3DA93C.6AD46637@pythonemproject.com> This one will be interesting. I'm going to convert the Fortran ASAP MoM simulator to Numpy. I've already started and its bringing back bad memories of punch cards, etc. ASAP is a viable alternative to NEC2 for wire antennas, as it uses the Sommerfield conditions for imperfect ground. And its not too long of a program to port. The heavy stuff will be I/O. Fortunately, I can use f2c to decode the format statements :) Rob. -- The Numeric Python EM Project www.pythonemproject.com From pearu at cens.ioc.ee Thu Jan 10 07:09:02 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Thu Jan 10 07:09:02 2002 Subject: [Numpy-discussion] taking on new EM simulator project- ASAP In-Reply-To: <3C3DA93C.6AD46637@pythonemproject.com> Message-ID: On Thu, 10 Jan 2002, Rob wrote: > This one will be interesting. I'm going to convert the Fortran ASAP MoM > simulator to Numpy. I've already started and its bringing back bad > memories of punch cards, etc. ASAP is a viable alternative to NEC2 for > wire antennas, as it uses the Sommerfield conditions for imperfect > ground. And its not too long of a program to port. The heavy stuff > will be I/O. Fortunately, I can use f2c to decode the format statements > :) But why don't you use Fortran/Python binding tools like pyfort or f2py? Pearu From rob at pythonemproject.com Thu Jan 10 12:23:31 2002 From: rob at pythonemproject.com (rob at pythonemproject.com) Date: Thu Jan 10 12:23:31 2002 Subject: [Numpy-discussion] taking on new EM simulator project- ASAP Message-ID: <200201102024.PAA82239> An embedded and charset-unspecified text was scrubbed... Name: not available URL: From vanandel at atd.ucar.edu Thu Jan 17 06:47:21 2002 From: vanandel at atd.ucar.edu (Joe Van Andel) Date: Thu Jan 17 06:47:21 2002 Subject: [Numpy-discussion] viewing Gridded or Polar data? Message-ID: <3C46E3DC.B1F5F1A2@atd.ucar.edu> Can anyone recommend a visualization package that would display arrays of numbers by mapping values to a set of colors? That is, we define 32 colors, where the first color represents values 0-10, the second color represents 10-20, etc, and then each input value gets displayed as its corresponding color. I'd like to read netCDF files, if possible. Thanks much! -- Joe VanAndel National Center for Atmospheric Research http://www.atd.ucar.edu/~vanandel/ Internet: vanandel at ucar.edu From rob at pythonemproject.com Thu Jan 17 07:09:06 2002 From: rob at pythonemproject.com (Rob) Date: Thu Jan 17 07:09:06 2002 Subject: [Numpy-discussion] viewing Gridded or Polar data? References: <3C46E3DC.B1F5F1A2@atd.ucar.edu> Message-ID: <3C46E847.FC885A0C@pythonemproject.com> Animabob does that but I don't think it specificially handles NetCDF files. You can define your own color mapping file with Animabob. OpenDX has that capasbility. I've tried OpenDX, but all versions FreeBSD, and Cygwin that I tried dumped core upon trying to plot. Perhaps they've fixed that- it was fixed in CVS but not in the dist files. Rob. Joe Van Andel wrote: > > Can anyone recommend a visualization package that would display arrays > of numbers by mapping values to a set of colors? That is, we define 32 > colors, where the first color represents values 0-10, the second color > represents 10-20, etc, and then each input value gets displayed as its > corresponding color. > > I'd like to read netCDF files, if possible. > > Thanks much! > > -- > Joe VanAndel > National Center for Atmospheric Research > http://www.atd.ucar.edu/~vanandel/ > Internet: vanandel at ucar.edu > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- The Numeric Python EM Project www.pythonemproject.com From jsaenz at wm.lc.ehu.es Thu Jan 17 07:59:23 2002 From: jsaenz at wm.lc.ehu.es (Jon Saenz) Date: Thu Jan 17 07:59:23 2002 Subject: [Numpy-discussion] viewing Gridded or Polar data? In-Reply-To: <3C46E3DC.B1F5F1A2@atd.ucar.edu> Message-ID: I have never tried exactly this type of plot, but I guess that a combination of GrADS commands like the following ones in a script: grads -l << EOF sdfopen yourfile.nc (for COARDS-compliant netCDF files) set gxout grfill (to get a colour-filled grid) set clevs 10 20 30 40 50 (and so on to define your intervals...) d variable_name (of the netCDF file) enable print your_plot_name.gm print disable print quit EOF gxeps -c your_plot_name should do the work and prepare an EPS file. GrADS handles netCDF files, in case they conform to the COARDS standard. Otherwise, you can define the data in the file and use the xdfopen command, but this is more complex than above. http://grads.iges.org is their homepage. Hope this helps. Jon Saenz. | Tfno: +34 946012445 Depto. Fisica Aplicada II | Fax: +34 944648500 Facultad de Ciencias. \\ Universidad del Pais Vasco \\ Apdo. 644 \\ 48080 - Bilbao \\ SPAIN On Thu, 17 Jan 2002, Joe Van Andel wrote: > Can anyone recommend a visualization package that would display arrays > of numbers by mapping values to a set of colors? That is, we define 32 > colors, where the first color represents values 0-10, the second color > represents 10-20, etc, and then each input value gets displayed as its > corresponding color. > > I'd like to read netCDF files, if possible. > > Thanks much! > > -- > Joe VanAndel > National Center for Atmospheric Research > http://www.atd.ucar.edu/~vanandel/ > Internet: vanandel at ucar.edu > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From ransom at spock.physics.mcgill.ca Thu Jan 17 08:01:14 2002 From: ransom at spock.physics.mcgill.ca (Scott Ransom) Date: Thu Jan 17 08:01:14 2002 Subject: [Numpy-discussion] viewing Gridded or Polar data? In-Reply-To: <3C46E3DC.B1F5F1A2@atd.ucar.edu> References: <3C46E3DC.B1F5F1A2@atd.ucar.edu> Message-ID: <20020117160037.GA6390@spock.physics.mcgill.ca> Hi Joe, There are C and Python wrappers around the very nice PGPLOT library that do exactly what you want (and a whole lot more). You can find them here: ftp://cfa-ftp.harvard.edu/pub/ransom The files you need are: ppgplot-0.11.tar.gz (the C wrappers) Pgplot.py (the Python wrappers around the C wrappers) Lots of additional information about the capabilities of PGPLOT can be found at its web page: http://www.astro.caltech.edu/~tjp/pgplot/ With my current setup, you can do a lot of plotting (except 3D stuff) in a manner that is very much like IDL (a single function call with lots of optional keyword arguments). I haven't modified it in quite awhile, but still use it in my own work almost every day. As for reading netCDF, I would recommend Konrad Hinsen's Scientific Python: http://starship.python.net/crew/hinsen/scientific.html Good luck, Scott On Thu, Jan 17, 2002 at 07:46:52AM -0700, Joe Van Andel wrote: > Can anyone recommend a visualization package that would display arrays > of numbers by mapping values to a set of colors? That is, we define 32 > colors, where the first color represents values 0-10, the second color > represents 10-20, etc, and then each input value gets displayed as its > corresponding color. > > I'd like to read netCDF files, if possible. > > Thanks much! > > -- > Joe VanAndel > National Center for Atmospheric Research > http://www.atd.ucar.edu/~vanandel/ > Internet: vanandel at ucar.edu > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- -- Scott M. Ransom Address: McGill Univ. Physics Dept. Phone: (514) 398-6492 3600 University St., Rm 338 email: ransom at physics.mcgill.ca Montreal, QC Canada H3A 2T8 GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From nodwell at physics.ubc.ca Thu Jan 17 08:52:02 2002 From: nodwell at physics.ubc.ca (Eric Nodwell) Date: Thu Jan 17 08:52:02 2002 Subject: [Numpy-discussion] viewing Gridded or Polar data? In-Reply-To: <3C46E3DC.B1F5F1A2@atd.ucar.edu> References: <3C46E3DC.B1F5F1A2@atd.ucar.edu> Message-ID: <20020117165106.GA13128@holler.physics.ubc.ca> You might want to have a look at Gri (http://gri.sourceforge.net/). Gri can read netCDF files. Here's the little blurb from their site: "Gri is a language for scientific graphics programming. The word "language" is important: Gri is command-driven, not point/click. Some users consider Gri similar to LaTeX, since both provide extensive power as a reward for tolerating a learning curve. Gri can make x-y graphs, contour graphs, and image graphs. The output is in PostScript..." Since gri is command-driven, it's easy to call it from within python. Gri won't display the data for you - it will only generate the PostScript file, which you can then display with any number of tools. Eric On Thu, Jan 17, 2002 at 07:46:52AM -0700, Joe Van Andel wrote: > Can anyone recommend a visualization package that would display arrays > of numbers by mapping values to a set of colors? That is, we define 32 > colors, where the first color represents values 0-10, the second color > represents 10-20, etc, and then each input value gets displayed as its > corresponding color. > > I'd like to read netCDF files, if possible. > > Thanks much! > From paul at pfdubois.com Thu Jan 17 09:16:04 2002 From: paul at pfdubois.com (Paul F. Dubois) Date: Thu Jan 17 09:16:04 2002 Subject: [Numpy-discussion] viewing Gridded or Polar data? In-Reply-To: <3C46E3DC.B1F5F1A2@atd.ucar.edu> Message-ID: <000101c19f7a$1a476e80$0c01a8c0@freedom> We will answer you again shortly and include an example of how to do this in cdat (cdat.sf.net). You can do it using our GUI or from the Python command line. cdat includes a colormap editor for getting things just the way you want, and a variety of graphics methods including box fill, isofill, isoline, etc. It can open a wide variety of data files including netcdf. Our name, Climate Data Analysis Tools, is a bit misleading. There are special facilities for climate modelers, and a variety of statistics and other algorithms that they need, but the package is a general-purpose one. -----Original Message----- From: numpy-discussion-admin at lists.sourceforge.net [mailto:numpy-discussion-admin at lists.sourceforge.net] On Behalf Of Joe Van Andel Sent: Thursday, January 17, 2002 6:47 AM To: numpy-discussion Subject: [Numpy-discussion] viewing Gridded or Polar data? Can anyone recommend a visualization package that would display arrays of numbers by mapping values to a set of colors? That is, we define 32 colors, where the first color represents values 0-10, the second color represents 10-20, etc, and then each input value gets displayed as its corresponding color. I'd like to read netCDF files, if possible. Thanks much! -- Joe VanAndel National Center for Atmospheric Research http://www.atd.ucar.edu/~vanandel/ Internet: vanandel at ucar.edu _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From feiguin at magnet.fsu.edu Thu Jan 17 12:47:08 2002 From: feiguin at magnet.fsu.edu (Adrian Feiguin) Date: Thu Jan 17 12:47:08 2002 Subject: [Numpy-discussion] viewing Gridded or Polar data? In-Reply-To: <3C46E3DC.B1F5F1A2@atd.ucar.edu> Message-ID: You can take a look at SciGraphica at http://scigraphica.sourceforge.net. Although it is mainly GUI based, it has a python interface under development. You can ask in the scigraphica mailing list for a plugin for reading netcdf files, I remember someone posting it. Saludos, On Thu, 17 Jan 2002, Joe Van Andel wrote: > Can anyone recommend a visualization package that would display arrays > of numbers by mapping values to a set of colors? That is, we define 32 > colors, where the first color represents values 0-10, the second color > represents 10-20, etc, and then each input value gets displayed as its > corresponding color. > > I'd like to read netCDF files, if possible. > > Thanks much! > > -- > Joe VanAndel > National Center for Atmospheric Research > http://www.atd.ucar.edu/~vanandel/ > Internet: vanandel at ucar.edu > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From paul at pfdubois.com Thu Jan 17 14:09:24 2002 From: paul at pfdubois.com (Paul F. Dubois) Date: Thu Jan 17 14:09:24 2002 Subject: [Numpy-discussion] Please help me test a new setup.py In-Reply-To: <02011719534500.00818@taco.polycnrs-gre.fr> Message-ID: <000201c19fa3$1fb92ec0$0c01a8c0@freedom> Working from something given to me by Gerard Vermeulen I have tried to improve Numeric's setup.py. This is now in CVS. I could use help testing it, particularly by the those whose brains are RPM-enabled. Those Numpy developers who have not done a damn thing lately are granted this chance for redemption. (:-> There is no more setup_all.py; rather, setup.py builds everything. It has, however, a structure that enables easy commenting-out of any optional packages. I have made and tested the Windows packages and installer; and made and tested Linux packages and have made but not tested the Linux RPM. (sh makedist.sh). If trying this on a Python will Numeric already installed, remove first from site-packages: Numeric.pth, Numeric, FFT, RNG, MA, kinds, PropertiedClasses Under the new scheme, Numeric.pth points to Numeric and the directories for the others are INSIDE Numeric. Everything should be backward compatible. I like this; it reduces the visible pollution in site-packages. The RPM produced does not contain the documentation (that runs into a distutils bug in older Pythons). It does not have a name beginning with python- because that screwed up something else. In short, it is a compromise. Once we're all convinced that what I have works then we can worry about the finer stuff. A nice side benefit is that there is now only one Windows .exe installer, and only one thing to uninstall in add/remove programs. From rob at hooft.net Thu Jan 17 23:07:03 2002 From: rob at hooft.net (Rob Hooft) Date: Thu Jan 17 23:07:03 2002 Subject: [Numpy-discussion] Re: viewing Gridded or Polar data? Message-ID: <3C47C941.9000008@hooft.net> I wrote a program to display X-ray diffraction images. The display engine is available as "dis.py" from http://starship.python.net/crew/hooft/ Regards, Rob Hooft -- Rob W.W. Hooft || rob at hooft.net || http://www.hooft.net/people/rob/ From gvermeul at polycnrs-gre.fr Fri Jan 18 00:57:08 2002 From: gvermeul at polycnrs-gre.fr (Gerard Vermeulen) Date: Fri Jan 18 00:57:08 2002 Subject: [Numpy-discussion] Please help me test a new setup.py In-Reply-To: <000201c19fa3$1fb92ec0$0c01a8c0@freedom> References: <000201c19fa3$1fb92ec0$0c01a8c0@freedom> Message-ID: <02011809564200.01995@taco.polycnrs-gre.fr> If I do setup.py bdist_rpm then 'Numeric.pth' does not end up in the binary RPM with Python-2.1.1 The reason is that rpm does something like setup.py install --root=~/tmp --record=record.txt and apparently the distutils part that handles --record is broken as shown by 'grep pth record.txt'. Paul, do you want me to 'fix' distutils (meaning overriding the install command, so that record.txt contains all files)? Or are you willing to use the pth_install_data class that does not work on Windows? Gerard On Thursday 17 January 2002 23:05, Paul F. Dubois wrote: > Working from something given to me by Gerard Vermeulen I have tried to > improve Numeric's setup.py. This is now in CVS. I could use help testing > it, particularly by the those whose brains are RPM-enabled. Those Numpy > developers who have not done a damn thing lately are granted this chance > for redemption. (:-> > > There is no more setup_all.py; rather, setup.py builds everything. It > has, however, a structure that enables easy commenting-out of any > optional packages. > > I have made and tested the Windows packages and installer; and made and > tested Linux packages and have made but not tested the Linux RPM. (sh > makedist.sh). > > If trying this on a Python will Numeric already installed, remove first > from site-packages: > Numeric.pth, Numeric, FFT, RNG, MA, kinds, PropertiedClasses > > Under the new scheme, Numeric.pth points to Numeric and the directories > for the others are INSIDE Numeric. Everything should be backward > compatible. I like this; it reduces the visible pollution in > site-packages. > > The RPM produced does not contain the documentation (that runs into a > distutils bug in older Pythons). It does not have a name beginning with > python- because that screwed up something else. In short, it is a > compromise. Once we're all convinced that what I have works then we can > worry about the finer stuff. > > A nice side benefit is that there is now only one Windows .exe > installer, and only one thing to uninstall in add/remove programs. > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From paul at pfdubois.com Mon Jan 21 20:43:06 2002 From: paul at pfdubois.com (Paul F. Dubois) Date: Mon Jan 21 20:43:06 2002 Subject: [Numpy-discussion] cvs checkins for rpms Message-ID: <000001c1a2fe$d3df3c60$0c01a8c0@freedom> Version 21.0a2 is now in cvs awaiting community testing. Especially to be tested are rpm creation, Windows installer creation. Includes new files setup.cfg and rpm_install.sh from Gerard Vermeulen. From jochen at jochen-kuepper.de Mon Jan 21 21:27:02 2002 From: jochen at jochen-kuepper.de (Jochen =?iso-8859-1?q?K=FCpper?=) Date: Mon Jan 21 21:27:02 2002 Subject: [Numpy-discussion] cvs checkins for rpms In-Reply-To: <000001c1a2fe$d3df3c60$0c01a8c0@freedom> References: <000001c1a2fe$d3df3c60$0c01a8c0@freedom> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mon, 21 Jan 2002 20:39:47 -0800 Paul F Dubois wrote: Paul> Includes new files setup.cfg and rpm_install.sh from Gerard Paul> Vermeulen. ,----[python setup.py bdist_rpm] | [...] | building 'kinds._kinds' extension | gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -IInclude -IPackages/FFT/Include -IPackages/RNG/Include -I/usr/local/include/python2.2 -c Packages/kinds/Src/_kindsmodule.c -o build/temp.linux-i686-2.2/_kindsmodule.o -O2 -march=i386 -mcpu=i686 | Packages/kinds/Src/_kindsmodule.c:15: warning: function declaration isn't a prototype | gcc -shared build/temp.linux-i686-2.2/_kindsmodule.o -o build/lib.linux-i686-2.2/kinds/_kinds.so | MA Version 11.1.0 | Properties Version 2.2 | kinds Version 1.1 | Numeric Version 21.0a2 | + exit 0 | Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.35738 | + umask 022 | + cd /home/software/programming/numeric/NumPy/build/bdist.linux-i686/rpm/BUILD | + cd Numeric-21.0a2 | + setup.py install --root=/var/tmp/Numeric-buildroot | /var/tmp/rpm-tmp.35738: setup.py: command not found | error: Bad exit status from /var/tmp/rpm-tmp.35738 (%install) `---- Greetings, Jochen - -- Einigkeit und Recht und Freiheit http://www.Jochen-Kuepper.de Libert?, ?galit?, Fraternit? GnuPG key: 44BCCD8E Sex, drugs and rock-n-roll -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Processed by Mailcrypt and GnuPG iD8DBQE8TPgIiJ/aUUS8zY4RArEoAJ0SsWQT+F3DNfh+YTaqk8lXpXsbzgCfQe27 GjZ+/DrL9sYbjPj3Ma+VLV4= =17Dr -----END PGP SIGNATURE----- From jochen at jochen-kuepper.de Mon Jan 21 21:42:02 2002 From: jochen at jochen-kuepper.de (Jochen =?iso-8859-1?q?K=FCpper?=) Date: Mon Jan 21 21:42:02 2002 Subject: [Numpy-discussion] cvs checkins for rpms In-Reply-To: References: <000001c1a2fe$d3df3c60$0c01a8c0@freedom> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 22 Jan 2002 00:26:33 -0500 Jochen K?pper wrote: Jochen> ,----[python setup.py bdist_rpm] [...] Jochen> | + setup.py install --root=/var/tmp/Numeric-buildroot Jochen> | /var/tmp/rpm-tmp.35738: setup.py: command not found Jochen> | error: Bad exit status from /var/tmp/rpm-tmp.35738 (%install) Jochen> `---- Looking at my own message I realize that the following patch is all that is needed. '.' shouldn't be in your path, it definitely won't be in mine. Index: rpm_install.sh =================================================================== RCS file: /cvsroot/numpy/Numerical/rpm_install.sh,v retrieving revision 1.1 diff -u -r1.1 rpm_install.sh - --- rpm_install.sh 2002/01/22 04:39:53 1.1 +++ rpm_install.sh 2002/01/22 05:39:50 @@ -1,4 +1,4 @@ - -setup.py install --root=$RPM_BUILD_ROOT +./setup.py install --root=$RPM_BUILD_ROOT cat >INSTALLED_FILES < iD8DBQE8TPuQiJ/aUUS8zY4RApgcAJ4mlgsdCbJoiXnF4s4IyENE1m+D2ACfXdGU dtpO/3TQEiV+kjY79ZmDUPE= =/wKD -----END PGP SIGNATURE----- From paul at pfdubois.com Wed Jan 23 11:59:45 2002 From: paul at pfdubois.com (Paul Dubois) Date: Wed Jan 23 11:59:45 2002 Subject: [Numpy-discussion] yacvsu Message-ID: <000701c1a448$1e69df20$09860cc0@CLENHAM> yet-another-cvs-update Disregarding advice from wiser heads, I made setup.py generate rpm_install.sh so that it would execute setup.py with the "right" python. Works for me (TM) in the sense that the rpm builds. I'm afraid to test it because I don't want my system ruined. We need some words someplace about how one uses these rpms. I suppose the web site is the right place. Also new in this issue is an upgrade to MA.average to allow axial weights as Numeric now does. I replaced all memcpy's with memmove's due to a bug report. If someone thinks this is a hideous decision just let me know why. From kragen at pobox.com Wed Jan 23 22:24:08 2002 From: kragen at pobox.com (Kragen Sitaker) Date: Wed Jan 23 22:24:08 2002 Subject: [Numpy-discussion] memory-mapped Numeric arrays: arrayfrombuffer version 2 Message-ID: <20020124012343.A13898@canonical.org> I would be very happy if this got included in the Numpy distribution, so that people don't have to do an extra download to get this facility. I just submitted it as a patch on SourceForge, but my browser failed to upload the file. http://sourceforge.net/tracker/index.php?func=detail&aid=507847&group_id=1369&atid=301369 There is one thing in arrayfrombuffer.c I'm not sure about, and I could use some help here. arrayobj is the return value from PyArray_FromDimsAndData: /* do I need to incref arrayobj?! fromstring does... */ return (PyObject*)arrayobj; } So, do I? Or not? It seems to work as it is. Following is the announcement I posted to comp.lang.python and (hopefully) comp.lang.python.announce. The 'arrayfrombuffer' package features support for Numerical Python arrays whose contents are stored in buffer objects, including memory-mapped files. This has the following advantages: - loading your array from a file is easy --- a module import and a single function call --- and doesn't use excessive amounts of memory. - loading your array is quick; it doesn't need to be copied from one part of memory to another in order to be loaded. - your array gets demand-loaded; parts you aren't using don't need to be in memory or in swap. - under memory-pressure conditions, your array doesn't use up swap, and parts of it you haven't modified can be evicted from RAM without the need for a disk write - your arrays can be bigger than your physical memory - when you modify your array, only the parts you modify get written back out to disk This is something that's been requested on the Numpy list a few times a year since 1999. arrayfrombuffer lives at http://pobox.com/~kragen/sw/arrayfrombuffer/ The current version is version 2; it is released under the X11 license (the BSD license without the advertising clause). From dubois1 at llnl.gov Fri Jan 25 09:44:04 2002 From: dubois1 at llnl.gov (Paul F. Dubois) Date: Fri Jan 25 09:44:04 2002 Subject: [Numpy-discussion] RE: memory-mapped Numeric arrays: arrayfrombuffer version 2 In-Reply-To: Message-ID: <000001c1a5c7$5d4bacc0$0c01a8c0@freedom> I have verified that this package seems to work on Windows. I says seems only because I didn't try enough to uncover anything subtle. Unless or until we are convinced as a community that this is (a) the right way to do this and (b) that the package is portable, it would not be wise to put it in the main distribution. I would like to hear from the community about this so that I will know whether or not to add this package as a separate SourceForge 'package' within the Numerical Python area. Meantime I will add a link to the web page. -----Original Message----- From: python-announce-list-admin at python.org [mailto:python-announce-list-admin at python.org] On Behalf Of Kragen Sitaker Sent: Wednesday, January 23, 2002 9:40 PM To: python-announce-list at python.org Subject: memory-mapped Numeric arrays: arrayfrombuffer version 2 The 'arrayfrombuffer' package features support for Numerical Python arrays whose contents are stored in buffer objects, including memory-mapped files. This has the following advantages: - loading your array from a file is easy --- a module import and a single function call --- and doesn't use excessive amounts of memory. - loading your array is quick; it doesn't need to be copied from one part of memory to another in order to be loaded. - your array gets demand-loaded; parts you aren't using don't need to be in memory or in swap. - under memory-pressure conditions, your array doesn't use up swap, and parts of it you haven't modified can be evicted from RAM without the need for a disk write - your arrays can be bigger than your physical memory - when you modify your array, only the parts you modify get written back out to disk This is something that's been requested on the Numpy list a few times a year since 1999. arrayfrombuffer lives at http://pobox.com/~kragen/sw/arrayfrombuffer/ The current version is version 2; it is released under the X11 license (the BSD license without the advertising clause).

arrayfrombuffer 2 - creates Numeric arrays from memory-mapped files. (23-Jan-02) -- http://mail.python.org/mailman/listinfo/python-announce-list From nodwell at physics.ubc.ca Fri Jan 25 10:41:08 2002 From: nodwell at physics.ubc.ca (Eric Nodwell) Date: Fri Jan 25 10:41:08 2002 Subject: [Numpy-discussion] RE: memory-mapped Numeric arrays: arrayfrombuffer version 2 In-Reply-To: <000001c1a5c7$5d4bacc0$0c01a8c0@freedom> References: <000001c1a5c7$5d4bacc0$0c01a8c0@freedom> Message-ID: <20020125184013.GA28841@holler.physics.ubc.ca> Since I have a 2.4GB data file handy, I thought I'd try this package with it. (Normally I process this data file by reading it in a chunk at a time, which is perfectly adequate.) Not surprisinly, it chokes: File "/home/eric/lib/python2.2/site-packages/maparray.py", line 15, in maparray m = mmap.mmap(fn, os.fstat(fn)[stat.ST_SIZE]) OverflowError: memory mapped size is too large (limited by C int) (details: Python 2.2, numpy 20.3, Pentium III, Debian Woody, Linux kernel 2.4.13, gcc 2.95.4) I'm not a big C programmer, but I wonder if there is some way for this package to overcome the 2GB limit on 32-bit systems. That could be useful in some situations. Eric On Fri, Jan 25, 2002 at 09:40:21AM -0800, Paul F. Dubois wrote: > > I have verified that this package seems to work on Windows. I says seems > only because I didn't try enough to uncover anything subtle. > > Unless or until we are convinced as a community that this is (a) the > right way to do this and (b) that the package is portable, it would not > be wise to put it in the main distribution. > > I would like to hear from the community about this so that I will know > whether or not to add this package as a separate SourceForge 'package' > within the Numerical Python area. Meantime I will add a link to the web > page. > > From: python-announce-list-admin at python.org > [mailto:python-announce-list-admin at python.org] On Behalf Of Kragen > Sitaker > Sent: Wednesday, January 23, 2002 9:40 PM > To: python-announce-list at python.org > Subject: memory-mapped Numeric arrays: arrayfrombuffer version 2 > > > The 'arrayfrombuffer' package features support for Numerical Python > arrays whose contents are stored in buffer objects, including > memory-mapped files. This has the following advantages: > > - loading your array from a file is easy --- a module import and a > single function call --- and doesn't use excessive amounts of > memory. > - loading your array is quick; it doesn't need to be copied from one > part of memory to another in order to be loaded. > - your array gets demand-loaded; parts you aren't using don't need to > be in memory or in swap. > - under memory-pressure conditions, your array doesn't use up swap, > and parts of it you haven't modified can be evicted from RAM without > the need for a disk write > - your arrays can be bigger than your physical memory > - when you modify your array, only the parts you modify get written > back out to disk > > This is something that's been requested on the Numpy list a few times a > year since 1999. > > arrayfrombuffer lives at http://pobox.com/~kragen/sw/arrayfrombuffer/ > The current version is version 2; it is released under the X11 license > (the BSD license without the advertising clause). > > > >

HREF="http://pobox.com/~kragen/sw/arrayfrombuffer/">arrayfrombuffer > 2 - creates Numeric arrays from memory-mapped files. (23-Jan-02) > -- ******************************** Eric Nodwell Ph.D. candidate Department of Physics University of British Columbia tel: 604-822-5425 fax: 604-822-4750 nodwell at physics.ubc.ca From hinsen at cnrs-orleans.fr Fri Jan 25 11:26:05 2002 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Fri Jan 25 11:26:05 2002 Subject: [Numpy-discussion] memory-mapped Numeric arrays: arrayfrombuffer version 2 In-Reply-To: <20020124012343.A13898@canonical.org> References: <20020124012343.A13898@canonical.org> Message-ID: Kragen Sitaker writes: > There is one thing in arrayfrombuffer.c I'm not sure about, and > I could use some help here. arrayobj is the return value from > PyArray_FromDimsAndData: > /* do I need to incref arrayobj?! fromstring does... */ > return (PyObject*)arrayobj; > } > > So, do I? Or not? It seems to work as it is. And it is correct, in my opinion. Your routine is the owner of the array (and no one else keeps a reference), so you just pass on ownership. Doing an INCREF here would mean that the array is never freed. (Keep in mind that the data area is never freed anyway!) Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From jjl at pobox.com Fri Jan 25 15:12:01 2002 From: jjl at pobox.com (John J. Lee) Date: Fri Jan 25 15:12:01 2002 Subject: [Numpy-discussion] amap: new function / ufunc method? Message-ID: Having written all this down, I'm half-expecting somebody is going to point out some simple way of doing this with existing builtin functions, but here goes... In the process of 'porting' Konrad Hinsen's Histogram class (from the Scientific package) to N dimensions, I ran across the problem of how to apply a function to an array for a specified set of indices. In this case, the problem is to increment an N-dimensional array for each of a set of indices. In general, it occurred to me that it would be useful to have a function (named 'amap' here) to loop over a specified set of indices, applying a function to a set of arguments taken from a set of argument arrays (similar to the map buitin function in this respect), and filling in the corresponding elements in a target array with the results. For example, here is a bit of the histogram problem: [The original 1D Histogram class uses a clever array-broadcasting trick which creates intermediate rank-2 arrays, feeding in the data in chunks to avoid quadratic behaviour. This is fine with 1D arrays, but in poor taste for the N dimensional case, if it works at all.] #!/usr/bin/env python import Numeric as N def flatten_indices(shape, indices): dims = indices.shape[0] facs = pow(shape, N.arange(dims))[::-1,N.NewAxis] return N.add.reduce(facs*indices) def amap(target, function, indices, *arg_arrays): indices = N.asarray(indices) arg_arrays = map(N.asarray, arg_arrays) if not isinstance(target, N.ArrayType): raise TypeError, "target must be an array" if not target.iscontiguous(): raise ValueError, "target must be contiguous" if not N.alltrue([aa.iscontiguous() for aa in arg_arrays]): raise ValueError, "all arg_arrays must be contiguous" if not len(indices.shape) == 1: raise ValueError, "indices must be rank-1" tf = target.flat for index in indices: args = [arg_array.flat[index] for arg_array in arg_arrays] tf[index] = function(*args) # construct a 2 dimensional histogram given indices, ind, of data points nbins = (5, 4) # 5 bins along one axis, 4 along the other histo = N.zeros(nbins) # empty histogram # these are the indices into the histogram at which data points lie: # histo[1,0] should be incremented twice, histo[2,2] should be incremented # once, etc. ind = N.array([[1, 1, 2, 4], [0, 0, 2, 3]]) flat_ind = flatten_indices(histo.shape, ind) assert flat_ind == N.array([4, 4, 10, 19]) # Now we want to increment histo for each index pair ind[:,n] (in this # 2D case) def increment(x): return x+1 amap(histo, increment, flat_ind, histo) print histo Actually, I just realised that calling it something related to 'map' is rather misleading, because the builtin Python map generates a list of the same length as the sequence passed in; this function doesn't: it 'generates' (fills in, in fact) an array of the same size as the target passed in. Can't think of a better name, though. So: is there a way of doing this (implementing amap) efficiently already? I don't think there is, but I'd be interested to be proved wrong. I admit I don't like the relatively large number of arguments, but I still think it is worthwhile. You could make the *arg_arrays a single argument, and make target an optional argument, at the end of the arguments: def amap(function, indices, arg_arrays, target=None): so that by default, the target array is first created as zeros, then filled in and returned, or perhaps even better: def amap(function, arg_arrays, indices=None, target=None): though the special case where you don't give the indices argument is already achievable without this function, so it could be argued that hiding away the indices argument as a mere optional argument obscures the real point of the function. The last form also has the (debatable) benefit of being similar to the built-in Python map: def map(function, *args): The example call above then looks like this: histo = amap(increment, [histo], indices=flat_ind) Another couple of tweaks might be to allow scalar values to replace the arg_arrays argument, and to allow normal indices rather than flattened ones (the only reason I started out with flat indices was that put() did the same): def amap(function, args, indices=None, target=None): So that the example call looks like: histo = amap(increment, [histo], indices=ind) or histo = amap(Numeric.add, [histo, 1], ind) I like those last two. Even if there is already an efficient way of doing this with existing Numeric functions, I think this is a natural way of thinking about some operations on arrays -- this one, at least -- and would be useful to have in Numeric. For real efficiency, I suppose the right thing to do would be to have a map method on ufuncs, as is the case for reduce: histo = Numeric.add.map([histo, 1], ind) but it would be more work to implement than just a standard function, and less general. I've made a first attempt at the function, in C (will upload diff against 20.2.0 if anyone so requests). It only accepts one arg array, probably doesn't really do exactly the same as the Python function above (for better or worse), and it's a mess of a function, since I rarely write anything in C. It is also incorrect in at least some ways (most obviously, it only deals with Python floats / C doubles / Numpy PyArray_DOUBLEs), though there are some tests (which pass). I'm wasn't sure how fast it would be, since after all it's only using ordinary Python functions, not ufuncs, but my crude tests appeared to show an improvement of about a factor of two over an optimised version of the Python function above. The slow bit seems to be constructing the argument list / passing arg list to function, so I removed that to make the comparison, along with an unecessary attribute access in the loop. Presumably the general (> 1 arg_array) version would show a bigger improvement over the corresponding Python version. By the way, does anyone understand what the reduceat ufunc method is supposed to do? Seems to be the Beale cipher of Numeric -- having studied its output to see if it was the function I was looking for, I can find no sense in it and kind of suspect that it was constructed only to keep us puzzling. ;) Is Jim Hugunin not available for comment on the matter, or is this somebody else's dirty work? John From rob at pythonemproject.com Fri Jan 25 17:59:01 2002 From: rob at pythonemproject.com (Rob) Date: Fri Jan 25 17:59:01 2002 Subject: [Numpy-discussion] Column in IEEE trans Antennas and Propagation Message-ID: <3C520C85.E0938CCD@pythonemproject.com> I've been asked to write a short column about Numpy and my web site in IEEE AP, for the Educational Corner. I still consider myself a Numpy neophyte, but perhaps thats the beauty of Python, being able to do neat things without being an expert. I am pondering what to write about on the Numpy side. I am soliciting any genious ideas for topics :) The first thing I thought of is that I just plain enjoy being able to tweek code without worrying about compilers. Second, the code ends up so much more readable than Fortran or C. Thirdly, vectorizing the ported code makes it look much more like the mathematical basis of the program. Lastly, I think its the perfect language for students learning algorithms. I'm open to any ideas. Thanks, Rob. -- The Numeric Python EM Project www.pythonemproject.com From cavallo at kip.uni-heidelberg.de Sat Jan 26 14:01:02 2002 From: cavallo at kip.uni-heidelberg.de (cavallo at kip.uni-heidelberg.de) Date: Sat Jan 26 14:01:02 2002 Subject: [Numpy-discussion] kdf file format I/O routine In-Reply-To: Message-ID: Hy, this is my first attempt to give some back to the python/numerical-python community. Here (http://kdfio.sourceforge.net) you can find an early release of my I/O routine for reading and writing kdf file as in khoros environment. Khoros (from http://www.khoral.com) is an interactive program for digital image processing, we are using in our lab (university of heidelberg). All the time i've found hard to use such environment because it lacks some flexibility i need. Now i've been using numerical python for some time and i would other will use it as standard base for numerical computation, but for my purpose it was lacking I/O with such an environment (khoros). Khoros is not free, but anyway if you are using it i hope you will like these routines to read and write their files. These are in an early stage, so expect a lot of bugs, missing feature and so on: it's only my first project so feedbacks, suggestions and complains are welcome. Ops, i've forgotten: i'm personally using this rotines for my PHD, so they should be reasonably stable. The license should be BSD? GPL? LGPL? any comment is welcome, because i never understood these problem (it is my first project, didn't i tell you?). best regards to all GREAT folks in this group, antonio cavallo ps. you can download khoros if you are a student from: http://www.khoral.com/khoros/kp2001_student From magnus at hetland.org Sat Jan 26 14:06:04 2002 From: magnus at hetland.org (Magnus Lie Hetland) Date: Sat Jan 26 14:06:04 2002 Subject: [Numpy-discussion] kdf file format I/O routine In-Reply-To: ; from cavallo@kip.uni-heidelberg.de on Sat, Jan 26, 2002 at 10:50:38PM +0100 References: Message-ID: <20020126230551.B3021@idi.ntnu.no> cavallo at kip.uni-heidelberg.de : ... > The license should be BSD? GPL? LGPL? any comment is welcome, because i > never understood these problem (it is my first project, didn't i tell > you?). If you don't specifically want to follow the philosophy of the GPL, I'd suggest that you go with some BSD variant (e.g. MIT, which is even simpler). That way, you place less restrictions on its use/distributions, and more people can/will use it. (YMMV, of course.) -- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org From jochen at unc.edu Sun Jan 27 07:36:03 2002 From: jochen at unc.edu (Jochen =?iso-8859-1?q?K=FCpper?=) Date: Sun Jan 27 07:36:03 2002 Subject: [Numpy-discussion] Character typecode Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 According to the manual there is a typecode 'Character'. But the following program gives an error: ,---- | import Numeric as num | data = num.array([1,2,3], num.Character) `---- ,---- | Traceback (most recent call last): | File "", line 2, in ? | AttributeError: 'module' object has no attribute 'Character' `---- This is python-2.2, latest NumPy cvs. The following line fixes that. Any reason that isn't in there? Index: Lib/Precision.py =================================================================== RCS file: /cvsroot/numpy/Numerical/Lib/Precision.py,v retrieving revision 1.1.1.1 diff -u -r1.1.1.1 Precision.py - --- Lib/Precision.py 2000/01/13 21:23:06 1.1.1.1 +++ Lib/Precision.py 2002/01/27 15:30:50 @@ -33,6 +33,7 @@ return typecode raise PrecisionError, key+" of "+str(required_bits)+" bits not available on this system" +Character = 'c' UnsignedInt8 = 'b' try: Int0 = _lookup(_code_table, 'Integer', 0) Greetings, Jochen PS: The manual page http://pfdubois.com/numpy/html2/numpy-6.html is really messed up right at the spot where typecodes are explained. That is with Netscape-4.7 and Mozilla-0.9.7 on Linux and Win2000. PPS: What format is the manual master deocument in? Would it be feasible to generate info pages from that? - -- University of North Carolina phone: +1-919-962-4403 Department of Chemistry phone: +1-919-962-1579 Venable Hall CB#3290 (Kenan C148) fax: +1-919-843-6041 Chapel Hill, NC 27599, USA GnuPG key: 44BCCD8E -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6-cygwin-fcn-1 (Cygwin) Comment: Processed by Mailcrypt and GnuPG iD8DBQE8VB5XiJ/aUUS8zY4RAhIxAJ43vXeTvN0hlbVmCsSCGCGYrt2m/wCeOvH8 lodh8d0YmJy0eCqMyglNdj0= =z/Wo -----END PGP SIGNATURE----- From oliphant.travis at ieee.org Sun Jan 27 22:10:04 2002 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Jan 27 22:10:04 2002 Subject: [Numpy-discussion] amap: new function / ufunc method? In-Reply-To: References: Message-ID: > In the process of 'porting' Konrad Hinsen's Histogram class (from the > Scientific package) to N dimensions, I ran across the problem of how to > apply a function to an array for a specified set of indices. If I understand what you are trying to do, there is a function called arraymap in the SciPy package (special module) (it was in the Numeric source for a short time but I think we decided to take it out --- I can't remember why). I think that using this function and a combination of take and put can do what you describe. It is available as scipy.special.arraymap Best regards, Travis Oliphant From nwagner at mecha.uni-stuttgart.de Mon Jan 28 00:37:02 2002 From: nwagner at mecha.uni-stuttgart.de (Nils Wagner) Date: Mon Jan 28 00:37:02 2002 Subject: [Numpy-discussion] Matlab to Numpy Message-ID: <3C551C48.E5CEF797@mecha.uni-stuttgart.de> Hi all, Who can send me a program written in Numpy (+Vpython for graphics), that reproduces the results of the following Matlab code x=linspace(0,1,25); t=linspace(0,2,50); [X,T] = meshgrid(x,t); z=exp(-abs((X-.5)*(T-1)))+sin(X.*T); subplot(3,2,1) surf(X,T,z) axis([0,1,0,2,0.4,2.1]) xlabel('x'),ylabel('t'),zlabel('z'),title('Actual surface') [u,s,v] = svd(z); for k =1:3 zz=u(:,1:k)*s(1:k,1:k)*v(:,1:k)'; subplot(3,2,k+1) surf(X,T,zz),axis([0,1,0,2,0.4,2.1]) xlabel('x'),ylabel('t'),zlabel('z') title(['Rank',num2str(k),' approximation']) Thanks in advance Nils From nwagner at mecha.uni-stuttgart.de Mon Jan 28 06:48:14 2002 From: nwagner at mecha.uni-stuttgart.de (Nils Wagner) Date: Mon Jan 28 06:48:14 2002 Subject: [Numpy-discussion] Truncated Singular value decomposition Message-ID: <3C557347.5CCB9569@mecha.uni-stuttgart.de> Hi all, how can I obtain a truncated SVD (dyadic decomposition) of V,S,WT = singular_value_decomposition(Z) ? I am interested in a low rank approximation of Z. Thanks in advance. Nils From hinsen at cnrs-orleans.fr Mon Jan 28 08:22:03 2002 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Mon Jan 28 08:22:03 2002 Subject: [Numpy-discussion] Truncated Singular value decomposition In-Reply-To: <3C557347.5CCB9569@mecha.uni-stuttgart.de> References: <3C557347.5CCB9569@mecha.uni-stuttgart.de> Message-ID: Nils Wagner writes: > V,S,WT = singular_value_decomposition(Z) ? > > I am interested in a low rank approximation of Z. The singular values in S are sorted by decreasing order of magnitude, so you just replace as many "small" ones by zero as you wish, and multiply the three matrices back together. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From magnus at hetland.org Mon Jan 28 09:44:03 2002 From: magnus at hetland.org (Magnus Lie Hetland) Date: Mon Jan 28 09:44:03 2002 Subject: [Numpy-discussion] Odd numarray compile problem Message-ID: <20020128184327.H2276@idi.ntnu.no> I'm trying to compile numarray 0.11 with Python 2.2 on a linux box but somehow the compilation gets stuck on this line: gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -IInclude/numarray -I/home/mlh/python/Python-2.2/include/python2.2 -c Src/_ufuncmodule.c -o build/temp.linux-i686-2.2/_ufuncmodule.o No error message, no crash; it just never finishes -- it freezes. I compiled/installed it on a Solaris box earlier, with no problems whatsoever. I've never seen gcc freeze like this; can there be some circular dependencies or something? (This machine had some problems with its clock earlier, which made make go in a loop...) I tried running touch on all files, but that didn't change anything... Any ideas, or tips for finding out what's wrong? (BTW, Numeric compiles and installs just fine...) -- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org From michaell.taylor at reis.com Mon Jan 28 12:58:04 2002 From: michaell.taylor at reis.com (Michaell Taylor) Date: Mon Jan 28 12:58:04 2002 Subject: [Numpy-discussion] conditional array indexing Message-ID: Very new user to Numpy, so excuse the question. It relates to a fundamental issue which I have having trouble getting a grip on. Specifically, say I have the following: a = ([1,2,1,2,4,3,1,2,3,2,3,2]) b = ([4,3,5,2,4,5,3,6,3,2,5,6]) c = random variable of length=len(b) I would like to get the value of C associated with the lowest value of b within groupings of a. That is, there are three incidents of "1" in the a array. The lowest value of b associated with the value 1 in the a index is "3". This occurs in index value 6 on the a and b arrays. Now, I would like to then know the value of c[6]. Obviously the real problem is far more complex with len(a)==5000. Any ideas? Thanks in advance. Michaell From paul at pfdubois.com Mon Jan 28 13:21:02 2002 From: paul at pfdubois.com (Paul F. Dubois) Date: Mon Jan 28 13:21:02 2002 Subject: [Numpy-discussion] conditional array indexing In-Reply-To: Message-ID: <000101c1a841$2047c6c0$0c01a8c0@freedom> import Numeric, sys a = Numeric.array([1,2,1,2,4,3,1,2,3,2,3,2]) b = Numeric.array([4,3,5,2,4,5,3,6,3,2,5,6]) c = Numeric.arange(len(b)) print c[Numeric.argmin(Numeric.where(a==1, b, sys.maxint))] -----Original Message----- From: numpy-discussion-admin at lists.sourceforge.net [mailto:numpy-discussion-admin at lists.sourceforge.net] On Behalf Of Michaell Taylor Sent: Monday, January 28, 2002 12:59 PM To: numpy-discussion at lists.sourceforge.net Subject: [Numpy-discussion] conditional array indexing Very new user to Numpy, so excuse the question. It relates to a fundamental issue which I have having trouble getting a grip on. Specifically, say I have the following: a = ([1,2,1,2,4,3,1,2,3,2,3,2]) b = ([4,3,5,2,4,5,3,6,3,2,5,6]) c = random variable of length=len(b) I would like to get the value of C associated with the lowest value of b within groupings of a. That is, there are three incidents of "1" in the a array. The lowest value of b associated with the value 1 in the a index is "3". This occurs in index value 6 on the a and b arrays. Now, I would like to then know the value of c[6]. Obviously the real problem is far more complex with len(a)==5000. Any ideas? Thanks in advance. Michaell _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From Barrett at stsci.edu Tue Jan 29 05:27:06 2002 From: Barrett at stsci.edu (Paul Barrett) Date: Tue Jan 29 05:27:06 2002 Subject: [Numpy-discussion] Odd numarray compile problem References: <20020128184327.H2276@idi.ntnu.no> Message-ID: <3C56A2B6.1060904@STScI.Edu> Magnus Lie Hetland wrote: > I'm trying to compile numarray 0.11 with Python 2.2 on a linux box but > somehow the compilation gets stuck on this line: > > gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -IInclude/numarray > -I/home/mlh/python/Python-2.2/include/python2.2 -c Src/_ufuncmodule.c > -o build/temp.linux-i686-2.2/_ufuncmodule.o > > No error message, no crash; it just never finishes -- it freezes. I > compiled/installed it on a Solaris box earlier, with no problems > whatsoever. > > I've never seen gcc freeze like this; can there be some circular > dependencies or something? (This machine had some problems with its > clock earlier, which made make go in a loop...) I tried running touch > on all files, but that didn't change anything... > > Any ideas, or tips for finding out what's wrong? (BTW, Numeric > compiles and installs just fine...) Yes, just wait several minutes for it to finish compiling. I had to wait over 5 minutes on 233 MHz i586. -- Paul Barrett, PhD Space Telescope Science Institute Phone: 410-338-4475 ESS/Science Software Group FAX: 410-338-4767 Baltimore, MD 21218 From jjl at pobox.com Tue Jan 29 08:40:04 2002 From: jjl at pobox.com (John J. Lee) Date: Tue Jan 29 08:40:04 2002 Subject: [Numpy-discussion] amap: new function / ufunc method? In-Reply-To: Message-ID: On Sun, 27 Jan 2002, Travis Oliphant wrote: [...] > If I understand what you are trying to do, there is a function called > arraymap in the SciPy package (special module) (it was in the Numeric source [...] > I think that using this function and a combination of take and put can do > what you describe. I don't see how, though I may very well be missing something. To restate the problem very quickly (though the Python function I posted is much clearer -- it's only two or three lines of actual code): for every element in a list of array indices, the element in an output array at that index needs to be incremented; the tricky part is that this needs to happen once for *every time the index appears in the list*. Eg., in a 1D example, [0,1,1,1,4,5] might give you [1,3,0,0,1,1] (again, see my previous post for an actual tested example) I don't see how you can do this with arraymap, take and put. John From michaell.taylor at reis.com Tue Jan 29 08:46:02 2002 From: michaell.taylor at reis.com (Michaell Taylor) Date: Tue Jan 29 08:46:02 2002 Subject: [Numpy-discussion] conditional array indexing In-Reply-To: <000101c1a841$2047c6c0$0c01a8c0@freedom> References: <000101c1a841$2047c6c0$0c01a8c0@freedom> Message-ID: <20020129164505.5A969170201@newman.bestweb.net> Thanks for your quick reply. This looks vaguely S-Plus-like, with which I am familar. I wasn't too clear in my question however, I would actually like a vector returned which would contain the values of c for the minimun value of b, within each category of a. Something like: for i in range(1:5000): # range of values of a d = c[Numeric.argmin(Numeric.where(a==i, b, sys.maxint))] Seems like I could hack a fix using such a vector, but I would guess that speed would be an issue. Essentially the functionality implied by GROUP BY in sql egen in stata lapply() in S-Plus. Paul F. Dubois wrote: > import Numeric, sys > a = Numeric.array([1,2,1,2,4,3,1,2,3,2,3,2]) > b = Numeric.array([4,3,5,2,4,5,3,6,3,2,5,6]) > c = Numeric.arange(len(b)) > > print c[Numeric.argmin(Numeric.where(a==1, b, sys.maxint))] > > > -----Original Message----- > From: numpy-discussion-admin at lists.sourceforge.net > [mailto:numpy-discussion-admin at lists.sourceforge.net] On Behalf Of > Michaell Taylor > Sent: Monday, January 28, 2002 12:59 PM > To: numpy-discussion at lists.sourceforge.net > Subject: [Numpy-discussion] conditional array indexing > > > > Very new user to Numpy, so excuse the question. It relates to a > fundamental > issue which I have having trouble getting a grip on. > > Specifically, say I have the following: > > a = ([1,2,1,2,4,3,1,2,3,2,3,2]) > b = ([4,3,5,2,4,5,3,6,3,2,5,6]) > c = random variable of length=len(b) > > I would like to get the value of C associated with the lowest value of b > > within groupings of a. That is, there are three incidents of "1" in the > a > array. The lowest value of b associated with the value 1 in the a index > is > "3". This occurs in index value 6 on the a and b arrays. Now, I would > like > to then know the value of c[6]. > > Obviously the real problem is far more complex with len(a)==5000. > Any ideas? > > Thanks in advance. > > Michaell > > _______________________________________________ > Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- ========================================= Michaell Taylor, PhD Senior Economist, Reis, New York, USA From DavidA at ActiveState.com Tue Jan 29 10:49:01 2002 From: DavidA at ActiveState.com (David Ascher) Date: Tue Jan 29 10:49:01 2002 Subject: [Numpy-discussion] CFP: Python/Zope track at OSCON 2002 References: <000101c1a841$2047c6c0$0c01a8c0@freedom> <20020129164505.5A969170201@newman.bestweb.net> Message-ID: <3C56EE7A.97F621C0@activestate.com> The O'Reilly Open Source Convention (July 22-26, 2002 -- San Diego, CA) is accepting proposals for tutorials, talks, panels, and lightning talks. See the Call for Participation in the Python and Zope track on python.org. Proposals are due by March 1, so don't wait a moment longer! CFP URL: http://www.python.org/workshops/oscon2002/cfp.html --david ascher From magnus at hetland.org Tue Jan 29 12:34:04 2002 From: magnus at hetland.org (Magnus Lie Hetland) Date: Tue Jan 29 12:34:04 2002 Subject: [Numpy-discussion] What are the chances of getting numarray properties? Message-ID: <20020129213340.F22910@idi.ntnu.no> I see in the docs that numarray uses accessor methods instead of properties, mainly because of performance (to avoid __getattr__ and __setattr__). With the new property type in Python 2.2, that shouldn't be as important anymore... Perhaps it would be possible to use properties where this type is available, and accessors elsewhere? -- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org From magnus at hetland.org Tue Jan 29 12:40:04 2002 From: magnus at hetland.org (Magnus Lie Hetland) Date: Tue Jan 29 12:40:04 2002 Subject: [Numpy-discussion] Accessors Message-ID: <20020129213922.G22910@idi.ntnu.no> According to the (probably outdated) user guide for numarray[1] attributes have accessors such as a.foo() --> get foo a.foo(val) --> set foo to val but this doesn't seem to be the case. I see that the shape() method doesn't take any arguments; I guess the reshape method is the thing to use here. How about other attributes? Will they have special names for their setters too, or is this only true for the shape attribute? It seems to me that if we are going to abandon plain Python attributes (or 2.2 properties), it would be nice to handle the transition in a consistent manner... Perhaps? (But then again, I'm sure I'm missing something :) [1] http://stsdas.stsci.edu/numarray/Userguide.html -- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org From jmiller at stsci.edu Tue Jan 29 13:01:01 2002 From: jmiller at stsci.edu (Jay T Miller) Date: Tue Jan 29 13:01:01 2002 Subject: [Numpy-discussion] Re: What are the chances of getting numarray properties? References: Message-ID: <3C570D31.4090308@stsci.edu> Numarray should soon support the following properties: .shape .flat .real .imag .imaginary We're testing them here at STSCI. If all goes well, there should be a new numarray release within 1-3 weeks. Todd -- Todd Miller jmiller at stsci.edu STSCI / SSG (410) 338 4576 From magnus at hetland.org Tue Jan 29 13:07:11 2002 From: magnus at hetland.org (Magnus Lie Hetland) Date: Tue Jan 29 13:07:11 2002 Subject: [Numpy-discussion] Re: What are the chances of getting numarray properties? In-Reply-To: <3C570D31.4090308@stsci.edu>; from jmiller@stsci.edu on Tue, Jan 29, 2002 at 03:59:29PM -0500 References: <3C570D31.4090308@stsci.edu> Message-ID: <20020129220647.I22910@idi.ntnu.no> Jay T Miller : > > Numarray should soon support the following properties: > > .shape > .flat > .real > .imag > .imaginary > > We're testing them here at STSCI. If all goes well, there should be > a new numarray release within 1-3 weeks. Yay! :)) > Todd -- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org From magnus at hetland.org Tue Jan 29 14:38:03 2002 From: magnus at hetland.org (Magnus Lie Hetland) Date: Tue Jan 29 14:38:03 2002 Subject: [Numpy-discussion] RandomArray and numarray Message-ID: <20020129233707.A28608@idi.ntnu.no> Is there some support for random arrays in numarray? Or, if it becomes part of the standard library, will there be a separate module (such as Numeric's RandomArray)? -- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org From magnus at hetland.org Tue Jan 29 14:52:06 2002 From: magnus at hetland.org (Magnus Lie Hetland) Date: Tue Jan 29 14:52:06 2002 Subject: [Numpy-discussion] numarray bug... Message-ID: <20020129235152.C28608@idi.ntnu.no> My numarray dumps core... >>> from numarray import * >>> a = zeros([10, 10]) >>> a = zeros([10, 10]) + 0.5 Bus Error (core dumped) Numeric tackles this just fine... What gives? -- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org From perry at stsci.edu Tue Jan 29 14:54:04 2002 From: perry at stsci.edu (Perry Greenfield) Date: Tue Jan 29 14:54:04 2002 Subject: [Numpy-discussion] RandomArray and numarray In-Reply-To: <20020129233707.A28608@idi.ntnu.no> Message-ID: > > Is there some support for random arrays in numarray? Or, if it becomes > part of the standard library, will there be a separate module (such as > Numeric's RandomArray)? > > -- > Magnus Lie Hetland The Anygui Project > http://hetland.org http://anygui.org > Not at the moment, but I imagine we will add a separate module as it is done now for Numeric. While I'm replying, our short range development plans are (now that we are back working on it): 1) adding properties for .shape, .flat, etc. This was done last week but we are testing this and will put it out as soon as possible (probably after the Python Conference). By the way, to make it usable from Python <2.2, we will add accessor functions with different names than we used previously (e.g., .getshape(), setshape(), .getflat()...) 2) reworking numarray to make it safe against array overruns and anything that might cause segfaults or buserrors (in principle should only happen now if you twiddle directly with private attributes). 3) provide examples of how to add new ufuncs and interface C code and such. We may decide to add a module or two in the process. And perhaps some limited benchmarking. 4) rework the implementation of complex types to be in C. 5) do more extensive benchmarking and look at how to optimize smaller arrays. Perry Greenfield From kragen at pobox.com Tue Jan 29 23:39:03 2002 From: kragen at pobox.com (Kragen Sitaker) Date: Tue Jan 29 23:39:03 2002 Subject: [Numpy-discussion] Re: memory-mapped Numeric arrays: arrayfrombuffer version 2 Message-ID: <20020130073834.AFB65BDC8@panacea.canonical.org> Paul Dubois writes: > I have verified that this package seems to work on Windows. I says seems > only because I didn't try enough to uncover anything subtle. Thanks! If you run maparray.py from the command-line, it runs a basic regression test suite, which doesn't really try enough to uncover anything subtle either. > Unless or until we are convinced as a community that this is (a) the > right way to do this and (b) that the package is portable, it would not > be wise to put it in the main distribution. Well, I'm not the community, but I'll state my arguments. On (a): I don't know about the right way to do it; as I'm sure is obvious, I'm new to extending Numerical Python in C, but I doubt there's a simpler way to do it ("it" being seeing the contents of files as Numeric arrays), and I think it does work as well as it's possible to get it to work without major hacking of Numeric (to support read-only arrays) or the mmap module (to prevent closing an open object, to allow read-only mmapping on Windows). The other things on the wishlist are basically features to add --- "start" and "size" arguments to maparray(), a "create-a-file" argument to maparray(), etc., and don't change the basic structure. On (b): it depends on two-argument mmap.mmap(), open(), .fileno(), and os.fstat(). The major portability hurdle there is probably mmap.mmap(), but that's OK. > I would like to hear from the community about this so that I will know > whether or not to add this package as a separate SourceForge 'package' > within the Numerical Python area. Meantime I will add a link to the web > page. I would too. There have been downloads from something like 50 IP addresses, but I've only heard from three people. From yoonseo at altavista.net Thu Jan 31 14:33:02 2002 From: yoonseo at altavista.net (SungPil Yoon) Date: Thu Jan 31 14:33:02 2002 Subject: [Numpy-discussion] numpy installation on HP-UX Message-ID: <02013116191902.23816@dhcp-243> Hi, all I have been trying to install numpy. Python is installed in my home directory ~/Python-2.2 and when I do % python setup.py install in the directory ~/Numeric-20.3, I get the following error message distutils.errors.DistutilsPlatformError: invalid Python installation: unable to open /usr/local/lib/python2.2/config/Makefile (No such file or directory) Can anybody figure out what I am doing wrong? sungpil From jh at oobleck.astro.cornell.edu Wed Jan 2 11:28:14 2002 From: jh at oobleck.astro.cornell.edu (Joe Harrington) Date: Wed Jan 2 11:28:14 2002 Subject: [Numpy-discussion] RPMs out of date, have problems Message-ID: <200201021929.g02JTKF30031@oobleck.astro.cornell.edu> Hi folks, Problem: The latest Numeric release on the web site is 20.3. The latest with an RPM is 20.1, and that RPM has a problem: it creates a directory in the system root directory. Paul D. says he will implement a solution but doesn't have the experience with RPMs (or the time) to find the problem quickly. I haven't dealt with building Python packages or distutils (is distutils a separate thing or part of Python?) at all. Can someone with the relevant experience fix the current problem and help Paul implement the solution so he can post current RPMs that install right? Ditto anyone who knows how to make packages for Debian, Solaris, and other popular package managers. Rationale: As I've mentionned previously, I'm getting an increasing number of queries from astronomers who want to play with Numeric. At this stage many of the converts will be application code contributors who will help build a library of discipline-specific routines. In talking to these people, I am finding them less than patient with the good 'ol tarball (a position I take myself, following the experience of maintaining the Clue Files, see ftp://oobleck.astro.cornell.edu/pub/clues.tar.gz). To them, it's not serious software if it isn't prepared under their system's installation manager. We need these (very) early adopters, so I think that having a current Numeric RPM for i386 Linux (and the equivalent for i386 Debian GNU/Linux and Solaris Sparc architectures, if someone knows how to build them) would be a Good Thing. Trivial install -> more users, more users -> more volunteers and more contributed code. Also, it would be more consistent with the RPM naming scheme to call the RPM "python-Numeric" (or "python-numeric", or even "numpy") rather than just "Numeric". If that's hard or philosophically undesirable, don't bother, but the name has changed a few times, so I hope it isn't a big deal. Sysadmins have to deal with more than 1000 packages now, and knowing what a package is just by looking at the name is a big help. Also, you can do things like 'rpm -qa | grep python' and get a list of all the python-related packages on your system. "Numeric" is too general outside the context of Python. All of the above goes for Numarray, when its developers are ready for the community at large to start writing code that uses it. Thanks, --jh-- From ransom at physics.mcgill.ca Wed Jan 2 11:37:08 2002 From: ransom at physics.mcgill.ca (Scott Ransom) Date: Wed Jan 2 11:37:08 2002 Subject: [Numpy-discussion] RPMs out of date, have problems In-Reply-To: <200201021929.g02JTKF30031@oobleck.astro.cornell.edu> References: <200201021929.g02JTKF30031@oobleck.astro.cornell.edu> Message-ID: Hello All, Debian (unstable -- which is actually quite stable) provides slightly more up-to-date packages of numeric (20.2) and its extensions. You can find them here: http://packages.debian.org/unstable/math/python-numeric.html http://packages.debian.org/unstable/interpreters/python-numeric-ext.html For a quick hack at a more up-to-date RPM, it _might_ be possible to use alien to convert the *.debs to *.rpms... Scott PS: As an astronomer myself, I am also seeing an increasing interest in my collegues towards python and numeric (especially since I keep preaching the Python gospel to them ;) On January 2, 2002 02:29 pm, Joe Harrington wrote: > Hi folks, > > Problem: > > The latest Numeric release on the web site is 20.3. The latest with > an RPM is 20.1, and that RPM has a problem: it creates a directory in > the system root directory. Paul D. says he will implement a solution > but doesn't have the experience with RPMs (or the time) to find the > problem quickly. I haven't dealt with building Python packages or > distutils (is distutils a separate thing or part of Python?) at all. > Can someone with the relevant experience fix the current problem and > help Paul implement the solution so he can post current RPMs that > install right? Ditto anyone who knows how to make packages for Debian, > Solaris, and other popular package managers. > > Rationale: > > As I've mentionned previously, I'm getting an increasing number of > queries from astronomers who want to play with Numeric. At this stage > many of the converts will be application code contributors who will > help build a library of discipline-specific routines. In talking to > these people, I am finding them less than patient with the good 'ol > tarball (a position I take myself, following the experience of > maintaining the Clue Files, see > ftp://oobleck.astro.cornell.edu/pub/clues.tar.gz). To them, it's not > serious software if it isn't prepared under their system's > installation manager. We need these (very) early adopters, so I think > that having a current Numeric RPM for i386 Linux (and the equivalent > for i386 Debian GNU/Linux and Solaris Sparc architectures, if someone > knows how to build them) would be a Good Thing. Trivial install -> > more users, more users -> more volunteers and more contributed code. > > Also, it would be more consistent with the RPM naming scheme to call > the RPM "python-Numeric" (or "python-numeric", or even "numpy") rather > than just "Numeric". If that's hard or philosophically undesirable, > don't bother, but the name has changed a few times, so I hope it isn't > a big deal. Sysadmins have to deal with more than 1000 packages now, > and knowing what a package is just by looking at the name is a big > help. Also, you can do things like 'rpm -qa | grep python' and get a > list of all the python-related packages on your system. "Numeric" is > too general outside the context of Python. > > All of the above goes for Numarray, when its developers are ready for > the community at large to start writing code that uses it. > > Thanks, > > --jh-- > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Scott M. Ransom Address: McGill Univ. Physics Dept. Phone: (514) 398-6492 3600 University St., Rm 338 email: ransom at physics.mcgill.ca Montreal, QC Canada H3A 2T8 GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From gvermeul at labs.polycnrs-gre.fr Thu Jan 3 00:20:03 2002 From: gvermeul at labs.polycnrs-gre.fr (Gerard Vermeulen) Date: Thu Jan 3 00:20:03 2002 Subject: [Numpy-discussion] RPMs out of date, have problems In-Reply-To: <200201021929.g02JTKF30031@oobleck.astro.cornell.edu> References: <200201021929.g02JTKF30031@oobleck.astro.cornell.edu> Message-ID: <02010309191300.12231@taco.polycnrs-gre.fr> Hi, Try python setup.py bdist_rpm Works like a charm. The same for the subdirectories/subpackages (maybe after installation of the main RPMs). Gerard PS: it is impossible to provide binary RPMs, because there are too many different Linux distributions. On Wednesday 02 January 2002 20:29, Joe Harrington wrote: > Hi folks, > > Problem: > > The latest Numeric release on the web site is 20.3. The latest with > an RPM is 20.1, and that RPM has a problem: it creates a directory in > the system root directory. Paul D. says he will implement a solution > but doesn't have the experience with RPMs (or the time) to find the > problem quickly. I haven't dealt with building Python packages or > distutils (is distutils a separate thing or part of Python?) at all. > Can someone with the relevant experience fix the current problem and > help Paul implement the solution so he can post current RPMs that > install right? Ditto anyone who knows how to make packages for Debian, > Solaris, and other popular package managers. > > Rationale: > > As I've mentionned previously, I'm getting an increasing number of > queries from astronomers who want to play with Numeric. At this stage > many of the converts will be application code contributors who will > help build a library of discipline-specific routines. In talking to > these people, I am finding them less than patient with the good 'ol > tarball (a position I take myself, following the experience of > maintaining the Clue Files, see > ftp://oobleck.astro.cornell.edu/pub/clues.tar.gz). To them, it's not > serious software if it isn't prepared under their system's > installation manager. We need these (very) early adopters, so I think > that having a current Numeric RPM for i386 Linux (and the equivalent > for i386 Debian GNU/Linux and Solaris Sparc architectures, if someone > knows how to build them) would be a Good Thing. Trivial install -> > more users, more users -> more volunteers and more contributed code. > > Also, it would be more consistent with the RPM naming scheme to call > the RPM "python-Numeric" (or "python-numeric", or even "numpy") rather > than just "Numeric". If that's hard or philosophically undesirable, > don't bother, but the name has changed a few times, so I hope it isn't > a big deal. Sysadmins have to deal with more than 1000 packages now, > and knowing what a package is just by looking at the name is a big > help. Also, you can do things like 'rpm -qa | grep python' and get a > list of all the python-related packages on your system. "Numeric" is > too general outside the context of Python. > > All of the above goes for Numarray, when its developers are ready for > the community at large to start writing code that uses it. > > Thanks, > > --jh-- > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From hinsen at cnrs-orleans.fr Fri Jan 4 02:32:02 2002 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Fri Jan 4 02:32:02 2002 Subject: [Numpy-discussion] RPMs out of date, have problems References: <200201021929.g02JTKF30031@oobleck.astro.cornell.edu> Message-ID: <200201041032.g04AWQg08269@localhost.localdomain> Joe Harrington writes: > The latest Numeric release on the web site is 20.3. The latest with > an RPM is 20.1, and that RPM has a problem: it creates a directory in > the system root directory. Paul D. says he will implement a solution > but doesn't have the experience with RPMs (or the time) to find the > problem quickly. I haven't dealt with building Python packages or If someone can give a precise description of the problem, I'll look at it, I think I have done enough RPMs to find my way... The RPMs generated by Distutils are often good enough, but sometimes need some manual cleanup. > distutils (is distutils a separate thing or part of Python?) at all. It is a part of Python since Python 1.6. > installation manager. We need these (very) early adopters, so I think > that having a current Numeric RPM for i386 Linux (and the equivalent I agree in principle, but providing Linux binary RPMs is becoming more and more messy: it depends on the Python version and on the distribution, sometimes even the distribution version number. That's a lot of RPMs. Source RPMs are easier, with some care they should work everywhere, but installation requires a compiler plus some of the "-devel" packages that not everyone has. > Also, it would be more consistent with the RPM naming scheme to call > the RPM "python-Numeric" (or "python-numeric", or even "numpy") rather > than just "Numeric". If that's hard or philosophically undesirable, Agreed. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From jh at oobleck.astro.cornell.edu Fri Jan 4 10:25:04 2002 From: jh at oobleck.astro.cornell.edu (Joe Harrington) Date: Fri Jan 4 10:25:04 2002 Subject: [Numpy-discussion] RPMs out of date, have problems In-Reply-To: <200201041032.g04AWQg08269@localhost.localdomain> (message from Konrad Hinsen on Fri, 4 Jan 2002 11:32:26 +0100) References: <200201021929.g02JTKF30031@oobleck.astro.cornell.edu> <200201041032.g04AWQg08269@localhost.localdomain> Message-ID: <200201041824.g04IODa15381@oobleck.astro.cornell.edu> In the exchange below, my query is >>, Gerard's response is >, and my present response to that is unquoted. >>From: Gerard Vermeulen >>Subject: Re: [Numpy-discussion] RPMs out of date, have problems >>Date: Fri, 4 Jan 2002 09:44:21 +0100 >> Could you check that it produces an RPM with files that go in >> /usr/lib/python2.1 rather than /pcmdi/dubois/linux/lib/python2.1 and >> /usr/include/python2.1 rather than /pcmdi/dubois/linux/include/python2.1? >Yes, mine go into /usr/lib/python2.1 >> Paul D. isn't sure why /pcmdi is being created. It's probably that >> way on his machine and he doesn't know how to tell distutils to use a >> different default prefix. Neither do I. That is the issue. Do you? >Well, the paths are figured out in /usr/lib/python2.1/distutils/sysconfig.py, >using sys.prefix or sys.exec_prefix. >So, I think that Paul's python has been build with >./configure --prefix=/pcmdi/dubois/linux >and that it still lives there, or has been moved to /usr afterwards. >Or that it resides on an NFS mounted volume??? >Or some python startup file that messes sys.(exec_)prefix. Ok, Paul, Konrad, is this what you need? >Anyhow, I think that with respect to binary packages >Windows has the edge over Linux (I know from experience, >because I am author of a Python wrapper to Qwt - a Qt library >allowing to do interactive plotting and it is SO easy to >produce a Windows). >Every combination of Distro-Version-Python-Version would >require its own binary RPM. If you consider Microsoft Windows one OS, Red Hat Linux another, Debian GNU/Linux a third, and so on, the distinction goes away. The fact that Microsoft has a monopoly is definitely convenient to producers. However, as users, we lose and shouldn't support it. >(A) IMHO, it is better to add a README.DIY.RPM: >(I am going through the steps based on Numeric-20.2.1) >(1) unpack Numeric-20.2.1.tar.gz (or whatever) >(2) cd Numeric-20.2.1 >(2) python setup.py bdist_rpm >(3) cd dist >(4) rpm -Uvh Numeric-20.2.1-1.i?86.rpm (as root, i?86 could be another >architecture) >But it is not so easy to build the extension packages (FFT & friends) like >this, without knowledge of RPM. I could try to hack setup.py to make it >compatible with RPM (means making 1 big package without subpackages) >(B) The other possibility is to include a generic python-numeric.spec file >(you build an RPM with rpm -ba python-numeric.spec). I can adapt mine. >Gerard >PS: do you have a RPM based Linux, yourself? Yes, I run Red Hat. So do the vast majority of US astronomers I have spoken to. I understand that SuSE is equally popular in Europe. A few other dists just copy one of them and add a few packages, and so don't need separate RPMs of their own. If we configure RPMs with the version of python included in these dists, we're not talking about a large number of packages, after all, to get 98% of the users or better. I think you are onto a solution: We build RPMs (and .deb files for Debian, if we think there are enough Debian users to make it worthwhile, and the same for Solaris if it uses a different package manager) for the current release of each popular distribution, with that release's python. That's maybe 4 binary packages per Numeric release (Red Hat, SuSE, Debian, Solaris). We also make a source RPM and distribute with it instructions for building a new binary RPM with appropriately-named version ID (e.g., python-numeric-20.4py2.2glibc6-1). We solicit users of unsupported combinations of OS and Python version to contribute those RPMs along with a simple ASCII form specifying what kind of system made it, contact info, etc. I predict we will not get a huge number of such submissions from outside this list, as the vast majority of Numeric users who upgrade their Python when a new Python release comes out (as opposed to a new OS release) are developers who don't care about the RPM anyway. The ultimate solution is to get this thing incorporated into the python core. --jh-- From paul at pfdubois.com Fri Jan 4 10:58:03 2002 From: paul at pfdubois.com (Paul F. Dubois) Date: Fri Jan 4 10:58:03 2002 Subject: [Numpy-discussion] RPMs out of date, have problems In-Reply-To: <200201041824.g04IODa15381@oobleck.astro.cornell.edu> Message-ID: <001101c19551$37d3c6c0$0c01a8c0@freedom> I can confirm that the Python I use is in /pcmdi/dubois/python. I *never* build into /usr or /usr/local. There are lots of people like me that have multiple pythons around with other stuff added in. I suppose what was bothering people was having to use --prefix because the default will not work because my Python is not in some standard place (said standard place seeming to me to be a mythical location given how I see people actually working). I continue to be less than impressed by RPMs. Every time I try one I get version conflicts with other software. From rossini at blindglobe.net Fri Jan 4 11:30:01 2002 From: rossini at blindglobe.net (A.J. Rossini) Date: Fri Jan 4 11:30:01 2002 Subject: [Numpy-discussion] RPMs out of date, have problems In-Reply-To: References: <200201021929.g02JTKF30031@oobleck.astro.cornell.edu> Message-ID: <87itahapbo.fsf@jeeves.blindglobe.net> >>>>> "SR" == Scott Ransom writes: SR> Hello All, Debian (unstable -- which is actually quite stable) SR> provides slightly more up-to-date packages of numeric (20.2) SR> and its extensions. You can find them here: SR> http://packages.debian.org/unstable/math/python-numeric.html SR> http://packages.debian.org/unstable/interpreters/python-numeric-ext.html Nothing like 2 days to make it now 20.3 in Debian unstable/Sid. best, -tony -- A.J. Rossini Rsrch. Asst. Prof. of Biostatistics U. of Washington Biostatistics rossini at u.washington.edu FHCRC/SCHARP/HIV Vaccine Trials Net rossini at scharp.org -------------- http://software.biostat.washington.edu/ -------------- FHCRC: M-W: 206-667-7025 (fax=4812)|Voicemail is pretty sketchy/use Email UW: T-Th: 206-543-1044 (fax=3286)|Change last 4 digits of phone to FAX Rosen: (Mullins' Lab) Fridays, and I'm unreachable except by email. From jh at oobleck.astro.cornell.edu Fri Jan 4 12:14:30 2002 From: jh at oobleck.astro.cornell.edu (Joe Harrington) Date: Fri Jan 4 12:14:30 2002 Subject: [Numpy-discussion] RPMs out of date, have problems In-Reply-To: <001101c19551$37d3c6c0$0c01a8c0@freedom> (paul@pfdubois.com) References: <001101c19551$37d3c6c0$0c01a8c0@freedom> Message-ID: <200201042009.g04K9ha15764@oobleck.astro.cornell.edu> > I *never* build into /usr or /usr/local. Good! You should leave /usr for your package manager to deal with. However, you should *release* software to install there, which you can with RPM without having to build it in /usr. /usr/local is yours to play with. The only thing in mine is IDL. (...which is why I'd like to delete it!) > I suppose > what was bothering people was having to use --prefix because the default > will not work because my Python is not in some standard place (said > standard place seeming to me to be a mythical location given how I see > people actually working). There was no documentation anywhere in the download area suggesting the use of --prefix. Even if there were, RPMs get collected and put in archives (rpmfind.net) without any other associated docs. They should always install into the system following the Linux file system standards. Nobody expects to have to configure an RPM. They're just supposed to work, and the vast majority just do. > There are lots of people like me > that have multiple pythons around with other stuff added in. There are lots if *developers* like you. The current group of Numpy users is demographically distinct from the future user base, if Numpy becomes a player for numerical work in science and engineering. We will have many users who have never written a line of C or C++ or FORTRAN and who will run screaming at the mention of the word "tar". We will have high school teachers and students. We will have lots of people who have zero time and zero tolerance for learning unrelated topics like system administration or software tweaking. If we don't care about them, they won't use our software. > I continue to be less than impressed by RPMs. Every time I try one I get > version conflicts with other software. I understand your frustration with the bad RPMs, but it's not the fault of RPM, it's the fault of either the package developer or the person installing the package. If the package developer links against a library that is itself under rapid development, there is a problem waiting to happen. It's worse if that library is included in some or all distributions. The simple solution is to statically link that library, or rename it and include it in the package. If the user has an out-of-date system, it's also trouble. If we build against the C library and python that are in the OS releases, we're in good shape. I don't think we need to bend over backwards to provide RPMs for people who install their own Python, unless a new version of Numeric requires use of an updated Python RPM. Very few (maybe 0.5%?) "mass-market" users will install their own Python. The experts who do can handle a tarball install, or one will build an RPM and distribute it if it's a particularly important release. The conflict problems pale in comparison to the problems with manually building all packages you want to install, particularly if you don't know how to fix makefile problems or don't have a compiler. If you're only installing a few packages, it's not that big a deal. If you're installing 90, it's a month's work by hand or a day with RPM, if you don't know the install procedure for the software beforehand. See ftp://oobleck.astro.cornell.edu/pub/clues.tar.gz for my attempt to tame this problem. As you can see from the dates of the files therein, I abandonned maintaining each package page as it came out under RPM. It was just impossible to keep up. It's trivial under RPM. Plus, I can uninstall! What matters is what the general users want, and they have spoken loudly in favor of package managers. Konrad writes: > RedHat > is rather conservative among the Linux distributors, but that is also > the reason for still using Python 1.5 in their latest versions. /usr/bin/python2.1 exists under 7.2, and claims in its startup that it existed in 7.1. You get 1.5.2 if you just type "python", but you get 2.1.1 if you type "python2.1". alias python python2.1... --jh-- From hinsen at cnrs-orleans.fr Fri Jan 4 13:45:02 2002 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Fri Jan 4 13:45:02 2002 Subject: [Numpy-discussion] RPMs out of date, have problems References: <001101c19551$37d3c6c0$0c01a8c0@freedom> <200201042009.g04K9ha15764@oobleck.astro.cornell.edu> Message-ID: <200201042144.g04LiqR01772@localhost.localdomain> Joe Harrington writes: > RPM. Plus, I can uninstall! What matters is what the general users > want, and they have spoken loudly in favor of package managers. I couldn't agree more. I tend to make many of my own RPMs, and still I prefer RPMs to straightforward installation in /usr/local. Clean updating and uninstalling is a big advantage. > /usr/bin/python2.1 exists under 7.2, and claims in its startup that it > existed in 7.1. You get 1.5.2 if you just type "python", but you get > 2.1.1 if you type "python2.1". alias python python2.1... Definitely not in 7.1, there was not Python 2.1 at all (before I installed it by hand). Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From tneidt at fidnet.com Sat Jan 5 02:40:04 2002 From: tneidt at fidnet.com (Tod M. Neidt) Date: Sat Jan 5 02:40:04 2002 Subject: [Numpy-discussion] Obscure setup question Message-ID: <1010227506.1141.6.camel@Q.neidt.net> Hi! I have noticed an interesting difference when installing Numeric-20.3 against python-2.1 vs. python-2.2. When installing against python-2.1, setup trys to write to /usr/lib/python2.1/distutils/command/install_headers.pyc But when installing against python-2.2, this doesn't happen. Note: this question came up because I am making an ebuild (basically a compile and install script) for Gentoo linux, http://www.gentoo.org. The attempted write to install_headers.pyc with python-2.1 violates Gentoo's sandboxed build environment causing the merge to fail. If you are interested see http://bugs.gentoo.org/show_bug.cgi?id=21 Just curious if there is a known explanation for this. The simple fix on my end, is to set the dependency >=python-2.2. Thanks for the great program! tod From gvermeul at labs.polycnrs-gre.fr Sat Jan 5 06:25:02 2002 From: gvermeul at labs.polycnrs-gre.fr (gvermeul at labs.polycnrs-gre.fr) Date: Sat Jan 5 06:25:02 2002 Subject: [Numpy-discussion] Obscure setup question Message-ID: <200201051424.g05EOCS26584@labs.polycnrs-gre.fr> Hi, This is very strange. I am also making packages (RPMs in my case). I do this as a normal user without write access to /usr/..... If running setup.py would have as a consequence that /usr/lib/....../install_headers.pyc would get overwritten, then setup.py would die. Can you tell if your python-2.1 is really 2.1 or 2.1.1? It happens that I am tweaking the setup.py script of Numeric and testing with python-2.1.1 and I am sure that distutils works. I suggest the following test: login as a normal user, unpack Numeric, cd Numeric and do python2.1 setup.py install --root=~/tmp This will install Numeric in ~/tmp/usr/lib/python2.1 and you will see if setup.py still tries to overwrite install_headers.pyc If so, your python2.1 is broken for some reason, if not your build system and/or sandbox are broken. Gerard > Hi! > > I have noticed an interesting difference when installing Numeric-20.3 > against python-2.1 vs. python-2.2. > > When installing against python-2.1, setup trys to write to > /usr/lib/python2.1/distutils/command/install_headers.pyc > > But when installing against python-2.2, this doesn't happen. > > Note: this question came up because I am making an ebuild (basically a > compile and install script) for Gentoo linux, http://www.gentoo.org. > The attempted write to install_headers.pyc with python-2.1 violates > Gentoo's sandboxed build environment causing the merge to fail. If you > are interested see > > http://bugs.gentoo.org/show_bug.cgi?id=21 > > Just curious if there is a known explanation for this. The simple fix > on my end, is to set the dependency >=python-2.2. > > Thanks for the great program! > > tod > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > --------------------------------------------- This message was sent using Endymion MailMan. http://www.endymion.com/products/mailman/ From gvermeul at labs.polycnrs-gre.fr Sat Jan 5 07:38:02 2002 From: gvermeul at labs.polycnrs-gre.fr (gvermeul at labs.polycnrs-gre.fr) Date: Sat Jan 5 07:38:02 2002 Subject: [Numpy-discussion] RPMs out of date, have problems Message-ID: <200201051536.g05FawS31545@labs.polycnrs-gre.fr> > > > I continue to be less than impressed by RPMs. Every time I try one I get > > version conflicts with other software. > Agreed, I always use RPMs because they help me to keep my systems in a reversible state. BUT, I never install binary RPMs that are not taylored to the Distro-Version that I am running (Try to install RH-7.2 packages on a RH-7.0). I may take source RPMs from other distributions as a starting point for my own tweaks, but I NEVER install 'foreign' RPMs. Attached you'll find a replacement for the setup.py script allowing to build a single binary and source RPM for Numeric + subpackages. The README.RPM for a normal user would be something like: (1) unpack python-numeric-XXX.YY.Z.tar.gz (2) cd python-numeric-XXX.YY.Z (3) python setup.py bdist_rpm This will build a binary and source RPM adapted to YOUR system (even yours, Paul, with your non-standard Python location) in the subdirectory dist (to be created automatically) (4) install with rpm -Uvh python-numeric-XXX.YY.Z-1.i?86.rpm You can always uninstall with rpm -e python-numeric. IMHO, this is a better way than trying to distibute binary RPMs. It happens that my favorite distribution is not on JH's list :-( But also, http://www.mandrakesoft.com/products/81/gaming-edition , the favorite toy of 11 years old whizkids, doing pygame-NumPy programming and who are certainly future students of astronomy. If you want to test the script: (1) replace the old setup.py with mine (2) rename Lib/Numeric.py to Lib/__init__.py (Numeric will become a real Python package). (3) python setup.py bdist (4) and install the RPM Caveats: (1) renaming Numeric to python-numeric has as consequence that the header files go into .../include/python2.1/python-numeric instead of .../include/python2.1/Numeric. Personally I don't like it (being author of a package that builds on Numeric). (I think this can be fixed, if needed). (2) I tried to acknowledge Paul Dubois' contributions in a long_description (the subpackages do not build to separate packages anymore). (3) The setup.py script shows how to add documentation to an RPM (requires to work around a bug in distutils). Personnally, I think that HTML and PDF documentation should also be added. In the current state of distutils this requires inclusion of the documentation in the python-numeric-XXX.YY.Z.tar.gz. Gerard --------------------------------------------- This message was sent using Endymion MailMan. http://www.endymion.com/products/mailman/ -------------- next part -------------- A non-text attachment was scrubbed... Name: setup.py.gz Type: application/x-gzip Size: 2553 bytes Desc: not available URL: From hinsen at cnrs-orleans.fr Sat Jan 5 09:00:02 2002 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Sat Jan 5 09:00:02 2002 Subject: [Numpy-discussion] RPMs out of date, have problems References: <200201051536.g05FawS31545@labs.polycnrs-gre.fr> Message-ID: <200201051659.g05Gxvf01911@localhost.localdomain> gvermeul at labs.polycnrs-gre.fr writes: > Caveats: > (1) renaming Numeric to python-numeric has as consequence that > the header files go into .../include/python2.1/python-numeric > instead of .../include/python2.1/Numeric. Personally I don't > like it (being author of a package that builds on Numeric). Me neither. But the fix is simple: take the distutils-generated spec file, edit the package name, and put it into the tar ball. The added advantage is that the binary RPM can be generated by rpm --rebuild numpy_xxx.tar.gz. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From gvermeul at labs.polycnrs-gre.fr Sat Jan 5 15:33:09 2002 From: gvermeul at labs.polycnrs-gre.fr (gvermeul at labs.polycnrs-gre.fr) Date: Sat Jan 5 15:33:09 2002 Subject: [Numpy-discussion] RPMs out of date, have problems Message-ID: <200201052332.g05NWnS28660@labs.polycnrs-gre.fr> > gvermeul at labs.polycnrs-gre.fr writes: > > > Caveats: > > (1) renaming Numeric to python-numeric has as consequence that > > the header files go into .../include/python2.1/python-numeric > > instead of .../include/python2.1/Numeric. Personally I don't > > like it (being author of a package that builds on Numeric). > > Me neither. But the fix is simple: take the distutils-generated spec > file, edit the package name, and put it into the tar ball. The added > advantage is that the binary RPM can be generated by rpm --rebuild > numpy_xxx.tar.gz. > You mean rpm -ta numpy_xxx.tar.gz Disadvantage: by default, the package will go into a distro dependent place (Mandrake, Red Hat and Suse are all different) and the package builder has to have write access to that place (Suse-7.3 grants it, Mandrake not, Red Hat don't know). This invites to building packages as root (DON'T). Building packages as a normal user protects you from erratic writes in /usr, /, ... Remember, we want to satisfy the needs of astronomer newbies. Users like you will allways find a solution. python setup.py bdist_rpm needs only write access to the user's home directory. Of course, adding a generic spec file is the easy 2 minutes solution, but I have choosen not to do that because it will invite bad policy (providing binary RPMs is asking for trouble). IMHO, the current setup.py/setup_all.py are not permitting distutils's way of 'RPM building'. It is better to fix that. I will try to see renaming the package from Numeric to python-numeric allows to install the headers in ../python2.1/Numeric instead of ./python2.1/python-numeric by changing my setup.py file. Gerard --------------------------------------------- This message was sent using Endymion MailMan. http://www.endymion.com/products/mailman/ From hinsen at cnrs-orleans.fr Sun Jan 6 02:25:02 2002 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Sun Jan 6 02:25:02 2002 Subject: [Numpy-discussion] RPMs out of date, have problems In-Reply-To: <200201052332.g05NWnS28660@labs.polycnrs-gre.fr> (gvermeul@labs.polycnrs-gre.fr) References: <200201052332.g05NWnS28660@labs.polycnrs-gre.fr> Message-ID: <200201061024.g06AOsY01387@localhost.localdomain> > Disadvantage: by default, the package will go into a distro dependent place > (Mandrake, Red Hat and Suse are all different) and the package builder > has to have write access to that place (Suse-7.3 grants it, Mandrake not, > Red Hat don't know). This invites to building packages as root (DON'T). RedHat doesn't by default (I changed that on my system). > python setup.py bdist_rpm needs only write access to the user's home > directory. Everywhere? It didn't when I first tried (that was an early distutils release under RedHat 6.2). > Of course, adding a generic spec file is the easy 2 minutes solution, > but I have choosen not to do that because it will invite bad policy > (providing binary RPMs is asking for trouble). And yet many people will not accept anything but binary RPMs, out of ignorance or fear. We won't make everyone happy... I admit that I never looked into fully automatic RPM generation by distutils as I usually need to add some manual steps to the build/install process. For example, distutils doesn't let me specify compiler options, but I want time-critical code to be compiled with the highest optimization level. Hopefully this will be addressed in distutils one day. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From gvermeul at labs.polycnrs-gre.fr Sun Jan 6 06:48:05 2002 From: gvermeul at labs.polycnrs-gre.fr (gvermeul at labs.polycnrs-gre.fr) Date: Sun Jan 6 06:48:05 2002 Subject: [Numpy-discussion] RPMs out of date, have problems Message-ID: <200201061447.g06ElbS26541@labs.polycnrs-gre.fr> > > > python setup.py bdist_rpm needs only write access to the user's home > > directory. > > Everywhere? It didn't when I first tried (that was an early distutils > release under RedHat 6.2). > Can you check which version of Python/distutils. If it is a version python that does not come with distutils, we can count it out. > > > Of course, adding a generic spec file is the easy 2 minutes solution, > > but I have choosen not to do that because it will invite bad policy > > (providing binary RPMs is asking for trouble). > > And yet many people will not accept anything but binary RPMs, out of > ignorance or fear. We won't make everyone happy... > Yes, and they will complain to JH if they are failing to install JH's binary RPMs. The best we can do is explain the motivation of NOT providing binary RPMs. If that is not sufficient for some, we can only advise that they switch to Windows or pester their distro providers for RPMs of the latest NumPy. > > I admit that I never looked into fully automatic RPM generation by > distutils as I usually need to add some manual steps to the > build/install process. For example, distutils doesn't let me specify > compiler options, but I want time-critical code to be compiled with > the highest optimization level. Hopefully this will be addressed in > distutils one day. > There are two ways to do this: (1) Extension( ..., extra_compile_args = [ "-O17" ], ...) This will append the optimization level 17 to the compiler flag (the last -O option counts). (2) export CFLAGS=-O3; python setup.py build This will append -O3 The result of (1) and (2) is gcc .... -017 -03 Gerard PS: distutils may look intimidating at first, but it is really simple compared to automake/autoconf. The disadvantage is that it is not very well tested. --------------------------------------------- This message was sent using Endymion MailMan. http://www.endymion.com/products/mailman/ From hinsen at cnrs-orleans.fr Sun Jan 6 13:34:03 2002 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Sun Jan 6 13:34:03 2002 Subject: [Numpy-discussion] RPMs out of date, have problems In-Reply-To: <200201061447.g06ElbS26541@labs.polycnrs-gre.fr> (gvermeul@labs.polycnrs-gre.fr) References: <200201061447.g06ElbS26541@labs.polycnrs-gre.fr> Message-ID: <200201062134.g06LY1G02583@localhost.localdomain> > Can you check which version of Python/distutils. If it is a version python > that does not come with distutils, we can count it out. I don't remember, and I don't have any of that stuff left. > Yes, and they will complain to JH if they are failing to install JH's > binary RPMs. The best we can do is explain the motivation of NOT > providing binary RPMs. If that is not sufficient for some, we can Somebody else will provide them, and you will still get complaints. I also get them. Fortunately there is a standard answer to send out. > There are two ways to do this: > > (1) Extension( ..., extra_compile_args = [ "-O17" ], ...) > This will append the optimization level 17 to the compiler flag > (the last -O option counts). But that will add the argument for all platforms. I need to specify compilation flags separately for each platform/compiler. I might even want to compile some modules with the new Intel compiler for Linux. > (2) export CFLAGS=-O3; python setup.py build > This will append -O3 For all extension modules - I need to specify different flags for each extension module, in one case even for a single source code file. My current solution is to build everything with distutils and then recompile the critical modules manually in the spec file. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From eric at enthought.com Sun Jan 6 20:29:03 2002 From: eric at enthought.com (eric) Date: Sun Jan 6 20:29:03 2002 Subject: [Numpy-discussion] RPMs out of date, have problems References: <200201061447.g06ElbS26541@labs.polycnrs-gre.fr> <200201062134.g06LY1G02583@localhost.localdomain> Message-ID: <08f601c1972b$3af04fa0$777ba8c0@ericlaptop> > > (1) Extension( ..., extra_compile_args = [ "-O17" ], ...) > > This will append the optimization level 17 to the compiler flag > > (the last -O option counts). > > But that will add the argument for all platforms. I need to specify > compilation flags separately for each platform/compiler. I might even > want to compile some modules with the new Intel compiler for Linux. > > > (2) export CFLAGS=-O3; python setup.py build > > This will append -O3 > > For all extension modules - I need to specify different flags for each > extension module, in one case even for a single source code file. This seems to be one of the big complaints about distutils for scientist (and maybe others). I think it is also one of Paul Dubois' major reasons for not liking distutils for Fortran files. I've run into this occasionally too and wished for a dictionary argument, maybe file_special_instructions, that had file names as keys and a dictionary of options for that file as the value. I think this might go some distance to solving the problem. file_special_instructions = { 'really_fast.c': { compiler = 'icc', override_compiler_flags = ['O2', 'tpp7'] } 'sorta_fast.c': { extra_compile_args = ['-O2']} } So an extension module that used this and had 'slow.c', 'really_fast.c', and 'kinda_fast.c' would use the standard compile tools for 'slow.c', use the specified compiler with the specified flags for 'really_fast.c', and add the specified compiler flags to the standard flags for 'sorta_fast.c'. This approach would provide quite a bit of flexiblity. To handle platform specific versions of file_special_instructions, I think you'd have to use if/thens within the setup.py file to build a separate set of instructions for each platform. There is one other major issue I've run into with distutils that I haven't found a way around. Sometimes you need compiler/linker flags inter-mingled with source or library files in a certain way. This can occur on SunOS when you want to link statically to some libraries and dynamically to others. distutils just doesn't provide a way of doing this that I have found. SciPy now has a package of extensions/changes to distutils called scipy_distutils helpful for building fortran based extension modules and other things needed by SciPy. We could experiment with adding something like the file_special_instructions flag if others thought it useful. Later, it could be folded back into distutils if the rest of the community wanted it. Thoughts? eric From paul at pfdubois.com Sun Jan 6 22:09:06 2002 From: paul at pfdubois.com (Paul F. Dubois) Date: Sun Jan 6 22:09:06 2002 Subject: distutils for scientific stuff, was RE: [Numpy-discussion] RPMs out of date, have problems In-Reply-To: <200201062134.g06LY1G02583@localhost.localdomain> Message-ID: <000901c19741$4c90a1e0$0c01a8c0@freedom> About doing different things in distutils for different platforms, that is easy. It is a Python script so you can set things depending on sys.platform. E.g. from Numerical, # You might need to add a case here for your system if sys.platform == 'win32': mathlibs = [] define_macros = [] undef_macros = ['HAVE_INVERSE_HYPERBOLIC'] elif sys.platform == 'beos5': mathlibs = [] Later the setup call uses these variables in its argument list. Adding Fortran support is quite a bit more difficult because the whole idea of distutils is to piggy-back off the Python configure, which doesn't configure Fortran compiler options or paths. I don't think distutils ought to try, really. You just compile the Fortran into a library and then use that in your setup.py. I think a possible way out is scons, but that is just a preliminary impression. It bears a strong resemblance to the system I was working on in 1998 and abandoned when I changed jobs. My theory was that the build should be rule-based, with finer and finer rules for special cases or platforms. The highest priority rule that governs a particular file, wins. E.g., Rule 1: To compile a .c file, do cc -O ... Rule 2: To compile foo.c, do cc -O3 ... Rule 3: To compile foo.c on platform win32, do cc -O1 ... Rule 4: compile bar.c but only on platform win32 There was more to it, because one of the tricky points is that nowadays files produce multiple outputs per execution and may need to be processed in more than one way. Note that if you add a new .c file or a new platform, you're covered unless it needs special treatment. Anyway, this is actually not a design area the numerical community ought to try to get into; there are people who have spent a lot of time on this already. Given that sentence, I can't explain why I was doing it, other than that I hate make so badly I was driven to it. The project was called MMD: Make Must Die. From pearu at cens.ioc.ee Sun Jan 6 23:02:04 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Sun Jan 6 23:02:04 2002 Subject: distutils for scientific stuff, was RE: [Numpy-discussion] RPMs out of date, have problems In-Reply-To: <000901c19741$4c90a1e0$0c01a8c0@freedom> Message-ID: On Sun, 6 Jan 2002, Paul F. Dubois wrote: > Adding Fortran support is quite a bit more difficult because the whole > idea of distutils is to piggy-back off the Python configure, which > doesn't configure Fortran compiler options or paths. I don't think > distutils ought to try, really. You just compile the Fortran into a > library and then use that in your setup.py. Saying that "just compile the Fortran into .." sounds like a common step for using Fortran, then I don't see why distutils shouldn't try to support this. As Eric mentioned in the previous message, scipy_distutils already does this step for an intermediate user. We have tested this to work for different compilers like Gnu,Intel F90,VAST,NAG F95,Absoft,etc and it works like a charm. Regards, Pearu From rob at pythonemproject.com Mon Jan 7 09:31:08 2002 From: rob at pythonemproject.com (Rob) Date: Mon Jan 7 09:31:08 2002 Subject: distutils for scientific stuff, was RE: [Numpy-discussion] RPMsout of date, have problems References: Message-ID: <3C39DA94.827E90EA@pythonemproject.com> Please when writing these utilities keep us BSD users in mind. I had to severely hack SciPy to get it to compile on FreeBSD, part of which was the Fortran utility. Rob. Pearu Peterson wrote: > > On Sun, 6 Jan 2002, Paul F. Dubois wrote: > > > Adding Fortran support is quite a bit more difficult because the whole > > idea of distutils is to piggy-back off the Python configure, which > > doesn't configure Fortran compiler options or paths. I don't think > > distutils ought to try, really. You just compile the Fortran into a > > library and then use that in your setup.py. > > Saying that "just compile the Fortran into .." sounds like a common step > for using Fortran, then I don't see why distutils shouldn't try to support > this. As Eric mentioned in the previous message, scipy_distutils already > does this step for an intermediate user. We have tested this to work for > different compilers like Gnu,Intel F90,VAST,NAG F95,Absoft,etc and it > works like a charm. > > Regards, > Pearu > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- The Numeric Python EM Project www.pythonemproject.com From bsder at allcaps.org Mon Jan 7 22:19:05 2002 From: bsder at allcaps.org (Andrew P. Lentvorski) Date: Mon Jan 7 22:19:05 2002 Subject: [Numpy-discussion] Sparse matricies Message-ID: <20020107221215.R65587-100000@mail.allcaps.org> I just looked through the archives and the web searches and didn't really find anything current on sparse matricies. Is there anything current for Python which handles sparse matrix codes? If someone is looking for a C implementation of sparse matrix handlers, I would refer to the sparse matrix package developed at Berekeley which is part of the Spice 3f4 circuit simulator (released under a modified old-style BSD license). I can provide pointers if they are desired. My goal is to actually create a useful circuit simulator wholly in Python so that it would be useful for both teaching and modification in special cases. Andy L. From oliphant at ee.byu.edu Tue Jan 8 08:01:12 2002 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Jan 8 08:01:12 2002 Subject: [Numpy-discussion] Sparse matricies In-Reply-To: <20020107221215.R65587-100000@mail.allcaps.org> Message-ID: > > Is there anything current for Python which handles sparse matrix codes? > > If someone is looking for a C implementation of sparse matrix handlers, I > would refer to the sparse matrix package developed at Berekeley which > is part of the Spice 3f4 circuit simulator (released under a modified > old-style BSD license). I can provide pointers if they are desired. I've done about three Sparse matrix packages built on SPARSEKIT, UMFPACK, and some of my own rolled stuff. SciPy has a subpackage for Sparse matrices, but it is not ready yet. I'd be interested in the package you are talking about. -Travis Oliphant From bsder at allcaps.org Tue Jan 8 18:45:02 2002 From: bsder at allcaps.org (Andrew P. Lentvorski) Date: Tue Jan 8 18:45:02 2002 Subject: [Numpy-discussion] Sparse matricies In-Reply-To: Message-ID: <20020108183038.O67212-100000@mail.allcaps.org> On Tue, 8 Jan 2002, Travis Oliphant wrote: > SciPy has a subpackage for Sparse matrices, but it is not ready yet. Well, I'm in the process of setting up the simulator now. I don't need the sparse matrix package immediately, but if there isn't anything useful in a week or so, I'll have to figure out a workaround. I'd *really* rather not. Even something which isn't quite ready would be better than what I would write (and then have to throw away). > I'd be interested in the package you are talking about. http://www-cad.eecs.berkeley.edu:80/Software/software.html The packge is SPICE, which is an electronic circuit simulator. Spice3f4 in particular. Download the Spice3f4 kit. Unpack it. The entire package is in src/lib/sparse. Andy L. From bsder at allcaps.org Tue Jan 8 18:52:01 2002 From: bsder at allcaps.org (Andrew P. Lentvorski) Date: Tue Jan 8 18:52:01 2002 Subject: [Numpy-discussion] Sparse matricies In-Reply-To: Message-ID: <20020108185024.V67212-100000@mail.allcaps.org> Perhaps next time I should dig on the web a little more ... On Tue, 8 Jan 2002, Travis Oliphant wrote: > I'd be interested in the package you are talking about. Also available as a separate package, http://buffy.eecs.berkeley.edu/IRO/Software/Catalog/Description/sparse1.3.html -a From bsder at allcaps.org Tue Jan 8 19:02:02 2002 From: bsder at allcaps.org (Andrew P. Lentvorski) Date: Tue Jan 8 19:02:02 2002 Subject: [Numpy-discussion] Sparse matricies In-Reply-To: Message-ID: <20020108190019.G67264-100000@mail.allcaps.org> Travis, Here is some of the most recent stuff I've found on sparse matricies. http://buffy.EECS.Berkeley.EDU/IRO/Summary/01abstracts/chapter18index.html http://buffy.EECS.Berkeley.EDU/IRO/Summary/01abstracts/ejr.1.html Andy L. From rob at pythonemproject.com Thu Jan 10 06:49:16 2002 From: rob at pythonemproject.com (Rob) Date: Thu Jan 10 06:49:16 2002 Subject: [Numpy-discussion] taking on new EM simulator project- ASAP Message-ID: <3C3DA93C.6AD46637@pythonemproject.com> This one will be interesting. I'm going to convert the Fortran ASAP MoM simulator to Numpy. I've already started and its bringing back bad memories of punch cards, etc. ASAP is a viable alternative to NEC2 for wire antennas, as it uses the Sommerfield conditions for imperfect ground. And its not too long of a program to port. The heavy stuff will be I/O. Fortunately, I can use f2c to decode the format statements :) Rob. -- The Numeric Python EM Project www.pythonemproject.com From pearu at cens.ioc.ee Thu Jan 10 07:09:02 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Thu Jan 10 07:09:02 2002 Subject: [Numpy-discussion] taking on new EM simulator project- ASAP In-Reply-To: <3C3DA93C.6AD46637@pythonemproject.com> Message-ID: On Thu, 10 Jan 2002, Rob wrote: > This one will be interesting. I'm going to convert the Fortran ASAP MoM > simulator to Numpy. I've already started and its bringing back bad > memories of punch cards, etc. ASAP is a viable alternative to NEC2 for > wire antennas, as it uses the Sommerfield conditions for imperfect > ground. And its not too long of a program to port. The heavy stuff > will be I/O. Fortunately, I can use f2c to decode the format statements > :) But why don't you use Fortran/Python binding tools like pyfort or f2py? Pearu From rob at pythonemproject.com Thu Jan 10 12:23:31 2002 From: rob at pythonemproject.com (rob at pythonemproject.com) Date: Thu Jan 10 12:23:31 2002 Subject: [Numpy-discussion] taking on new EM simulator project- ASAP Message-ID: <200201102024.PAA82239> An embedded and charset-unspecified text was scrubbed... Name: not available URL: From vanandel at atd.ucar.edu Thu Jan 17 06:47:21 2002 From: vanandel at atd.ucar.edu (Joe Van Andel) Date: Thu Jan 17 06:47:21 2002 Subject: [Numpy-discussion] viewing Gridded or Polar data? Message-ID: <3C46E3DC.B1F5F1A2@atd.ucar.edu> Can anyone recommend a visualization package that would display arrays of numbers by mapping values to a set of colors? That is, we define 32 colors, where the first color represents values 0-10, the second color represents 10-20, etc, and then each input value gets displayed as its corresponding color. I'd like to read netCDF files, if possible. Thanks much! -- Joe VanAndel National Center for Atmospheric Research http://www.atd.ucar.edu/~vanandel/ Internet: vanandel at ucar.edu From rob at pythonemproject.com Thu Jan 17 07:09:06 2002 From: rob at pythonemproject.com (Rob) Date: Thu Jan 17 07:09:06 2002 Subject: [Numpy-discussion] viewing Gridded or Polar data? References: <3C46E3DC.B1F5F1A2@atd.ucar.edu> Message-ID: <3C46E847.FC885A0C@pythonemproject.com> Animabob does that but I don't think it specificially handles NetCDF files. You can define your own color mapping file with Animabob. OpenDX has that capasbility. I've tried OpenDX, but all versions FreeBSD, and Cygwin that I tried dumped core upon trying to plot. Perhaps they've fixed that- it was fixed in CVS but not in the dist files. Rob. Joe Van Andel wrote: > > Can anyone recommend a visualization package that would display arrays > of numbers by mapping values to a set of colors? That is, we define 32 > colors, where the first color represents values 0-10, the second color > represents 10-20, etc, and then each input value gets displayed as its > corresponding color. > > I'd like to read netCDF files, if possible. > > Thanks much! > > -- > Joe VanAndel > National Center for Atmospheric Research > http://www.atd.ucar.edu/~vanandel/ > Internet: vanandel at ucar.edu > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- The Numeric Python EM Project www.pythonemproject.com From jsaenz at wm.lc.ehu.es Thu Jan 17 07:59:23 2002 From: jsaenz at wm.lc.ehu.es (Jon Saenz) Date: Thu Jan 17 07:59:23 2002 Subject: [Numpy-discussion] viewing Gridded or Polar data? In-Reply-To: <3C46E3DC.B1F5F1A2@atd.ucar.edu> Message-ID: I have never tried exactly this type of plot, but I guess that a combination of GrADS commands like the following ones in a script: grads -l << EOF sdfopen yourfile.nc (for COARDS-compliant netCDF files) set gxout grfill (to get a colour-filled grid) set clevs 10 20 30 40 50 (and so on to define your intervals...) d variable_name (of the netCDF file) enable print your_plot_name.gm print disable print quit EOF gxeps -c your_plot_name should do the work and prepare an EPS file. GrADS handles netCDF files, in case they conform to the COARDS standard. Otherwise, you can define the data in the file and use the xdfopen command, but this is more complex than above. http://grads.iges.org is their homepage. Hope this helps. Jon Saenz. | Tfno: +34 946012445 Depto. Fisica Aplicada II | Fax: +34 944648500 Facultad de Ciencias. \\ Universidad del Pais Vasco \\ Apdo. 644 \\ 48080 - Bilbao \\ SPAIN On Thu, 17 Jan 2002, Joe Van Andel wrote: > Can anyone recommend a visualization package that would display arrays > of numbers by mapping values to a set of colors? That is, we define 32 > colors, where the first color represents values 0-10, the second color > represents 10-20, etc, and then each input value gets displayed as its > corresponding color. > > I'd like to read netCDF files, if possible. > > Thanks much! > > -- > Joe VanAndel > National Center for Atmospheric Research > http://www.atd.ucar.edu/~vanandel/ > Internet: vanandel at ucar.edu > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From ransom at spock.physics.mcgill.ca Thu Jan 17 08:01:14 2002 From: ransom at spock.physics.mcgill.ca (Scott Ransom) Date: Thu Jan 17 08:01:14 2002 Subject: [Numpy-discussion] viewing Gridded or Polar data? In-Reply-To: <3C46E3DC.B1F5F1A2@atd.ucar.edu> References: <3C46E3DC.B1F5F1A2@atd.ucar.edu> Message-ID: <20020117160037.GA6390@spock.physics.mcgill.ca> Hi Joe, There are C and Python wrappers around the very nice PGPLOT library that do exactly what you want (and a whole lot more). You can find them here: ftp://cfa-ftp.harvard.edu/pub/ransom The files you need are: ppgplot-0.11.tar.gz (the C wrappers) Pgplot.py (the Python wrappers around the C wrappers) Lots of additional information about the capabilities of PGPLOT can be found at its web page: http://www.astro.caltech.edu/~tjp/pgplot/ With my current setup, you can do a lot of plotting (except 3D stuff) in a manner that is very much like IDL (a single function call with lots of optional keyword arguments). I haven't modified it in quite awhile, but still use it in my own work almost every day. As for reading netCDF, I would recommend Konrad Hinsen's Scientific Python: http://starship.python.net/crew/hinsen/scientific.html Good luck, Scott On Thu, Jan 17, 2002 at 07:46:52AM -0700, Joe Van Andel wrote: > Can anyone recommend a visualization package that would display arrays > of numbers by mapping values to a set of colors? That is, we define 32 > colors, where the first color represents values 0-10, the second color > represents 10-20, etc, and then each input value gets displayed as its > corresponding color. > > I'd like to read netCDF files, if possible. > > Thanks much! > > -- > Joe VanAndel > National Center for Atmospheric Research > http://www.atd.ucar.edu/~vanandel/ > Internet: vanandel at ucar.edu > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- -- Scott M. Ransom Address: McGill Univ. Physics Dept. Phone: (514) 398-6492 3600 University St., Rm 338 email: ransom at physics.mcgill.ca Montreal, QC Canada H3A 2T8 GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From nodwell at physics.ubc.ca Thu Jan 17 08:52:02 2002 From: nodwell at physics.ubc.ca (Eric Nodwell) Date: Thu Jan 17 08:52:02 2002 Subject: [Numpy-discussion] viewing Gridded or Polar data? In-Reply-To: <3C46E3DC.B1F5F1A2@atd.ucar.edu> References: <3C46E3DC.B1F5F1A2@atd.ucar.edu> Message-ID: <20020117165106.GA13128@holler.physics.ubc.ca> You might want to have a look at Gri (http://gri.sourceforge.net/). Gri can read netCDF files. Here's the little blurb from their site: "Gri is a language for scientific graphics programming. The word "language" is important: Gri is command-driven, not point/click. Some users consider Gri similar to LaTeX, since both provide extensive power as a reward for tolerating a learning curve. Gri can make x-y graphs, contour graphs, and image graphs. The output is in PostScript..." Since gri is command-driven, it's easy to call it from within python. Gri won't display the data for you - it will only generate the PostScript file, which you can then display with any number of tools. Eric On Thu, Jan 17, 2002 at 07:46:52AM -0700, Joe Van Andel wrote: > Can anyone recommend a visualization package that would display arrays > of numbers by mapping values to a set of colors? That is, we define 32 > colors, where the first color represents values 0-10, the second color > represents 10-20, etc, and then each input value gets displayed as its > corresponding color. > > I'd like to read netCDF files, if possible. > > Thanks much! > From paul at pfdubois.com Thu Jan 17 09:16:04 2002 From: paul at pfdubois.com (Paul F. Dubois) Date: Thu Jan 17 09:16:04 2002 Subject: [Numpy-discussion] viewing Gridded or Polar data? In-Reply-To: <3C46E3DC.B1F5F1A2@atd.ucar.edu> Message-ID: <000101c19f7a$1a476e80$0c01a8c0@freedom> We will answer you again shortly and include an example of how to do this in cdat (cdat.sf.net). You can do it using our GUI or from the Python command line. cdat includes a colormap editor for getting things just the way you want, and a variety of graphics methods including box fill, isofill, isoline, etc. It can open a wide variety of data files including netcdf. Our name, Climate Data Analysis Tools, is a bit misleading. There are special facilities for climate modelers, and a variety of statistics and other algorithms that they need, but the package is a general-purpose one. -----Original Message----- From: numpy-discussion-admin at lists.sourceforge.net [mailto:numpy-discussion-admin at lists.sourceforge.net] On Behalf Of Joe Van Andel Sent: Thursday, January 17, 2002 6:47 AM To: numpy-discussion Subject: [Numpy-discussion] viewing Gridded or Polar data? Can anyone recommend a visualization package that would display arrays of numbers by mapping values to a set of colors? That is, we define 32 colors, where the first color represents values 0-10, the second color represents 10-20, etc, and then each input value gets displayed as its corresponding color. I'd like to read netCDF files, if possible. Thanks much! -- Joe VanAndel National Center for Atmospheric Research http://www.atd.ucar.edu/~vanandel/ Internet: vanandel at ucar.edu _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From feiguin at magnet.fsu.edu Thu Jan 17 12:47:08 2002 From: feiguin at magnet.fsu.edu (Adrian Feiguin) Date: Thu Jan 17 12:47:08 2002 Subject: [Numpy-discussion] viewing Gridded or Polar data? In-Reply-To: <3C46E3DC.B1F5F1A2@atd.ucar.edu> Message-ID: You can take a look at SciGraphica at http://scigraphica.sourceforge.net. Although it is mainly GUI based, it has a python interface under development. You can ask in the scigraphica mailing list for a plugin for reading netcdf files, I remember someone posting it. Saludos, On Thu, 17 Jan 2002, Joe Van Andel wrote: > Can anyone recommend a visualization package that would display arrays > of numbers by mapping values to a set of colors? That is, we define 32 > colors, where the first color represents values 0-10, the second color > represents 10-20, etc, and then each input value gets displayed as its > corresponding color. > > I'd like to read netCDF files, if possible. > > Thanks much! > > -- > Joe VanAndel > National Center for Atmospheric Research > http://www.atd.ucar.edu/~vanandel/ > Internet: vanandel at ucar.edu > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From paul at pfdubois.com Thu Jan 17 14:09:24 2002 From: paul at pfdubois.com (Paul F. Dubois) Date: Thu Jan 17 14:09:24 2002 Subject: [Numpy-discussion] Please help me test a new setup.py In-Reply-To: <02011719534500.00818@taco.polycnrs-gre.fr> Message-ID: <000201c19fa3$1fb92ec0$0c01a8c0@freedom> Working from something given to me by Gerard Vermeulen I have tried to improve Numeric's setup.py. This is now in CVS. I could use help testing it, particularly by the those whose brains are RPM-enabled. Those Numpy developers who have not done a damn thing lately are granted this chance for redemption. (:-> There is no more setup_all.py; rather, setup.py builds everything. It has, however, a structure that enables easy commenting-out of any optional packages. I have made and tested the Windows packages and installer; and made and tested Linux packages and have made but not tested the Linux RPM. (sh makedist.sh). If trying this on a Python will Numeric already installed, remove first from site-packages: Numeric.pth, Numeric, FFT, RNG, MA, kinds, PropertiedClasses Under the new scheme, Numeric.pth points to Numeric and the directories for the others are INSIDE Numeric. Everything should be backward compatible. I like this; it reduces the visible pollution in site-packages. The RPM produced does not contain the documentation (that runs into a distutils bug in older Pythons). It does not have a name beginning with python- because that screwed up something else. In short, it is a compromise. Once we're all convinced that what I have works then we can worry about the finer stuff. A nice side benefit is that there is now only one Windows .exe installer, and only one thing to uninstall in add/remove programs. From rob at hooft.net Thu Jan 17 23:07:03 2002 From: rob at hooft.net (Rob Hooft) Date: Thu Jan 17 23:07:03 2002 Subject: [Numpy-discussion] Re: viewing Gridded or Polar data? Message-ID: <3C47C941.9000008@hooft.net> I wrote a program to display X-ray diffraction images. The display engine is available as "dis.py" from http://starship.python.net/crew/hooft/ Regards, Rob Hooft -- Rob W.W. Hooft || rob at hooft.net || http://www.hooft.net/people/rob/ From gvermeul at polycnrs-gre.fr Fri Jan 18 00:57:08 2002 From: gvermeul at polycnrs-gre.fr (Gerard Vermeulen) Date: Fri Jan 18 00:57:08 2002 Subject: [Numpy-discussion] Please help me test a new setup.py In-Reply-To: <000201c19fa3$1fb92ec0$0c01a8c0@freedom> References: <000201c19fa3$1fb92ec0$0c01a8c0@freedom> Message-ID: <02011809564200.01995@taco.polycnrs-gre.fr> If I do setup.py bdist_rpm then 'Numeric.pth' does not end up in the binary RPM with Python-2.1.1 The reason is that rpm does something like setup.py install --root=~/tmp --record=record.txt and apparently the distutils part that handles --record is broken as shown by 'grep pth record.txt'. Paul, do you want me to 'fix' distutils (meaning overriding the install command, so that record.txt contains all files)? Or are you willing to use the pth_install_data class that does not work on Windows? Gerard On Thursday 17 January 2002 23:05, Paul F. Dubois wrote: > Working from something given to me by Gerard Vermeulen I have tried to > improve Numeric's setup.py. This is now in CVS. I could use help testing > it, particularly by the those whose brains are RPM-enabled. Those Numpy > developers who have not done a damn thing lately are granted this chance > for redemption. (:-> > > There is no more setup_all.py; rather, setup.py builds everything. It > has, however, a structure that enables easy commenting-out of any > optional packages. > > I have made and tested the Windows packages and installer; and made and > tested Linux packages and have made but not tested the Linux RPM. (sh > makedist.sh). > > If trying this on a Python will Numeric already installed, remove first > from site-packages: > Numeric.pth, Numeric, FFT, RNG, MA, kinds, PropertiedClasses > > Under the new scheme, Numeric.pth points to Numeric and the directories > for the others are INSIDE Numeric. Everything should be backward > compatible. I like this; it reduces the visible pollution in > site-packages. > > The RPM produced does not contain the documentation (that runs into a > distutils bug in older Pythons). It does not have a name beginning with > python- because that screwed up something else. In short, it is a > compromise. Once we're all convinced that what I have works then we can > worry about the finer stuff. > > A nice side benefit is that there is now only one Windows .exe > installer, and only one thing to uninstall in add/remove programs. > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From paul at pfdubois.com Mon Jan 21 20:43:06 2002 From: paul at pfdubois.com (Paul F. Dubois) Date: Mon Jan 21 20:43:06 2002 Subject: [Numpy-discussion] cvs checkins for rpms Message-ID: <000001c1a2fe$d3df3c60$0c01a8c0@freedom> Version 21.0a2 is now in cvs awaiting community testing. Especially to be tested are rpm creation, Windows installer creation. Includes new files setup.cfg and rpm_install.sh from Gerard Vermeulen. From jochen at jochen-kuepper.de Mon Jan 21 21:27:02 2002 From: jochen at jochen-kuepper.de (Jochen =?iso-8859-1?q?K=FCpper?=) Date: Mon Jan 21 21:27:02 2002 Subject: [Numpy-discussion] cvs checkins for rpms In-Reply-To: <000001c1a2fe$d3df3c60$0c01a8c0@freedom> References: <000001c1a2fe$d3df3c60$0c01a8c0@freedom> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mon, 21 Jan 2002 20:39:47 -0800 Paul F Dubois wrote: Paul> Includes new files setup.cfg and rpm_install.sh from Gerard Paul> Vermeulen. ,----[python setup.py bdist_rpm] | [...] | building 'kinds._kinds' extension | gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -IInclude -IPackages/FFT/Include -IPackages/RNG/Include -I/usr/local/include/python2.2 -c Packages/kinds/Src/_kindsmodule.c -o build/temp.linux-i686-2.2/_kindsmodule.o -O2 -march=i386 -mcpu=i686 | Packages/kinds/Src/_kindsmodule.c:15: warning: function declaration isn't a prototype | gcc -shared build/temp.linux-i686-2.2/_kindsmodule.o -o build/lib.linux-i686-2.2/kinds/_kinds.so | MA Version 11.1.0 | Properties Version 2.2 | kinds Version 1.1 | Numeric Version 21.0a2 | + exit 0 | Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.35738 | + umask 022 | + cd /home/software/programming/numeric/NumPy/build/bdist.linux-i686/rpm/BUILD | + cd Numeric-21.0a2 | + setup.py install --root=/var/tmp/Numeric-buildroot | /var/tmp/rpm-tmp.35738: setup.py: command not found | error: Bad exit status from /var/tmp/rpm-tmp.35738 (%install) `---- Greetings, Jochen - -- Einigkeit und Recht und Freiheit http://www.Jochen-Kuepper.de Libert?, ?galit?, Fraternit? GnuPG key: 44BCCD8E Sex, drugs and rock-n-roll -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Processed by Mailcrypt and GnuPG iD8DBQE8TPgIiJ/aUUS8zY4RArEoAJ0SsWQT+F3DNfh+YTaqk8lXpXsbzgCfQe27 GjZ+/DrL9sYbjPj3Ma+VLV4= =17Dr -----END PGP SIGNATURE----- From jochen at jochen-kuepper.de Mon Jan 21 21:42:02 2002 From: jochen at jochen-kuepper.de (Jochen =?iso-8859-1?q?K=FCpper?=) Date: Mon Jan 21 21:42:02 2002 Subject: [Numpy-discussion] cvs checkins for rpms In-Reply-To: References: <000001c1a2fe$d3df3c60$0c01a8c0@freedom> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 22 Jan 2002 00:26:33 -0500 Jochen K?pper wrote: Jochen> ,----[python setup.py bdist_rpm] [...] Jochen> | + setup.py install --root=/var/tmp/Numeric-buildroot Jochen> | /var/tmp/rpm-tmp.35738: setup.py: command not found Jochen> | error: Bad exit status from /var/tmp/rpm-tmp.35738 (%install) Jochen> `---- Looking at my own message I realize that the following patch is all that is needed. '.' shouldn't be in your path, it definitely won't be in mine. Index: rpm_install.sh =================================================================== RCS file: /cvsroot/numpy/Numerical/rpm_install.sh,v retrieving revision 1.1 diff -u -r1.1 rpm_install.sh - --- rpm_install.sh 2002/01/22 04:39:53 1.1 +++ rpm_install.sh 2002/01/22 05:39:50 @@ -1,4 +1,4 @@ - -setup.py install --root=$RPM_BUILD_ROOT +./setup.py install --root=$RPM_BUILD_ROOT cat >INSTALLED_FILES < iD8DBQE8TPuQiJ/aUUS8zY4RApgcAJ4mlgsdCbJoiXnF4s4IyENE1m+D2ACfXdGU dtpO/3TQEiV+kjY79ZmDUPE= =/wKD -----END PGP SIGNATURE----- From paul at pfdubois.com Wed Jan 23 11:59:45 2002 From: paul at pfdubois.com (Paul Dubois) Date: Wed Jan 23 11:59:45 2002 Subject: [Numpy-discussion] yacvsu Message-ID: <000701c1a448$1e69df20$09860cc0@CLENHAM> yet-another-cvs-update Disregarding advice from wiser heads, I made setup.py generate rpm_install.sh so that it would execute setup.py with the "right" python. Works for me (TM) in the sense that the rpm builds. I'm afraid to test it because I don't want my system ruined. We need some words someplace about how one uses these rpms. I suppose the web site is the right place. Also new in this issue is an upgrade to MA.average to allow axial weights as Numeric now does. I replaced all memcpy's with memmove's due to a bug report. If someone thinks this is a hideous decision just let me know why. From kragen at pobox.com Wed Jan 23 22:24:08 2002 From: kragen at pobox.com (Kragen Sitaker) Date: Wed Jan 23 22:24:08 2002 Subject: [Numpy-discussion] memory-mapped Numeric arrays: arrayfrombuffer version 2 Message-ID: <20020124012343.A13898@canonical.org> I would be very happy if this got included in the Numpy distribution, so that people don't have to do an extra download to get this facility. I just submitted it as a patch on SourceForge, but my browser failed to upload the file. http://sourceforge.net/tracker/index.php?func=detail&aid=507847&group_id=1369&atid=301369 There is one thing in arrayfrombuffer.c I'm not sure about, and I could use some help here. arrayobj is the return value from PyArray_FromDimsAndData: /* do I need to incref arrayobj?! fromstring does... */ return (PyObject*)arrayobj; } So, do I? Or not? It seems to work as it is. Following is the announcement I posted to comp.lang.python and (hopefully) comp.lang.python.announce. The 'arrayfrombuffer' package features support for Numerical Python arrays whose contents are stored in buffer objects, including memory-mapped files. This has the following advantages: - loading your array from a file is easy --- a module import and a single function call --- and doesn't use excessive amounts of memory. - loading your array is quick; it doesn't need to be copied from one part of memory to another in order to be loaded. - your array gets demand-loaded; parts you aren't using don't need to be in memory or in swap. - under memory-pressure conditions, your array doesn't use up swap, and parts of it you haven't modified can be evicted from RAM without the need for a disk write - your arrays can be bigger than your physical memory - when you modify your array, only the parts you modify get written back out to disk This is something that's been requested on the Numpy list a few times a year since 1999. arrayfrombuffer lives at http://pobox.com/~kragen/sw/arrayfrombuffer/ The current version is version 2; it is released under the X11 license (the BSD license without the advertising clause). From dubois1 at llnl.gov Fri Jan 25 09:44:04 2002 From: dubois1 at llnl.gov (Paul F. Dubois) Date: Fri Jan 25 09:44:04 2002 Subject: [Numpy-discussion] RE: memory-mapped Numeric arrays: arrayfrombuffer version 2 In-Reply-To: Message-ID: <000001c1a5c7$5d4bacc0$0c01a8c0@freedom> I have verified that this package seems to work on Windows. I says seems only because I didn't try enough to uncover anything subtle. Unless or until we are convinced as a community that this is (a) the right way to do this and (b) that the package is portable, it would not be wise to put it in the main distribution. I would like to hear from the community about this so that I will know whether or not to add this package as a separate SourceForge 'package' within the Numerical Python area. Meantime I will add a link to the web page. -----Original Message----- From: python-announce-list-admin at python.org [mailto:python-announce-list-admin at python.org] On Behalf Of Kragen Sitaker Sent: Wednesday, January 23, 2002 9:40 PM To: python-announce-list at python.org Subject: memory-mapped Numeric arrays: arrayfrombuffer version 2 The 'arrayfrombuffer' package features support for Numerical Python arrays whose contents are stored in buffer objects, including memory-mapped files. This has the following advantages: - loading your array from a file is easy --- a module import and a single function call --- and doesn't use excessive amounts of memory. - loading your array is quick; it doesn't need to be copied from one part of memory to another in order to be loaded. - your array gets demand-loaded; parts you aren't using don't need to be in memory or in swap. - under memory-pressure conditions, your array doesn't use up swap, and parts of it you haven't modified can be evicted from RAM without the need for a disk write - your arrays can be bigger than your physical memory - when you modify your array, only the parts you modify get written back out to disk This is something that's been requested on the Numpy list a few times a year since 1999. arrayfrombuffer lives at http://pobox.com/~kragen/sw/arrayfrombuffer/ The current version is version 2; it is released under the X11 license (the BSD license without the advertising clause).

arrayfrombuffer 2 - creates Numeric arrays from memory-mapped files. (23-Jan-02) -- http://mail.python.org/mailman/listinfo/python-announce-list From nodwell at physics.ubc.ca Fri Jan 25 10:41:08 2002 From: nodwell at physics.ubc.ca (Eric Nodwell) Date: Fri Jan 25 10:41:08 2002 Subject: [Numpy-discussion] RE: memory-mapped Numeric arrays: arrayfrombuffer version 2 In-Reply-To: <000001c1a5c7$5d4bacc0$0c01a8c0@freedom> References: <000001c1a5c7$5d4bacc0$0c01a8c0@freedom> Message-ID: <20020125184013.GA28841@holler.physics.ubc.ca> Since I have a 2.4GB data file handy, I thought I'd try this package with it. (Normally I process this data file by reading it in a chunk at a time, which is perfectly adequate.) Not surprisinly, it chokes: File "/home/eric/lib/python2.2/site-packages/maparray.py", line 15, in maparray m = mmap.mmap(fn, os.fstat(fn)[stat.ST_SIZE]) OverflowError: memory mapped size is too large (limited by C int) (details: Python 2.2, numpy 20.3, Pentium III, Debian Woody, Linux kernel 2.4.13, gcc 2.95.4) I'm not a big C programmer, but I wonder if there is some way for this package to overcome the 2GB limit on 32-bit systems. That could be useful in some situations. Eric On Fri, Jan 25, 2002 at 09:40:21AM -0800, Paul F. Dubois wrote: > > I have verified that this package seems to work on Windows. I says seems > only because I didn't try enough to uncover anything subtle. > > Unless or until we are convinced as a community that this is (a) the > right way to do this and (b) that the package is portable, it would not > be wise to put it in the main distribution. > > I would like to hear from the community about this so that I will know > whether or not to add this package as a separate SourceForge 'package' > within the Numerical Python area. Meantime I will add a link to the web > page. > > From: python-announce-list-admin at python.org > [mailto:python-announce-list-admin at python.org] On Behalf Of Kragen > Sitaker > Sent: Wednesday, January 23, 2002 9:40 PM > To: python-announce-list at python.org > Subject: memory-mapped Numeric arrays: arrayfrombuffer version 2 > > > The 'arrayfrombuffer' package features support for Numerical Python > arrays whose contents are stored in buffer objects, including > memory-mapped files. This has the following advantages: > > - loading your array from a file is easy --- a module import and a > single function call --- and doesn't use excessive amounts of > memory. > - loading your array is quick; it doesn't need to be copied from one > part of memory to another in order to be loaded. > - your array gets demand-loaded; parts you aren't using don't need to > be in memory or in swap. > - under memory-pressure conditions, your array doesn't use up swap, > and parts of it you haven't modified can be evicted from RAM without > the need for a disk write > - your arrays can be bigger than your physical memory > - when you modify your array, only the parts you modify get written > back out to disk > > This is something that's been requested on the Numpy list a few times a > year since 1999. > > arrayfrombuffer lives at http://pobox.com/~kragen/sw/arrayfrombuffer/ > The current version is version 2; it is released under the X11 license > (the BSD license without the advertising clause). > > > >

HREF="http://pobox.com/~kragen/sw/arrayfrombuffer/">arrayfrombuffer > 2 - creates Numeric arrays from memory-mapped files. (23-Jan-02) > -- ******************************** Eric Nodwell Ph.D. candidate Department of Physics University of British Columbia tel: 604-822-5425 fax: 604-822-4750 nodwell at physics.ubc.ca From hinsen at cnrs-orleans.fr Fri Jan 25 11:26:05 2002 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Fri Jan 25 11:26:05 2002 Subject: [Numpy-discussion] memory-mapped Numeric arrays: arrayfrombuffer version 2 In-Reply-To: <20020124012343.A13898@canonical.org> References: <20020124012343.A13898@canonical.org> Message-ID: Kragen Sitaker writes: > There is one thing in arrayfrombuffer.c I'm not sure about, and > I could use some help here. arrayobj is the return value from > PyArray_FromDimsAndData: > /* do I need to incref arrayobj?! fromstring does... */ > return (PyObject*)arrayobj; > } > > So, do I? Or not? It seems to work as it is. And it is correct, in my opinion. Your routine is the owner of the array (and no one else keeps a reference), so you just pass on ownership. Doing an INCREF here would mean that the array is never freed. (Keep in mind that the data area is never freed anyway!) Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From jjl at pobox.com Fri Jan 25 15:12:01 2002 From: jjl at pobox.com (John J. Lee) Date: Fri Jan 25 15:12:01 2002 Subject: [Numpy-discussion] amap: new function / ufunc method? Message-ID: Having written all this down, I'm half-expecting somebody is going to point out some simple way of doing this with existing builtin functions, but here goes... In the process of 'porting' Konrad Hinsen's Histogram class (from the Scientific package) to N dimensions, I ran across the problem of how to apply a function to an array for a specified set of indices. In this case, the problem is to increment an N-dimensional array for each of a set of indices. In general, it occurred to me that it would be useful to have a function (named 'amap' here) to loop over a specified set of indices, applying a function to a set of arguments taken from a set of argument arrays (similar to the map buitin function in this respect), and filling in the corresponding elements in a target array with the results. For example, here is a bit of the histogram problem: [The original 1D Histogram class uses a clever array-broadcasting trick which creates intermediate rank-2 arrays, feeding in the data in chunks to avoid quadratic behaviour. This is fine with 1D arrays, but in poor taste for the N dimensional case, if it works at all.] #!/usr/bin/env python import Numeric as N def flatten_indices(shape, indices): dims = indices.shape[0] facs = pow(shape, N.arange(dims))[::-1,N.NewAxis] return N.add.reduce(facs*indices) def amap(target, function, indices, *arg_arrays): indices = N.asarray(indices) arg_arrays = map(N.asarray, arg_arrays) if not isinstance(target, N.ArrayType): raise TypeError, "target must be an array" if not target.iscontiguous(): raise ValueError, "target must be contiguous" if not N.alltrue([aa.iscontiguous() for aa in arg_arrays]): raise ValueError, "all arg_arrays must be contiguous" if not len(indices.shape) == 1: raise ValueError, "indices must be rank-1" tf = target.flat for index in indices: args = [arg_array.flat[index] for arg_array in arg_arrays] tf[index] = function(*args) # construct a 2 dimensional histogram given indices, ind, of data points nbins = (5, 4) # 5 bins along one axis, 4 along the other histo = N.zeros(nbins) # empty histogram # these are the indices into the histogram at which data points lie: # histo[1,0] should be incremented twice, histo[2,2] should be incremented # once, etc. ind = N.array([[1, 1, 2, 4], [0, 0, 2, 3]]) flat_ind = flatten_indices(histo.shape, ind) assert flat_ind == N.array([4, 4, 10, 19]) # Now we want to increment histo for each index pair ind[:,n] (in this # 2D case) def increment(x): return x+1 amap(histo, increment, flat_ind, histo) print histo Actually, I just realised that calling it something related to 'map' is rather misleading, because the builtin Python map generates a list of the same length as the sequence passed in; this function doesn't: it 'generates' (fills in, in fact) an array of the same size as the target passed in. Can't think of a better name, though. So: is there a way of doing this (implementing amap) efficiently already? I don't think there is, but I'd be interested to be proved wrong. I admit I don't like the relatively large number of arguments, but I still think it is worthwhile. You could make the *arg_arrays a single argument, and make target an optional argument, at the end of the arguments: def amap(function, indices, arg_arrays, target=None): so that by default, the target array is first created as zeros, then filled in and returned, or perhaps even better: def amap(function, arg_arrays, indices=None, target=None): though the special case where you don't give the indices argument is already achievable without this function, so it could be argued that hiding away the indices argument as a mere optional argument obscures the real point of the function. The last form also has the (debatable) benefit of being similar to the built-in Python map: def map(function, *args): The example call above then looks like this: histo = amap(increment, [histo], indices=flat_ind) Another couple of tweaks might be to allow scalar values to replace the arg_arrays argument, and to allow normal indices rather than flattened ones (the only reason I started out with flat indices was that put() did the same): def amap(function, args, indices=None, target=None): So that the example call looks like: histo = amap(increment, [histo], indices=ind) or histo = amap(Numeric.add, [histo, 1], ind) I like those last two. Even if there is already an efficient way of doing this with existing Numeric functions, I think this is a natural way of thinking about some operations on arrays -- this one, at least -- and would be useful to have in Numeric. For real efficiency, I suppose the right thing to do would be to have a map method on ufuncs, as is the case for reduce: histo = Numeric.add.map([histo, 1], ind) but it would be more work to implement than just a standard function, and less general. I've made a first attempt at the function, in C (will upload diff against 20.2.0 if anyone so requests). It only accepts one arg array, probably doesn't really do exactly the same as the Python function above (for better or worse), and it's a mess of a function, since I rarely write anything in C. It is also incorrect in at least some ways (most obviously, it only deals with Python floats / C doubles / Numpy PyArray_DOUBLEs), though there are some tests (which pass). I'm wasn't sure how fast it would be, since after all it's only using ordinary Python functions, not ufuncs, but my crude tests appeared to show an improvement of about a factor of two over an optimised version of the Python function above. The slow bit seems to be constructing the argument list / passing arg list to function, so I removed that to make the comparison, along with an unecessary attribute access in the loop. Presumably the general (> 1 arg_array) version would show a bigger improvement over the corresponding Python version. By the way, does anyone understand what the reduceat ufunc method is supposed to do? Seems to be the Beale cipher of Numeric -- having studied its output to see if it was the function I was looking for, I can find no sense in it and kind of suspect that it was constructed only to keep us puzzling. ;) Is Jim Hugunin not available for comment on the matter, or is this somebody else's dirty work? John From rob at pythonemproject.com Fri Jan 25 17:59:01 2002 From: rob at pythonemproject.com (Rob) Date: Fri Jan 25 17:59:01 2002 Subject: [Numpy-discussion] Column in IEEE trans Antennas and Propagation Message-ID: <3C520C85.E0938CCD@pythonemproject.com> I've been asked to write a short column about Numpy and my web site in IEEE AP, for the Educational Corner. I still consider myself a Numpy neophyte, but perhaps thats the beauty of Python, being able to do neat things without being an expert. I am pondering what to write about on the Numpy side. I am soliciting any genious ideas for topics :) The first thing I thought of is that I just plain enjoy being able to tweek code without worrying about compilers. Second, the code ends up so much more readable than Fortran or C. Thirdly, vectorizing the ported code makes it look much more like the mathematical basis of the program. Lastly, I think its the perfect language for students learning algorithms. I'm open to any ideas. Thanks, Rob. -- The Numeric Python EM Project www.pythonemproject.com From cavallo at kip.uni-heidelberg.de Sat Jan 26 14:01:02 2002 From: cavallo at kip.uni-heidelberg.de (cavallo at kip.uni-heidelberg.de) Date: Sat Jan 26 14:01:02 2002 Subject: [Numpy-discussion] kdf file format I/O routine In-Reply-To: Message-ID: Hy, this is my first attempt to give some back to the python/numerical-python community. Here (http://kdfio.sourceforge.net) you can find an early release of my I/O routine for reading and writing kdf file as in khoros environment. Khoros (from http://www.khoral.com) is an interactive program for digital image processing, we are using in our lab (university of heidelberg). All the time i've found hard to use such environment because it lacks some flexibility i need. Now i've been using numerical python for some time and i would other will use it as standard base for numerical computation, but for my purpose it was lacking I/O with such an environment (khoros). Khoros is not free, but anyway if you are using it i hope you will like these routines to read and write their files. These are in an early stage, so expect a lot of bugs, missing feature and so on: it's only my first project so feedbacks, suggestions and complains are welcome. Ops, i've forgotten: i'm personally using this rotines for my PHD, so they should be reasonably stable. The license should be BSD? GPL? LGPL? any comment is welcome, because i never understood these problem (it is my first project, didn't i tell you?). best regards to all GREAT folks in this group, antonio cavallo ps. you can download khoros if you are a student from: http://www.khoral.com/khoros/kp2001_student From magnus at hetland.org Sat Jan 26 14:06:04 2002 From: magnus at hetland.org (Magnus Lie Hetland) Date: Sat Jan 26 14:06:04 2002 Subject: [Numpy-discussion] kdf file format I/O routine In-Reply-To: ; from cavallo@kip.uni-heidelberg.de on Sat, Jan 26, 2002 at 10:50:38PM +0100 References: Message-ID: <20020126230551.B3021@idi.ntnu.no> cavallo at kip.uni-heidelberg.de : ... > The license should be BSD? GPL? LGPL? any comment is welcome, because i > never understood these problem (it is my first project, didn't i tell > you?). If you don't specifically want to follow the philosophy of the GPL, I'd suggest that you go with some BSD variant (e.g. MIT, which is even simpler). That way, you place less restrictions on its use/distributions, and more people can/will use it. (YMMV, of course.) -- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org From jochen at unc.edu Sun Jan 27 07:36:03 2002 From: jochen at unc.edu (Jochen =?iso-8859-1?q?K=FCpper?=) Date: Sun Jan 27 07:36:03 2002 Subject: [Numpy-discussion] Character typecode Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 According to the manual there is a typecode 'Character'. But the following program gives an error: ,---- | import Numeric as num | data = num.array([1,2,3], num.Character) `---- ,---- | Traceback (most recent call last): | File "", line 2, in ? | AttributeError: 'module' object has no attribute 'Character' `---- This is python-2.2, latest NumPy cvs. The following line fixes that. Any reason that isn't in there? Index: Lib/Precision.py =================================================================== RCS file: /cvsroot/numpy/Numerical/Lib/Precision.py,v retrieving revision 1.1.1.1 diff -u -r1.1.1.1 Precision.py - --- Lib/Precision.py 2000/01/13 21:23:06 1.1.1.1 +++ Lib/Precision.py 2002/01/27 15:30:50 @@ -33,6 +33,7 @@ return typecode raise PrecisionError, key+" of "+str(required_bits)+" bits not available on this system" +Character = 'c' UnsignedInt8 = 'b' try: Int0 = _lookup(_code_table, 'Integer', 0) Greetings, Jochen PS: The manual page http://pfdubois.com/numpy/html2/numpy-6.html is really messed up right at the spot where typecodes are explained. That is with Netscape-4.7 and Mozilla-0.9.7 on Linux and Win2000. PPS: What format is the manual master deocument in? Would it be feasible to generate info pages from that? - -- University of North Carolina phone: +1-919-962-4403 Department of Chemistry phone: +1-919-962-1579 Venable Hall CB#3290 (Kenan C148) fax: +1-919-843-6041 Chapel Hill, NC 27599, USA GnuPG key: 44BCCD8E -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6-cygwin-fcn-1 (Cygwin) Comment: Processed by Mailcrypt and GnuPG iD8DBQE8VB5XiJ/aUUS8zY4RAhIxAJ43vXeTvN0hlbVmCsSCGCGYrt2m/wCeOvH8 lodh8d0YmJy0eCqMyglNdj0= =z/Wo -----END PGP SIGNATURE----- From oliphant.travis at ieee.org Sun Jan 27 22:10:04 2002 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Jan 27 22:10:04 2002 Subject: [Numpy-discussion] amap: new function / ufunc method? In-Reply-To: References: Message-ID: > In the process of 'porting' Konrad Hinsen's Histogram class (from the > Scientific package) to N dimensions, I ran across the problem of how to > apply a function to an array for a specified set of indices. If I understand what you are trying to do, there is a function called arraymap in the SciPy package (special module) (it was in the Numeric source for a short time but I think we decided to take it out --- I can't remember why). I think that using this function and a combination of take and put can do what you describe. It is available as scipy.special.arraymap Best regards, Travis Oliphant From nwagner at mecha.uni-stuttgart.de Mon Jan 28 00:37:02 2002 From: nwagner at mecha.uni-stuttgart.de (Nils Wagner) Date: Mon Jan 28 00:37:02 2002 Subject: [Numpy-discussion] Matlab to Numpy Message-ID: <3C551C48.E5CEF797@mecha.uni-stuttgart.de> Hi all, Who can send me a program written in Numpy (+Vpython for graphics), that reproduces the results of the following Matlab code x=linspace(0,1,25); t=linspace(0,2,50); [X,T] = meshgrid(x,t); z=exp(-abs((X-.5)*(T-1)))+sin(X.*T); subplot(3,2,1) surf(X,T,z) axis([0,1,0,2,0.4,2.1]) xlabel('x'),ylabel('t'),zlabel('z'),title('Actual surface') [u,s,v] = svd(z); for k =1:3 zz=u(:,1:k)*s(1:k,1:k)*v(:,1:k)'; subplot(3,2,k+1) surf(X,T,zz),axis([0,1,0,2,0.4,2.1]) xlabel('x'),ylabel('t'),zlabel('z') title(['Rank',num2str(k),' approximation']) Thanks in advance Nils From nwagner at mecha.uni-stuttgart.de Mon Jan 28 06:48:14 2002 From: nwagner at mecha.uni-stuttgart.de (Nils Wagner) Date: Mon Jan 28 06:48:14 2002 Subject: [Numpy-discussion] Truncated Singular value decomposition Message-ID: <3C557347.5CCB9569@mecha.uni-stuttgart.de> Hi all, how can I obtain a truncated SVD (dyadic decomposition) of V,S,WT = singular_value_decomposition(Z) ? I am interested in a low rank approximation of Z. Thanks in advance. Nils From hinsen at cnrs-orleans.fr Mon Jan 28 08:22:03 2002 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Mon Jan 28 08:22:03 2002 Subject: [Numpy-discussion] Truncated Singular value decomposition In-Reply-To: <3C557347.5CCB9569@mecha.uni-stuttgart.de> References: <3C557347.5CCB9569@mecha.uni-stuttgart.de> Message-ID: Nils Wagner writes: > V,S,WT = singular_value_decomposition(Z) ? > > I am interested in a low rank approximation of Z. The singular values in S are sorted by decreasing order of magnitude, so you just replace as many "small" ones by zero as you wish, and multiply the three matrices back together. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From magnus at hetland.org Mon Jan 28 09:44:03 2002 From: magnus at hetland.org (Magnus Lie Hetland) Date: Mon Jan 28 09:44:03 2002 Subject: [Numpy-discussion] Odd numarray compile problem Message-ID: <20020128184327.H2276@idi.ntnu.no> I'm trying to compile numarray 0.11 with Python 2.2 on a linux box but somehow the compilation gets stuck on this line: gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -IInclude/numarray -I/home/mlh/python/Python-2.2/include/python2.2 -c Src/_ufuncmodule.c -o build/temp.linux-i686-2.2/_ufuncmodule.o No error message, no crash; it just never finishes -- it freezes. I compiled/installed it on a Solaris box earlier, with no problems whatsoever. I've never seen gcc freeze like this; can there be some circular dependencies or something? (This machine had some problems with its clock earlier, which made make go in a loop...) I tried running touch on all files, but that didn't change anything... Any ideas, or tips for finding out what's wrong? (BTW, Numeric compiles and installs just fine...) -- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org From michaell.taylor at reis.com Mon Jan 28 12:58:04 2002 From: michaell.taylor at reis.com (Michaell Taylor) Date: Mon Jan 28 12:58:04 2002 Subject: [Numpy-discussion] conditional array indexing Message-ID: Very new user to Numpy, so excuse the question. It relates to a fundamental issue which I have having trouble getting a grip on. Specifically, say I have the following: a = ([1,2,1,2,4,3,1,2,3,2,3,2]) b = ([4,3,5,2,4,5,3,6,3,2,5,6]) c = random variable of length=len(b) I would like to get the value of C associated with the lowest value of b within groupings of a. That is, there are three incidents of "1" in the a array. The lowest value of b associated with the value 1 in the a index is "3". This occurs in index value 6 on the a and b arrays. Now, I would like to then know the value of c[6]. Obviously the real problem is far more complex with len(a)==5000. Any ideas? Thanks in advance. Michaell From paul at pfdubois.com Mon Jan 28 13:21:02 2002 From: paul at pfdubois.com (Paul F. Dubois) Date: Mon Jan 28 13:21:02 2002 Subject: [Numpy-discussion] conditional array indexing In-Reply-To: Message-ID: <000101c1a841$2047c6c0$0c01a8c0@freedom> import Numeric, sys a = Numeric.array([1,2,1,2,4,3,1,2,3,2,3,2]) b = Numeric.array([4,3,5,2,4,5,3,6,3,2,5,6]) c = Numeric.arange(len(b)) print c[Numeric.argmin(Numeric.where(a==1, b, sys.maxint))] -----Original Message----- From: numpy-discussion-admin at lists.sourceforge.net [mailto:numpy-discussion-admin at lists.sourceforge.net] On Behalf Of Michaell Taylor Sent: Monday, January 28, 2002 12:59 PM To: numpy-discussion at lists.sourceforge.net Subject: [Numpy-discussion] conditional array indexing Very new user to Numpy, so excuse the question. It relates to a fundamental issue which I have having trouble getting a grip on. Specifically, say I have the following: a = ([1,2,1,2,4,3,1,2,3,2,3,2]) b = ([4,3,5,2,4,5,3,6,3,2,5,6]) c = random variable of length=len(b) I would like to get the value of C associated with the lowest value of b within groupings of a. That is, there are three incidents of "1" in the a array. The lowest value of b associated with the value 1 in the a index is "3". This occurs in index value 6 on the a and b arrays. Now, I would like to then know the value of c[6]. Obviously the real problem is far more complex with len(a)==5000. Any ideas? Thanks in advance. Michaell _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From Barrett at stsci.edu Tue Jan 29 05:27:06 2002 From: Barrett at stsci.edu (Paul Barrett) Date: Tue Jan 29 05:27:06 2002 Subject: [Numpy-discussion] Odd numarray compile problem References: <20020128184327.H2276@idi.ntnu.no> Message-ID: <3C56A2B6.1060904@STScI.Edu> Magnus Lie Hetland wrote: > I'm trying to compile numarray 0.11 with Python 2.2 on a linux box but > somehow the compilation gets stuck on this line: > > gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -IInclude/numarray > -I/home/mlh/python/Python-2.2/include/python2.2 -c Src/_ufuncmodule.c > -o build/temp.linux-i686-2.2/_ufuncmodule.o > > No error message, no crash; it just never finishes -- it freezes. I > compiled/installed it on a Solaris box earlier, with no problems > whatsoever. > > I've never seen gcc freeze like this; can there be some circular > dependencies or something? (This machine had some problems with its > clock earlier, which made make go in a loop...) I tried running touch > on all files, but that didn't change anything... > > Any ideas, or tips for finding out what's wrong? (BTW, Numeric > compiles and installs just fine...) Yes, just wait several minutes for it to finish compiling. I had to wait over 5 minutes on 233 MHz i586. -- Paul Barrett, PhD Space Telescope Science Institute Phone: 410-338-4475 ESS/Science Software Group FAX: 410-338-4767 Baltimore, MD 21218 From jjl at pobox.com Tue Jan 29 08:40:04 2002 From: jjl at pobox.com (John J. Lee) Date: Tue Jan 29 08:40:04 2002 Subject: [Numpy-discussion] amap: new function / ufunc method? In-Reply-To: Message-ID: On Sun, 27 Jan 2002, Travis Oliphant wrote: [...] > If I understand what you are trying to do, there is a function called > arraymap in the SciPy package (special module) (it was in the Numeric source [...] > I think that using this function and a combination of take and put can do > what you describe. I don't see how, though I may very well be missing something. To restate the problem very quickly (though the Python function I posted is much clearer -- it's only two or three lines of actual code): for every element in a list of array indices, the element in an output array at that index needs to be incremented; the tricky part is that this needs to happen once for *every time the index appears in the list*. Eg., in a 1D example, [0,1,1,1,4,5] might give you [1,3,0,0,1,1] (again, see my previous post for an actual tested example) I don't see how you can do this with arraymap, take and put. John From michaell.taylor at reis.com Tue Jan 29 08:46:02 2002 From: michaell.taylor at reis.com (Michaell Taylor) Date: Tue Jan 29 08:46:02 2002 Subject: [Numpy-discussion] conditional array indexing In-Reply-To: <000101c1a841$2047c6c0$0c01a8c0@freedom> References: <000101c1a841$2047c6c0$0c01a8c0@freedom> Message-ID: <20020129164505.5A969170201@newman.bestweb.net> Thanks for your quick reply. This looks vaguely S-Plus-like, with which I am familar. I wasn't too clear in my question however, I would actually like a vector returned which would contain the values of c for the minimun value of b, within each category of a. Something like: for i in range(1:5000): # range of values of a d = c[Numeric.argmin(Numeric.where(a==i, b, sys.maxint))] Seems like I could hack a fix using such a vector, but I would guess that speed would be an issue. Essentially the functionality implied by GROUP BY in sql egen in stata lapply() in S-Plus. Paul F. Dubois wrote: > import Numeric, sys > a = Numeric.array([1,2,1,2,4,3,1,2,3,2,3,2]) > b = Numeric.array([4,3,5,2,4,5,3,6,3,2,5,6]) > c = Numeric.arange(len(b)) > > print c[Numeric.argmin(Numeric.where(a==1, b, sys.maxint))] > > > -----Original Message----- > From: numpy-discussion-admin at lists.sourceforge.net > [mailto:numpy-discussion-admin at lists.sourceforge.net] On Behalf Of > Michaell Taylor > Sent: Monday, January 28, 2002 12:59 PM > To: numpy-discussion at lists.sourceforge.net > Subject: [Numpy-discussion] conditional array indexing > > > > Very new user to Numpy, so excuse the question. It relates to a > fundamental > issue which I have having trouble getting a grip on. > > Specifically, say I have the following: > > a = ([1,2,1,2,4,3,1,2,3,2,3,2]) > b = ([4,3,5,2,4,5,3,6,3,2,5,6]) > c = random variable of length=len(b) > > I would like to get the value of C associated with the lowest value of b > > within groupings of a. That is, there are three incidents of "1" in the > a > array. The lowest value of b associated with the value 1 in the a index > is > "3". This occurs in index value 6 on the a and b arrays. Now, I would > like > to then know the value of c[6]. > > Obviously the real problem is far more complex with len(a)==5000. > Any ideas? > > Thanks in advance. > > Michaell > > _______________________________________________ > Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- ========================================= Michaell Taylor, PhD Senior Economist, Reis, New York, USA From DavidA at ActiveState.com Tue Jan 29 10:49:01 2002 From: DavidA at ActiveState.com (David Ascher) Date: Tue Jan 29 10:49:01 2002 Subject: [Numpy-discussion] CFP: Python/Zope track at OSCON 2002 References: <000101c1a841$2047c6c0$0c01a8c0@freedom> <20020129164505.5A969170201@newman.bestweb.net> Message-ID: <3C56EE7A.97F621C0@activestate.com> The O'Reilly Open Source Convention (July 22-26, 2002 -- San Diego, CA) is accepting proposals for tutorials, talks, panels, and lightning talks. See the Call for Participation in the Python and Zope track on python.org. Proposals are due by March 1, so don't wait a moment longer! CFP URL: http://www.python.org/workshops/oscon2002/cfp.html --david ascher From magnus at hetland.org Tue Jan 29 12:34:04 2002 From: magnus at hetland.org (Magnus Lie Hetland) Date: Tue Jan 29 12:34:04 2002 Subject: [Numpy-discussion] What are the chances of getting numarray properties? Message-ID: <20020129213340.F22910@idi.ntnu.no> I see in the docs that numarray uses accessor methods instead of properties, mainly because of performance (to avoid __getattr__ and __setattr__). With the new property type in Python 2.2, that shouldn't be as important anymore... Perhaps it would be possible to use properties where this type is available, and accessors elsewhere? -- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org From magnus at hetland.org Tue Jan 29 12:40:04 2002 From: magnus at hetland.org (Magnus Lie Hetland) Date: Tue Jan 29 12:40:04 2002 Subject: [Numpy-discussion] Accessors Message-ID: <20020129213922.G22910@idi.ntnu.no> According to the (probably outdated) user guide for numarray[1] attributes have accessors such as a.foo() --> get foo a.foo(val) --> set foo to val but this doesn't seem to be the case. I see that the shape() method doesn't take any arguments; I guess the reshape method is the thing to use here. How about other attributes? Will they have special names for their setters too, or is this only true for the shape attribute? It seems to me that if we are going to abandon plain Python attributes (or 2.2 properties), it would be nice to handle the transition in a consistent manner... Perhaps? (But then again, I'm sure I'm missing something :) [1] http://stsdas.stsci.edu/numarray/Userguide.html -- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org From jmiller at stsci.edu Tue Jan 29 13:01:01 2002 From: jmiller at stsci.edu (Jay T Miller) Date: Tue Jan 29 13:01:01 2002 Subject: [Numpy-discussion] Re: What are the chances of getting numarray properties? References: Message-ID: <3C570D31.4090308@stsci.edu> Numarray should soon support the following properties: .shape .flat .real .imag .imaginary We're testing them here at STSCI. If all goes well, there should be a new numarray release within 1-3 weeks. Todd -- Todd Miller jmiller at stsci.edu STSCI / SSG (410) 338 4576 From magnus at hetland.org Tue Jan 29 13:07:11 2002 From: magnus at hetland.org (Magnus Lie Hetland) Date: Tue Jan 29 13:07:11 2002 Subject: [Numpy-discussion] Re: What are the chances of getting numarray properties? In-Reply-To: <3C570D31.4090308@stsci.edu>; from jmiller@stsci.edu on Tue, Jan 29, 2002 at 03:59:29PM -0500 References: <3C570D31.4090308@stsci.edu> Message-ID: <20020129220647.I22910@idi.ntnu.no> Jay T Miller : > > Numarray should soon support the following properties: > > .shape > .flat > .real > .imag > .imaginary > > We're testing them here at STSCI. If all goes well, there should be > a new numarray release within 1-3 weeks. Yay! :)) > Todd -- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org From magnus at hetland.org Tue Jan 29 14:38:03 2002 From: magnus at hetland.org (Magnus Lie Hetland) Date: Tue Jan 29 14:38:03 2002 Subject: [Numpy-discussion] RandomArray and numarray Message-ID: <20020129233707.A28608@idi.ntnu.no> Is there some support for random arrays in numarray? Or, if it becomes part of the standard library, will there be a separate module (such as Numeric's RandomArray)? -- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org From magnus at hetland.org Tue Jan 29 14:52:06 2002 From: magnus at hetland.org (Magnus Lie Hetland) Date: Tue Jan 29 14:52:06 2002 Subject: [Numpy-discussion] numarray bug... Message-ID: <20020129235152.C28608@idi.ntnu.no> My numarray dumps core... >>> from numarray import * >>> a = zeros([10, 10]) >>> a = zeros([10, 10]) + 0.5 Bus Error (core dumped) Numeric tackles this just fine... What gives? -- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org From perry at stsci.edu Tue Jan 29 14:54:04 2002 From: perry at stsci.edu (Perry Greenfield) Date: Tue Jan 29 14:54:04 2002 Subject: [Numpy-discussion] RandomArray and numarray In-Reply-To: <20020129233707.A28608@idi.ntnu.no> Message-ID: > > Is there some support for random arrays in numarray? Or, if it becomes > part of the standard library, will there be a separate module (such as > Numeric's RandomArray)? > > -- > Magnus Lie Hetland The Anygui Project > http://hetland.org http://anygui.org > Not at the moment, but I imagine we will add a separate module as it is done now for Numeric. While I'm replying, our short range development plans are (now that we are back working on it): 1) adding properties for .shape, .flat, etc. This was done last week but we are testing this and will put it out as soon as possible (probably after the Python Conference). By the way, to make it usable from Python <2.2, we will add accessor functions with different names than we used previously (e.g., .getshape(), setshape(), .getflat()...) 2) reworking numarray to make it safe against array overruns and anything that might cause segfaults or buserrors (in principle should only happen now if you twiddle directly with private attributes). 3) provide examples of how to add new ufuncs and interface C code and such. We may decide to add a module or two in the process. And perhaps some limited benchmarking. 4) rework the implementation of complex types to be in C. 5) do more extensive benchmarking and look at how to optimize smaller arrays. Perry Greenfield From kragen at pobox.com Tue Jan 29 23:39:03 2002 From: kragen at pobox.com (Kragen Sitaker) Date: Tue Jan 29 23:39:03 2002 Subject: [Numpy-discussion] Re: memory-mapped Numeric arrays: arrayfrombuffer version 2 Message-ID: <20020130073834.AFB65BDC8@panacea.canonical.org> Paul Dubois writes: > I have verified that this package seems to work on Windows. I says seems > only because I didn't try enough to uncover anything subtle. Thanks! If you run maparray.py from the command-line, it runs a basic regression test suite, which doesn't really try enough to uncover anything subtle either. > Unless or until we are convinced as a community that this is (a) the > right way to do this and (b) that the package is portable, it would not > be wise to put it in the main distribution. Well, I'm not the community, but I'll state my arguments. On (a): I don't know about the right way to do it; as I'm sure is obvious, I'm new to extending Numerical Python in C, but I doubt there's a simpler way to do it ("it" being seeing the contents of files as Numeric arrays), and I think it does work as well as it's possible to get it to work without major hacking of Numeric (to support read-only arrays) or the mmap module (to prevent closing an open object, to allow read-only mmapping on Windows). The other things on the wishlist are basically features to add --- "start" and "size" arguments to maparray(), a "create-a-file" argument to maparray(), etc., and don't change the basic structure. On (b): it depends on two-argument mmap.mmap(), open(), .fileno(), and os.fstat(). The major portability hurdle there is probably mmap.mmap(), but that's OK. > I would like to hear from the community about this so that I will know > whether or not to add this package as a separate SourceForge 'package' > within the Numerical Python area. Meantime I will add a link to the web > page. I would too. There have been downloads from something like 50 IP addresses, but I've only heard from three people. From yoonseo at altavista.net Thu Jan 31 14:33:02 2002 From: yoonseo at altavista.net (SungPil Yoon) Date: Thu Jan 31 14:33:02 2002 Subject: [Numpy-discussion] numpy installation on HP-UX Message-ID: <02013116191902.23816@dhcp-243> Hi, all I have been trying to install numpy. Python is installed in my home directory ~/Python-2.2 and when I do % python setup.py install in the directory ~/Numeric-20.3, I get the following error message distutils.errors.DistutilsPlatformError: invalid Python installation: unable to open /usr/local/lib/python2.2/config/Makefile (No such file or directory) Can anybody figure out what I am doing wrong? sungpil