From thomas.heller@ion-tof.com Fri Sep 1 03:35:01 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri Sep 1 02:35:01 2000 Subject: [Distutils] some small bugs References: <399BD06F.13FD9EAA@gmx.de> <20000821215505.A1131@beelzebub> <39ACDE11.B7F8AEE2@gmx.de> <20000831183138.A31473@python.net> Message-ID: <001101c013de$af7f8010$4500a8c0@thomasnb> > > The patch fixes this. It also changes two other problems. > > First it raises an PlatformException if you try to use it with > > modules which contain compiled C-extensions. Distutils doesn't > > support crosscompiling. > > Ooh, but wouldn't it be cool if I could compile Windows extensions on Linux > and vice-versa... oh, never mind! > I remember having heard that it is possible to cross-compile the windows version of tcl on linux... Maybe someone with more experience than me on could find out? Thomas From akuchlin@mems-exchange.org Fri Sep 1 12:34:01 2000 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Fri Sep 1 11:34:01 2000 Subject: [Distutils] Python interpreter path and bdist_rpm Message-ID: The first Python interpreter on my path is in /www/python/bin. I was building some RPMs today where I wanted to use the interpreter that comes with Red Hat, so I ran '/usr/bin/python1.5 setup.py bdist_rpm', and found that the RPM build script just uses 'python', so it found the first one on my path, which isn't the same as the one I wanted. Question: should the RPM build script hard-core the full path of the interpreter binary, or does it cause problems? If specifying the full path is OK, here's a patch. (Harry, or someone else who knows RPM, should approve of this patch before considering checking it in.) --amk Index: bdist_rpm.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/distutils/command/bdist_rpm.py,v retrieving revision 1.17 diff -C2 -r1.17 bdist_rpm.py *** bdist_rpm.py 2000/08/15 13:05:35 1.17 --- bdist_rpm.py 2000/09/01 15:28:35 *************** *** 8,12 **** __revision__ = "$Id: bdist_rpm.py,v 1.17 2000/08/15 13:05:35 gward Exp $" ! import os, string import glob from types import * --- 8,12 ---- __revision__ = "$Id: bdist_rpm.py,v 1.17 2000/08/15 13:05:35 gward Exp $" ! import os, string, sys import glob from types import * *************** *** 400,406 **** # figure out default build script if self.use_rpm_opt_flags: ! def_build = 'env CFLAGS="$RPM_OPT_FLAGS" python setup.py build' else: ! def_build = 'python setup.py build' # insert contents of files --- 400,407 ---- # figure out default build script if self.use_rpm_opt_flags: ! def_build = ('env CFLAGS="$RPM_OPT_FLAGS" %s setup.py build' ! % sys.executable ) else: ! def_build = '%s setup.py build' % sys.executable # insert contents of files From hgebel@inet.net Fri Sep 1 14:51:07 2000 From: hgebel@inet.net (Harry Henry Gebel) Date: Fri Sep 1 13:51:07 2000 Subject: [Distutils] Python interpreter path and bdist_rpm In-Reply-To: ; from akuchlin@mems-exchange.org on Fri, Sep 01, 2000 at 11:33:02AM -0400 References: Message-ID: <20000901135047.A4018@inet.net> On Fri, Sep 01, 2000 at 11:33:02AM -0400, Andrew Kuchling wrote: > Question: should the RPM build script hard-core the full path of the > interpreter binary, or does it cause problems? If specifying the full > path is OK, here's a patch. (Harry, or someone else who knows RPM, > should approve of this patch before considering checking it in.) The problem with hard coding the path into the spec file is that it prevents the source RPM from being built on a computer that does not have python located in that place. I think a better solution would be something like: env PATH=/usr/bin/:$PATH python setup.py bdist_rpm Of course this only works if /usr/bin/python -> /usr/bin/python1.5 , if /usr/bin/python -> /www/python/bin/python it won't work. If that is the case, then maybe hard coding the path should be implemented as an option, to be invoked with the understanding that it could hamper source RPM portability. In practice I think most RPM-based distributions put python in /usr/bin (I use RedHat, Mandrake, and SuSE on a fairly regular basis and they all put it there), so it might only be breaking portability on a theoretical basis (at least until python > 1.5 becomes common in distributions.) -- Harry Henry Gebel, Senior Developer, Landon House SBS ICQ# 76308382 West Dover Hundred, Delaware From hgebel@inet.net Fri Sep 1 14:55:01 2000 From: hgebel@inet.net (Harry Henry Gebel) Date: Fri Sep 1 13:55:01 2000 Subject: [Distutils] bdist_debian Message-ID: <20000901135430.B4018@inet.net> I am switching to Debian on my desktop system, and I was wondering if anybody out there is working on bdist_debian? If not I would like to give it a try. I will still be running Mandrake on the other computers on my network, so I will still be able to work on bdist_rpm as well. -- Harry Henry Gebel, Senior Developer, Landon House SBS ICQ# 76308382 West Dover Hundred, Delaware From mal@lemburg.com Sat Sep 2 06:51:01 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat Sep 2 05:51:01 2000 Subject: [Distutils] First steps with distutils... References: <39AFC048.6E963F82@lemburg.com> Message-ID: <39B0CD44.6B79F246@lemburg.com> Here is a list of things I've found while playing with the 0.9.2 snapshot. I've also posted this to Greg directly, but on second thought I think it might be worthwhile getting some feedback from distutilers as well: * distutils doesn't seem to like diretory names in MANIFEST * the setup.py file itself is not included in the source distribution per default (why not ?) * bdist_wininst wants zlib but the installer has a note that it is not using it anymore * bdist_wininst fails with a setup.py file which does not define long_description * bdist_rpm fails with an error in the rpm command (the source rpm is built, but it fails to build to the binary rpm for some reason -- it says, that it can't find the file ??); do I need some new rpm release for this to work ? * the bdist commands don't compile the .py files into .pyo but this is needed in order to use them with python -O * bdist (the dumb version) defaults to creating a tar.gz archive which is relative to / -- it shouldn't use the usr/local/ prefix to make the installation relocateable (e.g. to install under /usr, /opt, /opt/local, etc.) * how can I define the compiler settings for extension compilation (it defaults to -m486 for RPMs -- -mcpu=pentium would be more appropriate; default for standard bdist seems to be -O2 and no architecture) ? And a general question: how can I define which files are documentation, examples and auxiliary files ? More to come :-) -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gward@python.net Sat Sep 2 11:18:01 2000 From: gward@python.net (Greg Ward) Date: Sat Sep 2 10:18:01 2000 Subject: [Distutils] About version discussion In-Reply-To: <20000831155213.A1233@inet.net>; from hgebel@inet.net on Thu, Aug 31, 2000 at 03:52:13PM -0400 References: <20000831155213.A1233@inet.net> Message-ID: <20000902101554.A1221@beelzebub> On 31 August 2000, Harry Henry Gebel said: > I was wondering if it might be a good idea to put a function in Distutils > that could query if the currently installed version has certain > abilities. Each ability could be assigned a name, and it could be > implemented very simply: I'm skeptical. This could be useful for situations where you need version X because feature foo was added, but what about bug fixes? The ability to not crash when compiling extensions described the "old way" (pre-Extension object) on Windows is certainly important; isn't it just easier to say, "I need Distutils 0.9.2"? Greg -- Greg Ward - Unix bigot gward@python.net http://starship.python.net/~gward/ I am deeply CONCERNED and I want something GOOD for BREAKFAST! From gward@python.net Sat Sep 2 11:52:02 2000 From: gward@python.net (Greg Ward) Date: Sat Sep 2 10:52:02 2000 Subject: [Distutils] First steps with distutils... In-Reply-To: <39B0CD44.6B79F246@lemburg.com>; from mal@lemburg.com on Sat, Sep 02, 2000 at 11:49:56AM +0200 References: <39AFC048.6E963F82@lemburg.com> <39B0CD44.6B79F246@lemburg.com> Message-ID: <20000902104944.B1221@beelzebub> On 02 September 2000, M.-A. Lemburg said: > I've also posted this to Greg directly, but on second > thought I think it might be worthwhile getting some feedback from > distutilers as well: Then I'll reply publicly too; seems more useful that way. > * distutils doesn't seem to like diretory names in MANIFEST Correct. MANIFEST is a simple list of files, one per line. If you want to include whole directories, you should write a MANIFEST.in file. Eg. here is the Distutils' MANIFEST.in; I have (right now) 536 files in my Distutils working tree, and this MANIFEST.in whittles it down to the 89 files you see in the 0.9.2 source tarball: include *.txt TODO include MANIFEST.in recursive-include examples *.txt *.py prune examples/sample*/build include doc/dist/*.tex doc/inst/*.tex graft misc exclude misc/*.zip global-exclude *~ The docs for MANIFEST.in are sketchy right now; see http://www.python.org/sigs/distutils-sig/doc/dist/sdist-cmd.html for a start. > * the setup.py file itself is not included in the source > distribution per default (why not ?) It should be if you have a MANIFEST.in file. One current weakness is that the "sdist" command isn't very smart in the absence of a MANIFEST.in file. I always seem to notice this in the final hours up to a release, which is not the time or place for major surgery to fix this. Sigh. > * bdist_wininst wants zlib but the installer has a note that > it is not using it anymore Thomas, is this true? > * bdist_wininst fails with a setup.py file which does not > define long_description I thought that was fixed in 0.9.2 -- !@&%@!#&!@#&%! > * bdist_rpm fails with an error in the rpm command (the source > rpm is built, but it fails to build to the binary rpm for > some reason -- it says, that it can't find the file ??); > do I need some new rpm release for this to work ? bdist_rpm requires RPM 3.0, which all current RPM-based Linux distributions use (AFAIK). It does *not* work on Red Hat 5.x, which is what finally prompted me to upgrade to 6.2. ;-) Can you show the exact output when you run "setup.py bdist_rpm"? > * the bdist commands don't compile the .py files into .pyo > but this is needed in order to use them with python -O Yeah, known problem. How big do you want your RPM to be again? ;-( > * bdist (the dumb version) defaults to creating a tar.gz > archive which is relative to / -- it shouldn't use the > usr/local/ prefix to make the installation relocateable > (e.g. to install under /usr, /opt, /opt/local, etc.) There's no one right way to do this; I've tried them both, and they're both wrong. Originally, bdist_dumb created an archive relative to -- relocatable, but it didn't work if prefix != exec-prefix. (I can't remember if it bombed or created a bogus archive, but the bottom line is it didn't work.) Now it creates an archive relative to /, which handles prefix != exec-prefix, but is not relocatable. ;-( I think the answer is to let the packager decide, ie. add an option to make the archive relative to / or prefix. (And it should blow up if prefix != exec-prefix and you try to make a prefix-relative archive.) > * how can I define the compiler settings for extension > compilation (it defaults to -m486 for RPMs -- -mcpu=pentium > would be more appropriate; default for standard bdist > seems to be -O2 and no architecture) ? For a straight Distutil build (ie. no RPM involved), setting CFLAGS should work, but it only applies to extensions (not C libraries). When RPM is involved, I'm not sure what the Right Way to do it is. Maybe CFLAGS will work? Maybe editing /etc/rpmrc or /usr/lib/rpm/rpmrc? I dunno. > And a general question: how can I define which files are > documentation, examples and auxiliary files ? Doc files can be defined with the doc_files option to bdist_rpm. The easiest way to set this is with a setup.cfg in your distribution root, eg. the Distutils setup.cfg has: [bdist_rpm] release = 1 packager = Greg Ward doc_files = CHANGES.txt README.txt USAGE.txt doc/ examples/ ... Although I have been thinking that putting making every RPM option a Distutils option was a bad idea. Hmmmm. > More to come :-) Oh no! You might want to take a look in the Distutils-0.9.2/TODO file to see what's on "the list". Greg -- Greg Ward - nerd gward@python.net http://starship.python.net/~gward/ Does your DRESSING ROOM have enough ASPARAGUS? From gward@python.net Sat Sep 2 12:09:01 2000 From: gward@python.net (Greg Ward) Date: Sat Sep 2 11:09:01 2000 Subject: [Distutils] Python interpreter path and bdist_rpm In-Reply-To: <20000901135047.A4018@inet.net>; from hgebel@inet.net on Fri, Sep 01, 2000 at 01:50:48PM -0400 References: <20000901135047.A4018@inet.net> Message-ID: <20000902110727.C1221@beelzebub> On 01 September 2000, Harry Henry Gebel said: > The problem with hard coding the path into the spec file is that it > prevents the source RPM from being built on a computer that does not have > python located in that place. I think a better solution would be something > like: That *might* be a feature: if I build an RPM with /usr/bin/python, and encode that path in the spec file, then I expect any target systems to have a similar Python installation, ie. /usr/bin/python had better be there. If not, we have problems. Of course, this argument breaks down in the face of relocatable RPMs. I would suggest two wrinkles on Andrew's patch: * use sys.executable or "python"; sys.executable is not 100% reliable in my experience (although I've only seen it empty in one particular FastCGI script... AMK will know what I'm talking about...) * have an option to allow "whatever python is first on the path", but make sys.executable the default IOW, something like this: if not self.fix_interpreter_path: python = "python" else: python = sys.executable or "python" ... and then build the command that goes in the .spec file. Greg -- Greg Ward - geek-at-large gward@python.net http://starship.python.net/~gward/ I just heard the SEVENTIES were over!! And I was just getting in touch with my LEISURE SUIT!! From gward@python.net Sat Sep 2 12:15:01 2000 From: gward@python.net (Greg Ward) Date: Sat Sep 2 11:15:01 2000 Subject: [Distutils] bdist_debian In-Reply-To: <20000901135430.B4018@inet.net>; from hgebel@inet.net on Fri, Sep 01, 2000 at 01:54:30PM -0400 References: <20000901135430.B4018@inet.net> Message-ID: <20000902111323.D1221@beelzebub> On 01 September 2000, Harry Henry Gebel said: > I am switching to Debian on my desktop system, and I was wondering if > anybody out there is working on bdist_debian? If not I would like to give > it a try. I will still be running Mandrake on the other computers on my > network, so I will still be able to work on bdist_rpm as well. I still have some messages in my inbox from a couple of the Python/Debian maintainers. Note there is a mailing list debian-python@lists.debian.org which would probably be a good source of wisdom; I thoroughly approve of (and encourage) cross-posting between debian-python and distutils-sig. What the hell, I'll go ahead and cross-post this one, so we can find out if any of the Debian Python maintainers have done anything more than make Debian packages of the Distutils. If nothing else, we might end up making their lives a lot easier. strongly-approve-of-debian-in-principle-but-was-thoroughly-mystified- by-dselect-the-one-time-i-tried-it... Greg -- Greg Ward - Unix nerd gward@python.net http://starship.python.net/~gward/ There are no stupid questions -- only stupid people. From hgebel@inet.net Sat Sep 2 12:46:11 2000 From: hgebel@inet.net (Harry Henry Gebel) Date: Sat Sep 2 11:46:11 2000 Subject: [Distutils] First steps with distutils... In-Reply-To: <39B0CD44.6B79F246@lemburg.com>; from mal@lemburg.com on Sat, Sep 02, 2000 at 11:49:56AM +0200 References: <39AFC048.6E963F82@lemburg.com> <39B0CD44.6B79F246@lemburg.com> Message-ID: <20000902114558.A6277@inet.net> On Sat, Sep 02, 2000 at 11:49:56AM +0200, M.-A. Lemburg wrote: > * bdist_rpm fails with an error in the rpm command (the source > rpm is built, but it fails to build to the binary rpm for > some reason -- it says, that it can't find the file ??); > do I need some new rpm release for this to work ? bdist_rpm requires RPM version 3, there was some discussion about supporting RPM 2; but it did not seem practical to me to support RPM 2 in a way that would work across different RPM based platforms. It probably would not be hard to make it produce version 2 spec files, but the packager would then have to build them outside of the Distutils. > * the bdist commands don't compile the .py files into .pyo > but this is needed in order to use them with python -O There were some attempts made to do this, but none has been satisfactory. > * how can I define the compiler settings for extension > compilation (it defaults to -m486 for RPMs -- -mcpu=pentium > would be more appropriate; default for standard bdist > seems to be -O2 and no architecture) ? The compiler options for RPMs are set by RPM (using CFLAGS) not by Distutils, you can change these by modifying your RPM configuration (the exact modifications depend on which version of RPM you are using.) Otherwise you can use CFLAGS to set them, I think they can be set in setup.py, but I'm not sure about that. -- Harry Henry Gebel, Senior Developer, Landon House SBS ICQ# 76308382 West Dover Hundred, Delaware From hgebel@inet.net Sat Sep 2 12:51:09 2000 From: hgebel@inet.net (Harry Henry Gebel) Date: Sat Sep 2 11:51:09 2000 Subject: [Distutils] bdist_debian In-Reply-To: <20000902111323.D1221@beelzebub>; from gward@python.net on Sat, Sep 02, 2000 at 11:13:23AM -0400 References: <20000901135430.B4018@inet.net> <20000902111323.D1221@beelzebub> Message-ID: <20000902115036.B6277@inet.net> On Sat, Sep 02, 2000 at 11:13:23AM -0400, Greg Ward wrote: > strongly-approve-of-debian-in-principle-but-was-thoroughly-mystified- > by-dselect-the-one-time-i-tried-it... Same here, but the packaging system is supposed to be alot more usable in 2.2; so I figured I would give it a try. -- Harry Henry Gebel, Senior Developer, Landon House SBS ICQ# 76308382 West Dover Hundred, Delaware From hgebel@inet.net Sat Sep 2 13:07:01 2000 From: hgebel@inet.net (Harry Henry Gebel) Date: Sat Sep 2 12:07:01 2000 Subject: [Distutils] Python interpreter path and bdist_rpm In-Reply-To: <20000902110727.C1221@beelzebub>; from gward@python.net on Sat, Sep 02, 2000 at 11:07:27AM -0400 References: <20000901135047.A4018@inet.net> <20000902110727.C1221@beelzebub> Message-ID: <20000902120644.C6277@inet.net> On Sat, Sep 02, 2000 at 11:07:27AM -0400, Greg Ward wrote: > On 01 September 2000, Harry Henry Gebel said: > > The problem with hard coding the path into the spec file is that it > > prevents the source RPM from being built on a computer that does not have > > python located in that place. I think a better solution would be something > > like: > That *might* be a feature: if I build an RPM with /usr/bin/python, and > encode that path in the spec file, then I expect any target systems to > have a similar Python installation, ie. /usr/bin/python had better be > there. If not, we have problems. This is fine for binary RPMs, but source RPMs can (and should, I believe) to be buildable on any platform using the same or a later version of RPM. In fact a great number of the packages on my system are built from source RPMs designed for other distributions; many of these I had to alter to build on Mandrake; which always annoyed me since the alterations to make them not depend on a particular distribution are usually pretty minor. The source RPMs produced by bdist_rpm now should build without modifications on any system using RPM 3 or later, no matter where python is located. > * have an option to allow "whatever python is first on the path", > but make sys.executable the default I think this is a good option, but I think source RPM compatibility should be the default, and sys.executable should be the option. -- Harry Henry Gebel, Senior Developer, Landon House SBS ICQ# 76308382 West Dover Hundred, Delaware From hgebel@inet.net Sat Sep 2 13:14:02 2000 From: hgebel@inet.net (Harry Henry Gebel) Date: Sat Sep 2 12:14:02 2000 Subject: [Distutils] About version discussion In-Reply-To: <20000902101554.A1221@beelzebub>; from gward@python.net on Sat, Sep 02, 2000 at 10:15:54AM -0400 References: <20000831155213.A1233@inet.net> <20000902101554.A1221@beelzebub> Message-ID: <20000902121319.D6277@inet.net> On Sat, Sep 02, 2000 at 10:15:54AM -0400, Greg Ward wrote: > On 31 August 2000, Harry Henry Gebel said: > > I was wondering if it might be a good idea to put a function in Distutils > > that could query if the currently installed version has certain > > abilities. Each ability could be assigned a name, and it could be > > implemented very simply: > I'm skeptical. This could be useful for situations where you need > version X because feature foo was added, but what about bug fixes? The > ability to not crash when compiling extensions described the "old way" > (pre-Extension object) on Windows is certainly important; isn't it just > easier to say, "I need Distutils 0.9.2"? My concern was for some future time when everybody who makes python packages uses Distutils, but most of them will probably not have been members of the Distutils SIG long enough to know when different features were implemented. I also have doubts it would be useful (or used) enough to justify putting it in; but I figured it wouldn't hurt to get people's opinions. -- Harry Henry Gebel, Senior Developer, Landon House SBS ICQ# 76308382 West Dover Hundred, Delaware From gward@python.net Sat Sep 2 23:37:07 2000 From: gward@python.net (Greg Ward) Date: Sat Sep 2 22:37:07 2000 Subject: [Distutils] Python interpreter path and bdist_rpm In-Reply-To: <20000902120644.C6277@inet.net>; from hgebel@inet.net on Sat, Sep 02, 2000 at 12:06:45PM -0400 References: <20000901135047.A4018@inet.net> <20000902110727.C1221@beelzebub> <20000902120644.C6277@inet.net> Message-ID: <20000902223542.A2097@beelzebub> On 02 September 2000, Harry Henry Gebel said: > > * have an option to allow "whatever python is first on the path", > > but make sys.executable the default > > I think this is a good option, but I think source RPM compatibility should > be the default, and sys.executable should be the option. OK, we'll flip a coin. *flip* There, I won. Wasn't that easy? >grin< But seriously, does anyone have an opinion here? As I see it, the issue is this: * I create an RPM pair (source & binary) using /usr/bin/python * but the build instructions in the .spec file just say "python setup.py build", not "/usr/bin/python setup.py build" * you build from my source RPM on a machine where the first python in the PATH != /usr/bin/python -- eg. in Andrew's example, /www/python/bin/python, because that's how we setup our development machines at work * my modules get installed somewhere unexpected on your machine You could flip things around and ponder what happens when I build an RPM using /foo/bar/bin/python, but I won't do that because I'm afraid it would negate my argument and obliterate my position. (Hmmm.) The question is, under what circumstances should the .spec file refer to "/usr/bin/python" (ie. sys.executable at the time the .spec file and the RPMs are generated). My hunch is we should make this the default; Harry says this behaviour should be available, but not the default. Inertia and backwards compatibility support Harry's position. Anyone feel strongly one way or the other? Greg -- Greg Ward - Linux geek gward@python.net http://starship.python.net/~gward/ Support bacteria -- it's the only culture some people have! From gward@python.net Sat Sep 2 23:41:01 2000 From: gward@python.net (Greg Ward) Date: Sat Sep 2 22:41:01 2000 Subject: [Distutils] About version discussion In-Reply-To: <20000902121319.D6277@inet.net>; from hgebel@inet.net on Sat, Sep 02, 2000 at 12:13:19PM -0400 References: <20000831155213.A1233@inet.net> <20000902101554.A1221@beelzebub> <20000902121319.D6277@inet.net> Message-ID: <20000902223905.B2097@beelzebub> On 02 September 2000, Harry Henry Gebel said: > My concern was for some future time when everybody who makes python > packages uses Distutils, but most of them will probably not have been > members of the Distutils SIG long enough to know when different features > were implemented. I also have doubts it would be useful (or used) enough to > justify putting it in; but I figured it wouldn't hurt to get people's > opinions. But you have the same problem for any piece of software, and I'm not familiar with any other tool exposing feature lists like this instead of version numbers. Your point is valid -- it's hard to know which features were added when, and which bugs were fixed when -- but I suspect compiling and maintaining a list of what's present in each version of a given tool would be a thankless and herculean task. (But I bet if you suggest it for Perl, it'll get implemented someday...) Greg -- Greg Ward - Linux geek gward@python.net http://starship.python.net/~gward/ Excuse me, but didn't I tell you there's NO HOPE for the survival of OFFSET PRINTING? From calvin@cs.uni-sb.de Sun Sep 3 17:42:00 2000 From: calvin@cs.uni-sb.de (Bastian Kleineidam) Date: Sun Sep 3 16:42:00 2000 Subject: [Distutils] bdist_debian In-Reply-To: <20000902111323.D1221@beelzebub> Message-ID: Debian Packages with Distutils: 1) You are on a Debian system a) execute "dh_make --native" or see dh_make(1) b) adjust your debian/rules file: debian/rules: #... build: #... rm -rf debian/tmp # Distutils chokes on broken symlinks python setup.py build #... clean: #... python setup.py clean --all #... install: build #... python setup.py install --root=`pwd`/debian/tmp #... 2) But I am running a non-Debian system a) Read something about the .deb format in deb(5): The file is an ar archive with a magic number of !. The first member is named debian-binary and contains a series of lines, separated by newlines. The second required member is named control.tar.gz. It is a gzipped tar archive containing the package control information. The third, last required member is named data.tar.gz . It contains the filesystem archive as a gzipped tar archive. b) Be sure that you follow all of the Debian standards and requirements so that the package fits smoothly in any Debian installation. Look at the documents in the debian-policy package. There are scripts to do this automatically: see debhelper(1). All this can be checked with the lintian Debian package. Number 2) has two disadvantages: 1. All this code to produce Debian packages is existing, tested and working code. bdist_debain would only duplicate this effort. Espacially for 2b) there are numerous existing helper scripts in a Debian system. 2. Okay, lets assume implementing this whole number 2) thingy is not very hard (or you have enough spare time to do it), but then you remember deep in your mind the Debian FAQ 6.2: This internal format is subject to change (between major releases of Debian GNU/Linux), therefore please always use dpkg-deb(8) for manipulating .deb files. So you will run into problems sooner or later and you have to update your software as the internal format changes. You can look at the discussions about bdist_rpm and the RPM versions, thats pretty much the same. Bastian Kleineidam From akuchlin@mems-exchange.org Sun Sep 3 23:09:02 2000 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Sun Sep 3 22:09:02 2000 Subject: [Distutils] Python interpreter path and bdist_rpm In-Reply-To: <20000902223542.A2097@beelzebub>; from gward@python.net on Sat, Sep 02, 2000 at 10:35:42PM -0400 References: <20000901135047.A4018@inet.net> <20000902110727.C1221@beelzebub> <20000902120644.C6277@inet.net> <20000902223542.A2097@beelzebub> Message-ID: <20000903220840.A3432@newcnri.cnri.reston.va.us> On Sat, Sep 02, 2000 at 10:35:42PM -0400, Greg Ward wrote: >says this behaviour should be available, but not the default. Inertia >and backwards compatibility support Harry's position. Anyone feel >strongly one way or the other? > In truth, it's not hard to work around this problem for building RPMs on a system with a non-standard extra Python installation; I simply reset my $PATH. So, while I brought the issue up in the first place, I agree that it's probably not worth fixing. --amk From mal@lemburg.com Mon Sep 4 08:13:03 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon Sep 4 07:13:03 2000 Subject: [Distutils] First steps with distutils... References: <39AFC048.6E963F82@lemburg.com> <39B0CD44.6B79F246@lemburg.com> <20000902104944.B1221@beelzebub> Message-ID: <39B38398.1B4079D3@lemburg.com> Greg Ward wrote: > > On 02 September 2000, M.-A. Lemburg said: > > > * distutils doesn't seem to like diretory names in MANIFEST > > Correct. MANIFEST is a simple list of files, one per line. If you want > to include whole directories, you should write a MANIFEST.in file. I didn't mean that distutils should scan the goven directory -- it should simply ignore it instead of raising an exception. The reason is that generation tools for MANIFEST could sometimes include directory entries in the list (e.g. take a ZIP file directory listing or ls -1p). > Eg. here is the Distutils' MANIFEST.in; I have (right now) 536 files in > my Distutils working tree, and this MANIFEST.in whittles it down to the > 89 files you see in the 0.9.2 source tarball: > > include *.txt TODO > include MANIFEST.in > recursive-include examples *.txt *.py > prune examples/sample*/build > include doc/dist/*.tex doc/inst/*.tex > graft misc > exclude misc/*.zip > global-exclude *~ > > The docs for MANIFEST.in are sketchy right now; see > > http://www.python.org/sigs/distutils-sig/doc/dist/sdist-cmd.html > > for a start. I'm currently using my own tools for generating the MANIFEST file. The most important difference is that they allow per directory MANIFEST.in style files which are appended to the general MANIFEST.in logic while scanning the directory. > > * the setup.py file itself is not included in the source > > distribution per default (why not ?) > > It should be if you have a MANIFEST.in file. Ahh, ok. So that's why it doesn't get included: I don't have a MANIFEST.in file. > One current weakness is that the "sdist" command isn't very smart in the > absence of a MANIFEST.in file. I always seem to notice this in the > final hours up to a release, which is not the time or place for major > surgery to fix this. Sigh. > > > * bdist_wininst wants zlib but the installer has a note that > > it is not using it anymore > > Thomas, is this true? I would make the usage of zlib optional with distutils defaulting to not using it (since it isn't enabled by default in the Python distribution). > > * bdist_wininst fails with a setup.py file which does not > > define long_description > > I thought that was fixed in 0.9.2 -- !@&%@!#&!@#&%! I used the latest snapshot, not the final 0.9.2 release version. I'll check this again. > > * bdist_rpm fails with an error in the rpm command (the source > > rpm is built, but it fails to build to the binary rpm for > > some reason -- it says, that it can't find the file ??); > > do I need some new rpm release for this to work ? > > bdist_rpm requires RPM 3.0, which all current RPM-based Linux > distributions use (AFAIK). It does *not* work on Red Hat 5.x, which is > what finally prompted me to upgrade to 6.2. ;-) My version if rpm 3.0.3. > Can you show the exact output when you run "setup.py bdist_rpm"? I'll attach a copy below. I also noted another bug: When building an RPM which contains more than one Extension(), bdist_rpm fails on the second Extension(): it can't find the C file. """ running build_ext building 'mx.DateTime.mxDateTime.mxDateTime' extension creating build/temp.linux2 creating build/temp.linux2/mx creating build/temp.linux2/mx/DateTime creating build/temp.linux2/mx/DateTime/mxDateTime gcc -g -O2 -fpic -Imx/DateTime/mxDateTime -I/usr/local/include/python2.0 -c mx/DateTime/mxDateTime/mxDateTime.c -o build/temp.linux2/mx/DateTime/mxDateTime/mxDateTime.o -O2 -m486 -fno-strength-reduce creating build/lib.linux2/mx/DateTime creating build/lib.linux2/mx/DateTime/mxDateTime gcc -shared build/temp.linux2/mx/DateTime/mxDateTime/mxDateTime.o -o build/lib.linux2/mx/DateTime/mxDateTime/mxDateTime.so building 'mx.Proxy.mxProxy.mxProxy' extension error: file 'mx/Proxy/mxProxy/mxProxy.c' does not exist Bad exit status from /var/tmp/rpm-tmp.85854 (%build) error: command 'rpm' failed with exit status 1 """ > > * the bdist commands don't compile the .py files into .pyo > > but this is needed in order to use them with python -O > > Yeah, known problem. How big do you want your RPM to be again? ;-( I don't really care for the size of the file as long as it runs in all possible settings (don't want a FAQ along the lines of "mx.DateTime fails to import..."). A simple solution would be to compile the Python code using os.system('python -O compileall.py') with the optimization level being a distutils option. > > * bdist (the dumb version) defaults to creating a tar.gz > > archive which is relative to / -- it shouldn't use the > > usr/local/ prefix to make the installation relocateable > > (e.g. to install under /usr, /opt, /opt/local, etc.) > > There's no one right way to do this; I've tried them both, and they're > both wrong. Originally, bdist_dumb created an archive relative to > -- relocatable, but it didn't work if prefix != exec-prefix. > (I can't remember if it bombed or created a bogus archive, but the > bottom line is it didn't work.) Now it creates an archive relative to > /, which handles prefix != exec-prefix, but is not relocatable. ;-( > > I think the answer is to let the packager decide, ie. add an option to > make the archive relative to / or prefix. (And it should blow up if > prefix != exec-prefix and you try to make a prefix-relative archive.) That would be ok. > > * how can I define the compiler settings for extension > > compilation (it defaults to -m486 for RPMs -- -mcpu=pentium > > would be more appropriate; default for standard bdist > > seems to be -O2 and no architecture) ? > > For a straight Distutil build (ie. no RPM involved), setting CFLAGS > should work, but it only applies to extensions (not C libraries). When > RPM is involved, I'm not sure what the Right Way to do it is. Maybe > CFLAGS will work? Maybe editing /etc/rpmrc or /usr/lib/rpm/rpmrc? I > dunno. Wouldn't an option to distutils be the right way for these kind of things ? > > And a general question: how can I define which files are > > documentation, examples and auxiliary files ? > > Doc files can be defined with the doc_files option to bdist_rpm. The > easiest way to set this is with a setup.cfg in your distribution root, > eg. the Distutils setup.cfg has: > > [bdist_rpm] > release = 1 > packager = Greg Ward > doc_files = CHANGES.txt > README.txt > USAGE.txt > doc/ > examples/ > ... > > Although I have been thinking that putting making every RPM option a > Distutils option was a bad idea. Hmmmm. Well, documentation is usually installed under /usr/doc/packages so this option is important. Perhaps you could add a generic pass-through-to-rpm option (much like the -Wl, option on gcc) ?! > > More to come :-) > > Oh no! You might want to take a look in the Distutils-0.9.2/TODO file > to see what's on "the list". Will do. Thanks for the feedback. > Greg > -- > Greg Ward - nerd gward@python.net > http://starship.python.net/~gward/ > Does your DRESSING ROOM have enough ASPARAGUS? > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG@python.org > http://www.python.org/mailman/listinfo/distutils-sig -- setup.py bdist_rpm output: rpm -ba --define _topdir /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm --clean build/bdist.linux2/rpm/SPECS/mx-Extensions-BASE.spec Executing: %prep + umask 022 + cd /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/BUILD + cd /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/BUILD + rm -rf mx-Extensions-BASE-1.0.0 + tar -xvvf - + /bin/gzip -dc /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/SOURCES/mx-Extensions-BASE-1.0.0.tar.gz drwxr-xr-x lemburg/users 0 2000-09-04 13:01 mx-Extensions-BASE-1.0.0/ drwxr-xr-x lemburg/users 0 2000-09-04 13:01 mx-Extensions-BASE-1.0.0/mx/ drwxr-xr-x lemburg/users 0 2000-09-04 13:01 mx-Extensions-BASE-1.0.0/mx/DateTime/ drwxr-xr-x lemburg/users 0 2000-09-04 13:01 mx-Extensions-BASE-1.0.0/mx/DateTime/Doc/ -rw-r--r-- lemburg/users 12532 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Doc/mxDateTime-History.html -rw-r--r-- lemburg/users 103704 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Doc/mxDateTime.html drwxr-xr-x lemburg/users 0 2000-09-04 13:01 mx-Extensions-BASE-1.0.0/mx/DateTime/Examples/ -rw-r--r-- lemburg/users 1577 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Examples/AtomicClock.py -rw-r--r-- lemburg/users 12973 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Examples/CommandLine.py -rw-r--r-- lemburg/users 399 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Examples/Y2000.py -rw-r--r-- lemburg/users 0 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Examples/__init__.py -rwxr-xr-x lemburg/users 1465 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Examples/alarm.py -rw-r--r-- lemburg/users 564 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Examples/lifespan.py drwxr-xr-x lemburg/users 0 2000-09-04 13:01 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/ -rw-r--r-- lemburg/users 10228 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/Makefile.pre.in -rw-r--r-- lemburg/users 655 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/Setup.in -rw-r--r-- lemburg/users 155 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/W95.bat -rw-r--r-- lemburg/users 521 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/__init__.py -rwxr-xr-x lemburg/users 5965 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/ld_so_aix -rwxr-xr-x lemburg/users 3101 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/makexp_aix -rw-r--r-- lemburg/users 19753 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mx.h -rw-r--r-- lemburg/users 99513 2000-09-01 15:08 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mxDateTime.c -rw-r--r-- lemburg/users 24 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mxDateTime.def -rw-r--r-- lemburg/users 10058 2000-09-01 15:08 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mxDateTime.h -rw-r--r-- lemburg/users 7039 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mxDateTime.mak -rwxr-xr-x lemburg/users 57344 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mxDateTime.pyd -rw-r--r-- lemburg/users 24804 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mxDateTime.sit.hqx -rw-r--r-- lemburg/users 18248 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mxDateTime_Python.py -rw-r--r-- lemburg/users 1554 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mxh.h -rw-r--r-- lemburg/users 8552 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mxpyapi.h -rw-r--r-- lemburg/users 6903 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mxstdlib.h -rw-r--r-- lemburg/users 9202 2000-09-01 15:08 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/test.py -rw-r--r-- lemburg/users 426 2000-09-01 15:08 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/testcmp.py -rw-r--r-- lemburg/users 385 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/testcomdates.py -rw-r--r-- lemburg/users 147 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/teststrftime.py -rw-r--r-- lemburg/users 1358 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/testticks.py -rw-r--r-- lemburg/users 480 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/win32mk.bat -rw-r--r-- lemburg/users 7290 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/ARPA.py -rw-r--r-- lemburg/users 26902 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/DateTime.py -rw-r--r-- lemburg/users 2724 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Feasts.py -rw-r--r-- lemburg/users 10374 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/ISO.py -rw-r--r-- lemburg/users 143 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/LICENSE -rw-r--r-- lemburg/users 2561 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/LazyModule.py -rw-r--r-- lemburg/users 3760 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Locale.py -rw-r--r-- lemburg/users 863 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Makefile.pkg -rw-r--r-- lemburg/users 14232 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/NIST.py -rw-r--r-- lemburg/users 5262 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/ODMG.py -rw-r--r-- lemburg/users 15580 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Parser.py -rw-r--r-- lemburg/users 142 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/README -rw-r--r-- lemburg/users 4353 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Timezone.py -rw-r--r-- lemburg/users 2110 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/__init__.py -rw-r--r-- lemburg/users 2753 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/timegm.py drwxr-xr-x lemburg/users 0 2000-09-04 13:01 mx-Extensions-BASE-1.0.0/mx/Misc/ -rw-r--r-- lemburg/users 2561 2000-09-01 15:10 mx-Extensions-BASE-1.0.0/mx/Misc/LazyModule.py -rw-r--r-- lemburg/users 0 2000-09-01 15:10 mx-Extensions-BASE-1.0.0/mx/Misc/__init__.py -rw-r--r-- lemburg/users 891 2000-09-01 15:08 mx-Extensions-BASE-1.0.0/mx/__init__.py -rwxr-xr-x lemburg/users 1587 2000-09-04 13:01 mx-Extensions-BASE-1.0.0/setup.py + STATUS=0 + '[' 0 -ne 0 ']' + cd mx-Extensions-BASE-1.0.0 ++ /usr/bin/id -u + '[' 500 = 0 ']' ++ /usr/bin/id -u + '[' 500 = 0 ']' + /bin/chmod -Rf a+rX,g-w,o-w . + exit 0 Executing: %build + umask 022 + cd /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/BUILD + cd mx-Extensions-BASE-1.0.0 + env 'CFLAGS=-O2 -m486 -fno-strength-reduce' python setup.py build running build running build_py creating build creating build/lib.linux2 creating build/lib.linux2/mx copying mx/__init__.py -> build/lib.linux2/mx running build_ext building 'mx.DateTime.mxDateTime.mxDateTime' extension creating build/temp.linux2 creating build/temp.linux2/mx creating build/temp.linux2/mx/DateTime creating build/temp.linux2/mx/DateTime/mxDateTime gcc -g -O2 -fpic -Imx/DateTime/mxDateTime -I/usr/local/include/python2.0 -c mx/DateTime/mxDateTime/mxDateTime.c -o build/temp.linux2/mx/DateTime/mxDateTime/mxDateTime.o -O2 -m486 -fno-strength-reduce creating build/lib.linux2/mx/DateTime creating build/lib.linux2/mx/DateTime/mxDateTime gcc -shared build/temp.linux2/mx/DateTime/mxDateTime/mxDateTime.o -o build/lib.linux2/mx/DateTime/mxDateTime/mxDateTime.so + exit 0 Executing: %install + umask 022 + cd /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/BUILD + cd mx-Extensions-BASE-1.0.0 + python setup.py install --root=/var/tmp/mx-Extensions-BASE-buildroot --record=INSTALLED_FILES running install running build running build_py not copying mx/__init__.py (output up-to-date) running build_ext skipping 'mx.DateTime.mxDateTime.mxDateTime' extension (up-to-date) running install_lib not copying build/lib.linux2/mx/__init__.py (output up-to-date) copying build/lib.linux2/mx/DateTime/mxDateTime/mxDateTime.so -> /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/DateTime/mxDateTime skipping byte-compilation of /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/__init__.py writing list of installed files to 'INSTALLED_FILES' warning: install: modules installed to '/var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/', which is not in Python's module search path (sys.path) -- you'll have to change the search path yourself + exit 0 Processing files: mx-Extensions-BASE Finding provides... Finding requires... Provides: mxDateTime.so Requires: ld-linux.so.2 libc.so.6 libc.so.6(GLIBC_2.0) libc.so.6(GLIBC_2.1) Wrote: /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/SRPMS/mx-Extensions-BASE-1.0.0-1.src.rpm Could not open /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/RPMS/i386/mx-Extensions-BASE-1.0.0-1.i386.rpm error: command 'rpm' failed with exit status 1 -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal@lemburg.com Mon Sep 4 08:19:01 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon Sep 4 07:19:01 2000 Subject: [Distutils] First steps with distutils... References: <39AFC048.6E963F82@lemburg.com> <39B0CD44.6B79F246@lemburg.com> <20000902114558.A6277@inet.net> Message-ID: <39B384DB.41CA8C45@lemburg.com> Harry Henry Gebel wrote: > > On Sat, Sep 02, 2000 at 11:49:56AM +0200, M.-A. Lemburg wrote: > > * bdist_rpm fails with an error in the rpm command (the source > > rpm is built, but it fails to build to the binary rpm for > > some reason -- it says, that it can't find the file ??); > > do I need some new rpm release for this to work ? > > bdist_rpm requires RPM version 3, there was some discussion about > supporting RPM 2; but it did not seem practical to me to support RPM 2 in a > way that would work across different RPM based platforms. It probably would > not be hard to make it produce version 2 spec files, but the packager would > then have to build them outside of the Distutils. I'm using rpm 3.0.3. FYI, I've attached a copy of the shell output to my reply to Greg. > > * the bdist commands don't compile the .py files into .pyo > > but this is needed in order to use them with python -O > > There were some attempts made to do this, but none has been satisfactory. See my reply to Greg... this should be possible via os.system('python -O'). > > * how can I define the compiler settings for extension > > compilation (it defaults to -m486 for RPMs -- -mcpu=pentium > > would be more appropriate; default for standard bdist > > seems to be -O2 and no architecture) ? > > The compiler options for RPMs are set by RPM (using CFLAGS) not by > Distutils, you can change these by modifying your RPM configuration (the > exact modifications depend on which version of RPM you are using.) > Otherwise you can use CFLAGS to set them, I think they can be set in > setup.py, but I'm not sure about that. Hmm, I would rather not have to edit system files for RPM since this would make the setup.py file rely on the system's config files. A distutils option would be my preferred solution. Should be simple to add: the option would just have to set os.environ['CFLAGS'] to whatever value is specified in setup.py. setup.py can then include all the necessary logic to determine the compiler flags (e.g. by checking the default setting of the compiler classes). Thanks, -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From R.Liebscher@gmx.de Mon Sep 4 09:48:02 2000 From: R.Liebscher@gmx.de (Rene Liebscher) Date: Mon Sep 4 08:48:02 2000 Subject: [Distutils] some small bugs References: <399BD06F.13FD9EAA@gmx.de> <20000821215505.A1131@beelzebub> <39ACDE11.B7F8AEE2@gmx.de> <20000831183138.A31473@python.net> <001101c013de$af7f8010$4500a8c0@thomasnb> Message-ID: <39B39955.4353081@gmx.de> Thomas Heller wrote: > > > > The patch fixes this. It also changes two other problems. > > > First it raises an PlatformException if you try to use it with > > > modules which contain compiled C-extensions. Distutils doesn't > > > support crosscompiling. > > > > Ooh, but wouldn't it be cool if I could compile Windows extensions on > Linux > > and vice-versa... oh, never mind! > > > I remember having heard that it is possible to cross-compile the windows > version > of tcl on linux... > Maybe someone with more experience than me on could find out? > It would be cool, but then you probably need special options for the compiler or may be a new cross-compiler class. (It would use different extensions than the standard unix compiler '.dll'<>'.so') Until someone has a solution for that, we should prevent users from creating installers that don't work. (Maybe we need then an option 'target_platform' for build_ext, this could then also be used to allow creation of the installer.) Kind regards Rene Liebscher From thomas.heller@ion-tof.com Mon Sep 4 12:31:13 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Mon Sep 4 11:31:13 2000 Subject: [Distutils] First steps with distutils... References: <39AFC048.6E963F82@lemburg.com> <39B0CD44.6B79F246@lemburg.com> <20000902104944.B1221@beelzebub> <39B38398.1B4079D3@lemburg.com> Message-ID: <029a01c01685$01db13d0$4500a8c0@thomasnb> > > > * bdist_wininst wants zlib but the installer has a note that > > > it is not using it anymore > > > > Thomas, is this true? > > I would make the usage of zlib optional with distutils defaulting > to not using it (since it isn't enabled by default in the Python > distribution). > > > > * bdist_wininst fails with a setup.py file which does not > > > define long_description > > > > I thought that was fixed in 0.9.2 -- !@&%@!#&!@#&%! The current CVS version, which is the same as 0.9.2, does NOT use zlib any longer, also long_description is no longer needed. Thomas From gward@python.net Mon Sep 4 16:20:02 2000 From: gward@python.net (Greg Ward) Date: Mon Sep 4 15:20:02 2000 Subject: [Distutils] bdist_debian In-Reply-To: ; from calvin@cs.uni-sb.de on Sun, Sep 03, 2000 at 10:35:44PM +0200 References: <20000902111323.D1221@beelzebub> Message-ID: <20000904151758.A1887@beelzebub> On 03 September 2000, Bastian Kleineidam said: > Debian Packages with Distutils: > > 1) You are on a Debian system I think we can assume this most of the time. To build RPMs, you have to be a on a system with "rpm"; to build Windows installers with binary code, you have to be on a Windows system. The fact that you can build pure Python Windows installers under Linux is a happy accident. > 2) But I am running a non-Debian system > a) Read something about the .deb format in deb(5): > The file is an ar archive with a magic number of !. As soon as I saw the word "magic", I knew that trying to do it this way was Wrong Wrong Wrong. > This internal format is subject to change (between major releases > of Debian GNU/Linux), therefore please always use dpkg-deb(8) for > manipulating .deb files. More confirmation. Bottom line: go ahead and assume the presence of Debian package-building tools for building Debian packages. Now, here's the interesting question: is it possible to install RPM on a Debian system, or the Debian tools on an RPM system, solely for use in producing the "other" type of package? Seems like it ought to be possible at least for RPM-under-Debian, since AFAIK creating RPMs doesn't affect or use the RPM database in /var/lib/rpm. Can anyone speak for creating Debian packages on an RPM system? Greg -- Greg Ward - geek-at-large gward@python.net http://starship.python.net/~gward/ "One world, one web, one program" --Microsoft "Eine volk, eine reich, eine führer" --Hitler From gward@python.net Mon Sep 4 16:53:01 2000 From: gward@python.net (Greg Ward) Date: Mon Sep 4 15:53:01 2000 Subject: [Distutils] First steps with distutils... In-Reply-To: <39B38398.1B4079D3@lemburg.com>; from mal@lemburg.com on Mon, Sep 04, 2000 at 01:12:24PM +0200 References: <39AFC048.6E963F82@lemburg.com> <39B0CD44.6B79F246@lemburg.com> <20000902104944.B1221@beelzebub> <39B38398.1B4079D3@lemburg.com> Message-ID: <20000904155056.B1887@beelzebub> On 04 September 2000, M.-A. Lemburg said: > > Correct. MANIFEST is a simple list of files, one per line. If you want > > to include whole directories, you should write a MANIFEST.in file. > > I didn't mean that distutils should scan the goven directory -- > it should simply ignore it instead of raising an exception. Hmm, are you really getting an exception? When I put a directory name into a MANIFEST file, I get: error: can't copy 'pkg': doesn't exist or not a regular file and immediate termination of the setup script (hey, it's an error, not a warning). I'll admit that's not as useful as it could be, but as long as there's no traceback it's not a bug! ;-) Actually, I spent some time documenting the "sdist" command today, and tracked down some bugs in the handling of the manifest files. I also now understand the rules a bit better -- amazing what reading the code will tell you. ;-) Here are the rules as implemented in the current code (not even checked in yet, because of the 2.0b1 code freeze): cases: 1) no manifest, template exists: generate manifest (covered by 2a: no manifest == template newer) 2) manifest & template exist: 2a) template or setup script newer than manifest: regenerate manifest 2b) manifest newer than both: do nothing (unless --force or --manifest-only) 3) manifest exists, no template: do nothing (unless --force or --manifest-only) 4) no manifest, no template: generate w/ warning ("defaults only") In the 0.9.2 code (and all Distutils releases since at least 0.2), case (4) wasn't handled properly. A workaround is to use the -f (--force-manifest) option, which causes the sdist command to unconditionally regenerate the MANIFEST file. Also, case (3) implies that the "default file set" -- setup.py, README.txt, and any source code specified in the setup script -- will *not* be used if you supply your own MANIFEST file. Additionally, the default "prune list" -- which strips RCS and CVS directories, as well as Distutils-generated temporary directories -- does not apply. IOW, if you generate your own MANIFEST, you have to get it exactly right. I think this is the right thing to do, because only control freaks will want to supply their own MANIFEST file, and taking away any control from control freaks makes them, well, freak out. ;-) See the 'get_file_list()' method in distutils/command/sdist.py if you're curious about the implementation. (And see if you can spot the bug that I fixed this morning.) (No fair watching python-checkins!) > I'm currently using my own tools for generating the MANIFEST > file. The most important difference is that they allow per > directory MANIFEST.in style files which are appended to the > general MANIFEST.in logic while scanning the directory. Hmmm, another good reason to roll your own manifest -- maybe it's not just for control freaks? > My version if rpm 3.0.3. [...] > I also noted another bug: > When building an RPM which contains more than one Extension(), > bdist_rpm fails on the second Extension(): it can't find > the C file. > """ > running build_ext > building 'mx.DateTime.mxDateTime.mxDateTime' extension > creating build/temp.linux2 > creating build/temp.linux2/mx > creating build/temp.linux2/mx/DateTime > creating build/temp.linux2/mx/DateTime/mxDateTime > gcc -g -O2 -fpic -Imx/DateTime/mxDateTime -I/usr/local/include/python2.0 -c mx/DateTime/mxDateTime/mxDateTime.c -o build/temp.linux2/mx/DateTime/mxDateTime/mxDateTime.o -O2 -m486 -fno-strength-reduce > creating build/lib.linux2/mx/DateTime > creating build/lib.linux2/mx/DateTime/mxDateTime > gcc -shared build/temp.linux2/mx/DateTime/mxDateTime/mxDateTime.o -o build/lib.linux2/mx/DateTime/mxDateTime/mxDateTime.so > building 'mx.Proxy.mxProxy.mxProxy' extension > error: file 'mx/Proxy/mxProxy/mxProxy.c' does not exist > Bad exit status from /var/tmp/rpm-tmp.85854 (%build) > error: command 'rpm' failed with exit status 1 > """ ??? Very weird. Does "python setup.py build" work on its own, ie. when run by you rather than by rpm in its build directory? One simple possibility is that you forgot to include mx/Proxy/mxProxy/mxProxy.c in your tarball -- make sure that you can unpack your tarball in a fresh directory and run "python setup.py build", which is just what rpm is doing. > > For a straight Distutil build (ie. no RPM involved), setting CFLAGS > > should work, but it only applies to extensions (not C libraries). When > > RPM is involved, I'm not sure what the Right Way to do it is. Maybe > > CFLAGS will work? Maybe editing /etc/rpmrc or /usr/lib/rpm/rpmrc? I > > dunno. > > Wouldn't an option to distutils be the right way for these kind > of things ? Yes. Definitely. It's on the TODO list. Now, on to your other "bdist_rpm" failure... > setup.py bdist_rpm output: > > rpm -ba --define _topdir /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm --clean build/bdist.linux2/rpm/SPECS/mx-Extensions-BASE.spec > Executing: %prep [...] > Wrote: /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/SRPMS/mx-Extensions-BASE-1.0.0-1.src.rpm > Could not open /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/RPMS/i386/mx-Extensions-BASE-1.0.0-1.i386.rpm > > error: command 'rpm' failed with exit status 1 Hmmm, this smells like the archetypal Distutils bug: someone's trying to write a file into a directory that doesn't exist yet. In this case, it would be rpm's fault, since the "bdist_rpm" command doesn't know the name of the architecture directory that rpm will try to write to. Things to look for: * does /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/RPMS/i386 exist? * if you create it, does the bdist_rpm command then run successfully? I haven't had problems with rpm failing to create directories -- I have RPM 3.0.4 on my Red Hat 6.2 box, and I just cranked out an RPM of mxDateTime 1.3.0 without a hitch. Perhaps RPM 3.0.3 joins RPM 2.x on the index of forbidden versions... sigh... Greg -- Greg Ward - Unix bigot gward@python.net http://starship.python.net/~gward/ If at first you don't succeed, redefine success. From akuchlin@mems-exchange.org Mon Sep 4 17:47:01 2000 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Mon Sep 4 16:47:01 2000 Subject: [Distutils] First steps with distutils... In-Reply-To: <20000904155056.B1887@beelzebub>; from gward@python.net on Mon, Sep 04, 2000 at 03:50:56PM -0400 References: <39AFC048.6E963F82@lemburg.com> <39B0CD44.6B79F246@lemburg.com> <20000902104944.B1221@beelzebub> <39B38398.1B4079D3@lemburg.com> <20000904155056.B1887@beelzebub> Message-ID: <20000904164647.A8349@newcnri.cnri.reston.va.us> On Mon, Sep 04, 2000 at 03:50:56PM -0400, Greg Ward wrote: >I haven't had problems with rpm failing to create directories -- I have >RPM 3.0.4 on my Red Hat 6.2 box, and I just cranked out an RPM of >mxDateTime 1.3.0 without a hitch. Perhaps RPM 3.0.3 joins RPM 2.x on >the index of forbidden versions... sigh... I think it does; I noticed this problem on my machine, kronos, which has RPM 3.0.3 installed. --amk From mal@lemburg.com Mon Sep 4 18:51:19 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon Sep 4 17:51:19 2000 Subject: [Distutils] First steps with distutils... References: <39AFC048.6E963F82@lemburg.com> <39B0CD44.6B79F246@lemburg.com> <20000902104944.B1221@beelzebub> <39B38398.1B4079D3@lemburg.com> <20000904155056.B1887@beelzebub> Message-ID: <39B41915.5BA7B56C@lemburg.com> Greg Ward wrote: > > On 04 September 2000, M.-A. Lemburg said: > > > Correct. MANIFEST is a simple list of files, one per line. If you want > > > to include whole directories, you should write a MANIFEST.in file. > > > > I didn't mean that distutils should scan the goven directory -- > > it should simply ignore it instead of raising an exception. > > Hmm, are you really getting an exception? When I put a directory name > into a MANIFEST file, I get: > > error: can't copy 'pkg': doesn't exist or not a regular file > > and immediate termination of the setup script (hey, it's an error, not a > warning). I'll admit that's not as useful as it could be, but as long > as there's no traceback it's not a bug! ;-) You're right; it's not an exception, but still something which causes distutils to stop ;-) Would it hurt much adding a "continue" somewhere to remedy this minor "caveat" :-) > Actually, I spent some time documenting the "sdist" command today, and > tracked down some bugs in the handling of the manifest files. I also > now understand the rules a bit better -- amazing what reading the code > will tell you. ;-) Here are the rules as implemented in the current > code (not even checked in yet, because of the 2.0b1 code freeze): > > cases: > 1) no manifest, template exists: generate manifest > (covered by 2a: no manifest == template newer) > 2) manifest & template exist: > 2a) template or setup script newer than manifest: > regenerate manifest > 2b) manifest newer than both: > do nothing (unless --force or --manifest-only) > 3) manifest exists, no template: > do nothing (unless --force or --manifest-only) > 4) no manifest, no template: generate w/ warning ("defaults only") > > In the 0.9.2 code (and all Distutils releases since at least 0.2), case > (4) wasn't handled properly. A workaround is to use the -f > (--force-manifest) option, which causes the sdist command to > unconditionally regenerate the MANIFEST file. > > Also, case (3) implies that the "default file set" -- setup.py, > README.txt, and any source code specified in the setup script -- will > *not* be used if you supply your own MANIFEST file. Additionally, the > default "prune list" -- which strips RCS and CVS directories, as well as > Distutils-generated temporary directories -- does not apply. IOW, if > you generate your own MANIFEST, you have to get it exactly right. I > think this is the right thing to do, because only control freaks will > want to supply their own MANIFEST file, and taking away any control from > control freaks makes them, well, freak out. ;-) True ;-) > See the 'get_file_list()' method in distutils/command/sdist.py if you're > curious about the implementation. (And see if you can spot the bug that > I fixed this morning.) (No fair watching python-checkins!) > > > I'm currently using my own tools for generating the MANIFEST > > file. The most important difference is that they allow per > > directory MANIFEST.in style files which are appended to the > > general MANIFEST.in logic while scanning the directory. > > Hmmm, another good reason to roll your own manifest -- maybe it's not > just for control freaks? > > > My version if rpm 3.0.3. > [...] > > I also noted another bug: > > When building an RPM which contains more than one Extension(), > > bdist_rpm fails on the second Extension(): it can't find > > the C file. > > """ > > running build_ext > > building 'mx.DateTime.mxDateTime.mxDateTime' extension > > creating build/temp.linux2 > > creating build/temp.linux2/mx > > creating build/temp.linux2/mx/DateTime > > creating build/temp.linux2/mx/DateTime/mxDateTime > > gcc -g -O2 -fpic -Imx/DateTime/mxDateTime -I/usr/local/include/python2.0 -c mx/DateTime/mxDateTime/mxDateTime.c -o build/temp.linux2/mx/DateTime/mxDateTime/mxDateTime.o -O2 -m486 -fno-strength-reduce > > creating build/lib.linux2/mx/DateTime > > creating build/lib.linux2/mx/DateTime/mxDateTime > > gcc -shared build/temp.linux2/mx/DateTime/mxDateTime/mxDateTime.o -o build/lib.linux2/mx/DateTime/mxDateTime/mxDateTime.so > > building 'mx.Proxy.mxProxy.mxProxy' extension > > error: file 'mx/Proxy/mxProxy/mxProxy.c' does not exist > > Bad exit status from /var/tmp/rpm-tmp.85854 (%build) > > error: command 'rpm' failed with exit status 1 > > """ > > ??? Very weird. Does "python setup.py build" work on its own, ie. when > run by you rather than by rpm in its build directory? Yes. Works just fine. > One simple possibility is that you forgot to include > mx/Proxy/mxProxy/mxProxy.c in your tarball -- make sure that you can > unpack your tarball in a fresh directory and run "python setup.py > build", which is just what rpm is doing. That could have been the cause. While testing this I stumbled over another bugglet: Executing: %build + umask 022 + cd /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/BUILD + cd mx-Extensions-BASE-1.0.0 + env 'CFLAGS=-O2 -m486 -fno-strength-reduce' python setup.py build Traceback (innermost last): File "setup.py", line 7, in ? from distutils.core import setup, Extension File "/home/lemburg/lib/distutils/core.py", line 146 raise ^ SyntaxError: invalid syntax This is due to RPM finding a Python 1.5 version under the simple name "python".... distutils doesn't seem to be compatible with multiple installed Python versions :-( [I did run the setup.py file using Python 2.0]. I then tried: projects/tmp> python2.0 setup.py bdist_rpm --prep-script ./prep-script invalid command name './prep-script' Looks like somethings wrong there: I can't seem to pass a script filename to the command. I had to hack the bdist_rpm.py file to make it work (perhaps time for yet another option ?-) ... OK, after that hack the RPM compile runs through, but it now finishes with the following lines (and no apparent reason): creating /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools creating /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools/mxTools copying build/lib.linux2/mx/Tools/mxTools/mxTools.so -> /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools/mxTools byte-compiling /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/__init__.py to __init__.pyc writing list of installed files to 'INSTALLED_FILES' warning: install: modules installed to '/var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/', which is not in Python's module search path (sys.path) -- you'll have to change the search path yourself + exit 0 Processing files: mx-Extensions-BASE File listed twice: /usr/local/lib/python2.0/site-packages/mx/Tools/mxTools/mxTools.so Finding provides... Finding requires... Provides: mxDateTime.so mxProxy.so mxQueue.so mxStack.so mxTextTools.so mxTools.so Requires: ld-linux.so.2 libc.so.6 libc.so.6(GLIBC_2.0) libc.so.6(GLIBC_2.1) error: command 'rpm' failed with exit status 1 Does this mean that my system doesn't provide the "required" packages ? (SuSE has a somewhat different naming scheme than RedHat so this could be a possible cause) > > > For a straight Distutil build (ie. no RPM involved), setting CFLAGS > > > should work, but it only applies to extensions (not C libraries). When > > > RPM is involved, I'm not sure what the Right Way to do it is. Maybe > > > CFLAGS will work? Maybe editing /etc/rpmrc or /usr/lib/rpm/rpmrc? I > > > dunno. > > > > Wouldn't an option to distutils be the right way for these kind > > of things ? > > Yes. Definitely. It's on the TODO list. Great :-) > Now, on to your other "bdist_rpm" failure... > > > setup.py bdist_rpm output: > > > > rpm -ba --define _topdir /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm --clean build/bdist.linux2/rpm/SPECS/mx-Extensions-BASE.spec > > Executing: %prep > [...] > > Wrote: /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/SRPMS/mx-Extensions-BASE-1.0.0-1.src.rpm > > Could not open /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/RPMS/i386/mx-Extensions-BASE-1.0.0-1.i386.rpm > > > > error: command 'rpm' failed with exit status 1 > > Hmmm, this smells like the archetypal Distutils bug: someone's trying to > write a file into a directory that doesn't exist yet. In this case, it > would be rpm's fault, since the "bdist_rpm" command doesn't know the > name of the architecture directory that rpm will try to write to. > Things to look for: > * does /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/RPMS/i386 > exist? No. > * if you create it, does the bdist_rpm command then run successfully? Yes. > I haven't had problems with rpm failing to create directories -- I have > RPM 3.0.4 on my Red Hat 6.2 box, and I just cranked out an RPM of > mxDateTime 1.3.0 without a hitch. Perhaps RPM 3.0.3 joins RPM 2.x on > the index of forbidden versions... sigh... Dunno. It did work after I manually created the i386 directory and applied the python2.0 hack to bdist_rpm.py. Thanks, -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gward@python.net Mon Sep 4 23:47:03 2000 From: gward@python.net (Greg Ward) Date: Mon Sep 4 22:47:03 2000 Subject: [Distutils] First steps with distutils... In-Reply-To: <39B41915.5BA7B56C@lemburg.com>; from mal@lemburg.com on Mon, Sep 04, 2000 at 11:50:13PM +0200 References: <39AFC048.6E963F82@lemburg.com> <39B0CD44.6B79F246@lemburg.com> <20000902104944.B1221@beelzebub> <39B38398.1B4079D3@lemburg.com> <20000904155056.B1887@beelzebub> <39B41915.5BA7B56C@lemburg.com> Message-ID: <20000904224526.A2683@beelzebub> On 04 September 2000, M.-A. Lemburg said: > > error: can't copy 'pkg': doesn't exist or not a regular file > > > > and immediate termination of the setup script (hey, it's an error, not a > > warning). I'll admit that's not as useful as it could be, but as long > > as there's no traceback it's not a bug! ;-) > > You're right; it's not an exception, but still something which > causes distutils to stop ;-) It is fixable, I was just making sure that you and I were seeing the same problem -- "that's not a bug, it's a misfeature". Another thing I discovered this morning is that if MANIFEST is empty, then you *do* get an exception -- definitely a bug. So at least a "! empty" check of the file list is required before trying to copy files and create the tarball; might as well throw in a "isfile or warn" check on everything at the same time. > Would it hurt much adding a "continue" somewhere to remedy > this minor "caveat" :-) Well, it'll print a warning that "foo is not a regular file (skipped)". Not nice to silently skip things! > That could have been the cause. While testing this I stumbled > over another bugglet: > > Executing: %build > + umask 022 > + cd /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/BUILD > + cd mx-Extensions-BASE-1.0.0 > + env 'CFLAGS=-O2 -m486 -fno-strength-reduce' python setup.py build > Traceback (innermost last): > File "setup.py", line 7, in ? > from distutils.core import setup, Extension > File "/home/lemburg/lib/distutils/core.py", line 146 > raise > ^ > SyntaxError: invalid syntax > > This is due to RPM finding a Python 1.5 version under the > simple name "python".... distutils doesn't seem to be compatible > with multiple installed Python versions :-( [I did run the > setup.py file using Python 2.0]. Yup, there's another thread about this very issue going on at this very moment. Everyone who has spoken up (me, AMK, and Harry Gebel) thinks that it should at least be an option to put the value of sys.executable in the spec file, instead of hard-coding "python". I think you've raised another good argument in favour of making this the default, but I'm biased -- I do think it should be the default. ;-) > I then tried: > > projects/tmp> python2.0 setup.py bdist_rpm --prep-script ./prep-script > invalid command name './prep-script' > > Looks like somethings wrong there: I can't seem to pass a script > filename to the command. The RPM script options are, ummm, not well-thought-out. Possibly unfinished, I'm not sure. Those are the options I was thinking of when I said it might have been unwise to attempt to mirror *all* of RPM's spec file in Distutils options. My inclination is to delete them, rather than expose a broken interface. > OK, after that hack the RPM compile runs through, but it now finishes > with the following lines (and no apparent reason): > > creating /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools > creating /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools/mxTools > copying build/lib.linux2/mx/Tools/mxTools/mxTools.so -> /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools/mxTools > byte-compiling /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/__init__.py to __init__.pyc > writing list of installed files to 'INSTALLED_FILES' > warning: install: modules installed to '/var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/', which is not in Python's module search path (sys.path) -- you'll have to change the search path yourself > + exit 0 > Processing files: mx-Extensions-BASE > File listed twice: /usr/local/lib/python2.0/site-packages/mx/Tools/mxTools/mxTools.so > Finding provides... > Finding requires... > Provides: mxDateTime.so mxProxy.so mxQueue.so mxStack.so mxTextTools.so mxTools.so > Requires: ld-linux.so.2 libc.so.6 libc.so.6(GLIBC_2.0) libc.so.6(GLIBC_2.1) > error: command 'rpm' failed with exit status 1 ??? I'm confused again. Try running "bdist_rpm" with the --source-only option, and then see if you can build from the source RPM directly. About the only way to debug these sort of problems is to break the thing down, and a *heck* of a lot of stuff happens between you typing "setup.py bdist_rpm" and the finished RPM files being moved into dist/. (Just listen to your hard drive go!) > > * does /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/RPMS/i386 > > exist? > > No. > > > * if you create it, does the bdist_rpm command then run successfully? > > Yes. OK, based on that and Andrew's feedback, I'd say we blame RPM 3.0.3. Bottom line: building RPMs with the Distutils requires RPM 3.0.4 or better. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From mal@lemburg.com Tue Sep 5 04:55:04 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue Sep 5 03:55:04 2000 Subject: [Distutils] First steps with distutils... References: <39AFC048.6E963F82@lemburg.com> <39B0CD44.6B79F246@lemburg.com> <20000902104944.B1221@beelzebub> <39B38398.1B4079D3@lemburg.com> <20000904155056.B1887@beelzebub> <39B41915.5BA7B56C@lemburg.com> <20000904224526.A2683@beelzebub> Message-ID: <39B4A6B0.3FAB1C21@lemburg.com> Greg Ward wrote: > > On 04 September 2000, M.-A. Lemburg said: > > > error: can't copy 'pkg': doesn't exist or not a regular file > > > > > > and immediate termination of the setup script (hey, it's an error, not a > > > warning). I'll admit that's not as useful as it could be, but as long > > > as there's no traceback it's not a bug! ;-) > > > > You're right; it's not an exception, but still something which > > causes distutils to stop ;-) > > It is fixable, I was just making sure that you and I were seeing the > same problem -- "that's not a bug, it's a misfeature". Another thing I > discovered this morning is that if MANIFEST is empty, then you *do* get > an exception -- definitely a bug. So at least a "! empty" check of the > file list is required before trying to copy files and create the > tarball; might as well throw in a "isfile or warn" check on everything > at the same time. > > > Would it hurt much adding a "continue" somewhere to remedy > > this minor "caveat" :-) > > Well, it'll print a warning that "foo is not a regular file (skipped)". > Not nice to silently skip things! Ok. > > That could have been the cause. While testing this I stumbled > > over another bugglet: > > > > Executing: %build > > + umask 022 > > + cd /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/BUILD > > + cd mx-Extensions-BASE-1.0.0 > > + env 'CFLAGS=-O2 -m486 -fno-strength-reduce' python setup.py build > > Traceback (innermost last): > > File "setup.py", line 7, in ? > > from distutils.core import setup, Extension > > File "/home/lemburg/lib/distutils/core.py", line 146 > > raise > > ^ > > SyntaxError: invalid syntax > > > > This is due to RPM finding a Python 1.5 version under the > > simple name "python".... distutils doesn't seem to be compatible > > with multiple installed Python versions :-( [I did run the > > setup.py file using Python 2.0]. > > Yup, there's another thread about this very issue going on at this very > moment. Everyone who has spoken up (me, AMK, and Harry Gebel) thinks > that it should at least be an option to put the value of sys.executable > in the spec file, instead of hard-coding "python". I think you've > raised another good argument in favour of making this the default, but > I'm biased -- I do think it should be the default. ;-) Hmm, this depends on how the spec file is created. [Warning: I don't know much about RPM...] If the contents of the spec are included in the RPM file then it is probably a bad idea to include sys.executable of the building Python interpreter. In that case, I'd leave the hard- coded "python" in there and issue a warning that the executable found as "python" is not identical to sys.executable. The reason is simple: you cannot know what Python version a user uses to rebuild from SRPM. If the spec is *always* recreated, then I'd opt for the sys.executable default too, since it is natural that the Python version running setup.py will also take care of building the RPM sources. > > I then tried: > > > > projects/tmp> python2.0 setup.py bdist_rpm --prep-script ./prep-script > > invalid command name './prep-script' > > > > Looks like somethings wrong there: I can't seem to pass a script > > filename to the command. > > The RPM script options are, ummm, not well-thought-out. Possibly > unfinished, I'm not sure. Those are the options I was thinking of when > I said it might have been unwise to attempt to mirror *all* of RPM's > spec file in Distutils options. My inclination is to delete them, > rather than expose a broken interface. Fine with me... as long as you do publish a working interface to pass arbitrary RPM options to "rpm" ;-) > > OK, after that hack the RPM compile runs through, but it now finishes > > with the following lines (and no apparent reason): > > > > creating /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools > > creating /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools/mxTools > > copying build/lib.linux2/mx/Tools/mxTools/mxTools.so -> /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools/mxTools > > byte-compiling /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/__init__.py to __init__.pyc > > writing list of installed files to 'INSTALLED_FILES' > > warning: install: modules installed to '/var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/', which is not in Python's module search path (sys.path) -- you'll have to change the search path yourself > > + exit 0 > > Processing files: mx-Extensions-BASE > > File listed twice: /usr/local/lib/python2.0/site-packages/mx/Tools/mxTools/mxTools.so > > Finding provides... > > Finding requires... > > Provides: mxDateTime.so mxProxy.so mxQueue.so mxStack.so mxTextTools.so mxTools.so > > Requires: ld-linux.so.2 libc.so.6 libc.so.6(GLIBC_2.0) libc.so.6(GLIBC_2.1) > > error: command 'rpm' failed with exit status 1 > > ??? I'm confused again. > > Try running "bdist_rpm" with the --source-only option, and then see if > you can build from the source RPM directly. About the only way to debug > these sort of problems is to break the thing down, and a *heck* of a lot > of stuff happens between you typing "setup.py bdist_rpm" and the > finished RPM files being moved into dist/. (Just listen to your hard > drive go!) This works. Hmm, runnning /home/lemburg> rpm -q --whatprovides ld-linux.so.2 libc.so.6 "libc.so.6(GLIBC_2.0)" "libc.so.6(GLIBC_2.1)" I get: shlibs-2.1.1-4 shlibs-2.1.1-4 javarunt-1.1.7v1a-11 shlibs-2.1.1-4 shlibs-2.1.1-4 This looks ok, so it's not the "Requires" part which causes the problem either. > > > * does /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/RPMS/i386 > > > exist? > > > > No. > > > > > * if you create it, does the bdist_rpm command then run successfully? > > > > Yes. > > OK, based on that and Andrew's feedback, I'd say we blame RPM 3.0.3. > > Bottom line: building RPMs with the Distutils requires RPM 3.0.4 or > better. I'll try upgrading to rpm 3.0.4... -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From hgebel@conectiv.net Tue Sep 5 05:04:01 2000 From: hgebel@conectiv.net (Harry Henry Gebel) Date: Tue Sep 5 04:04:01 2000 Subject: [Distutils] First steps with distutils... In-Reply-To: <39B4A6B0.3FAB1C21@lemburg.com>; from mal@lemburg.com on Tue, Sep 05, 2000 at 09:54:24AM +0200 References: <39AFC048.6E963F82@lemburg.com> <39B0CD44.6B79F246@lemburg.com> <20000902104944.B1221@beelzebub> <39B38398.1B4079D3@lemburg.com> <20000904155056.B1887@beelzebub> <39B41915.5BA7B56C@lemburg.com> <20000904224526.A2683@beelzebub> <39B4A6B0.3FAB1C21@lemburg.com> Message-ID: <20000905040326.B4702@inet.net> On Tue, Sep 05, 2000 at 09:54:24AM +0200, M.-A. Lemburg wrote: > If the contents of the spec are included in the RPM file then > it is probably a bad idea to include sys.executable of the > building Python interpreter. In that case, I'd leave the hard- > coded "python" in there and issue a warning that the > executable found as "python" is not identical to sys.executable. > The reason is simple: you cannot know what Python version > a user uses to rebuild from SRPM. > > If the spec is *always* recreated, then I'd opt for the > sys.executable default too, since it is natural that > the Python version running setup.py will also take care > of building the RPM sources. The contents of the spec file are included in the source RPM, that is why A think that using sys.executable should be the option and not the default. -- Harry Henry Gebel, Senior Developer, Landon House SBS ICQ# 76308382 West Dover Hundred, Delaware From mal@lemburg.com Tue Sep 5 05:39:01 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue Sep 5 04:39:01 2000 Subject: [Distutils] First steps with distutils... References: <39AFC048.6E963F82@lemburg.com> <39B0CD44.6B79F246@lemburg.com> <20000902104944.B1221@beelzebub> <39B38398.1B4079D3@lemburg.com> <20000904155056.B1887@beelzebub> <39B41915.5BA7B56C@lemburg.com> <20000904224526.A2683@beelzebub> <39B4A6B0.3FAB1C21@lemburg.com> Message-ID: <39B4B108.B1F628C0@lemburg.com> "M.-A. Lemburg" wrote: > > > > OK, after that hack the RPM compile runs through, but it now finishes > > > with the following lines (and no apparent reason): > > > > > > creating /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools > > > creating /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools/mxTools > > > copying build/lib.linux2/mx/Tools/mxTools/mxTools.so -> /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools/mxTools > > > byte-compiling /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/__init__.py to __init__.pyc > > > writing list of installed files to 'INSTALLED_FILES' > > > warning: install: modules installed to '/var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/', which is not in Python's module search path (sys.path) -- you'll have to change the search path yourself > > > + exit 0 > > > Processing files: mx-Extensions-BASE > > > File listed twice: /usr/local/lib/python2.0/site-packages/mx/Tools/mxTools/mxTools.so > > > Finding provides... > > > Finding requires... > > > Provides: mxDateTime.so mxProxy.so mxQueue.so mxStack.so mxTextTools.so mxTools.so > > > Requires: ld-linux.so.2 libc.so.6 libc.so.6(GLIBC_2.0) libc.so.6(GLIBC_2.1) > > > error: command 'rpm' failed with exit status 1 > > > > ??? I'm confused again. > > > > Try running "bdist_rpm" with the --source-only option, and then see if > > you can build from the source RPM directly. About the only way to debug > > these sort of problems is to break the thing down, and a *heck* of a lot > > of stuff happens between you typing "setup.py bdist_rpm" and the > > finished RPM files being moved into dist/. (Just listen to your hard > > drive go!) > > This works. > > Hmm, runnning > > /home/lemburg> rpm -q --whatprovides ld-linux.so.2 libc.so.6 "libc.so.6(GLIBC_2.0)" "libc.so.6(GLIBC_2.1)" > > I get: > > shlibs-2.1.1-4 > shlibs-2.1.1-4 > javarunt-1.1.7v1a-11 > shlibs-2.1.1-4 > shlibs-2.1.1-4 > > This looks ok, so it's not the "Requires" part which causes the > problem either. Ok, I found the bug... RPM didn't like a mistake I made in the setup.py file: I listed one Extension twice. While distutils didn't warn about this, RPM fails to produce a binary RPM if a file is "listed twice". Removing the duplicate in setup.py remedies the problem. Now I have another problem ;-): the created RPM doesn't include all needed files, just the base package's __init__.py file and the compiled .so files: tmp/dist> rpm -qpl mx-Extensions-BASE-1.0.0-1.i386.rpm /usr/local/lib/python2.0/site-packages/mx/DateTime/mxDateTime/mxDateTime.so /usr/local/lib/python2.0/site-packages/mx/Proxy/mxProxy/mxProxy.so /usr/local/lib/python2.0/site-packages/mx/Queue/mxQueue/mxQueue.so /usr/local/lib/python2.0/site-packages/mx/Stack/mxStack/mxStack.so /usr/local/lib/python2.0/site-packages/mx/TextTools/mxTextTools/mxTextTools.so /usr/local/lib/python2.0/site-packages/mx/Tools/mxTools/mxTools.so /usr/local/lib/python2.0/site-packages/mx/__init__.py /usr/local/lib/python2.0/site-packages/mx/__init__.pyc Here's my setup.py file: #!/usr/bin/env python2.0 """ Distutils Setup File for the mx Extensions BASE distribution. """ import string from distutils.core import setup, Extension from distutils.command.config import config __version__ = '1.0.0' setup (name = "mx-Extensions-BASE", version = __version__, description = "", long_description = "", author = "Marc-André Lemburg", author_email = "mal@lemburg.com", url = "http://starship.python.net/~lemburg/mxBASE.html", packages = ['mx'], ext_modules = [ Extension('mx.DateTime.mxDateTime.mxDateTime', ['mx/DateTime/mxDateTime/mxDateTime.c'], include_dirs=['mx/DateTime/mxDateTime']), Extension('mx.Proxy.mxProxy.mxProxy', ['mx/Proxy/mxProxy/mxProxy.c'], include_dirs=['mx/Proxy/mxProxy']), Extension('mx.Queue.mxQueue.mxQueue', ['mx/Queue/mxQueue/mxQueue.c'], include_dirs=['mx/Queue/mxQueue']), Extension('mx.Stack.mxStack.mxStack', ['mx/Stack/mxStack/mxStack.c'], include_dirs=['mx/Stack/mxStack']), Extension('mx.TextTools.mxTextTools.mxTextTools', ['mx/TextTools/mxTextTools/mxTextTools.c'], include_dirs=['mx/TextTools/mxTextTools']), Extension('mx.Tools.mxTools.mxTools', ['mx/Tools/mxTools/mxTools.c'], include_dirs=['mx/Tools/mxTools']), ] ) Do I need to include *all* subpackages in the packages list ? (distutils seems to work without it.) -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From Moshe Zadka Tue Sep 5 10:23:03 2000 From: Moshe Zadka (Moshe Zadka) Date: Tue Sep 5 09:23:03 2000 Subject: [Distutils] bdist_debian In-Reply-To: <20000904151758.A1887@beelzebub> Message-ID: On Mon, 4 Sep 2000, Greg Ward wrote: > tools for building Debian packages. > > Now, here's the interesting question: is it possible to install RPM on a > Debian system, or the Debian tools on an RPM system, solely for use in > producing the "other" type of package? Seems like it ought to be > possible at least for RPM-under-Debian, since AFAIK creating RPMs > doesn't affect or use the RPM database in /var/lib/rpm. Can anyone > speak for creating Debian packages on an RPM system? I'm sure about RPM-on-Debian. In fact, via a package called alien, it is also possible to convert .rpm's to .deb's. (That of course necessitates rpm since "nothing but rpm can parse RPM". -- Moshe Zadka There is no IGLU cabal. http://advogato.org/person/moshez From gward@python.net Tue Sep 5 22:37:02 2000 From: gward@python.net (Greg Ward) Date: Tue Sep 5 21:37:02 2000 Subject: [Distutils] Default format for "bdist" Message-ID: <20000905213429.A841@beelzebub> Hi all -- as no doubt many of you are aware, the default behaviour for the "bdist" command is to create a "dumb" built distribution -- a /-relative tarball on Unix, and a prefix-relative ZIP file on Windows. I have at times had vague plans for making the Unix default whatever is sensible on the current platform: an RPM, a Debian package, a Free BSD package, a Solaris package, etc. etc. I propose two changes to this: * make a dumb tarball the permanent and everlasting default for Unix: I don't want to write code that attempts to figure out if the current system is RPM-based or Debian-based or uses some other packaging tool (and I'm not keen on adding Marc-André's über-"grok this platform" module -- it's just too big) * make "wininst" the default format on Windows; I'm not aware of any problems with this spiffy little self-installing ZIP file generator, so why the heck not make it the default? Any opinions? If not, I'll go ahead and make this change and check it in as soon as the Python 2.0b1 code freeze is lifted (err, if it isn't already...) Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ If ignorance is bliss, why aren't there more happy people? From jeremy@beopen.com Tue Sep 5 22:40:01 2000 From: jeremy@beopen.com (Jeremy Hylton) Date: Tue Sep 5 21:40:01 2000 Subject: [Distutils] Default format for "bdist" In-Reply-To: <20000905213429.A841@beelzebub> References: <20000905213429.A841@beelzebub> Message-ID: <14773.41332.688868.725091@bitdiddle.concentric.net> >>>>> "GW" == Greg Ward writes: GW> Any opinions? If not, I'll go ahead and make this change and GW> check it in as soon as the Python 2.0b1 code freeze is lifted GW> (err, if it isn't already...) Have at it, Greg! We've already built the tar ball, the Windows installer, and the RPM. We're just finishing up the HTML now. Jeremy From mal@lemburg.com Wed Sep 6 07:28:02 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed Sep 6 06:28:02 2000 Subject: [Distutils] Default format for "bdist" References: <20000905213429.A841@beelzebub> Message-ID: <39B5FF79.CD45EF09@lemburg.com> Greg Ward wrote: > > Hi all -- > > as no doubt many of you are aware, the default behaviour for the "bdist" > command is to create a "dumb" built distribution -- a /-relative tarball > on Unix, and a prefix-relative ZIP file on Windows. I have at times had > vague plans for making the Unix default whatever is sensible on the > current platform: an RPM, a Debian package, a Free BSD package, a > Solaris package, etc. etc. > > I propose two changes to this: > > * make a dumb tarball the permanent and everlasting default for Unix: > I don't want to write code that attempts to figure out if the > current system is RPM-based or Debian-based or uses some other > packaging tool (and I'm not keen on adding Marc-André's über-"grok > this platform" module -- it's just too big) Ok, but could you then at least make the archives platform name an option then ? I wrote platform.py for just this reason (it is used in the mxCGIPython project) and would of course like to reuse its functionality for distutils too. Perhaps you could make distutils use platform.py on an availability basis: try to import it and fall back to sys.platform otherwise. > * make "wininst" the default format on Windows; I'm not aware of any > problems with this spiffy little self-installing ZIP file generator, > so why the heck not make it the default? It's cool, yes... but it didn't work for me (I seem to be the unlucky guy in this forum ;-() with Tim's Python 2.0 pre-release: it says that it can't find a usable Python installation and the [...] button doesn't allow me to select a proper target. I'll try again with the final 1.6 Windows version. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From thomas.heller@ion-tof.com Wed Sep 6 09:09:01 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Wed Sep 6 08:09:01 2000 Subject: Status of bdist_wininst (Was: Re: [Distutils] Default format for "bdist") References: <20000905213429.A841@beelzebub> <39B5FF79.CD45EF09@lemburg.com> Message-ID: <00f801c017fb$1a111090$4500a8c0@thomasnb> Marc-Andre wrote: > > > > > * make "wininst" the default format on Windows; I'm not aware of any > > problems with this spiffy little self-installing ZIP file generator, > > so why the heck not make it the default? > > It's cool, yes... but it didn't work for me (I seem to be the unlucky > guy in this forum ;-() with Tim's Python 2.0 pre-release: it says > that it can't find a usable Python installation and the [...] > button doesn't allow me to select a proper target. > > I'll try again with the final 1.6 Windows version. > The status of bdist_wininst in the 1.6 and 2.0b distributions is a little bit messy. This is partly because I have been very busy with private things over the last four weeks. My apologies for this. 1.6 final has distutils version 0.9.1. bdist_wininst still uses the old GUI, and only (hardcoded) knows about the 1.5 and 1.6 python versions in the registy. It also has some bugs which were fixed in the meantime: 'setup.py bdist_wininst' fails when the 'dist' directory does not exist. 2.0b1 has distutils version 0.9.2. bdist_wininst uses the new GUI, no hardcoded python version strings any more, no requirement for zlib. Seems to work fine, although some features are missing: - does not take care of the 'extra_path' argument. - does not compile pyc and/or pyo files. I have prepared a new version, which I will check in as soon as I get ssh installed and working correctly. Thomas From gward@python.net Thu Sep 7 00:02:06 2000 From: gward@python.net (Greg Ward) Date: Wed Sep 6 23:02:06 2000 Subject: [Distutils] Updated Distutils docs Message-ID: <20000906225943.A1266@beelzebub> Hi all -- I've been slowly plugging away at the "Distributing Python Modules" manual over the last couple of days, and just uploaded the results to python.org. See http://www.python.org/sigs/distutils-sig/doc/ Especially the following sections of the "Distributing" manual: http://www.python.org/sigs/distutils-sig/doc/dist/describing-extensions.html http://www.python.org/sigs/distutils-sig/doc/dist/setup-config.html http://www.python.org/sigs/distutils-sig/doc/dist/source-dist.html http://www.python.org/sigs/distutils-sig/doc/dist/built-dist.html The "Installing Python Modules" manual has also been updated from the old copy that was on python.org, but this is mainly typo fixes (for now). Any feedback is welcome! Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From thomas.heller@ion-tof.com Thu Sep 7 13:07:01 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Thu Sep 7 12:07:01 2000 Subject: [Distutils] bdist_wininst updated in CVS repository Message-ID: <038201c018e5$8da0ffd0$4500a8c0@thomasnb> I just checked in a new version of bdist_wininst. Changes: distutils/command/bdist_wininst.py: - the windows installer is again able to compile after installing the files. Note: The default has changed, the packager has to give --no-target-compile/--no-target-optimize to NOT compile on the target system. (Another note: install_lib's --compile --optimize options have the same semantics to switch off the compilation. Shouldn't the names change?) - All references to specific python versions are gone. - A small bug: raise DistutilsPlatformError ("...") instead of raise DistutilsPlatformError, ("...") - When bdist_wininst creates an installer for one specific python version, this is reflected in the name: Distutils-0.9.2.win32-py15.exe instead of Distutils-0.9.2.win32.exe - bdist_wininst, when run as script, reads the wininst.exe file and rewrites itself. Previously this was done by hand. misc/install.c - All the changes needed for compilation - Deleted a lot of debug/dead code Thomas Heller From akuchlin@mems-exchange.org Thu Sep 7 13:13:03 2000 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Thu Sep 7 12:13:03 2000 Subject: [Distutils] bdist_wininst updated in CVS repository In-Reply-To: <038201c018e5$8da0ffd0$4500a8c0@thomasnb>; from thomas.heller@ion-tof.com on Thu, Sep 07, 2000 at 06:06:16PM +0200 References: <038201c018e5$8da0ffd0$4500a8c0@thomasnb> Message-ID: <20000907121219.B15388@kronos.cnri.reston.va.us> On Thu, Sep 07, 2000 at 06:06:16PM +0200, Thomas Heller wrote: >- A small bug: > raise DistutilsPlatformError ("...") > instead of > raise DistutilsPlatformError, ("...") Are you sure that is in fact a bug? If DistutilsPlatformError is a class-based exception, the first form is quite legal. --amk From jeremy@beopen.com Thu Sep 7 13:17:01 2000 From: jeremy@beopen.com (Jeremy Hylton) Date: Thu Sep 7 12:17:01 2000 Subject: [Distutils] bdist_wininst updated in CVS repository In-Reply-To: <20000907121219.B15388@kronos.cnri.reston.va.us> References: <038201c018e5$8da0ffd0$4500a8c0@thomasnb> <20000907121219.B15388@kronos.cnri.reston.va.us> Message-ID: <14775.49313.587137.35684@bitdiddle.concentric.net> >>>>> "AMK" == Andrew Kuchling writes: AMK> On Thu, Sep 07, 2000 at 06:06:16PM +0200, Thomas Heller wrote: >>- A small bug: >> raise DistutilsPlatformError ("...") >> instead of >> raise DistutilsPlatformError, ("...") AMK> Are you sure that is in fact a bug? If DistutilsPlatformError AMK> is a class-based exception, the first form is quite legal. Except that the space between DisutilsPlatformError and the open paren is bad Python style. Jeremy From fdrake@beopen.com Thu Sep 7 13:19:00 2000 From: fdrake@beopen.com (Fred L. Drake, Jr.) Date: Thu Sep 7 12:19:00 2000 Subject: [Distutils] bdist_wininst updated in CVS repository In-Reply-To: <20000907121219.B15388@kronos.cnri.reston.va.us> References: <038201c018e5$8da0ffd0$4500a8c0@thomasnb> <20000907121219.B15388@kronos.cnri.reston.va.us> Message-ID: <14775.49024.689788.673025@cj42289-a.reston1.va.home.com> Andrew Kuchling writes: > On Thu, Sep 07, 2000 at 06:06:16PM +0200, Thomas Heller wrote: > >- A small bug: > > raise DistutilsPlatformError ("...") > > instead of > > raise DistutilsPlatformError, ("...") > > Are you sure that is in fact a bug? If DistutilsPlatformError is a > class-based exception, the first form is quite legal. This is a style issue. The first form is normally recommended for exceptions that are always classes. But the spaces between the name of something callable and the opening parenthesis of the parameter list should go -- that's in Guido's style "essay". -Fred -- Fred L. Drake, Jr. BeOpen PythonLabs Team Member From robin@jessikat.fsnet.co.uk Sat Sep 9 12:23:02 2000 From: robin@jessikat.fsnet.co.uk (Robin Becker) Date: Sat Sep 9 11:23:02 2000 Subject: [Distutils] win32 default installation Message-ID: Is there any reason why the default directory for .pyds seems to be the base python install dir rather than the DLLs directory underneath it? Normally .lib files would go to say c:\Python\Libs and the corresponding .pyd would go to c:\Python\DLLs I have a pure C extension and it therefore has no natural package home so I would normally put that into DLLs, but it seems to be copied into the base dir. -- Robin Becker From gward@python.net Sat Sep 9 13:32:03 2000 From: gward@python.net (Greg Ward) Date: Sat Sep 9 12:32:03 2000 Subject: [Distutils] First steps with distutils... In-Reply-To: <20000905040326.B4702@inet.net>; from hgebel@conectiv.net on Tue, Sep 05, 2000 at 04:03:27AM -0400 References: <39AFC048.6E963F82@lemburg.com> <39B0CD44.6B79F246@lemburg.com> <20000902104944.B1221@beelzebub> <39B38398.1B4079D3@lemburg.com> <20000904155056.B1887@beelzebub> <39B41915.5BA7B56C@lemburg.com> <20000904224526.A2683@beelzebub> <39B4A6B0.3FAB1C21@lemburg.com> <20000905040326.B4702@inet.net> Message-ID: <20000909123012.A12352@beelzebub> On 05 September 2000, Harry Henry Gebel said: > The contents of the spec file are included in the source RPM, that is why A > think that using sys.executable should be the option and not the default. Dammit, I'm still waiting for a light to flash on so I can understand where you guys are coming from on this. Please indulge me as I think out loud... Assertions (please tell me if I'm misunderstanding something): * this isn't really all *that* important, as the build instructions in the .spec file only apply to people building the source RPM -- IOW, the build instructions in the .spec file in no way affect the majority of people who use the RPM, ie. those who just install the "built" RPM * in fact, the only person who *must* build the source RPM is the person who creates the RPM in the first place (although anyone creating binary RPMs for other distributions or other architectures would probably start from the source RPM) I've been thinking hard about this -- hey, my brain is slow this weekend -- and I think I understand it a little better. Case 1 (the status quo): put "python setup.py ..." in the build/install/ clean instructions in the .spec file. This is bad when the packager, P, uses a "non-standard" python (anything other than /usr/bin/python) to create an RPM that is intended to go in the non-standard location. This is mainly a problem when P immediately creates a built RPM from his new source RPM (the usual case, in fact); if a builder, B, turns source RPM -> built RPM on a separate system, then using the first python in the path -- most likely /usr/bin/python -- might well be the right thing to do. Case 2: put sys.executable + " setup.py ..." in the .spec file. This fixes the above problem, but is bad in the case where P accidentally uses a non-standard python to create an RPM that is supposed to go to the standard python installation (/usr). Eg. if I forget that /usr/local/bin/python is first in my path, then any source RPMs I create will refer to /usr/local/bin/python in the .spec file, and building those source RPMs will either fail (on systems that don't have /usr/local/bin/python, probably the vast majority of installed Linux boxen out there) or will generate an RPM with the "non-standard" destination of /usr/local. In the latter case -- accidentally using the wrong python -- it would be best if the Distutil issued a warning. I don't see an easy way to detect this, especially if someone is deliberately creating an RPM to install modules to a non-standard location. (Eg. Andrew's case of having python in /www/python/bin/python: he might want to make RPMs of common modules to install on all the web developers' workstations in /www/python.) Both situations are subtle errors on the packager's part, and neither seem to have obvious automatic solutions. The "fix" is social engineering: let the packager decide what he wants to do with options to the bdist_rpm command, and make sure the rationale for each option is carefully documented. The question is: which failure is more obvious and happens closest to the packager, the one who made the mistake (failed to RTFM, etc.)? That one should be the default. It seems to me that creating a source RPM (spec file) that immediately fails because of an explicit /path/to/python is better than one that works, but possibly wrongly, because of implicitly using the first python on the path. IOW, "Explicit is better than implicit", even in snippets of shell code included in a .spec file bundled in a source RPM. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ "I came, I saw, she conquered." (The original Latin seems to have been garbled.) From gward@python.net Sat Sep 9 13:34:01 2000 From: gward@python.net (Greg Ward) Date: Sat Sep 9 12:34:01 2000 Subject: [Distutils] First steps with distutils... In-Reply-To: <20000905040326.B4702@inet.net>; from hgebel@conectiv.net on Tue, Sep 05, 2000 at 04:03:27AM -0400 References: <39AFC048.6E963F82@lemburg.com> <39B0CD44.6B79F246@lemburg.com> <20000902104944.B1221@beelzebub> <39B38398.1B4079D3@lemburg.com> <20000904155056.B1887@beelzebub> <39B41915.5BA7B56C@lemburg.com> <20000904224526.A2683@beelzebub> <39B4A6B0.3FAB1C21@lemburg.com> <20000905040326.B4702@inet.net> Message-ID: <20000909123214.A12374@beelzebub> On 05 September 2000, Harry Henry Gebel said: > The contents of the spec file are included in the source RPM, that is why A > think that using sys.executable should be the option and not the default. But it only affects people building from the source RPM, which is usually done by the same person as created the source RPM and on the same system. See my last post for details. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Sat Sep 9 13:39:02 2000 From: gward@python.net (Greg Ward) Date: Sat Sep 9 12:39:02 2000 Subject: [Distutils] win32 default installation In-Reply-To: ; from robin@jessikat.fsnet.co.uk on Sat, Sep 09, 2000 at 04:20:32PM +0100 References: Message-ID: <20000909123742.B12374@beelzebub> On 09 September 2000, Robin Becker said: > Is there any reason why the default directory for .pyds seems to be the > base python install dir rather than the DLLs directory underneath it? Because of this code in install.py: INSTALL_SCHEMES = { ... 'nt': { 'purelib': '$base', 'platlib': '$base', 'headers': '$base\\Include\\$dist_name', 'scripts': '$base\\Scripts', 'data' : '$base', }, ... } Which is based on what I understood about Python-on-Windows after grilling Guido mercilessly for several hours one day last summer. That is, third-party libraries -- modules and extensions both -- go straight to the dir. (It's spelled "base" here because the base library dir == prefix only on Windows and Mac OS; on Unix, "base" is /lib/pythonX.Y.) Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From mal@lemburg.com Sat Sep 9 14:05:00 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat Sep 9 13:05:00 2000 Subject: [Distutils] First steps with distutils... References: <39AFC048.6E963F82@lemburg.com> <39B0CD44.6B79F246@lemburg.com> <20000902104944.B1221@beelzebub> <39B38398.1B4079D3@lemburg.com> <20000904155056.B1887@beelzebub> <39B41915.5BA7B56C@lemburg.com> <20000904224526.A2683@beelzebub> <39B4A6B0.3FAB1C21@lemburg.com> <20000905040326.B4702@inet.net> <20000909123012.A12352@beelzebub> Message-ID: <39BA6D52.51D1CF58@lemburg.com> Greg Ward wrote: > > On 05 September 2000, Harry Henry Gebel said: > > The contents of the spec file are included in the source RPM, that is why A > > think that using sys.executable should be the option and not the default. > > Dammit, I'm still waiting for a light to flash on so I can understand > where you guys are coming from on this. Please indulge me as I think > out loud... > > Assertions (please tell me if I'm misunderstanding something): > > * this isn't really all *that* important, as the build instructions in > the .spec file only apply to people building the source RPM -- IOW, > the build instructions in the .spec file in no way affect the > majority of people who use the RPM, ie. those who just install the > "built" RPM > > * in fact, the only person who *must* build the source RPM is > the person who creates the RPM in the first place (although > anyone creating binary RPMs for other distributions or other > architectures would probably start from the source RPM) > > I've been thinking hard about this -- hey, my brain is slow this weekend > -- and I think I understand it a little better. > > Case 1 (the status quo): put "python setup.py ..." in the build/install/ > clean instructions in the .spec file. This is bad when the packager, > P, uses a "non-standard" python (anything other than /usr/bin/python) > to create an RPM that is intended to go in the non-standard location. > This is mainly a problem when P immediately creates a built RPM from > his new source RPM (the usual case, in fact); if a builder, B, turns > source RPM -> built RPM on a separate system, then using the first > python in the path -- most likely /usr/bin/python -- might well be the > right thing to do. > > Case 2: put sys.executable + " setup.py ..." in the .spec file. > This fixes the above problem, but is bad in the case where P > accidentally uses a non-standard python to create an RPM that > is supposed to go to the standard python installation (/usr). > Eg. if I forget that /usr/local/bin/python is first in my path, > then any source RPMs I create will refer to /usr/local/bin/python > in the .spec file, and building those source RPMs will either > fail (on systems that don't have /usr/local/bin/python, probably > the vast majority of installed Linux boxen out there) or will > generate an RPM with the "non-standard" destination of /usr/local. > > In the latter case -- accidentally using the wrong python -- it would be > best if the Distutil issued a warning. I don't see an easy way to > detect this, especially if someone is deliberately creating an RPM to > install modules to a non-standard location. (Eg. Andrew's case of > having python in /www/python/bin/python: he might want to make RPMs of > common modules to install on all the web developers' workstations in > /www/python.) > > Both situations are subtle errors on the packager's part, and neither > seem to have obvious automatic solutions. The "fix" is social > engineering: let the packager decide what he wants to do with options to > the bdist_rpm command, and make sure the rationale for each option is > carefully documented. > > The question is: which failure is more obvious and happens closest to > the packager, the one who made the mistake (failed to RTFM, etc.)? That > one should be the default. It seems to me that creating a source RPM > (spec file) that immediately fails because of an explicit > /path/to/python is better than one that works, but possibly wrongly, > because of implicitly using the first python on the path. > > IOW, "Explicit is better than implicit", even in snippets of shell code > included in a .spec file bundled in a source RPM. Why all the fuzz ? ;-) Keep the status quo and add an option for those who want to play with different Python versions, e.g. --set-executable=/home/lemburg/tmp/betas/python2.0 This will make everybody happy and hide the feature well enough to not cause frequent RTFM replies. Those who need it will find it and those who can't find it won't need it ;-) KISS, -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From thomas.heller@ion-tof.com Sat Sep 9 15:28:01 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Sat Sep 9 14:28:01 2000 Subject: [Distutils] win32 default installation References: <20000909123742.B12374@beelzebub> Message-ID: <001701c01a8b$a4680270$4500a8c0@thomasnb> > On 09 September 2000, Robin Becker said: > > Is there any reason why the default directory for .pyds seems to be the > > base python install dir rather than the DLLs directory underneath it? > [Greg] > Because of this code in install.py: > > INSTALL_SCHEMES = { > ... > 'nt': { > 'purelib': '$base', > 'platlib': '$base', > 'headers': '$base\\Include\\$dist_name', > 'scripts': '$base\\Scripts', > 'data' : '$base', > }, > ... > } > > Which is based on what I understood about Python-on-Windows after > grilling Guido mercilessly for several hours one day last summer. That > is, third-party libraries -- modules and extensions both -- go straight > to the dir. (It's spelled "base" here because the base library > dir == prefix only on Windows and Mac OS; on Unix, "base" is > /lib/pythonX.Y.) > One problem with this is the following (for modules/packages, not only for .pyd extensions): distutils (on windows), as supplied in python 1.6 and 2.0, is installed in \lib. If I download a new distutils distribution and run 'setup.py install' it will install the new version in . The default python path on windows is something like ['', '\\DLLs', '\\lib', '\\libplat-win', '\\lib\\lib-tk', '']. So the old version in '\\lib', which should replace the old version in '', will be found first on the python-path, which is clearly wrong. What should we do here? Try to import the old package/module, and use their __path__ as the destination directory? Thomas From mwa@gate.net Sat Sep 9 21:25:01 2000 From: mwa@gate.net (Mark W. Alexander) Date: Sat Sep 9 20:25:01 2000 Subject: [Distutils] First steps with distutils... In-Reply-To: <20000909123012.A12352@beelzebub> Message-ID: On Sat, 9 Sep 2000, Greg Ward wrote: > > On 05 September 2000, Harry Henry Gebel said: > > The contents of the spec file are included in the source RPM, that is why A > > think that using sys.executable should be the option and not the default. > > Dammit, I'm still waiting for a light to flash on so I can understand > where you guys are coming from on this. Please indulge me as I think > out loud... Making packages is not that hard. Making GOOD packages is very subtly tricky. My comments below may help your understanding. They may not help find a solution.... > Assertions (please tell me if I'm misunderstanding something): > > * this isn't really all *that* important, as the build instructions in > the .spec file only apply to people building the source RPM -- IOW, > the build instructions in the .spec file in no way affect the > majority of people who use the RPM, ie. those who just install the > "built" RPM Correct. > * in fact, the only person who *must* build the source RPM is > the person who creates the RPM in the first place (although > anyone creating binary RPMs for other distributions or other > architectures would probably start from the source RPM) Not exactly. No one ever *must* build a source RPM. If you have a .spec and a source tar/directory you can build a binary RPM without creating a source RPM. Distribution of the binary RPM satisfies 99.99% of the people who want to install an RPM. Those who want to tweak how the binary is created could use the source RPM, but even then (especially with pure python packages) it's not required. I've "rebuilt" RPM's by getting a source tarball and recreating most of the .spec from an rpm -qi query. > I've been thinking hard about this -- hey, my brain is slow this weekend > -- and I think I understand it a little better. > > Case 1 (the status quo): put "python setup.py ..." in the build/install/ > clean instructions in the .spec file. This is bad when the packager, > P, uses a "non-standard" python (anything other than /usr/bin/python) > to create an RPM that is intended to go in the non-standard location. > This is mainly a problem when P immediately creates a built RPM from > his new source RPM (the usual case, in fact); if a builder, B, turns > source RPM -> built RPM on a separate system, then using the first > python in the path -- most likely /usr/bin/python -- might well be the > right thing to do. Close. It really has nothing to do with the source RPM. It's just the usual RPM way of package building. I prefer to "force" my rpm installs (the fake install done by rpm -b during the install step) into a directory OTHER than where it will install on the target machine. Benefits are 1) I can create a binary RPM on a system where I don't hace root access (as if I'd work there...), and 2) you don't step on your production files untill you are satisfied with the RPM. This has the added advantage in that you can now make the RPM relocatable with a little more effort in the .spec file. To make the package relocatable, you capture the library path where you want it to go (presumably the site-packages directory of the default python, but it doesn't have to be) to the location where you just did the fake RPM install. RPM then works it's magic by creating the package from the files in the fake tree so that they will be installed by default into the real site-packages directory. Since the package is relocatable now, the installer could override it to go anywhere. This is really cool, in that now the installer doesn't have to be root. Instead, they can relocate the package anywhere they can write and just add that path to their PYTHONPATH. > Case 2: put sys.executable + " setup.py ..." in the .spec file. > This fixes the above problem, but is bad in the case where P > accidentally uses a non-standard python to create an RPM that > is supposed to go to the standard python installation (/usr). > Eg. if I forget that /usr/local/bin/python is first in my path, > then any source RPMs I create will refer to /usr/local/bin/python > in the .spec file, and building those source RPMs will either > fail (on systems that don't have /usr/local/bin/python, probably > the vast majority of installed Linux boxen out there) or will > generate an RPM with the "non-standard" destination of /usr/local. There's a little more confusion here, and I'm not exactly sure how it works with bdist_rpm. RPM does some additional hacking beyond what's in the .spec file. It looks especially for #! magic in package files but also for referenced libraries and anything else it thinks the package is going to need. RPM then (theoritcally, in a helpful fashion) adds these things to the "Requires:" tag in the binary package. SO, if it identifies /usr/local/bin/python as a requirement for the package, it will not install (without --force) on a target system that does not have /usr/local/bin/python. (Yes, I have done this...the original pygresql package went out this way...sigh). I don't think RPM deduces requires from processors used in the setup, config, build, install steps, but if the packager is using a /usr/local/bin python, it's very likely to be references somewhere in the package, too. > In the latter case -- accidentally using the wrong python -- it would be > best if the Distutil issued a warning. I don't see an easy way to > detect this, especially if someone is deliberately creating an RPM to > install modules to a non-standard location. (Eg. Andrew's case of > having python in /www/python/bin/python: he might want to make RPMs of > common modules to install on all the web developers' workstations in > /www/python.) This is why I'm struggling with pkgtool (and even worse, sdux). If the bdist_whatever's make the package relocatable, it doesn't matter what python was used in the build (location-wise anyway). Then the issue shift's to installation time. How do you determin the TARGET system's "prefered" python? This could be retrieved either by executing python or querying RPM in a preinstall step, then automatically reloacting the package where it should go. Except you also have to be prepared to skip the auto-relocation if the installer is relocating it with RPM options. > Both situations are subtle errors on the packager's part, and neither > seem to have obvious automatic solutions. The "fix" is social > engineering: let the packager decide what he wants to do with options to > the bdist_rpm command, and make sure the rationale for each option is > carefully documented. Like I said...Packaging is easy. Making easy packages are hard. I'd vote for making bdist_rpm make things relocatable with a default pre-install to do auto-relocation IF I still had any sanity from actually trying to do this with other package types. Maybe I'm too anal about my packages, but good ones make it sooooo much easier to administer large number of machines. > The question is: which failure is more obvious and happens closest to > the packager, the one who made the mistake (failed to RTFM, etc.)? That > one should be the default. It seems to me that creating a source RPM > (spec file) that immediately fails because of an explicit > /path/to/python is better than one that works, but possibly wrongly, > because of implicitly using the first python on the path. Agreed, except that the source RPM is still only going to be used by people who are already prepared to get in and hack at it anyway. > IOW, "Explicit is better than implicit", even in snippets of shell code > included in a .spec file bundled in a source RPM. Hey, you're not trying to suggest that maybe, possible, ONE thing from bdist_pkgtool was almost, kinda, maybe on the right track ;-) Anyway I said this probably wouldn't be a solution, but I hope I clarified the problem al little.... mwa From gward@python.net Sat Sep 9 21:59:02 2000 From: gward@python.net (Greg Ward) Date: Sat Sep 9 20:59:02 2000 Subject: [Distutils] First steps with distutils... In-Reply-To: <39BA6D52.51D1CF58@lemburg.com>; from mal@lemburg.com on Sat, Sep 09, 2000 at 07:03:14PM +0200 References: <39B0CD44.6B79F246@lemburg.com> <20000902104944.B1221@beelzebub> <39B38398.1B4079D3@lemburg.com> <20000904155056.B1887@beelzebub> <39B41915.5BA7B56C@lemburg.com> <20000904224526.A2683@beelzebub> <39B4A6B0.3FAB1C21@lemburg.com> <20000905040326.B4702@inet.net> <20000909123012.A12352@beelzebub> <39BA6D52.51D1CF58@lemburg.com> Message-ID: <20000909201949.A12708@beelzebub> On 09 September 2000, M.-A. Lemburg said: > Why all the fuzz ? ;-) 'cause I spent waaaay too much time Friday night and Saturday morning worrying about this extremely trivial issue, and I wanted the rest of the SIG to share my pain. So there! > Keep the status quo and add an option for those who want to > play with different Python versions, e.g. > --set-executable=/home/lemburg/tmp/betas/python2.0 Done. Here's the help text from "bdist_rpm --help": --python path to Python interpreter to hard-code in the .spec file (default: "python") --fix-python hard-code the exact path to the current Python interpreter in the .spec file Seem reasonable? There's no opposite to --fix-python, because you can go back to the default with "--python=python". Should be checked in by the time you read this... Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Sat Sep 9 21:59:07 2000 From: gward@python.net (Greg Ward) Date: Sat Sep 9 20:59:07 2000 Subject: [Distutils] Rewritten 'get_platform()' Message-ID: <20000909200139.A12684@beelzebub> Hi all -- for months I have resisted the urge to turn the 'util.get_platform()' function into a miniaturized version of Marc-André's platform.py or the GNU script that spits out the old familar "os-vendor-hardware" triple. Well, today I succumbed. The idea is to generate a string that does a better job of describing the current platform than sys.platform or os.name. This has (so far) two purposes in the distutils: naming the build directories where platform-specific files (.o, .so, etc.) go, and naming built distributions. IMHO the exact information to include depends on the OS: eg. for Linux, the exact kernel version isn't very relevant, whereas for IRIX the version is crucial but the hardware isn't. Typically, though, OS, OS version, and hardware are all relevant. This information is, AFAIK, always available from the 'uname()' system call, exposed in Python as 'os.uname()'. However, it typically needs a bit of massaging to become something that we all know and recognize: eg. "SunOS 5.x" is really "Solaris 2.x". So, here's my initial crack at a rewritten 'get_platform()'. Currently it only handles POSIX platforms; on Windows and Mac OS it'll just return sys.platform (as it has always done). ------------------------------------------------------------------------ def get_platform (): """Return a string that identifies the current platform. This is used mainly to distinguish platform-specific build directories and platform-specific built distributions. Typically includes the OS name and version and the architecture (as supplied by 'os.uname()'), although the exact information included depends on the OS; eg. for IRIX the architecture isn't particularly important (IRIX only runs on SGI hardware), but for Linux the kernel version isn't particularly important. Examples of returned values: linux-i586 linux-alpha (?) solaris-2.6-sun4u irix-5.3 irix64-6.2 For non-POSIX platforms, currently just returns 'sys.platform'. """ if os.name != "posix": # XXX what about the architecture? NT is Intel or Alpha, # Mac OS is M68k or PPC, etc. return sys.platform # Try to distinguish various flavours of Unix (os, host, release, version, machine) = os.uname() os = string.lower(os) if os[:5] == "linux": # At least on Linux/Intel, 'machine' is the processor -- # i386, etc. # XXX what about Alpha, SPARC, etc? return "%s-%s" % (os, machine) elif os[:5] == "sunos": if release[0] >= "5": # SunOS 5 == Solaris 2 os = "solaris" release = "%d.%s" % (int(release[0]) - 3, release[2:]) # fall through to standard os-release-machine representation elif os[:4] == "irix": # could be "irix64"! return "%s-%s" % (os, release) return "%s-%s-%s" % (os, release, machine) ------------------------------------------------------------------------ There are a heck of a lot of POSIX-ish platforms out there, and I'll be very happy if this function returns something sensible for the following: * Linux on any architecture * {Free,Net,Open}BSD on any architecture * Solaris on SPARC, Intel * IRIX * AIX * HP-UX ...plus of course Windows and Mac OS. Like the rest of the world, I'm a little unclear on just how many versions of Windows there really are. ;-) I suspect we can get away with Win32 vs. Win64, but what about NT on Alpha -- is that relevant? Are there any major Unices that I'm missing here? As long as the os-release-machine triple is sensible, it's OK to leave them out of the switch on OS name up there, but I suspect most will need some special-case tweaking like the three that I'm familar with. And yes, I'm going to go dig up MAL's platform.py and see what it does. My informal rule is that, at least for POSIX-ish platforms, we should rely solely on os.uname(). It remains to be seen how foolishly optimistic that is. ;-) Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ War is Peace; Freedom is Slavery; Ignorance is Knowledge From robin@jessikat.fsnet.co.uk Sat Sep 9 22:00:02 2000 From: robin@jessikat.fsnet.co.uk (Robin Becker) Date: Sat Sep 9 21:00:02 2000 Subject: [Distutils] win32 default installation In-Reply-To: <20000909123742.B12374@beelzebub> References: <20000909123742.B12374@beelzebub> Message-ID: In article <20000909123742.B12374@beelzebub>, Greg Ward writes ... >Which is based on what I understood about Python-on-Windows after >grilling Guido mercilessly for several hours one day last summer. That >is, third-party libraries -- modules and extensions both -- go straight >to the dir. (It's spelled "base" here because the base library >dir == prefix only on Windows and Mac OS; on Unix, "base" is >/lib/pythonX.Y.) .... In this case I fear the BDFL is inaccurate. zlib.pyd et al don't go into $base they go into $base/DLLs so if the BDFL were to start distributing newer versions of zlib etc these would end up in the wrong location. I thing the powers that be have got it wrong for this instance. These singleton .pyd packages seem better off in DLLs to me anyway. Anyhow if I were to desire the .pyd to end up in $base/DLLs how do I make it do that under win32; must I mess with the defaults in my setup.py? -- Robin Becker From gward@python.net Sat Sep 9 22:13:02 2000 From: gward@python.net (Greg Ward) Date: Sat Sep 9 21:13:02 2000 Subject: [Distutils] win32 default installation In-Reply-To: <001701c01a8b$a4680270$4500a8c0@thomasnb>; from thomas.heller@ion-tof.com on Sat, Sep 09, 2000 at 08:27:32PM +0200 References: <20000909123742.B12374@beelzebub> <001701c01a8b$a4680270$4500a8c0@thomasnb> Message-ID: <20000909211136.B12708@beelzebub> On 09 September 2000, Thomas Heller said: > One problem with this is the following (for modules/packages, > not only for .pyd extensions): > distutils (on windows), as supplied in python 1.6 and 2.0, > is installed in \lib. > If I download a new distutils distribution and run 'setup.py install' > it will install the new version in . Same problem on Unix, and same problem for any modules in the standard library that also have a "life of their own" -- distutils, asyncore, who knows what else. I wanted to sneak in a feature that would let you upgrade bits of the standard library, but Guido wouldn't let me. ;-( My recollection of our BDFL's reasoning: the Python 1.6 (or 1.5, or 2.0, or whatever) standard library IS the Python 1.6 (or 1.5, or 2.0, or whatever) standard library, and I'm not supposed to muck with that. Period. Perl's MakeMaker supports this with the "install_dirs=perl" option (default is "install_dirs=site"). I think this is because most of the Perl standard library grew out of popular and important CPAN modules that all have lives of their own. This phenomenon seems to be less widespread in Python-land. Note that the Distutils README.txt does address the issue: """ INSTALLATION UNDER PYTHON 1.6/2.0 --------------------------------- The Distutils have been included with Python since 1.6a1, and Distutils 0.9.2 is the same as the code included with Python 2.0b1 (modulo bug fixes). Thus, there's generally no need to install the Distutils under Python 1.6/2.0. However, Distutils releases may occasionally get ahead of Python releases, so if you really like life on the bleeding edge, you might want to install this Distutils release into your Python 1.6/2.0 library. To do this, you'll need to hide the original Distutils package directory from Python, so it will find the version you install. For example, under a typical Unix installation, the "stock" Distutils directory is /usr/local/lib/python1.6/distutils; you could hide this from Python as follows: cd /usr/local/lib/python1.6 # or 2.0 mv distutils distutils-orig On Windows, the stock Distutils installation is "Lib\distutils" under the Python directory ("C:\Python" by default with Python 1.6a2 and later). Again, you should just rename this directory, eg. to "distutils-orig", so that Python won't find it. Once you have renamed the stock Distutils directory, you can install the Distutils as described above. """ Correct? Sure. Satisfactory? Not really... > What should we do here? Try to import the old package/module, > and use their __path__ as the destination directory? As I implied above, my preferred solution would be an option in the setup script, eg. "install_dirs = 'standard'" (default 'site'), so that certain module distributions could declare themselves as being part of the standard library. But that was shot down, and I think your idea is even more likely to be Frowned Upon. (Strikes me as a bit too sneaky and subtle.) Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Sat Sep 9 22:21:04 2000 From: gward@python.net (Greg Ward) Date: Sat Sep 9 21:21:04 2000 Subject: [Distutils] win32 default installation In-Reply-To: ; from robin@jessikat.fsnet.co.uk on Sun, Sep 10, 2000 at 01:58:12AM +0100 References: <20000909123742.B12374@beelzebub> Message-ID: <20000909211939.C12708@beelzebub> On 10 September 2000, Robin Becker said: > In this case I fear the BDFL is inaccurate. zlib.pyd et al don't go into > $base they go into $base/DLLs because they are part of the standard library. Standard library modules do not get installed to the same location as third-party, site-specific modules on any Python platform. > so if the BDFL were to start distributing > newer versions of zlib etc these would end up in the wrong location. Correct. I think this is (currently) more a problem for modules that start out as not part of the standard library and maintain a life of their own, but it could become a problem if bits of the standard library start to be maintained separately. > I thing the powers that be have got it wrong for this instance. These > singleton .pyd packages seem better off in DLLs to me anyway. Not if they're third-party modules, they don't! I agree that dumping third-party modules right into is, umm, bogus. But that's the way Python's sys.path is on Windows, so what are ya gonna do? > Anyhow if I were to desire the .pyd to end up in $base/DLLs how do I > make it do that under win32; must I mess with the defaults in my > setup.py? Try this: python setup.py install --install-platlib='$sys_prefix/DLLs' Hmm, I wish I could remember why I thought "sys_prefix" was the right name for that variable (instead of just "prefix"). Oh well. Note that $sys_prefix is expanded by the Distutils "install" command, so if you're using a shell that expands variables in the Unix way (say, bash under Cygwin), you need the single quotes! Greg PS. to the person who asked some time ago why I used the "$var" syntax instead of "%(var)s": now do you buy it? ;-) -- Greg Ward gward@python.net http://starship.python.net/~gward/ From dubois@users.sourceforge.net Mon Sep 11 01:48:01 2000 From: dubois@users.sourceforge.net (Paul F. Dubois) Date: Mon Sep 11 00:48:01 2000 Subject: [Distutils] Rewritten 'get_platform()' In-Reply-To: <20000909200139.A12684@beelzebub> Message-ID: Just a thought related to what you are doing here; I have had some experiences where the LOCATION mattered too. For example, a certain library was in /usr/local on one of our systems, and in someone's home directory on another, even though otherwise the systems were identical. Of course, I don't know how you could "detect" the location except as a possible configuration variable. Indeed, I did just that in one make system I wrote. > -----Original Message----- > From: distutils-sig-admin@python.org > [mailto:distutils-sig-admin@python.org]On Behalf Of Greg Ward > Sent: Saturday, September 09, 2000 5:02 PM > To: distutils-sig@python.org > Subject: [Distutils] Rewritten 'get_platform()' > > > Hi all -- > > for months I have resisted the urge to turn the 'util.get_platform()' > function into a miniaturized version of Marc-André's platform.py or the > GNU script that spits out the old familar "os-vendor-hardware" triple. > > Well, today I succumbed. The idea is to generate a string that does a > better job of describing the current platform than sys.platform or > os.name. This has (so far) two purposes in the distutils: naming the > build directories where platform-specific files (.o, .so, etc.) go, and > naming built distributions. > > IMHO the exact information to include depends on the OS: eg. for Linux, > the exact kernel version isn't very relevant, whereas for IRIX the > version is crucial but the hardware isn't. Typically, though, OS, OS > version, and hardware are all relevant. This information is, AFAIK, > always available from the 'uname()' system call, exposed in Python as > 'os.uname()'. However, it typically needs a bit of massaging to become > something that we all know and recognize: eg. "SunOS 5.x" is really > "Solaris 2.x". > > So, here's my initial crack at a rewritten 'get_platform()'. Currently > it only handles POSIX platforms; on Windows and Mac OS it'll just return > sys.platform (as it has always done). > > ------------------------------------------------------------------------ > def get_platform (): > """Return a string that identifies the current platform. This is used > mainly to distinguish platform-specific build directories and > platform-specific built distributions. Typically includes the OS name > and version and the architecture (as supplied by 'os.uname()'), > although the exact information included depends on the OS; > eg. for IRIX > the architecture isn't particularly important (IRIX only runs on SGI > hardware), but for Linux the kernel version isn't particularly > important. > > Examples of returned values: > linux-i586 > linux-alpha (?) > solaris-2.6-sun4u > irix-5.3 > irix64-6.2 > > For non-POSIX platforms, currently just returns 'sys.platform'. > """ > if os.name != "posix": > # XXX what about the architecture? NT is Intel or Alpha, > # Mac OS is M68k or PPC, etc. > return sys.platform > > # Try to distinguish various flavours of Unix > > (os, host, release, version, machine) = os.uname() > os = string.lower(os) > if os[:5] == "linux": > # At least on Linux/Intel, 'machine' is the processor -- > # i386, etc. > # XXX what about Alpha, SPARC, etc? > return "%s-%s" % (os, machine) > elif os[:5] == "sunos": > if release[0] >= "5": # SunOS 5 == Solaris 2 > os = "solaris" > release = "%d.%s" % (int(release[0]) - 3, release[2:]) > # fall through to standard os-release-machine representation > elif os[:4] == "irix": # could be "irix64"! > return "%s-%s" % (os, release) > > return "%s-%s-%s" % (os, release, machine) > ------------------------------------------------------------------------ > > There are a heck of a lot of POSIX-ish platforms out there, and I'll be > very happy if this function returns something sensible for the > following: > * Linux on any architecture > * {Free,Net,Open}BSD on any architecture > * Solaris on SPARC, Intel > * IRIX > * AIX > * HP-UX > > ...plus of course Windows and Mac OS. Like the rest of the world, I'm a > little unclear on just how many versions of Windows there really are. > ;-) I suspect we can get away with Win32 vs. Win64, but what about NT > on Alpha -- is that relevant? Are there any major Unices that I'm > missing here? As long as the os-release-machine triple is sensible, > it's OK to leave them out of the switch on OS name up there, but I > suspect most will need some special-case tweaking like the three that > I'm familar with. > > And yes, I'm going to go dig up MAL's platform.py and see what it does. > My informal rule is that, at least for POSIX-ish platforms, we should > rely solely on os.uname(). It remains to be seen how foolishly > optimistic that is. ;-) > > Greg > -- > Greg Ward gward@python.net > http://starship.python.net/~gward/ > War is Peace; Freedom is Slavery; Ignorance is Knowledge > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG@python.org > http://www.python.org/mailman/listinfo/distutils-sig > From calvin@cs.uni-sb.de Mon Sep 11 05:01:01 2000 From: calvin@cs.uni-sb.de (Bastian Kleineidam) Date: Mon Sep 11 04:01:01 2000 Subject: [Distutils] Rewritten 'get_platform()' In-Reply-To: Message-ID: Paul F. Dubois wrote: >Indeed, I did just that in one make system I wrote. I have the following solution: first I look in /usr/lib and in /usr/local/lib. If I still do not find the library, the user will need to use the --library-dirs option. Same thing for include dirs. Bastian From calvin@cs.uni-sb.de Mon Sep 11 06:20:01 2000 From: calvin@cs.uni-sb.de (Bastian Kleineidam) Date: Mon Sep 11 05:20:01 2000 Subject: [Distutils] Minor issues Message-ID: 1) Why is the --formats list comma-separated and the --include-dirs and --library-dirs lists not? They are separated by os.pathsep. Well, at least its documented :) 2) The os.pathsep separator is not documented in the help text: > python setup.py build_ext --help ... --library-dirs (-L) directories to search for external C libraries This should be: ... --library-dirs (-L) colon-separated list of directories to search for external C libraries or whatever os.pathsep is on the current platform. 3) bdist_wininst does not seem to install data files and scripts. If you execute "strings" with the .exe, you will only see Python module files. And I did not found a file with the install log. The installer just copied the files in the Python dir. Bastian From calvin@cs.uni-sb.de Mon Sep 11 06:45:01 2000 From: calvin@cs.uni-sb.de (Bastian Kleineidam) Date: Mon Sep 11 05:45:01 2000 Subject: [Distutils] Patch for clean.py to remove build_scripts directory Message-ID: In Distutils-0.9.2 the clean command does not remove the build_scripts dir. The patch is fixing this. Bastian --- clean.py Sat Aug 5 03:31:54 2000 +++ /usr/local/src/Distutils-0.9.2/distutils/command/clean.py Mon Sep 11 11:37:02 2000 @@ -20,6 +20,8 @@ "build directory for all modules (default: 'build.build-lib')"), ('build-temp=', 't', "temporary build directory (default: 'build.build-temp')"), + ('build-scripts=', None, + "build directory for scripts (default: 'build.build-scripts')"), ('bdist-base=', None, "temporary directory for built distributions"), ('all', 'a', @@ -30,6 +32,7 @@ self.build_base = None self.build_lib = None self.build_temp = None + self.build_scripts = None self.bdist_base = None self.all = None @@ -37,6 +40,7 @@ self.set_undefined_options('build', ('build_base', 'build_base'), ('build_lib', 'build_lib'), + ('build_scripts', 'build_scripts'), ('build_temp', 'build_temp')) self.set_undefined_options('bdist', ('bdist_base', 'bdist_base')) @@ -50,27 +54,16 @@ self.warn ("'%s' does not exist -- can't clean it" % self.build_temp) - - - if self.all: - # remove the module build directory (unless already gone) - if os.path.exists (self.build_lib): - remove_tree (self.build_lib, self.verbose, self.dry_run) - else: - self.warn ("'%s' does not exist -- can't clean it" % - self.build_lib) - - # remove the temporary directory used for creating built - # distributions (default "build/bdist") -- eg. type of - # built distribution will have its own subdirectory under - # "build/bdist", but they'll be taken care of by - # 'remove_tree()'. - if os.path.exists (self.bdist_base): - remove_tree (self.bdist_base, self.verbose, self.dry_run) - else: - self.warn ("'%s' does not exist -- can't clean it" % - self.bdist_base) + # remove build directories + for directory in (self.build_lib, + self.bdist_base, + self.build_scripts): + if os.path.exists (directory): + remove_tree (directory, self.verbose, self.dry_run) + else: + self.warn ("'%s' does not exist -- can't clean it" % + directory) # just for the heck of it, try to remove the base build directory: # we might have emptied it right now, but if not we don't care From gward@python.net Mon Sep 11 20:47:02 2000 From: gward@python.net (Greg Ward) Date: Mon Sep 11 19:47:02 2000 Subject: [Distutils] Default format for "bdist" In-Reply-To: <39B5FF79.CD45EF09@lemburg.com>; from mal@lemburg.com on Wed, Sep 06, 2000 at 10:25:29AM +0200 References: <20000905213429.A841@beelzebub> <39B5FF79.CD45EF09@lemburg.com> Message-ID: <20000910210533.A891@beelzebub> On 06 September 2000, M.-A. Lemburg said: > Ok, but could you then at least make the archives platform > name an option then ? I wrote platform.py for just this reason > (it is used in the mxCGIPython project) and would of course > like to reuse its functionality for distutils too. Good idea -- done, at least for bdist_dumb. Now you can run python setup.py bdist --formats=gztar,zip \ --plat-name=linux-2.2-redhat-glibc2.1-i586 or, equivalently (except for the multitude of formats): python setup.py bdist_dumb --plat-name=linux-2.2-redhat-glibc2.1-i586 if you are determined to assault your users with that level of detail. Note that thanks to recent changes to util.get_platform() (not yet checked in, awaiting comment from the SIG), the default platform name is now more useful than Python's vanilla sys.platform, but not as exhaustive (or expensive to compute, I'm guessing) as whatever your platform.py spits out. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Mon Sep 11 21:03:01 2000 From: gward@python.net (Greg Ward) Date: Mon Sep 11 20:03:01 2000 Subject: [Distutils] Rewritten 'get_platform()' In-Reply-To: ; from dubois1@llnl.gov on Sun, Sep 10, 2000 at 09:45:21PM -0700 References: <20000909200139.A12684@beelzebub> Message-ID: <20000911200131.A791@beelzebub> On 10 September 2000, Paul F. Dubois said: > Just a thought related to what you are doing here; I have had some > experiences where the LOCATION mattered too. For example, a certain library > was in /usr/local on one of our systems, and in someone's home directory on > another, even though otherwise the systems were identical. Of course, I > don't know how you could "detect" the location except as a possible > configuration variable. Indeed, I did just that in one make system I wrote. Umm, I'm not talking about auto-configuration -- I'm talking about generating a reasonable "platform ID" string on a variety of Unices. Requirements: * should establish enough binary compatibility for most situations (ie. the OS name, usually the OS release, and quite often the hardware architecture) * for a given OS, should be codable in a few lines of Python based entirely on information available from Python's os and sys modules (eg. os.uname()) In short, I'm looking for something more informative than sys.platform ("linux2" doesn't cut it), but less expensive than what MAL's platform.py computes. Regarding finding required libraries for auto-configuration (since you brought it up), I mostly agree with Bastian. I would add to that: take a stab at searching the prefix directory; if someone installs everything to prefix=/opt/funky/path, then you might just find that key library in /opt/funky/path/lib. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Mon Sep 11 23:31:01 2000 From: gward@python.net (Greg Ward) Date: Mon Sep 11 22:31:01 2000 Subject: [Distutils] Patch for clean.py to remove build_scripts directory In-Reply-To: ; from calvin@cs.uni-sb.de on Mon, Sep 11, 2000 at 11:43:37AM +0200 References: Message-ID: <20000911222917.A977@beelzebub> On 11 September 2000, Bastian Kleineidam said: > In Distutils-0.9.2 the clean command does not remove the build_scripts > dir. The patch is fixing this. Thanks -- just checked it in. Greg From R.Liebscher@gmx.de Tue Sep 12 07:51:01 2000 From: R.Liebscher@gmx.de (Rene Liebscher) Date: Tue Sep 12 06:51:01 2000 Subject: [Distutils] More minor issues Message-ID: <39BE09E2.F699662B@gmx.de> This is a multi-part message in MIME format. --------------CB01264E7BC9B98B6895F141 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit There are two features which could be very useful. 1) If the install command had a '--force' option, which lets install ignore file dates, you could overwrite manipulated installations with the original files. Currently if there are changed some files for any reason, you have to find and delete them and then reinstall the package. (Distutils doesn't overwrite newer files.) (Using this feature you could also install an older version of the package.) 2) If you are using two or more different python versions on your computer, and you want to install a package which contains C-compiled extensions, then you have to delete the build directory before you can install the package with another version of python or you use build with the '--force' option. In any case you have to force distutils to rebuild all extensions because they were compiled using a wrong version of python. If you only build this package once, this may be not a problem, but if you develop a package on your own, you have to compile it more often, and then you don't want to compile everything new to test it. You could use here different build directories, this is not too much work. But you could forget these options, so why doesn't let distutils do it for us. We simply have to change the default build directory to something like 'build_' + sys.version[0:3] There is attached a patch for it. (build.patch) (Using this, I found 'build_rpm' using a different version of python than I have used to start the setup. I probably never had spotted it, if both had used the same build directory.) Kind regards Rene Liebscher --------------CB01264E7BC9B98B6895F141 Content-Type: text/plain; charset=us-ascii; name="build.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="build.patch" diff -BurN --exclude=*.pyc --minimal distutils.orig/distutils/command/build.py distutils.patched/distutils/command/build.py --- distutils.orig/distutils/command/build.py Sat Jun 24 03:23:37 2000 +++ distutils.patched/distutils/command/build.py Tue Sep 12 12:05:46 2000 @@ -48,7 +48,7 @@ ] def initialize_options (self): - self.build_base = 'build' + self.build_base = 'build_' + sys.version[0:3] # these are decided only after 'build_base' has its final value # (unless overridden by the user or client) self.build_purelib = None --------------CB01264E7BC9B98B6895F141-- From thomas.heller@ion-tof.com Tue Sep 12 08:04:01 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue Sep 12 07:04:01 2000 Subject: [Distutils] New version of bdist_wininst Message-ID: <013e01c01ca8$fdc36b60$4500a8c0@thomasnb> I've checked in a slightly changed bdist_wininst version (already on saturday). Here is the log message: The installer now displays info about version of distutils used to create the distribution and the creation date. Takes care of the extra_path argument to the setup function, installs the modules into /extra_path and creates a -pth file (like install_lib does). Thomas Heller From R.Liebscher@gmx.de Tue Sep 12 08:55:01 2000 From: R.Liebscher@gmx.de (Rene Liebscher) Date: Tue Sep 12 07:55:01 2000 Subject: [Distutils] CCompiler interfaces Message-ID: <39BE18E1.DD782434@gmx.de> Last weekend I tried to use Distutils' compiler classes to build an executable on Win32. Beside some problems with the compilers, I found that parameters of some methods of CCompiler are a little bit confusing. Output filenames: * create_static_lib needs a library name, which becomes intern a real file name. (Is anyone using this method?) * link_shared_library is similar to create_static_lib. * link_shared_object needs a real file name. * link_executable wants the basename of its output file. We have two methods shared_object_filename() and library_filename() (for static and shared libraries). Wouldn't it be better to add a method executable_filename() and use for all of the linker methods the real filename as parameter? For link_executable() we also need a parameter export_symbols. (An executable could export symbols to be able to be extended by plug-ins.) If we would make both changes I mentioned above, the interfaces of the three link_* methods would be the same. Also the implementation is almost the same. ( link_shared_library() calls link_shared_object(), and link_executable() differs only in one line in UnixCCompiler. The other compilers are similar, they only change some options to compile an executable instead a shared library.) The best way would to merge these three methods into one new method and remove the old ones. (Or let the old ones call the new and give a warning the user should use the new method.) This new method would have following parameters: (It is better to see a shared libray as an half-build executable than to compare it with static libraries.) def link (self, target, objects, output_libname, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None): target would be one of 'shared_object', 'shared_library' or 'executable'. The compiler classes would then be much simpler and smaller. For example UnixCCompiler. * Remove link_shared_library() and link_executable() * Rename link_shared_object() to link() and add target to the parameter list. * Change the spawn() call in it to the following piece of code. try: if target == 'executable': self.spawn (self.linker_exe + ld_args) else: self.spawn (self.linker_so + ld_args) except DistutilsExecError, msg: raise LinkError, msg This saves about 70 lines of redundant code. For BCPPCompiler and MSVCCompiler it is similar. Are there any objections? If not, I would send Greg a patch for it. ########################################################### And now a short piece of Changes.txt: > Release 0.9 (29 June, 2000): > ---------------------------- ... > * added the ability to compile and link in resource files with > Visual C++ on Windows (Thomas Heller) However, I can't find the place where it is done. Maybe it wasn't checked in ? I can do the same using BCPPCompiler and CygwinCCompiler, and will send a patch after reorganizing the compiler classes. Kind regards Rene Liebscher From gward@python.net Tue Sep 12 21:57:42 2000 From: gward@python.net (Greg Ward) Date: Tue Sep 12 20:57:42 2000 Subject: [Distutils] New code snapshot Message-ID: <20000912204646.A1283@beelzebub> Hi all -- I've just put together a new code snapshot. From CHANGES.txt: * added --plat-name option to bdist and bdist_dumb commands * added --python and --fix-python options to bdist_rpm, for better control over which Python interpreter is hard-coded in the .spec file * fixed default installation directories for Mac OS; now all modules and extensions go to :Lib:site-packages * fixed the "clean" command so it cleans up the build directory for scripts (build/scripts by default) (Bastian Kleineidam) * more improvements to the bdist_wininst command and the installers it generates: ....... (Thomas Heller) * fix the "sdist" command so it deals better with missing, empty, or broken MANIFEST and MANIFEST.in files Hmm, this list is getting long: time for 0.9.3 already? Thomas, can you provide a 25-words-or-less blurb to put in place of that "....."? It's at the usual place: http://www.python.org/sigs/distutils-sig/download.html -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Tue Sep 12 22:30:45 2000 From: gward@python.net (Greg Ward) Date: Tue Sep 12 21:30:45 2000 Subject: [Distutils] More minor issues In-Reply-To: <39BE09E2.F699662B@gmx.de>; from R.Liebscher@gmx.de on Tue, Sep 12, 2000 at 12:48:02PM +0200 References: <39BE09E2.F699662B@gmx.de> Message-ID: <20000912212202.A1082@beelzebub> On 12 September 2000, Rene Liebscher said: > 1) If the install command had a '--force' option, > which lets install ignore file dates, you could > overwrite manipulated installations with the > original files. Good idea, especially since the "install" command needs to get *more* paranoid about overwriting old files. ...OK, I've just made that change to install*.py. Boy, we need a better way to add an option to a whole family of commands. ;-( Anyways, it works for me when installing the Distutils. Please give it a try on something that includes data, scripts, etc. Hmmm, maybe I'll give it a shot on my Distutil-ized IDLE... that's got modules, data files, and a script! ...OK, it works, but it revealed a subtle bug in my IDLE setup script. D'ohh! Oh well, this is an excuse to repost said setup script to python-dev and prod Guido about distutil-izing IDLE. ;-) Good demo of packaging a real Python app, if nothing else. > 2) If you are using two or more different python versions > on your computer, and you want to install a package > which contains C-compiled extensions, then you have > to delete the build directory before you can install > the package with another version of python or you > use build with the '--force' option. > In any case you have to force distutils to rebuild > all extensions because they were compiled using a wrong > version of python. Hmmm, good point. I'm always wary of over-differentiation leading to unworkably long directory names, but I think this one is worthy. However, I don't like the idea of having multiple top-level "build" directories; I'd rather the sub-directories of "build" incorporated the Python version along with the platform info that they already include. This means there should probably be a function somewhere to take a plain vanilla directory name and "differentiate" it -- eg. build/lib -> build/lib-1.6 or build/temp -> build/temp.linux-i586-1.6. Or something like that. Can you rework your patch to do this? I think the commads to hit are build_* and bdist_*. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Tue Sep 12 22:38:07 2000 From: gward@python.net (Greg Ward) Date: Tue Sep 12 21:38:07 2000 Subject: [Distutils] New version of bdist_wininst In-Reply-To: <013e01c01ca8$fdc36b60$4500a8c0@thomasnb>; from thomas.heller@ion-tof.com on Tue, Sep 12, 2000 at 01:02:38PM +0200 References: <013e01c01ca8$fdc36b60$4500a8c0@thomasnb> Message-ID: <20000912212650.B1082@beelzebub> On 12 September 2000, Thomas Heller said: > I've checked in a slightly changed bdist_wininst > version (already on saturday). [...] > Takes care of the extra_path argument to the setup function, > installs the modules into /extra_path and creates > a -pth file (like install_lib does). Hmmm. And someone else recently pointed out that bdist_wininst doesn't handle scripts (or was it data files? or both? whatever). Sounds like bdist_wininst isn't playing by the rules and running a fake install before putting together its built distribution. It shouldn't have to do any work to handle extra_path/.pth files -- that should Just Work (TM). (rummage rummage, peek prod, etc.) HmmMMMMmmm. It looks like bdist_wininst *is* doing the right thing: running the "install" command directly, which should take care of this. Something seems to be going wrong. Thomas, have you looked into the scripts/data (whichever) problem? I'm confused... well, I've added an entry to the TODO file, which is about all I have the brainpower left to do tonight. Greg From jeremy@beopen.com Tue Sep 12 23:41:14 2000 From: jeremy@beopen.com (Jeremy Hylton) Date: Tue Sep 12 22:41:14 2000 Subject: [Distutils] define_macros ignored? Message-ID: <14782.59101.828231.905056@bitdiddle.concentric.net> I am working on a setup script for Tkinter, which requires a -DWITH_APPINIT flag at compile time. The docs suggest that I should add the following definition to my Extension constructor call: define_macros=[('WITH_APPINIT', None),], I did this, but it had no effect. It looks like there is no where that the define_macros attribute of the Extension object is actually used. I assumed this is a bug and tried to figure out where the macros definitions ought to be processed. I'm not familiar with the code, so I may have put it in the wrong place. A patch is included below, which allows _tkinter to build. Jeremy *** build_ext.py Tue Sep 12 20:58:48 2000 --- /home/jeremy/src/python/dist/src/Lib/distutils/command/build_ext.py Thu Sep 7 11:09:18 2000 *************** *** 401,407 **** # any sensible compiler will give precedence to later # command line args. Hence we combine them in order: extra_args = ext.extra_compile_args or [] - macros = ext.define_macros or [] # XXX and if we support CFLAGS, why not CC (compiler # executable), CPPFLAGS (pre-processor options), and LDFLAGS --- 401,406 ---- *************** *** 410,419 **** if os.environ.has_key('CFLAGS'): extra_args.extend(string.split(os.environ['CFLAGS'])) ! objects = self.compiler.compile (sources, output_dir=self.build_temp, ! macros=macros, include_dirs=ext.include_dirs, debug=self.debug, extra_postargs=extra_args) --- 409,418 ---- if os.environ.has_key('CFLAGS'): extra_args.extend(string.split(os.environ['CFLAGS'])) ! objects = self.compiler.compile (sources, output_dir=self.build_temp, ! #macros=macros, include_dirs=ext.include_dirs, debug=self.debug, extra_postargs=extra_args) From calvin@cs.uni-sb.de Wed Sep 13 07:30:07 2000 From: calvin@cs.uni-sb.de (Bastian Kleineidam) Date: Wed Sep 13 06:30:07 2000 Subject: [Distutils] bdist_wininst and data files Message-ID: Hello, I just discovered the "bug" which prevents bdist_wininst from installing data files and scripts. bdist_wininst.py:109: root_dir = install.install_lib I noticed this comment in the source: # XXX hack! Our archive MUST be relative to sys.prefix # XXX What about .install_data, .install_scripts, ...? So what about data and scripts? Why dont they get in the archive? Bastian From thomas.heller@ion-tof.com Wed Sep 13 07:48:01 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Wed Sep 13 06:48:01 2000 Subject: [Distutils] bdist_wininst and data files References: Message-ID: <006d01c01d6f$fdae2e40$4500a8c0@thomasnb> I will look after this... Thomas Heller ----- Original Message ----- From: "Bastian Kleineidam" To: Sent: Wednesday, September 13, 2000 12:29 PM Subject: [Distutils] bdist_wininst and data files > Hello, > > I just discovered the "bug" which prevents bdist_wininst from installing > data files and scripts. > bdist_wininst.py:109: root_dir = install.install_lib > > I noticed this comment in the source: > # XXX hack! Our archive MUST be relative to sys.prefix > # XXX What about .install_data, .install_scripts, ...? > > So what about data and scripts? Why dont they get in the archive? > > Bastian > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG@python.org > http://www.python.org/mailman/listinfo/distutils-sig From thomas.heller@ion-tof.com Wed Sep 13 11:34:01 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Wed Sep 13 10:34:01 2000 Subject: [Distutils] setup.py sdist hangs on windows with 1.6 and 2.0b1 Message-ID: <027501c01d8f$a24f51d0$4500a8c0@thomasnb> This may not be directly related to distutils, it may also be a bug in 1.6 and 2.0b1 re implementation. 'setup.py sdist' with the current distutils CVS version hangs while parsing MANIFEST.in, executing the re.sub command in these lines in text_file.py: # collapse internal whitespace (*after* joining lines!) if self.collapse_ws: line = re.sub (r'(\S)\s+(\S)', r'\1 \2', line) Has anyone else noticed this, or is smething wrong on my side? Thomas From R.Liebscher@gmx.de Thu Sep 14 06:19:01 2000 From: R.Liebscher@gmx.de (Rene Liebscher) Date: Thu Sep 14 05:19:01 2000 Subject: [Distutils] More minor issues References: <39BE09E2.F699662B@gmx.de> <20000912212202.A1082@beelzebub> Message-ID: <39C09744.133BDC89@gmx.de> This is a multi-part message in MIME format. --------------0C7CAED1776BBF03C2141E8C Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Greg Ward wrote: > > On 12 September 2000, Rene Liebscher said: > > 1) If the install command had a '--force' option, > > which lets install ignore file dates, you could > > overwrite manipulated installations with the > > original files. > > Good idea, especially since the "install" command needs to get *more* > paranoid about overwriting old files. > > ...OK, I've just made that change to install*.py. Boy, we need a better > way to add an option to a whole family of commands. ;-( > > Anyways, it works for me when installing the Distutils. Please give it > a try on something that includes data, scripts, etc. Hmmm, maybe I'll > give it a shot on my Distutil-ized IDLE... that's got modules, data > files, and a script! > > ...OK, it works, but it revealed a subtle bug in my IDLE setup script. > D'ohh! Oh well, this is an excuse to repost said setup script to > python-dev and prod Guido about distutil-izing IDLE. ;-) Good demo of > packaging a real Python app, if nothing else. > > > 2) If you are using two or more different python versions > > on your computer, and you want to install a package > > which contains C-compiled extensions, then you have > > to delete the build directory before you can install > > the package with another version of python or you > > use build with the '--force' option. > > In any case you have to force distutils to rebuild > > all extensions because they were compiled using a wrong > > version of python. > > Hmmm, good point. I'm always wary of over-differentiation leading to > unworkably long directory names, but I think this one is worthy. > However, I don't like the idea of having multiple top-level "build" > directories; I'd rather the sub-directories of "build" incorporated the > Python version along with the platform info that they already include. > > This means there should probably be a function somewhere to take a plain > vanilla directory name and "differentiate" it -- eg. build/lib -> > build/lib-1.6 or build/temp -> build/temp.linux-i586-1.6. Or something > like that. Can you rework your patch to do this? I think the commads > to hit are build_* and bdist_*. It is only build.py, all other command get their paths from build. I changed only the platform specific directories. lib.linux2 -> lib-1.5.linux2 temp.linux2 -> temp-1.5.linux2 If you try this with distutils, you will not see any changes, because distutils is a pure-python package. (file: lib-x.x.patch) ----------------------------------- In the CVS are the source files for the windows installer. So everyone can create his own, I think most people would like to change the bitmap (PythonPowered) and use a more specific bitmap for their package. Creating the installer executable is not the problem, but to use it you have to change bdist_wininst.py. (If you only maintain one package this is not a problem but if there are more, you had to change it everytime you want to create an installer for the other package.) So I added an 'install-exe' option to bdist_wininst, which can be used to specify the executable which is used as base for the installer. Using this you can have own installers for every package. (You could set it in setup.cfg.) This should also help to test new installers, I'm right, Thomas? (file: wininst.patch) -------------------------------------- Another problem: python setup.py bdist --formats=zip,gztar doesn't work. It deletes the build tree after creating the first archive. There is a simple solution for that. First find out all commands to run. Then execute all commands, check for every command if it has to be run later again (eg. bdist_dumb with other format), if it has, check if there is a keep_tree option and set it. This way you keep the tree for following runs, only the last one will remove the tree because there is no following command and the keep_tree will not be set. (file: bdist.patch) After this you could even run this command: python setup.py bdist --formats=rpm,gztar,bztar,ztar,tar,wininst,zip All, except rpm and wininst, would use the same tree. Kind regards Rene Liebscher --------------0C7CAED1776BBF03C2141E8C Content-Type: text/plain; charset=us-ascii; name="lib-x.x.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="lib-x.x.patch" diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/command/build.py distutils.patched/distutils/command/build.py --- distutils.orig/distutils/command/build.py Sat Jun 24 03:23:37 2000 +++ distutils.patched/distutils/command/build.py Wed Sep 13 13:58:56 2000 @@ -74,7 +74,7 @@ self.build_purelib = os.path.join (self.build_base, 'lib') if self.build_platlib is None: self.build_platlib = os.path.join (self.build_base, - 'lib.' + self.plat) + 'lib-' + sys.version[0:3] + '.' + self.plat) # 'build_lib' is the actual directory that we will use for this # particular module distribution -- if user didn't supply it, pick @@ -89,7 +89,7 @@ # "build/temp." if self.build_temp is None: self.build_temp = os.path.join (self.build_base, - 'temp.' + self.plat) + 'temp-' + sys.version[0:3] + '.' + self.plat) if self.build_scripts is None: self.build_scripts = os.path.join (self.build_base, 'scripts') # finalize_options () --------------0C7CAED1776BBF03C2141E8C Content-Type: text/plain; charset=us-ascii; name="wininst.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="wininst.patch" diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/command/bdist_wininst.py distutils.patched/distutils/command/bdist_wininst.py --- distutils.orig/distutils/command/bdist_wininst.py Tue Sep 12 11:50:01 2000 +++ distutils.patched/distutils/command/bdist_wininst.py Wed Sep 13 18:05:13 2000 @@ -31,6 +31,8 @@ "do not compile .py to .pyo (optimized) on the target system"), ('dist-dir=', 'd', "directory to put final built distributions in"), + ('install-exe=', None, + "executable used as base for installer"), ] def initialize_options (self): @@ -40,6 +42,7 @@ self.no_target_optimize = 0 self.target_version = None self.dist_dir = None + self.install_exe = None # initialize_options() @@ -59,6 +62,10 @@ self.set_undefined_options('bdist', ('dist_dir', 'dist_dir')) + if self.install_exe: + if not os.path.isfile(self.install_exe): + raise DistutilsFileError, "%s not found." % self.install_exe + # finalize_options() @@ -193,8 +200,11 @@ # create_exe() def get_exe_bytes (self): - import base64 - return base64.decodestring (EXEDATA) + if self.install_exe: + return open(self.install_exe,"rb").read() + else: + import base64 + return base64.decodestring (EXEDATA) # class bdist_wininst --------------0C7CAED1776BBF03C2141E8C Content-Type: text/plain; charset=us-ascii; name="bdist.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="bdist.patch" diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/command/bdist.py distutils.patched/distutils/command/bdist.py --- distutils.orig/distutils/command/bdist.py Tue Sep 12 11:50:01 2000 +++ distutils.patched/distutils/command/bdist.py Wed Sep 13 15:39:08 2000 @@ -109,16 +109,25 @@ def run (self): + commands = [] + # which commands do we need for format in self.formats: try: - cmd_name = self.format_command[format][0] + commands.append(self.format_command[format][0]) except KeyError: raise DistutilsOptionError, \ "invalid format '%s'" % format + # execute commands + for i in range(len(self.formats)): + cmd_name = commands[i] sub_cmd = self.reinitialize_command(cmd_name) if cmd_name not in self.no_format_option: - sub_cmd.format = format + sub_cmd.format = self.formats[i] + # Do we need this command later again? Does it have a keep_tree + # option? If it has, keep the tree for next run of this command. + if hasattr(sub_cmd,"keep_tree") and (cmd_name in commands[i+1:]): + sub_cmd.keep_tree = 1 self.run_command (cmd_name) # run() --------------0C7CAED1776BBF03C2141E8C-- From akuchlin@mems-exchange.org Thu Sep 14 09:01:01 2000 From: akuchlin@mems-exchange.org (A.M. Kuchling) Date: Thu Sep 14 08:01:01 2000 Subject: [Distutils] Path caching when running multiple commands Message-ID: I've written a script to apply a command across a bunch of setup.py scripts in a subdirectory, and found a problem: distutils.dir_utils._path_cache stores a list of directories that have already been created. The directory name is just stored as 'dist', not a full path name, so if two subdirectories need to create a 'dist' directory, the second one thinks it's already been created, and dies. Fix: storing full path names in the cache seems reasonable. (The setup.py script is included below, since it may be instructive; feel free to use it as an example.) --amk # Top-level script for running a command-line over all the subpackages import sys, os, glob from distutils.core import run_setup files = os.listdir('.') dirs = filter(os.path.isdir, files) packages = [] for dir in dirs: if os.path.exists( os.path.join( dir, 'setup.py') ): packages.append( dir ) top_dir = os.getcwd() for package_dir in packages: print 'Running', package_dir os.chdir( package_dir ) dist = run_setup('setup.py', script_args=sys.argv[1:], stop_after="run") os.chdir( top_dir ) From thomas.heller@ion-tof.com Thu Sep 14 09:35:01 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Thu Sep 14 08:35:01 2000 Subject: [Distutils] Minor issues References: Message-ID: <020901c01e48$19659460$4500a8c0@thomasnb> > 3) bdist_wininst does not seem to install data files and scripts. > If you execute "strings" with the .exe, you will only see Python module > files. See separate post. BTW: On which system did you create the installer? > And I did not found a file with the install log. The installer just > copied the files in the Python dir. Why do you want this file? There are hooks in the installer code to create such a file, but they are still unused. Thomas From R.Liebscher@gmx.de Thu Sep 14 09:54:01 2000 From: R.Liebscher@gmx.de (Rene Liebscher) Date: Thu Sep 14 08:54:01 2000 Subject: [Distutils] CCompiler interfaces References: <39BE18E1.DD782434@gmx.de> Message-ID: <39C09BBC.C03D815E@gmx.de> This is a multi-part message in MIME format. --------------2B7CFCB61910F88B9072AE46 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Rene Liebscher wrote: > > Last weekend I tried to use Distutils' compiler > classes to build an executable on Win32. > Beside some problems with the compilers, > I found that parameters of some methods of > CCompiler are a little bit confusing. > > Output filenames: > > * create_static_lib needs a library name, which > becomes intern a real file name. > (Is anyone using this method?) > > * link_shared_library is similar to > create_static_lib. > > * link_shared_object needs a real file name. > > * link_executable wants the basename of its > output file. > > We have two methods shared_object_filename() > and library_filename() (for static and shared > libraries). Wouldn't it be better to add > a method executable_filename() and use for > all of the linker methods the real filename > as parameter? > > For link_executable() we also need a parameter > export_symbols. (An executable could export > symbols to be able to be extended by plug-ins.) > > If we would make both changes I mentioned above, > the interfaces of the three link_* methods would > be the same. Also the implementation is almost > the same. ( link_shared_library() calls > link_shared_object(), and link_executable() differs > only in one line in UnixCCompiler. The other compilers > are similar, they only change some options to > compile an executable instead a shared library.) > > The best way would to merge these three methods > into one new method and remove the old ones. > (Or let the old ones call the new and give a warning > the user should use the new method.) > > This new method would have following parameters: > (It is better to see a shared libray as an half-build > executable than to compare it with static libraries.) > > def link (self, > target, > objects, > output_libname, > output_dir=None, > libraries=None, > library_dirs=None, > runtime_library_dirs=None, > export_symbols=None, > debug=0, > extra_preargs=None, > extra_postargs=None, > build_temp=None): > > target would be one of 'shared_object', > 'shared_library' or 'executable'. > > The compiler classes would then be much simpler > and smaller. > > For example UnixCCompiler. > * Remove link_shared_library() and link_executable() > * Rename link_shared_object() to link() and add > target to the parameter list. > > * Change the spawn() call in it to the following > piece of code. > > try: > if target == 'executable': > self.spawn (self.linker_exe + ld_args) > else: > self.spawn (self.linker_so + ld_args) > except DistutilsExecError, msg: > raise LinkError, msg > > This saves about 70 lines of redundant code. > For BCPPCompiler and MSVCCompiler it is similar. > > Are there any objections? > If not, I would send Greg a patch for it. > > ########################################################### > And now a short piece of Changes.txt: > > Release 0.9 (29 June, 2000): > > ---------------------------- > ... > > > * added the ability to compile and link in resource files with > > Visual C++ on Windows (Thomas Heller) > > However, I can't find the place where it is done. Maybe it wasn't > checked in ? > I can do the same using BCPPCompiler and CygwinCCompiler, > and will send a patch after reorganizing the compiler > classes. > Here is now the patch for both, reorganzing the compiler classes and adding the ability to use resource files with Borland C++ and Cygwin. (file: complier.link.patch) It is a huge patch (43k) because it has to delete a lot of old redundant code. unixccompiler.py 12926 -> 10407 Bytes msvccompiler.py 17251 -> 14321 # next both can now compile resources bcppcompiler.py 14992 -> 14237 cygwinccompiler.py 13294 -> 17178 (this is the new code to compile resources.) ccompiler.py 40680 -> 42175 (Without old methods link_shared_*() it would be 39508. These old methods are still there for compatibility with existing external code.) Kind regards Rene Liebscher --------------2B7CFCB61910F88B9072AE46 Content-Type: text/plain; charset=us-ascii; name="compiler.link.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="compiler.link.patch" diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/bcppcompiler.py distutils.patched/distutils/bcppcompiler.py --- distutils.orig/distutils/bcppcompiler.py Mon Sep 4 11:58:57 2000 +++ distutils.patched/distutils/bcppcompiler.py Wed Sep 13 15:09:52 2000 @@ -63,7 +63,7 @@ # indicate their installation locations. self.cc = "bcc32.exe" - self.link = "ilink32.exe" + self.linker = "ilink32.exe" self.lib = "tlib.exe" self.preprocess_options = None @@ -73,6 +73,8 @@ self.ldflags_shared = ['/Tpd', '/Gn', '/q', '/x'] self.ldflags_shared_debug = ['/Tpd', '/Gn', '/q', '/x'] self.ldflags_static = [] + self.ldflags_exe = ['/Gn', '/q', '/x'] + self.ldflags_exe_debug = ['/Gn', '/q', '/x','/r'] # -- Worker methods ------------------------------------------------ @@ -108,16 +110,32 @@ if skip_sources[src]: self.announce ("skipping %s (%s up-to-date)" % (src, obj)) else: + src = os.path.normpath(src) + obj = os.path.normpath(obj) + self.mkpath(os.path.dirname(obj)) + + if ext == '.res': + # this is already a binary file + continue # the 'for' loop + if ext == '.rc': + # this needs to be compiled to a .res file + try: + self.spawn (["brcc32"] + ["-fo"]+ [obj] + [src]) + except DistutilsExecError, msg: + raise CompileError, msg + continue # the 'for' loop + + # next both are for the real compiler if ext in self._c_extensions: input_opt = "" elif ext in self._cpp_extensions: input_opt = "-P" + else: + # set default. it is an unknown extension, + # probably the compiler will fail + input_opt = "" - src = os.path.normpath(src) - obj = os.path.normpath(obj) - output_opt = "-o" + obj - self.mkpath(os.path.dirname(obj)) # Compiler command line syntax is: "bcc32 [options] file(s)". # Note that the source file names must appear at the end of @@ -163,45 +181,20 @@ # create_static_lib () - - def link_shared_lib (self, - objects, - output_libname, - output_dir=None, - libraries=None, - library_dirs=None, - runtime_library_dirs=None, - export_symbols=None, - debug=0, - extra_preargs=None, - extra_postargs=None, - build_temp=None): - - self.link_shared_object (objects, - self.shared_library_name(output_libname), - output_dir=output_dir, - libraries=libraries, - library_dirs=library_dirs, - runtime_library_dirs=runtime_library_dirs, - export_symbols=export_symbols, - debug=debug, - extra_preargs=extra_preargs, - extra_postargs=extra_postargs, - build_temp=build_temp) - - def link_shared_object (self, - objects, - output_filename, - output_dir=None, - libraries=None, - library_dirs=None, - runtime_library_dirs=None, - export_symbols=None, - debug=0, - extra_preargs=None, - extra_postargs=None, - build_temp=None): + def link (self, + target, + objects, + output_filename, + output_dir=None, + libraries=None, + library_dirs=None, + runtime_library_dirs=None, + export_symbols=None, + debug=0, + extra_preargs=None, + extra_postargs=None, + build_temp=None): # XXX this ignores 'build_temp'! should follow the lead of # msvccompiler.py @@ -213,45 +206,62 @@ if runtime_library_dirs: self.warn ("I don't know what to do with 'runtime_library_dirs': " + str (runtime_library_dirs)) - + + if output_dir is not None: output_filename = os.path.join (output_dir, output_filename) if self._need_link (objects, output_filename): - if debug: - ld_args = self.ldflags_shared_debug[:] + # target specific code + if target == CCompiler.EXECUTABLE: + startup_obj = 'c0w32' + if debug: + ld_args = self.ldflags_exe_debug[:] + else: + ld_args = self.ldflags_exe[:] else: - ld_args = self.ldflags_shared[:] + startup_obj = 'c0d32' + if debug: + ld_args = self.ldflags_shared_debug[:] + else: + ld_args = self.ldflags_shared[:] + # Create a temporary exports file for use by the linker - head, tail = os.path.split (output_filename) - modname, ext = os.path.splitext (tail) - temp_dir = os.path.dirname(objects[0]) # preserve tree structure - def_file = os.path.join (temp_dir, '%s.def' % modname) - contents = ['EXPORTS'] - for sym in (export_symbols or []): - contents.append(' %s=_%s' % (sym, sym)) - self.execute(write_file, (def_file, contents), - "writing %s" % def_file) + if export_symbols is None: + def_file = '' + else: + head, tail = os.path.split (output_filename) + modname, ext = os.path.splitext (tail) + temp_dir = os.path.dirname(objects[0]) # preserve tree structure + def_file = os.path.join (temp_dir, '%s.def' % modname) + contents = ['EXPORTS'] + for sym in (export_symbols or []): + contents.append(' %s=_%s' % (sym, sym)) + self.execute(write_file, (def_file, contents), + "writing %s" % def_file) # Borland C++ has problems with '/' in paths - objects = map(os.path.normpath, objects) - startup_obj = 'c0d32' - objects.insert(0, startup_obj) - - # either exchange python15.lib in the python libs directory against - # a Borland-like one, or create one with name bcpp_python15.lib - # there and remove the pragmas from config.h - libraries.append ('import32') - libraries.append ('cw32mt') - - # Start building command line flags and options. - + objects2 = map(os.path.normpath, objects) + # split objects in .obj and .res files + # Borland C++ needs them at different positions in the command line + objects = [startup_obj] + resources = [] + for file in objects2: + (base, ext) = os.path.splitext(os.path.normcase(file)) + if ext == '.res': + resources.append(file) + else: + objects.append(file) + + for l in library_dirs: ld_args.append("/L%s" % os.path.normpath(l)) - - ld_args.extend(objects) # list of object files + ld_args.append("/L.") # we use sometimes relative paths + + # list of object files + ld_args.extend(objects) # XXX the command-line syntax for Borland C++ is a bit wonky; # certain filenames are jammed together in one big string, but @@ -263,14 +273,14 @@ # because 'spawn()' would quote any filenames with spaces in # them. Arghghh!. Apparently it works fine as coded... - # name of dll file + # name of dll/exe file ld_args.extend([',',output_filename]) # no map file and start libraries ld_args.append(',,') for lib in libraries: # see if we find it and if there is a bcpp specific lib - # (bcpp_xxx.lib) + # (xxx_bcpp.lib) libfile = self.find_library_file(library_dirs, lib, debug) if libfile is None: ld_args.append(lib) @@ -279,8 +289,17 @@ else: # full name which prefers bcpp_xxx.lib over xxx.lib ld_args.append(libfile) + + # some default libraries + ld_args.append ('import32') + ld_args.append ('cw32mt') + # def file for export symbols ld_args.extend([',',def_file]) + # add resource files + ld_args.append(',') + ld_args.extend(resources) + if extra_preargs: ld_args[:0] = extra_preargs @@ -289,88 +308,24 @@ self.mkpath (os.path.dirname (output_filename)) try: - self.spawn ([self.link] + ld_args) + self.spawn ([self.linker] + ld_args) except DistutilsExecError, msg: raise LinkError, msg else: self.announce ("skipping %s (up-to-date)" % output_filename) - # link_shared_object () - - - def link_executable (self, - objects, - output_progname, - output_dir=None, - libraries=None, - library_dirs=None, - runtime_library_dirs=None, - debug=0, - extra_preargs=None, - extra_postargs=None): - - (objects, output_dir) = self._fix_object_args (objects, output_dir) - (libraries, library_dirs, runtime_library_dirs) = \ - self._fix_lib_args (libraries, library_dirs, runtime_library_dirs) - - if runtime_library_dirs: - self.warn ("I don't know what to do with 'runtime_library_dirs': " - + str (runtime_library_dirs)) - - lib_opts = gen_lib_options (self, - library_dirs, runtime_library_dirs, - libraries) - output_filename = output_progname + self.exe_extension - if output_dir is not None: - output_filename = os.path.join (output_dir, output_filename) - - if self._need_link (objects, output_filename): - - if debug: - ldflags = self.ldflags_shared_debug[1:] - else: - ldflags = self.ldflags_shared[1:] - - ld_args = ldflags + lib_opts + \ - objects + ['/OUT:' + output_filename] - - if extra_preargs: - ld_args[:0] = extra_preargs - if extra_postargs: - ld_args.extend (extra_postargs) - - self.mkpath (os.path.dirname (output_filename)) - try: - self.spawn ([self.link] + ld_args) - except DistutilsExecError, msg: - raise LinkError, msg - else: - self.announce ("skipping %s (up-to-date)" % output_filename) - + # link () # -- Miscellaneous methods ----------------------------------------- - # These are all used by the 'gen_lib_options() function, in - # ccompiler.py. - - def library_dir_option (self, dir): - return "-L" + dir - - def runtime_library_dir_option (self, dir): - raise DistutilsPlatformError, \ - ("don't know how to set runtime library search path " - "for Borland C++") - - def library_option (self, lib): - return self.library_filename (lib) def find_library_file (self, dirs, lib, debug=0): # List of effective library names to try, in order of preference: - # bcpp_xxx.lib is better than xxx.lib + # xxx_bcpp.lib is better than xxx.lib # and xxx_d.lib is better than xxx.lib if debug is set # - # The "bcpp_" prefix is to handle a Python installation for people + # The "_bcpp" suffix is to handle a Python installation for people # with multiple compilers (primarily Distutils hackers, I suspect # ;-). The idea is they'd have one static library for each # compiler they care about, since (almost?) every Windows compiler @@ -390,3 +345,31 @@ # Oops, didn't find it in *any* of 'dirs' return None + # overwrite the one from CCompiler to support rc and res-files + def object_filenames (self, + source_filenames, + strip_dir=0, + output_dir=''): + if output_dir is None: output_dir = '' + obj_names = [] + for src_name in source_filenames: + # use normcase to make sure '.rc' is really '.rc' and not '.RC' + (base, ext) = os.path.splitext (os.path.normcase(src_name)) + if ext not in (self.src_extensions + ['.rc','.res']): + raise UnknownFileError, \ + "unknown file type '%s' (from '%s')" % \ + (ext, src_name) + if strip_dir: + base = os.path.basename (base) + if ext == '.res': + # these can go unchanged + obj_names.append (os.path.join (output_dir, base + ext)) + elif ext == '.rc': + # these need to be compiled to .res-files + obj_names.append (os.path.join (output_dir, base + '.res')) + else: + obj_names.append (os.path.join (output_dir, + base + self.obj_extension)) + return obj_names + + # object_filenames () \ No newline at end of file diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/ccompiler.py distutils.patched/distutils/ccompiler.py --- distutils.orig/distutils/ccompiler.py Tue Aug 8 13:18:37 2000 +++ distutils.patched/distutils/ccompiler.py Wed Sep 13 15:00:54 2000 @@ -561,9 +561,15 @@ pass - def link_shared_lib (self, + # values for target parameter in link() + SHARED_OBJECT = "shared_object" + SHARED_LIBRARY = "shared_library" + EXECUTABLE = "executable" + + def link (self, + target, objects, - output_libname, + output_filename, output_dir=None, libraries=None, library_dirs=None, @@ -573,12 +579,15 @@ extra_preargs=None, extra_postargs=None, build_temp=None): - """Link a bunch of stuff together to create a shared library file. - Similar semantics to 'create_static_lib()', with the addition of - other libraries to link against and directories to search for them. - Also, of course, the type and name of the generated file will - almost certainly be different, as will the program used to create - it. + """Link a bunch of stuff together to create an executable or + shared library file. + + The "bunch of stuff" consists of the list of object files supplied + as 'objects'. + 'output_filename' should be a filename. + If 'output_dir' is supplied, 'output_filename' is relative to it + (i.e. 'output_filename' can provide directory components if + needed). 'libraries' is a list of libraries to link against. These are library names, not filenames, since they're translated into @@ -610,7 +619,30 @@ Raises LinkError on failure. """ - pass + raise NotImplementedError + + + # old methods, rewritten to use the new link() method. + + def link_shared_lib (self, + objects, + output_libname, + output_dir=None, + libraries=None, + library_dirs=None, + runtime_library_dirs=None, + export_symbols=None, + debug=0, + extra_preargs=None, + extra_postargs=None, + build_temp=None): + self.warn("link_shared_lib(...) is depreciated, " + "use link(CCompiler.SHARED_LIBRARY,...) instead.") + self.link (CCompiler.SHARED_LIBRARY, objects, + self.library_filename (output_libname, lib_type='shared'), + output_dir, + libraries, library_dirs, runtime_library_dirs, export_symbols, + debug, extra_preargs, extra_postargs, build_temp) def link_shared_object (self, @@ -625,16 +657,11 @@ extra_preargs=None, extra_postargs=None, build_temp=None): - """Link a bunch of stuff together to create a shared object file. - Much like 'link_shared_lib()', except the output filename is - explicitly supplied as 'output_filename'. If 'output_dir' is - supplied, 'output_filename' is relative to it - (i.e. 'output_filename' can provide directory components if - needed). - - Raises LinkError on failure. - """ - pass + self.warn("link_shared_object(...) is depreciated, " + "use link(CCompiler.SHARED_OBJECT,...) instead.") + self.link (CCompiler.SHARED_OBJECT, objects, output_filename, output_dir, + libraries, library_dirs, runtime_library_dirs, export_symbols, + debug, extra_preargs, extra_postargs, build_temp) def link_executable (self, @@ -647,15 +674,12 @@ debug=0, extra_preargs=None, extra_postargs=None): - """Link a bunch of stuff together to create a binary executable - file. The "bunch of stuff" is as for 'link_shared_lib()'. - 'output_progname' should be the base name of the executable - program--e.g. on Unix the same as the output filename, but on - DOS/Windows ".exe" will be appended. - - Raises LinkError on failure. - """ - pass + self.warn("link_executable(...) is depreciated, " + "use link(CCompiler.EXECUTABLE,...) instead.") + self.link (CCompiler.EXECUTABLE, objects, + self.executable_filename(output_progname), output_dir, + libraries, library_dirs, runtime_library_dirs, None, + debug, extra_preargs, extra_postargs, None) @@ -756,6 +780,14 @@ basename = os.path.basename (basename) return os.path.join (output_dir, basename + self.shared_lib_extension) + def executable_filename (self, + basename, + strip_dir=0, + output_dir=''): + if output_dir is None: output_dir = '' + if strip_dir: + basename = os.path.basename (basename) + return os.path.join (output_dir, basename + (self.exe_extension or '')) def library_filename (self, libname, diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/command/build_ext.py distutils.patched/distutils/command/build_ext.py --- distutils.orig/distutils/command/build_ext.py Thu Sep 7 11:51:25 2000 +++ distutils.patched/distutils/command/build_ext.py Tue Sep 12 16:08:32 2000 @@ -425,7 +425,7 @@ extra_args = ext.extra_link_args or [] - self.compiler.link_shared_object ( + self.compiler.link (self.compiler.SHARED_OBJECT, objects, ext_filename, libraries=self.get_libraries(ext), library_dirs=ext.library_dirs, diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/cygwinccompiler.py distutils.patched/distutils/cygwinccompiler.py --- distutils.orig/distutils/cygwinccompiler.py Mon Sep 4 11:58:57 2000 +++ distutils.patched/distutils/cygwinccompiler.py Wed Sep 13 14:55:09 2000 @@ -45,8 +45,10 @@ __revision__ = "$Id: cygwinccompiler.py,v 1.7 2000/09/01 01:24:31 gward Exp $" import os,sys,copy +from distutils.ccompiler import gen_preprocess_options, gen_lib_options from distutils.unixccompiler import UnixCCompiler from distutils.file_util import write_file +from distutils.errors import DistutilsExecError,CompileError,UnknownFileError class CygwinCCompiler (UnixCCompiler): @@ -111,18 +113,77 @@ # __init__ () - def link_shared_object (self, - objects, - output_filename, - output_dir=None, - libraries=None, - library_dirs=None, - runtime_library_dirs=None, - export_symbols=None, - debug=0, - extra_preargs=None, - extra_postargs=None, - build_temp=None): + # not much different of the compile method in UnixCCompiler, + # but we have to insert some lines in the middle of it, so + # we put here a adapted version of it. + # (If we would call compile() in the base class, it would do some + # initializations a second time, this is why all is done here.) + def compile (self, + sources, + output_dir=None, + macros=None, + include_dirs=None, + debug=0, + extra_preargs=None, + extra_postargs=None): + + (output_dir, macros, include_dirs) = \ + self._fix_compile_args (output_dir, macros, include_dirs) + (objects, skip_sources) = self._prep_compile (sources, output_dir) + + # Figure out the options for the compiler command line. + pp_opts = gen_preprocess_options (macros, include_dirs) + cc_args = pp_opts + ['-c'] + if debug: + cc_args[:0] = ['-g'] + if extra_preargs: + cc_args[:0] = extra_preargs + if extra_postargs is None: + extra_postargs = [] + + # Compile all source files that weren't eliminated by + # '_prep_compile()'. + for i in range (len (sources)): + src = sources[i] ; obj = objects[i] + ext = (os.path.splitext (src))[1] + if skip_sources[src]: + self.announce ("skipping %s (%s up-to-date)" % (src, obj)) + else: + self.mkpath (os.path.dirname (obj)) + if ext == '.rc' or ext == '.res': + # gcc needs '.res' and '.rc' compiled to object files !!! + try: + self.spawn (["windres","-i",src,"-o",obj]) + except DistutilsExecError, msg: + raise CompileError, msg + else: # for other files use the C-compiler + try: + self.spawn (self.compiler_so + cc_args + + [src, '-o', obj] + + extra_postargs) + except DistutilsExecError, msg: + raise CompileError, msg + + # Return *all* object filenames, not just the ones we just built. + return objects + + # compile () + + + # XXXX needs to be checked with executables + def link (self, + target, + objects, + output_filename, + output_dir=None, + libraries=None, + library_dirs=None, + runtime_library_dirs=None, + export_symbols=None, + debug=0, + extra_preargs=None, + extra_postargs=None, + build_temp=None): # use separate copies, so we can modify the lists extra_preargs = copy.copy(extra_preargs or []) @@ -137,46 +198,49 @@ # where are the object files temp_dir = os.path.dirname(objects[0]) - # name of dll to give the helper files (def, lib, exp) the same name - (dll_name, dll_extension) = os.path.splitext( - os.path.basename(output_filename)) + # using export symbols with executables is not tested yet + if target <> UnixCCompiler.EXECUTABLE: + # name of dll to give the helper files (def, lib, exp) the same name + (dll_name, dll_extension) = os.path.splitext( + os.path.basename(output_filename)) - # generate the filenames for these files - def_file = None # this will be done later, if necessary - exp_file = os.path.join(temp_dir, dll_name + ".exp") - lib_file = os.path.join(temp_dir, 'lib' + dll_name + ".a") + # generate the filenames for these files + def_file = None # this will be done later, if necessary + exp_file = os.path.join(temp_dir, dll_name + ".exp") + lib_file = os.path.join(temp_dir, 'lib' + dll_name + ".a") - #extra_preargs.append("--verbose") - if self.linker == "dllwrap": - extra_preargs.extend([#"--output-exp",exp_file, - "--output-lib",lib_file, - ]) - else: - # doesn't work: bfd_close build\...\libfoo.a: Invalid operation - extra_preargs.extend([#"-Wl,--out-implib,%s" % lib_file, - ]) + if self.linker == "dllwrap": + extra_preargs.extend([#"--output-exp",exp_file, + "--output-lib",lib_file, + ]) + else: + # doesn't work: bfd_close build\...\libfoo.a: Invalid operation + extra_preargs.extend([#"-Wl,--out-implib,%s" % lib_file, + ]) - # check what we got in export_symbols - if export_symbols is not None: - # Make .def file - # (It would probably better to check if we really need this, - # but for this we had to insert some unchanged parts of - # UnixCCompiler, and this is not what we want.) - def_file = os.path.join(temp_dir, dll_name + ".def") - contents = [ - "LIBRARY %s" % os.path.basename(output_filename), - "EXPORTS"] - for sym in export_symbols: - contents.append(sym) - self.execute(write_file, (def_file, contents), - "writing %s" % def_file) + # check what we got in export_symbols + if export_symbols is not None: + # Make .def file + # (It would probably better to check if we really need this, + # but for this we had to insert some unchanged parts of + # UnixCCompiler, and this is not what we want.) + def_file = os.path.join(temp_dir, dll_name + ".def") + contents = [ + "LIBRARY %s" % os.path.basename(output_filename), + "EXPORTS"] + for sym in export_symbols: + contents.append(sym) + self.execute(write_file, (def_file, contents), + "writing %s" % def_file) - if def_file: - if self.linker == "dllwrap": - # for dllwrap we have to use a special option - extra_preargs.append("--def") - # for gcc/ld it is specified as any other object file - extra_preargs.append(def_file) + if def_file: + if self.linker == "dllwrap": + # for dllwrap we have to use a special option + extra_preargs.append("--def") + # for gcc/ld it is specified as any other object file + extra_preargs.append(def_file) + + # end: if target <> UnixCCompiler.EXECUTABLE # who wants symbols and a many times larger output file # should explicitly switch the debug mode on @@ -186,7 +250,8 @@ if not debug: extra_preargs.append("-s") - UnixCCompiler.link_shared_object(self, + UnixCCompiler.link(self, + target, objects, output_filename, output_dir, @@ -200,6 +265,34 @@ build_temp) # link_shared_object () + + # -- Miscellaneous methods ----------------------------------------- + + # overwrite the one from CCompiler to support rc and res-files + def object_filenames (self, + source_filenames, + strip_dir=0, + output_dir=''): + if output_dir is None: output_dir = '' + obj_names = [] + for src_name in source_filenames: + # use normcase to make sure '.rc' is really '.rc' and not '.RC' + (base, ext) = os.path.splitext (os.path.normcase(src_name)) + if ext not in (self.src_extensions + ['.rc','.res']): + raise UnknownFileError, \ + "unknown file type '%s' (from '%s')" % \ + (ext, src_name) + if strip_dir: + base = os.path.basename (base) + if ext == '.res' or ext == '.rc': + # these need to be compiled to object files + obj_names.append (os.path.join (output_dir, base + ext + self.obj_extension)) + else: + obj_names.append (os.path.join (output_dir, + base + self.obj_extension)) + return obj_names + + # object_filenames () # class CygwinCCompiler diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/msvccompiler.py distutils.patched/distutils/msvccompiler.py --- distutils.orig/distutils/msvccompiler.py Mon Sep 4 11:58:57 2000 +++ distutils.patched/distutils/msvccompiler.py Wed Sep 13 13:52:26 2000 @@ -201,7 +201,7 @@ version = versions[0] # highest version self.cc = find_exe("cl.exe", version) - self.link = find_exe("link.exe", version) + self.linker = find_exe("link.exe", version) self.lib = find_exe("lib.exe", version) set_path_env_var ('lib', version) set_path_env_var ('include', version) @@ -215,7 +215,7 @@ else: # devstudio not found in the registry self.cc = "cl.exe" - self.link = "link.exe" + self.linker = "link.exe" self.lib = "lib.exe" self.preprocess_options = None @@ -313,45 +313,19 @@ # create_static_lib () - - def link_shared_lib (self, - objects, - output_libname, - output_dir=None, - libraries=None, - library_dirs=None, - runtime_library_dirs=None, - export_symbols=None, - debug=0, - extra_preargs=None, - extra_postargs=None, - build_temp=None): - - self.link_shared_object (objects, - self.shared_library_name(output_libname), - output_dir=output_dir, - libraries=libraries, - library_dirs=library_dirs, - runtime_library_dirs=runtime_library_dirs, - export_symbols=export_symbols, - debug=debug, - extra_preargs=extra_preargs, - extra_postargs=extra_postargs, - build_temp=build_temp) - - - def link_shared_object (self, - objects, - output_filename, - output_dir=None, - libraries=None, - library_dirs=None, - runtime_library_dirs=None, - export_symbols=None, - debug=0, - extra_preargs=None, - extra_postargs=None, - build_temp=None): + def link (self, + target, + objects, + output_filename, + output_dir=None, + libraries=None, + library_dirs=None, + runtime_library_dirs=None, + export_symbols=None, + debug=0, + extra_preargs=None, + extra_postargs=None, + build_temp=None): (objects, output_dir) = self._fix_object_args (objects, output_dir) (libraries, library_dirs, runtime_library_dirs) = \ @@ -369,10 +343,16 @@ if self._need_link (objects, output_filename): - if debug: - ldflags = self.ldflags_shared_debug + if target == CCompiler.EXECUTABLE: + if debug: + ldflags = self.ldflags_shared_debug[1:] + else: + ldflags = self.ldflags_shared[1:] else: - ldflags = self.ldflags_shared + if debug: + ldflags = self.ldflags_shared_debug + else: + ldflags = self.ldflags_shared export_opts = [] for sym in (export_symbols or []): @@ -386,12 +366,13 @@ # needed! Make sure they are generated in the temporary build # directory. Since they have different names for debug and release # builds, they can go into the same directory. - (dll_name, dll_ext) = os.path.splitext( - os.path.basename(output_filename)) - implib_file = os.path.join( - os.path.dirname(objects[0]), - self.library_filename(dll_name)) - ld_args.append ('/IMPLIB:' + implib_file) + if export_symbols is not None: + (dll_name, dll_ext) = os.path.splitext( + os.path.basename(output_filename)) + implib_file = os.path.join( + os.path.dirname(objects[0]), + self.library_filename(dll_name)) + ld_args.append ('/IMPLIB:' + implib_file) if extra_preargs: ld_args[:0] = extra_preargs @@ -400,65 +381,15 @@ self.mkpath (os.path.dirname (output_filename)) try: - self.spawn ([self.link] + ld_args) + self.spawn ([self.linker] + ld_args) except DistutilsExecError, msg: raise LinkError, msg else: self.announce ("skipping %s (up-to-date)" % output_filename) - # link_shared_object () - - - def link_executable (self, - objects, - output_progname, - output_dir=None, - libraries=None, - library_dirs=None, - runtime_library_dirs=None, - debug=0, - extra_preargs=None, - extra_postargs=None): - - (objects, output_dir) = self._fix_object_args (objects, output_dir) - (libraries, library_dirs, runtime_library_dirs) = \ - self._fix_lib_args (libraries, library_dirs, runtime_library_dirs) - - if runtime_library_dirs: - self.warn ("I don't know what to do with 'runtime_library_dirs': " - + str (runtime_library_dirs)) - - lib_opts = gen_lib_options (self, - library_dirs, runtime_library_dirs, - libraries) - output_filename = output_progname + self.exe_extension - if output_dir is not None: - output_filename = os.path.join (output_dir, output_filename) - - if self._need_link (objects, output_filename): - - if debug: - ldflags = self.ldflags_shared_debug[1:] - else: - ldflags = self.ldflags_shared[1:] - - ld_args = ldflags + lib_opts + \ - objects + ['/OUT:' + output_filename] - - if extra_preargs: - ld_args[:0] = extra_preargs - if extra_postargs: - ld_args.extend (extra_postargs) + # link () - self.mkpath (os.path.dirname (output_filename)) - try: - self.spawn ([self.link] + ld_args) - except DistutilsExecError, msg: - raise LinkError, msg - else: - self.announce ("skipping %s (up-to-date)" % output_filename) - # -- Miscellaneous methods ----------------------------------------- # These are all used by the 'gen_lib_options() function, in diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/unixccompiler.py distutils.patched/distutils/unixccompiler.py --- distutils.orig/distutils/unixccompiler.py Tue Aug 8 13:18:38 2000 +++ distutils.patched/distutils/unixccompiler.py Tue Sep 12 16:06:19 2000 @@ -190,45 +190,19 @@ # create_static_lib () - def link_shared_lib (self, - objects, - output_libname, - output_dir=None, - libraries=None, - library_dirs=None, - runtime_library_dirs=None, - export_symbols=None, - debug=0, - extra_preargs=None, - extra_postargs=None, - build_temp=None): - - self.link_shared_object ( - objects, - self.library_filename (output_libname, lib_type='shared'), - output_dir, - libraries, - library_dirs, - runtime_library_dirs, - export_symbols, - debug, - extra_preargs, - extra_postargs, - build_temp) - - - def link_shared_object (self, - objects, - output_filename, - output_dir=None, - libraries=None, - library_dirs=None, - runtime_library_dirs=None, - export_symbols=None, - debug=0, - extra_preargs=None, - extra_postargs=None, - build_temp=None): + def link (self, + target, + objects, + output_filename, + output_dir=None, + libraries=None, + library_dirs=None, + runtime_library_dirs=None, + export_symbols=None, + debug=0, + extra_preargs=None, + extra_postargs=None, + build_temp=None): (objects, output_dir) = self._fix_object_args (objects, output_dir) (libraries, library_dirs, runtime_library_dirs) = \ @@ -253,54 +227,16 @@ ld_args.extend (extra_postargs) self.mkpath (os.path.dirname (output_filename)) try: - self.spawn (self.linker_so + ld_args) - except DistutilsExecError, msg: - raise LinkError, msg - else: - self.announce ("skipping %s (up-to-date)" % output_filename) - - # link_shared_object () - - - def link_executable (self, - objects, - output_progname, - output_dir=None, - libraries=None, - library_dirs=None, - runtime_library_dirs=None, - debug=0, - extra_preargs=None, - extra_postargs=None): - - (objects, output_dir) = self._fix_object_args (objects, output_dir) - (libraries, library_dirs, runtime_library_dirs) = \ - self._fix_lib_args (libraries, library_dirs, runtime_library_dirs) - - lib_opts = gen_lib_options (self, - library_dirs, runtime_library_dirs, - libraries) - output_filename = output_progname # Unix-ism! - if output_dir is not None: - output_filename = os.path.join (output_dir, output_filename) - - if self._need_link (objects, output_filename): - ld_args = objects + self.objects + lib_opts + ['-o', output_filename] - if debug: - ld_args[:0] = ['-g'] - if extra_preargs: - ld_args[:0] = extra_preargs - if extra_postargs: - ld_args.extend (extra_postargs) - self.mkpath (os.path.dirname (output_filename)) - try: - self.spawn (self.linker_exe + ld_args) + if target == CCompiler.EXECUTABLE: + self.spawn (self.linker_exe + ld_args) + else: + self.spawn (self.linker_so + ld_args) except DistutilsExecError, msg: raise LinkError, msg else: self.announce ("skipping %s (up-to-date)" % output_filename) - # link_executable () + # link () # -- Miscellaneous methods ----------------------------------------- --------------2B7CFCB61910F88B9072AE46-- From thomas.heller@ion-tof.com Thu Sep 14 11:04:01 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Thu Sep 14 10:04:01 2000 Subject: [Distutils] More minor issues References: <39BE09E2.F699662B@gmx.de> <20000912212202.A1082@beelzebub> <39C09744.133BDC89@gmx.de> Message-ID: <002c01c01e54$800807f0$4500a8c0@thomasnb> [Rene] > In the CVS are the source files for the windows installer. > So everyone can create his own, I think most people > would like to change the bitmap (PythonPowered) and use > a more specific bitmap for their package. I would happily accept a better (general) bitmap! > Creating the installer executable is not the problem, but > to use it you have to change bdist_wininst.py. > (If you only maintain one package this is not a problem > but if there are more, you had to change it everytime > you want to create an installer for the other package.) > Recompiling the installer executable only to change the bitmap seems overkill to me. You could use win32api calls (if available) to replace the bitmap in the executable without need for a compiler, better would be to pack the bitmap into the installer archive at runtime. This could also be done on Linux. Thomas From gward@python.net Thu Sep 14 21:58:04 2000 From: gward@python.net (Greg Ward) Date: Thu Sep 14 20:58:04 2000 Subject: [Distutils] BeOS library location? Message-ID: <20000914205621.A1068@beelzebub> Hi all -- just noticed the BeOS-specific code in sysconfig._init_posix(): g['LDSHARED'] = ("%s -L%s/lib -lpython%s" % (linkerscript, PREFIX, sys.version[0:3])) Should that be EXEC_PREFIX? I dunno in one would ever do a prefix != exec-prefix installation on BeOS, but it seems like good practice to respect the "exec-prefix for binaries" convention in any event. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Thu Sep 14 22:53:02 2000 From: gward@python.net (Greg Ward) Date: Thu Sep 14 21:53:02 2000 Subject: [Distutils] Re: Distutils documentation Mac mods In-Reply-To: <20000913090239.00289303181@snelboot.oratrix.nl>; from jack@oratrix.nl on Wed, Sep 13, 2000 at 11:02:38AM +0200 References: <20000913090239.00289303181@snelboot.oratrix.nl> Message-ID: <20000914215049.C1016@beelzebub> On 13 September 2000, Jack Jansen said: > Greg, > I've cc-d Corran on this reply, it's a bit silly to keep two conversations > going with similar content. Don't worry, we'll keep our esoteric Mac-talk to > ourselves:-) And I'm bringing the whole SIG in on this -- surely you two aren't the only Mac hackers in the world! [I opined:] > I like the model we've used for Windows installers: developers and > packagers have to put up with a command line interface -- ie. open a > "DOS box" and run "python setup.py bdist_wininst" -- but then end users > have a nice little Windows-friendly executable installer. They're not > even aware of such a thing as the Distutils, which is how it should be > for end-users. [Jack replies] > Ok, that's fine. From reading the document I got the impression that the > various command-line switches to install (in the end-user situation) would > actually be necessary relatively often, but you're saying that they're rare, > and that the end user will normally just do "python setup.py". Right? Not quite. You're right that the options to install will not be needed very often, especially on single-user systems with no file permissions. But the normal case is "python setup.py install" -- you CANNOT use the Distutils without providing at least one argument to the setup script. The rationale for this reveals my deep-seated Unix geek prejudice: how am I supposed to find out what this program does? Well, I run it without arguments and see what it says. As long as it's not a filter (remember trying to learn how to use grep before you understood what a filter was?), you're fine. Well-behaved command-line programs give you a usage summary when run with bad args or no args, and any Distutils setup script automatically plays by those rules. This is bad on Mac OS, where providing *any* command-line args is a pain. However, I have a cunning plan... When a setup script is run under Mac OS, instead of printing a usage summary to stdout and exiting, throw up a dialog with the user's options -- ie. what commands they can run. You can get a barebones list of commands from distutils.command.__all__, and a list of all commands (including non-standard commands provided by the current setup script) by poking around in the Distribution object. See 'print_commands()' in dist.py for an example, and 'print_command_list()' for how to get the description of each command in turn. (The command module has to be imported, which is why running "setup.py --help-commands" is slow and chunky: it has to import an entire module for each line of help text!) However, the barebones list is probably sub-optimal; you'd want to shuffle it around a bit so "install", "build", "clean", "sdist", and "bdist" are up at the top. Then the sub-commands can come below, grouped as they already are (which is a function of the ordering of distutils.command.__all__). This reshuffling is purely a usability issue; 90% of users will just run "install" 90% of the time, so it should be first, and checked by default. Eg. here's a rough sketch of the dialog I have in mind: +-------------------------------------------------------------+ | (x) install install everything from build directory | | ( ) build build everything needed to install | | ( ) clean clean up output of 'build' command | | ( ) sdist create a source distribution ... | | ( ) bdist create a built (binary) distribution | | ----------------------------------------------------------- | | ( ) build_py ... | | ( ) build_ext ... | | ( ) build_clib ... | | ( ) build_scripts ... | | ( ) install_lib ... | | ( ) install_headers | | ( ) ... | | ( ) bdist_dumb | | ( ) ... | | | | [ Ok ] [ Cancel ] | +-------------------------------------------------------------+ (Memo to myself: if this ordering makes more sense in a GUI dialog, it makes more sense in the output of --help-commands. Hmmm.) I *think* the commands should be radiobuttons, even though you can run multiple commands in one go, because of the need to associate command-line options with each command. Hmmm, maybe not. If you pick multiple commands *and* choose to "Supply additional command-line arguments" (this would probably have to be another checkbox, off by default), then you could get a dialog with space for multiple argument lists: Arguments for: build_clib ____________________________ install_headers ____________________________ if the user selected to run "build_clib" and "install_headers". You might want to use distutils.util.split_quoted() to parse those argument strings into words, in case -- gasp! -- people use directories with spaces in them. (Which will have to be quoted -- ugh! I can just see Joe Mac User saying, "Whaddya mean, the standard shell quoting convention?!??!") Of course, if the user doesn't check the "Supply additional command-line arguments" dialog, then the requested command(s) immediately run. stdout should be preserved and presented to the user in a big scrolling window -- yeah it's ugly, yeah it'll look like someone took a big ol' Unix program and ported to Mac OS as quickly and cheaply as possible. But that's what (will have) happened! So whaddya think? How outrageously horrible and Unix-oid is that? I figure it's not too bad as long as the user doesn't want to supply any extra args, and I don't think that'll be necessary for most end-user installation situations. > Mac installers don't have to worry about HKEY_USER and HKEY_MACHINE and the > difference between OS version X and OS version Y and such: there's always a > single place to put things, and if that somehow changes "the old way" > continues to work. Usually through Apple-supplied magic, sometimes through > VISE-supplied magic. What, you mean you have a *sensible* operating system? I thought they were banned by the secret Microsoft World Domination Act of 1994 (passed by the hidden One World Government, of course). (*wink*... I think.) The issue of where to install header files affects other platforms as well, so I think I'll bring that up in a new thread on the SIG. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Thu Sep 14 23:04:01 2000 From: gward@python.net (Greg Ward) Date: Thu Sep 14 22:04:01 2000 Subject: [Distutils] Re: Distutils under Mac OS In-Reply-To: ; from cwebster@nevada.edu on Wed, Sep 13, 2000 at 02:54:41PM -0700 References: <20000913090239.00289303181@snelboot.oratrix.nl> Message-ID: <20000914220135.D1016@beelzebub> [cc'ing the SIG again, for the same reason] On 13 September 2000, Corran Webster said: > At the moment to install you have to use the command line interface - > you can't just double click on setup.py. What Greg is saying is that > one way to avoid this on the mac side is to have some version of the > bdist command which builds an intelligent installer of some sort (for > example, it may bundle everything up into a VISE installer) which > then puts everything in the right place. That only helps installers -- see my previous post for a possible way to help all Distutils users (developers and packagers too). > I can see two slightly simpler ways of dealing with this. The > easiest to implement is to have an applet which you can just drop > setup.py files on and it handles everything from there (easy to do > thanks to the "run_setup" function - I have a 5 line applet which > does this now). This "Install package" applet could be added to the > standard mac distribution of Python much like the "Build applet" and > "Build application" applets are now. Hmm, neat idea. You could even make your "Install package" applet smart enough to unpack a Distutils-style archive and dig up setup.py there -- just download a source release off the net, drop it onto the applet, and away-we-go. How's *that* for ease of use? ;-) > A slightly more sophisticated approach would be to have a bdist-type > command which creates an installer applet for the package, and then > bundles everything up in a zip/tar/stuffit file. All the end users > would just unpack the archive and run the applet and it would take > care of everything automatically. That might be more sophisticated, but it sounds inferior to the previous option in usability terms: the user has to unpack the archive and "do something" with what he finds there. > An even more sophisticated approach would bundle everything up using > zip/tar/stuffit and then store the data inside a python script > somehow (a really big string?). This script would know how to unpack > and install the archived data. This script would then be turned into > an applet (or even a standalone application) and distributed - > essentially making a pure python poor man's version of VISE. This is essentially what Thomas Heller's bdist_wininst does, except the wrapper is a C program -- really a self-expanding ZIP file with a bit of GUI to make it look pretty. Despite the fact that I have to carry around C source and a Windows .exe in the Distutils CVS and source distribution, this strikes me as more elegant than making a big Python script with a little bit of code and a lot of encoded ZIP/StuffIt/whatever data. > The most sophisticated would be to package everything up in a VISE > Installer. Jack would know far more than I about how > possible/practical that would be. The original idea for Windows was to generate WISE installers. (Any relation to VISE?) The impetus for this has dropped off with the success of the bdist_wininst command; it sounds like not many module distributions need the sophistication of a full-blown commercial installer. I suspect the same would hold on Mac OS. However, when it comes time to deal with full-blown Python applications, we may have to reexamine the issue. Again, I'll deal with header file installation in a separate post. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Thu Sep 14 23:07:02 2000 From: gward@python.net (Greg Ward) Date: Thu Sep 14 22:07:02 2000 Subject: [Distutils] Re: Distutils on Mac OS In-Reply-To: <20000914083321.3D6C5303181@snelboot.oratrix.nl>; from jack@oratrix.nl on Thu, Sep 14, 2000 at 10:33:20AM +0200 References: <20000914083321.3D6C5303181@snelboot.oratrix.nl> Message-ID: <20000914220446.E1016@beelzebub> On 14 September 2000, Jack Jansen said: > is there a reason that simply starting setup.py doesn't do an install? > That may be the easiets way to go... From README.txt: """The Python Distribution Utilities, or Distutils for short, are a collection of modules that aid in the development, distribution, and installation of Python modules.""" Installation isn't the only game in town here! ;-) [Corran's idea] > I can see two slightly simpler ways of dealing with this. The > easiest to implement is to have an applet which you can just drop > setup.py files on and it handles everything from there (easy to do > thanks to the "run_setup" function - I have a 5 line applet which > does this now). [Jack responds warmly] > I think that of the solutions listed I like this one best. VISE is > overkill (and moreover will make it difficult for people to create a > distribution on a whim, because they'll have to get VISE first) and > the distribute-as-applet has potential backward/forward compatibility > problems. > > But I think that the solution I really like best is that we just whip up a > simple user interface for setup and put that in the distutils package... Yeah, as I implied in my last post, I prefer this option too. (Easy for me to say, from my impregnable bastion of ignorance about Mac OS.) Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Thu Sep 14 23:28:01 2000 From: gward@python.net (Greg Ward) Date: Thu Sep 14 22:28:01 2000 Subject: [Distutils] Brewing controversy over header file installation Message-ID: <20000914222612.A1500@beelzebub> Jack Jansen has been hacking on the Mac OS support in the Distutils, and brings up an interesting -- but annoying point: installing extension-related header files under Python's include directory could be perceived as interfering with Python's source. I disagree, but let me see if I can summarize what's been going on in email between Jack, Corran Webster (the other person working on Distutils Mac support), and me. I mentioned how files are installed on Mac OS (by quoting the source, of course): > 'mac': { > 'purelib': '$base:Lib:site-packages', > 'platlib': '$base:Lib:site-packages', > 'headers': '$base:Include:$dist_name', > 'scripts': '$base:Scripts', Regarding the 'headers' install dir -- where extension-related header files go -- Jack says: > This one is wrong. Where do you put these things on Unix? I don't > particularly like it if distutils scripts go mucking about in my > source directories (which is what $base:Include would be). On Unix, the answer is: 'unix_prefix': { 'purelib': '$base/lib/python$py_version_short/site-packages', 'platlib': '$platbase/lib/python$py_version_short/site-packages', 'headers': '$base/include/python$py_version_short/$dist_name', 'scripts': '$base/bin', 'data' : '$base', }, In the "prefix" installation scheme, $base expands to sys.prefix, so in a standard Python 2.0 installation on Unix, the 'headers' install dir is /usr/local/include/python2.0/$dist_name. $dist_name in turn expands to the name of the Distribution -- "Numerical", "Imaging", whatever. So yes, we are installing files *under* one of Python's directories, but not *into* it -- the include/python2.0 directory continues to provide just the header files installed by Python, namely Python.h and friends (abstract.h, bitset.h, ..., unicodeobject.h). And that's why I think this isn't a bad thing to do. The real reason it was done this way, of course, was laziness. Any time you compile a Python extension, you have to include "-I/usr/local/include/python2.0" (or local equivalent). With Numerical Python's headers installed under this location, an extension that needs access to Numerical's C structures just has to say #include and it will Just Work. Jack had an interesting suggestion: > Moreover, for packages to use one another I would guess that more than just > include files is needed (like the location of the shared libraries). I think > that my preference (but I haven't thought about it very long) would be to put > "distribution data for other distributions" all together in a single place, > something like > $base/Extensions/packagename/Include > $base/Extensions/packagename/Lib If the shared libraries that go in Extensions/$dist_name/Lib (Jack misspelled it ;-) are C libraries, that's fine -- if they're Python extensions, I'm confused. This makes sense, but it does require adding another directory to the standard include search path when building Python extensions. And the directory would probably be named something different on Unix. But it's in interesting idea, and I'm not going to throw it out willy-nilly. What does everyone else think? Corran's response to this was: > I agree, though, that this would be a natural way to go. It might > make more sense to have it all inside the site-packages directory, > though: headers live in > > $base:Lib:site-packages:$package:Include > > while modules (both python and C shared libraries) live in > > $base:Lib:site-packages:$package: I disagree: site-packages should be for Python modules and extensions. If we have to install C headers and libraries -- and I suspect we probably do -- then they should find their own home. On Unix, perhaps another directory under /lib/python2.0, but I'm not sure what to call it that makes it clear these are for *C* programming, not Python programming. (Granted it's the specialized domain of C programming for developing Python extensions, but still.) On Mac OS and Windows, it's easy -- I'd say put 'em in Extensions/Includes and Extensions/Lib under (where that / is really a : or a \, of course). Jack later follows up: > Ok, so then we have (assuming we're running in a source tree) an "Include" > which contains standard Python include files and "includes" which contains > package-deposited include files. I'm not too thrilled with the similar names, > but I guess I can live with this. Yechh -- I would be *very* wary of depending on case distinction in filenames! I still don't see what's so wrong with Include/*.h being standard Python headers, and Include/*/*.h being headers provided by whatever extensions are installed on the system. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From mfletch@tpresence.com Fri Sep 15 05:11:01 2000 From: mfletch@tpresence.com (Mike Fletcher) Date: Fri Sep 15 04:11:01 2000 Subject: [Distutils] Help: request for assistance with PyOpenGL distribution using dis tutils on NT Message-ID: Background: While stuck on my real work, I'm trying to get a PyOpenGL distribution (1.5.6alpha1 from CVS) working using distutils. Rene Liebscher kindly created an installation script that seems to work for him on Win32 for the 1.5.5 PyOpenGL, and for Jason Petrone (on Linux) using the 1.5.6 source from CVS. I'm working on NT 4.0 sp3, using VC++ 6.0sp3, (I get similar effects on Cygnus, but since I'm not sure the package builds under that compiler I'm not focussing on it yet). When trying to build, I get a long set of errors (see below), the elimination of which is well beyond my meagre VC++ or distutils skills to fix. According to MSDN, the problems I'm getting are normally caused by having a C++ file compiled as a C file. To get around it, you apparently put an ifdef at the top and bottom of the file, but the file concerned (GL.h) has this check, and still reports the errors. I'm assuming there's some flag that needs to be set somewhere, but I don't know what it is. Note: everything builds fine using the old setup script for the latest CVS version on this machine (the one for which I'm trying to get a distutils script). D:\bin\lang\studio\VC98\BIN\cl.exe /c /nologo /Ox /MD /W3 /GX -Isrc -ID:\bin\lang\Python\Include /Tcsrc/_glumodule.c /Fobuild\temp.win32\Release\src/_glumodule.obj Doesn't work. The GUI under the old BUILD.py uses: /nologo /MD /W3 /GX /O2 /Ob2 /I "D:\bin\lang\Python\include" /D "WIN32" /D "_WINDOWS" /D "NDEBUG" /Fp"tmp\./_glu.pch" /YX /Fo"tmp\./" /Fd"tmp\./" /FD /c and works fine. GL.h has the following code near the top (and close at the bottom): #ifdef __cplusplus extern "C" { #endif Which, according to MSDN, should avoid the errors, but doesn't seem to. Here's the output from running setup.py build... S:\PyOpenGL>setup build running build running build_py warning: build_py: package init file 'OpenGL\__init__.py' not found (or not a re gular file) warning: build_py: package init file 'OpenGL\Demo\__init__.py' not found (or not a regular file) warning: build_py: package init file 'OpenGL\Demo\da\__init__.py' not found (or not a regular file) warning: build_py: package init file 'OpenGL\Demo\dek\__init__.py' not found (or not a regular file) warning: build_py: package init file 'OpenGL\Demo\dek\OglSurface\__init__.py' no t found (or not a regular file) warning: build_py: package init file 'OpenGL\Demo\srenner\__init__.py' not found (or not a regular file) warning: build_py: package init file 'OpenGL\Demo\srenner\Images\__init__.py' no t found (or not a regular file) warning: build_py: package init file 'OpenGL\Demo\tom\__init__.py' not found (or not a regular file) warning: build_py: package init file 'OpenGL\GL\__init__.py' not found (or not a regular file) warning: build_py: package init file 'OpenGL\GLU\__init__.py' not found (or not a regular file) warning: build_py: package init file 'OpenGL\GLUT\__init__.py' not found (or not a regular file) warning: build_py: package init file 'OpenGL\Tk\__init__.py' not found (or not a regular file) warning: build_py: package init file 'OpenGL\shared\__init__.py' not found (or n ot a regular file) running build_ext skipping '_opengl' extension (up-to-date) skipping 'openglutil' extension (up-to-date) building '_glu' extension D:\bin\lang\studio\VC98\BIN\cl.exe /c /nologo /Ox /MD /W3 /GX -Isrc -ID:\bin\lan g\Python\Include /Tcsrc/_glumodule.c /Fobuild\temp.win32\Release\src/_glumodule. obj _glumodule.c src\GL/gl.h(1135) : error C2054: expected '(' to follow 'WINGDIAPI' src\GL/gl.h(1135) : error C2085: 'APIENTRY' : not in formal parameter list src\GL/gl.h(1135) : error C2146: syntax error : missing ',' before identifier 'g lAccum' src\GL/gl.h(1135) : error C2143: syntax error : missing ';' before '(' src\GL/gl.h(1135) : error C2059: syntax error : ')' src\GL/gl.h(1136) : error C2054: expected '(' to follow 'WINGDIAPI' src\GL/gl.h(1136) : error C2085: 'APIENTRY' : not in formal parameter list src\GL/gl.h(1136) : error C2146: syntax error : missing ',' before identifier 'g lAlphaFunc' src\GL/gl.h(1136) : error C2143: syntax error : missing ';' before '(' src\GL/gl.h(1136) : error C2059: syntax error : ')' src\GL/gl.h(1137) : error C2054: expected '(' to follow 'WINGDIAPI' ... (100 errors, then stops due to having so many) Does anyone have any suggestions on how to fix this (keeping in mind that I'm not a C/C++ programmer, so the instructions should be pretty basic)? Thoughts appreciated, Mike __________________________________ Mike C. Fletcher Designer, VR Plumber http://members.home.com/mcfletch From calvin@cs.uni-sb.de Fri Sep 15 05:22:01 2000 From: calvin@cs.uni-sb.de (Bastian Kleineidam) Date: Fri Sep 15 04:22:01 2000 Subject: [Distutils] Minor issues In-Reply-To: <020901c01e48$19659460$4500a8c0@thomasnb> Message-ID: >> 3) bdist_wininst does not seem to install data files and scripts. >> If you execute "strings" with the .exe, you will only see Python module >> files. >See separate post. > >BTW: On which system did you create the installer? Debian Linux 2.2, Python 1.5.2 >> And I did not found a file with the install log. The installer just >> copied the files in the Python dir. >Why do you want this file? When I did not found the data files I was curious what had been installed (and where). Bastian From thomas.heller@ion-tof.com Fri Sep 15 05:42:01 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri Sep 15 04:42:01 2000 Subject: [Distutils] Minor issues References: Message-ID: <00e601c01ef0$ad98caf0$4500a8c0@thomasnb> > >> 3) bdist_wininst does not seem to install data files and scripts. > >> If you execute "strings" with the .exe, you will only see Python module > >> files. > >See separate post. > > > >BTW: On which system did you create the installer? > Debian Linux 2.2, Python 1.5.2 The problem is that install, which is run inside bdist_wininst, creates an installation tree in the build directory, and uses the unix installation scheme for it, while bdist_wininst should force the NT installation scheme. On NT, data and scripts are subdirectories of lib, on unix, it is different. Thomas From mwa@gate.net Fri Sep 15 11:05:01 2000 From: mwa@gate.net (Mark W. Alexander) Date: Fri Sep 15 10:05:01 2000 Subject: [Distutils] Brewing controversy over header file installation In-Reply-To: <20000914222612.A1500@beelzebub> Message-ID: On Thu, 14 Sep 2000, Greg Ward wrote: > I disagree: site-packages should be for Python modules and extensions. > If we have to install C headers and libraries -- and I suspect we > probably do -- then they should find their own home. On Unix, perhaps > another directory under /lib/python2.0, but I'm not sure > what to call it that makes it clear these are for *C* programming, not > Python programming. (Granted it's the specialized domain of C > programming for developing Python extensions, but still.) I disagree as well. To vastly over-simplify, why would a python package be installing headers and libraries for C only? Just because someone is installing stuff to help them code python, doesn't mean they want to load everything necessary for C development. This is why the bdist stuff distributes compiled libraries, so the installer doesn't have to deal with rebuilding. So, if someone wants to provide C interfaces, it should be done seperate from the python interfaces, obviously with the python package dependent on the C libraries. The C interface should also have seperate run-time and development sub-packages, so most people, including those using the python package would only require the run-time. For example,you have gkt+, gtk+-devel, and python-pygtk. python-pygtk would require gkt+, but not gtk+-devel. I think getting distutils involved with C headers and (C only) libraries is going beyond it's scope without good cause. Mark Alexander mwa@gate.net From akuchlin@mems-exchange.org Fri Sep 15 11:21:01 2000 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Fri Sep 15 10:21:01 2000 Subject: [Distutils] Brewing controversy over header file installation In-Reply-To: ; from mwa@gate.net on Fri, Sep 15, 2000 at 10:04:53AM -0400 References: <20000914222612.A1500@beelzebub> Message-ID: <20000915102033.C9605@kronos.cnri.reston.va.us> On Fri, Sep 15, 2000 at 10:04:53AM -0400, Mark W. Alexander wrote: >I disagree as well. To vastly over-simplify, why would a python >package be installing headers and libraries for C only? Just Counterexample: ExtensionClass from Digital Creations is a Python-only C module for writing Python classes in C. It needs to install an ExtensionClass.h, which is then used by other C modules (such as ZODB's cPersistence, I think installing the files under the Python include directory is fine, at least on Unix, since it means if C code can find Python.h, it can also find ExtensionClass/ExtensionClass.h. I'm not sure what to do for Jack's problem. --amk From mwa@gate.net Fri Sep 15 11:51:01 2000 From: mwa@gate.net (Mark W. Alexander) Date: Fri Sep 15 10:51:01 2000 Subject: [Distutils] Brewing controversy over header file installation In-Reply-To: <20000915102033.C9605@kronos.cnri.reston.va.us> Message-ID: On Fri, 15 Sep 2000, Andrew Kuchling wrote: > On Fri, Sep 15, 2000 at 10:04:53AM -0400, Mark W. Alexander wrote: > >I disagree as well. To vastly over-simplify, why would a python > >package be installing headers and libraries for C only? Just > > Counterexample: ExtensionClass from Digital Creations is a Python-only > C module for writing Python classes in C. It needs to install an > ExtensionClass.h, which is then used by other C modules (such as > ZODB's cPersistence, Then maybe I'm lost, but wouldn't the same concept apply? The only "users" of the C headers/libs would be other C developers; even if they are doing C development for python. Do python users who use the Extension class need the headers, or just the shared libraries that distutils already handles properly? > I think installing the files under the Python include directory is > fine, at least on Unix, since it means if C code can find Python.h, it > can also find ExtensionClass/ExtensionClass.h. I'm not sure what to > do for Jack's problem. Since the python tree already contains stuff to do C modules, I don't see anything at all "wrong" with putting additional stuff there for use by C developers. I'm just questioning whether it's something that distutils should be responsible for. I think the answer is a definite maybe. It seems like a lot of effort to provide something that may not be a good fit. Mark Mark From calvin@cs.uni-sb.de Fri Sep 15 18:14:01 2000 From: calvin@cs.uni-sb.de (Bastian Kleineidam) Date: Fri Sep 15 17:14:01 2000 Subject: [Distutils] Distutils on Mac In-Reply-To: <20000914220135.D1016@beelzebub> Message-ID: >However, when it comes time to deal with full-blown Python applications, >we may have to reexamine the issue. I am distributing for some time now my Linkchecker with Distutils which is a "full blown" app (data files, python module files, pyhton packages, scripts, and C-extensions). It installs nice on Unix and Windows boxen. Feel free to play with it to test the Distutils on a Mac. Homepage at linkchecker.sourceforge.net. Bastian From gward@python.net Fri Sep 15 22:40:01 2000 From: gward@python.net (Greg Ward) Date: Fri Sep 15 21:40:01 2000 Subject: [Distutils] Brewing controversy over header file installation In-Reply-To: ; from mwa@gate.net on Fri, Sep 15, 2000 at 10:50:55AM -0400 References: <20000915102033.C9605@kronos.cnri.reston.va.us> Message-ID: <20000915213820.A776@beelzebub> On 15 September 2000, Mark W. Alexander said: > Since the python tree already contains stuff to do C modules, I > don't see anything at all "wrong" with putting additional stuff > there for use by C developers. I'm just questioning whether it's > something that distutils should be responsible for. I think the > answer is a definite maybe. It seems like a lot of effort to > provide something that may not be a good fit. Installing C headers is definitely something the Distutils has to do. Just as people write modules that depend on other modules, they also write extensions that depend on other extensions. In C, that kind of dependence usually means two things: include a header file and link against a library. In the specialized domain of writing Python extensions in C, I *think* all the header file is the only problem; the linking issue will be taken care of when both extensions are loaded into the Python interpreter. (This is merely informed speculation; I've never actually written an extension that depends on facilities provided by another extension! Things could well be different on platforms that don't follow the usual Unix model of shared libraries, where .so files can depend on each other, but the dependencies don't have to be resolved until load-time.) That's why the "install" command can install C header files (using the "install_headers" command). We install .py and .so/.pyd files because they will be needed by Python programmers writing Python modules and applications; we install .h files because they might be needed by C programmers writing Python extensions. We doesn't install any C libraries, either shared or static, because I have not yet seen a need for them. Greg From gward@python.net Fri Sep 15 22:45:10 2000 From: gward@python.net (Greg Ward) Date: Fri Sep 15 21:45:10 2000 Subject: [Distutils] Minor issues In-Reply-To: ; from calvin@cs.uni-sb.de on Mon, Sep 11, 2000 at 11:19:12AM +0200 References: Message-ID: <20000915214324.B776@beelzebub> On 11 September 2000, Bastian Kleineidam said: > 1) Why is the --formats list comma-separated and the --include-dirs and > --library-dirs lists not? They are separated by os.pathsep. > Well, at least its documented :) Just because. No seriously: MS-DOS and Unix each have well-established conventions for lists of directories; semicolon is the separator under DOS, and colon under Unix. So we use the local convention. The formats list is just a list of strings, and I've always thought that comma is a perfectly good delimiter for lists of strings. > 2) The os.pathsep separator is not documented in the help text: > > python setup.py build_ext --help Fixed, thanks. Greg From gward@python.net Fri Sep 15 23:50:09 2000 From: gward@python.net (Greg Ward) Date: Fri Sep 15 22:50:09 2000 Subject: [Distutils] More minor issues In-Reply-To: <39C09744.133BDC89@gmx.de>; from R.Liebscher@gmx.de on Thu, Sep 14, 2000 at 11:15:48AM +0200 References: <39BE09E2.F699662B@gmx.de> <20000912212202.A1082@beelzebub> <39C09744.133BDC89@gmx.de> Message-ID: <20000915224829.C776@beelzebub> On 14 September 2000, Rene Liebscher said: > It is only build.py, all other command get their paths from build. > I changed only the platform specific directories. > > lib.linux2 -> lib-1.5.linux2 > temp.linux2 -> temp-1.5.linux2 > > If you try this with distutils, you will not see any changes, > because distutils is a pure-python package. > (file: lib-x.x.patch) OK, thanks -- I reworked your patch a bit and checked it in. Note that with the recent changes to util.get_platform(), those directories are actually named (eg.) "lib-1.5-linux-i586" and "temp-1.5-linux-i586". (I changed your . to a -.) Two things to note: * if we ever start byte-compiling at build-time, rather than install- time, then the Python version will matter for pure Python distributions * the "bdist" command also generates "platform-dependent" directory names; I can't think of any reason to put the Python version in them, but I could be missing something. It's a wee bit inconsistent to have "lib-1.5-linux-i586" next to "bdist.linux-i586". Arg! I just noticed more -/. confusion there. Now it's "lib.linux-i586-1.5". > In the CVS are the source files for the windows installer. > So everyone can create his own, I think most people > would like to change the bitmap (PythonPowered) and use > a more specific bitmap for their package. I agree, but it does seem silly to create a new executable for every bitmap. Why not fix things so you can supply a bitmap file? (But being able to specify the exe file seems useful in its own right, so don't throw that patch away!) I'll let you and Thomas work it out, and Thomas can check it in when he's happy with it. > python setup.py bdist --formats=zip,gztar > > doesn't work. It deletes the build tree after > creating the first archive. Son of a gun, so it doesn't. I thought I tested that... sigh... > There is a simple solution for that. Unfortunately, the bug is deeper than that. Your patch is a nice optimization, but this should work without it (redoing the build and install for each format). I think the blame lies on bdist_dumb, but it needs some help from Command and Distribution to get everything right. Working on it now, but I need sleep... If I get it working, then I'll see about applying your optimization patch. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From mwa@gate.net Sat Sep 16 00:49:01 2000 From: mwa@gate.net (Mark W. Alexander) Date: Fri Sep 15 23:49:01 2000 Subject: [Distutils] Brewing controversy over header file installation In-Reply-To: <20000915213820.A776@beelzebub> Message-ID: On Fri, 15 Sep 2000, Greg Ward wrote: > On 15 September 2000, Mark W. Alexander said: > > Since the python tree already contains stuff to do C modules, I > > don't see anything at all "wrong" with putting additional stuff > > there for use by C developers. I'm just questioning whether it's > > something that distutils should be responsible for. I think the > > answer is a definite maybe. It seems like a lot of effort to > > provide something that may not be a good fit. > > Installing C headers is definitely something the Distutils has to do. > Just as people write modules that depend on other modules, they also > write extensions that depend on other extensions. In C, that kind of > dependence usually means two things: include a header file and link > against a library. In the specialized domain of writing Python > extensions in C, I *think* all the header file is the only problem; the > linking issue will be taken care of when both extensions are loaded into > the Python interpreter. (This is merely informed speculation; I've > never actually written an extension that depends on facilities provided > by another extension! Things could well be different on platforms that > don't follow the usual Unix model of shared libraries, where .so files > can depend on each other, but the dependencies don't have to be resolved > until load-time.) Ok, I WAS taking my position for the sake of simplicity. Now I'm just confused. Distutils installs headers, but not libraries, because a C module developer may be making a module that requires another module. But, if he's only got the headers of the required module, what does he link against? If I follow what you're saying, you're suggesting that somehow this occurs when python loads both modules. I don't think so. If the required module has it's own headers, it follows that it must also have it's own library which actually implements the functions defined in the headers. Python's really good, but it can't resolve symbols that don't exist, so it needs both libraries to resolve the dependencies. So, it doesn't make sense to me to have headers without libraries. The exceptions to this being the headers that provide the python extension api. I just double checked, and on my system, those and libpython are in the python-devel package. So even those are not required on the typical user machine. Why would they need headers for any other C mods they install? > That's why the "install" command can install C header files (using the > "install_headers" command). We install .py and .so/.pyd files because > they will be needed by Python programmers writing Python modules and > applications; we install .h files because they might be needed by C > programmers writing Python extensions. We doesn't install any C > libraries, either shared or static, because I have not yet seen a need > for them. I'm only providing opinions here, not code, so I'm very easily out-voted, but I think distutils should do neither or both. I still feel that the libraries are better provided by a generic (non-python) run-time package, the headers (and usage docs) by a devel package, and the python implementation by distutils. My bigest concern is that if distutils starts doing the devel stuff, why shouldn't is do the run-time stuff .....It's just a matter of time before distutils becomes the universal build/install/package everything tool. I recommend you don't take that first step. If you're going to do headers, though, then you need to do the libraries that implement them also. This would be a nice feature for developers who want to share C modules, but I tend to think they would just to tarballs or cvs. Mark From gward@python.net Sat Sep 16 13:10:01 2000 From: gward@python.net (Greg Ward) Date: Sat Sep 16 12:10:01 2000 Subject: [Distutils] More minor issues In-Reply-To: <39C09744.133BDC89@gmx.de>; from R.Liebscher@gmx.de on Thu, Sep 14, 2000 at 11:15:48AM +0200 References: <39BE09E2.F699662B@gmx.de> <20000912212202.A1082@beelzebub> <39C09744.133BDC89@gmx.de> Message-ID: <20000916120820.A2385@beelzebub> On 14 September 2000, Rene Liebscher said: > Another problem: > > python setup.py bdist --formats=zip,gztar > > doesn't work. It deletes the build tree after > creating the first archive. OK, I've fixed the bug. The problem was that when you ask to reinitialize the "install" command, it doesn't reinitialize "install_lib", "install_headers", etc. Part of reinitializing a command is resetting its "have run" flag to 0, so on the second run of bdist_dumb, it looked like "install" had not run, but "install_lib" had. So nothing actually got installed, and the directory that was supposed to become the root of the archive didn't even exist. The solution was to add a 'reinit_subcommands' flag to Distribution's 'reinitialize_command()' method. Making this work required promoting the "sub-commands" machinery -- which already had informal standing in the "install" command -- up to the Command base class. Now the Distutils have formal recognition of command "families", something that I've known for a while is needed, but hadn't gotten around to until now. > There is a simple solution for that. > First find out all commands to run. > Then execute all commands, check for every command > if it has to be run later again (eg. bdist_dumb > with other format), if it has, check if there > is a keep_tree option and set it. OK, I applied your patch as an optimization. It's not essential at all, but what the hell. Also, I renamed --keep-tree to --keep-temp in the two bdist commands that had it (bdist_dumb and bdist_wininst). And I changed bdist_rpm so it, too, has a --keep-tree option (it used to have --clean/--no-clean). Now, all bdist commands must support the --keep-tree option, so there's no need for bdist.py to check if it exists before assigning to it. Greg From gward@python.net Sat Sep 16 22:13:01 2000 From: gward@python.net (Greg Ward) Date: Sat Sep 16 21:13:01 2000 Subject: [Distutils] New snapshot needs testing Message-ID: <20000916211055.A3907@beelzebub> Hi all -- I've just put out a new code snapshot that needs some widespread testing. (More so than usual, I mean.) This is mainly because I tweaked the sysconfig module to parse Python's makefile lazily rather than eagerly, and to store what it finds in a dictionary rather than global variables. This facility of sysconfig was only used in a few places throughout the Distutils, and I'm pretty sure I got all of them -- as usual, It Works For Me (TM). But I just have my one Linux box at home; at least now I test with Python 1.6 as well as 1.5.2. (And I have that enormous RPM of 2.0b1 ready to install one of these days...) So please, download tonight's snapshot and give it a good workout -- build stuff, lots of stuff, on lots of platforms. Please! Oh, a neat feature added today: I devised a brilliant and entirely original new method for describing extensions to build. I call it "the Setup file". This amazingly innovative mechanism lets you describe the extension name, source files, include paths, macros to define/undefine, and a few other goodies in a simple, compact, familiar (to Unix geeks at least) notation. For example, you could describe the "_tkinter" extension as follows: _tkinter _tkinter.c tkappinit.c -DWITH_APPINIT -UFOO \ -I/usr/include \ -I/usr/X11R6/include \ -DWITH_TIX -ltix4.1.8.0 \ -L/usr/lib \ -ltk8.0 -ltcl8.0 \ -L/usr/X11R6/lib \ -lX11 Then, a setup script to build this extension is quite simple: from distutils.core import setup from distutils.extension import read_setup_file exts = read_setup_file("Setup.tk") setup(name = "Tkinter", ext_modules = exts) Hmmm... on second thought, this syntax seems vaguely familiar... where *have* I seen it before? ;-) Greg P.S. yes, it does support comments just like in Modules/Setup! -- Greg Ward gward@python.net http://starship.python.net/~gward/ From calvin@cs.uni-sb.de Sun Sep 17 08:00:01 2000 From: calvin@cs.uni-sb.de (Bastian Kleineidam) Date: Sun Sep 17 07:00:01 2000 Subject: [Distutils] snapshot patch In-Reply-To: <20000916211055.A3907@beelzebub> Message-ID: diff -BurN --exclude=*.pyc distutils-snap/distutils/command/sdist.py distutils-20000916/distutils/command/sdist.py --- distutils-snap/distutils/command/sdist.py Wed Sep 6 04:18:59 2000 +++ distutils-20000916/distutils/command/sdist.py Sun Sep 17 11:20:31 2000 @@ -332,7 +332,7 @@ join_lines=1, lstrip_ws=1, rstrip_ws=1, - collapse_ws=1) + collapse_join=1) while 1: line = template.readline() From gward@python.net Sun Sep 17 21:22:01 2000 From: gward@python.net (Greg Ward) Date: Sun Sep 17 20:22:01 2000 Subject: [Distutils] CCompiler interfaces In-Reply-To: <20000917181959.A593@beelzebub>; from greg on Sun, Sep 17, 2000 at 06:19:59PM -0400 References: <39BE18E1.DD782434@gmx.de> <20000917181959.A593@beelzebub> Message-ID: <20000917201921.B593@beelzebub> On 17 September 2000, I said: > > > * added the ability to compile and link in resource files with > > > Visual C++ on Windows (Thomas Heller) > > > > However, I can't find the place where it is done. Maybe it wasn't > > checked in ? > > I think this was a change to msvccompiler.py. Can't remember exactly > what the change was -- was there anything in the CVS log? Aiee! I goofed -- looks like that patch was never applied and checked in; it's still sitting in my patches directory. Arghgh! I'll attach the patch: can someone look it over, bring it up-to-date, test it, and send me a new version? Otherwise I'll have to do it myself, and who knows what damage I'll cause... ;-) Greg From gward@python.net Sun Sep 17 21:22:05 2000 From: gward@python.net (Greg Ward) Date: Sun Sep 17 20:22:05 2000 Subject: [Distutils] CCompiler interfaces In-Reply-To: <39BE18E1.DD782434@gmx.de>; from R.Liebscher@gmx.de on Tue, Sep 12, 2000 at 01:52:01PM +0200 References: <39BE18E1.DD782434@gmx.de> Message-ID: <20000917181959.A593@beelzebub> On 12 September 2000, Rene Liebscher said: > Output filenames: > > * create_static_lib needs a library name, which > becomes intern a real file name. > (Is anyone using this method?) Yes, this is used by the "build_clib" command. It's important functionality (IMHO). > * link_shared_library is similar to > create_static_lib. Right, because shared libraries and static libraries are (at least conceptually) similar. In implementation terms -- how they're created, what you can do with them, how they're formatted -- they are indeed quite different, as you observed later. > * link_shared_object needs a real file name. > > * link_executable wants the basename of its > output file. 'link_shared_object()' is the odd man out here: the other three all want an "abstract" name -- library or executable name -- rather than a concrete filename. 'link_shared_object()' was mainly conceived as a back-door to let you generate whatever filename you please, if the high-level interface ('link_shared_library()') doesn't cut it. > We have two methods shared_object_filename() > and library_filename() (for static and shared > libraries). Wouldn't it be better to add > a method executable_filename() and use for > all of the linker methods the real filename > as parameter? 'shared_object_filename()' is a relic -- I just checked, and it's only defined by CCompiler, and not used anywhere. Fine to delete it. I agree about 'executable_filename()' -- good idea to add this for consistency. > For link_executable() we also need a parameter > export_symbols. (An executable could export > symbols to be able to be extended by plug-ins.) OK, I didn't know that. Sounds like a reasonable change. > The best way would to merge these three methods > into one new method and remove the old ones. > (Or let the old ones call the new and give a warning > the user should use the new method.) Agreed. The backwards compatibility is a pain because these methods take so many parameters, but I think the 'link_this()' and 'link_that()' interface is nice to support. I wouldn't bother with a warning, though. > This new method would have following parameters: > (It is better to see a shared libray as an half-build > executable than to compare it with static libraries.) > > def link (self, > target, [...] > > target would be one of 'shared_object', > 'shared_library' or 'executable'. Then it should be called 'target_desc' -- when I see "target", I think "filename". > The compiler classes would then be much simpler > and smaller. Good -- they need it. > This saves about 70 lines of redundant code. > For BCPPCompiler and MSVCCompiler it is similar. Excellent! > Are there any objections? > If not, I would send Greg a patch for it. OK -- I'll take a look at the patch. > > Release 0.9 (29 June, 2000): > > ---------------------------- > ... > > > * added the ability to compile and link in resource files with > > Visual C++ on Windows (Thomas Heller) > > However, I can't find the place where it is done. Maybe it wasn't > checked in ? I think this was a change to msvccompiler.py. Can't remember exactly what the change was -- was there anything in the CVS log? Greg From gward@python.net Sun Sep 17 21:24:00 2000 From: gward@python.net (Greg Ward) Date: Sun Sep 17 20:24:00 2000 Subject: [Distutils] define_macros ignored? In-Reply-To: <14782.59101.828231.905056@bitdiddle.concentric.net>; from jeremy@beopen.com on Tue, Sep 12, 2000 at 10:30:53PM -0400 References: <14782.59101.828231.905056@bitdiddle.concentric.net> Message-ID: <20000917202146.C593@beelzebub> On 12 September 2000, Jeremy Hylton said: > I am working on a setup script for Tkinter, which requires a > -DWITH_APPINIT flag at compile time. The docs suggest that I should > add the following definition to my Extension constructor call: > define_macros=[('WITH_APPINIT', None),], > > I did this, but it had no effect. It looks like there is no where > that the define_macros attribute of the Extension object is actually > used. For the record: this bug has now been fixed; it's in the latest code snapshot. Thanks to Jeremy for pointing it out this time. (And I think to AMK for pointing it out a couple months ago... oops...) Greg From gward@python.net Sun Sep 17 21:40:01 2000 From: gward@python.net (Greg Ward) Date: Sun Sep 17 20:40:01 2000 Subject: [Distutils] setup.py sdist hangs on windows with 1.6 and 2.0b1 In-Reply-To: <027501c01d8f$a24f51d0$4500a8c0@thomasnb>; from thomas.heller@ion-tof.com on Wed, Sep 13, 2000 at 04:33:43PM +0200 References: <027501c01d8f$a24f51d0$4500a8c0@thomasnb> Message-ID: <20000917203808.D593@beelzebub> On 13 September 2000, Thomas Heller said: > This may not be directly related to distutils, > it may also be a bug in 1.6 and 2.0b1 re implementation. > > 'setup.py sdist' with the current distutils CVS version > hangs while parsing MANIFEST.in, > executing the re.sub command in these lines in text_file.py: > > # collapse internal whitespace (*after* joining lines!) > if self.collapse_ws: > line = re.sub (r'(\S)\s+(\S)', r'\1 \2', line) That was a very suboptimal use of regex substitution; I discovered (and fixed) this several months ago in the version of text_file.py we use at work, but forgot to propagate the change to the distutils version. Coincidentally, I finally got around to doing that this weekend: revision 1.10 date: 2000/09/16 18:04:55; author: gward; state: Exp; lines: +31 -27 [change from 2000/04/17, propagating now to distutils copy] Dropped the 'collapse_ws' option and replaced it with 'collapse_join' -- it's *much* faster (no 're.sub()') and this is the reason I really added 'collapse_ws', ie. to remove leading whitespace from a line being joined to the previous line. So this problem should no longer be there. However, if the sre engine really did *hang* (whereas pcre was just slow), that might be an sre bug. [...a few moments pass...] Yep, you're right: if I do this single, simple regex substitution with Python 2.0, it hangs. I've submitted a bug to SourceForge -- thanks! Greg From gward@python.net Sun Sep 17 21:42:10 2000 From: gward@python.net (Greg Ward) Date: Sun Sep 17 20:42:10 2000 Subject: [Distutils] snapshot patch In-Reply-To: ; from calvin@cs.uni-sb.de on Sun, Sep 17, 2000 at 12:58:51PM +0200 References: <20000916211055.A3907@beelzebub> Message-ID: <20000917204000.E593@beelzebub> On 17 September 2000, Bastian Kleineidam said: > - collapse_ws=1) > + collapse_join=1) D'ohh! Paper-bag-over-head time. Good thing it was just a snapshot -- thanks! Greg From thomas.heller@ion-tof.com Mon Sep 18 05:43:02 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Mon Sep 18 04:43:02 2000 Subject: [Distutils] Help: request for assistance with PyOpenGL distribution using distutils on NT References: Message-ID: <01aa01c0214c$428ea690$4500a8c0@thomasnb> > > D:\bin\lang\studio\VC98\BIN\cl.exe /c /nologo /Ox /MD /W3 /GX -Isrc > -ID:\bin\lang\Python\Include /Tcsrc/_glumodule.c > /Fobuild\temp.win32\Release\src/_glumodule.obj > > Doesn't work. The GUI under the old BUILD.py uses: > > /nologo /MD /W3 /GX /O2 /Ob2 /I "D:\bin\lang\Python\include" /D "WIN32" /D > "_WINDOWS" /D "NDEBUG" /Fp"tmp\./_glu.pch" /YX /Fo"tmp\./" /Fd"tmp\./" /FD > /c > > and works fine. This is not related to C/C++ problems. The problem is that WIN32 and _WINDOWS is not defined in the first run. Insert the define_macros parameter: Extension("_glu", ["src/_glumodule.c"], libraries=glu_libs, library_dirs=glu_lib_dirs, define_macros=[("WIN32", None), ("_WINDOWS", None)], ), in your setup-script and it should work. (This may also be necessary for other extensions). Thomas From R.Liebscher@gmx.de Mon Sep 18 07:58:01 2000 From: R.Liebscher@gmx.de (Rene Liebscher) Date: Mon Sep 18 06:58:01 2000 Subject: [Distutils] More minor issues References: <39BE09E2.F699662B@gmx.de> <20000912212202.A1082@beelzebub> <39C09744.133BDC89@gmx.de> <002c01c01e54$800807f0$4500a8c0@thomasnb> Message-ID: <39C5F470.C05FADB9@gmx.de> Thomas Heller wrote: > > [Rene] > > In the CVS are the source files for the windows installer. > > So everyone can create his own, I think most people > > would like to change the bitmap (PythonPowered) and use > > a more specific bitmap for their package. > I would happily accept a better (general) bitmap! What about a kind of contest? Let's collect some drafts, put them on a web page and let the people decide which looks at best. Anyone who wants to manage this? (For people who want create such bitmaps some information. There are four dialogs, they can use the same bitmap or different. 1. "This Wizard will install %s on your computer. ..." 2. "Select python installation to use:" 3. "Click Next to begin the installation. ..." 4. "Click the Finish button to exit the Setup wizard." Maximum size seems to be 98(width)x155(height) pixels. At least are these the values in the install.rc file for the bitmap. ) > > Creating the installer executable is not the problem, but > > to use it you have to change bdist_wininst.py. > > (If you only maintain one package this is not a problem > > but if there are more, you had to change it everytime > > you want to create an installer for the other package.) > > > Recompiling the installer executable only to change the bitmap > seems overkill to me. You could use win32api calls (if available) > to replace the bitmap in the executable without need for a compiler, > better would be to pack the bitmap into the installer archive > at runtime. This could also be done on Linux. Replacing the bitmap was bad example. What I wanted to say was that people sometimes want to use a modified installer and need a simple way to get bdist_wininst use this installer instead its default. The new installer had to compiled only once. And this could even be done with the help of a short python script which uses distutils compiler classes. (I have already written such a script but it uses my new compiler interface (executable_filename(),link()) so you have to wait until distutils also supports it.) kind regards Rene Liebscher From hoel@germanlloyd.org Mon Sep 18 07:59:01 2000 From: hoel@germanlloyd.org (Berthold =?iso-8859-1?q?H=F6llmann?=) Date: Mon Sep 18 06:59:01 2000 Subject: [Distutils] test command for setup.py Message-ID: Hello, I'm thinking about providing a framework for pre-install testing for distutils enabled Python packages. I think about a new "test" command, which inserts the lib build path into PYTHONPATH and runs specified tests. Is anyone else working on something like this, and what are your opinions and requirements on this? Thanks Berthold -- email: hoel@GermanLloyd.org ) tel. : +49 (40) 3 61 49 - 73 74 ( C[_] These opinions might be mine, but never those of my employer. From thomas.heller@ion-tof.com Mon Sep 18 08:09:01 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Mon Sep 18 07:09:01 2000 Subject: [Distutils] test command for setup.py References: Message-ID: <03f801c02160$d3cf7760$4500a8c0@thomasnb> > Hello, > > I'm thinking about providing a framework for pre-install testing for > distutils enabled Python packages. I think about a new "test" command, > which inserts the lib build path into PYTHONPATH and runs specified > tests. Is anyone else working on something like this, and what are > your opinions and requirements on this? > A good idea! Related: IMHO, distutils needs a test suite for itself! Thomas From R.Liebscher@gmx.de Mon Sep 18 08:23:01 2000 From: R.Liebscher@gmx.de (Rene Liebscher) Date: Mon Sep 18 07:23:01 2000 Subject: [Distutils] Help: request for assistance with PyOpenGL distribution using distutils on NT References: <01aa01c0214c$428ea690$4500a8c0@thomasnb> Message-ID: <39C5FA15.668A71DB@gmx.de> Thomas Heller wrote: > > > > > D:\bin\lang\studio\VC98\BIN\cl.exe /c /nologo /Ox /MD /W3 /GX -Isrc > > -ID:\bin\lang\Python\Include /Tcsrc/_glumodule.c > > /Fobuild\temp.win32\Release\src/_glumodule.obj > > > > Doesn't work. The GUI under the old BUILD.py uses: > > > > /nologo /MD /W3 /GX /O2 /Ob2 /I "D:\bin\lang\Python\include" /D "WIN32" /D > > "_WINDOWS" /D "NDEBUG" /Fp"tmp\./_glu.pch" /YX /Fo"tmp\./" /Fd"tmp\./" /FD > > /c > > > > and works fine. > > This is not related to C/C++ problems. > The problem is that WIN32 and _WINDOWS is not defined in the first run. > Insert the define_macros parameter: > Extension("_glu", > ["src/_glumodule.c"], > libraries=glu_libs, > library_dirs=glu_lib_dirs, > define_macros=[("WIN32", None), ("_WINDOWS", None)], > ), > in your setup-script and it should work. > (This may also be necessary for other extensions). 'define_macros' only works with the latest version of distutils. May be you should use the same include order as in the other c-files. (This file fails to compile.) * _glumodule.c ******************* ... #ifdef WIN32 #include #define MCALLBACK (void (__stdcall *)(void)) #define GLUCALLBACK WINAPI #else #define MCALLBACK #define GLUCALLBACK #endif #include #include #include "Python.h" ... ********************************** (This one compiles without problems.) * _openglmodule.c **************** ... #include "Python.h" #ifdef MS_WIN32 #include #endif #include #include ... ********************************** If you include "Python.h" first and use then MS_WIN32 instead of WIN32, you don't need to define WIN32 and don't get problems with older versions of distutils, which doesn't handle 'define_macros'. (There are some other occurences of WIN32 in your file, either you replace them or you define WIN32 in your c-file if MS_WIN32 is defined.) kind regards Rene Liebscher From thomas.heller@ion-tof.com Mon Sep 18 08:24:01 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Mon Sep 18 07:24:01 2000 Subject: [Distutils] CCompiler interfaces References: <39BE18E1.DD782434@gmx.de> <20000917181959.A593@beelzebub> Message-ID: <043801c02162$ce6d6000$4500a8c0@thomasnb> > On 12 September 2000, Rene Liebscher said: > > For link_executable() we also need a parameter > > export_symbols. (An executable could export > > symbols to be able to be extended by plug-ins.) > But not on windows... (?) Thomas From R.Liebscher@gmx.de Mon Sep 18 08:48:01 2000 From: R.Liebscher@gmx.de (Rene Liebscher) Date: Mon Sep 18 07:48:01 2000 Subject: [Distutils] CCompiler interfaces References: <39BE18E1.DD782434@gmx.de> <20000917181959.A593@beelzebub> <043801c02162$ce6d6000$4500a8c0@thomasnb> Message-ID: <39C60011.21AAA23F@gmx.de> This is a multi-part message in MIME format. --------------1A9774B975004582D6999413 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Thomas Heller wrote: > > > On 12 September 2000, Rene Liebscher said: > > > For link_executable() we also need a parameter > > > export_symbols. (An executable could export > > > symbols to be able to be extended by plug-ins.) > > > But not on windows... (?) > Have you ever tried to use 'quikview' (German: Schnellansicht) with a executable or a dll? It shows you all exported symbols. For example most Borland C++ compiled executable export a hook for the debugger. netscape.exe exports about 8000 symbols. (see attached file for some examples) To programm plugins, which call functions in the main program, you link against the main program as you would do to link against a dll. (file: netscape_exports.txt) Kind regards Rene Liebscher --------------1A9774B975004582D6999413 Content-Type: text/plain; charset=us-ascii; name="netscape_exports.txt" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="netscape_exports.txt" ??0CPrefStore@@QAE@ABV0@@Z ??0CPrefStoreLocal@@QAE@ABV0@@Z ??0CPrefStoreLocalLI@@QAE@ABV0@@Z ??0CPrefStoreLockFile@@QAE@ABV0@@Z ??0CProfile@@QAE@ABV0@@Z ??0CProfileAuthKey@@QAE@ABV0@@Z ??0CProfileManager@@QAE@ABV0@@Z ??1CPrefStore@@UAE@XZ ??1CPrefStoreLocalLI@@UAE@XZ ??1CPrefStoreLockFile@@UAE@XZ ??4CPrefStore@@QAEAAV0@ABV0@@Z ??4CPrefStoreLocal@@QAEAAV0@ABV0@@Z ??4CPrefStoreLocalLI@@QAEAAV0@ABV0@@Z ??4CPrefStoreLockFile@@QAEAAV0@ABV0@@Z ??4CProfile@@QAEAAV0@ABV0@@Z ??4CProfileAuthKey@@QAEAAV0@ABV0@@Z ??4CProfileManager@@QAEAAV0@ABV0@@Z ??_7CPrefStore@@6B@ ??_7CPrefStoreLocal@@6B@ ??_7CPrefStoreLocalLI@@6B@ ??_7CPrefStoreLockFile@@6B@ ??_7CProfile@@6B@ ??_7CProfileAuthKey@@6B@ ??_7CProfileManager@@6B@ ??_ECPrefStore@@UAEPAXI@Z ??_ECPrefStoreLocal@@UAEPAXI@Z ??_ECPrefStoreLocalLI@@UAEPAXI@Z ??_ECPrefStoreLockFile@@UAEPAXI@Z ??_ECProfile@@UAEPAXI@Z ??_ECProfileAuthKey@@UAEPAXI@Z ??_ECProfileManager@@UAEPAXI@Z ??_GCPrefStore@@UAEPAXI@Z ??_GCPrefStoreLocal@@UAEPAXI@Z ??_GCPrefStoreLocalLI@@UAEPAXI@Z ??_GCPrefStoreLockFile@@UAEPAXI@Z ??_GCProfile@@UAEPAXI@Z ??_GCProfileAuthKey@@UAEPAXI@Z ??_GCProfileManager@@UAEPAXI@Z ?AttachCurrentThread@JavaVM_@@QAGJPAPAUJNIEnv_@@PAX@Z ?DestroyJavaVM@JavaVM_@@QAGJXZ ?DetachCurrentThread@JavaVM_@@QAGJXZ ?GetAuthLogin@CProfileAuthKey@@QAEPBDXZ ?GetAuthToken@CProfileAuthKey@@QAEPBDXZ ?GetConnectionInfo@CProfile@@UAEPAUProfileConnectionInfo@@XZ ... NET_ParseDate NET_ParseHTMLHelpFree NET_ParseHTMLHelpInit NET_ParseHTMLHelpLoadHelpDoc NET_ParseMimeHeader NET_ParseNetHelpURL NET_ParseURL NET_ParseUploadURL NET_PlainTextConverter NET_PlusToSpace NET_Pop3Load NET_PrefChangedFunc NET_PrintDirectory NET_PrintFileType NET_PrintNetlibStatus NET_PrintRawToDisk NET_ProcessAddressBook NET_ProcessCallbackURL NET_ProcessDataURL NET_ProcessFTP NET_ProcessFile ... xp_MakeHTMLDialogPass1 xp_MakeHTMLDialogPass2 xp_MakeHTMLDialogWindow zip_close zip_get zip_get_no_of_elements zip_list_files zip_lock zip_open zip_stat zip_unlock zlibVersion --------------1A9774B975004582D6999413-- From mfletch@tpresence.com Tue Sep 19 07:04:04 2000 From: mfletch@tpresence.com (Mike Fletcher) Date: Tue Sep 19 06:04:04 2000 Subject: [Distutils] PyOpenGL -- setup files checked into repository, demos added Message-ID: Thanks to Rene and Thomas, the current PyOpenGL (CVS) now builds the extensions under the latest official distutils for Python 1.5.2 under VC++ *yay*. [Note: I made a very minor change to the setup script to build the latest openglutils module (requires glu)] I don't have time to get the .py stuff working just now (from what I can see it looks simple, but I'm on a short timeline, and I spent my time this evening trying to figure out what's going on with the PyOpenGL CVS tree (it's a mess!)). I've checked the 1.5.5 demos into the repository. If any of the authors would like them _not_ included, let someone know and we'll pull them out again. If someone who knows wants to let people know what all those directories are about it might be advisable... I'm thinking of just deleting most of them if they're not needed. Things to do (for PyOpenGL people): Add the .py files to the setup scripts (I assume you just specify the directory, at least, that's what it looks like) Build and test the results. Check back into the repository. Start creating distributables. I'll be busy until next week, so no chance I'll get to this before then. Thanks distutils people, this is cool stuff. Enjoy yourselves, Mike From thomas.heller@ion-tof.com Tue Sep 19 08:16:01 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue Sep 19 07:16:01 2000 Subject: [Distutils] bdist_wininst: data and scripts Message-ID: <002d01c0222a$e53b6660$4500a8c0@thomasnb> I just check in the following patch for bdist_wininst: (This should fix Bastian's complaints about not including data and scripts into the windows installer) Modified Files: bdist_wininst.py Log Message: Set the 'nt' installation scheme for the install command even if run on other systems, so that data, headers, scripts are included in the installer. Index: bdist_wininst.py =================================================================== RCS file: /cvsroot/python/distutils/distutils/command/bdist_wininst.py,v retrieving revision 1.12 retrieving revision 1.13 diff -C2 -r1.12 -r1.13 *** bdist_wininst.py 2000/09/16 15:56:32 1.12 --- bdist_wininst.py 2000/09/19 11:10:23 1.13 *************** *** 75,78 **** --- 75,89 ---- install = self.reinitialize_command('install') install.root = self.bdist_dir + if os.name != 'nt': + # must force install to use the 'nt' scheme + install.select_scheme ('nt') + # change the backslash to the current pathname separator + for key in ('purelib', 'platlib', 'headers', 'scripts', + 'data'): + attrname = 'install_' + key + attr = getattr (install, attrname) + if attr: + attr = string.replace (attr, '\\', os.sep) + setattr (install, attrname, attr) install_lib = self.reinitialize_command('install_lib') *************** *** 100,111 **** "%s.win32" % fullname) ! # XXX hack! Our archive MUST be relative to sys.prefix ! # XXX What about .install_data, .install_scripts, ...? ! # [Perhaps require that all installation dirs be under sys.prefix ! # on Windows? this will be acceptable until we start dealing ! # with Python applications, at which point we should zip up ! # the application directory -- and again everything can be ! # under one dir --GPW] ! root_dir = install.install_lib arcname = self.make_archive (archive_basename, "zip", root_dir=root_dir) --- 111,128 ---- "%s.win32" % fullname) ! # Our archive MUST be relative to sys.prefix, which is the ! # same as install_lib in the 'nt' scheme. ! root_dir = os.path.normpath (install.install_lib) ! ! # Sanity check: Make sure everything is included ! for key in ('purelib', 'platlib', 'headers', 'scripts', 'data'): ! attrname = 'install_' + key ! install_x = getattr (install, attrname) ! # (Use normpath so that we can string.find to look for ! # subdirectories) ! install_x = os.path.normpath (install_x) ! if string.find (install_x, root_dir) != 0: ! raise DistutilsInternalError \ ! ("'%s' not included in install_lib" % key) arcname = self.make_archive (archive_basename, "zip", root_dir=root_dir) Thomas From hoel@germanlloyd.org Tue Sep 19 08:28:03 2000 From: hoel@germanlloyd.org (Berthold =?iso-8859-1?q?H=F6llmann?=) Date: Tue Sep 19 07:28:03 2000 Subject: [Distutils] test command for setup.py In-Reply-To: hoel@germanlloyd.org's message of "18 Sep 2000 12:56:05 +0200" References: Message-ID: --=-=-= Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable hoel@germanlloyd.org (Berthold H=F6llmann) writes: > Hello, >=20 > I'm thinking about providing a framework for pre-install testing for > distutils enabled Python packages. I think about a new "test" command, > which inserts the lib build path into PYTHONPATH and runs specified > tests. Is anyone else working on something like this, and what are > your opinions and requirements on this? Attatched you find a first version of test.py. This copied into ../distutils/command allows python setup.py test First this command calls "build" to enshure a correctly build disttribution. It then modifies "sys.path" to point it to the build directory, imports files from a subdirectory, and calls a function named "test" in the imported module. The default name of the subdirectory is "test". From this directory by default all modules named "test_*.py" are imported. Additional options are: - test_dir (test-dir): name of the directory containg the test files (default: "test") - test_prefix (test-prefix): prefix for the test files. (default: "test_") - test_suffixes (test-suffixes): suffixes used to generate filenames for the tests (default: None, which means: look for all files in test_dir named test_prefix.py and list all s for test_suffixes). I took build.py as a template for writing this. Berthold --=-=-= Content-Type: application/octet-stream Content-Disposition: attachment; filename=test.py """distutils.command.test Implements the Distutils 'test' command.""" # created 2000/09/18, Greg Ward __revision__ = "$Id" import sys, os from distutils.core import Command from distutils.util import get_platform class test (Command): description = "test the distribution prior to install" user_options = [ ('build-base=', 'b', "base directory for build library"), ('build-purelib=', None, "build directory for platform-neutral distributions"), ('build-platlib=', None, "build directory for platform-specific distributions"), ('build-lib=', None, "build directory for all distribution (defaults to either " + "build-purelib or build-platlib"), ('build-scripts=', None, "build directory for scripts"), ('test-dir=', None, "directory that contains the test definitions"), ('test-prefix=', None, "prefix to the testcase filename"), ('test-suffix=', None, "a list of suffixes used to generate names the of the testcases") ] def initialize_options (self): self.build_base = 'build' # these are decided only after 'build_base' has its final value # (unless overridden by the user or client) self.build_purelib = None self.build_platlib = None self.build_lib = None self.build_scripts = None self.test_dir = 'test' self.test_prefix = 'test_' self.test_suffixes = None def finalize_options (self): # Need this to name platform-specific directories, but sys.platform # is not enough -- it only names the OS and version, not the # hardware architecture! self.plat = get_platform () # 'build_purelib' and 'build_platlib' just default to 'lib' and # 'lib.' under the base build directory. We only use one of # them for a given distribution, though -- if self.build_purelib is None: self.build_purelib = os.path.join (self.build_base, 'lib') if self.build_platlib is None: self.build_platlib = os.path.join (self.build_base, 'lib.' + self.plat) # 'build_lib' is the actual directory that we will use for this # particular module distribution -- if user didn't supply it, pick # one of 'build_purelib' or 'build_platlib'. if self.build_lib is None: if self.distribution.ext_modules: self.build_lib = self.build_platlib else: self.build_lib = self.build_purelib if self.build_scripts is None: self.build_scripts = os.path.join (self.build_base, 'scripts') if self.test_suffixes is None: self.test_suffixes = [] pref_len = len(self.test_prefix) for file in os.listdir(self.test_dir): if (file[-3:] == ".py" and file[:pref_len]==self.test_prefix): self.test_suffixes.append(file[pref_len:-3]) # finalize_options () def run (self): # Invoke the 'build' command to "build" pure Python modules # (ie. copy 'em into the build tree) self.run_command ('build') # remember old sys.path to restore it afterwards old_path = sys.path[:] # extend sys.path sys.path.insert(0, self.build_purelib) sys.path.insert(0, self.build_platlib) sys.path.insert(0, self.test_dir) # build include path for test for case in self.test_suffixes: TEST = __import__(self.test_prefix+case, globals(), locals(), ['']) TEST.test() # restore sys.path sys.path = old_path[:] # class test --=-=-= -- email: hoel@GermanLloyd.org ) tel. : +49 (40) 3 61 49 - 73 74 ( C[_] These opinions might be mine, but never those of my employer. --=-=-=-- From gward@python.net Tue Sep 19 21:16:01 2000 From: gward@python.net (Greg Ward) Date: Tue Sep 19 20:16:01 2000 Subject: [Distutils] Re: [Python-checkins] CVS: distutils/distutils/command bdist_wininst.py,1.12,1.13 In-Reply-To: <200009191110.EAA13195@slayer.i.sourceforge.net>; from theller@users.sourceforge.net on Tue, Sep 19, 2000 at 04:10:27AM -0700 References: <200009191110.EAA13195@slayer.i.sourceforge.net> Message-ID: <20000919201346.A587@beelzebub> On 19 September 2000, Thomas Heller said: > Modified Files: > bdist_wininst.py > Log Message: > Set the 'nt' installation scheme for the install command even if run > on other systems, so that data, headers, scripts are included in > the installer. Ahh good, I was hoping you'd take care of this. But: > *** bdist_wininst.py 2000/09/16 15:56:32 1.12 > --- bdist_wininst.py 2000/09/19 11:10:23 1.13 > *************** > *** 75,78 **** > --- 75,89 ---- > install = self.reinitialize_command('install') > install.root = self.bdist_dir > + if os.name != 'nt': > + # must force install to use the 'nt' scheme > + install.select_scheme ('nt') > + # change the backslash to the current pathname separator > + for key in ('purelib', 'platlib', 'headers', 'scripts', > + 'data'): > + attrname = 'install_' + key > + attr = getattr (install, attrname) > + if attr: > + attr = string.replace (attr, '\\', os.sep) > + setattr (install, attrname, attr) I think we can get rid of this hack by changing the INSTALL_SCHEMES dictionary in install.py to store paths in Unix form, and then convert to native when they're needed. That is, the install path "templates" in install.py would be treated just like install paths in the setup script (or on the command line) -- convert from foo/bar/baz to foo\bar\baz on Windows, and foo:bar:baz on Mac OS. (Modulo Mac OS' peculiar convention for absolute/relative directories: not sure if 'convert_path()' handles that completely right.) Greg Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Tue Sep 19 22:16:00 2000 From: gward@python.net (Greg Ward) Date: Tue Sep 19 21:16:00 2000 Subject: [Distutils] Brewing controversy over header file installation In-Reply-To: ; from mwa@gate.net on Fri, Sep 15, 2000 at 11:53:03PM -0400 References: <20000915213820.A776@beelzebub> Message-ID: <20000919211357.A648@beelzebub> On 15 September 2000, Mark W. Alexander said: > I'm only providing opinions here, not code, so I'm very easily out-voted, > but I think distutils should do neither or both. I still feel that the > libraries are better provided by a generic (non-python) run-time package, > the headers (and usage docs) by a devel package, and the python > implementation by distutils. But the headers installed by the Distutils are *specifically* for Python extensions: we install Numeric/*.h so that other extension authors can use Numeric's C structs and functions, not so that general C programmers can do so. Thus, this is a concern for a Python extension-building and -installing system. I do not know for certain, but I suspect that installing the Python extension -- eg. _numpy.so -- suffices for the "library" portion. I have not heard any reaction, positive or negative, from anyone using the installed Numeric/*.h files. So who knows if it's actually working? > .....It's just a matter of time before distutils becomes the universal > build/install/package everything tool. Curses! You've seen to the heart of my devious master plan. All who read this post will have to swear to uttermost silence, else my minions will track them down and... SILENCE THEM!! bwa-hAHAHAHAHAA!!! > I recommend you don't take that > first step. Oops, too late. ;-) Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Tue Sep 19 22:28:12 2000 From: gward@python.net (Greg Ward) Date: Tue Sep 19 21:28:12 2000 Subject: [Distutils] test command for setup.py In-Reply-To: ; from hoel@germanlloyd.org on Mon, Sep 18, 2000 at 12:56:05PM +0200 References: Message-ID: <20000919212537.B648@beelzebub> On 18 September 2000, Berthold Höllmann said: > I'm thinking about providing a framework for pre-install testing for > distutils enabled Python packages. I think about a new "test" command, > which inserts the lib build path into PYTHONPATH and runs specified > tests. Is anyone else working on something like this, and what are > your opinions and requirements on this? I've been thinking about it from Day 1, personally. Here are some of my opinions/requirements: * serious tests don't belong in the module being tested, because test code tends to be at least as long as the code being tested * however, ensuring that each module imports (and possibly runs as a script) is not a bad idea. Running modules as scripts should be an option that the developer can turn off. * the convention I've always leaned towards is that test/test*.py is your test suite; run each of those scripts in turn and analyze the results to determine if the entire module distribution passes * the requirements on what a test script may output should be fairly strict. My favourite is loosely based on Perl's testing convention, which is a great success largely because the standard-way-to-build- Perl-modules includes a "make test" step, and it's always immediately clear if everything passed. My proposed convention is this: each test script outputs a line saying how many tests are expected, then a series of lines that start with either "ok" or "not ok". If there are any "not ok" lines, the test script fails; if the number of "ok" lines != the number of expected tests, the test script fails. Each "ok" or "not ok" may be followed by ":" and an arbitrary string, which makes it a lot easier to track down problems. (The Perl convention is "ok 1", "ok 2", ... "ok N", which makes it really awkward to track down problems.) Any lines not starting with "ok" or "not ok" are ignored. * there should be a nice framework that makes it easy to write good tests and play with the above rules. Such a framework exists in prototype form and has been used quite successfully for several months internally where I work; it suffers from code bloat and creeping featuritis but is extremely useful. * but developers should be allowed to go their own way and ignore the "official" testing framework: some things don't fit into frameworks! * we might also want to accomodate a Python-style test suite, where you compare "known good" output of a script with the current output. I'm not a fan of this methodology, but sometimes it's easier, and it may have some currency in the Python community outside of Lib/test. It should be the developer's choice what style of test suite he wants to write. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Tue Sep 19 22:29:01 2000 From: gward@python.net (Greg Ward) Date: Tue Sep 19 21:29:01 2000 Subject: [Distutils] test command for setup.py In-Reply-To: <03f801c02160$d3cf7760$4500a8c0@thomasnb>; from thomas.heller@ion-tof.com on Mon, Sep 18, 2000 at 01:08:46PM +0200 References: <03f801c02160$d3cf7760$4500a8c0@thomasnb> Message-ID: <20000919212706.C648@beelzebub> On 18 September 2000, Thomas Heller said: > Related: > > IMHO, distutils needs a test suite for itself! Yeah, I've been working on it sporadically. It's even less fun than writing documentation, and normally I get a perverse enjoyment out of writing tests. Well, actually, the enjoyment comes from seeing them pass. You know what I mean. Haven't checked anything in because it's so sporadic, skeletal, and incomplete. Lousy excuse. Someone should threaten to take away my keyboard until I get the tests to a check-in-able state... Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From loewis@informatik.hu-berlin.de Wed Sep 20 06:57:01 2000 From: loewis@informatik.hu-berlin.de (Martin von Loewis) Date: Wed Sep 20 05:57:01 2000 Subject: [Distutils] Synchronizing version information Message-ID: <200009200956.LAA04691@pandora.informatik.hu-berlin.de> In my package, I have two places which carry version information: in the setup.py, when passing the version= keyword argument to setup, and in the package's __init__, to allow runtime access to the available version. I'd like these to be synchronized, having the information only in a single place, and fetching it from there in the other. Is there any mechanism in distutils to facilitate such synchronization? One approach would be a autoconf-style replacement mechanism, where I put version_info = distutils.version.StrictVersion(@VERSION@) into the Python source, and version = version_string, substitute = {'VERSION': repr(version_string)} into setup.py. 'build', 'sdist', etc would update the string appropriately. Any comments? TIA, Martin From calvin@cs.uni-sb.de Wed Sep 20 07:44:01 2000 From: calvin@cs.uni-sb.de (Bastian Kleineidam) Date: Wed Sep 20 06:44:01 2000 Subject: [Distutils] Synchronizing version information In-Reply-To: <200009200956.LAA04691@pandora.informatik.hu-berlin.de> Message-ID: This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. Send mail to mime@docserver.cac.washington.edu for more info. --1619418364-707833344-969446609=:31224 Content-Type: TEXT/PLAIN; charset=US-ASCII >Is there any mechanism in distutils to facilitate such synchronization? You could use the attached Template.py file. Then copy __init__.py to __init__.py.tmpl, insert $version and execute t = Template.Template('__init__.py.tmpl') f = open('__init__.py','w') f.write(t.fill_in({'version', 0.1}) f.close and $version gets replaced. Bastian --1619418364-707833344-969446609=:31224 Content-Type: TEXT/PLAIN; charset=US-ASCII; name="Template.py" Content-Transfer-Encoding: BASE64 Content-ID: Content-Description: Content-Disposition: attachment; filename="Template.py" IiIiU3RyaW5nIGludGVycG9sYXRpb24NCjIwIE9jdCAxOTk2DQogICAgSW5p dGlhbCB2ZXJzaW9uIGNhbGxlZCBJdHBsIGJ5IEthLVBpbmcgWWVlDQoxMyBK YW4gMjAwMA0KICAgIE1vZGlmaWVkIGJ5IEJhc3RpYW4gS2xlaW5laWRhbSB0 byBlbXVsYXRlIFBlcmxzIFRleHQ6OlRlbXBsYXRlDQogICANClVzYWdlOg0K VGVtcGxhdGUgZGF0YSBjb25zaXN0cyBvZiBub3JtYWwgdGV4dCBhbmQgcHl0 aG9uIGNvZGUgZnJhZ21lbnRzLg0KQ29kZSBmcmFnbWVudHMgc3RhcnQgd2l0 aCBhIGRvbGxhciBzaWduIGZvbGxvd2VkIGVpdGhlciBieQ0KMSkgdmFyaWFi bGUgb3IgYXR0cmlidXRlIHJlZmVyZW5jZSBvciBzdWJzY3JpcHRpb24gb3Ig ZnVuY3Rpb24gY2FsbA0KICAgZXZhbHVhdGVzIHRvIHRoZSBzdHIoKSB2YWx1 ZSBvZiB0aGUgZ2l2ZW4gZXhwcmVzc2lvbg0KICAgZXhhbXBsZTogJHByaW50 aXQoKQ0KICAgDQoyKSB7cHl0aG9uIHN0YXRlbWVudHN9DQogICBhcHBlbmQg YXJiaXRyYXJ5IG9iamVjdHMgdG8gdGhlIG91dHB1dCBsaXN0DQoNCkxpbWl0 YXRpb25zOg0KMSkgQ29kZSBmcmFnbWVudHMgYXJlIGFyYml0cmFyeSBzdHJp bmdzIHdpdGggbWF0Y2hpbmcgInt9IiBwYXJhbnRoZXNlcy4NCiAgIE5vIGZ1 cnRoZXIgc3ludGF4IHZlcmlmaWNhdGlvbiBpcyBkb25lLg0KMikgUGFyYW1l dGVycyB0byBmdW5jdGlvbiBjYWxscyBtdXN0IG5vdCBoYXZlIHBhcmFudGhl c2VzIGluc2lkZS4NCiAgIFBhcmFtZXRlcnMgdG8gc3Vic2NyaXB0aW9ucyBt dXN0IG5vdCBoYXZlIGJyYWNrZXRzIGluc2lkZS4NCiAgIFNvIHRoaXMgaXMg bm90IGFsbG93ZWQ6DQogICAkZnVuYygodHVwbGUxLHR1cGxlMikpICRmdW5j KG1hcFtrZXldKQ0KICAgQnV0IHRoaXMgaXMgYWxsb3dlZDoNCiAgICRmdW5j KHBhcmFtKVsxXSAkYVt4XVt5XVt6XShibHViYikNCiAgIEluIGNvZGUgZnJh Z21lbnRzLCB5b3UgY2FuIHdyaXRlIGV2ZXJ5IFB5dGhvbiBjb2RlOg0KICAg JHtPVVQgPSBPVVQgKyB0cnVua1tpZHhbMV0oZGF0YSgpKV19DQoNCkV4YW1w bGU6DQppbXBvcnQgVGVtcGxhdGUNCg0KdCA9IFRlbXBsYXRlLlRlbXBsYXRl KCJibGFibGEudG1wbCIpDQpvcGVuKCJibGFibGEiLCJ3Iikud3JpdGUodC5m aWxsX2luKHsidmFyMSI6MSwidmFyMiI6J2h1aSd9KSkNCg0KYmxhYmxhLnRt cGw6DQpIZWxsbywgdGhpcyBpcyBhIGRlbW8uIHZhcjEgaXMgJHZhcjEsIGFu ZCB2YXIyIGlzICR2YXIyLg0KIiIiDQppbXBvcnQgc3lzLCBzdHJpbmcsIHJl DQpmcm9tIHR5cGVzIGltcG9ydCBTdHJpbmdUeXBlDQoNCiMgdHlwZSBhcmd1 bWVudCAod2UgY2Fubm90IGRpc3Rpbmd1aXNoIGJldHdlZW4gZmlsZW5hbWUg YW5kIHN0cmluZyB0eXBlKQ0KRklMRU5BTUUgPSAwDQpBUlJBWSAgICA9IDEN CkZJTEUgICAgID0gMg0KU1RSSU5HICAgPSAzDQoNCiMgcmVndWxhciBleHBy ZXNzaW9ucyBmb3IgbGV4aWNhbCBhbmFseXNpcw0KIyBub3RlIHdlIG9ubHkg bWF0Y2ggYSBzdWJzZXQgb2YgdGhlIGxhbmd1YWdlIGRlZmluaXRpb24NCiMg d2UgbWF0Y2ggaWRlbnRpZmllcnMsIGF0dHJpYnV0ZSByZWZlcmVuY2UsIHN1 YnNjcmlwdGlvbnMsDQojIHNsaWNpbmdzIGFuZCBmdW5jdGlvbiBjYWxscw0K aWRlbnRpZmllciA9ICJbYS16QS1aX11bYS16QS1aMC05X10qIg0KYXR0cmli dXRlcmVmID0gaWRlbnRpZmllciArICIoXC4iICsgaWRlbnRpZmllciArICIp KiINCnN1YnNjcmlwdGlvbiA9ICJcWy4qXF0iDQpjYWxsYXJncyA9ICJcKC4q XCkiDQojIG5vdyB0aGUgd2hvbGUgcmVnZXgNCnJlZ2V4dmFyID0gcmUuY29t cGlsZSgiXiIrYXR0cmlidXRlcmVmKyIoIitzdWJzY3JpcHRpb24rInwiK2Nh bGxhcmdzKyIpKiIpDQoNCg0KY2xhc3MgVGVtcGxhdGVFcnJvcihTdGFuZGFy ZEVycm9yKToNCiAgICBwYXNzDQoNCg0KY2xhc3MgVGVtcGxhdGU6DQogICAg ZGVmIF9faW5pdF9fKHNlbGYsIGRhdGEsIGFyZ3R5cGUgPSBGSUxFTkFNRSk6 DQogICAgICAgICJsb2FkIHRoZSB0ZW1wbGF0ZSBpbiBzZWxmLmRhdGEiDQog ICAgICAgIGlmIGFyZ3R5cGUgPT0gRklMRU5BTUU6DQogICAgICAgICAgICBm aWxlID0gb3BlbihkYXRhKQ0KICAgICAgICAgICAgc2VsZi5kYXRhID0gc3Ry aW5nLmpvaW4oZmlsZS5yZWFkbGluZXMoKSwgIiIpDQogICAgICAgICAgICBm aWxlLmNsb3NlKCkNCiAgICAgICAgZWxpZiBhcmd0eXBlID09IEZJTEU6DQog ICAgICAgICAgICBpZiBkYXRhLmNsb3NlZDoNCiAgICAgICAgICAgICAgICB3 YXNjbG9zZWQgPSAxDQogICAgICAgICAgICAgICAgb3BlbihkYXRhKQ0KICAg ICAgICAgICAgZWxzZToNCiAgICAgICAgICAgICAgICB3YXNjbG9zZWQgPSAw DQogICAgICAgICAgICBzZWxmLmRhdGEgPSBzdHJpbmcuam9pbihkYXRhLnJl YWRsaW5lcygpLCAiIikNCiAgICAgICAgICAgIGlmIHdhc2Nsb3NlZDoNCiAg ICAgICAgICAgICAgICBkYXRhLmNsb3NlKCkNCiAgICAgICAgZWxpZiBhcmd0 eXBlID09IEFSUkFZOg0KICAgICAgICAgICAgc2VsZi5kYXRhID0gc3RyaW5n LmpvaW4oZGF0YSwgIlxuIikNCiAgICAgICAgZWxzZToNCiAgICAgICAgICAg IHNlbGYuZGF0YSA9IGRhdGENCiAgICAgICAgaWYgdHlwZShzZWxmLmRhdGEp ICE9IFN0cmluZ1R5cGU6DQogICAgICAgICAgICByYWlzZSBUZW1wbGF0ZUVy cm9yLCAiY291bGQgbm90IHJlYWQgZGF0YSINCg0KDQogICAgZGVmIGZpbGxf aW4oc2VsZiwgZGljdD17fSk6DQogICAgICAgICJwYXJzZSB0aGUgdGVtcGxh dGUgYW5kIGZpbGwgaW4gdmFsdWVzIg0KICAgICAgICB0cnk6DQogICAgICAg ICAgICBuYW1lY2hhcnMgPSAnYWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXon IFwNCiAgICAgICAgICAgICAgICAnQUJDREVGR0hJSktMTU5PUFFSU1RVVldY WVowMTIzNDU2Nzg5Xyc7DQogICAgICAgICAgICBvdXQgPSAiIg0KICAgICAg ICAgICAgcG9zID0gMA0KICAgICAgICAgICAgd2hpbGUgcG9zIDwgbGVuKHNl bGYuZGF0YSkgYW5kIFwNCiAgICAgICAgICAgICAgICAgIHN0cmluZy5maW5k KHNlbGYuZGF0YSwgIiQiLCBwb3MpICE9IC0xOg0KICAgICAgICAgICAgICAg IGRvbGxhciA9IHN0cmluZy5maW5kKHNlbGYuZGF0YSwgIiQiLCBwb3MpDQog ICAgICAgICAgICAgICAgIyBhcHBlbmQgcHVyZSB0ZXh0IHdpdGhvdXQgZG9s bGFyDQogICAgICAgICAgICAgICAgb3V0ID0gb3V0ICsgc2VsZi5kYXRhW3Bv czpkb2xsYXJdDQogICAgICAgICAgICAgICAgcG9zID0gZG9sbGFyICsgMQ0K ICAgICAgICAgICAgICAgIG5leHRjaGFyID0gc2VsZi5kYXRhW2RvbGxhcisx XQ0KDQogICAgICAgICAgICAgICAgaWYgbmV4dGNoYXIgPT0gJ3snOg0KICAg ICAgICAgICAgICAgICAgICAjIGFwcGx5IGEgY29kZSBmcmFnbWVudA0KICAg ICAgICAgICAgICAgICAgICBsZXZlbCA9IDENCiAgICAgICAgICAgICAgICAg ICAgd2hpbGUgbGV2ZWw6DQogICAgICAgICAgICAgICAgICAgICAgICBwb3Mg PSBwb3MgKyAxDQogICAgICAgICAgICAgICAgICAgICAgICBpZiBzZWxmLmRh dGFbcG9zXSA9PSAieyI6DQogICAgICAgICAgICAgICAgICAgICAgICAgICAg bGV2ZWwgPSBsZXZlbCArIDENCiAgICAgICAgICAgICAgICAgICAgICAgIGVs aWYgc2VsZi5kYXRhW3Bvc10gPT0gIn0iOg0KICAgICAgICAgICAgICAgICAg ICAgICAgICAgIGxldmVsID0gbGV2ZWwgLSAxDQogICAgICAgICAgICAgICAg ICAgIG91dHB1dCA9IFtdDQogICAgICAgICAgICAgICAgICAgIGV4ZWMgc2Vs Zi5kYXRhWyhkb2xsYXIrMik6cG9zXSBpbiBkaWN0LCB7Im91dHB1dCI6b3V0 cHV0fQ0KICAgICAgICAgICAgICAgICAgICBmb3Igb2JqIGluIG91dHB1dDoN CiAgICAgICAgICAgICAgICAgICAgICAgIG91dCA9IG91dCArIG9iag0KICAg ICAgICAgICAgICAgICAgICBwb3MgPSBwb3MgKyAxDQoNCiAgICAgICAgICAg ICAgICBlbGlmIG5leHRjaGFyIGluIG5hbWVjaGFyczoNCiAgICAgICAgICAg ICAgICAgICAgIyBzY2FuIGEgdmFyaWFibGUgYWNjZXNzDQogICAgICAgICAg ICAgICAgICAgIHZhciA9IHJlZ2V4dmFyLm1hdGNoKHNlbGYuZGF0YVtwb3M6 XSkNCiAgICAgICAgICAgICAgICAgICAgaWYgdmFyOg0KICAgICAgICAgICAg ICAgICAgICAgICAgIyBIb3VzdG9uLCB3ZSBoYXZlIGEgbWF0Y2ghDQogICAg ICAgICAgICAgICAgICAgICAgICAjIGV4dHJhY3QgdGhlIHN0cmluZyBhbmQg ZXZhbHVhdGUgaXQNCiAgICAgICAgICAgICAgICAgICAgICAgIHZhciA9IHZh ci5ncm91cCgpDQogICAgICAgICAgICAgICAgICAgICAgICBwb3MgPSBwb3Mg KyBsZW4odmFyKQ0KICAgICAgICAgICAgICAgICAgICAgICAgb3V0ID0gb3V0 ICsgc3RyKGV2YWwodmFyLCBkaWN0KSkNCiAgICAgICAgICAgICAgICBlbHNl Og0KICAgICAgICAgICAgICAgICAgICBpZiBuZXh0Y2hhciA9PSAiJCI6DQog ICAgICAgICAgICAgICAgICAgICAgICBwb3MgPSBwb3MgKyAxDQogICAgICAg ICAgICAgICAgICAgICAgICBvdXQgPSBvdXQgKyAiJCINCg0KICAgICAgICAg ICAgaWYgcG9zIDwgbGVuKHNlbGYuZGF0YSk6IA0KICAgICAgICAgICAgICAg IG91dCA9IG91dCArIHNlbGYuZGF0YVtwb3M6XQ0KICAgICAgICBleGNlcHQ6 DQogICAgICAgICAgICBzeXMuc3RkZXJyLndyaXRlKCJUZW1wbGF0ZS5weTog cGFyc2UgZXJyb3IgYXQgcG9zICIrYHBvc2ArIjpcbiIrXA0KICAgICAgICAg ICAgICAgICAgICAgICAgICAgICBvdXQrIlxuIikNCg0KICAgICAgICByZXR1 cm4gb3V0DQoNCg== --1619418364-707833344-969446609=:31224-- From mal@lemburg.com Wed Sep 20 08:26:01 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed Sep 20 07:26:01 2000 Subject: [Distutils] Patch to have distutils generate PYC *and* PYO files Message-ID: <39C89E84.64ECE3A5@lemburg.com> This is a multi-part message in MIME format. --------------BFA08A6E6BF635BA05A3FB1C Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Attached you find a patch against the current CVS tree which allows distutils to include PYC as well as PYO files. I haven't dug deep into the option system yet, but it would be nice if there also were an option which lets the packager set the optimization level she want to use for the PYO files. Currently level 1 is used. Level 2 would also eliminate doc-strings which can sometimes be useful to reduce package size or to hide interface information. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ --------------BFA08A6E6BF635BA05A3FB1C Content-Type: text/plain; charset=us-ascii; name="distutils-pyo.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="distutils-pyo.patch" --- CVS-Python/Lib/distutils/util.py Mon Sep 18 10:31:17 2000 +++ Python+Unicode/Lib/distutils/util.py Wed Sep 20 13:14:27 2000 @@ -5,11 +5,11 @@ one of the other *util.py modules.""" # created 1999/03/08, Greg Ward __revision__ = "$Id: util.py,v 1.44 2000/09/15 01:16:14 gward Exp $" -import sys, os, string, re, shutil +import sys, os, string, re, shutil, py_compile from distutils.errors import * from distutils.spawn import spawn def get_platform (): @@ -265,5 +265,39 @@ def execute (func, args, msg=None, verbo # And do it, as long as we're not in dry-run mode if not dry_run: apply(func, args) # execute() + + +def compile(filename, optimize=None): + + """ Compile the Python script filename. + + If optimize is not given, the current optimization setting is + used. Otherwise, optimize must be an integer stating the value + of the optimization level to use: + + 0 - no optimization (generates .pyc file) + 1 - normal optimization (generates .pyo file, same as -O) + 2 - optimize + remove doc-strings (generates .pyo file, -OO) + + """ + if (optimize is None) or \ + (not optimize and __debug__) or \ + (optimize and not __debug__): + return py_compile.compile(filename) + + # Otherwise will have to run a system command + python = sys.executable + filename = os.path.abspath(filename) + if optimize: + options = '-' + 'O' * optimize + else: + options = '' + command = "import py_compile; py_compile.compile('%s')" % filename + rc = os.system('%s %s -c "%s"' % (python, options, command)) + if rc != 0: + # py_compile currently does not pass back errors, so this + # is not really executed... + raise SyntaxError,'could not compile "%s"' % filename + --- CVS-Python/Lib/distutils/command/install_lib.py Mon Sep 18 10:31:17 2000 +++ Python+Unicode/Lib/distutils/command/install_lib.py Wed Sep 20 13:19:11 2000 @@ -61,27 +61,24 @@ class install_lib (Command): else: self.warn("'%s' does not exist -- no Python modules to install" % self.build_dir) return - # (Optionally) compile .py to .pyc - # XXX hey! we can't control whether we optimize or not; that's up - # to the invocation of the current Python interpreter (at least - # according to the py_compile docs). That sucks. - + # (Optionally) compile .py to .pyc and .pyo if self.compile: - from py_compile import compile + from distutils.util import compile for f in outfiles: # only compile the file if it is actually a .py file if f[-3:] == '.py': - out_fn = f + (__debug__ and "c" or "o") - compile_msg = "byte-compiling %s to %s" % \ - (f, os.path.basename (out_fn)) - skip_msg = "skipping byte-compilation of %s" % f - self.make_file (f, out_fn, compile, (f,), - compile_msg, skip_msg) + for optimize in (0, 1): + out_fn = f + (optimize and "o" or "c") + compile_msg = "byte-compiling %s to %s" % \ + (f, os.path.basename (out_fn)) + skip_msg = "skipping byte-compilation of %s" % f + self.make_file (f, out_fn, compile, (f, optimize), + compile_msg, skip_msg) # run () def _mutate_outputs (self, has_any, build_cmd, cmd_option, output_dir): @@ -102,12 +99,12 @@ class install_lib (Command): # _mutate_outputs () def _bytecode_filenames (self, py_filenames): bytecode_files = [] for py_file in py_filenames: - bytecode = py_file + (__debug__ and "c" or "o") - bytecode_files.append(bytecode) + bytecode_files.append(py_file + "c") + bytecode_files.append(py_file + "o") return bytecode_files def get_outputs (self): """Return the list of files that would be installed if this command --------------BFA08A6E6BF635BA05A3FB1C-- From mal@lemburg.com Wed Sep 20 09:13:01 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed Sep 20 08:13:01 2000 Subject: [Distutils] Bug in parsing of .cfg files Message-ID: <39C8A99D.DB32FF8F@lemburg.com> The following .cfg file produces an error: [sdist] manifest = MANIFEST no-defaults = 1 keep-tree = 1 formats=bztar,gztar,tar,zip,ztar ... error: error in setup.cfg: command 'sdist' has no such option 'keep-tree' Looks as if the config file parser doesn't like options with embedded hyphens (or maybe it's just me not knowing how to pass the options ?). -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From R.Liebscher@gmx.de Wed Sep 20 10:05:15 2000 From: R.Liebscher@gmx.de (Rene Liebscher) Date: Wed Sep 20 09:05:15 2000 Subject: [Distutils] Bug in parsing of .cfg files References: <39C8A99D.DB32FF8F@lemburg.com> Message-ID: <39C8B541.6B3CF067@gmx.de> "M.-A. Lemburg" wrote: > > The following .cfg file produces an error: > > [sdist] > manifest = MANIFEST > no-defaults = 1 > keep-tree = 1 > formats=bztar,gztar,tar,zip,ztar > > ... > error: error in setup.cfg: command 'sdist' has no such option 'keep-tree' > > Looks as if the config file parser doesn't like options with > embedded hyphens (or maybe it's just me not knowing how to > pass the options ?). > You have to call it keep_tree. It is documented in "Distributing Python Modules" "4. Writing the Setup Configuration File". ( Note that an option spelled --foo-bar on the command-line is spelled foo_bar in configuration files. ) May be distutils should accept both, foo-bar and foo_bar. It is not to hard to change it, lets see what is Gregs opinion. kind regards Rene Liebscher From mal@lemburg.com Wed Sep 20 10:18:02 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed Sep 20 09:18:02 2000 Subject: [Distutils] Bug in parsing of .cfg files References: <39C8A99D.DB32FF8F@lemburg.com> <39C8B541.6B3CF067@gmx.de> Message-ID: <39C8B8E2.82B6E5E8@lemburg.com> Rene Liebscher wrote: > > "M.-A. Lemburg" wrote: > > > > The following .cfg file produces an error: > > > > [sdist] > > manifest = MANIFEST > > no-defaults = 1 > > keep-tree = 1 > > formats=bztar,gztar,tar,zip,ztar > > > > ... > > error: error in setup.cfg: command 'sdist' has no such option 'keep-tree' > > > > Looks as if the config file parser doesn't like options with > > embedded hyphens (or maybe it's just me not knowing how to > > pass the options ?). > > > You have to call it keep_tree. > It is documented in "Distributing Python Modules" > "4. Writing the Setup Configuration File". > ( Note that an option spelled --foo-bar on the command-line is > spelled foo_bar in configuration files. ) Ah ok. Thanks... after having made the change I now get: error: error in setup.cfg: command 'sdist' has no such option 'no_defaults' > May be distutils should accept both, foo-bar and foo_bar. > It is not to hard to change it, lets see what is Gregs opinion. Would be easier to understand, I guess. Thanks, -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From fdrake@beopen.com Wed Sep 20 10:25:02 2000 From: fdrake@beopen.com (Fred L. Drake, Jr.) Date: Wed Sep 20 09:25:02 2000 Subject: [Distutils] Bug in parsing of .cfg files In-Reply-To: <39C8B8E2.82B6E5E8@lemburg.com> References: <39C8A99D.DB32FF8F@lemburg.com> <39C8B541.6B3CF067@gmx.de> <39C8B8E2.82B6E5E8@lemburg.com> Message-ID: <14792.47642.653468.987083@cj42289-a.reston1.va.home.com> Rene Liebscher wrote: > May be distutils should accept both, foo-bar and foo_bar. > It is not to hard to change it, lets see what is Gregs opinion. M.-A. Lemburg writes: > Would be easier to understand, I guess. Wouldn't it be easiest to understand if it just used the hyphen in both places? The underscore is a programmers' character, not meant for end users! -Fred -- Fred L. Drake, Jr. BeOpen PythonLabs Team Member From R.Liebscher@gmx.de Wed Sep 20 10:28:01 2000 From: R.Liebscher@gmx.de (Rene Liebscher) Date: Wed Sep 20 09:28:01 2000 Subject: [Distutils] Bug in parsing of .cfg files References: <39C8A99D.DB32FF8F@lemburg.com> <39C8B541.6B3CF067@gmx.de> <39C8B8E2.82B6E5E8@lemburg.com> Message-ID: <39C8BA80.E246E530@gmx.de> "M.-A. Lemburg" wrote: > > Rene Liebscher wrote: > > > > "M.-A. Lemburg" wrote: > > > > > > The following .cfg file produces an error: > > > > > > [sdist] > > > manifest = MANIFEST > > > no-defaults = 1 > > > keep-tree = 1 > > > formats=bztar,gztar,tar,zip,ztar > > > > > > ... ... > > Ah ok. Thanks... after having made the change I now get: > > error: error in setup.cfg: command 'sdist' has no such option 'no_defaults' > There are two options 'use_defaults' and 'no_defaults'. 'use_defaults' is normally 1 and will be set to 0 if you give the 'no-defaults' option. 'use_defaults = 0' instead 'no_defaults = 1' should work. kind regards Rene Liebscher From calvin@cs.uni-sb.de Wed Sep 20 10:33:01 2000 From: calvin@cs.uni-sb.de (Bastian Kleineidam) Date: Wed Sep 20 09:33:01 2000 Subject: [Distutils] Bug in parsing of .cfg files In-Reply-To: <14792.47642.653468.987083@cj42289-a.reston1.va.home.com> Message-ID: Hello Fred, > Wouldn't it be easiest to understand if it just used the hyphen in >both places? The underscore is a programmers' character, not meant >for end users! It is very easy to "parse" config files if they are written in correct Python code. You just use the Python builtin program parser. Unfortunately Python does not allow a hyphen in a variable name. Bastian From fdrake@beopen.com Wed Sep 20 10:38:01 2000 From: fdrake@beopen.com (Fred L. Drake, Jr.) Date: Wed Sep 20 09:38:01 2000 Subject: [Distutils] Bug in parsing of .cfg files In-Reply-To: References: <14792.47642.653468.987083@cj42289-a.reston1.va.home.com> Message-ID: <14792.48423.114914.403502@cj42289-a.reston1.va.home.com> Bastian Kleineidam writes: > It is very easy to "parse" config files if they are written in correct > Python code. You just use the Python builtin program parser. > Unfortunately Python does not allow a hyphen in a variable name. That doesn't seem like the right way to parse config data. Why not use the ConfigParser module? Hyphens are allowed, and the syntax seems to match. (Greg might even have written it this way; not sure.) -Fred -- Fred L. Drake, Jr. BeOpen PythonLabs Team Member From mal@lemburg.com Wed Sep 20 10:48:02 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed Sep 20 09:48:02 2000 Subject: [Distutils] Bug in parsing of .cfg files References: <39C8A99D.DB32FF8F@lemburg.com> <39C8B541.6B3CF067@gmx.de> <39C8B8E2.82B6E5E8@lemburg.com> <39C8BA80.E246E530@gmx.de> Message-ID: <39C8BFEA.6AB7E1F9@lemburg.com> Rene Liebscher wrote: > > "M.-A. Lemburg" wrote: > > > > Rene Liebscher wrote: > > > > > > "M.-A. Lemburg" wrote: > > > > > > > > The following .cfg file produces an error: > > > > > > > > [sdist] > > > > manifest = MANIFEST > > > > no-defaults = 1 > > > > keep-tree = 1 > > > > formats=bztar,gztar,tar,zip,ztar > > > > > > > > ... > ... > > > > Ah ok. Thanks... after having made the change I now get: > > > > error: error in setup.cfg: command 'sdist' has no such option 'no_defaults' > > > There are two options 'use_defaults' and 'no_defaults'. > 'use_defaults' is normally 1 and will be set to 0 > if you give the 'no-defaults' option. > > 'use_defaults = 0' instead 'no_defaults = 1' should work. Weird, but works ;-) Thanks, -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal@lemburg.com Wed Sep 20 10:51:01 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed Sep 20 09:51:01 2000 Subject: [Distutils] Bug in parsing of .cfg files References: <39C8A99D.DB32FF8F@lemburg.com> <39C8B541.6B3CF067@gmx.de> <39C8B8E2.82B6E5E8@lemburg.com> <14792.47642.653468.987083@cj42289-a.reston1.va.home.com> Message-ID: <39C8C071.6F196F00@lemburg.com> "Fred L. Drake, Jr." wrote: > > Rene Liebscher wrote: > > May be distutils should accept both, foo-bar and foo_bar. > > It is not to hard to change it, lets see what is Gregs opinion. > > M.-A. Lemburg writes: > > Would be easier to understand, I guess. > > Wouldn't it be easiest to understand if it just used the hyphen in > both places? The underscore is a programmers' character, not meant > for end users! I guess it would break existing setup.cfg files, but having both and just documenting one (the hyphen version) should "fix" this for future setup.cfg writers ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From fdrake@beopen.com Wed Sep 20 11:39:01 2000 From: fdrake@beopen.com (Fred L. Drake, Jr.) Date: Wed Sep 20 10:39:01 2000 Subject: [Distutils] Bug in parsing of .cfg files In-Reply-To: <39C8C071.6F196F00@lemburg.com> References: <39C8A99D.DB32FF8F@lemburg.com> <39C8B541.6B3CF067@gmx.de> <39C8B8E2.82B6E5E8@lemburg.com> <14792.47642.653468.987083@cj42289-a.reston1.va.home.com> <39C8C071.6F196F00@lemburg.com> Message-ID: <14792.52080.167792.80929@cj42289-a.reston1.va.home.com> M.-A. Lemburg writes: > I guess it would break existing setup.cfg files, but having both > and just documenting one (the hyphen version) should "fix" this > for future setup.cfg writers ;-) Annoying, but I could live with that. -Fred -- Fred L. Drake, Jr. BeOpen PythonLabs Team Member From thomas.heller@ion-tof.com Wed Sep 20 12:11:04 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Wed Sep 20 11:11:04 2000 Subject: [Distutils] Re: [Python-checkins] CVS: distutils/distutils/command bdist_wininst.py,1.12,1.13 References: <200009191110.EAA13195@slayer.i.sourceforge.net> <20000919201346.A587@beelzebub> Message-ID: <00e201c02314$e65ac3c0$4500a8c0@thomasnb> > On 19 September 2000, Thomas Heller said: > > Modified Files: > > bdist_wininst.py > > Log Message: > > Set the 'nt' installation scheme for the install command even if run > > on other systems, so that data, headers, scripts are included in > > the installer. > > Ahh good, I was hoping you'd take care of this. But: > > > *** bdist_wininst.py 2000/09/16 15:56:32 1.12 > > --- bdist_wininst.py 2000/09/19 11:10:23 1.13 > > *************** > > *** 75,78 **** > > --- 75,89 ---- > > install = self.reinitialize_command('install') > > install.root = self.bdist_dir > > + if os.name != 'nt': > > + # must force install to use the 'nt' scheme > > + install.select_scheme ('nt') > > + # change the backslash to the current pathname separator > > + for key in ('purelib', 'platlib', 'headers', 'scripts', > > + 'data'): > > + attrname = 'install_' + key > > + attr = getattr (install, attrname) > > + if attr: > > + attr = string.replace (attr, '\\', os.sep) > > + setattr (install, attrname, attr) > [Greg] > I think we can get rid of this hack by changing the INSTALL_SCHEMES > dictionary in install.py to store paths in Unix form, and then convert > to native when they're needed. That is, the install path "templates" in > install.py would be treated just like install paths in the setup script > (or on the command line) -- convert from foo/bar/baz to foo\bar\baz on > Windows, and foo:bar:baz on Mac OS. (Modulo Mac OS' peculiar convention > for absolute/relative directories: not sure if 'convert_path()' handles > that completely right.) > I'll remove this hack in bdist_wininst, and you take care of changing INSTALL_SCHEMES and converting paths ;-) OK? Thomas From uche.ogbuji@fourthought.com Wed Sep 20 15:06:02 2000 From: uche.ogbuji@fourthought.com (Uche Ogbuji) Date: Wed Sep 20 14:06:02 2000 Subject: [Distutils] Appears bdist_rpm completely broken Message-ID: <39C8FC9F.3EE734AA@fourthought.com> Maybe I'm missing something, but I don't see how bdist_rpm works for anyone. I've tried 0.9.2, the 9/12 and 9/16 snapshots but I have a problem with the generated pycs. First the symptom. I build 4Suite using distutils. The sdist, bdist and bdist_win packages work just fine, but when I move the generated RPM to another machine and try to use 4Suite: [uogbuji@dynip-23 uogbuji]$ python Python 1.5.2 (#1, Jul 12 2000, 10:39:14) [GCC 2.96 20000702 (experimental)] on linux-i386 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>> import sys >>> sys.path ['', '/usr/lib/python1.5/', '/usr/lib/python1.5/plat-linux-i386', '/usr/lib/python1.5/lib-tk', '/usr/lib/python1.5/lib-dynload', '/usr/lib/python1.5/site-packages'] >>> import xml.xslt Traceback (innermost last): File "", line 1, in ? File "/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages/xml/xslt/__init__.py", line 3, in ? File "/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages/xml/xpath/__init__.py", line 3, in ? ImportError: cannot import name XPathParserBase >>> The problem is the phantom "/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages"... [uogbuji@dynip-23 uogbuji]$ ls /var/tmp [uogbuji@dynip-23 uogbuji]$ Further investigation turns up the culprit: The generated pyc files from the RPM have the "/var/tmp/..." hard-coded in to their imports. So am I doing something wrong, or is distutils broken for RPM build? I'm not on the distutils list so ccs to me would be appreciated. Otherwise I can check the archives. Thanks. -- Uche Ogbuji Principal Consultant uche.ogbuji@fourthought.com +1 303 583 9900 x 101 Fourthought, Inc. http://Fourthought.com 4735 East Walnut St, Ste. C, Boulder, CO 80301-2537, USA Software-engineering, knowledge-management, XML, CORBA, Linux, Python From akuchlin@mems-exchange.org Wed Sep 20 15:17:06 2000 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Wed Sep 20 14:17:06 2000 Subject: [Distutils] Appears bdist_rpm completely broken In-Reply-To: <39C8FC9F.3EE734AA@fourthought.com>; from uche.ogbuji@fourthought.com on Wed, Sep 20, 2000 at 12:06:23PM -0600 References: <39C8FC9F.3EE734AA@fourthought.com> Message-ID: <20000920141639.C12159@kronos.cnri.reston.va.us> On Wed, Sep 20, 2000 at 12:06:23PM -0600, Uche Ogbuji wrote: >Maybe I'm missing something, but I don't see how bdist_rpm works for >anyone. I've tried 0.9.2, the 9/12 and 9/16 snapshots but I have a >problem with the generated pycs. It generated ZODB & ExtensionClass RPMs that worked fine, and those modules both involve mixed Python and C module, which is about as complicated as you can get. >line 3, in ? >ImportError: cannot import name XPathParserBase >>>> What exactly is the bug? Do you not expect that ImportError to be raised? Or is it just that the path names shown in tracebacks are wrong? >Further investigation turns up the culprit: The generated pyc files >from the RPM have the "/var/tmp/..." hard-coded in to their imports. Presumably bdist_rpm shouldn't contain .pyc files, and should instead use a postinstall script to compile the .py files. --amk From uche.ogbuji@fourthought.com Wed Sep 20 15:29:03 2000 From: uche.ogbuji@fourthought.com (Uche Ogbuji) Date: Wed Sep 20 14:29:03 2000 Subject: [Distutils] Appears bdist_rpm completely broken References: <39C8FC9F.3EE734AA@fourthought.com> <20000920141639.C12159@kronos.cnri.reston.va.us> Message-ID: <39C901BE.36C986C9@fourthought.com> Andrew Kuchling wrote: > >line 3, in ? > >ImportError: cannot import name XPathParserBase > >>>> > > What exactly is the bug? Do you not expect that ImportError to be > raised? Or is it just that the path names shown in tracebacks are > wrong? The ImportError is being raised _because_ of the wrong pathnames. If I perform the exact same procedure on the exact same machine with a 4Suite installed from the distutils sdist, I get no such ImportError. > >Further investigation turns up the culprit: The generated pyc files > >from the RPM have the "/var/tmp/..." hard-coded in to their imports. > > Presumably bdist_rpm shouldn't contain .pyc files, and should instead > use a postinstall script to compile the .py files. Incorrect. Here is the .spec that bdist_rpm generates: %define name 4Suite %define version 0.9.0 %define release 1 Summary: 4Suite Description Name: %{name} Version: %{version} Release: %{release} Source0: %{name}-%{version}.tar.gz Copyright: UNKNOWN Group: Development/Libraries BuildRoot: %{_tmppath}/%{name}-buildroot Prefix: %{_prefix} Vendor: Fourthought, Inc. Url: http://www.4suite.org %description 4Suite is a collection of Python tools for XML processing and object-databases. 4Suite is an integrated packaging of 4DOM, 4XPath, 4XSLT, 4RDF, and 4ODS. %prep %setup %build env CFLAGS="$RPM_OPT_FLAGS" python setup.py build %install python setup.py install --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES %clean rm -rf $RPM_BUILD_ROOT %files -f INSTALLED_FILES %defattr(-,root,root) As you can see, there is no post-installation script registration. Note that I can probably solve this probblem by building the RPM by hand without the BuildRoot line or the "--root=$RPM_BUILD_ROOT" argument in the "python setup.py...". Of course I'll need a smarter %clean section as well. If so, I'll post a working .spec file for additional data. In case it helps anyone to look at the package, there is a 4Suite sdist available at ftp://ftp.fourthought.com/pub/4Suite/4Suite-0.9.0rc1.tar.gz Note the prerequisite of ftp://ftp.fourthought.com/pub/4Suite/BisonGen-0.5.0rc1.tar.gz -- Uche Ogbuji Principal Consultant uche.ogbuji@fourthought.com +1 303 583 9900 x 101 Fourthought, Inc. http://Fourthought.com 4735 East Walnut St, Ste. C, Boulder, CO 80301-2537, USA Software-engineering, knowledge-management, XML, CORBA, Linux, Python From Mike.Olson@fourthought.com Wed Sep 20 18:52:02 2000 From: Mike.Olson@fourthought.com (Mike Olson) Date: Wed Sep 20 17:52:02 2000 Subject: [Distutils] Extension Modules Message-ID: <39C9315A.36F0574@FourThought.com> Hello, We are working towards converting 4Suite over to use distutils. We are almost there, but have run into a problem. We needed to implement our own parser to generate some files using a script we call BisonGen. This generates both C and py files. Then we hand the C off to build_ext to get compilied. The .so (or pyd) is installed fine. However, the additional py files generated are not copied into the installation directory. Is there a simple way of doing this? Looking through the code, build_ext only uses Extension.name to generate a list of outputs. I've tried a couple of approaches, but all seem to break someother portion of the script (usually bdist --format=rpm). We have the same problem with shadow files generated via swig..... Help is appreciated. Mike -- Mike Olson Principal Consultant mike.olson@fourthought.com (303)583-9900 x 102 Fourthought, Inc. http://Fourthought.com Software-engineering, knowledge-management, XML, CORBA, Linux, Python From uche.ogbuji@fourthought.com Wed Sep 20 21:06:48 2000 From: uche.ogbuji@fourthought.com (Uche Ogbuji) Date: Wed Sep 20 20:06:48 2000 Subject: [Distutils] How I sorted things out Message-ID: <39C950E7.DEFA0661@fourthought.com> I ended up doing things by hand. I'd like to have distutils help automate things, and if it's any help, here's what I ended up doing. I had to do all the following as root, which I know distutils tries to avoid. Maybe there's another way around. I copied the spec file from build/bdist.linux-i686/rpm/SOURCES/4Suite-0.9.0.tar.gz to /usr/src/redhat/SOURCES and used the following spec file (mods from distutils) -----------------------------------%-------------------------------------- %define name 4Suite %define version 0.9.0 %define release 1 Summary: Tools for XML, DOM, XPath, XSLT, RDF and object database Name: %{name} Version: %{version} Release: %{release} Source0: %{name}-%{version}.tar.gz Copyright: FourThought, Inc Group: Development/Languages/Python Vendor: Fourthought, Inc. Url: http://www.4suite.org %description 4Suite is a collection of Python tools for XML processing and object-databases. 4Suite is an integrated packaging of 4DOM, 4XPath, 4XSLT, 4RDF, and 4ODS. %prep %setup %build env CFLAGS="$RPM_OPT_FLAGS" python setup.py build %install python setup.py install --record=INSTALLED_FILES #%clean %files -f INSTALLED_FILES /usr/lib/python1.5/site-packages/xml/xpath/XPath.py /usr/lib/python1.5/site-packages/xml/xslt/XPattern.py /usr/lib/python1.5/site-packages/Ft/Ods/Parsers/Odl/Odl.py /usr/lib/python1.5/site-packages/Ft/Ods/Parsers/Oql/Oql.py /usr/lib/python1.5/site-packages/Ft/Ods/Parsers/Oif/Oif.py /usr/lib/python1.5/site-packages/xml/xpath/XPathParserBase.py /usr/lib/python1.5/site-packages/xml/xslt/XPatternParserBase.py /usr/lib/python1.5/site-packages/Ft/Ods/Parsers/Odl/OdlParserBase.py /usr/lib/python1.5/site-packages/Ft/Ods/Parsers/Odl/OdlParseTreeMapping.py /usr/lib/python1.5/site-packages/Ft/Ods/Parsers/Oql/OqlParserBase.py /usr/lib/python1.5/site-packages/Ft/Ods/Parsers/Oql/OqlParseTreeTuple.py /usr/lib/python1.5/site-packages/Ft/Ods/Parsers/Oif/OifParserBase.py /usr/lib/python1.5/site-packages/Ft/Ods/Parsers/Oif/OifParseTreeMapping.py /usr/lib/python1.5/site-packages/xml/xpath/XPath.py? /usr/lib/python1.5/site-packages/xml/xslt/XPattern.py? /usr/lib/python1.5/site-packages/Ft/Ods/Parsers/Odl/Odl.py? /usr/lib/python1.5/site-packages/Ft/Ods/Parsers/Oql/Oql.py? /usr/lib/python1.5/site-packages/Ft/Ods/Parsers/Oif/Oif.py? /usr/lib/python1.5/site-packages/xml/xpath/XPathParserBase.py? /usr/lib/python1.5/site-packages/xml/xslt/XPatternParserBase.py? /usr/lib/python1.5/site-packages/Ft/Ods/Parsers/Odl/OdlParserBase.py? /usr/lib/python1.5/site-packages/Ft/Ods/Parsers/Odl/OdlParseTreeMapping.py? /usr/lib/python1.5/site-packages/Ft/Ods/Parsers/Oql/OqlParserBase.py? /usr/lib/python1.5/site-packages/Ft/Ods/Parsers/Oql/OqlParseTreeTuple.py? /usr/lib/python1.5/site-packages/Ft/Ods/Parsers/Oif/OifParserBase.py? /usr/lib/python1.5/site-packages/Ft/Ods/Parsers/Oif/OifParseTreeMapping.py? %defattr(-,root,root) %changelog * Wed Sep 20 2000 Uche Ogbuji - initial RPM release -----------------------------------%-------------------------------------- Note that we have to manually list out more files than setup.py recognizes. This is because of the limitation of distutils with regard to extensions with multiple output files (see Mike Olson's message to distutils-sig today). I don't know if things can be made smoother. Are we missing something? Thanks. -- Uche Ogbuji Principal Consultant uche.ogbuji@fourthought.com +1 303 583 9900 x 101 Fourthought, Inc. http://Fourthought.com 4735 East Walnut St, Ste. C, Boulder, CO 80301-2537, USA Software-engineering, knowledge-management, XML, CORBA, Linux, Python From thomas.heller@ion-tof.com Thu Sep 21 12:12:01 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Thu Sep 21 11:12:01 2000 Subject: [Distutils] test command for setup.py References: Message-ID: <009901c023de$3df93cf0$4500a8c0@thomasnb> I like the simplicity of your test command, however I would like (and have partly implmented) the following changes: - don't use so many options, the build directories could be retrieved from the 'build' command - wrap 'try' blocks around running the tests - output a summary of failed and passed tests I will probably post my script later. Thomas From bhoel@starship.python.net Thu Sep 21 13:14:02 2000 From: bhoel@starship.python.net (Berthold =?iso-8859-1?q?H=F6llmann?=) Date: Thu Sep 21 12:14:02 2000 Subject: [Distutils] test command for setup.py In-Reply-To: "Greg Ward"'s message of "Tue, 19 Sep 2000 21:25:37 -0400" References: <20000919212537.B648@beelzebub> Message-ID: >>>>> "Greg" =3D=3D Greg Ward writes: Greg> On 18 September 2000, Berthold H=F6llmann said: >> I'm thinking about providing a framework for pre-install >> testing for distutils enabled Python packages. I think about a >> new "test" command, which inserts the lib build path into >> PYTHONPATH and runs specified tests. Is anyone else working on >> something like this, and what are your opinions and >> requirements on this? Greg> I've been thinking about it from Day 1, personally. Here are Greg> some of my opinions/requirements: Greg> * serious tests don't belong in the module being tested, Greg> because test code tends to be at least as long as the code Greg> being tested I think it is easier for the developer to have code and tests together. One can leave out the tests from any distribution by using a Manifest file. But for the developer it is helpful to have a set of tests that one can run after changes have been made to the sources to enshure that "little" changes have not unexpected side effects. Greg> ... Greg> * the requirements on what a test script may output should Greg> be fairly strict. My favourite is loosely based on Perl's Greg> testing convention, which is a great success largely because Greg> the standard-way-to-build- Perl-modules includes a "make Greg> test" step, and it's always immediately clear if everything Greg> passed. My proposed convention is this: each test script Greg> outputs a line saying how many tests are expected, then a Greg> series of lines that start with either "ok" or "not ok". If Greg> there are any "not ok" lines, the test script fails; if the Greg> number of "ok" lines !=3D the number of expected tests, the Greg> test script fails. Each "ok" or "not ok" may be followed by Greg> ":" and an arbitrary string, which makes it a lot easier to Greg> track down problems. (The Perl convention is "ok 1", "ok 2", Greg> ... "ok N", which makes it really awkward to track down Greg> problems.) Any lines not starting with "ok" or "not ok" are Greg> ignored. Thats not always an option. I'm writing modules using numerical opreations. These require test results to be looked over very carefully to decide, whether the differences are significant or not. Also the underlaying Fortran code makes different decisions on different platforms. That makes automatic testing nearly impossible, but I want at least a constistant way to generate the needed output, and I want to test everything before I install it. Greg> ... Greg> * we might also want to accomodate a Python-style test Greg> suite, where you compare "known good" output of a script Greg> with the current output. I'm not a fan of this methodology, Greg> but sometimes it's easier, and it may have some currency in Greg> the Python community outside of Lib/test. It should be the Greg> developer's choice what style of test suite he wants to Greg> write. All this can be done with my posted, extremly lightweight test.py extension for the _command_ subdirectory. The test scripts can written to implement/use any testframework you like. Maybe a >python setup.py test command is only a convenience command but it gives you a starting point when maintaining code you wrote ages ago, or somebody else wrote. Greetings Berthold --=20 bhoel@starship.python.net / http://starship.python.net/crew/bhoel/ It is unlawful to use this email address for unsolicited ads (USC Title 47 Sec.227). I will assess a US$500 charge for reviewing and deleting each unsolicited ad. From gward@python.net Thu Sep 21 22:34:01 2000 From: gward@python.net (Greg Ward) Date: Thu Sep 21 21:34:01 2000 Subject: [Distutils] Re: [Python-checkins] CVS: distutils/distutils/command bdist_wininst.py,1.12,1.13 In-Reply-To: <00e201c02314$e65ac3c0$4500a8c0@thomasnb>; from thomas.heller@ion-tof.com on Wed, Sep 20, 2000 at 05:10:17PM +0200 References: <200009191110.EAA13195@slayer.i.sourceforge.net> <20000919201346.A587@beelzebub> <00e201c02314$e65ac3c0$4500a8c0@thomasnb> Message-ID: <20000921213224.A937@beelzebub> On 20 September 2000, Thomas Heller said: > I'll remove this hack in bdist_wininst, and you take care of > changing INSTALL_SCHEMES and converting paths ;-) > OK? OK, done. I went ahead and changed bdist_wininst; also set install.prefix so that the pseudo-installation is done to "build/bdist.linux-i586/wininst/Python" rather than "build/bdist.linux-i586/wininst/usr", which works fine but looks silly. Greg From gward@python.net Thu Sep 21 22:58:01 2000 From: gward@python.net (Greg Ward) Date: Thu Sep 21 21:58:01 2000 Subject: [Distutils] test command for setup.py In-Reply-To: ; from bhoel@starship.python.net on Thu, Sep 21, 2000 at 10:39:34AM +0200 References: <20000919212537.B648@beelzebub> Message-ID: <20000921215623.A991@beelzebub> On 21 September 2000, Berthold Höllmann said: > I think it is easier for the developer to have code and tests > together. Not if the tests are really exhaustive! When I write good tests (or at least what I think are good tests), the test script is roughly the size of the module being tested. I don't want my "if __name__ == '__main__'" block to be as big as the rest of the module. I think the "test" command should support "in-situ" testing, but it's *not* enough for serious tests. > One can leave out the tests from any distribution by using a > Manifest file. Huh? That loses half the point of writing tests, which is to assure people building and installing your modules that they probably work as advertised on their platform. > But for the developer it is helpful to have a set of > tests that one can run after changes have been made to the sources to > enshure that "little" changes have not unexpected side effects. Yes! That's the other half of the reason for writing tests. (You can define "half" here as 40-60, 20-80, whatever.) > Thats not always an option. I'm writing modules using numerical > opreations. These require test results to be looked over very > carefully to decide, whether the differences are significant or not. Oooh, yes, in a previous life I struggled (briefly) with automated testing of numerical algorithms. Not fun. But asking end-users to examine the test output and make sure that the difference between 1.0234324 and 1.0234821 is less than the machine epsilon is surely even less fun. > Also the underlaying Fortran code makes different decisions on > different platforms. That makes automatic testing nearly impossible, > but I want at least a constistant way to generate the needed output, > and I want to test everything before I install it. Here's my extreme position: If a test isn't fully automated -- i.e., if it takes more than epsilon human intelligence to determine if the test passed or failed -- then it's not a proper test, it's just a demo. Granted there are real-world situations that cannot meet this tough criterion -- testing a curses module, say, or some (many? most?) numerical algorithms. However, it is a worthy goal that, with some work, can often be achieved. Witness the number of Perl modules on CPAN for which you can run "make test" and immediately be assured that everything Just Works on the current platform and Perl interpreter. Thus, fully automated testing is to be encouraged! I like the "ok, ok, not ok, ok" method because it's immediately obvious what's going on, fairly easy (if tedious) to write tests with no external support, and shouldn't be too hard to write some sort of test harness framework. The framework would have two purposes: * make it easy for developers to write tests that output "ok, ok, not ok, ok" * support the Distutils "test" command that analyses that output and ensures that exactly the correct number of "ok" lines, and no "not ok" lines, were seen > All this can be done with my posted, extremly lightweight test.py > extension for the _command_ subdirectory. The test scripts can written > to implement/use any testframework you like. Maybe a OK, lightweight is good. (Something I occasionally need to be reminded of...) I'll definitely take a look at it. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Thu Sep 21 23:05:07 2000 From: gward@python.net (Greg Ward) Date: Thu Sep 21 22:05:07 2000 Subject: [Distutils] Latest snapshot Message-ID: <20000921220334.B991@beelzebub> Hi all -- I've just released another code snapshot. The main change is that Windows installers created on non-Windows platforms should now include scripts and data correctly. This involved some subtle rearrangement of the "install" command; as usual, It Works For Me (TM), but please try it out and see if it works for you. Also, some patches for Mac OS support have snuck in over the last couple of days. Download it from the usual place: http://www.python.org/sigs/distutils-sig/download.html#snapshot Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Thu Sep 21 23:11:17 2000 From: gward@python.net (Greg Ward) Date: Thu Sep 21 22:11:17 2000 Subject: [Distutils] Synchronizing version information In-Reply-To: <200009200956.LAA04691@pandora.informatik.hu-berlin.de>; from loewis@informatik.hu-berlin.de on Wed, Sep 20, 2000 at 11:56:53AM +0200 References: <200009200956.LAA04691@pandora.informatik.hu-berlin.de> Message-ID: <20000921220933.C991@beelzebub> On 20 September 2000, Martin von Loewis said: > In my package, I have two places which carry version information: in > the setup.py, when passing the version= keyword argument to setup, and > in the package's __init__, to allow runtime access to the available > version. I have the same problem -- I always have to update the Distutils setup.py and distutils/__init__.py in sync. Yuck. > I'd like these to be synchronized, having the information only in a > single place, and fetching it from there in the other. Is there any > mechanism in distutils to facilitate such synchronization? I just had an idea which requires no funky substitution mechanism, and no extra code in the Distutils: in the setup file, do this (or equivalent for your scenario): from distutils import core import distutils setup(name = "Distutils", version = distutils.__version__, ...) > One approach would be a autoconf-style replacement mechanism, where I > put > > version_info = distutils.version.StrictVersion(@VERSION@) Yuck. I'm not keen on this. It Just Seems Wrong to go hacking up Python source like that. (Although, mind you, the "build_scripts" command does hack up Python scripts a bit. But only the first line!) Another idea, stolen (as usual) from Perl's MakeMaker: have a "version_from" option in the setup script, eg. setup(name = "Distutils", version_from = "distutils/__init__.py", ...) A couple of models suggest themselves here: * grep the named file for /^__version__ = "([^"]+)"/ and use $1 * execfile the named file and extract __version__ from the module dict * don't give a filename, give a module name, and import __version__ from it Obviously, 'version' and 'version_from' would be mutually exclusive. Opinions? Greg Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Thu Sep 21 23:12:02 2000 From: gward@python.net (Greg Ward) Date: Thu Sep 21 22:12:02 2000 Subject: [Distutils] Synchronizing version information In-Reply-To: ; from calvin@cs.uni-sb.de on Wed, Sep 20, 2000 at 12:43:29PM +0200 References: <200009200956.LAA04691@pandora.informatik.hu-berlin.de> Message-ID: <20000921221025.D991@beelzebub> On 20 September 2000, Bastian Kleineidam said: > >Is there any mechanism in distutils to facilitate such synchronization? > You could use the attached Template.py file. > Then copy __init__.py to __init__.py.tmpl, insert $version and execute > t = Template.Template('__init__.py.tmpl') > f = open('__init__.py','w') > f.write(t.fill_in({'version', 0.1}) > f.close > > and $version gets replaced. As I suggested in my previous post, I'm "-1" on Autoconf-style substitution on Python source. It just doesn't seem necessary, and feels like it introduces all sorts of potential problems Greg From gward@python.net Fri Sep 22 00:05:01 2000 From: gward@python.net (Greg Ward) Date: Thu Sep 21 23:05:01 2000 Subject: [Distutils] Bug in parsing of .cfg files In-Reply-To: <39C8B8E2.82B6E5E8@lemburg.com>; from mal@lemburg.com on Wed, Sep 20, 2000 at 03:17:22PM +0200 References: <39C8A99D.DB32FF8F@lemburg.com> <39C8B541.6B3CF067@gmx.de> <39C8B8E2.82B6E5E8@lemburg.com> Message-ID: <20000921230332.A1158@beelzebub> On 20 September 2000, M.-A. Lemburg said: > Ah ok. Thanks... after having made the change I now get: > > error: error in setup.cfg: command 'sdist' has no such option 'no_defaults' I'm open to config files accepting hyphens as well as underscores. The current config parsing code was thrown together pretty quickly, and yes it *does* use ConfigParser. Accepting hyphens would make the documentation a bit simpler, which is always a good metric of software complexity. The no_default vs. use_defaults could definitely be massaged a bit. It's not documented at *all*; in fact, the docs as written would lead you to believe that anything you see in "--help" output can be used in a config file (modulo the underscore/hyphen thingy). But that's not so: some options are just "negative aliases" for others. The underlying reason for this is sound (--use-defaults and --no-default both set the same instance attribute: 'use_defaults'), but that's just an explanation, not an excuse. IMHO the config file should accept as close to what the command-line accepts as possible: this means hyphens too, and it means handling "negative alias" options. Hmmm. I'll look into it -- just added it to the TODO file, FWIW. Anyone who wants to give it a shot and submit a patch should look at the 'parse_config_files()' method in dist.py. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Fri Sep 22 00:23:02 2000 From: gward@python.net (Greg Ward) Date: Thu Sep 21 23:23:02 2000 Subject: [Distutils] Appears bdist_rpm completely broken In-Reply-To: <39C8FC9F.3EE734AA@fourthought.com>; from uche.ogbuji@fourthought.com on Wed, Sep 20, 2000 at 12:06:23PM -0600 References: <39C8FC9F.3EE734AA@fourthought.com> Message-ID: <20000921232109.B1158@beelzebub> On 20 September 2000, Uche Ogbuji said: > First the symptom. I build 4Suite using distutils. The sdist, bdist > and bdist_win packages work just fine, but when I move the generated RPM > to another machine and try to use 4Suite: > > [uogbuji@dynip-23 uogbuji]$ python > Python 1.5.2 (#1, Jul 12 2000, 10:39:14) [GCC 2.96 20000702 > (experimental)] on linux-i386 > Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam > >>> import sys > >>> sys.path > ['', '/usr/lib/python1.5/', '/usr/lib/python1.5/plat-linux-i386', > '/usr/lib/python1.5/lib-tk', '/usr/lib/python1.5/lib-dynload', > '/usr/lib/python1.5/site-packages'] > >>> import xml.xslt > Traceback (innermost last): > File "", line 1, in ? > File > "/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages/xml/xslt/__init__.py", > line 3, in ? > File > "/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages/xml/xpath/__init__.py", > line 3, in ? > ImportError: cannot import name XPathParserBase > >>> > > The problem is the phantom > "/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages"... That's bizarre. Having the wrong filename in the .pyc files is annoying, but I can't for the life of me see how it could be causing an ImportError. I've downloaded your two source dists to see if I can figure out what the heck's going on. Don't have anything yet, but I'm glad to see someone is using all those extensibility hooks I put in! AFAIK you're the first ones to define a new Distribution class -- cool! Thought I do wonder if all the heavy lifting you're doing to beat the Distutils into submission could be causing problems. It's idle speculation right now; I haven't even reproduced your problems yet, much less tried to debug them. Tomorrow... my brain is shutting down for the night right now. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From uche.ogbuji@fourthought.com Fri Sep 22 00:53:01 2000 From: uche.ogbuji@fourthought.com (Uche Ogbuji) Date: Thu Sep 21 23:53:01 2000 Subject: [Distutils] Appears bdist_rpm completely broken References: <39C8FC9F.3EE734AA@fourthought.com> <20000921232109.B1158@beelzebub> Message-ID: <39CAD7A6.2706EC3D@fourthought.com> Greg Ward wrote: "/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages/xml/xpath/__init__.py", > > line 3, in ? > > ImportError: cannot import name XPathParserBase > > >>> > > > > The problem is the phantom > > "/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages"... > > That's bizarre. Having the wrong filename in the .pyc files is > annoying, but I can't for the life of me see how it could be causing an > ImportError. We just found out that this is indeed not the problem. The problem was that RPM wasn't getting the right lists of installed files, so it was getting confused about where things were. > I've downloaded your two source dists to see if I can figure out what > the heck's going on. Don't have anything yet, but I'm glad to see > someone is using all those extensibility hooks I put in! AFAIK you're > the first ones to define a new Distribution class -- cool! You ain't seen nothing yet! We fixed a whole bunch of our additional packaging problems with more distutils subclassing. Check out ftp://ftp.fourthought.com/pub/4Suite/4Suite-0.9.0.2.tar.gz For the sordid mess... > Thought I do > wonder if all the heavy lifting you're doing to beat the Distutils into > submission could be causing problems. It's idle speculation right now; > I haven't even reproduced your problems yet, much less tried to debug > them. Tomorrow... my brain is shutting down for the night right now. We'd love to hear any feedback about our distutils efforts. We hardly thought we were doing anything extraordinary, but distutils does seem to have several limitations as is that we had to sub-class to get around. Maybe we were trying too hard in some cases. -- Uche Ogbuji Principal Consultant uche.ogbuji@fourthought.com +1 303 583 9900 x 101 Fourthought, Inc. http://Fourthought.com 4735 East Walnut St, Ste. C, Boulder, CO 80301-2537, USA Software-engineering, knowledge-management, XML, CORBA, Linux, Python From Mike.Olson@fourthought.com Fri Sep 22 01:00:15 2000 From: Mike.Olson@fourthought.com (Mike Olson) Date: Fri Sep 22 00:00:15 2000 Subject: [Distutils] Appears bdist_rpm completely broken References: <39C8FC9F.3EE734AA@fourthought.com> <20000921232109.B1158@beelzebub> <39CAD7A6.2706EC3D@fourthought.com> Message-ID: <39CAD927.3E3E1C7A@FourThought.com> Uche Ogbuji wrote: > > Greg Ward wrote: > > We'd love to hear any feedback about our distutils efforts. We hardly > thought we were doing anything extraordinary, but distutils does seem to > have several limitations as is that we had to sub-class to get around. > Maybe we were trying too hard in some cases. There were a couple of spots were if we could abstract core distutils code into a function call (that we could override) it would make our lives that much nicer. Ex, I needed on little hook into install_lib.run (after the build and before the copy), but I had to reproduce the entire function call... another really useful thing for us on copy_file would be if the function told you if the file was copied.... Mike > > -- > Uche Ogbuji Principal Consultant > uche.ogbuji@fourthought.com +1 303 583 9900 x 101 > Fourthought, Inc. http://Fourthought.com > 4735 East Walnut St, Ste. C, Boulder, CO 80301-2537, USA > Software-engineering, knowledge-management, XML, CORBA, Linux, Python > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG@python.org > http://www.python.org/mailman/listinfo/distutils-sig -- Mike Olson Principal Consultant mike.olson@fourthought.com (303)583-9900 x 102 Fourthought, Inc. http://Fourthought.com Software-engineering, knowledge-management, XML, CORBA, Linux, Python From mal@lemburg.com Fri Sep 22 05:04:00 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri Sep 22 04:04:00 2000 Subject: [Distutils] Bug in parsing of .cfg files References: <39C8A99D.DB32FF8F@lemburg.com> <39C8B541.6B3CF067@gmx.de> <39C8B8E2.82B6E5E8@lemburg.com> <20000921230332.A1158@beelzebub> Message-ID: <39CB1238.B879C132@lemburg.com> Greg Ward wrote: > > On 20 September 2000, M.-A. Lemburg said: > > Ah ok. Thanks... after having made the change I now get: > > > > error: error in setup.cfg: command 'sdist' has no such option 'no_defaults' > > I'm open to config files accepting hyphens as well as underscores. The > current config parsing code was thrown together pretty quickly, and yes > it *does* use ConfigParser. Accepting hyphens would make the > documentation a bit simpler, which is always a good metric of software > complexity. Indeed :-) > The no_default vs. use_defaults could definitely be massaged a bit. > It's not documented at *all*; in fact, the docs as written would lead > you to believe that anything you see in "--help" output can be used in a > config file (modulo the underscore/hyphen thingy). Right, that's what I thought... didn't work out all the way, but would certainly make understanding distutils a lot easier. > But that's not so: some options are just "negative aliases" for others. > The underlying reason for this is sound (--use-defaults and --no-default > both set the same instance attribute: 'use_defaults'), but that's just > an explanation, not an excuse. IMHO the config file should accept as > close to what the command-line accepts as possible: this means hyphens > too, and it means handling "negative alias" options. Right again :-) > Hmmm. I'll look into it -- just added it to the TODO file, FWIW. > Anyone who wants to give it a shot and submit a patch should look at the > 'parse_config_files()' method in dist.py. Apropos patch: have you seen the patch I posted here which allows distutils to compile PYC and PYO files in one go ? It basically adds a new util function compile() which takes care of the logic. Python's sys module should really support e.g. sys.setoptimization() for this kind of task... it's on the plate, but we're in a feature freeze so it won't make it into 2.0. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal@lemburg.com Fri Sep 22 05:42:01 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri Sep 22 04:42:01 2000 Subject: [Distutils] Synchronizing version information References: <200009200956.LAA04691@pandora.informatik.hu-berlin.de> <20000921220933.C991@beelzebub> Message-ID: <39CB1B1C.A4394DD3@lemburg.com> Greg Ward wrote: > > On 20 September 2000, Martin von Loewis said: > > In my package, I have two places which carry version information: in > > the setup.py, when passing the version= keyword argument to setup, and > > in the package's __init__, to allow runtime access to the available > > version. > > I have the same problem -- I always have to update the Distutils > setup.py and distutils/__init__.py in sync. Yuck. > > > I'd like these to be synchronized, having the information only in a > > single place, and fetching it from there in the other. Is there any > > mechanism in distutils to facilitate such synchronization? > > I just had an idea which requires no funky substitution mechanism, and > no extra code in the Distutils: in the setup file, do this (or > equivalent for your scenario): > > from distutils import core > import distutils > > setup(name = "Distutils", > version = distutils.__version__, > ...) Uhm... isn't that the obvious solution ;-) Note that I've just moved a thread about package versioning from xml-sig to python-dev. I think we need a PEP on this. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gward@python.net Fri Sep 22 10:47:01 2000 From: gward@python.net (Greg Ward) Date: Fri Sep 22 09:47:01 2000 Subject: [Distutils] Extension Modules In-Reply-To: <39C9315A.36F0574@FourThought.com>; from Mike.Olson@fourthought.com on Wed, Sep 20, 2000 at 03:51:22PM -0600 References: <39C9315A.36F0574@FourThought.com> Message-ID: <20000922092446.A605@beelzebub> On 20 September 2000, Mike Olson said: > We are working towards converting 4Suite over to use distutils. We are > almost there, but have run into a problem. We needed to implement our > own parser to generate some files using a script we call BisonGen. Mike -- first off, I'm not clear which distribution you're talking about here: BisonGen or 4Suite? (Or something else? I downloaded those two source tarballs from your ftp site last night, so those are my only points of reference right now.) > This generates both C and py files. Then we hand the C off to > build_ext to get compilied. The .so (or pyd) is installed fine. > However, the additional py files generated are not copied into the > installation directory. Is there a simple way of doing this? Where does BisonGen put the generated .c and .py files? In the source tree or the build tree? You might consider doing all preprocessing (run SWIG, BisonGen, lex, yacc, etc.) *before* you put together the source tarball. This would make source distributions bigger, but would require less funky custom code to run on the installer's machine, away from your ability to debug. Then your custom stuff could just be run by a custom "sdist" command. (Or you could do it manually, before running the stock "sdist".) This would also reduce the number of bits and pieces that people have to install merely to build 4Suite. Note that there's no rule that says you can only have one setup script per module distribution. You might have a big complex beast to handle your local build needs -- or you might use a Makefile for that, since the Distutils' dependency analysis of .c files is very primitive (no header file analysis, in particular). For example, I have one "official" setup script that I use to create real releases, and another one for creating code snapshots. (They also have different manifests.) Only the "official" setup script gets distributed in official releases. > Looking through the > code, build_ext only uses Extension.name to generate a list of outputs. If you're trying to figure out who decides what gets installed, you should be looking in the install_* commands -- specifically, install_lib for Python modules and extensions. I think the code you're looking for is: def run (self): [...] # Install everything: simply dump the entire contents of the build # directory to the installation directory (that's the beauty of # having a build directory!) if os.path.isdir(self.build_dir): outfiles = self.copy_tree (self.build_dir, self.install_dir) IOW, the "install_lib" command should be driven entirely by what it finds in the "build" tree. (This is a general principle which all the "install_*" commands should follow.) Anyways, I'm having a hard time understanding what's going on. Once I have SWIG installed and can actually run the 4Suite build, hopefully I'll get clued in. Greg From gward@python.net Fri Sep 22 10:47:07 2000 From: gward@python.net (Greg Ward) Date: Fri Sep 22 09:47:07 2000 Subject: [Distutils] Patch to have distutils generate PYC *and* PYO files In-Reply-To: <39C89E84.64ECE3A5@lemburg.com>; from mal@lemburg.com on Wed, Sep 20, 2000 at 01:24:52PM +0200 References: <39C89E84.64ECE3A5@lemburg.com> Message-ID: <20000922094418.B605@beelzebub> On 20 September 2000, M.-A. Lemburg said: > Attached you find a patch against the current CVS tree which allows > distutils to include PYC as well as PYO files. Looks like it should work, but I really don't like the idea of spawning an interpreter for *every* *single* *file* we want to compile. This sounds very expensive. Doesn't look like it would be too hard to rework your 'compile()' function to work on a bunch of files at once, and the "install_lib" command already sorta-kinda works that way. The trouble is generating a single Python command that compiles an arbitrary number of scripts -- it's not impossible, just hairy. Maybe it's time to reconsider when we do byte-compilation. The reasons for doing it at install time were: * so the .py filename encoded in the .pyc file is correct * to make built distributions smaller But the former is a red herring, since it turns out that we do pseudo-installations to temporary directories to create RPM and wininst built distributions, and will probably do the same for other smart bdist formats. And the latter seems to be a lower priority than making what happens at install time as simple as possible. Most people (myself included) seem to be more concerned that an RPM come with "batteries included" (ie. .pyc and .pyo) files than that it be as small as possible. Byte-compiling at build time would also (I think) make it easier to distribute *only* bytecode, for those poor unenlightened souls who somehow think that keeping their source code to themselves will make the world a better place. (Obviously I disagree with those people, but I don't want to bar them from using the Distutils.) If we byte-compile at build time, then using the standard compileall module/script is a no-brainer: the build directory is purely the domain of this particular module distribution, so if we blindly walk over it compiling everything, that's just fine. (Doing that in the real install directory would be quite rude, which is why the current byte-compilation code doesn't use compileall.py.) > I haven't dug deep into the option system yet, but it would be > nice if there also were an option which lets the packager > set the optimization level she want to use for the PYO files. I think the best way to do that would be to support something like build --optimize # .pyc and .pyo (level 1) build --optimize=2 # .pyc and .pyo (level 2) unfortunately, getopt (and therefore fancy_getopt) doesn't support default option values. ;-( So this would have to be something else, alas. I think this is a frill compared to figuring out when to compile and how to get both .pyc and .pyo in there, though. Greg From gward@python.net Fri Sep 22 10:47:13 2000 From: gward@python.net (Greg Ward) Date: Fri Sep 22 09:47:13 2000 Subject: [Distutils] BisonGen, 4Suite, and Distutils Message-ID: <20000922091326.A607@beelzebub> Uche, Mike -- I'm digging into BisonGen and 4Suite now. I managed to install BisonGen (from the sdist) without problems, but there's a dependency you forgot to document: $ /usr/local/bin/python Python 1.5.2 (#5, Sep 8 2000, 22:21:43) [GCC egcs-2.91.66 19990314/Linux (egcs- on linux2 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>> import BisonGen.BisonGen Traceback (innermost last): File "", line 1, in ? File "/usr/local/python/lib/python1.5/site-packages/BisonGen/BisonGen.py", line 80, in ? import BGenReader File "/usr/local/python/lib/python1.5/site-packages/BisonGen/BGenReader.py", line 24, in ? from xml.sax import saxlib, saxexts ImportError: No module named xml.sax >>> Luckily, I also have PyXML's source dist. sitting around, so I can install it. Still, you should probably mention this in your README. Another thing: setting "verbose = 0" in the setup script is the moral equivalent of setting ".SILENT" in a Makefile -- and IMHO it's Considered Harmful. Building software *should* be noisy, because so many things can go wrong. You're free to run the setup script with -q, but if people are used to seeing noisy builds, then one particular build being quiet is unsettling. (It unsettled me, that's for sure!) Yet another: requiring SWIG to be installed in order to build your software is the moral equivalent of requiring bison and flex on a non-GNU-powered Unix box -- ie. it's a bother, it's a pain, and it's completely unnecessary since the software that I now have to go find, download, build, and install is used *only* to build this particular package! (I'm speaking here from the narrow, just-want-to-get-the- silly-thing-working perspective of someone who needs to install 4Suite in order to get work done. I'm a little embarassed to admit that I don't even have SWIG installed on my home machine, but it does let me discover lapses in other people's source dists. ;-) Now, if SWIG is turning 20k .i files into 400k .c files, that's a good reason for not distributing the .c files. And I'm starting to get a grasp of the enormity of 4Suite; this is a boatload-and-a-half of a code, so I can see why you might want to cut down on the amount of stuff distributed. Anyways, looks like I won't be able to debug your bdist_rpm problem this morning; have to go get SWIG, etc. etc. Grumble. BTW, congratulations on finding -- and making use of -- every known "backdoor" entry into the Distutils. At least it looks like the 4Suite setup.py uses every sneaky trick that I can remember slipping in to the code at some point. Very gratifying to find that all those tricks are finally useful to someone! Also, it looks like 4Suite's setup.py is so complex it will be a good shaker-out of Distutils bugs. Thanks (I think)! Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From mal@lemburg.com Fri Sep 22 11:29:08 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri Sep 22 10:29:08 2000 Subject: [Distutils] Patch to have distutils generate PYC *and* PYO files References: <39C89E84.64ECE3A5@lemburg.com> <20000922094418.B605@beelzebub> Message-ID: <39CB6C75.1DE8445D@lemburg.com> Greg Ward wrote: > > On 20 September 2000, M.-A. Lemburg said: > > Attached you find a patch against the current CVS tree which allows > > distutils to include PYC as well as PYO files. > > Looks like it should work, but I really don't like the idea of spawning > an interpreter for *every* *single* *file* we want to compile. This > sounds very expensive. It's not as expensive as it sounds... on my machine I can't even notice a difference. > Doesn't look like it would be too hard to rework your 'compile()' > function to work on a bunch of files at once, and the "install_lib" > command already sorta-kinda works that way. The trouble is generating a > single Python command that compiles an arbitrary number of scripts -- > it's not impossible, just hairy. Right. It should be possible by providing a helper in distutils which is then used instead of py_compile. > Maybe it's time to reconsider when we do byte-compilation. The reasons > for doing it at install time were: > * so the .py filename encoded in the .pyc file is correct > * to make built distributions smaller > > But the former is a red herring, since it turns out that we do > pseudo-installations to temporary directories to create RPM and wininst > built distributions, and will probably do the same for other smart bdist > formats. > > And the latter seems to be a lower priority than making what happens at > install time as simple as possible. Most people (myself included) seem > to be more concerned that an RPM come with "batteries included" > (ie. .pyc and .pyo) files than that it be as small as possible. Right. I don't care about the RPM size at all: bandwidth is there, harddisk's are cheap. Size is not as much an argument anymore as it was some years ago. > Byte-compiling at build time would also (I think) make it easier to > distribute *only* bytecode, for those poor unenlightened souls who > somehow think that keeping their source code to themselves will make the > world a better place. (Obviously I disagree with those people, but I > don't want to bar them from using the Distutils.) Ah, you are forgetting about the few of us who have to make a living by writing commercial closed source software, e.g. my apps are only shipped with PYO files with doc-strings stripped. Most of the underlying helpers are Open Source though, ie. the mx Extensions were written for just this purpose -- not only to get a warm fuzzy feeling from community feedback ;-) > If we byte-compile at build time, then using the standard compileall > module/script is a no-brainer: the build directory is purely the domain > of this particular module distribution, so if we blindly walk over it > compiling everything, that's just fine. (Doing that in the real install > directory would be quite rude, which is why the current byte-compilation > code doesn't use compileall.py.) Hmm, the existing logic works just fine -- why change it ? > > I haven't dug deep into the option system yet, but it would be > > nice if there also were an option which lets the packager > > set the optimization level she want to use for the PYO files. > > I think the best way to do that would be to support something like > > build --optimize # .pyc and .pyo (level 1) > build --optimize=2 # .pyc and .pyo (level 2) > > unfortunately, getopt (and therefore fancy_getopt) doesn't support > default option values. ;-( So this would have to be something else, > alas. But it does support -O and -OO -- which is what Python itself uses. Wouldn't that be an option ? If not, I'd also be satisfied with --optimize=0|1|2 without default value. > I think this is a frill compared to figuring out when to compile and how > to get both .pyc and .pyo in there, though. Not necessarily... some people might not want to get doc-strings into production code. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From thomas.heller@ion-tof.com Fri Sep 22 11:31:01 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri Sep 22 10:31:01 2000 Subject: [Distutils] Patch to have distutils generate PYC *and* PYO files References: <39C89E84.64ECE3A5@lemburg.com> <20000922094418.B605@beelzebub> Message-ID: <021a01c024a1$9cca6ba0$4500a8c0@thomasnb> > Maybe it's time to reconsider when we do byte-compilation. The reasons > for doing it at install time were: > * so the .py filename encoded in the .pyc file is correct > * to make built distributions smaller > > But the former is a red herring, since it turns out that we do > pseudo-installations to temporary directories to create RPM and wininst > built distributions, and will probably do the same for other smart bdist > formats. Not exactly true: wininst switches off compilation in the install_lib command which runs at build (pseudo installation) time: install_lib.compile = 0 install_lib.optimize = 0 and compiles at install time both to pyc and pyo, which is correct IMO, because we now have the right filenames in the compiled files. Of course, since the windows installer embeds python, I have a backdoor into the interpreter :-) Thomas From bhoel@starship.python.net Fri Sep 22 13:14:01 2000 From: bhoel@starship.python.net (Berthold =?iso-8859-1?q?H=F6llmann?=) Date: Fri Sep 22 12:14:01 2000 Subject: [Distutils] test command for setup.py In-Reply-To: "Greg Ward"'s message of "Thu, 21 Sep 2000 21:56:23 -0400" References: <20000919212537.B648@beelzebub> <20000921215623.A991@beelzebub> Message-ID: >>>>> "Greg" =3D=3D Greg Ward writes: Greg> On 21 September 2000, Berthold H=F6llmann said: >> I think it is easier for the developer to have code and tests >> together. Greg> Not if the tests are really exhaustive! When I write good Greg> tests (or at least what I think are good tests), the test Greg> script is roughly the size of the module being tested. I Greg> don't want my "if __name__ =3D=3D '__main__'" block to be as big Greg> as the rest of the module. I think the "test" command should Greg> support "in-situ" testing, but it's *not* enough for serious Greg> tests. >> One can leave out the tests from any distribution by using a >> Manifest file. Greg> Huh? That loses half the point of writing tests, which is to Greg> assure people building and installing your modules that they Greg> probably work as advertised on their platform. My Idea here was, to distinguish between test to enshure correct compilation on the "customers" computer and tests to help the developer my the Manifest file, but to keep all tests with your code in one (CVS?)repository. >> But for the developer it is helpful to have a set of tests that >> one can run after changes have been made to the sources to >> enshure that "little" changes have not unexpected side effects. Greg> Yes! That's the other half of the reason for writing tests. Greg> (You can define "half" here as 40-60, 20-80, whatever.) >> ... Greg> Oooh, yes, in a previous life I struggled (briefly) with Greg> automated testing of numerical algorithms. Not fun. But Greg> asking end-users to examine the test output and make sure Greg> that the difference between 1.0234324 and 1.0234821 is less Greg> than the machine epsilon is surely even less fun. It begins with the difference between 1.00000E+00 and 1.0000E+000 (where is it?). >> Also the underlaying Fortran code makes different decisions on >> different platforms. That makes automatic testing nearly >> impossible, but I want at least a constistant way to generate >> the needed output, and I want to test everything before I >> install it. Greg> Here's my extreme position: Greg> If a test isn't fully automated -- i.e., if it takes more Greg> than epsilon human intelligence to determine if the test Greg> passed or failed -- then it's not a proper test, it's just a Greg> demo. Accepted for "functionality" tests, to be executed by the "customer", but I don't want to distinguish these from heavier tests, to be executed by the developer. Greg> ... >> All this can be done with my posted, extremly lightweight >> test.py extension for the _command_ subdirectory. The test >> scripts can written to implement/use any testframework you >> like. Maybe a Greg> OK, lightweight is good. (Something I occasionally need to Greg> be reminded of...) I'll definitely take a look at it. Thanks Greetings Berthold --=20 bhoel@starship.python.net / http://starship.python.net/crew/bhoel/ It is unlawful to use this email address for unsolicited ads (USC Title 47 Sec.227). I will assess a US$500 charge for reviewing and deleting each unsolicited ad. From dlists@inbed.org Fri Sep 22 15:00:01 2000 From: dlists@inbed.org (Dragons Spam Account) Date: Fri Sep 22 14:00:01 2000 Subject: [Distutils] How to add include dirs for compiler Message-ID: <200009221755.MAA62239@inbed.org> How do I inform distutils of addition include directories to find headers in when compiling c source in a python module. This is outside the package. I'm a user not the author. My scenerio: I installed one module that provides a header file. Rather than inflict my additions on all the users of the system I'm installing locally to my own home directory with --home=~ This pops the libraries and such into ~/lib/python but drops the header file into ~/include python. For the sake of python finding ~/lib/python, PYTHONPATH suffices. Now I attempt to install a second seperate module that relies on the first module. Regardless of whether I install this second module to the system or just to my home directory it needs to find that header and I can't seem to find the interface through which to provide distutils with additional include directories to use when compiling source. Which in this case would be ~/include/python. The whole scenerio works fine if I'm installing everything to the system python directories but currently this is not an option for these modules. Any help? From Mike.Olson@fourthought.com Fri Sep 22 17:44:01 2000 From: Mike.Olson@fourthought.com (Mike Olson) Date: Fri Sep 22 16:44:01 2000 Subject: [Distutils] Re: BisonGen, 4Suite, and Distutils References: <20000922091326.A607@beelzebub> Message-ID: <39CBC3DC.72549C8D@FourThought.com> Greg Ward wrote: > > Uche, Mike -- > > I'm digging into BisonGen and 4Suite now. I managed to install BisonGen > (from the sdist) without problems, but there's a dependency you forgot > to document: > > $ /usr/local/bin/python > Python 1.5.2 (#5, Sep 8 2000, 22:21:43) [GCC egcs-2.91.66 19990314/Linux (egcs- on linux2 > Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam > >>> import BisonGen.BisonGen > Traceback (innermost last): > File "", line 1, in ? > File "/usr/local/python/lib/python1.5/site-packages/BisonGen/BisonGen.py", line 80, in ? > import BGenReader > File "/usr/local/python/lib/python1.5/site-packages/BisonGen/BGenReader.py", line 24, in ? > from xml.sax import saxlib, saxexts > ImportError: No module named xml.sax > >>> > > Luckily, I also have PyXML's source dist. sitting around, so I can > install it. Still, you should probably mention this in your README. I'll add this to the reqs. > > Another thing: setting "verbose = 0" in the setup script is the moral > equivalent of setting ".SILENT" in a Makefile -- and IMHO it's > Considered Harmful. Building software *should* be noisy, because so > many things can go wrong. You're free to run the setup script with -q, > but if people are used to seeing noisy builds, then one particular build > being quiet is unsettling. (It unsettled me, that's for sure!) I wasn't sure which way that should be. I'll set it to verbose. > > Yet another: requiring SWIG to be installed in order to build your > software is the moral equivalent of requiring bison and flex on a > non-GNU-powered Unix box -- ie. it's a bother, it's a pain, and it's > completely unnecessary since the software that I now have to go find, > download, build, and install is used *only* to build this particular > package! (I'm speaking here from the narrow, just-want-to-get-the- > silly-thing-working perspective of someone who needs to install 4Suite > in order to get work done. I'm a little embarassed to admit that I > don't even have SWIG installed on my home machine, but it does let me > discover lapses in other people's source dists. ;-) We finally bit the bullet. We don't have to bgen, swig, flex, or bison that often so we checked in the final set of .c files. This should make life easier... > > Now, if SWIG is turning 20k .i files into 400k .c files, that's a good > reason for not distributing the .c files. And I'm starting to get a > grasp of the enormity of 4Suite; this is a boatload-and-a-half of a > code, so I can see why you might want to cut down on the amount of stuff > distributed. I don't think its that big, besides, c files should zip up pretty good... > > Anyways, looks like I won't be able to debug your bdist_rpm problem this > morning; have to go get SWIG, etc. etc. Grumble. > > BTW, congratulations on finding -- and making use of -- every known > "backdoor" entry into the Distutils. At least it looks like the 4Suite > setup.py uses every sneaky trick that I can remember slipping in to the > code at some point. Very gratifying to find that all those tricks are > finally useful to someone! Also, it looks like 4Suite's setup.py is so > complex it will be a good shaker-out of Distutils bugs. Thanks (I > think)! Your welcome:) Unfortunalty I simplified it alot this morning when I removed the bgen stuff. Mike > > Greg > -- > Greg Ward gward@python.net > http://starship.python.net/~gward/ -- Mike Olson Principal Consultant mike.olson@fourthought.com (303)583-9900 x 102 Fourthought, Inc. http://Fourthought.com Software-engineering, knowledge-management, XML, CORBA, Linux, Python From gward@python.net Fri Sep 22 22:30:01 2000 From: gward@python.net (Greg Ward) Date: Fri Sep 22 21:30:01 2000 Subject: [Distutils] Appears bdist_rpm completely broken In-Reply-To: <39CAD7A6.2706EC3D@fourthought.com>; from uche.ogbuji@fourthought.com on Thu, Sep 21, 2000 at 09:53:10PM -0600 References: <39C8FC9F.3EE734AA@fourthought.com> <20000921232109.B1158@beelzebub> <39CAD7A6.2706EC3D@fourthought.com> Message-ID: <20000922212732.A1316@beelzebub> On 21 September 2000, Uche Ogbuji said: > We just found out that this is indeed not the problem. The problem was > that RPM wasn't getting the right lists of installed files, so it was > getting confused about where things were. ...which probably has something to do with your highly customized build process... > You ain't seen nothing yet! We fixed a whole bunch of our additional > packaging problems with more distutils subclassing. Check out > > ftp://ftp.fourthought.com/pub/4Suite/4Suite-0.9.0.2.tar.gz ...OK, I'll download this one once I've got SWIG installed. *Then* I'll look into your sordid mess! > We'd love to hear any feedback about our distutils efforts. We hardly > thought we were doing anything extraordinary, but distutils does seem to > have several limitations as is that we had to sub-class to get around. > Maybe we were trying too hard in some cases. No build system can cover every possibility out of the box, that's why they have to be extensible. Distutils covers pure Python modules, vanilla Python extensions, SWIGged Python extensions, and C libraries that are statically linked into Python extensions pretty nicely (if I do say so myself). Anything outside of that box and you've got to write custom code. (The theory was that Python programmers would prefer to write such custom code in Python, rather than cobbling together a nightmarish mix of m4, shell, and make that a traditional Autoconf/Automake system implies -- and of course, doing it all in Python makes it portable to other platforms. In practice, it turns out that extending a large object framework requires a bit more in-depth knowledge than cranking out a few more Makefile targets. Well, if nothing else it's an interesting experiment, and I think it's fairly successful.) Anyways, I already got some feedback from Mike about refactoring the install_lib command for easier extensibility -- and in fact I'm about to checkin a change that should help there. See my next post. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Fri Sep 22 22:57:01 2000 From: gward@python.net (Greg Ward) Date: Fri Sep 22 21:57:01 2000 Subject: [Distutils] Appears bdist_rpm completely broken In-Reply-To: <39CAD927.3E3E1C7A@FourThought.com>; from Mike.Olson@fourthought.com on Thu, Sep 21, 2000 at 09:59:35PM -0600 References: <39C8FC9F.3EE734AA@fourthought.com> <20000921232109.B1158@beelzebub> <39CAD7A6.2706EC3D@fourthought.com> <39CAD927.3E3E1C7A@FourThought.com> Message-ID: <20000922215534.B1316@beelzebub> On 21 September 2000, Mike Olson said: > There were a couple of spots were if we could abstract core distutils > code into a function call (that we could override) it would make our > lives that much nicer. Ex, I needed on little hook into install_lib.run > (after the build and before the copy), but I had to reproduce the entire > function call... OK, I've split install_lib.run() up. It's now a three-liner: self.build() outfiles = self.install() self.bytecompile(outfiles) and I have no problems with making people cut-n-paste three lines of code in order to insert one. ;-) I *think* this approach is preferable to sticking in "pre_build()" and "post_install()" hooks, which are empty methods by default. Try this out with the CVS version, but don't count on the interface staying the same -- ie. that 'bytecompile()' method might disappear, if byte-compilation moves to the build phase. Whatever goes out in 0.9.3 should be final. > another really useful thing for us on copy_file would be if the function > told you if the file was copied.... Hmm, every time this comes up I can never remember why I didn't change it the last time around. Or maybe I did change it last time, and now you want the old behaviour back. Argh! Good thing I have CVS here to cover my you-know-what... ah! here we go: revision 1.4 date: 2000/06/23 01:42:40; author: gward; state: Exp; lines: +6 -7 Bastian Kleineidam: 'copy_file()' now returns the output filename, rather than a boolean indicating whether it did the copy. This bloody return value has bounced back and forth between "was it copied?" and "output filename", and I'm *not* changing it again! It makes my head hurt, which is not worth it over such a silly, trivial little thing. There is code out there that relies on getting the output filename back, regardless of whether the file was copied or not. Greg From gward@python.net Fri Sep 22 23:04:01 2000 From: gward@python.net (Greg Ward) Date: Fri Sep 22 22:04:01 2000 Subject: [Distutils] How to add include dirs for compiler In-Reply-To: <200009221755.MAA62239@inbed.org>; from dlists@inbed.org on Fri, Sep 22, 2000 at 12:55:42PM -0500 References: <200009221755.MAA62239@inbed.org> Message-ID: <20000922220230.C1316@beelzebub> On 22 September 2000, Dragons Spam Account said: > How do I inform distutils of addition include directories to find headers in > when compiling c source in a python module. Use the --include (-I) option to "build_ext". > I installed one module that provides a header file. > > Rather than inflict my additions on all the users of the system > I'm installing locally to my own home directory with --home=~ OK, sounds straightforward so far. > This pops the libraries and such into ~/lib/python but drops the header > file into ~/include python. Yup, all working as planned. > Now I attempt to install a second seperate module that relies > on the first module. Regardless of whether I install this second > module to the system or just to my home directory it needs to > find that header and I can't seem to find the interface through > which to provide distutils with additional include directories > to use when compiling source. Which in this case would be ~/include/python. D'ohh! Well, Distutils really can't figure this out automatically: it can only presume that what it needs is installed to the standard Python directories, and if you put stuff elsewhere, you'll have to tell it. Luckily, this isn't hard. Instead of running just python setup.py build you would tack on some "build_ext" options to control how extensions are built: python setup.py build build_ext -n -I/your/home/dir/include/python Note that you can't use ~ here; only the "install" command expands ~ and $variables. Greg From mwa@gate.net Fri Sep 22 23:50:01 2000 From: mwa@gate.net (Mark W. Alexander) Date: Fri Sep 22 22:50:01 2000 Subject: [Distutils] Patch to have distutils generate PYC *and* PYO files In-Reply-To: <39CB6C75.1DE8445D@lemburg.com> Message-ID: On Fri, 22 Sep 2000, M.-A. Lemburg wrote: > > Greg Ward wrote: > > > > Looks like it should work, but I really don't like the idea of spawning > > an interpreter for *every* *single* *file* we want to compile. This > > sounds very expensive. > > It's not as expensive as it sounds... on my machine I can't even > notice a difference. I suggest that this is probably a result of the interpreter remaining in the buffer cache. On some of the newer drives with large on-board memory caches, you probably wouldn't even see the disk light blink. All machines not being equal, though, it would be very expensive on (even slightly) older hardware. If there's another way, it's probably better. $.02 Mark From gward@python.net Sun Sep 24 22:50:02 2000 From: gward@python.net (Greg Ward) Date: Sun Sep 24 21:50:02 2000 Subject: [Distutils] Patch to have distutils generate PYC *and* PYO files In-Reply-To: <39CB6C75.1DE8445D@lemburg.com>; from mal@lemburg.com on Fri, Sep 22, 2000 at 04:28:05PM +0200 References: <39C89E84.64ECE3A5@lemburg.com> <20000922094418.B605@beelzebub> <39CB6C75.1DE8445D@lemburg.com> Message-ID: <20000924214757.A6955@beelzebub> On 22 September 2000, M.-A. Lemburg said: > It's not as expensive as it sounds... on my machine I can't even > notice a difference. I tried it on my machine: spawning a new interpreter to byte-compile every file in the Distutils is nearly a 100% penalty: 13 sec vs a little under 7 sec. So I think it's worth the trouble. > Right. I don't care about the RPM size at all: bandwidth is > there, harddisk's are cheap. Size is not as much an argument > anymore as it was some years ago. I mostly agree, but turning off byte-compilation should be an option. (Certainly not the default!) > > Byte-compiling at build time would also (I think) make it easier to > > distribute *only* bytecode, for those poor unenlightened souls who > > somehow think that keeping their source code to themselves will make the > > world a better place. (Obviously I disagree with those people, but I > > don't want to bar them from using the Distutils.) > > Ah, you are forgetting about the few of us who have to make > a living by writing commercial closed source software, e.g. > my apps are only shipped with PYO files with doc-strings stripped. No, I am proposing to change the situation to accomodate people who only distribute byte-code. Right now, all built distributions always include source, because they must pseudo-install the source in order to compile it. If we do compilation at build-time, that should make it easier to distribute only byte-code. If you want to distribute only byte-code, you'd have to sneak some code into the appropriate bdist commands to delete source files after pseudo-installation but before constructing the archive. I still think software wants to be free, but I'm not going to shove that philosophy down anyone's throat. ;-) > Most of the underlying helpers are Open Source though, ie. > the mx Extensions were written for just this purpose -- not > only to get a warm fuzzy feeling from community feedback ;-) > > If we byte-compile at build time, then using the standard compileall > > module/script is a no-brainer: the build directory is purely the domain > > of this particular module distribution, so if we blindly walk over it > > compiling everything, that's just fine. (Doing that in the real install > > directory would be quite rude, which is why the current byte-compilation > > code doesn't use compileall.py.) > > Hmm, the existing logic works just fine -- why change it ? I can see a couple reasons to change compilation to build time: * it just seems right: installation should be limited to copying files around, possibly setting modes, and nothing more. More complicated stuff, including compilation, should be done at *build* time... that's why it's called "build" time! * should make it easier to support closed source distributions And one good reason not to change it (apart from inertia): * makes it trickier to put the right source filename into the .pyc file (not that that's being done currently, as you can see from a traceback in code installed by a Distutils-generated RPM file; the .pyc files refer to /var/tmp/.../usr/lib/python1.5/... rather than /usr/lib/python1.5/...) Am I missing anything? Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Sun Sep 24 22:59:03 2000 From: gward@python.net (Greg Ward) Date: Sun Sep 24 21:59:03 2000 Subject: [Distutils] Changes to config file handling Message-ID: <20000924215602.A6998@beelzebub> Hi all -- I think I've fixed all the problems Marc-André pointed out with config files last week, specifically: * options in config files can now be spelled either "foo-bar" (like on the command line, minus the leading "--") or "foo_bar" (like in the code) * negative alias options are now respected, eg. "quiet=1" is the same as "verbose=0" Plus I fixed a bug nobody had reported: * boolean options now work: previously, "force=0" resulted in an attribute 'force' being set to the string "0", which of course is true. As an extra bonus, you can now use a variety of sensible representations of true and false: yes, y, t, true, 1, on for true, and no, n, f, false, 0, off for false. And added a feature that was supposed to be in there from the beginning, but I just plain forgot about: * a "[global]" section where you can set global options; currently, only 'verbose' and 'dry-run' are supported here (and of course 'quiet', since it's the opposite of 'verbose') As soon as I've checked everything in, I'll put out a new code snapshot. Please give it a try and let me know if config files now work as you'd expect them to work. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From Mike.Olson@fourthought.com Mon Sep 25 13:22:02 2000 From: Mike.Olson@fourthought.com (Mike Olson) Date: Mon Sep 25 12:22:02 2000 Subject: [Distutils] Appears bdist_rpm completely broken References: <39C8FC9F.3EE734AA@fourthought.com> <20000921232109.B1158@beelzebub> <39CAD7A6.2706EC3D@fourthought.com> <39CAD927.3E3E1C7A@FourThought.com> <20000922215534.B1316@beelzebub> Message-ID: <39CF7B93.2E6ACD70@FourThought.com> Greg Ward wrote: > > On 21 September 2000, Mike Olson said: > > There were a couple of spots were if we could abstract core distutils > > code into a function call (that we could override) it would make our > > lives that much nicer. Ex, I needed on little hook into install_lib.run > > (after the build and before the copy), but I had to reproduce the entire > > function call... > > OK, I've split install_lib.run() up. It's now a three-liner: > > self.build() > outfiles = self.install() > self.bytecompile(outfiles) > > and I have no problems with making people cut-n-paste three lines of > code in order to insert one. ;-) I *think* this approach is preferable > to sticking in "pre_build()" and "post_install()" hooks, which are empty > methods by default. > > Try this out with the CVS version, but don't count on the interface > staying the same -- ie. that 'bytecompile()' method might disappear, if > byte-compilation moves to the build phase. Whatever goes out in 0.9.3 > should be final. This looks like it will help, I'll try out the latest CVS version today yet. > > > another really useful thing for us on copy_file would be if the function > > told you if the file was copied.... > > Hmm, every time this comes up I can never remember why I didn't change > it the last time around. Or maybe I did change it last time, and now > you want the old behaviour back. Argh! Good thing I have CVS here to > cover my you-know-what... ah! here we go: > > revision 1.4 > date: 2000/06/23 01:42:40; author: gward; state: Exp; lines: +6 -7 > Bastian Kleineidam: 'copy_file()' now returns the output filename, rather > than a boolean indicating whether it did the copy. > > This bloody return value has bounced back and forth between "was it > copied?" and "output filename", and I'm *not* changing it again! It > makes my head hurt, which is not worth it over such a silly, trivial > little thing. There is code out there that relies on getting the output > filename back, regardless of whether the file was copied or not. I've noticed the code that relies on copy_file returning the dest file. I also understand the need to have that code in place. However, (even though this could be a futile attmept), what about returning a tuple, best of both worlds, or even better, a new interface that returns the tuple. Rename copy_file to the new interface name and modify it to return the tuple of dest name and a boolean copied flag. Then recreate the function copy_file to work as it does..... def new_copy_file(src,dest,....): return (dest,copied) def copy_file(src,dest,....): return new_copy_file(src,dest,...)[0] We'd also want to do the same for copy_tree where is can return a list of tuples.... Just a thought... Mike > > Greg -- Mike Olson Principal Consultant mike.olson@fourthought.com (303)583-9900 x 102 Fourthought, Inc. http://Fourthought.com Software-engineering, knowledge-management, XML, CORBA, Linux, Python From hoel@germanlloyd.org Mon Sep 25 13:42:01 2000 From: hoel@germanlloyd.org (Berthold =?iso-8859-1?q?H=F6llmann?=) Date: Mon Sep 25 12:42:01 2000 Subject: [Distutils] test command for setup.py In-Reply-To: "Thomas Heller"'s message of "Thu, 21 Sep 2000 17:11:28 +0200" References: <009901c023de$3df93cf0$4500a8c0@thomasnb> Message-ID: "Thomas Heller" writes: > I like the simplicity of your test command, > however I would like (and have partly implmented) > the following changes: > - don't use so many options, the build directories could > be retrieved from the 'build' command I would, if I could figure out how. )-: > - wrap 'try' blocks around running the tests I think, if a test failed, the script should abort. If you want it to continue, the try blocks should go into the test files themselves. > - output a summary of failed and passed tests > I will probably post my script later. This again should go into the test implementations, or into a test framework. I think these are only good for the "enduser" testing. But I would like to handle some "developer" testing also by the distutils test command. Berthold -- email: hoel@GermanLloyd.org ) tel. : +49 (40) 3 61 49 - 73 74 ( C[_] These opinions might be mine, but never those of my employer. From gward@python.net Mon Sep 25 23:22:02 2000 From: gward@python.net (Greg Ward) Date: Mon Sep 25 22:22:02 2000 Subject: [Distutils] My (slightly goofy) coding standard Message-ID: <20000925221954.A997@beelzebub> Hi all -- I have just completed one pass over the Distutils source to enforce one minor coding convention: whitespace next to the open parentheses when calling a function (or other callable). Anyone hacking on the Distutils needs to know about this convention, which is slightly odd but quite defensible (IMHO): * when *defining* a callable, use a space: def foo (): ... def foo (arg1, arg2, ...): ... class SubThing (Thing): ... * corollary: for long functions that need a "# end"-style line, it looks like this: def foo (): # > 40 lines of code (say) # foo () because the "# foo ()" line is part of the definition * when *using* a callable, no space: foo() thing.foo() foo(2, "hello") this.that.get_something().foo.name_of_long_method( arg1, arg2, arg3, ...) Rationale: this makes it trivial (grep-wise) to distinguish between *definition* and *use*. I *know* what Guido's coding standard says, but the problem with "def foo()" is that you can't search for use only without a negative lookbehind to exclude "def" lines... and my "grep" doesn't have negative lookbehind. Anyways, that's the Distutils convention as of tonight. Please respect it when hacking on the code! (And please fix deviance if you submit patches: I've only fixed up distutils/*.py, excluding *ccompiler.py because of a pending patch. distutils/command/*.py is still to go; I can only do so much brain-dead search 'n replace work at once.) Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From jeremy@beopen.com Mon Sep 25 23:40:01 2000 From: jeremy@beopen.com (Jeremy Hylton) Date: Mon Sep 25 22:40:01 2000 Subject: [Distutils] My (slightly goofy) coding standard In-Reply-To: <20000925221954.A997@beelzebub> References: <20000925221954.A997@beelzebub> Message-ID: <14800.3529.212812.820237@bitdiddle.concentric.net> >>>>> "GW" == Greg Ward writes: GW> Hi all -- I have just completed one pass over the Distutils GW> source to enforce one minor coding convention: whitespace next GW> to the open parentheses when calling a function (or other GW> callable). Anyone hacking on the Distutils needs to know about GW> this convention, which is slightly odd [...] It is really unfortunate that you have chosen to go this route. We are trying to make all the Python code in the standard library follow Guido's coding style. http://www.python.org/doc/essays/styleguide.html We will slow fix code in the library to follow these conventions, which will make distutils the only part of the core that doesn't follow them. Jeremy From thomas.heller@ion-tof.com Tue Sep 26 03:54:01 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue Sep 26 02:54:01 2000 Subject: [Distutils] Changes to config file handling References: <20000924215602.A6998@beelzebub> Message-ID: <009501c02786$7e1b1960$4500a8c0@thomasnb> > Plus I fixed a bug nobody had reported: > > * boolean options now work: previously, "force=0" resulted in an > attribute 'force' being set to the string "0", which of course > is true. As an extra bonus, you can now use a variety of sensible > representations of true and false: yes, y, t, true, 1, on for true, > and no, n, f, false, 0, off for false. > Does not work with command line options. Example: C:\sf\lpfw>setup.py build_ext -f running build_ext Traceback (innermost last): File "C:\sf\lpfw\setup.py", line 41, in ? ext_modules = [_ctypes, ExtensionClass, xxtype], File "c:\work\orion\src\Fpanel\Python\distutils\core.py", line 138, in setup dist.run_commands() File "c:\work\orion\src\Fpanel\Python\distutils\dist.py", line 828, in run_commands self.run_command(cmd) File "c:\work\orion\src\Fpanel\Python\distutils\dist.py", line 846, in run_command cmd_obj = self.get_command_obj(command) File "c:\work\orion\src\Fpanel\Python\distutils\dist.py", line 726, in get_command_obj self._set_command_options(cmd_obj, options) File "c:\work\orion\src\Fpanel\Python\distutils\dist.py", line 761, in _set_command_options setattr(command_obj, option, strtobool(value)) File "c:\work\orion\src\Fpanel\Python\distutils\util.py", line 284, in strtobool val = string.lower(val) TypeError: read-only character buffer, int string.lower(val) is called with the integer 1 (not the string "1"). Thomas From thomas.heller@ion-tof.com Tue Sep 26 15:55:02 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue Sep 26 14:55:02 2000 Subject: [Distutils] test command for setup.py References: <009901c023de$3df93cf0$4500a8c0@thomasnb> Message-ID: <002301c027eb$2319dff0$4500a8c0@thomasnb> > "Thomas Heller" writes: > > > I like the simplicity of your test command, > > however I would like (and have partly implmented) > > the following changes: > > - don't use so many options, the build directories could > > be retrieved from the 'build' command > > I would, if I could figure out how. )-: This is what I currently have in one of my setup scripts: class test (Command): # Original version of this class posted # by Berthold Höllmann to distutils-sig@python.org description = "test the distribution prior to install" user_options = [ ('test-dir=', None, "directory that contains the test definitions"), ('test-prefix=', None, "prefix to the testcase filename"), ('test-suffix=', None, "a list of suffixes used to generate names the of the testcases") ] def initialize_options (self): self.build_base = 'build' # these are decided only after 'build_base' has its final value # (unless overridden by the user or client) self.test_dir = 'test' self.test_prefix = 'test_' self.test_suffixes = None # initialize_options () def finalize_options (self): import os if self.test_suffixes is None: self.test_suffixes = [] pref_len = len(self.test_prefix) for file in os.listdir(self.test_dir): if (file[-3:] == ".py" and file[:pref_len]==self.test_prefix): self.test_suffixes.append(file[pref_len:-3]) build = self.get_finalized_command('build') self.build_purelib = build.build_purelib self.build_platlib = build.build_platlib # finalize_options () def run (self): import sys # Invoke the 'build' command to "build" pure Python modules # (ie. copy 'em into the build tree) self.run_command ('build') # remember old sys.path to restore it afterwards old_path = sys.path[:] # extend sys.path sys.path.insert(0, self.build_purelib) sys.path.insert(0, self.build_platlib) sys.path.insert(0, self.test_dir) # build include path for test for case in self.test_suffixes: TEST = __import__(self.test_prefix+case, globals(), locals(), ['']) TEST.test() # restore sys.path sys.path = old_path[:] # run () # class test Thomas From gward@python.net Tue Sep 26 19:16:44 2000 From: gward@python.net (Greg Ward) Date: Tue Sep 26 18:16:44 2000 Subject: [Distutils] ANNOUNCE: Distutils 0.9.3 Message-ID: <20000926181415.A935@beelzebub> Python Distribution Utilities release 0.9.3 September 26, 2000 The Python Distribution Utilities, or Distutils for short, are a collection of modules that aid in the development, distribution, and installation of Python modules and (ultimately) applications. The Distutils are a standard part of Python 2.0; if you are running 2.0 (or one of the 1.6 alpha releases that preceded it), you don't need to install the Distutils separately. This release is primarily so that you can add the Distutils to a Python 1.5.2 installation -- you will then be able to install modules that require the Distutils, or use the Distutils to distribute your own modules. Version 0.9.3 is identical to the Distutils code included with Python 2.0b2. More information is available at the Distutils web page: http://www.python.org/sigs/distutils-sig/ and in the README.txt included in the Distutils source distribution. You can download the Distutils from http://www.python.org/sigs/distutils-sig/download.html Trivial patches can be sent to me (Greg Ward) at gward@python.net. Larger patches should be discussed on the Distutils mailing list: distutils-sig@python.org. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Tue Sep 26 21:30:01 2000 From: gward@python.net (Greg Ward) Date: Tue Sep 26 20:30:01 2000 Subject: [Distutils] Changes to config file handling In-Reply-To: <009501c02786$7e1b1960$4500a8c0@thomasnb>; from thomas.heller@ion-tof.com on Tue, Sep 26, 2000 at 08:53:30AM +0200 References: <20000924215602.A6998@beelzebub> <009501c02786$7e1b1960$4500a8c0@thomasnb> Message-ID: <20000926202755.A1423@beelzebub> On 26 September 2000, Thomas Heller said: > Does not work with command line options. > Example: > C:\sf\lpfw>setup.py build_ext -f > running build_ext [...] > val = string.lower(val) > TypeError: read-only character buffer, int ARRGGHHH!!!! Thank you Thomas; I only wish I had tested this myself. Duh. It's too late for Distutils 0.9.3 or, more importantly, Python 2.0b2 -- they both include the broken code. Damn, damn, damn! I'm uploading Distutils 0.9.4 to python.org as I type. Fixed three lines of code, and that was it. !@$#%#^@#@&!@#%@^!#$%!@%@!#% Did actually test it this time. Good thing I don't have one of those annoying mail programs that warn you when your language is potentially offensive... Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Tue Sep 26 21:55:01 2000 From: gward@python.net (Greg Ward) Date: Tue Sep 26 20:55:01 2000 Subject: [Distutils] Distutils 1.0 code freeze: Oct 1 Message-ID: <20000926205312.A1470@beelzebub> Considering the following schedule of events: Oct 4: I go out of town (away from email, off the net, etc.) Oct 10: planned release of Python 2.0 Oct 12: I'm back in town, ready to hack! (and wondering why it's so quiet around here...) the Distutils 1.0 release will go out October 1 or 2. I don't need quite as much code freeze time as the full Python release, but let's put it this way: if there are features you want added to the Distutils that I don't already know about, forget about it. Changes currently under consideration: * Rene Liebscher's rearrangement of the CCompiler classes; most of this is just reducing the amount of code, but it does add some minor features, so it's under consideration. * making byte-compilation more flexible: should be able to generate both .pyc and .pyo files, and should be able to do it at build time or install time (developer's and packager's discretion) If you know about any outstanding Distutils bugs, please tell me *now*. Put 'em in the SourceForge bug database if you're wondering why I haven't fixed them yet -- they might have gotten lost, I might not know about 'em, etc. If you're not sure, put it in SourceForge. Stuff that will definitely have to wait until after 1.0: * a "test" command (standard test framework for Python modules) * finishing the "config" command (auto-configuration) * installing package meta-data, to support "what *do* I have installed, anyways?" queries, uninstallation, upgrades, etc. Blue-sky projects: * standard documentation processing * intra-module dependencies Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Tue Sep 26 22:03:01 2000 From: gward@python.net (Greg Ward) Date: Tue Sep 26 21:03:01 2000 Subject: [Distutils] Synchronizing version information In-Reply-To: <39CB1B1C.A4394DD3@lemburg.com>; from mal@lemburg.com on Fri, Sep 22, 2000 at 10:41:00AM +0200 References: <200009200956.LAA04691@pandora.informatik.hu-berlin.de> <20000921220933.C991@beelzebub> <39CB1B1C.A4394DD3@lemburg.com> Message-ID: <20000926210109.G975@beelzebub> [MAL, responding to my "import __version__ from distutils" trick] > Uhm... isn't that the obvious solution ;-) No, but it's doable without making any changes in the Distutils (minimal) support for version numbers. > Note that I've just moved a thread about package versioning from > xml-sig to python-dev. I think we need a PEP on this. Good idea. There are any number of ways to do this, and we (the community) should pick one and settle on it. The whole point of standardizing this stuff is not that it'll be perfect, but it'll be standard. I don't recall seeing the thread develop properly on python-dev... maybe you should try again, cross-posting distutils-sig this time. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From akuchlin@mems-exchange.org Tue Sep 26 22:38:01 2000 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Tue Sep 26 21:38:01 2000 Subject: [Distutils] Distutils 1.0 code freeze: Oct 1 In-Reply-To: <20000926205312.A1470@beelzebub>; from gward@python.net on Tue, Sep 26, 2000 at 08:53:12PM -0400 References: <20000926205312.A1470@beelzebub> Message-ID: <20000926213726.A20834@newcnri.cnri.reston.va.us> On Tue, Sep 26, 2000 at 08:53:12PM -0400, Greg Ward wrote: >If you know about any outstanding Distutils bugs, please tell me *now*. How about the directory caching problem when doing multiple builds in a single Python process? (Reported by me in http://www.python.org/pipermail/distutils-sig/2000-September/001915.html) I'm willing to write a patch, if given guidance on what to write. --amk From gward@python.net Tue Sep 26 23:37:01 2000 From: gward@python.net (Greg Ward) Date: Tue Sep 26 22:37:01 2000 Subject: [Distutils] New code snapshot Message-ID: <20000926223435.A2157@beelzebub> Hi all -- I've just checked in Rene's rather hefty patch to collapse a bunch of mostly-redundant 'link_*()' methods in the various CCompiler classes into one 'link()'. This is big enough to warrant a whole new code snapshot, so I put one out: two-and-a-half releases in one day, a personal record! ;-) It works for me building a few things on my Linux box; please try it out on other platforms (especially with a variety of compilers on Windows) and let me know how it goes. Thanks. Oh yeah: http://www.python.org/sigs/distutils-sig/download.html#snapshot Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Tue Sep 26 23:41:42 2000 From: gward@python.net (Greg Ward) Date: Tue Sep 26 22:41:42 2000 Subject: [Distutils] Distutils 1.0 code freeze: Oct 1 In-Reply-To: <20000926213726.A20834@newcnri.cnri.reston.va.us>; from akuchlin@cnri.reston.va.us on Tue, Sep 26, 2000 at 09:37:26PM -0400 References: <20000926205312.A1470@beelzebub> <20000926213726.A20834@newcnri.cnri.reston.va.us> Message-ID: <20000926223845.B1423@beelzebub> On 26 September 2000, Andrew Kuchling said: > How about the directory caching problem when doing multiple builds in > a single Python process? Yep, that one slipped through the cracks all right -- thanks! > (Reported by me in > http://www.python.org/pipermail/distutils-sig/2000-September/001915.html) > I'm willing to write a patch, if given guidance on what to write. grep dir_util.py for '_path_cache'. Couldn't hurt to grep *.py, but if any other module is referring to a leading-underscore variable, somebody needs to be slapped with the proverbial wet noodle! ... lessee... yep, just 'mkpath()' and 'remove_tree()' use it. Both will probably need to be changed to use absolute paths. Shouldn't be hard, and I approve. Go for it. Greg From gward@python.net Wed Sep 27 08:45:01 2000 From: gward@python.net (Greg Ward) Date: Wed Sep 27 07:45:01 2000 Subject: [Distutils] ANNOUNCE: Distutils 0.9.3 Message-ID: Python Distribution Utilities release 0.9.3 September 26, 2000 The Python Distribution Utilities, or Distutils for short, are a collection of modules that aid in the development, distribution, and installation of Python modules and (ultimately) applications. The Distutils are a standard part of Python 2.0; if you are running 2.0 (or one of the 1.6 alpha releases that preceded it), you don't need to install the Distutils separately. This release is primarily so that you can add the Distutils to a Python 1.5.2 installation -- you will then be able to install modules that require the Distutils, or use the Distutils to distribute your own modules. Version 0.9.3 is identical to the Distutils code included with Python 2.0b2. More information is available at the Distutils web page: http://www.python.org/sigs/distutils-sig/ and in the README.txt included in the Distutils source distribution. You can download the Distutils from http://www.python.org/sigs/distutils-sig/download.html Trivial patches can be sent to me (Greg Ward) at gward@python.net. Larger patches should be discussed on the Distutils mailing list: distutils-sig@python.org. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From pete@visionart.com Wed Sep 27 15:12:01 2000 From: pete@visionart.com (Pete Shinners) Date: Wed Sep 27 14:12:01 2000 Subject: [Distutils] setup.py for a wrapper extension? Message-ID: <005d01c028ae$ac7fa180$f73f93cd@visionart.com> i've got a python extension that is simply a wrapping around another C library. (not an uncommon case :]) i cannot figure out the 'right' way to check for and link/include this dependency into my distutils setup.py script. under linux, it's pretty easy to assume the correct header and shared library will be somewhere in the standard include and library directories /usr/local/include, /usr/local/lib, etc my main concern is getting it working with windows (and keeping it crossplatform happy for linux). my best bet in windows seems to be walk around the parent directory tree trying to find directories for the dependency?? if that is the best way, it would probably be nice for the linux version to use this method or look in the standard compiler directories ??????? i'm hoping someone out there has an example doing just this sort of stuff i could look at. the documentation doesn't mention anything about this sort of stuff. (except for adding library and include info to the Extensions() call thanks for any help and/or guidance. the distutils have just amazed me. (please help, i couldn't get any responses on c.l.python) From gward@python.net Wed Sep 27 23:37:01 2000 From: gward@python.net (Greg Ward) Date: Wed Sep 27 22:37:01 2000 Subject: [Distutils] My (slightly goofy) coding standard In-Reply-To: <14800.3529.212812.820237@bitdiddle.concentric.net>; from jeremy@beopen.com on Mon, Sep 25, 2000 at 10:45:29PM -0400 References: <20000925221954.A997@beelzebub> <14800.3529.212812.820237@bitdiddle.concentric.net> Message-ID: <20000927223524.A1139@beelzebub> [Jeremy responds to my coding convention] > It is really unfortunate that you have chosen to go this route. We > are trying to make all the Python code in the standard library follow > Guido's coding style. Two points in my defence: * the search-and-replace work I did the other night brings the Distutils code *closer* to Guido's style guide, not farther away. I changed a lot of "foo (...)" calls to "foo(...)", because I have recently seen the light over that particular use of whitespace. (Also, almost all docstrings are now consistent with the Docstring Gospel According to Barry.) * I didn't know there was an ongoing push to standardize the standard library. When I'm done fixing up the Distutils code, if you really want to make it conform, that can be done trivially: perl -pi~ -e 's/^( *)(def|class) *\(/$1$2\(/' *.py command/*.py (Oops, that doesn't catch my end-of-function "# foo ()" convention.) I still like my way better, but an imperfect standard is better than no standard at all. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Wed Sep 27 23:48:01 2000 From: gward@python.net (Greg Ward) Date: Wed Sep 27 22:48:01 2000 Subject: [Distutils] setup.py for a wrapper extension? In-Reply-To: <005d01c028ae$ac7fa180$f73f93cd@visionart.com>; from pete@visionart.com on Wed, Sep 27, 2000 at 11:13:44AM -0700 References: <005d01c028ae$ac7fa180$f73f93cd@visionart.com> Message-ID: <20000927224529.B1139@beelzebub> On 27 September 2000, Pete Shinners said: > i've got a python extension that is simply a wrapping > around another C library. (not an uncommon case :]) I take it you mean a C library that's already installed on the target system, not when you ship the source for. > i cannot figure out the 'right' way to check for and > link/include this dependency into my distutils > setup.py script. > > under linux, it's pretty easy to assume the correct > header and shared library will be somewhere in the > standard include and library directories > /usr/local/include, /usr/local/lib, etc I think you're getting at auto-configuation here. There are some initial stabs at general, cross-platform auto-configuration in the Distutils, but they are unfinished and will remain so until after Distutils 1.0. (Bummer.) > my main concern is getting it working with windows > (and keeping it crossplatform happy for linux). my best > bet in windows seems to be walk around the parent directory > tree trying to find directories for the dependency?? I think it's probably best to avoid trying to roll-your-own auto-configuration code. Well, maybe: if you do go this route, you will probably learn a lot about cross-platform auto-configuration, and maybe you can help out with the auto-configuration support for Distutils 1.1. The cheap and easy way to do this is: let the user worry about it. As of Distutils 0.9.3, there's a very nice way to let the user worry about -- include an old-fashioned Setup file! That's right, you can put something like this in your distribution: # uncomment this lines for Unix/Linux (possibly changing library dir) #foo foo.c -L/usr/lib -lfoo # or this one for Windows #foo foo.c -L"C:\Program Files\Foo" -lfoo (I'm assuming you're compiling foo.c to create the "foo" extension module, linking against in /usr/lib/libfoo.{a,so} or "C:\Program Files\Foo\foo.dll".) Of course, your README will have to tell users to edit the Setup file before building. This is icky, but it beats editing setup.py and will have to do until we have proper auto-configuration. Then, your setup script would do this: from distutils.extension import read_setup_file ... extensions = read_setup_file("Setup") if not extensions: sys.exit("you forgot to edit Setup!") setup(name = "foo", version = "1.0", ..., extensions = extensions) and away-you-go. Let me know how it works... this is pretty new code, and It Worked For Me (TM). This does tie you to Distutils 0.9.3, which of course ties you to Distutils 0.9.4 since 0.9.3 was broken (oops). Oh well, it keeps everyone in practice, constantly upgrading the Distutils. ;-) Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From mfletch@tpresence.com Thu Sep 28 01:52:03 2000 From: mfletch@tpresence.com (Mike Fletcher) Date: Thu Sep 28 00:52:03 2000 Subject: [Distutils] Bug report: debug builds on msvc compiler don't specify *_d.lib a s link sources Message-ID: In ccompiler, you have this code in gen_lib_options: for lib in libraries: (lib_dir, lib_name) = os.path.split (lib) if lib_dir: lib_file = compiler.find_library_file ([lib_dir], lib_name) if lib_file: lib_opts.append (lib_file) else: compiler.warn ("no library file corresponding to " "'%s' found (skipping)" % lib) else: lib_opts.append (compiler.library_option (lib)) Note that you aren't passing "debug" to compiler.find_library_file, so the debug libraries will never get used. Also note that msvccompiler doesn't provide for passing the debug flag to library_option, so it doesn't do the proper name mangling (adding _d) for the library name. I'm guessing you'll need to make gen_lib_options get passed the debug flag, and then make it pass that flag on to the compiler methods. No clue what to do for non-vc++ compilers (or even if they need debug flags). Enjoy, Mike From martin@loewis.home.cs.tu-berlin.de Thu Sep 28 05:17:01 2000 From: martin@loewis.home.cs.tu-berlin.de (Martin v. Loewis) Date: Thu Sep 28 04:17:01 2000 Subject: [Distutils] Re: [XML-SIG] [Bug #115544] Compile Problems on Mac OS X In-Reply-To: <200009280733.AAA02214@bush.i.sourceforge.net> (noreply@sourceforge.net) References: <200009280733.AAA02214@bush.i.sourceforge.net> Message-ID: <200009280810.KAA01330@loewis.home.cs.tu-berlin.de> > Details: I have successfully compiled and installed Python 2.0b2 on > my Mac OS X system, but I am having trouble building the current > Python/XML CVS version. During the linking stage, I get a warning > and an error: I assume that your Python build does not include the pyexpat module, right? If it did, PyXML should not attempt to build it itself. Since building is controlled by distutils, I've cc'ed this to the distutils-sig. Since likely fixes will also need to appear in Python 2.0, it would be good if we resolve this within the next week. > cc -bundle -prebind build/temp.darwin-1.2-Power Macintosh-2.0/extensions/pyexpat.o build/temp.darwin-1.2-Power Macintosh-2.0/extensions/expat/xmltok/xmltok.o build/temp.darwin-1.2-Power Macintosh-2.0/extensions/expat/xmltok/xmlrole.o build/temp.darwin-1.2-Power Macintosh-2.0/extensions/expat/xmlwf/xmlfile.o build/temp.darwin-1.2-Power Macintosh-2.0/extensions/expat/xmlwf/xmlwf.o build/temp.darwin-1.2-Power Macintosh-2.0/extensions/expat/xmlwf/codepage.o build/temp.darwin-1.2-Power Macintosh-2.0/extensions/expat/xmlparse/xmlparse.o build/temp.darwin-1.2-Power Macintosh-2.0/extensions/expat/xmlparse/hashtable.o build/temp.darwin-1.2-Power Macintosh-2.0/extensions/expat/xmlwf/unixfilemap.o -o build/lib.darwin-1.2-Power Macintosh-2.0/_xmlplus/parsers/pyexpat.so I guess I need some help in resolving this, as I have no MacOS X myself. Can you spot anything that is incorrect in this command line? > /usr/bin/ld: warning -prebind has no effect with -bundle What is the intended effect of -prebind and -bundle? Where do these options come from, anyway? Grepping the distutils sources, I see no mentioning of them, so what magic finds those options? > /usr/bin/ld: Undefined symbols: > _PyArg_ParseTuple > _PyArg_ParseTupleAndKeywords > ...*removed a few dozen more symbols*... It's certainly the case that these symbols are undefined - we are building an extension module, after all. On MacOS X, the compiler is gcc, right? Normally, you'll have to pass -shared to the compiler to have it build shared libraries. I notice that Python's configure.in has two alternatives for building extension modules on next/*: If dyld is used, then the options are "-bundle -prebind", otherwise they are "-nostdlib -r". Do you use WITH_DYLD? Can you try linking the module with "-nostdlib -r"? Thanks, Martin > For detailed info, follow this link: > http://sourceforge.net/bugs/?func=detailbug&bug_id=115544&group_id=6473 From thomas.heller@ion-tof.com Thu Sep 28 14:00:01 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Thu Sep 28 13:00:01 2000 Subject: [Distutils] Bug report: debug builds on msvc compiler don't specify *_d.lib as link sources References: Message-ID: <029f01c0296d$588aa830$4500a8c0@thomasnb> > In ccompiler, you have this code in gen_lib_options: > > for lib in libraries: > (lib_dir, lib_name) = os.path.split (lib) > if lib_dir: > lib_file = compiler.find_library_file ([lib_dir], lib_name) > if lib_file: > lib_opts.append (lib_file) > else: > compiler.warn ("no library file corresponding to " > "'%s' found (skipping)" % lib) > else: > lib_opts.append (compiler.library_option (lib)) > > Note that you aren't passing "debug" to compiler.find_library_file, so the > debug libraries will never get used. Also note that msvccompiler doesn't > provide for passing the debug flag to library_option, so it doesn't do the > proper name mangling (adding _d) for the library name. I'm guessing you'll > need to make gen_lib_options get passed the debug flag, and then make it > pass that flag on to the compiler methods. No clue what to do for non-vc++ > compilers (or even if they need debug flags). You are completely right: MSVCCompiler links with the wrong library on debug builds. I just tried and noticed that python15.lib is explicitely passed to the linker (for both debug and release builds) although this is completely unneccessary for MSVC, since the python import library is already set with in pragma in Python.h. Maybe you should try one of the older code snapshots (for a certain definition of 'old')... Thomas From thomas.heller@ion-tof.com Thu Sep 28 14:06:01 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Thu Sep 28 13:06:01 2000 Subject: [Distutils] Distutils 1.0 code freeze: Oct 1 References: <20000926205312.A1470@beelzebub> Message-ID: <02af01c0296e$40cf1b30$4500a8c0@thomasnb> > If you know about any outstanding Distutils bugs, please tell me *now*. > Put 'em in the SourceForge bug database if you're wondering why I > haven't fixed them yet -- they might have gotten lost, I might not know > about 'em, etc. If you're not sure, put it in SourceForge. Mike Fletcher found a another bug: Building extensions on windows (at least with MSVC) in debug mode link with the wrong python import library. This leads to crashes because the extension loads the wrong python dll at runtime. Will report this on sourceforge, although I doubt Greg will be able to fix this... Distutils code freeze: Greg, I have some time next week to work on this. Do you give me permission to check it in if I find a solution? Thomas From pete@visionart.com Thu Sep 28 15:12:13 2000 From: pete@visionart.com (Pete Shinners) Date: Thu Sep 28 14:12:13 2000 Subject: [Distutils] borland compiler Message-ID: <005701c02977$e5b84640$f73f93cd@visionart.com> i'm trying to get borland's free commandline compiler working with a sample module i'm testing with. everything goes fine in the compile, but the linker is giving me the following error: Error: 'D:\PYTHON\LIBS\PYTHON20.LIB' contains invalid OMF record, type 0x21 (possibly COFF) ok, sounds like the python20.lib is incompatible with borland's compiler (??). if so, that seems like a big pain. does anyone have the steps needed to get this compiler working? i've not found any help from the docs or source yet. i'm guessing i'll need to recompile python with borland? (i didn't think that was an option?) perhaps i'm better off going with cygnus or mingw? what is the most common free win32 compiler for use with distutils? From thomas.heller@ion-tof.com Thu Sep 28 15:37:02 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Thu Sep 28 14:37:02 2000 Subject: [Distutils] borland compiler References: <005701c02977$e5b84640$f73f93cd@visionart.com> Message-ID: <039001c0297a$fe285960$4500a8c0@thomasnb> Which distutils version are you using? > i'm trying to get borland's free commandline compiler > working with a sample module i'm testing with. > > everything goes fine in the compile, but the linker is > giving me the following error: > > > Error: 'D:\PYTHON\LIBS\PYTHON20.LIB' contains invalid > OMF record, type 0x21 (possibly COFF) > Just at this moment I'm looking into this issue (currently for MSVC, soon for Borland). > > ok, sounds like the python20.lib is incompatible with > borland's compiler (??). if so, that seems like a big > pain. does anyone have the steps needed to get this > compiler working? i've not found any help from the > docs or source yet. i'm guessing i'll need to recompile > python with borland? (i didn't think that was an option?) No, bcpp contains an utility to convert OMF to COFF, so you should be able to generate a valid library. The real problem seems to be that distutils currently has problems with library names, so this issue will have to be fixed. > > perhaps i'm better off going with cygnus or mingw? what > is the most common free win32 compiler for use with distutils? MSVC (which is the best compiler for win32), can be freely downloaded from MS if you don't care for (very) large downloads. There was a thread yesterday on c.l.p about this. Thomas From thomas.heller@ion-tof.com Thu Sep 28 15:53:01 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Thu Sep 28 14:53:01 2000 Subject: [Distutils] borland compiler References: <005701c02977$e5b84640$f73f93cd@visionart.com> <039001c0297a$fe285960$4500a8c0@thomasnb> Message-ID: <044901c0297d$37253510$4500a8c0@thomasnb> > No, bcpp contains an utility to convert OMF to COFF, > so you should be able to generate a valid library. > As borland compiler user you are required to read the source, since it isn't mentioned anywhere else: def find_library_file (self, dirs, lib, debug=0): # List of effective library names to try, in order of preference: # xxx_bcpp.lib is better than xxx.lib # and xxx_d.lib is better than xxx.lib if debug is set # # The "_bcpp" suffix is to handle a Python installation for people # with multiple compilers (primarily Distutils hackers, I suspect # ;-). The idea is they'd have one static library for each # compiler they care about, since (almost?) every Windows compiler # seems to have a different format for static libraries. if debug: dlib = (lib + "_d") try_names = (dlib + "_bcpp", lib + "_bcpp", dlib, lib) else: try_names = (lib + "_bcpp", lib) for dir in dirs: for name in try_names: libfile = os.path.join(dir, self.library_filename(name)) if os.path.exists(libfile): return libfile else: # Oops, didn't find it in *any* of 'dirs' return None Thomas From pete@visionart.com Thu Sep 28 15:53:06 2000 From: pete@visionart.com (Pete Shinners) Date: Thu Sep 28 14:53:06 2000 Subject: [Distutils] borland compiler References: <005701c02977$e5b84640$f73f93cd@visionart.com> <039001c0297a$fe285960$4500a8c0@thomasnb> Message-ID: <007b01c0297d$8fd844e0$f73f93cd@visionart.com> > Just at this moment I'm looking into this issue (currently > for MSVC, soon for Borland). > > The real problem seems to be that distutils currently has > problems with library names, so this issue will have to be fixed. > > > > perhaps i'm better off going with cygnus or mingw? what > > is the most common free win32 compiler for use with distutils? thanks thomas. i found a website with more info. http://www.cyberus.ca/~g_will/pyExtenDL.shtml using all the info garnerd. running a simple command, "coff2omf python20.lib python20_bcpp.lib", has yielded perfect results. i can now do "python setup.py build --c=bcpp" and get perfect PYD files from my extension source. distutils wins again! > MSVC (which is the best compiler for win32), can be freely > downloaded from MS if you don't care for (very) large downloads. > There was a thread yesterday on c.l.p about this. good golly, my newsfeed is ugly now, but i'll check for this on deja From thomas.heller@ion-tof.com Thu Sep 28 16:38:02 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Thu Sep 28 15:38:02 2000 Subject: [Distutils] Automatically detect wrong lib-format for borland compiler? Message-ID: <04f601c02983$7d877da0$4500a8c0@thomasnb> I just had the idea to make bcccompiler a little bit more robust. By default, python is delivered with pythonxx.lib and pythonxx_d.lib import libraries generated by MSVC, which are in COFF format. By inspection I found that these libraries start with the bytes '!./' (without the single quotes). Shouldn't we open these files and issue a warning or error if they start with these bytes? We could even invoke borland's coff2omf utility to convert them... Thomas From thomas.heller@ion-tof.com Thu Sep 28 16:38:07 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Thu Sep 28 15:38:07 2000 Subject: [Distutils] Fw: [Python-checkins] CVS: distutils/distutils/command build_ext.py,1.66,1.67 Message-ID: <04f501c02983$62f609c0$4500a8c0@thomasnb> > Update of /cvsroot/python/distutils/distutils/command > In directory slayer.i.sourceforge.net:/tmp/cvs-serv31686 > > Modified Files: > build_ext.py > Log Message: > Removed the implib_dir instance variable because it is unused. > > Removed get_ext_libname() because it is unused. > > Fixed get_libraries() to append an '_d' to the python debug > import library. If MSVC is used, do not add 'pythonxx.lib' to > the list of libraries, because this is handled better > by a pragma in config.h. > > This should fix bug #115595, but it needs some more testing. > > > Index: build_ext.py > =================================================================== > RCS file: /cvsroot/python/distutils/distutils/command/build_ext.py,v > retrieving revision 1.66 > retrieving revision 1.67 > diff -C2 -r1.66 -r1.67 > *** build_ext.py 2000/09/25 01:41:15 1.66 > --- build_ext.py 2000/09/28 19:28:35 1.67 > *************** > *** 159,163 **** > if os.name == 'nt': > self.library_dirs.append (os.path.join(sys.exec_prefix, 'libs')) > - self.implib_dir = self.build_temp > if self.debug: > self.build_temp = os.path.join (self.build_temp, "Debug") > --- 159,162 ---- > *************** > *** 544,556 **** > return apply (os.path.join, ext_path) + so_ext > > - def get_ext_libname (self, ext_name): > - # create a filename for the (unneeded) lib-file. > - # extensions in debug_mode are named 'module_d.pyd' under windows > - ext_path = string.split (ext_name, '.') > - if os.name == 'nt' and self.debug: > - return apply (os.path.join, ext_path) + '_d.lib' > - return apply (os.path.join, ext_path) + '.lib' > - > - > def get_export_symbols (self, ext): > """Return the list of symbols that a shared extension has to > --- 543,546 ---- > *************** > *** 574,580 **** > # config.h that MSVC groks. The other Windows compilers all seem > # to need it mentioned explicitly, though, so that's what we do. > ! if sys.platform == "win32": > ! pythonlib = ("python%d%d" % > ! (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff)) > # don't extend ext.libraries, it may be shared with other > # extensions, it is a reference to the original list > --- 564,576 ---- > # config.h that MSVC groks. The other Windows compilers all seem > # to need it mentioned explicitly, though, so that's what we do. > ! # Append '_d' to the python import library on debug builds. > ! from distutils.msvccompiler import MSVCCompiler > ! if sys.platform == "win32" and \ > ! not isinstance(self.compiler, MSVCCompiler): > ! template = "python%d%d" > ! if self.debug: > ! template = template + '_d' > ! pythonlib = (template % > ! (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff)) > # don't extend ext.libraries, it may be shared with other > # extensions, it is a reference to the original list > > > _______________________________________________ > Python-checkins mailing list > Python-checkins@python.org > http://www.python.org/mailman/listinfo/python-checkins From R.Liebscher@gmx.de Fri Sep 29 08:29:01 2000 From: R.Liebscher@gmx.de (Rene Liebscher) Date: Fri Sep 29 07:29:01 2000 Subject: [Distutils] Bug report: debug builds on msvc compiler don't specify *_d.lib as link sources References: <029f01c0296d$588aa830$4500a8c0@thomasnb> Message-ID: <39D47C1D.AB9BE910@gmx.de> This is a multi-part message in MIME format. --------------E7FFB86D1F6AAC725CEA32D5 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Thomas Heller wrote: > > > In ccompiler, you have this code in gen_lib_options: > > > > for lib in libraries: > > (lib_dir, lib_name) = os.path.split (lib) > > if lib_dir: > > lib_file = compiler.find_library_file ([lib_dir], lib_name) > > if lib_file: > > lib_opts.append (lib_file) > > else: > > compiler.warn ("no library file corresponding to " > > "'%s' found (skipping)" % lib) > > else: > > lib_opts.append (compiler.library_option (lib)) > > > > Note that you aren't passing "debug" to compiler.find_library_file, so the > > debug libraries will never get used. Also note that msvccompiler doesn't > > provide for passing the debug flag to library_option, so it doesn't do the > > proper name mangling (adding _d) for the library name. I'm guessing > you'll > > need to make gen_lib_options get passed the debug flag, and then make it > > pass that flag on to the compiler methods. No clue what to do for > non-vc++ > > compilers (or even if they need debug flags). BorlandCCompiler doesn't use gen_lib_options() and handles debug libraries correct. It uses its own find_library_file() which uses python15_d.lib (or python15_d_bcpp.lib) if it exists. MSVCCompiler has also its own find_library_file() method which works the same way. The only problem is that gen_lib_option() calls it without the debug parameter. I made the changes you describe above in a patch (gen_lib_options.patch). The only problem is now, it is not tested by me because I'm currently 20 km away from my msvccompiler-distutils-test-machine. > > You are completely right: MSVCCompiler links with the wrong library > on debug builds. > > I just tried and noticed that python15.lib is explicitely passed > to the linker (for both debug and release builds) although this > is completely unneccessary for MSVC, since the python import library > is already set with in pragma in Python.h. Explicitely passing 'python15' as library should not be problem if gen_lib_options() works correct. (You would name a library twice (pragma and explicit) for MSVC, but on the other side 'build_ext' doesn't need to know about special compiler classes.) kind regards Rene Liebscher --------------E7FFB86D1F6AAC725CEA32D5 Content-Type: text/plain; charset=us-ascii; name="gen_lib_options.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="gen_lib_options.patch" diff -BurN --exclude=*.pyc --minimal distutils/distutils/ccompiler.py distutils.patched/distutils/ccompiler.py --- distutils/distutils/ccompiler.py Wed Sep 27 11:26:08 2000 +++ distutils.patched/distutils/ccompiler.py Fri Sep 29 13:11:54 2000 @@ -969,7 +969,11 @@ # gen_preprocess_options () -def gen_lib_options (compiler, library_dirs, runtime_library_dirs, libraries): +def gen_lib_options (compiler, + library_dirs, + runtime_library_dirs, + libraries, + debug=0): """Generate linker options for searching library directories and linking with specific libraries. 'libraries' and 'library_dirs' are, respectively, lists of library names (not filenames!) and search @@ -993,7 +997,7 @@ for lib in libraries: (lib_dir, lib_name) = os.path.split (lib) if lib_dir: - lib_file = compiler.find_library_file ([lib_dir], lib_name) + lib_file = compiler.find_library_file ([lib_dir], lib_name, debug) if lib_file: lib_opts.append (lib_file) else: diff -BurN --exclude=*.pyc --minimal distutils/distutils/command/build_ext.py distutils.patched/distutils/command/build_ext.py --- distutils/distutils/command/build_ext.py Fri Sep 29 10:24:57 2000 +++ distutils.patched/distutils/command/build_ext.py Fri Sep 29 13:16:08 2000 @@ -568,8 +568,8 @@ if sys.platform == "win32" and \ not isinstance(self.compiler, MSVCCompiler): template = "python%d%d" - if self.debug: - template = template + '_d' + #if self.debug: + # template = template + '_d' pythonlib = (template % (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff)) # don't extend ext.libraries, it may be shared with other diff -BurN --exclude=*.pyc --minimal distutils/distutils/msvccompiler.py distutils.patched/distutils/msvccompiler.py --- distutils/distutils/msvccompiler.py Wed Sep 27 11:26:10 2000 +++ distutils.patched/distutils/msvccompiler.py Fri Sep 29 13:12:34 2000 @@ -420,7 +420,7 @@ lib_opts = gen_lib_options (self, library_dirs, runtime_library_dirs, - libraries) + libraries, debug) if output_dir is not None: output_filename = os.path.join (output_dir, output_filename) --------------E7FFB86D1F6AAC725CEA32D5-- From R.Liebscher@gmx.de Fri Sep 29 08:44:01 2000 From: R.Liebscher@gmx.de (Rene Liebscher) Date: Fri Sep 29 07:44:01 2000 Subject: [Distutils] Automatically detect wrong lib-format for borland compiler? References: <04f601c02983$7d877da0$4500a8c0@thomasnb> Message-ID: <39D47F9B.4518D61C@gmx.de> Thomas Heller wrote: > > I just had the idea to make bcccompiler a little bit > more robust. > > By default, python is delivered with pythonxx.lib > and pythonxx_d.lib import libraries generated by MSVC, > which are in COFF format. By inspection I found > that these libraries start with the bytes '!./' > (without the single quotes). '!\012/' > > Shouldn't we open these files and issue a warning > or error if they start with these bytes? > We could even invoke borland's coff2omf utility > to convert them... But where to write the converted files, in the python libs directory is not a good idea and if you would write them in the build-temp directory you had to add code to find_library_file(). It might be better to have script(s) in the misc directory, which checks for a certain compiler the libraries and converts them if necessary. The compiler classes would print a warning/error message about the wrong format and could mention how to convert the library. From thomas.heller@ion-tof.com Fri Sep 29 10:19:01 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri Sep 29 09:19:01 2000 Subject: [Distutils] Bug report: debug builds on msvc compiler don't specify *_d.lib as link sources References: <029f01c0296d$588aa830$4500a8c0@thomasnb> <39D47C1D.AB9BE910@gmx.de> Message-ID: <015a01c02a17$c677ac80$4500a8c0@thomasnb> > BorlandCCompiler doesn't use gen_lib_options() and handles debug > libraries > correct. It uses its own find_library_file() which uses python15_d.lib > (or python15_d_bcpp.lib) if it exists. > MSVCCompiler has also its own find_library_file() method which works the > same way. The only problem is that gen_lib_option() calls it without the > debug parameter. Your patch does not work for MSVC: msvccompiler->link() is called with libraries = [..., 'python15'] This calls gen_lib_options() in ccompiler, this calls msvccompiler->library_option() this calls msvccompiler->library_filename('python15'), which is implemented in the CCompiler superclass, and this returns 'python15.lib' EVEN IF debug is set. Bang! > > I made the changes you describe above in a patch > (gen_lib_options.patch). > The only problem is now, it is not tested by me because I'm currently > 20 km away from my msvccompiler-distutils-test-machine. You can test all compilers without having them by something like setup.py build_ext [-g] -n --compiler=[bcpp|msvc|...] (note the -n flag) and looking at the output. Nice feature! BTW: I already have checked in my patches (with the special casing of MSVCCompiler in build_ext removed again). Maybe this is not the right way to do it, but it works. I tested it with several distributions: Numerical-15.3 (from sourceforge), ExtensionClass-2.2.1 and ZODB-2.2.1 (distutilized by akuchling), PyXML-0.5.5.1 from xml-sig, LinkChecker-1.2.5 (sourceforge). in debug and release builds. Maybe I did it not the nicest way, but they work, and currently I'm more worried about the python 2.0 (and distutils 1.0) release schedule than anything else. Hopefully you (and maybe others) can also test the CVS version even more. Thomas From R.Liebscher@gmx.de Fri Sep 29 11:04:13 2000 From: R.Liebscher@gmx.de (Rene Liebscher) Date: Fri Sep 29 10:04:13 2000 Subject: [Distutils] Bug report: debug builds on msvc compiler don't specify *_d.lib as link sources References: <029f01c0296d$588aa830$4500a8c0@thomasnb> <39D47C1D.AB9BE910@gmx.de> <015a01c02a17$c677ac80$4500a8c0@thomasnb> Message-ID: <39D4A088.2883F5B9@gmx.de> Thomas Heller wrote: > > > BorlandCCompiler doesn't use gen_lib_options() and handles debug > > libraries > > correct. It uses its own find_library_file() which uses python15_d.lib > > (or python15_d_bcpp.lib) if it exists. > > MSVCCompiler has also its own find_library_file() method which works the > > same way. The only problem is that gen_lib_option() calls it without the > > debug parameter. > > Your patch does not work for MSVC: > > msvccompiler->link() is called with libraries = [..., 'python15'] > This calls gen_lib_options() in ccompiler, > this calls msvccompiler->library_option() > this calls msvccompiler->library_filename('python15'), which > is implemented in the CCompiler superclass, > and this returns 'python15.lib' EVEN IF debug is set. Bang! > I see, there is a second path in gen_lib_options() which uses library_option(). So we have now another candidate for a debug parameter. Good to know, I think, the problem will then have to be fixed after the release of Distutils 1.0. Rene From gward@python.net Fri Sep 29 22:46:01 2000 From: gward@python.net (Greg Ward) Date: Fri Sep 29 21:46:01 2000 Subject: [Distutils] Does the SWIG support work? Message-ID: <20000929214347.A2128@beelzebub> Hi all -- some months ago, I added completely experimental and untested support for compiling SWIG extensions to the Distutils. I even documented it! But I have been unable to verify for myself that it works, since I have been unable to build SWIG on my home PC. I haven't heard any reports of success, and the first reports of failure were from 4Thought last week. So: can anyone vouch for whether the Distutils SWIG support works at all, does the "right thing", etc? What's broken about it, what works? Should it even be documented? Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Sat Sep 30 17:53:02 2000 From: gward@python.net (Greg Ward) Date: Sat Sep 30 16:53:02 2000 Subject: [Distutils] Byte-compilation revisited Message-ID: <20000930165104.A1413@beelzebub> Hi all -- based on and inspired by recent patches from Marc-Andre Lemburg and Rene Liebscher, I've finally started tackling the byte-compilation problem in earnest. Here's the approach I'm taking: * new function 'byte_compile()' in distutils.util: this is the all- singing, all-dancing wrapper around py_compile that will do all the real work * reduce the 'bytecompile()' method in the install_lib command to a simple wrapper around 'util.byte_compile()', that does the Right Thing with respect to optimization and claimed source filename written to the .py{c,o} file * add similar functionality to the build_py command, so that you may optionally do byte-compilation at build time rather than install time. The first two steps are done and checked in, except that install_lib's 'bytecompile()' method doesn't yet take advantage of the fancy features in the new 'byte_compile()' -- it doesn't rewrite filenames or do optimization. The default will continue to be doing compilation at install time rather than build time. I'm still leaning towards build-time compilation, but it's too late in the Distutils 1.0 release cycle to change things like this. However, I want to have the *option* to do compilation at build time, so people can experiment with it, see if it works, figure out what other features are needed so it really works, etc. The idea is that developers could put settings in their setup.cfg that control when to do byte-compilation; I suspect developers who want to distribute closed-source modules will have to do build-time compilation. Probably the "install" command will need some sort of "don't install source" option, or maybe the build command should have a "blow away source after compiling it" option. Here's my 'byte_compile()' function: as usual, it works for me. Please review it, and if you're following CVS, try it out. (Should be enough to install any module distribution containing pure Python modules.) ------------------------------------------------------------------------ def byte_compile (py_files, optimize=0, force=0, prefix=None, base_dir=None, verbose=1, dry_run=0, direct=None): """Byte-compile a collection of Python source files to either .pyc or .pyo files in the same directory. 'optimize' must be one of the following: 0 - don't optimize (generate .pyc) 1 - normal optimization (like "python -O") 2 - extra optimization (like "python -OO") If 'force' is true, all files are recompiled regardless of timestamps. The source filename encoded in each bytecode file defaults to the filenames listed in 'py_files'; you can modify these with 'prefix' and 'basedir'. 'prefix' is a string that will be stripped off of each source filename, and 'base_dir' is a directory name that will be prepended (after 'prefix' is stripped). You can supply either or both (or neither) of 'prefix' and 'base_dir', as you wish. If 'verbose' is true, prints out a report of each file. If 'dry_run' is true, doesn't actually do anything that would affect the filesystem. Byte-compilation is either done directly in this interpreter process with the standard py_compile module, or indirectly by writing a temporary script and executing it. Normally, you should let 'byte_compile()' figure out to use direct compilation or not (see the source for details). The 'direct' flag is used by the script generated in indirect mode; unless you know what you're doing, leave it set to None. """ # First, if the caller didn't force us into direct or indirect mode, # figure out which mode we should be in. We take a conservative # approach: choose direct mode *only* if the current interpreter is # in debug mode and optimize is 0. If we're not in debug mode (-O # or -OO), we don't know which level of optimization this # interpreter is running with, so we can't do direct # byte-compilation and be certain that it's the right thing. Thus, # always compile indirectly if the current interpreter is in either # optimize mode, or if either optimization level was requested by # the caller. if direct is None: direct = (__debug__ and optimize == 0) # "Indirect" byte-compilation: write a temporary script and then # run it with the appropriate flags. if not direct: from tempfile import mktemp script_name = mktemp(".py") if verbose: print "writing byte-compilation script '%s'" % script_name if not dry_run: script = open(script_name, "w") script.write("""\ from distutils.util import byte_compile files = [ """) script.write(string.join(map(repr, py_files), ",\n") + "]\n") script.write(""" byte_compile(files, optimize=%s, force=%s, prefix=%s, base_dir=%s, verbose=%s, dry_run=0, direct=1) """ % (`optimize`, `force`, `prefix`, `base_dir`, `verbose`)) script.close() cmd = [sys.executable, script_name] if optimize == 1: cmd.insert(1, "-O") elif optimize == 2: cmd.insert(1, "-OO") spawn(cmd, verbose=verbose, dry_run=dry_run) # "Direct" byte-compilation: use the py_compile module to compile # right here, right now. Note that the script generated in indirect # mode simply calls 'byte_compile()' in direct mode, a weird sort of # cross-process recursion. Hey, it works! else: from py_compile import compile for file in py_files: if file[-3:] != ".py": raise ValueError, \ "invalid filename: %s doesn't end with '.py'" % `file` # Terminology from the py_compile module: # cfile - byte-compiled file # dfile - purported source filename (same as 'file' by default) cfile = file + (__debug__ and "c" or "o") dfile = file if prefix: if file[:len(prefix)] != prefix: raise ValueError, \ ("invalid prefix: filename %s doesn't start with %s" % (`file`, `prefix`)) dfile = dfile[len(prefix):] if base_dir: dfile = os.path.join(base_dir, dfile) cfile_base = os.path.basename(cfile) if direct: if force or newer(file, cfile): if verbose: print "byte-compiling %s to %s" % (file, cfile_base) if not dry_run: compile(file, cfile, dfile) else: if verbose: print "skipping byte-compilation of %s to %s" % \ (file, cfile_base) ------------------------------------------------------------------------ -- Greg Ward gward@python.net http://starship.python.net/~gward/ From gward@python.net Sat Sep 30 18:09:01 2000 From: gward@python.net (Greg Ward) Date: Sat Sep 30 17:09:01 2000 Subject: [Distutils] borland compiler In-Reply-To: <044901c0297d$37253510$4500a8c0@thomasnb>; from thomas.heller@ion-tof.com on Thu, Sep 28, 2000 at 08:52:08PM +0200 References: <005701c02977$e5b84640$f73f93cd@visionart.com> <039001c0297a$fe285960$4500a8c0@thomasnb> <044901c0297d$37253510$4500a8c0@thomasnb> Message-ID: <20000930170634.A1109@beelzebub> On 28 September 2000, Thomas Heller said: > > No, bcpp contains an utility to convert OMF to COFF, > > so you should be able to generate a valid library. > > > As borland compiler user you are required to read the source, > since it isn't mentioned anywhere else: Thanks for pointing this out. I've just added a skeletal section to the "Installing Python Modules" manual called "Building Extensions: Tips and Tricks". This is where these sort of questions should be answered; feel free to mail me suggestions, patches to the doc, etc. I'll write something up for the non-Microsoft compilers, but I'll need to help to test and fact-check what I write! Greg From gward@python.net Sat Sep 30 18:10:01 2000 From: gward@python.net (Greg Ward) Date: Sat Sep 30 17:10:01 2000 Subject: [Distutils] Automatically detect wrong lib-format for borland compiler? In-Reply-To: <39D47F9B.4518D61C@gmx.de>; from R.Liebscher@gmx.de on Fri, Sep 29, 2000 at 01:40:11PM +0200 References: <04f601c02983$7d877da0$4500a8c0@thomasnb> <39D47F9B.4518D61C@gmx.de> Message-ID: <20000930170823.B1109@beelzebub> On 29 September 2000, Rene Liebscher said: > It might be better to have script(s) in the misc directory, > which checks for a certain compiler the libraries and > converts them if necessary. > > The compiler classes would print a warning/error message > about the wrong format and could mention how to convert > the library. I think I prefer this approach -- easy to go overboard here. Being smart enough to *detect* a possible bad situation is reasonable, but trying to be smart enough to *correct* it is asking for trouble. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/