From yueqiang@earth.usc.edu Tue Feb 1 00:55:32 2000 From: yueqiang@earth.usc.edu (Yueqiang Huang) Date: Mon, 31 Jan 2000 16:55:32 -0800 (PST) Subject: [Distutils] installation error on UNIX Message-ID: I tried to install the latest version 0.1.3, but got the following message: python setup.py install --prefix=/auto/terra-06/wrk6/yueqiang/Python --exec-prefix=/auto/terra-06/wrk6/yueqiang/Python running install running build running build_py Traceback (innermost last): File "setup.py", line 13, in ? setup (name = "Distutils", File "distutils/core.py", line 97, in setup dist.run_commands () File "distutils/core.py", line 526, in run_commands self.run_command (cmd) File "distutils/core.py", line 575, in run_command cmd_obj.run () File "distutils/command/install.py", line 282, in run self.run_peer ('build') File "distutils/core.py", line 886, in run_peer self.distribution.run_command (command) File "distutils/core.py", line 575, in run_command cmd_obj.run () File "distutils/command/build.py", line 49, in run self.run_peer ('build_py') File "distutils/core.py", line 886, in run_peer self.distribution.run_command (command) File "distutils/core.py", line 575, in run_command cmd_obj.run () File "distutils/command/build_py.py", line 93, in run self.build_packages () File "distutils/command/build_py.py", line 290, in build_packages package_dir = self.get_package_dir (package) File "distutils/command/build_py.py", line 112, in get_package_dir return apply (os.path.join, path) TypeError: apply() 2nd argument must be tuple How to fix this? Thanks in advance! yueqiang _______________________________________________________________________ Yueqiang Huang, Department of Earth Sciences, University of Southern CA Los Angeles, CA 90089-0740 USA From gward@cnri.reston.va.us Tue Feb 1 13:51:51 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 1 Feb 2000 08:51:51 -0500 Subject: [Distutils] distutils on hpux In-Reply-To: ; from swalton@apollo.sfo.csun.edu on Wed, Jan 26, 2000 at 03:08:40PM -0800 References: Message-ID: <20000201085151.A17402@cnri.reston.va.us> On 26 January 2000, Stephen Walton said: > I just joined the list but am not sure I should have :-). I really just > want to get distutils working on our HP/UX 10.20 host so I can install > NumPy. > > Using the 0.1.2 release, all the tests fail, basically because "from > sysconfig import AR" fails. Where should that come from? It appears it > should be in the config.h set up when I built Python, but I can't find it. Hmmm. You're close, but `AR' is actually supposed to be set in the Makefile installed with Python. (Both config.h and Makefile are read by sysconfig, and all the "variables" -- C macros and make variables -- in those two files are exposed as Python global variables. Once the two files are parsed, there's no way to know if a given Python variable originated in config.h or Makefile.) Anyways, here's how things look on my system (Python 1.5.2 on Solaris 2.6): $ python Python 1.5.2 (#1, Dec 16 1999, 16:50:40) [GCC 2.8.1] on sunos5 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>> import os >>> from distutils import sysconfig >>> sysconfig.get_makefile_filename() '/usr/local/python/lib/python1.5/config/Makefile' >>> os.system ("grep '^AR' %s" % _) AR= ar 0 >>> sysconfig.AR 'ar' Would you mind trying that out on your machine and tell us what the results are? If AR is not defined in your Makefile, then that's a problem with portability-across-Python-platforms that I need to know about. If AR is defined and sysconfig isn't picking it up, that's a bug in sysconfig that I (and Fred Drake! ;-) need to know about. Sorry for the long delay in replying -- speculating about future features is so much more fun than fixing old bugs... *sigh*... Greg From gward@cnri.reston.va.us Tue Feb 1 13:57:07 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 1 Feb 2000 08:57:07 -0500 Subject: [Distutils] Complling on IRIX OS In-Reply-To: <3890E416.D97B745A@be-research.ucsd.edu>; from cjensen@bioeng.ucsd.edu on Thu, Jan 27, 2000 at 04:34:30PM -0800 References: <3890E416.D97B745A@be-research.ucsd.edu> Message-ID: <20000201085706.B17402@cnri.reston.va.us> On 27 January 2000, Curtis Jensen said: > To install on an IRIX OS, sometimes we need to compile with the -o32 > flag. I can't seem to find how to do this with distutils. Any ideas? Hmmm... I just checked the source, and it looks like there is no way to do this right now. ;-( Why do you need to? The Distutils default -- in fact, its *only* behaviour for now -- is to compile extensions the same way you compiled Python itself. I'm a bit fuzzy on the various 32/64 options for the IRIX 6+ compiler, but I expect you'd have a hard time loading .so's compiled one way into an executable compiled another way. What exactly are you trying to accomplish here -- better performance, ability to run on more platforms (ie. 32- and 64-bit IRIX), or what? Thanks -- Greg From gward@cnri.reston.va.us Tue Feb 1 14:00:06 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 1 Feb 2000 09:00:06 -0500 Subject: [Distutils] WinNT and distutils In-Reply-To: ; from robin@jessikat.demon.co.uk on Mon, Jan 31, 2000 at 03:13:08PM +0000 References: <20000128190216.B12965@cnri.reston.va.us> <20000131093428.A17133@cnri.reston.va.us> Message-ID: <20000201090005.C17402@cnri.reston.va.us> On 31 January 2000, Robin Becker said: > it seems that not all of the directories obtained from the keys are > actually rock solid eg they are part of the devstudio customisables. > > I'm fairly sure that you could add all of the Path dirs to the path > set Include=all the directories in Include Dirs (; sep) > set Lib=all in the lib dirs (; sep) > > and then cl.exe etc are supposed to work. ??? I don't follow. Are `Include' and `Lib' environment variables that you want me to set? If you could just supply a new patch to msvccompiler.py, I think we'd save a lot of bouncing back and forth. Thanks -- Greg From gward@cnri.reston.va.us Tue Feb 1 14:04:14 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 1 Feb 2000 09:04:14 -0500 Subject: [Distutils] WinNT and distutils In-Reply-To: ; from mhammond@skippinet.com.au on Mon, Jan 31, 2000 at 03:25:01PM -0800 References: Message-ID: <20000201090414.D17402@cnri.reston.va.us> On 31 January 2000, Mark Hammond said: > > Surely msvccompiler can tell if a particular file (say cl.exe) isn't in > > any of the well known places. Then it can just issue the correct > > commands with the path attached or tell the user it can't find > > cl.exe/link.exe and then ask for vcvars32.bat to be run first. I assume > > we're not going to search all known drives for the correct exe's. > > I agree. IMO, we could take the following approach: > * Look in C:\Program Files\The default place\... > * Look in \Program Files\The default place\... > * Try and import win32api, catching the import error Two things: * what is "The default place"? * if/when a standard Python 1.6 way for accessing the Windows registry comes into existence, should it be added that to list? Otherwise, that sounds good. Patches welcome, tested patches especially so. > However, it is worth noting that running "vcvars32.bat" wont work if run via > a seperate "system()" call - the environment variables set will not persist. > The only decent solution I can see is to create a temporary batch file, that > first calls "vcvars32.bat" before the builds. Depending on how distutils > does its thing, it may just be simpler to detect the condition, and politely > ask the user to run the batch file themselves... Yecch. 1) I don't like the idea of running a batch file (or shell script on the Unix side) for every compiler invocation -- I haven't worried much about performance on the Python side so far in the Distutils (since copying and compiling files is so expensive), but I might have to draw the line at this. 2) If I have no idea where cl.exe is, how am I supposed to know where vcvars32.bat is? Greg From gward@cnri.reston.va.us Tue Feb 1 14:21:38 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 1 Feb 2000 09:21:38 -0500 Subject: [Distutils] installing distutils In-Reply-To: <3895F09D.710839ED@be-research.ucsd.edu>; from cjensen@bioeng.ucsd.edu on Mon, Jan 31, 2000 at 12:29:17PM -0800 References: <200001312010.MAA22173@octane28.nas.nasa.gov> <3895F09D.710839ED@be-research.ucsd.edu> Message-ID: <20000201092138.E17402@cnri.reston.va.us> On 31 January 2000, Curtis Jensen said: > View the USAGE file. Near the end it tells how. It'll be a command > similar to: > setup.py -v install --prefix=/home/greg --exec-prefix=/home/greg > > If /home/greg was where you wanted to install to. Some caveats here: * installing to non-standard directories doesn't work very well in general; it was poorly thought out and needs to be redesigned for the next version * so, don't curse too hard trying to get it to work -- it's my fault, not yours! * if you use the --prefix/--exec-prefix options, be aware that the standard Python library structure will be mimicked, e.g. in the above example modules will be installed to /home/greg/lib/python1.5/site-packages * if you want finer control, you should use the --install-site-lib or --install-site-platlib options (example forthcoming). * when using either --prefix/exec-prefix (or --install-site-lib/ install-site-platlib), be aware of which one to supply: only *one* of them is used to install any given Python module distribution. If any extension modules are present, only --exec-prefix (or --install-site-platlib) need be supplied; if the distribution is 100% pure Python, only --prefix (or --install-site-lib) need be supplied. Requiring users to know this is a bug. * when installing non-packagized distributions that are supposed to go in their own directory under site-packages, Distutils ignores this request when installing to a non-standard directory. (I think. It probably depends on whether you use --prefix/exec-prefix or --install-sitelib/install-site-platlib.) This is a bug. Here's the recommended way to install Numeric Python to a non-standard directory *using Distutils 0.1.x*: python setup.py install --install-site-platlib=$HOME/lib/python/Numeric where you can supply whatever base you like instead of $HOME/lib/python; but putting it into a directory called Numeric is strongly recommended, because that's what Distutils would do if you were installing to the standard library directory. Finally, you have to make sure that the exact directory named -- $HOME/lib/python/Numeric -- is in your PYTHONPATH. Please don't count on this buggy, counterintuitive behaviour existing beyond Distutils 0.1.x... Greg From thomas.heller@ion-tof.com Tue Feb 1 14:27:50 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue, 1 Feb 2000 15:27:50 +0100 Subject: [Distutils] WinNT and distutils References: <20000201090414.D17402@cnri.reston.va.us> Message-ID: <01da01bf6cc0$85d25d20$4500a8c0@thomasnotebook> > > I agree. IMO, we could take the following approach: > > * Look in C:\Program Files\The default place\... > > * Look in \Program Files\The default place\... > > * Try and import win32api, catching the import error > > Two things: > * what is "The default place"? IMHO there is no default place (tm), because even "c:\Program Files\" is only correct under certain circmustances: For a german version of NT, it is "c:\Programme", if NT is installed on another drive,it may be D:\Programme, and so on. On the other hand, if we really have to guess, it is the best guess we can make. > * if/when a standard Python 1.6 way for accessing the Windows registry > comes into existence, should it be added that to list? I think distutils is a good reason why we need it. I'm currently working on a module which hopefully will be included. > > However, it is worth noting that running "vcvars32.bat" wont work if run via > > a seperate "system()" call - the environment variables set will not persist. > > The only decent solution I can see is to create a temporary batch file, that > > first calls "vcvars32.bat" before the builds. Depending on how distutils > > does its thing, it may just be simpler to detect the condition, and politely > > ask the user to run the batch file themselves... > > Yecch. 1) I don't like the idea of running a batch file (or shell > script on the Unix side) for every compiler invocation -- I haven't > worried much about performance on the Python side so far in the > Distutils (since copying and compiling files is so expensive), but I > might have to draw the line at this. I second that. 2) If I have no idea where cl.exe > is, how am I supposed to know where vcvars32.bat is? Exactly. Note that vcvars32.bat does more than extend the PATH to include the directory where cl.exe resides, it also sets INCLUDE and LIB environment vars (which we can also supply on the command line). Thomas Heller From gward@cnri.reston.va.us Tue Feb 1 14:31:02 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 1 Feb 2000 09:31:02 -0500 Subject: [Distutils] installation error on UNIX In-Reply-To: ; from yueqiang@earth.usc.edu on Mon, Jan 31, 2000 at 04:55:32PM -0800 References: Message-ID: <20000201093101.F17402@cnri.reston.va.us> On 31 January 2000, Yueqiang Huang said: > I tried to install the latest version 0.1.3, but got the following > message: Thanks for the bug report, but you forgot two vital pieces of information: * what version of Python are you running? * what OS are you running? (just "UNIX" isn't enough -- what version of what Unix variant? if you don't know, try the "uname -a" command) > python setup.py install --prefix=/auto/terra-06/wrk6/yueqiang/Python > --exec-prefix=/auto/terra-06/wrk6/yueqiang/Python > running install > running build > running build_py > Traceback (innermost last): > File "setup.py", line 13, in ? > setup (name = "Distutils", [...] > File "distutils/command/build_py.py", line 112, in get_package_dir > return apply (os.path.join, path) > TypeError: apply() 2nd argument must be tuple Hmmm... I can't reproduce this under Python 1.5.2. The 'path' variable is always a list -- it's either returned from 'string.split()' or coerced from a tuple to a list -- and 'apply()' doesn't seem to mind at all for me. Again, what version of Python are you running? I suspect the ability for 'apply()' to take lists must have been added recently, and you're using an old version. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From thomas.heller@ion-tof.com Tue Feb 1 14:37:41 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue, 1 Feb 2000 15:37:41 +0100 Subject: Finding VC (Was: Re: [Distutils] WinNT and distutils) References: <20000201090414.D17402@cnri.reston.va.us> Message-ID: <01fc01bf6cc1$e57d1110$4500a8c0@thomasnotebook> I propose the following: 1. Convince Guido to give python 1.6 registry access in the core. I'm currently working on such a module. 2. (>=1.6) Use the registry to find Path, Include, and Lib directories for VC, and supply them on the command line. 3. (<1.6) Try to import win32api and win32con, and use Robins code to find the directories. If this also fails, inform the user that the environment must be set up for VC by running vcvars32.bat, and restarting setup.py. Note that it also may be possible the VC is not installed at all! I think we should come to a conclusion here, and offer Greg some tested patches. I'm willing to work on this. Thomas Heller From dubois1@llnl.gov Tue Feb 1 14:41:16 2000 From: dubois1@llnl.gov (Paul F. Dubois) Date: Tue, 1 Feb 2000 06:41:16 -0800 Subject: [Distutils] Complling on IRIX OS In-Reply-To: <20000201085706.B17402@cnri.reston.va.us> Message-ID: Greg, This is a *very* typical instance: there is a bug in the way this platform compiles one file. You can't just compile everything the way you compiled Python. -----Original Message----- From: distutils-sig-admin@python.org [mailto:distutils-sig-admin@python.org]On Behalf Of Greg Ward Sent: Tuesday, February 01, 2000 5:57 AM To: distutils-sig@python.org Subject: Re: [Distutils] Complling on IRIX OS On 27 January 2000, Curtis Jensen said: > To install on an IRIX OS, sometimes we need to compile with the -o32 > flag. I can't seem to find how to do this with distutils. Any ideas? Hmmm... I just checked the source, and it looks like there is no way to do this right now. ;-( Why do you need to? The Distutils default -- in fact, its *only* behaviour for now -- is to compile extensions the same way you compiled Python itself. I'm a bit fuzzy on the various 32/64 options for the IRIX 6+ compiler, but I expect you'd have a hard time loading .so's compiled one way into an executable compiled another way. What exactly are you trying to accomplish here -- better performance, ability to run on more platforms (ie. 32- and 64-bit IRIX), or what? Thanks -- Greg _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://www.python.org/mailman/listinfo/distutils-sig From thomas.heller@ion-tof.com Tue Feb 1 15:35:17 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue, 1 Feb 2000 16:35:17 +0100 Subject: [Distutils] Proposal: Registry access for Python on Windows Message-ID: <02a501bf6cc9$f1681620$4500a8c0@thomasnotebook> I propose to include functions to access the windows registry into the python 1.6 core. I have thrown together some code which I will post hopefully tomorrow, but I would like to hear some comments before. ---------------------------------------------------------------------- winreg - registry access module Constants: HKEY_CLASSES_ROOT, HKEY_LOCAL_MACHINE, HKEY_CURRENT_USER, HKEY_USERS - bases of registry KEY_READ, KEY_WRITE, KEY_ALL_ACCESS - access rights REG_SZ, REG_DWORD, REG_BINARY - typecodes for values Exception: error - raised when a function fails. Will usually contain a windows error code and a textual description. Functions: OpenKey (base, subkey, [, rights=KEY_READ]) -> integer Opens an existing key with the specified access rights and returns an integer handle. The handle must be closed by a call to CloseKey(). CreateKey (base, subkey [, sam=KEY_ALL_ACCESS]) -> integer Creates a new subkey or opens an exsiting one and returns an integer handle. base must be one of the HKEY_ constants or an integer handle. The handle must be closed by a call to CloseKey(). CloseKey (handle) Closes a key handle. EnumValues (handle, subkey) -> sequence Returns a sequence containing name, value, typecode triples for each existing value. EnumKeys (handle, subkey) -> sequence Returns a sequence containing the names of all subkeys. QueryValue (handle, subkey) -> tuple Returns a tuple containing name, value and typecode. SetValue (handle, name, typecode, value) Sets the value of a name to value. value must be a string for REG_SZ or REG_BINARY, an integer for REG_DWORD. DeleteValue (handle, valuename) Deletes the value. DeleteKey (handle, name [,recursive=0) Deletes the named key if no subkeys exist. If recursive is given as a true value, subkeys are recursively deleted. This is done with Reg* calls, NOT with SHDeleteKey and SHDeleteEmptyKey functions, so should work under any flavor of NT and Win9x. Note: To use the SetValue and Delete* functions, the key must have been opened with KEY_WRITE_ACCESS or KEY_ALL_ACCESS. ---------------------------------------------------------------------- Open Questions: Is the recursive-flag for DeleteKey() too dangerous? Should I switch from an integer handle to a full blown python-object, which would call CloseKey() automatically in the destructor? A higher level interface would be nice, but this should probably implemented on top of this module in python. Comments appreciated! Thomas Heller ION-TOF GmbH From guido@CNRI.Reston.VA.US Tue Feb 1 15:49:05 2000 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 01 Feb 2000 10:49:05 -0500 Subject: Finding VC (Was: Re: [Distutils] WinNT and distutils) In-Reply-To: Your message of "Tue, 01 Feb 2000 15:37:41 +0100." <01fc01bf6cc1$e57d1110$4500a8c0@thomasnotebook> References: <20000201090414.D17402@cnri.reston.va.us> <01fc01bf6cc1$e57d1110$4500a8c0@thomasnotebook> Message-ID: <200002011549.KAA18724@eric.cnri.reston.va.us> > 1. Convince Guido to give python 1.6 registry access in the core. > I'm currently working on such a module. Sure. Mail it to me ASAP. You'll also have to snail-mail me a wet signature on the conbtribution release form: http://www.python.org/1.5/wetsign.html. (I apologize for this sillyness, our lawyers require it and who am I to argue with them...) > 2. (>=1.6) Use the registry to find Path, Include, and Lib > directories for VC, and supply them on the command line. > > 3. (<1.6) Try to import win32api and win32con, and use Robins > code to find the directories. If this also fails, inform the user > that the environment must be set up for VC by running > vcvars32.bat, and restarting setup.py. > Note that it also may be possible the VC is not installed at all! Or the wrong version. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@CNRI.Reston.VA.US Tue Feb 1 16:07:04 2000 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 01 Feb 2000 11:07:04 -0500 Subject: [Distutils] installation error on UNIX In-Reply-To: Your message of "Tue, 01 Feb 2000 09:31:02 EST." <20000201093101.F17402@cnri.reston.va.us> References: <20000201093101.F17402@cnri.reston.va.us> Message-ID: <200002011607.LAA18811@eric.cnri.reston.va.us> > > File "distutils/command/build_py.py", line 112, in get_package_dir > > return apply (os.path.join, path) > > TypeError: apply() 2nd argument must be tuple This smells of Python version 1.5.1 or older; in 1.5.2, apply() does allow any sequence as second arg. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@CNRI.Reston.VA.US Tue Feb 1 16:11:03 2000 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 01 Feb 2000 11:11:03 -0500 Subject: [Distutils] Re: [Python-Dev] Proposal: Registry access for Python on Windows In-Reply-To: Your message of "Tue, 01 Feb 2000 16:35:17 +0100." <02a501bf6cc9$f1681620$4500a8c0@thomasnotebook> References: <02a501bf6cc9$f1681620$4500a8c0@thomasnotebook> Message-ID: <200002011611.LAA18829@eric.cnri.reston.va.us> > I propose to include functions to access the windows registry > into the python 1.6 core. > > I have thrown together some code which I will post hopefully > tomorrow, but I would like to hear some comments before. > > ---------------------------------------------------------------------- > winreg - registry access module > > Constants: > HKEY_CLASSES_ROOT, HKEY_LOCAL_MACHINE, HKEY_CURRENT_USER, > HKEY_USERS - bases of registry > KEY_READ, KEY_WRITE, KEY_ALL_ACCESS - access rights > REG_SZ, REG_DWORD, REG_BINARY - typecodes for values > > Exception: > error - raised when a function fails. Will usually contain > a windows error code and a textual description. > > Functions: > OpenKey (base, subkey, [, rights=KEY_READ]) -> integer > Opens an existing key with the specified access rights > and returns an integer handle. The handle must be closed > by a call to CloseKey(). > > CreateKey (base, subkey [, sam=KEY_ALL_ACCESS]) -> integer > Creates a new subkey or opens an exsiting one > and returns an integer handle. > base must be one of the HKEY_ constants or an integer handle. > The handle must be closed by a call to CloseKey(). > > CloseKey (handle) > Closes a key handle. > > EnumValues (handle, subkey) -> sequence > Returns a sequence containing name, value, typecode triples > for each existing value. > > EnumKeys (handle, subkey) -> sequence > Returns a sequence containing the names of all subkeys. > > QueryValue (handle, subkey) -> tuple > Returns a tuple containing name, value and typecode. > > SetValue (handle, name, typecode, value) > Sets the value of a name to value. > value must be a string for REG_SZ or REG_BINARY, > an integer for REG_DWORD. > > DeleteValue (handle, valuename) > Deletes the value. > > DeleteKey (handle, name [,recursive=0) > Deletes the named key if no subkeys exist. If recursive is > given as a true value, subkeys are recursively deleted. > This is done with Reg* calls, NOT with SHDeleteKey and > SHDeleteEmptyKey functions, so should work under any > flavor of NT and Win9x. > > Note: To use the SetValue and Delete* functions, the key must have > been opened with KEY_WRITE_ACCESS or KEY_ALL_ACCESS. > ---------------------------------------------------------------------- > Open Questions: > Is the recursive-flag for DeleteKey() too dangerous? > Should I switch from an integer handle to a full blown > python-object, which would call CloseKey() automatically in the > destructor? > A higher level interface would be nice, but this should probably > implemented > on top of this module in python. > > Comments appreciated! As long as we're redesigning the API and not just copying the registry access functions fromn win32api, shouldn't this be made a little bit more OO? Things that return a handle should return an object, and things taking a handle should be methods of that object. Also, the structured return values should probably be turned into objects with attributes. It might be acceptable to do this as a Python wrapper; in that case perhaps the C module should be called _winreg. --Guido van Rossum (home page: http://www.python.org/~guido/) From robin@jessikat.demon.co.uk Tue Feb 1 16:14:45 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Tue, 1 Feb 2000 16:14:45 +0000 Subject: [Distutils] WinNT and distutils In-Reply-To: <20000201090005.C17402@cnri.reston.va.us> References: <20000128190216.B12965@cnri.reston.va.us> <20000131093428.A17133@cnri.reston.va.us> <20000201090005.C17402@cnri.reston.va.us> Message-ID: In article <20000201090005.C17402@cnri.reston.va.us>, Greg Ward writes >On 31 January 2000, Robin Becker said: >> it seems that not all of the directories obtained from the keys are >> actually rock solid eg they are part of the devstudio customisables. >> >> I'm fairly sure that you could add all of the Path dirs to the path >> set Include=all the directories in Include Dirs (; sep) >> set Lib=all in the lib dirs (; sep) >> >> and then cl.exe etc are supposed to work. > >??? > >I don't follow. Are `Include' and `Lib' environment variables that you >want me to set? If you could just supply a new patch to >msvccompiler.py, I think we'd save a lot of bouncing back and forth. > >Thanks -- > > Greg ... OK this patch works for me with none of the vcvars things set. I tried building Numeric and it works modulo complaints about missing arrayfns.def I don't have that so the dll hasn't got an entry point. *** msvccompiler.py.org Mon Jan 17 21:57:18 2000 --- msvccompiler.py Tue Feb 01 16:11:20 2000 *************** *** 10,19 **** --- 10,104 ---- import os import sys + import string from distutils.errors import * from distutils.ccompiler import \ CCompiler, gen_preprocess_options, gen_lib_options + def _devstudio_versions(): + "Get a list of devstudio versions" + try: + import win32api + import win32con + except ImportError: + return None + + K = 'Software\\Microsoft\\Devstudio' + L = [] + for base in (win32con.HKEY_CLASSES_ROOT,win32con.HKEY_LOCAL_MACHINE,win32con.HKEY_CURRENT_USER,win32con.HKEY_USERS): + try: + k = win32api.RegOpenKeyEx(base,K) + i = 0 + while 1: + try: + p = win32api.RegEnumKey(k,i) + if p[0] in '123456789' and p not in L: + L.append(p) + except win32api.error: + break + i = i + 1 + except win32api.error: + pass + L.sort() + return L + + def _msvc_get_paths(path, vNum='6.0', platform='x86'): + "Get a devstudio path (include, lib or path)" + try: + import win32api + import win32con + except ImportError: + return None + + L = [] + if path=='lib': path= 'Library' + path = string.upper(path + ' Dirs') + K = 'Software\\Microsoft\\Devstudio\\%s\\Build System\\Components\\Platforms\\Win32 (%s)\\Directories' \ + % (vNum,platform) + for base in (win32con.HKEY_CLASSES_ROOT,win32con.HKEY_LOCAL_MACHINE,win32con.HKEY_CURRENT_USER,win32con.HKEY_USERS): + try: + k = win32api.RegOpenKeyEx(base,K) + i = 0 + while 1: + try: + (p,v,t) = win32api.RegEnumValue(k,i) + if string.upper(p)==path: + V = string.split(v,';') + for v in V: + if v=='' or v in L: continue + L.append(v) + break + i = i + 1 + except win32api.error: + break + except win32api.error: + pass + return L + + def _find_exe(exe): + for v in _devstudio_versions(): + for p in _msvc_get_paths('bin',v): + fn=os.path.join(os.path.abspath(p),exe) + if os.path.isfile(fn): return fn + + #didn't find it; try existing path + try: + for p in string.split(os.environ['Path'],';'): + fn=os.path.join(os.path.abspath(p),exe) + if os.path.isfile(fn): return fn + except: + pass + return exe #last desperate hope + + def _find_SET(n): + for v in _devstudio_versions(): + p=_msvc_get_paths(n,v) + if p!=[]: return p + return [] + + def _do_SET(n): + p=_find_SET(n) + if p!=[]: os.environ[n]=string.join(p,';') class MSVCCompiler (CCompiler) : """Concrete class that implements an interface to Microsoft Visual C++, *************** *** 34,41 **** self.add_library ( "python" + sys.version[0] + sys.version[2] ) self.add_library_dir( os.path.join( sys.exec_prefix, 'libs' ) ) ! self.cc = "cl.exe" ! self.link = "link.exe" self.preprocess_options = None self.compile_options = [ '/nologo', '/Ox', '/MD' ] --- 119,135 ---- self.add_library ( "python" + sys.version[0] + sys.version[2] ) self.add_library_dir( os.path.join( sys.exec_prefix, 'libs' ) ) ! self.cc = _find_exe("cl.exe") ! self.link = _find_exe("link.exe") ! _do_SET('lib') ! _do_SET('include') ! path=_find_SET('path') ! try: ! for p in string.split(os.environ['path'],';'): ! path.append(p) ! except KeyError: ! pass ! os.environ['path'] = string.join(path,';') self.preprocess_options = None self.compile_options = [ '/nologo', '/Ox', '/MD' ] -- Robin Becker From thomas.heller@ion-tof.com Tue Feb 1 16:33:12 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue, 1 Feb 2000 17:33:12 +0100 Subject: Finding VC (Was: Re: [Distutils] WinNT and distutils) References: <20000201090414.D17402@cnri.reston.va.us> <01fc01bf6cc1$e57d1110$4500a8c0@thomasnotebook> <200002011549.KAA18724@eric.cnri.reston.va.us> Message-ID: <033e01bf6cd2$08e60ed0$4500a8c0@thomasnotebook> > > 1. Convince Guido to give python 1.6 registry access in the core. > > I'm currently working on such a module. > > Sure. Mail it to me ASAP. You'll also have to snail-mail me a wet > signature on the conbtribution release form: > http://www.python.org/1.5/wetsign.html. (I apologize for this > sillyness, our lawyers require it and who am I to argue with them...) How much time do we have to make this ready for 1.6? Hours, days, weeks? Thomas Heller From robin@jessikat.demon.co.uk Tue Feb 1 16:22:17 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Tue, 1 Feb 2000 16:22:17 +0000 Subject: [Distutils] Proposal: Registry access for Python on Windows In-Reply-To: <02a501bf6cc9$f1681620$4500a8c0@thomasnotebook> References: <02a501bf6cc9$f1681620$4500a8c0@thomasnotebook> Message-ID: In article <02a501bf6cc9$f1681620$4500a8c0@thomasnotebook>, Thomas Heller writes >I propose to include functions to access the windows registry >into the python 1.6 core. > >I have thrown together some code which I will post hopefully >tomorrow, but I would like to hear some comments before. > aaaaaargghhhhh just when I had thrown some code together to use the win32api stuff. mumble dribble >---------------------------------------------------------------------- >winreg - registry access module > >Constants: > HKEY_CLASSES_ROOT, HKEY_LOCAL_MACHINE, HKEY_CURRENT_USER, > HKEY_USERS - bases of registry > KEY_READ, KEY_WRITE, KEY_ALL_ACCESS - access rights > REG_SZ, REG_DWORD, REG_BINARY - typecodes for values > >Exception: > error - raised when a function fails. Will usually contain > a windows error code and a textual description. > >Functions: > OpenKey (base, subkey, [, rights=KEY_READ]) -> integer > Opens an existing key with the specified access rights > and returns an integer handle. The handle must be closed > by a call to CloseKey(). > > CreateKey (base, subkey [, sam=KEY_ALL_ACCESS]) -> integer > Creates a new subkey or opens an exsiting one > and returns an integer handle. > base must be one of the HKEY_ constants or an integer handle. > The handle must be closed by a call to CloseKey(). > > CloseKey (handle) > Closes a key handle. > > EnumValues (handle, subkey) -> sequence > Returns a sequence containing name, value, typecode triples > for each existing value. > > EnumKeys (handle, subkey) -> sequence > Returns a sequence containing the names of all subkeys. > > QueryValue (handle, subkey) -> tuple > Returns a tuple containing name, value and typecode. > > SetValue (handle, name, typecode, value) > Sets the value of a name to value. > value must be a string for REG_SZ or REG_BINARY, > an integer for REG_DWORD. > > DeleteValue (handle, valuename) > Deletes the value. > > DeleteKey (handle, name [,recursive=0) > Deletes the named key if no subkeys exist. If recursive is > given as a true value, subkeys are recursively deleted. > This is done with Reg* calls, NOT with SHDeleteKey and > SHDeleteEmptyKey functions, so should work under any > flavor of NT and Win9x. > >Note: To use the SetValue and Delete* functions, the key must have >been opened with KEY_WRITE_ACCESS or KEY_ALL_ACCESS. >---------------------------------------------------------------------- >Open Questions: > Is the recursive-flag for DeleteKey() too dangerous? > Should I switch from an integer handle to a full blown > python-object, which would call CloseKey() automatically in the >destructor? > A higher level interface would be nice, but this should probably >implemented > on top of this module in python. > >Comments appreciated! > >Thomas Heller >ION-TOF GmbH > > > >_______________________________________________ >Distutils-SIG maillist - Distutils-SIG@python.org >http://www.python.org/mailman/listinfo/distutils-sig -- Robin Becker From thomas.heller@ion-tof.com Tue Feb 1 16:41:47 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue, 1 Feb 2000 17:41:47 +0100 Subject: [Distutils] Proposal: Registry access for Python on Windows References: <02a501bf6cc9$f1681620$4500a8c0@thomasnotebook> Message-ID: <036101bf6cd3$3b781d10$4500a8c0@thomasnotebook> > >I propose to include functions to access the windows registry > >into the python 1.6 core. > > > >I have thrown together some code which I will post hopefully > >tomorrow, but I would like to hear some comments before. > > > > aaaaaargghhhhh just when I had thrown some code together to use the > win32api stuff. mumble dribble We can still use your stuff for 1.5 ;-) > Robin Becker Thomas Heller From gmcm@hypernet.com Tue Feb 1 16:46:36 2000 From: gmcm@hypernet.com (Gordon McMillan) Date: Tue, 1 Feb 2000 11:46:36 -0500 Subject: [Distutils] Re: [Python-Dev] Proposal: Registry access for Python on Windows In-Reply-To: <02a501bf6cc9$f1681620$4500a8c0@thomasnotebook> Message-ID: <1262698908-4003053@hypernet.com> Thomas Heller wrote: > I propose to include functions to access the windows registry > into the python 1.6 core. > > I have thrown together some code which I will post hopefully > tomorrow, but I would like to hear some comments before. [snip API - which appears to be a fairly low level wrap] > Open Questions: > Is the recursive-flag for DeleteKey() too dangerous? As long as it defaults to non-recursive, it's fine by me. > Should I switch from an integer handle to a full blown > python-object, which would call CloseKey() automatically in the > destructor? Absodefiposilutely Yes. Not so sure you need typecode. There are 10 types defined, but most tools only do 3 (string, dword and binary) and most usage is of 2 (dword and string - which can hold binary data). So I'd think you could infer pretty safely. Actually, I guess this boils down to intent - "Python as a complete registry tool" would require all the types, but "Python can use the registry" could automatically use dword or string, as appropriate. - Gordon From sanner@scripps.edu Tue Feb 1 17:36:34 2000 From: sanner@scripps.edu (Michel Sanner) Date: Tue, 1 Feb 2000 09:36:34 -0800 Subject: [Distutils] Complling on IRIX OS In-Reply-To: Greg Ward "Re: [Distutils] Complling on IRIX OS" (Feb 1, 8:57am) References: <3890E416.D97B745A@be-research.ucsd.edu> <20000201085706.B17402@cnri.reston.va.us> Message-ID: <1000201093634.ZM5785@noah.scripps.edu> On Feb 1, 8:57am, Greg Ward wrote: > Subject: Re: [Distutils] Complling on IRIX OS > On 27 January 2000, Curtis Jensen said: > > To install on an IRIX OS, sometimes we need to compile with the -o32 > > flag. I can't seem to find how to do this with distutils. Any ideas? > > Hmmm... I just checked the source, and it looks like there is no way to > do this right now. ;-( Why do you need to? The Distutils default -- in > fact, its *only* behaviour for now -- is to compile extensions the same > way you compiled Python itself. I'm a bit fuzzy on the various 32/64 > options for the IRIX 6+ compiler, but I expect you'd have a hard time > loading .so's compiled one way into an executable compiled another way. > What exactly are you trying to accomplish here -- better performance, > ability to run on more platforms (ie. 32- and 64-bit IRIX), or what? > Greg, sgi has created a real mess. They have 3 ABIs (Application Binary Interfaces): IRIX supports three ABIs: o32 The old 32-bit ABI which was standard on IRIX 5 systems. n64 The 64-bit ABI which was introduced on IRIX 6.0 systems. n32 The new high performance 32-bit ABI which was introduced on IRIX 6.2. Of course objects using a given ABI cannot be mixed. As long as one compiles extensions on an SGI running the same OS as the one the Python interpreter has been compiled on AND one uses default compilation flags everything works fine. Unfortunately this is not the always the case. I compile everything by explicitly specifying -o32 -mips2 in the compilation flags and -o32 in the LDFLAGS. This ensures maximum portability between various SGI configurations. Are you saying that it would difficult to specify at build time that I want these flags to be added to CPFLAGS and LDFLAGS ? or is the problem to find out "automatically" whether the Python interpreter running disutil was built o32 n32 or n64 ? For the second option, one could simple execute a "file" unix-command on the python binary to find out about the ABI. -Michel -- ----------------------------------------------------------------------- >>>>>>>>>> AREA CODE CHANGE <<<<<<<<< we are now 858 !!!!!!! Michel F. Sanner Ph.D. The Scripps Research Institute Assistant Professor Department of Molecular Biology 10550 North Torrey Pines Road Tel. (858) 784-2341 La Jolla, CA 92037 Fax. (858) 784-2860 sanner@scripps.edu http://www.scripps.edu/sanner ----------------------------------------------------------------------- From gward@cnri.reston.va.us Tue Feb 1 18:21:33 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 1 Feb 2000 13:21:33 -0500 Subject: [Distutils] Complling on IRIX OS In-Reply-To: <1000201093634.ZM5785@noah.scripps.edu>; from sanner@scripps.edu on Tue, Feb 01, 2000 at 09:36:34AM -0800 References: <3890E416.D97B745A@be-research.ucsd.edu> <20000201085706.B17402@cnri.reston.va.us> <1000201093634.ZM5785@noah.scripps.edu> Message-ID: <20000201132133.A17723@cnri.reston.va.us> On 01 February 2000, Michel Sanner said: > Greg, sgi has created a real mess. They have 3 ABIs (Application Binary > Interfaces): [...] Thanks for the explanation. > Of course objects using a given ABI cannot be mixed. As long as one > compiles extensions on an SGI running the same OS as the one the > Python interpreter has been compiled on AND one uses default > compilation flags everything works fine. That's what I suspected -- eg. if your python is built -n32, you can't link on extensions built with -n64 or -o32. > Are you saying that it would difficult to specify at build time that I want > these flags to be added to CPFLAGS and LDFLAGS ? Stop thinking in terms of CFLAGS and LDFLAGS -- they no longer exist. There are no makefiles. (We have always been at war with Oceania.) Conceptually, though, yes: it would be quite difficult to augment the compiler/linker flags that Distutils uses when building extensions, because I forgot to provide a "way in" for either module developers or installers to provide this information. That's needed for fairly innocuous cases -- eg. compile this big file with -Olimit=1500 (another annoying SGIism) -- or for people who enjoy playing with fire (try to mix and match -o32 and -n32 between python and an extension). Both of these should be supported, the former because it's necessary, the latter because people should be given the freedom to shoot themselves in the foot if they really insist. Might be nice to add a warning to the effect that "compiler flags differ -- if things blow up, don't blame me" though. > or is the problem to find out "automatically" whether the Python > interpreter running disutil was built o32 n32 or n64 ? For the second > option, one could simple execute a "file" unix-command on the python > binary to find out about the ABI. Eew, that's getting awfully platform-specific. What I prefer to do is just use the compiler flags used to build Python. That'll work most of the time; the only time it'll really fall down is for that big file that requires -Olimit=1500 on IRIX, or some other funky thing on other compilers. Another possibility: hairy code that confuses gcc's optimizer, so you need one of the -f switches to turn off that type of optimization for that file. One could imagine endless perverse examples, but it boils down to this: * the user (either module developer or installer) needs a way to circumvent the lovely platform-neutral compiler abstraction model, and just say, "on platform X, with compiler Y, shove *these* flags in when you run the compiler (and *these* other flags when you run the linker). It's crude, it's ugly, but it's essential. Incidentally, if you think you need this backdoor to (eg.) define a preprocessor macro or add another include directory for a particular C source file, think again: these things *are* handled by the lovely platform-neutral compiler abstraction model, because those are essential attributes of the language itself. Cranking up an optimization buffer, or turning off loop-unrolling, are inextricably linked to a particular compiler (often on a particular platform), so have no place in that abstraction model. Hence the occasional need for a backdoor. Greg From gward@cnri.reston.va.us Tue Feb 1 18:29:16 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 1 Feb 2000 13:29:16 -0500 Subject: Finding VC (Was: Re: [Distutils] WinNT and distutils) In-Reply-To: <01fc01bf6cc1$e57d1110$4500a8c0@thomasnotebook>; from thomas.heller@ion-tof.com on Tue, Feb 01, 2000 at 03:37:41PM +0100 References: <20000201090414.D17402@cnri.reston.va.us> <01fc01bf6cc1$e57d1110$4500a8c0@thomasnotebook> Message-ID: <20000201132915.B17723@cnri.reston.va.us> On 01 February 2000, Thomas Heller said: > I propose the following: > > 1. Convince Guido to give python 1.6 registry access in the core. > I'm currently working on such a module. > > 2. (>=1.6) Use the registry to find Path, Include, and Lib > directories for VC, and supply them on the command line. > > 3. (<1.6) Try to import win32api and win32con, and use Robins > code to find the directories. If this also fails, inform the user > that the environment must be set up for VC by running > vcvars32.bat, and restarting setup.py. > Note that it also may be possible the VC is not installed at all! That sounds like a good plan. > I think we should come to a conclusion here, and offer Greg some tested > patches. > I'm willing to work on this. Excellent! Probably a good place to start is with Robin's latest patch, which looks like it fulfills #3 on your list. Try applying that to 0.1.3 and see if it works for you as well as it did for Robin. Let us know how it goes. Thanks! Greg From vanandel@ucar.edu Tue Feb 1 21:05:19 2000 From: vanandel@ucar.edu (Joe Van Andel) Date: Tue, 01 Feb 2000 14:05:19 -0700 Subject: [Distutils] patch to unixccompiler.py Message-ID: <38974A8F.FA033F79@atd.ucar.edu> I've encountered a situation where I'd like to be able to over-ride the default compiler used by distutils. I have an extension which requires a newer version of gcc than I used to build python. The following patch uses the value of the CC environment variable to override the default. I'd like to see this in the next version of distutils, if possible: % diff -c unixccompiler.orig unixccompiler.py *** unixccompiler.orig Sun Dec 12 09:57:47 1999 --- unixccompiler.py Tue Feb 1 13:58:39 2000 *************** *** 22,27 **** --- 22,32 ---- from copy import copy from sysconfig import \ CC, CCSHARED, CFLAGS, OPT, LDSHARED, LDFLAGS, RANLIB, AR, SO + import os + tcc = os.environ.get('CC') + if tcc: + CC=tcc + from ccompiler import CCompiler, gen_preprocess_options, gen_lib_options from util import move_file, newer_pairwise, newer_group -- Joe VanAndel National Center for Atmospheric Research http://www.atd.ucar.edu/~vanandel/ Internet: vanandel@ucar.edu From mal@lemburg.com Tue Feb 1 17:51:01 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 01 Feb 2000 18:51:01 +0100 Subject: [Distutils] Complling on IRIX OS References: <3890E416.D97B745A@be-research.ucsd.edu> <20000201085706.B17402@cnri.reston.va.us> <1000201093634.ZM5785@noah.scripps.edu> Message-ID: <38971D05.F1CE0B70@lemburg.com> Michel Sanner wrote: > > Greg, sgi has created a real mess. They have 3 ABIs (Application Binary > Interfaces): > > IRIX supports three ABIs: > > o32 The old 32-bit ABI which was standard on IRIX 5 systems. > > n64 The 64-bit ABI which was introduced on IRIX 6.0 systems. > > n32 The new high performance 32-bit ABI which was introduced on IRIX > 6.2. > Just in case this hasn't come up yet: you can check for o32, n64 and n32 using the architecture() API in platform.py (available from my Python Pages). Other goodies in platform.py might also help to set some reasonable default options for other platforms as well. FYI, platform.py provides a platform independent way of querying platform dependent data... (well, look at the code then you'll know what I mean ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From vanandel@ucar.edu Tue Feb 1 22:11:00 2000 From: vanandel@ucar.edu (Joe Van Andel) Date: Tue, 01 Feb 2000 15:11:00 -0700 Subject: [Distutils] patch to build_py.py Message-ID: <389759F4.149FAC76@atd.ucar.edu> If the user mis-types a file name in the py_modules list, the current release gets a runtime error, while trying to print the warning message. The following patch properly produces the error message. *** build_py.orig Sun Dec 12 10:03:59 1999 --- build_py.py Tue Feb 1 15:05:10 2000 *************** *** 159,166 **** def check_module (self, module, module_file): if not os.path.isfile (module_file): ! self.warn ("file %s (for module %s) not found" % ! module_file, module) return 0 else: return 1 --- 159,166 ---- def check_module (self, module, module_file): if not os.path.isfile (module_file): ! self.warn ("file %s (for module %s) not found" % ! (module_file, module)) return 0 else: return 1 -- Joe VanAndel National Center for Atmospheric Research http://www.atd.ucar.edu/~vanandel/ Internet: vanandel@ucar.edu From gward@cnri.reston.va.us Tue Feb 1 23:56:32 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 1 Feb 2000 18:56:32 -0500 Subject: [Distutils] distutils on hpux In-Reply-To: ; from swalton@apollo.sfo.csun.edu on Wed, Jan 26, 2000 at 03:08:40PM -0800 References: Message-ID: <20000201185632.A18193@cnri.reston.va.us> On 26 January 2000, Stephen Walton said: > Using the 0.1.2 release, all the tests fail, basically because "from > sysconfig import AR" fails. Where should that come from? It appears it > should be in the config.h set up when I built Python, but I can't find it. For the record (ie. for the list archive and collective benefit of the SIG), here's how this one was resolved: Stephen rebuilt and reinstalled Python. It turns out that his $exec_prefix/lib/python1.5/config/Makefile was the wrong file; it looks to me as though it was generated from $exec_prefix/lib/python1.5/config/Makefile.pre.in. The only way I can see this happening is if someone runs "make -f Makefile.pre.in" *right in the $exec_prefix/lib/python1.5/config directory* -- which of course they're not supposed to do, but there's nothing stopping them from doing it. Oops. Anyways, a Python reinstall put the proper Makefile into the config directory, and now all is well for Stephen. Hooray. I feel like a Microsoft tech support weenie -- "just reinstall and everything will be Just Fine". Greg From gward@cnri.reston.va.us Wed Feb 2 00:11:14 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 1 Feb 2000 19:11:14 -0500 Subject: [Distutils] patch to unixccompiler.py In-Reply-To: <38974A8F.FA033F79@atd.ucar.edu>; from vanandel@atd.ucar.edu on Tue, Feb 01, 2000 at 02:05:19PM -0700 References: <38974A8F.FA033F79@atd.ucar.edu> Message-ID: <20000201191113.B18193@cnri.reston.va.us> On 01 February 2000, Joe Van Andel said: > I've encountered a situation where I'd like to be able to over-ride the > default compiler used by distutils. I have an extension which requires > a newer version of gcc than I used to build python. > > The following patch uses the value of the CC environment variable to > override the default. I'd like to see this in the next version of > distutils, if possible: Hmm. I'm a bit leery of introducing environment variables into the user interface; they're prone to action-at-a-distance and unintended consequence. However, it would be convenient to be able to adopt the time-honoured idiom of CC=foo_cc make to CC=foo_cc python setup.py build I think the proper way to do it would be to add an option to the build_ext command, so you'd have to do something like this: python setup.py build_ext --compiler=foo_cc (This might also be a good way to override compiler flags -- add a "--compiler-flags" option.) This would require rather more code than Joe's tiny little patch, though. > *** unixccompiler.orig Sun Dec 12 09:57:47 1999 > --- unixccompiler.py Tue Feb 1 13:58:39 2000 > *************** > *** 22,27 **** > --- 22,32 ---- > from copy import copy > from sysconfig import \ > CC, CCSHARED, CFLAGS, OPT, LDSHARED, LDFLAGS, RANLIB, AR, SO > + import os > + tcc = os.environ.get('CC') > + if tcc: > + CC=tcc > + > from ccompiler import CCompiler, gen_preprocess_options, > gen_lib_options > from util import move_file, newer_pairwise, newer_group Anyone have strong feelings one way or the other about accepting this patch and making environment variables a part of the Distutils user interface? Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gward@cnri.reston.va.us Wed Feb 2 00:11:36 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 1 Feb 2000 19:11:36 -0500 Subject: [Distutils] patch to build_py.py In-Reply-To: <389759F4.149FAC76@atd.ucar.edu>; from vanandel@atd.ucar.edu on Tue, Feb 01, 2000 at 03:11:00PM -0700 References: <389759F4.149FAC76@atd.ucar.edu> Message-ID: <20000201191135.C18193@cnri.reston.va.us> On 01 February 2000, Joe Van Andel said: > If the user mis-types a file name in the py_modules list, the current > release gets a runtime error, while trying to print the warning > message. The following patch properly produces the error message. Thanks! This fix is in the CVS tree now. Greg From thomas.heller@ion-tof.com Wed Feb 2 17:18:51 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Wed, 2 Feb 2000 18:18:51 +0100 Subject: [Distutils] WinNT and distutils References: <20000128190216.B12965@cnri.reston.va.us><20000131093428.A17133@cnri.reston.va.us><20000201090005.C17402@cnri.reston.va.us> Message-ID: <02de01bf6da1$93dba1b0$4500a8c0@thomasnotebook> Robin, there are some small typos in your patch, and a more severe problem. > + def _devstudio_versions(): > + "Get a list of devstudio versions" > + try: > + import win32api > + import win32con > + except ImportError: > + return None This should be except ImportError: return () > + def _msvc_get_paths(path, vNum='6.0', platform='x86'): > + "Get a devstudio path (include, lib or path)" > + try: > + import win32api > + import win32con > + except ImportError: > + return None Same as above: except ImportError: return () > + def _find_exe(exe): > + for v in _devstudio_versions(): > + for p in _msvc_get_paths('bin',v): Should be: for p in _msvc_get_paths('path',v): -------------------------------------------------------- The more severe problem is the following: The background is that in my PC I have registry entries for version 5.0 and 6.0, although VC5.0 is no longer installed. Note that I had 5.0 installed in different directories than 6.0! It seems the setup program did not remove the 5.0 entries. Now, what happens in your code is the following: > ! self.cc = _find_exe("cl.exe") > ! self.link = _find_exe("link.exe") find the 6.0 executables (because the 5.0 exes are no longer there). > ! _do_SET('lib') > ! _do_SET('include') > ! path=_find_SET('path') set the 5.0 environment (because _devstudio_versions() returns ['5.0', '6.0']. Now the self.spawn() call launches cl.exe, this displays a Box that some dll is not found, because this is not in the (invalid) path. Conclusion: - Your code may mix up different versions of compiler, linker and path, lib, and include directories. The code should first determine a version which is actually installed, and then set up everything, sticking to this version. - _devstudio_versions() returns version numbers in ascending order, so lower version numbers are preferred over higher version numbers. Should be the other way (although I'm not sure if it is possible to have different versions of VC installed at the same time). Do you want to fix this, or should I prepare another patch? Regards, Thomas Heller From thomas.heller@ion-tof.com Wed Feb 2 18:13:03 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Wed, 2 Feb 2000 19:13:03 +0100 Subject: [Distutils] WinNT and distutils References: <20000128190216.B12965@cnri.reston.va.us><20000131093428.A17133@cnri.reston.va.us><20000201090005.C17402@cnri.reston.va.us> <02de01bf6da1$93dba1b0$4500a8c0@thomasnotebook> Message-ID: <035401bf6da9$25e68c30$4500a8c0@thomasnotebook> [I wrote] > > + try: > > + import win32api > > + import win32con > > + except ImportError: > > + return None > This should be > except ImportError: > return () Sorry, correct is: except ImportError: return [] From robin@jessikat.demon.co.uk Wed Feb 2 18:15:22 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Wed, 2 Feb 2000 18:15:22 +0000 Subject: [Distutils] WinNT and distutils In-Reply-To: <02de01bf6da1$93dba1b0$4500a8c0@thomasnotebook> References: <20000128190216.B12965@cnri.reston.va.us> <20000131093428.A17133@cnri.reston.va.us> <20000201090005.C17402@cnri.reston.va.us> <02de01bf6da1$93dba1b0$4500a8c0@thomasnotebook> Message-ID: In article <02de01bf6da1$93dba1b0$4500a8c0@thomasnotebook>, Thomas Heller writes Agreed about the typos. I had some idea of checking for non-availability using None. It's simpler to return []. As for the other at the bottom of _devstudio_versions try L.reverse() after the L.sort(). I meant to do that originally, but failed. Then the versions should return the versions in latest first order. It's a bit of a pain to have multiple versions. Probably we should get a version just once and pass to the other lookups as required. if multiple versions are available there's always a choice to be made. Probably we should write a warning message into the output and use the latest throughout. Is there an obvious version flag setup.py? >Robin, >there are some small typos in your patch, >and a more severe problem. > >> + def _devstudio_versions(): >> + "Get a list of devstudio versions" >> + try: >> + import win32api >> + import win32con >> + except ImportError: >> + return None >This should be > except ImportError: > return () > >> + def _msvc_get_paths(path, vNum='6.0', platform='x86'): >> + "Get a devstudio path (include, lib or path)" >> + try: >> + import win32api >> + import win32con >> + except ImportError: >> + return None >Same as above: > except ImportError: > return () > >> + def _find_exe(exe): >> + for v in _devstudio_versions(): >> + for p in _msvc_get_paths('bin',v): >Should be: > for p in _msvc_get_paths('path',v): > >-------------------------------------------------------- >The more severe problem is the following: > >The background is that in my PC I have registry entries for >version 5.0 and 6.0, although VC5.0 is no longer installed. >Note that I had 5.0 installed in different directories than 6.0! >It seems the setup program did not remove the 5.0 entries. >Now, what happens in your code is the following: >> ! self.cc = _find_exe("cl.exe") >> ! self.link = _find_exe("link.exe") >find the 6.0 executables (because the 5.0 exes are no longer there). > >> ! _do_SET('lib') >> ! _do_SET('include') >> ! path=_find_SET('path') >set the 5.0 environment (because _devstudio_versions() returns ['5.0', >'6.0']. > >Now the self.spawn() call launches cl.exe, this displays a Box that >some dll is not found, because this is not in the (invalid) path. > >Conclusion: >- Your code may mix up different versions of compiler, linker >and path, lib, and include directories. The code should first >determine a version which is actually installed, and then >set up everything, sticking to this version. > >- _devstudio_versions() returns version numbers in ascending order, >so lower version numbers are preferred over higher version numbers. >Should be the other way (although I'm not sure if it is possible >to have different versions of VC installed at the same time). > >Do you want to fix this, or should I prepare another patch? > >Regards, > >Thomas Heller > > >_______________________________________________ >Distutils-SIG maillist - Distutils-SIG@python.org >http://www.python.org/mailman/listinfo/distutils-sig -- Robin Becker From robin@jessikat.demon.co.uk Wed Feb 2 20:47:32 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Wed, 2 Feb 2000 20:47:32 +0000 Subject: [Distutils] WinNT and distutils In-Reply-To: <035401bf6da9$25e68c30$4500a8c0@thomasnotebook> References: <20000128190216.B12965@cnri.reston.va.us> <20000131093428.A17133@cnri.reston.va.us> <20000201090005.C17402@cnri.reston.va.us> <02de01bf6da1$93dba1b0$4500a8c0@thomasnotebook> <035401bf6da9$25e68c30$4500a8c0@thomasnotebook> Message-ID: In article <035401bf6da9$25e68c30$4500a8c0@thomasnotebook>, Thomas Heller writes >[I wrote] >> > + try: >> > + import win32api >> > + import win32con >> > + except ImportError: >> > + return None >> This should be >> except ImportError: >> return () >Sorry, correct is: > except ImportError: > return [] > > > >_______________________________________________ >Distutils-SIG maillist - Distutils-SIG@python.org >http://www.python.org/mailman/listinfo/distutils-sig try this patch Thomas and see if it's ok with your registry. I put in a DistutilsPlatformError if it can't find any devstudio at all. *** msvccompiler.py.org Mon Jan 17 21:57:18 2000 --- msvccompiler.py Wed Feb 02 20:40:40 2000 *************** *** 10,19 **** --- 10,101 ---- import os import sys + import string from distutils.errors import * from distutils.ccompiler import \ CCompiler, gen_preprocess_options, gen_lib_options + def _devstudio_versions(): + "Get a list of devstudio versions" + try: + import win32api + import win32con + except ImportError: + return [] + + K = 'Software\\Microsoft\\Devstudio' + L = [] + for base in (win32con.HKEY_CLASSES_ROOT,win32con.HKEY_LOCAL_MACHINE,win32con.HKEY_CURRENT_USER,win32con.HKEY_USERS): + try: + k = win32api.RegOpenKeyEx(base,K) + i = 0 + while 1: + try: + p = win32api.RegEnumKey(k,i) + if p[0] in '123456789' and p not in L: + L.append(p) + except win32api.error: + break + i = i + 1 + except win32api.error: + pass + L.sort() + L.reverse() + return L + + def _msvc_get_paths(path, vNum='6.0', platform='x86'): + "Get a devstudio path (include, lib or path)" + try: + import win32api + import win32con + except ImportError: + return [] + + L = [] + if path=='lib': path= 'Library' + path = string.upper(path + ' Dirs') + K = 'Software\\Microsoft\\Devstudio\\%s\\Build System\\Components\\Platforms\\Win32 (%s)\\Directories' \ + % (vNum,platform) + for base in (win32con.HKEY_CLASSES_ROOT,win32con.HKEY_LOCAL_MACHINE,win32con.HKEY_CURRENT_USER,win32con.HKEY_USERS): + try: + k = win32api.RegOpenKeyEx(base,K) + i = 0 + while 1: + try: + (p,v,t) = win32api.RegEnumValue(k,i) + if string.upper(p)==path: + V = string.split(v,';') + for v in V: + if v=='' or v in L: continue + L.append(v) + break + i = i + 1 + except win32api.error: + break + except win32api.error: + pass + return L + + def _find_exe(exe,vNum): + for p in _msvc_get_paths('bin',vNum): + fn=os.path.join(os.path.abspath(p),exe) + if os.path.isfile(fn): return fn + + #didn't find it; try existing path + try: + for p in string.split(os.environ['Path'],';'): + fn=os.path.join(os.path.abspath(p),exe) + if os.path.isfile(fn): return fn + except: + pass + return exe #last desperate hope + + def _find_SET(n,vNum): + return _msvc_get_paths(n,vNum) + + def _do_SET(n,vNum): + p=_find_SET(n,vNum) + if p!=[]: os.environ[n]=string.join(p,';') class MSVCCompiler (CCompiler) : """Concrete class that implements an interface to Microsoft Visual C++, *************** *** 34,41 **** self.add_library ( "python" + sys.version[0] + sys.version[2] ) self.add_library_dir( os.path.join( sys.exec_prefix, 'libs' ) ) ! self.cc = "cl.exe" ! self.link = "link.exe" self.preprocess_options = None self.compile_options = [ '/nologo', '/Ox', '/MD' ] --- 116,136 ---- self.add_library ( "python" + sys.version[0] + sys.version[2] ) self.add_library_dir( os.path.join( sys.exec_prefix, 'libs' ) ) ! vNum = _devstudio_versions() ! if vNum==[]: ! raise DistutilsPlatformError, "Can't seem to find devstudio in the registry" ! vNum = vNum[0] #highest version ! self.cc = _find_exe("cl.exe",vNum) ! self.link = _find_exe("link.exe",vNum) ! _do_SET('lib',vNum) ! _do_SET('include',vNum) ! path=_find_SET('path',vNum) ! try: ! for p in string.split(os.environ['path'],';'): ! path.append(p) ! except KeyError: ! pass ! os.environ['path'] = string.join(path,';') self.preprocess_options = None self.compile_options = [ '/nologo', '/Ox', '/MD' ] -- Robin Becker From vanandel@ucar.edu Wed Feb 2 21:44:40 2000 From: vanandel@ucar.edu (Joe Van Andel) Date: Wed, 02 Feb 2000 14:44:40 -0700 Subject: [Distutils] sample setup.py for SWIG'd extensions in a package. References: <38974A8F.FA033F79@atd.ucar.edu> <20000201191113.B18193@cnri.reston.va.us> Message-ID: <3898A548.6B295F0D@atd.ucar.edu> This is a multi-part message in MIME format. --------------13C833499F5ECA2F221B74E6 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit If you are interested in a contribution to a HOWTO type document, here's a setup.py I wrote for a set of extensions for radar processing. The package has 3 sub-packages: 'a1pp','rec', and 'util', with both .py files and C++ extensions. I use 'SWIG' to generate wrappers for my C++ classes, which generates a Python wrapper, and an extension. I'm definitely not done with this, but I though anyone using SWIG's wrapped classes would like to see how the extensions are named. (It took some experimentation for me to figure it out). -- Joe VanAndel National Center for Atmospheric Research http://www.atd.ucar.edu/~vanandel/ Internet: vanandel@ucar.edu --------------13C833499F5ECA2F221B74E6 Content-Type: text/plain; charset=us-ascii; name="setup.py" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="setup.py" #!/usr/bin/env python # setup script for building Python Environment for Radar Processing # PERP extensions to Python # To use: # * run this to build : # python setup.py build # or this to build and install it # python setup.py install # created 01 Feb 2000 Joseph VanAndel import os from glob import glob from distutils.command import install from distutils.core import setup # the locate of the RDSS source tree. RDSS='/scr/canoe4/rdss' # location of include files, based on RDSS DDU_INC = RDSS + '/spol/ddutils' SPOL_INC = RDSS + '/spol/include' #DDM_LIB = RDSS + '/spol/ddutils/i386/libddm.a' DDM_LIB = RDSS + '/spol/ddutils/i386' setup (name = "Perp", version = "0.2", maintainer = "Joseph VanAndel", maintainer_email = "vanandel@ucar.edu", description = "Python Environment for Radar Processing(PERP)", # cmdclass = {'install': Install}, packages = ['','a1pp','rec','util'], install_path = 'Perp', include_dirs = ['./include',DDU_INC,SPOL_INC], # ext_package = 'Perp', ext_modules = [ ('a1pp.pulsepairc', {'sources': ['a1pp/pulsepair.cc', 'a1pp/pulsepair_wrap.cc'], 'libraries': ['gcc', 'stdc++','pthread'] }, ), ('a1pp.IIR_Filterc', {'sources': ['a1pp/IIR_Filter.cc', 'a1pp/IIR_Filter_wrap.cc'], 'libraries': ['gcc', 'stdc++','pthread'] }, ), ('rec.RadarFeaturec', {'sources': ['rec/RadarFeature.cc', 'rec/RadarFeature_wrap.cc'], 'libraries': ['gcc', 'stdc++','pthread'] }, ), ('util.du_SweepUtilc', {'sources': ['util/du_SweepUtil.cc', 'util/du_SweepUtil_wrap.cc'], 'library_dirs': [DDM_LIB], 'libraries': ['ddm','gcc', 'stdc++','pthread'], # 'extra_preargs': [DDM_LIB] }, ), ] ) --------------13C833499F5ECA2F221B74E6-- From yueqiang@earth.usc.edu Wed Feb 2 22:03:08 2000 From: yueqiang@earth.usc.edu (Yueqiang Huang) Date: Wed, 2 Feb 2000 14:03:08 -0800 (PST) Subject: [Distutils] Re: installation error on UNIX Message-ID: Greg, The hint at the end of your message is enough to solve the problem. I retried under Python 1.5.2, no problem at all. Previously I was using Python 1.5.1. Thanks! yueqiang From mhammond@skippinet.com.au Wed Feb 2 23:00:41 2000 From: mhammond@skippinet.com.au (Mark Hammond) Date: Thu, 3 Feb 2000 10:00:41 +1100 Subject: [Distutils] RE: [Python-Dev] Proposal: Registry access for Python on Windows In-Reply-To: <02a501bf6cc9$f1681620$4500a8c0@thomasnotebook> Message-ID: > I propose to include functions to access the windows registry > into the python 1.6 core. Agreed! > I have thrown together some code which I will post hopefully > tomorrow, but I would like to hear some comments before. > > ---------------------------------------------------------------------- > winreg - registry access module Im not convinced that we need either a) a new implementation, or b) a new API. I would propose that we use the existing registry access functions from win32api - as I didnt code them, Im not suggesting this due to my own ego. The reasons for my beliefs are: * The new API does not add any new value. The only changes I can see are that OpenKey() has had a reserved value dropped and QueryValue() has an extra redundant result. Indeed the new API does not appear a new API at all, so should be clearly stated if indeed it is (and in which case, exactly what the changes are) or be documented as a pythonic wrapping of the existing win32 registry API. Further, there is some useful functionality dropped. * Well tested. There are some obscure rules and code that experience has shown we need. It would be a shame to reinvent those wheels. It has taken a few years to get it "just right", and this was an implementation by people who know their stuff! Why not take the existing code as it stands (which _does_ include an auto-closing key) and add a new .py wrapper? At least Python will have a "complete, but low-level" registry API, and a higher level "easier to use, but maybe not complete" layer implemented in Python. Indeed, Guido has stated that the existing win32api functions would be suitable for the core. However, the issue is the "coding style" - and I admit that if I knew exectly what that meant I would make the change - eg, if it only means renaming the identifiers it would be trivial - however, it would also appear to be excessively anal, so Im sure there is more to it than that :-) And-I-think-I-will-stay-out-of-the-ternary-debate-ly, Mark. From guido@CNRI.Reston.VA.US Wed Feb 2 23:14:50 2000 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 02 Feb 2000 18:14:50 -0500 Subject: [Distutils] Re: [Python-Dev] Proposal: Registry access for Python on Windows In-Reply-To: Your message of "Thu, 03 Feb 2000 10:00:41 +1100." References: Message-ID: <200002022314.SAA24664@eric.cnri.reston.va.us> > Im not convinced that we need either a) a new implementation, or b) a new > API. > > I would propose that we use the existing registry access functions from > win32api - as I didnt code them, Im not suggesting this due to my own ego. > The reasons for my beliefs are: > > * The new API does not add any new value. The only changes I can see are > that OpenKey() has had a reserved value dropped and QueryValue() has an > extra redundant result. Indeed the new API does not appear a new API at > all, so should be clearly stated if indeed it is (and in which case, exactly > what the changes are) or be documented as a pythonic wrapping of the > existing win32 registry API. Further, there is some useful functionality > dropped. > > * Well tested. There are some obscure rules and code that experience has > shown we need. It would be a shame to reinvent those wheels. It has taken > a few years to get it "just right", and this was an implementation by people > who know their stuff! > > Why not take the existing code as it stands (which _does_ include an > auto-closing key) and add a new .py wrapper? At least Python will have a > "complete, but low-level" registry API, and a higher level "easier to use, > but maybe not complete" layer implemented in Python. Sounds good to me. > Indeed, Guido has stated that the existing win32api functions would be > suitable for the core. However, the issue is the "coding style" - and I > admit that if I knew exectly what that meant I would make the change - eg, > if it only means renaming the identifiers it would be trivial - however, it > would also appear to be excessively anal, so Im sure there is more to it > than that :-) I only vaguely recollect what caused me to object against the coding style, but I think it was a lot of little things: // comments, lines longer than 78 chars, unusual indentation style. It was also somehow requiring C++ (everything in the core is plain C). I tried to do a style conversion myself, but found it very painful -- plus there were some dependencies on other files or modules that seemed to require me to pull in a lot of other things besides win32api.c. However, the registry functions are only a tiny piece of win32api -- they could probably be cut out of win32api and moved into something I would tentatively call win32reg (unless that's already an existing name), which could then be cleaned up much easier than all of win32api. I still think that it can be done, but evidently it takes someone besides me and Mark to do it. --Guido van Rossum (home page: http://www.python.org/~guido/) From mhammond@skippinet.com.au Wed Feb 2 23:32:09 2000 From: mhammond@skippinet.com.au (Mark Hammond) Date: Thu, 3 Feb 2000 10:32:09 +1100 Subject: [Distutils] RE: [Python-Dev] Proposal: Registry access for Python on Windows In-Reply-To: <200002022314.SAA24664@eric.cnri.reston.va.us> Message-ID: > I only vaguely recollect what caused me to object against the coding > style, but I think it was a lot of little things: // comments, lines > longer than 78 chars, unusual indentation style. It was also somehow > requiring C++ (everything in the core is plain C). I tried to do a > style conversion myself, but found it very painful -- plus there were > some dependencies on other files or modules that seemed to require me > to pull in a lot of other things besides win32api.c. OK - that is a good enought list for me to get started. If no one else steps up within a week I will make an attempt at a module that meets these requirements.... Mark. From gstein@lyra.org Thu Feb 3 12:32:31 2000 From: gstein@lyra.org (Greg Stein) Date: Thu, 3 Feb 2000 04:32:31 -0800 (PST) Subject: [Distutils] who needs keyword params? (was: sample setup.py ...) In-Reply-To: <3898A548.6B295F0D@atd.ucar.edu> Message-ID: On Wed, 2 Feb 2000, Joe Van Andel wrote: >... example setup ... > > setup (name = "Perp", > version = "0.2", > maintainer = "Joseph VanAndel", > maintainer_email = "vanandel@ucar.edu", > description = "Python Environment for Radar Processing(PERP)", > # cmdclass = {'install': Install}, > packages = ['','a1pp','rec','util'], > install_path = 'Perp', > include_dirs = ['./include',DDU_INC,SPOL_INC], > # ext_package = 'Perp', > ext_modules = [ > ('a1pp.pulsepairc', > {'sources': ['a1pp/pulsepair.cc', > 'a1pp/pulsepair_wrap.cc'], > 'libraries': ['gcc', 'stdc++','pthread'] > }, > ), > ('a1pp.IIR_Filterc', > {'sources': ['a1pp/IIR_Filter.cc', > 'a1pp/IIR_Filter_wrap.cc'], > 'libraries': ['gcc', 'stdc++','pthread'] > }, > ), > ('rec.RadarFeaturec', > {'sources': ['rec/RadarFeature.cc', > 'rec/RadarFeature_wrap.cc'], > 'libraries': ['gcc', 'stdc++','pthread'] > }, > ), > ('util.du_SweepUtilc', > {'sources': ['util/du_SweepUtil.cc', > 'util/du_SweepUtil_wrap.cc'], > 'library_dirs': [DDM_LIB], > 'libraries': ['ddm','gcc', 'stdc++','pthread'], > # 'extra_preargs': [DDM_LIB] > }, > ), > ] > ) I just had a reasonably evil thought. Looking at the above code, I thought, "man. look at all those parameters. gets kind of ugly. it would be nice to not have to pass all those, but just define them at the top level." Of course, that's when the anti-Pythonic demon possessed me, and I figured that we could do something like the following (untested) code: def _internal_setup(...): # rename the old setup function to this ... def _get_caller_globals(): try: raise 'hi' except: return tb.tb_frame.f_back.f_back def setup(**kw): new_kw = { } frame = _get_caller_frame() code = _internal_setup.func_code for argname in code.co_varnames[:code.co_argcount]: new_kw[argname] = kw.get(argname) or \ frame.f_locals.get(argname) or \ frame.f_globals.get(argname) return apply(_internal_setup, (), new_kw) Now that is fun! :-) The only caveat is that any arguments passed to setup() must be keyword arguments -- it does not define any positional arguments (unlike the current setup function). Of course, this could be its "specification" and/or an extension of the above code could fix that. While this is generally Badness for most people, it could very well simplify things for people wanting to whip up a new setup.py. Cheers, -g -- Greg Stein, http://www.lyra.org/ From robin@jessikat.demon.co.uk Thu Feb 3 12:49:52 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Thu, 3 Feb 2000 12:49:52 +0000 Subject: [Distutils] RE: [Python-Dev] Proposal: Registry access for Python on Windows In-Reply-To: References: <200002022314.SAA24664@eric.cnri.reston.va.us> Message-ID: In article , Mark Hammond writes >> I only vaguely recollect what caused me to object against the coding >> style, but I think it was a lot of little things: // comments, lines >> longer than 78 chars, unusual indentation style. It was also somehow >> requiring C++ (everything in the core is plain C). I tried to do a >> style conversion myself, but found it very painful -- plus there were >> some dependencies on other files or modules that seemed to require me >> to pull in a lot of other things besides win32api.c. > >OK - that is a good enought list for me to get started. If no one else >steps up within a week I will make an attempt at a module that meets these >requirements.... > >Mark. > > >_______________________________________________ >Distutils-SIG maillist - Distutils-SIG@python.org >http://www.python.org/mailman/listinfo/distutils-sig I'm having a quick scan at this. It seems to go quite deeply into PyWinHandles, PyWinObjects etc etc. I question the wisdom of trying to rip out a chunk from win32api.cpp and making it exactly compatible with the remainder of Mark+others' stuff. Leaving two very similar registry api's seems a recipe for disaster&confusion. Another solution is to split win32api in two (win32api' + win32reg). That means duplicating all the bits like handle open/close unicode string handling etc etc. This looks just about doable, but will lead to maintenance problems and will no doubt interact with unicode features of 1.6. I see that win32api.pyd is 57kB which is smaller than bsddb.pyd. Including all of win32api will probably cause even more GvR style issues. -- Robin Becker From thomas.heller@ion-tof.com Thu Feb 3 13:26:39 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Thu, 3 Feb 2000 14:26:39 +0100 Subject: [Distutils] Re: [Python-Dev] Proposal: Registry access for Python on Windows References: Message-ID: <018b01bf6e4a$4db27700$4500a8c0@thomasnotebook> [Thomas] > > I propose to include functions to access the windows registry > > into the python 1.6 core. [Mark] > > Agreed! Fine. Thats the _main_ purpose of the proposal. > > winreg - registry access module > > Im not convinced that we need either a) a new implementation, or b) a new > API. > > I would propose that we use the existing registry access functions from > win32api - as I didnt code them, Im not suggesting this due to my own ego. > The reasons for my beliefs are: > > * The new API does not add any new value. The only changes I can see are > that OpenKey() has had a reserved value dropped and QueryValue() has an > extra redundant result. The redundant result value was a typo in the proposal. > Indeed the new API does not appear a new API at > all, so should be clearly stated if indeed it is (and in which case, exactly > what the changes are) or be documented as a pythonic wrapping of the > existing win32 registry API. Further, there is some useful functionality > dropped. I posted my proposal to start the discussion - so far it has been successfull - not as the final specification for the api. I admit: it has been too low level. I have posted (and implemented) a new proposal describing a more high level interface. In this I followed Gordon's suggestion: Provide the minimum needed. If one wants to do more special things, one probably needs your win32 extensions anyway. > > * Well tested. There are some obscure rules and code that experience has > shown we need. It would be a shame to reinvent those wheels. Already too late! (Programming is fun :-) (Is competition good ?) > It has taken > a few years to get it "just right", and this was an implementation by people > who know their stuff! The only obscure code I can see (having looked at win32api source AFTER writing my code) is the handling of REG_MULTI_SZ. These are returned as one normal string with embedded NULL bytes by my implementation. BTW: This is also how regedit.exe (but not regedt32) handles these. > > Why not take the existing code as it stands (which _does_ include an > auto-closing key) and add a new .py wrapper? At least Python will have a > "complete, but low-level" registry API, and a higher level "easier to use, > but maybe not complete" layer implemented in Python. > > Indeed, Guido has stated that the existing win32api functions would be > suitable for the core. Don't you think that the raw Win32 api functions are much too low level to belong into core python? How should they be documented? Pointers to msdn.microsoft.com? > However, the issue is the "coding style" - and I > admit that if I knew exectly what that meant I would make the change - eg, > if it only means renaming the identifiers it would be trivial - however, it > would also appear to be excessively anal, so Im sure there is more to it > than that :-) It seems (but I may be wrong) that my coding style is nearer to Guido's than that of win32api. Regards, Thomas From thomas.heller@ion-tof.com Thu Feb 3 13:35:36 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Thu, 3 Feb 2000 14:35:36 +0100 Subject: [Distutils] Revised proposal (and preliminary implementation): Registry access for Python on Windows Message-ID: <01fc01bf6e4b$8db42820$4500a8c0@thomasnotebook> Ok, at least the first proposal did start the discussion. Here is a revised one: A preliminary implementation is available at http://starship.python.net/crew/theller/ ---------------------------------------------------------------------- winreg - windows registry access module Exception: error - raised when a function fails. Will contain a windows error code and a textual description. Objects: regnode object - represents a open key in the registry. Functions: OpenKey (name) -> regnode object Opens an existing key with the specified access rights and returns a regnode object. name is specified like "HKLM\Software\Python" or "HKEY_LOCAL_MACHINE\Software\Python" CreateKey (name) -> regnode object Creates a new key or opens an existing one and returns a regnode object. For the name format see OpenKey regnode object methods: Values () -> dict Returns a dictionary mapping names to values. The or unnamed value has the key ''. The values are either strings or integers, depending on the REG_* type. GetValue ([name]) -> integer or string Returns a value specified by name or the default value. SetValue ([name,] value) Set a named or the value. Named values must be integers or string (which are stored as REG_DWORD or REG_SZ). Should an optional third parameter be used, allowing to store in other REG_* typecodes? I dont think so. DeleteValue ([name]) Deletes a named or the value. SubKeys () -> sequence Returns a sequence containing the names of all subkeys. DeleteKey (name [,recursive=0]) If recursive is 0, deletes the named key if no subkeys exist. If there are subkeys an error is raised. If recursive is not 0, the named key is deleted including subkeys. OpenKey (name) -> regnode object Openes an existing subkey and returns a regnode object pointing to it. CreateKey (name) -> regnode object Creates a new or openes an existing subkey and returns a regnode object pointing to it. regnode objects have the following properties: name - the name of the RegistryKey, something like "HKLM\Software\Python" hkey - the integer keyhandle ---------------------------------------------------------------------- Thomas Heller From thomas.heller@ion-tof.com Thu Feb 3 15:10:11 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Thu, 3 Feb 2000 16:10:11 +0100 Subject: [Distutils] WinNT and distutils References: <20000128190216.B12965@cnri.reston.va.us><20000131093428.A17133@cnri.reston.va.us><20000201090005.C17402@cnri.reston.va.us><02de01bf6da1$93dba1b0$4500a8c0@thomasnotebook><035401bf6da9$25e68c30$4500a8c0@thomasnotebook> Message-ID: <030701bf6e58$c4c670e0$4500a8c0@thomasnotebook> This is a multi-part message in MIME format. ------=_NextPart_000_0304_01BF6E61.260BE4E0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Robin, I have extended your patch a little. If it works for you, we should submit it to Greg. The attached file (I have not mastered diff on NT yet) does the following: 1. Try to find cl.exe and link.exe in the path. If this succeeds, it assumes everything is set up and starts the build. User in control! 2. Then it reads the versions from the registry, starting with the highest one until it finds cl and link. For debugging, it also prints the version number it uses. 3. If this also fails, DistutilError is raised. Thomas ------=_NextPart_000_0304_01BF6E61.260BE4E0 Content-Type: application/x-zip-compressed; name="msvccompiler.zip" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="msvccompiler.zip" UEsDBBQAAAAIADh/QyiA+PZOCAwAANIsAAAPAAAAbXN2Y2NvbXBpbGVyLnB57Rprb9tG8rt+xZRp QKpmaTu9tokPQpE6ThtEiYPYSXuwDYIiVxJjaklwKdn69zezD5JLUn4URXDAdYEkFGd2dt6PZRzH SVJRras0E0Ec56sizVg5Gh3nvIpSLuDd2efjY/3ah4hDuioytmIIrtKcQz6HaskgmomqjOIKamSI s0iI0TwvJcK7NC5zkc8r+JyKdZTBWbVO0jxwHGc0Gj2BuGRRxRI4fPHixf7B8/3DFz58YGW5RcQ8 yxBjFIZlLNIkDGECzrdvkiNYiU3Nc1Bs/Q0cBs/h2cHBwf7B4f7hz/Ds8OjHn4/wYXETlQmc3Bbw LZ6HMuRlBbkwT2LbPFZlyhejeZmvoFENcpKXAjTOd11wzYXBuBwBrZbqFoyHRcmKMo+ZEGFekPqE ep+lM/NiNErYHMKEbYRUULhhpSCANz6SNJ3fWAURZHg2Kb9GBIPoSLSq3Cp8Wpqpm5T/8Cwq0sH3 cc7le3Ybs6KCNxJ0QmI3dEpWrUsOF1cj+eotGsI9Q5uictnlZW3iy8tXhitXIk4RETfRI/nDLBIM Ug6eOTj4/e3Jf8Lj6cuzs5Oz8OPp6blvg6anxy+n4buXx7+/eX/SgR1/+vjx5P15+Ons5GMHRK/O xg3/llJoXSNjRivBR7Y4LRh/y7Yntx7x6L8dW9gpYh9Yb26WaFw4tInS6p1kVtE58YSvV3iid+2n 48EN6RyKi4Mr0pd7+OyHf/3408/PX7gYiQnS4nlFgOnwWbSmQVSgUIlX9MlrU9fsMNva7TXD8Lzu QUghKezBYQ15AMmC0gI9TAOBLuaN9Y+SkQcz/Vu72lTHA0V6uGBVWETVUnj0tw+b9+vVxP0pOHB9 KLKoQt/C37fPf3LtWGlihPaBl/I4WyfMxyCaAfojvR1/vbBpRQMZFw+fTFxkxT1SP8CdprMyKrcq eCTPE52XgjVas5Tyo95deJWWwh0/NBovL5+Ky8tf12mWwNlWVGx1eUkZKueYzxHyQSsRH/8g4cB7 Ksa4Py1ZXOVlyoRrMltrPQWPLOEbE4z/CXRcXuFv/Go8EO6fo2zN7gx429bjyYTsvTvGPzfuIYos rbyN7/7bHaZOi0yzIbt83k2TFnKyQdd0weBPjwBVXqV8ze7cWCedzW4mhjMKrX5WMesvJ6zHpKVO 5pmnPAnZLfPwj09+rt2MlFiQUrq5yaV/3DaqROeTXAQECr7kKffMD2yc6F+0so8HNOpC3RuUVMzR +bw5Hx8Z5ua8zSpu7DAbpjxUuY6Idhi2XAUPYXyTljm/cD8Q41fSd/5uxi1VP0SIs5Nzj7eVqNG6 2tY4pm3K+xuLSY/iSDNafDO5uDqClg741USrRwpbqEAayV7W6oXBqx/HoKuN42DbjI1sxVTzi41v VDUNs5DtM69YOY9iBlXeb4qP9/Z8o6pIYOFCzrEpnm1lD90cXrfb8hzVRNMe04aG1bZgVA9IW66C SQWhY6QVdtCeYNm8Pqq1sAbPcsEmB0PApNyG5ZoPA9G/YtyHWq+BNcNB52BzjG9I+mr7uNn7BP78 8084X6bYdaPmgEei2qIQlFcYj7ekvyhJAGcQka9YtUSjQY5qKm9SwVpksOmucGecsYgHAKt8gzWp ou0zKoQYLhWskWZJ5uE4ABQsTudpjBmhrIIWIbHEwppQ04DFGWshcIY/kU6W8muIFjQvVVBsq2XO D38MEK/eTEIHyG2oNm/BA0chOpjocPoIdPdOnZ794tkVjHfSCZO09MCKT7kZgymmaWOe3vpAvQUW 7nGLzsimGMfoKwPJw4kzIuV0GJDyDm8gkN6yOyMoJ8BTx7KNHQQSoU7aKDAuK3Be52gtGYuUzCgu VDeIgbBekRPQKx3PFHbSfTIsB8kW2UdbF05NlmWC2YeYKYrkGxrCLGSUjDpwA+1XojJCX8QmTc+J pr+S/aEPznHEXXQ4xlbkRqTOVq/aEk6qKY44HSZYVMZLCSrZAimXW8c6V1Z3zHJEYDdnxu7QNmRt 8Faa7G3qWr9l9B3b/i4HMKvfOvTtSOvv0j5HjerfwyrXnvlJkPtJz/ysFA9PhYP9ManF2mAKlWz7 B7RWw/WoMoQjZ4WmtLXbDgtvsEO9vxkohpqB9lJdwOBwqdstarCHO7S62TKrf3LT06pKTDGuirHZ I92kf6WCG9/jQNPJcKoQtXAuwN3neZYvcpwe3f3TW/nPu1cuXHWOyJJ5Fi1EqPM/bnX3X02nEl9T uNqxgy7JYvswpC6Rn8D338MfeXmNhYfKV54IfPO4pQl5qzUWnhlrWg3VM4j1TPYHTGithTFVO8ZF o4MgJoEVsCgGwDHJGSCsVkyYz77Iqon9RYDPakyVhcC8xWf9VilN3nAZYJJlBij1YwEpIIyCqAGg oHPzdVWsKyp2LlXlCPVFV23YGSwwNpsGyehRBmsqNBmpg28A3sAX0hNSovpPODkpDScagTFNleIm xWhvdVk80SQ+8fS2dZc3QxLUdKCb4WSIfUhcYfeWbeGa5zei1cwhxSzTNJbRRjZ9q+haNiA3aPsg UA2G/IsaNO2nO/szka+xTRIDkEZHE3L/AYxVRN3mLqjONERgJw4aqYyor4jKxX1IORrXYLV7QiwG ihGyHwHt7KBh9Q2J3tLmbnijhTGpL1tooY+yuHqNeu1CaPgP0e0xLRDIvtsYvrHV5UnzuacZHmqI +0vutBjds/geD3Nmsqy7H7em+vqBcrkoY5JPZnTlIrZ25C2HijFTbWXKx1ee3muncNSZJDhRlzh7 4FCoOyMLCQ1DNOlQksxKLwM1B+mdFsSCs38e2xWUZT1aVja6h1rRZYxjLBAUwQZxzyjJllOGjSH0 Onfw3Z6R3iYaxyF5NBWmoZKyN3A1ppblZbvRLmpe/FqAK1usuR2Cfa1oHi+ODqiAWsg7KOk43Ukq kEZI6CahjT+2dSM1IorohoN3oZs7mmQ0kZ5vmXg0rl37W4OpB/4Wcl05VWFQk+E19W35HNZ4Koap zOGoSJp6+HU4SzmOSd6Y7rD0ZgWgCWldRbOMKWBriMS6jL9X0RaLg0Ju6hQiazICT8kS2f+zW/oY g6M+1gBfdrQ3TNUamfM7nPwiCVCy79DePZTrpVRxV67RRQCJcSyS9yPeUS3MqsfdhyFu7ywgZj2o kHSQ7ygozUg+WBoacLestHm+a++2VVYMpH7QX+8I3PmYV48y5vy9hpdHVAyLhz2LpYdRcZ4KIuLg 0LM/ffPrh5fnvx/hZNKK4Syxclungd1rZBxOX9ozEYq98emn8yOXkqjyMcq/5I2W4u9IY5qVO1PY PelLk7g/dbXTFra7zXxLucvoRKWk5hJNhW3d1/4Ttnci7wpblcV1FsV8KSKe0jXdksXXspk3V2Wk kl/QhosAk21ru8C2fslapaW2njGOUjx49xvAItGYViqGzvdsC421S9BfXY8wh97jFPAQvwDoBtGD kB/gHvAID4HHOQk80k9gl6sYqOM4U3nDinMXj5dU7EW1ns9xmFowuvClqUr9/xHE0VO6Um77VFJh APBujSSy9Lqu7bW1sTb75uqCHFBps0uCrEClgt1i/xynVUZzdoHPeGgk6lnVoLryct4Q+J+vVV+z Ku2ujQ8oTMrK/++FqXWV9Nr45irii4z62EfdKrW+EcVLpMBaw5cnz/dr9xc+cHZzYuDQilWl71q1 3Xm7M6rWaA8fVuvt9kjRqyZVmRaYKebYjm9Vg04qIVq/WBu7DNcDiR57bUEbSazppNneqLGnCf3l S43nzXs70X1UZFX9U//DSxeUhlCclyUTRc4T+bUjBxbFy7ZQ+gsWxoc6rN5r5SItgZ62B2zeYdTX qOb2b9zIapW+escOiXcKbGXuJtf2xI2+nqyd28uWyCbddYU1HcJOIVVHbZqb3VIi8rCc7b5oSEhs 9uVXB8/wUgtj37b27feXZWp/GP3KMvUM1LMQFiRd44w8+KYlizminozokyyi9CnZVPBtn4pVGRs9 Em6LNfnRpo3V4kwYylbSRAAlTIL3ar4kMLG+AoOHmP5d3PQ+ZJrtMlcKiUQ7Bj7+aFE1QsNn/yPc EzjNC0FiJfSpTX5iU1fv32Fa/o6ym0siudYuTV9+xBkpMgMKG8v/Ndz7fxmj/wJQSwECFAAUAAAA CAA4f0MogPj2TggMAADSLAAADwAAAAAAAAABACAAtoEAAAAAbXN2Y2NvbXBpbGVyLnB5UEsFBgAA AAABAAEAPQAAADUMAAAAAA== ------=_NextPart_000_0304_01BF6E61.260BE4E0-- From robin@jessikat.demon.co.uk Thu Feb 3 17:32:14 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Thu, 3 Feb 2000 17:32:14 +0000 Subject: [Distutils] WinNT and distutils In-Reply-To: <030701bf6e58$c4c670e0$4500a8c0@thomasnotebook> References: <20000128190216.B12965@cnri.reston.va.us> <20000131093428.A17133@cnri.reston.va.us> <20000201090005.C17402@cnri.reston.va.us> <02de01bf6da1$93dba1b0$4500a8c0@thomasnotebook> <035401bf6da9$25e68c30$4500a8c0@thomasnotebook> <030701bf6e58$c4c670e0$4500a8c0@thomasnotebook> Message-ID: <+PceUDAeubm4EwfE@jessikat.demon.co.uk> In article <030701bf6e58$c4c670e0$4500a8c0@thomasnotebook>, Thomas Heller writes >Robin, > >I have extended your patch a little. >If it works for you, we should submit it to Greg. > >The attached file (I have not mastered diff on NT yet) >does the following: > >1. Try to find cl.exe and link.exe in the path. If this >succeeds, it assumes everything is set up and starts the build. >User in control! > >2. Then it reads the versions from the registry, >starting with the highest one until it finds cl and link. >For debugging, it also prints the version number it uses. > >3. If this also fails, DistutilError is raised. > >Thomas > >[ A MIME application / x-zip-compressed part was included here. ] > I've no problem with your version except that it may fail even though the path is set up. If cl.exe & link.exe are on the path and properly installed then they should findable via the registry. Even so Lib & Include may not be set up and the compile will fail. Searching the registry first won't fail provided it finds the best version and the Lib/Include dirs haven't been hacked to death inside devstudio. Both search orders will fail under particular circumstances. Perhaps it should try both in the manner of autoconf by attempting a trial compile. I'm not serious really. I feel my original search has more chance of success if things are 'properly' installed. -- Robin Becker From billtut@microsoft.com Thu Feb 3 22:34:20 2000 From: billtut@microsoft.com (Bill Tutt) Date: Thu, 3 Feb 2000 14:34:20 -0800 Subject: [Distutils] Re: High level Registry API Message-ID: <4D0A23B3F74DD111ACCD00805F31D8101D8BCCBB@RED-MSG-50> With respect to DeleteKey() allowing recursive deletes. Either require using SHDeleteKey() to do it correctly, or don't allow recursive deletes at all. The problem being is that when deleting registry keys recursively on NT the entire operation should completly succeed or completly fail. If you naively recursively delete using RegDeleteKey() you can end up with a partially completed delete operation which is completly annoying to go about fixing. (bad, bad, bad) Bill From thomas.heller@ion-tof.com Fri Feb 4 11:59:59 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 4 Feb 2000 12:59:59 +0100 Subject: [Distutils] Re: High level Registry API References: <4D0A23B3F74DD111ACCD00805F31D8101D8BCCBB@RED-MSG-50> Message-ID: <01bd01bf6f07$5d432940$4500a8c0@thomasnotebook> > With respect to DeleteKey() allowing recursive deletes. > Either require using SHDeleteKey() to do it correctly, or don't allow > recursive deletes at all. > The problem being is that when deleting registry keys recursively on NT the > entire operation should completly succeed or completly fail. > If you naively recursively delete using RegDeleteKey() you can end up with a > partially completed delete operation which is completly annoying to go about > fixing. > (bad, bad, bad) > > Bill This is a usefull hint. BTW: win32com has the same problem, see win32com\server\register.py recurse_delete_key() Thanks, Thomas From robin@jessikat.demon.co.uk Fri Feb 4 12:05:19 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Fri, 4 Feb 2000 12:05:19 +0000 Subject: [Distutils] Re: High level Registry API In-Reply-To: <01bd01bf6f07$5d432940$4500a8c0@thomasnotebook> References: <4D0A23B3F74DD111ACCD00805F31D8101D8BCCBB@RED-MSG-50> <01bd01bf6f07$5d432940$4500a8c0@thomasnotebook> Message-ID: In article <01bd01bf6f07$5d432940$4500a8c0@thomasnotebook>, Thomas Heller writes >> With respect to DeleteKey() allowing recursive deletes. >> Either require using SHDeleteKey() to do it correctly, or don't allow >> recursive deletes at all. >> The problem being is that when deleting registry keys recursively on NT >the >> entire operation should completly succeed or completly fail. >> If you naively recursively delete using RegDeleteKey() you can end up with >a >> partially completed delete operation which is completly annoying to go >about >> fixing. >> (bad, bad, bad) >> >> Bill >This is a usefull hint. > >BTW: win32com has the same problem, >see win32com\server\register.py recurse_delete_key() > >Thanks, > >Thomas ... As a matter of interest is the Win32 registry single writer? -- Robin Becker From thomas.heller@ion-tof.com Fri Feb 4 15:27:46 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 4 Feb 2000 16:27:46 +0100 Subject: [Distutils] WinNT and distutils References: <20000128190216.B12965@cnri.reston.va.us><20000131093428.A17133@cnri.reston.va.us><20000201090005.C17402@cnri.reston.va.us><02de01bf6da1$93dba1b0$4500a8c0@thomasnotebook><035401bf6da9$25e68c30$4500a8c0@thomasnotebook><030701bf6e58$c4c670e0$4500a8c0@thomasnotebook> <+PceUDAeubm4EwfE@jessikat.demon.co.uk> Message-ID: <025901bf6f24$64ede000$4500a8c0@thomasnotebook> > In article <030701bf6e58$c4c670e0$4500a8c0@thomasnotebook>, Thomas > Heller writes > >Robin, > > > >I have extended your patch a little. > >If it works for you, we should submit it to Greg. > > > >The attached file (I have not mastered diff on NT yet) > >does the following: > > > >1. Try to find cl.exe and link.exe in the path. If this > >succeeds, it assumes everything is set up and starts the build. > >User in control! > > > >2. Then it reads the versions from the registry, > >starting with the highest one until it finds cl and link. > >For debugging, it also prints the version number it uses. > > > >3. If this also fails, DistutilError is raised. > > > >Thomas > > > >[ A MIME application / x-zip-compressed part was included here. ] > > > > I've no problem with your version except that it may fail even though > the path is set up. If cl.exe & link.exe are on the path and properly > installed then they should findable via the registry. Even so Lib & > Include may not be set up and the compile will fail. > > Searching the registry first won't fail provided it finds the best > version and the Lib/Include dirs haven't been hacked to death inside > devstudio. > Ok, so we should use this. One point only: + def _find_exe(exe,vNum): + for p in _msvc_get_paths('bin',vNum): This should be: + for p in _msvc_get_paths('path',vNum): because there are no 'bin Dirs' entries in the registry. Thomas From thomas.heller@ion-tof.com Fri Feb 4 15:37:52 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 4 Feb 2000 16:37:52 +0100 Subject: [Distutils] Compiling / Installing extensions Message-ID: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> It seems, with Robins latest patch compiling extensions under windows goes smoothly. Some points are still nonstandard: 1. Usually extensions are named xxx.pyd, not xxx.dll Should this be changed? 2. install copies all files built: not only the dll but also .exp, .lib, everthing which is output by the linker. Is this also the case on unix? Or is my setup.py wrong? 3. It would really be nice if there were a possibility to compile debug versions. Thomas Heller From fdrake@acm.org Fri Feb 4 17:29:02 2000 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 4 Feb 2000 12:29:02 -0500 (EST) Subject: [Distutils] Compiling / Installing extensions In-Reply-To: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> Message-ID: <14491.3166.372627.683278@weyr.cnri.reston.va.us> Thomas Heller writes: > 1. Usually extensions are named xxx.pyd, not xxx.dll > Should this be changed? The PYD extension is good because it indicates that it isn't used as a "normal" DLL; this is useful because it makes file management easier. > 2. install copies all files built: not only the dll but also > .exp, .lib, everthing which is output by the linker. > Is this also the case on unix? > Or is my setup.py wrong? On Unix, linking doesn't typically leave a bunch of extra stuff. I don't even know what an EXP file is! ;) > 3. It would really be nice if there were a possibility to > compile debug versions. Agreed, but I'm just a Unix weenie. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From thomas.heller@ion-tof.com Fri Feb 4 18:19:40 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 4 Feb 2000 19:19:40 +0100 Subject: [Distutils] Compiling / Installing extensions References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> <14491.3166.372627.683278@weyr.cnri.reston.va.us> Message-ID: <034a01bf6f3c$671fee50$4500a8c0@thomasnotebook> > Thomas Heller writes: > > 1. Usually extensions are named xxx.pyd, not xxx.dll > > Should this be changed? > > The PYD extension is good because it indicates that it isn't used as > a "normal" DLL; this is useful because it makes file management > easier. It also makes possible (I think) to wrap the python interface for thirdparty.dll into thirdparty.pyd. So it should be changed. Greg, what do you think? > > 2. install copies all files built: not only the dll but also > > .exp, .lib, everthing which is output by the linker. > > Is this also the case on unix? > > Or is my setup.py wrong? > > On Unix, linking doesn't typically leave a bunch of extra stuff. On windows, it's usually not even possible to suppress this extra stuff. Depending on the linker/compiler flags, even more files are generated. It seems distutils should only copy the target files, not this extra stuff. Greg ? > I > don't even know what an EXP file is! ;) Same for the typical windows programmer ;) > > 3. It would really be nice if there were a possibility to > > compile debug versions. > > Agreed, but I'm just a Unix weenie. What do you mean? You don't need to debug under Unix? Is the debug version the default? Or what? Thomas From fdrake@acm.org Fri Feb 4 18:25:19 2000 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 4 Feb 2000 13:25:19 -0500 (EST) Subject: [Distutils] Compiling / Installing extensions In-Reply-To: <034a01bf6f3c$671fee50$4500a8c0@thomasnotebook> References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> <14491.3166.372627.683278@weyr.cnri.reston.va.us> <034a01bf6f3c$671fee50$4500a8c0@thomasnotebook> Message-ID: <14491.6543.878701.914328@weyr.cnri.reston.va.us> I said: > Agreed, but I'm just a Unix weenie. Thomas Heller writes: > What do you mean? You don't need to debug under Unix? > Is the debug version the default? Or what? Thomas, I meant that I write perfect code on the first pass, every time, of course. Unless it's in an email. ;) I was just saying that there should be some way for distutils to provide an option to do that. It should be there for all supported platforms, not just Windows. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From thomas.heller@ion-tof.com Fri Feb 4 18:36:58 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 4 Feb 2000 19:36:58 +0100 Subject: [Distutils] Compiling / Installing extensions References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook><14491.3166.372627.683278@weyr.cnri.reston.va.us><034a01bf6f3c$671fee50$4500a8c0@thomasnotebook> <14491.6543.878701.914328@weyr.cnri.reston.va.us> Message-ID: <037f01bf6f3e$d1fbd160$4500a8c0@thomasnotebook> > I was just saying that there should be some way for distutils to > provide an option to do that. It should be there for all supported > platforms, not just Windows. Yes. Thomas From robin@jessikat.demon.co.uk Fri Feb 4 16:26:22 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Fri, 4 Feb 2000 16:26:22 +0000 Subject: [Distutils] Compiling / Installing extensions In-Reply-To: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> Message-ID: In article <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook>, Thomas Heller writes >It seems, with Robins latest patch compiling extensions >under windows goes smoothly. > >Some points are still nonstandard: > >1. Usually extensions are named xxx.pyd, not xxx.dll >Should this be changed? > >2. install copies all files built: not only the dll but also >.exp, .lib, everthing which is output by the linker. >Is this also the case on unix? >Or is my setup.py wrong? > >3. It would really be nice if there were a possibility to >compile debug versions. > >Thomas Heller > > >_______________________________________________ >Distutils-SIG maillist - Distutils-SIG@python.org >http://www.python.org/mailman/listinfo/distutils-sig Python picks up .pyd's or .dll's for extensions. We could just change _shared_lib_ext to '.pyd'. -- Robin Becker From robin@jessikat.demon.co.uk Fri Feb 4 16:18:46 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Fri, 4 Feb 2000 16:18:46 +0000 Subject: [Distutils] WinNT and distutils In-Reply-To: <025901bf6f24$64ede000$4500a8c0@thomasnotebook> References: <20000128190216.B12965@cnri.reston.va.us> <20000131093428.A17133@cnri.reston.va.us> <20000201090005.C17402@cnri.reston.va.us> <02de01bf6da1$93dba1b0$4500a8c0@thomasnotebook> <035401bf6da9$25e68c30$4500a8c0@thomasnotebook> <030701bf6e58$c4c670e0$4500a8c0@thomasnotebook> <+PceUDAeubm4EwfE@jessikat.demon.co.uk> <025901bf6f24$64ede000$4500a8c0@thomasnotebook> Message-ID: <87ye5MAmvvm4Ew9l@jessikat.demon.co.uk> In article <025901bf6f24$64ede000$4500a8c0@thomasnotebook>, Thomas Heller writes >> In article <030701bf6e58$c4c670e0$4500a8c0@thomasnotebook>, Thomas >> Heller writes >> >Robin, >> > >> >I have extended your patch a little. >> >If it works for you, we should submit it to Greg. >> > >> >The attached file (I have not mastered diff on NT yet) >> >does the following: >> > >> >1. Try to find cl.exe and link.exe in the path. If this >> >succeeds, it assumes everything is set up and starts the build. >> >User in control! >> > >> >2. Then it reads the versions from the registry, >> >starting with the highest one until it finds cl and link. >> >For debugging, it also prints the version number it uses. >> > >> >3. If this also fails, DistutilError is raised. >> > >> >Thomas >> > >> >[ A MIME application / x-zip-compressed part was included here. ] >> > >> >> I've no problem with your version except that it may fail even though >> the path is set up. If cl.exe & link.exe are on the path and properly >> installed then they should findable via the registry. Even so Lib & >> Include may not be set up and the compile will fail. >> >> Searching the registry first won't fail provided it finds the best >> version and the Lib/Include dirs haven't been hacked to death inside >> devstudio. >> >Ok, so we should use this. >One point only: >+ def _find_exe(exe,vNum): >+ for p in _msvc_get_paths('bin',vNum): >This should be: >+ for p in _msvc_get_paths('path',vNum): >because there are no 'bin Dirs' entries in the registry. > >Thomas ... Mea Culpa. I put a check in to map 'bin' to 'Path' and 'lib' to 'Library' and the last patch has it missing. -- Robin Becker From thomas.heller@ion-tof.com Fri Feb 4 19:36:50 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 4 Feb 2000 20:36:50 +0100 Subject: [Distutils] Compiling / Installing extensions References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> Message-ID: <03be01bf6f47$2ec3ac80$4500a8c0@thomasnotebook> > Python picks up .pyd's or .dll's for extensions. We could just change > _shared_lib_ext to '.pyd'. But this does change nothing. Only after I changed g['SO'] = '.dll' to g['SO'] = '.pyd' in sysconfig.py: _init_nt() Have to look after this. Thomas From robin@jessikat.demon.co.uk Fri Feb 4 21:05:33 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Fri, 4 Feb 2000 21:05:33 +0000 Subject: [Distutils] Compiling / Installing extensions In-Reply-To: <03be01bf6f47$2ec3ac80$4500a8c0@thomasnotebook> References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> <03be01bf6f47$2ec3ac80$4500a8c0@thomasnotebook> Message-ID: In article <03be01bf6f47$2ec3ac80$4500a8c0@thomasnotebook>, Thomas Heller writes >> Python picks up .pyd's or .dll's for extensions. We could just change >> _shared_lib_ext to '.pyd'. >But this does change nothing. >Only after I changed > g['SO'] = '.dll' >to > g['SO'] = '.pyd' >in sysconfig.py: _init_nt() > >Have to look after this. >Thomas .... mmmmhhhh this code is dusty -- Robin Becker From mhammond@skippinet.com.au Sat Feb 5 02:53:20 2000 From: mhammond@skippinet.com.au (Mark Hammond) Date: Sat, 5 Feb 2000 13:53:20 +1100 Subject: [Distutils] RE: [Python-Dev] Proposal: Registry access for Python on Windows In-Reply-To: <018b01bf6e4a$4db27700$4500a8c0@thomasnotebook> Message-ID: [Thomas writes] > I admit: it has been too low level. I have posted > (and implemented) a new proposal describing a more > high level interface. Your high-level interface looks fine (except it should be coded in Python ;-) > In this I followed Gordon's > suggestion: Provide the minimum needed. I took that to mean "for the high-level interface" > If one wants to do more special things, one probably needs > your win32 extensions anyway. Im not really convinced about this. I would still rather see the complete win32api registry support added, seeing as the code exists. > Already too late! (Programming is fun :-) (Is competition good ?) Of course it is good - it means I will get my implementation done quicker now ;-) > Don't you think that the raw Win32 api functions are much too low > level to belong into core python? How should they be documented? You said you have seen the sources, so you should have seen there is also copious documentation - Im converting them to docstrings. Mark. From gward@cnri.reston.va.us Sat Feb 5 03:17:35 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Fri, 4 Feb 2000 22:17:35 -0500 Subject: [Distutils] Building PIL with Distutils Message-ID: <20000204221734.A22037@cnri.reston.va.us> --BOKacYhQ+x31HxR3 Content-Type: text/plain; charset=us-ascii Hi all -- I've just finished hacking in the feature that allows Distutils to build a standalone C library before building any extension modules. This model is used by PIL, and I've used it in the past for a Perl extension, so I'm sure it's a useful thing to have. To demonstrate, I've just released a new code snapshot: http://www.python.org/sigs/distutils-sig/distutils-20000204.tar.gz If you want to try it out, you'll need * the 2000-02-04 Distutils code snapshot installed on your system * the PIL source distribution * a C compiler It's entirely possible that this will work under Windows, but I expect it will take some messing around. The first hurdle is to generate libImaging/ImConfig.h; under Unix, this is a snap: cd libImaging ; ./configure Fredrik, do you perchance have a version of ImConfig.h that you use to build PIL on Windows? Next, you have to (ugh) edit the setup script. I hate this, but there's going to have to be a heckuva lot of code written before it's no longer required (can you say "Autoconf in Python"?). You need to a) specify which optional libraries (from jpeglib, zlib, tcl, and tk) you have installed, and b) where they are installed. This is the bit that, in principle, ought to work under Windows as long as you have the .h and .lib files and know where they are. Then just run the setup script. If you like doing things in small chunks: python setup.py build_lib python setup.py build_ext Or, if you prefer medium sized chunks: python setup.py build And if you really enjoy biting off a lot at once: python setup.py install Please let me know how this works for you -- it worked great for me under Linux, with the four optional libraries all in /usr/{lib,include}. When I tried it on Solaris, I had to hack up the setup script to have the big editable section at the top ("Oh yeah, not every Unix system has all of these libraries installed..."). Now it works for me on Solaris. As usual, I'd love to hear if anyone gets it to work under Windows. Greg PS. I've attached the example PIL setup script in case you're curious, but not curious enough to download the Distutils snapshot. Just to be sure you know: this WILL NOT WORK with Distutils 0.1.3! It needs the 2000-02-04 snapshot. --BOKacYhQ+x31HxR3 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="pil_setup.py" #!/usr/bin/env python """Example setup.py for Fredrik Lundh's Imaging library (the PIL distribution).""" # created 1999/09/19, Greg Ward __rcsid__ = "$Id: pil_setup.py,v 1.7 2000/02/05 03:00:07 gward Exp $" import os from distutils.core import setup from distutils.ccompiler import new_compiler from glob import glob lib_dir = 'libImaging' lib_name = "Imaging" lib_sources = glob (os.path.join (lib_dir, "[A-Z]*.c")) # PIL can build against several optional libraries: jpeglib, zlib, and # Tcl/Tk. Since the Distutils hasn't yet tackled the job of rewriting # Autoconf in Python, you have to supply some vital data here: which of # these libraries to use, where to find their header files, and where to # find the libraries themselves. # # 'optional_libs' just lists the library names -- in a Unix environment, # each of these strings will be translated to eg. "-ljpeg" on the # link command line, meaning you have to have libjpeg.a somewhere. # # 'include_dirs' is a list of directories to search for header files for # the optional libraries plus PIL's own internal support library # (libImaging/libImaging.a under Unix). You should only customize the # entries after 'lib_dir'. For a typical Unix installation # "/usr/local/include" is probably what you want, since that's the # traditional place to install third-party libraries. (Your mileage may # vary, so check with your system administrator if you're not sure # whether these libraries are installed and where they are installed.) # Most Linux distributions should include those libraries # out-of-the-box, so the header files should be in /usr/include, meaning # you don't have to add anything to include_dirs. # # Finally, 'library_dirs' lists the locations of PIL's internal support # library as well as the optional third-party libraries. Again, the # traditional Unix location is "/usr/local/lib", and you probably don't # need to list any extra library directories for most Linux # distributions. optional_libs = ['jpeg', 'z', 'tcl8.0', 'tk8.0'] # good sysadmin #optional_libs = [] # lazy sysadmin include_dirs = [lib_dir, '/usr/local/include'] # typical Unix #include_dirs = [lib_dir, '/depot/sundry/include'] # CNRI's setup #include_dirs = [lib_dir] # most Linux distributions library_dirs = [lib_dir, '/usr/local/lib'] # typical Unix #library_dirs = [lib_dir, '/depot/sundry/plat/lib']# CNRI's setup #library_dirs = [lib_dir] # most Linux distributions # ------------------------------------------------------------------------ # You should't have to change anything below this point! # Figure out macros to define based on whether Tcl/Tk are in the # 'optional_libs' list. use_tcl = use_tk = 0 for lib in optional_libs: if lib[0:3] == 'tcl': use_tcl = 1 if lib[0:2] == 'tk': use_tk = 1 if use_tcl and use_tk: ext_macros = [('WITH_TKINTER', None)] else: ext_macros = None # arg -- still haven't written autoconf in Python, so we rely on # the user to run the configure script config_header = os.path.join (lib_dir, "ImConfig.h") if not os.path.exists (config_header): raise SystemExit, \ "%s not found -- did you run the configure script in %s?" % \ (config_header, lib_dir) setup (name = "PIL", version = "1.0", description = "Python Imaging Library", author = "Fredrik Lundh", author_email = "fredrik@pythonware.com", url = "http://www.pythonware.com/downloads.htm", # Hmm, we don't install the sane module or PIL's "Scripts" # directory. We could handle sane (in a slightly ugly way -- since # it's in a directory of its own, it would necessarily be the only # top-level module in the distribution), but Distutils as yet has no # facilities for installing scripts. libraries = [('%s/%s' % (lib_dir, lib_name), { 'sources': lib_sources, 'include_dirs': include_dirs, 'macros': [('HAVE_CONFIG_H', None)], } )], packages = ['PIL'], ext_modules = \ [('_imaging', { 'sources': ['_imaging.c', 'decode.c', 'encode.c', 'map.c', 'display.c', 'outline.c', 'path.c'], # This must include the directories with the JPEG, # zlib, and Tcl/Tk header files (if installed) 'include_dirs': include_dirs, # Keep this for Tcl/Tk support 'macros': ext_macros, # This must include the directories with the JPEG, zlib, # and Tcl/Tk libraries (whatever's available) 'library_dirs': library_dirs, # And this, of course, lists which of those external # libraries to link against (plus libImaging, which *must* # be included!) 'libraries': ['libImaging/Imaging'] + optional_libs } )] ) --BOKacYhQ+x31HxR3-- From gward@cnri.reston.va.us Sat Feb 5 03:38:40 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Fri, 4 Feb 2000 22:38:40 -0500 Subject: [Distutils] project description files? (was: who needs keyword params?) In-Reply-To: ; from gstein@lyra.org on Thu, Feb 03, 2000 at 04:32:31AM -0800 References: <3898A548.6B295F0D@atd.ucar.edu> Message-ID: <20000204223839.A22123@cnri.reston.va.us> On 03 February 2000, Greg Stein said: > I just had a reasonably evil thought. Looking at the above code, I > thought, "man. look at all those parameters. gets kind of ugly. it would > be nice to not have to pass all those, but just define them at the top > level." Of course, that's when the anti-Pythonic demon possessed me, and I > figured that we could do something like the following (untested) code: [...evil code skipped...] > Now that is fun! :-) > > The only caveat is that any arguments passed to setup() must be keyword > arguments -- it does not define any positional arguments (unlike the > current setup function). Of course, this could be its "specification" > and/or an extension of the above code could fix that. Now Greg, I know you recently moved to California, but I don't know where. From reading that code, I'd have to guess you either live in Mountain View, just down the street from a certain Mr. Wall; or you have taken up residence in a low-rent district of San Francisco, and now spend your time smoking crack in between "joke" posts to various Python mailing lists. In a past life, I wrote Perl code that did something rather like what Greg's suggesting. It wasn't quite so evil, though, since Perl has a clearly understood "main" namespace in which "global global" variables (as opposed to mere module globals) can be found. It was handy, but I don't think I'm proud of it... But seriously folks... I don't really care for the deeply nested data structures you have to construct in order to specify a real-world set of extension modules, either. And I've been toying with some possible "solutions" to the "I need to set compiler flags for individual source files" problem that will only make things worse. Now, you can always make things look nicer with some temp. variable assignments just before the setup() call, eg. ext1 = ('ext1', { ... hairy build info dict ... }) ext2 = ('ext2', { ... another hairy build info dict ... }) setup (... ext_modules = [ext1, ext2] ) But that really only papers over the problem. The thing that keeps hitting me on the head is "What are these deeply nested data structure for?". Well, they describe a project: in this case, a set of Python extension modules. So perhaps the answer is a simple declarative "project description language". Meaning that people will have to write (simple declarative) "project description files" along with their setup scripts. Is this sounding to much like make-land for some people? Or do you think it's just the ticket? Greg From gward@cnri.reston.va.us Sat Feb 5 04:01:37 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Fri, 4 Feb 2000 23:01:37 -0500 Subject: [Distutils] Compiling / Installing extensions In-Reply-To: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook>; from thomas.heller@ion-tof.com on Fri, Feb 04, 2000 at 04:37:52PM +0100 References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> Message-ID: <20000204230137.B22123@cnri.reston.va.us> On 04 February 2000, Thomas Heller said: > It seems, with Robins latest patch compiling extensions > under windows goes smoothly. Great! I got a bit lost in the flurry of patches-to-patches, so could one of you mail me either a definitive patch or a replacement version of msvccompiler.py with the "let's go find the compiler" code included? (Send it to gward@python.net so I'll get it at home.) > 1. Usually extensions are named xxx.pyd, not xxx.dll > Should this be changed? Definitely -- the "foo.pyd wraps foo.dll" argument is compelling. FWIW, msvccompiler.py is *not* the place to fix this. The CCompiler classes are meant to apply to more than just compiling Python extensions, but rather to know "everything" about compiling and linking C on a given platform/compiler. Unfortunately, the rest of the Distutils tries really hard to be platform-neutral, which means this kind of stuff -- which is specific to building Python extensions on a given platform -- falls between the cracks. My preferred solution is to put platform-dependent code into build_ext.py, as you can see in the current kludge for handling .def files. That reminds me, I've been convinced on several occasions that .def files are not necessary (since Python extensions really only need to export one symbol, 'init_foo()' or whatever it is, which can easily be specified on the compiler command-line). I'd love to see that MSVC-specific kludge stripped out of build_ext.py, but I think this requires some other changes: * notion of "list of symbols to export" when building a shared object or library (in CCompiler and all subclasses -- it would just be ignored in UnixCCompiler) * build_ext passes ['init_%s' % extension_name] for that * MSVCCompiler generates the appropriate command-line options for cl.exe to export the listed symbols If any of you intrepid Windows hackers would like to tackle this, be my guest! And feel free to do whatever cleanup you like in msvccompiler.py; there is definitely some cruft in there. > 2. install copies all files built: not only the dll but also > .exp, .lib, everthing which is output by the linker. > Is this also the case on unix? > Or is my setup.py wrong? Wow, I had no idea this was happening. That should be the job of MSVCCompiler to clean up, since it "knows everything" about the compiler and platform, eg. what crufty turds it drops behind after compiling or linking something. Silly me, I blithely assumed that no compiler would be so stupid as to leave unnecessary temporary files around after a build. I should never have underestimated Microsoft's capacity for mind-numbingly stupid code. > 3. It would really be nice if there were a possibility to > compile debug versions. Shouldn't be too hard; just need to add: * debug flag to CCompiler instances * debug flag to CCompiler methods (the usual "method param overrides instance attribute" paradigm I've used everywhere in those classes) * UnixCCompiler sticks "-g" into the command line if debug is true * MSVCCompiler sticks "**something**" into the command line if debug is true * build_ext and build_lib should take 'debug' options, which can be set on the command line I think if all this is done, then setup.py build_ext --debug will do what you want on either Unix or Windows. If I don't get a patch for it this weekend, I'll do it myself... Greg From robin@jessikat.demon.co.uk Sat Feb 5 09:20:26 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Sat, 5 Feb 2000 09:20:26 +0000 Subject: [Distutils] Compiling / Installing extensions In-Reply-To: <20000204230137.B22123@cnri.reston.va.us> References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> <20000204230137.B22123@cnri.reston.va.us> Message-ID: <9qtONDAat+m4Ewsg@jessikat.demon.co.uk> In article <20000204230137.B22123@cnri.reston.va.us>, Greg Ward writes >On 04 February 2000, Thomas Heller said: >> It seems, with Robins latest patch compiling extensions >> under windows goes smoothly. > >Great! I got a bit lost in the flurry of patches-to-patches, so could >one of you mail me either a definitive patch or a replacement version of >msvccompiler.py with the "let's go find the compiler" code included? >(Send it to gward@python.net so I'll get it at home.) > Thomas has the latest working version >> 1. Usually extensions are named xxx.pyd, not xxx.dll >> Should this be changed? > >Definitely -- the "foo.pyd wraps foo.dll" argument is compelling. FWIW, >msvccompiler.py is *not* the place to fix this. The CCompiler classes >are meant to apply to more than just compiling Python extensions, but >rather to know "everything" about compiling and linking C on a given >platform/compiler. Unfortunately, the rest of the Distutils tries >really hard to be platform-neutral, which means this kind of stuff -- >which is specific to building Python extensions on a given platform -- >falls between the cracks. .... agreed >My preferred solution is to put platform-dependent code into >build_ext.py, as you can see in the current kludge for handling .def >files. > >That reminds me, I've been convinced on several occasions that .def >files are not necessary (since Python extensions really only need to >export one symbol, 'init_foo()' or whatever it is, which can easily be >specified on the compiler command-line). I'd love to see that >MSVC-specific kludge stripped out of build_ext.py, but I think this >requires some other changes: > > * notion of "list of symbols to export" when building a shared > object or library (in CCompiler and all subclasses -- it would > just be ignored in UnixCCompiler) > * build_ext passes ['init_%s' % extension_name] for that > * MSVCCompiler generates the appropriate command-line options for > cl.exe to export the listed symbols > This is certainly doable. The compile.py from used to do that. >If any of you intrepid Windows hackers would like to tackle this, be my >guest! And feel free to do whatever cleanup you like in >msvccompiler.py; there is definitely some cruft in there. > >> 2. install copies all files built: not only the dll but also >> .exp, .lib, everthing which is output by the linker. >> Is this also the case on unix? >> Or is my setup.py wrong? > >Wow, I had no idea this was happening. That should be the job of >MSVCCompiler to clean up, since it "knows everything" about the compiler >and platform, eg. what crufty turds it drops behind after compiling or >linking something. Silly me, I blithely assumed that no compiler would >be so stupid as to leave unnecessary temporary files around after a >build. I should never have underestimated Microsoft's capacity for >mind-numbingly stupid code. > >> 3. It would really be nice if there were a possibility to >> compile debug versions. > >Shouldn't be too hard; just need to add: > > * debug flag to CCompiler instances > * debug flag to CCompiler methods (the usual "method param overrides > instance attribute" paradigm I've used everywhere in those classes) > * UnixCCompiler sticks "-g" into the command line if debug is true > * MSVCCompiler sticks "**something**" into the command line if debug > is true > * build_ext and build_lib should take 'debug' options, which can > be set on the command line > >I think if all this is done, then > > setup.py build_ext --debug > >will do what you want on either Unix or Windows. If I don't get a patch >for it this weekend, I'll do it myself... > > Greg > >_______________________________________________ >Distutils-SIG maillist - Distutils-SIG@python.org >http://www.python.org/mailman/listinfo/distutils-sig -- Robin Becker From gstein@lyra.org Sat Feb 5 12:36:35 2000 From: gstein@lyra.org (Greg Stein) Date: Sat, 5 Feb 2000 04:36:35 -0800 (PST) Subject: [Distutils] project description files? In-Reply-To: <20000204223839.A22123@cnri.reston.va.us> Message-ID: On Fri, 4 Feb 2000, Greg Ward wrote: >... > > Now Greg, I know you recently moved to California, but I don't know > where. From reading that code, I'd have to guess you either live in > Mountain View, just down the street from a certain Mr. Wall; or you have I'm in Palo Alto. Next town over. Two in the same town might be... um... dangerous. :-) > taken up residence in a low-rent district of San Francisco, and now > spend your time smoking crack in between "joke" posts to various Python > mailing lists. Joke?! Joke!! I was serious. Really! > Ditto :-) >... > Now, you can always make things look nicer with some temp. variable > assignments just before the setup() call, eg. > > ext1 = ('ext1', { ... hairy build info dict ... }) > ext2 = ('ext2', { ... another hairy build info dict ... }) > > setup (... > ext_modules = [ext1, ext2] > ) > > But that really only papers over the problem. This would be my real suggestion to the issue, but that suggestion wouldn't have been as much fun :-) > The thing that keeps hitting me on the head is "What are these deeply > nested data structure for?". Well, they describe a project: in this > case, a set of Python extension modules. So perhaps the answer is a > simple declarative "project description language". Meaning that people > will have to write (simple declarative) "project description files" > along with their setup scripts. > > Is this sounding to much like make-land for some people? Or do you > think it's just the ticket? Ack! I thought we threw out the "new little language" a while back. But if you insist, maybe there is a way to break down the structure into something that fits nicely with ConfigParser? For example: [global] target = ooky [ext1] flag1 = foo modules = bar, baz [ext2] flag3 = bogle ... Then use "setup(conf='project.conf')". Of course, this throws out all possibility of computed values, which is why we kept the config "language" in Python. You could state that the .conf file is used IFF the corresponding parameter is not provided to setup(). Cheers, -g -- Greg Stein, http://www.lyra.org/ From gward@cnri.reston.va.us Sun Feb 6 15:17:11 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Sun, 6 Feb 2000 10:17:11 -0500 Subject: [Distutils] project description files? In-Reply-To: ; from gstein@lyra.org on Sat, Feb 05, 2000 at 04:36:35AM -0800 References: <20000204223839.A22123@cnri.reston.va.us> Message-ID: <20000206101710.A352@cnri.reston.va.us> On 05 February 2000, Greg Stein said: > Ack! I thought we threw out the "new little language" a while back. > > But if you insist, maybe there is a way to break down the structure into > something that fits nicely with ConfigParser? For example: For some reason, it keeps coming back to haunt me. I am not a big fan of little declarative languages, but sometimes they can be useful. Eg. for the limited range of tasks to which make is well-suited, I would much rather write a makefile than a Python script. The problem is that most such little languages give you no good way to break out; make at least gives you *a* way (shell commands), but no one would call that a *good* way. (GNU make gives you a few more ways [various forms of access to the external environment when defining variables], but again that'll only go so far until you hit the usual brick wall endemic to special-purpose languages.) I find Anthony Pfrunder's Zmake intriguing -- the idea is basically makefiles with Python code instead of shell commands. I just don't know the right way to integrate it, or any other little declarative language for that matter, into the Distutils' way of doing things. I'll continue to toy with the idea, and I'm certainly open to entertaining others' ideas. > [global] > target = ooky > > [ext1] > flag1 = foo > modules = bar, baz > > [ext2] > flag3 = bogle > ... > > Then use "setup(conf='project.conf')". > > Of course, this throws out all possibility of computed values, which is > why we kept the config "language" in Python. You could state that the > .conf file is used IFF the corresponding parameter is not provided to > setup(). That just might work. The idea of section headers being dynamically determined by the nature of the current module distribution (is "ext1" present or not?) seems a tiny bit dangerous; somewhere down that road lies the madness of programming languages with dynamic syntax. But the ConfigParser syntax is so simple I don't think this minor abuse could be carried very far. Greg From gward@cnri.reston.va.us Sun Feb 6 15:55:00 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Sun, 6 Feb 2000 10:55:00 -0500 Subject: [Distutils] Proposed new manifest scheme Message-ID: <20000206105459.A361@cnri.reston.va.us> Hi all -- right up at the top of my list of things to do for the Distutils is "fix the dist command". The main problems are that the syntax for the MANIFEST file (where you specify the files to go in a source distribution) is a bit goofy, and there's poor feedback about which files will actually be included. Part 1 of the fix is to rearrange how the "dist" command works. I've worked out a solution that appeals to me; give this a read and see if it sounds right to you. Step 1 (optional): Module developer creates MANIFEST.in file. This is a concise-but- readable specification of the files to be included in the source distribution with wildcards, directory recursion, and exclusion. (Syntax to be worked out in a future post; it should have the same features as the current syntax, but be more readable.) Step 2 (optional): Developer runs "dist" command with "--manifest-only" option. Distutils parses MANIFEST.in and spits out MANIFEST, containing a complete, explicit list of every file to be included in the source distribution. If MANIFEST.in doesn't exist -- ie. the developer skipped step 1 -- Distutils creates a MANIFEST using the "default fileset", mostly the pure Python modules and extension module source files mentioned in setup.py. Possibly: print a warning for every file in the source tree that *won't* be included in the source distribution; or write this information to another file (with a "files were excluded: see ..." warning). Step 3: Developer runs "dist" command. If MANIFEST doesn't exist or is out-of-date (ie. developer skipped step 2, or edited MANIFEST.in since running step 2), it is regenerated. Distutils then creates a source distibution (tarball, zipfile, whatever) containing exactly the files listed in MANIFEST. Possibly *this* is the step where the warning about excluded files should be generated. Open issues: * Obviously we should regenerate MANIFEST whenever MANIFEST.in is updated. (I.e. MANIFEST is only auto-generated, never edited -- unless you like playing with fire.) But what about when the filesystem changes? If I add or remove files or directories, MANIFEST should be regenerated. I seem to recall that detecting this in a portable way is pretty much impossible, especially in the presence of network filesystems (NFS, SMB). Two possible solutions: regenerate MANIFEST every time the "dist" command is run, or add a "--force-manifest" option. I prefer the latter because of the (presumed) expense of walking the directory tree. * Should MANIFEST be included with the source distribution? What about MANIFEST.in? The only reason I can see for including MANIFEST is that it would enable trivial integrity checking; it buys you zero security-wise, but at least ensures the download wasn't truncated. I think this is a bogus argument; a truncated ZIP or .tar.gz file simply won't unpack without errors, so I don't see a big need for checking for the presence of all files. A less trivial integrity check could be done by adding MD5 signatures (or something) to MANIFEST, but that would all-but-require regenerating the damn thing every time the "dist" command is run. Including MANIFEST.in is more defensible; it's one of the "source" files the developer uses to maintain the distribution, and third parties should be able to regenerate the author's source distribution if needed. (Just following the letter of the free software licences: forking should be an option, but hopefully not a commonly used one!) * When should the developer be warned about files in his development tree that weren't picked up by the MANIFEST-generating scan? If we do it when the MANIFEST is generated (step 2 above), that's when the developer is most likely to be watching -- why would you bother doing an explicit separate MANIFEST generation if you're not going to watch it closely? However, doing it as late as possible -- ie. when the MANIFEST is read and the source distribution is generated -- maximizes the chance of spotting any late additions not caught in MANIFEST.in's net. At the very least, this will remind the developer to re-run "dist --force-manifest", and it might well remind him to edit his MANIFEST.in to catch the late additions. * Speaking of warning about files not included: there should be a way to say, "Don't warn me about excluding X". X could be *~, *.bak, *.o, or whatever (depending on your platform, editor, compiler, development style, ...). One possibility: any file explicitly excluded by MANIFEST.in would be exempt from "not included" warnings. This would require you to explicitly exclude *~, *.o, etc. -- normally they would not be caught in MANIFEST.in's web at all, so no need to exclude them. Alternately, we could add a bit more syntax to MANIFEST.in that says "Don't necessarily include or exclude this file, but don't warn me if it happens to be excluded". I don't see any need for this, and it could be confusing. Thoughts? * Should the "default fileset" be included even when you have a MANIFEST.in? This seems obvious to me, but others (hi Fred!) have disagreed in the past. Since you can always exclude files from the default set (a feature of the present syntax that will be included in any future syntax), I see no need to make the great utility of a default fileset disappear just because you need to distribute files *not* mentioned in setup.py. Coming soon: proposed new syntax for the MANIFEST.in file. Greg From gward@cnri.reston.va.us Mon Feb 7 00:56:29 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Sun, 6 Feb 2000 19:56:29 -0500 Subject: [Distutils] And a proposed new manifest syntax Message-ID: <20000206195629.A487@cnri.reston.va.us> ...picking up where I left earlier today, here's my proposed new syntax for the MANIFEST.in file. Note that the semantics are the same as the existing MANIFEST, but it should be a) easier to read, b) easier to write, and c) easier to understand the results (thanks to the division of labour between MANIFEST.in and MANIFEST -- see my earlier post). The requirements for the MANIFEST.in syntax is that the developer must be able to include or exclude files by - exact path (including wildcards -- "foo/bar.py" is just a special case of "foo/*.py") - wildcard applied recursively throughout a directory subtree And of course the MANIFEST.in file must be clear, concise, and expressive. The existing syntax is already concise and expressive, but pssbly a lttl too cncse. In my proposed syntax, there are four types of lines in the MANIFEST.in file: include exactly the specified file(s) (path can be a complete directory specification ending with a wildcard, eg. "foo.py", "bar/README.txt", "qux/*.c") exclude exclude exactly the specified file(s) recursive split into and ; recursively include all files under that match recursive-exclude you figure it out ;-) Possible points of contention (not "open issues" because I've pretty much made up my mind -- but I am willing to consider the existence of unenlightened heathens who dare to disagree ;-): * no "include" keyword -- clarity/concision tradeoff * no wildcards on directories, ie. "foo/bar/*.py" is OK, but not "foo/*/bar.py" A subtlety that I don't think I've mentioned very widely: like in the setup script, paths would be delimited with a forward slash rather than using the local convention. (Distutils explicitly does "string.split (..., '/')" in many places -- this is a feature, as the resulting list is then put back together with 'os.path.join()' before being used in the real world.) The purpose is the same for both the setup script and MANIFEST.in -- portability. As an example, here's how the MANIFEST file for Distutils 0.1.3 would look with the new syntax: *.txt examples/*.txt examples/*.py exclude distutils/command/build_lib.py So -- opinions? I'll probably implement the new 'dist' command in the next few days, so get back to me soon if you think I'm doing it wrong. Greg From fredrik@pythonware.com Mon Feb 7 09:15:15 2000 From: fredrik@pythonware.com (Fredrik Lundh) Date: Mon, 7 Feb 2000 10:15:15 +0100 Subject: [Distutils] Re: Building PIL with Distutils References: <20000204221734.A22037@cnri.reston.va.us> Message-ID: <00bc01bf714b$d92c53b0$f29b12c2@secret.pythonware.com> Greg Ward wrote: > Fredrik, do you perchance have a version of ImConfig.h that you use to > build PIL on Windows? it's in the source distribution: ImConfig.h.win (don't have time to look at the new distutils kit right now, but I'll make a serious attempt to add it to the upcoming PIL 1.0.1 release -- maybe as soon as later this week. please let me know if you think it's too soon...). From robin@jessikat.demon.co.uk Mon Feb 7 11:04:35 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Mon, 7 Feb 2000 11:04:35 +0000 Subject: [Distutils] Compiling / Installing extensions In-Reply-To: <9qtONDAat+m4Ewsg@jessikat.demon.co.uk> References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> <20000204230137.B22123@cnri.reston.va.us> <9qtONDAat+m4Ewsg@jessikat.demon.co.uk> Message-ID: In article <9qtONDAat+m4Ewsg@jessikat.demon.co.uk>, Robin Becker writes ... the change to get the initXXX exported currently needs to be done in build_extensions at the point where the check for a .def file is done. In my view it is wrong for the compiler specific stuff to be anywhere other than the specific compiler file, but there you go. so in build_ext.py line 225 should look like if self.compiler.compiler_type == 'msvc': extra_args.append('/export:init%s'%extension_name) What's the point of all the definitions in msvccompiler if they aren't used? -- Robin Becker From thomas.heller@ion-tof.com Mon Feb 7 12:54:51 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Mon, 7 Feb 2000 13:54:51 +0100 Subject: [Distutils] Compiling / Installing extensions References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> <20000204230137.B22123@cnri.reston.va.us> Message-ID: <049a01bf716a$863ab650$4500a8c0@thomasnotebook> > > 1. Usually extensions are named xxx.pyd, not xxx.dll > > Should this be changed? > > Definitely -- the "foo.pyd wraps foo.dll" argument is compelling. FWIW, > msvccompiler.py is *not* the place to fix this. The CCompiler classes > are meant to apply to more than just compiling Python extensions, but > rather to know "everything" about compiling and linking C on a given > platform/compiler. Unfortunately, the rest of the Distutils tries > really hard to be platform-neutral, which means this kind of stuff -- > which is specific to building Python extensions on a given platform -- > falls between the cracks. > Only one change neccessary, in sysconfig.py line 150: g['SO'] = '.dll' make this: g['SO'] = '.pyd' > > 2. install copies all files built: not only the dll but also > > .exp, .lib, everthing which is output by the linker. > > Is this also the case on unix? > > Or is my setup.py wrong? > > Wow, I had no idea this was happening. That should be the job of > MSVCCompiler to clean up, since it "knows everything" about the compiler > and platform, eg. what crufty turds it drops behind after compiling or > linking something. Silly me, I blithely assumed that no compiler would > be so stupid as to leave unnecessary temporary files around after a > build. I should never have underestimated Microsoft's capacity for > mind-numbingly stupid code. > The current msvccompiler.py I sent to you adds the /INCREMENTAL:No flag to the linker, which will suppress the .ilk file (used for incremental linking). The generation of .exp and .lib files cannot be suppressed by command line flags. If you want to use msvccompiler.py in a way you described above, we cannot simply delete the .exp and .lib files because they may be needed for dependent modules to be build. Wouldn't it be better to change install_ext to only copy the expected files? > > 3. It would really be nice if there were a possibility to > > compile debug versions. > > Shouldn't be too hard; just need to add: > > * debug flag to CCompiler instances > * debug flag to CCompiler methods (the usual "method param overrides > instance attribute" paradigm I've used everywhere in those classes) > * UnixCCompiler sticks "-g" into the command line if debug is true > * MSVCCompiler sticks "**something**" into the command line if debug > is true > * build_ext and build_lib should take 'debug' options, which can > be set on the command line > > I think if all this is done, then > > setup.py build_ext --debug > > will do what you want on either Unix or Windows. If I don't get a patch > for it this weekend, I'll do it myself... > > Greg If we start to compile debug version, we will have to struggle with the .pdb file, the 'program database' containing debugging information. What do other windows developers think? Should we link with /pdb:None, which will write the debugging info into the executable? Or should we live with the .pdb file in the same directory as the .pyd? Thomas From robin@jessikat.demon.co.uk Mon Feb 7 14:10:08 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Mon, 7 Feb 2000 14:10:08 +0000 Subject: [Distutils] Compiling / Installing extensions In-Reply-To: <049a01bf716a$863ab650$4500a8c0@thomasnotebook> References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> <20000204230137.B22123@cnri.reston.va.us> <049a01bf716a$863ab650$4500a8c0@thomasnotebook> Message-ID: In article <049a01bf716a$863ab650$4500a8c0@thomasnotebook>, Thomas Heller writes ... >> >Only one change neccessary, in sysconfig.py line 150: > g['SO'] = '.dll' >make this: > g['SO'] = '.pyd' > It's bad to do that here. This is supposed to be for general stuff. DLLs are called .dll; python extensions are called .pyd. The output extension is supposedly set in msvccompiler, but then build_extension doesn't use it. >> >The current msvccompiler.py I sent to you >adds the /INCREMENTAL:No flag to the linker, >which will suppress the .ilk file (used for incremental linking). > >The generation of .exp and .lib files cannot be suppressed >by command line flags. > >If you want to use msvccompiler.py in a way you described above, >we cannot simply delete the .exp and .lib files because they >may be needed for dependent modules to be build. >Wouldn't it be better to change install_ext to only copy the expected >files? > .... From the MSDN docs You can use LIB with the /DEF option to create an import library and an export file. LINK uses the export file to build a program that contains exports (usually a dynamic-link library (DLL)), and it uses the import library to resolve references to those exports in other programs. In most situations, you do not need to use LIB to create your import library. When you link a program (either an executable file or a DLL) that contains exports, LINK automatically creates an import library that describes the exports. Later, when you link a program that references those exports, you specify the import library. However, when a DLL exports to a program that it also imports from, whether directly or indirectly, you must use LIB to create one of the import libraries. When LIB creates an import library, it also creates an export file. You must use the export file when linking one of the DLLs. >If we start to compile debug version, we will have to struggle >with the .pdb file, the 'program database' containing debugging >information. >What do other windows developers think? Should we link with /pdb:None, >which will write the debugging info into the executable? Or should we >live with the .pdb file in the same directory as the .pyd? > >Thomas > > We could try and use /Z7 type debugging instead of /Zi /Z7 C 7.0- Compatible Produces an .OBJ file and an .EXE file containing line numbers and full symbolic debugging information for use with the debugger. The symbolic debugging information includes the names and types of variables, as well as functions and line numbers. Note that according to the standard we need to do stuff like using the _d versions of dll/pyd names for our link. >_______________________________________________ >Distutils-SIG maillist - Distutils-SIG@python.org >http://www.python.org/mailman/listinfo/distutils-sig -- Robin Becker From thomas.heller@ion-tof.com Mon Feb 7 15:33:53 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Mon, 7 Feb 2000 16:33:53 +0100 Subject: [Distutils] Compiling / Installing extensions References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook><20000204230137.B22123@cnri.reston.va.us><049a01bf716a$863ab650$4500a8c0@thomasnotebook> Message-ID: <053e01bf7180$bdee8020$4500a8c0@thomasnotebook> > In article <049a01bf716a$863ab650$4500a8c0@thomasnotebook>, Thomas > Heller writes > ... > >> > >Only one change neccessary, in sysconfig.py line 150: > > g['SO'] = '.dll' > >make this: > > g['SO'] = '.pyd' > > > > It's bad to do that here. This is supposed to be for general stuff. > DLLs are called .dll; python extensions are called .pyd. > The output extension is supposedly set in msvccompiler, but then > build_extension doesn't use it. Yes, you are right of course. > > >> > >The current msvccompiler.py I sent to you > >adds the /INCREMENTAL:No flag to the linker, > >which will suppress the .ilk file (used for incremental linking). > > > >The generation of .exp and .lib files cannot be suppressed > >by command line flags. > > > >If you want to use msvccompiler.py in a way you described above, > >we cannot simply delete the .exp and .lib files because they > >may be needed for dependent modules to be build. > >Wouldn't it be better to change install_ext to only copy the expected > >files? > > > .... > > >From the MSDN docs [deleted] I know this stuff (or at least I can read it in my msdn). The problem is: The linker generates an .exp and .lib file for every python extension built. Should msvccompiler.py delete these, or should install_ext delete them? Or should install_ext only copy the .pyd files instead of the whole tree? > We could try and use /Z7 type debugging instead of /Zi > > /Z7 C 7.0- Compatible Produces an .OBJ file and an .EXE file containing > line numbers and full symbolic debugging information for use with the > debugger. The symbolic debugging information includes the names and > types of variables, as well as functions and line numbers. > Is it true that /Zi to the compiler and /pdb:None to the linker does the same as /Z7 to the compiler? I'm not sure... > Note that according to the standard we need to do stuff like using the > _d versions of dll/pyd names for our link. At least for python15.lib and pyton15_d.lib there is some magic in config.h, so this is automatic. Works also for /D_DEBUG. We should drop the line self.add_library ( "python" + sys.version[0] + sys.version[2] ) in msvccompiler.py. Thomas From robin@jessikat.demon.co.uk Mon Feb 7 19:02:41 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Mon, 7 Feb 2000 19:02:41 +0000 Subject: [Distutils] Compiling / Installing extensions In-Reply-To: <053e01bf7180$bdee8020$4500a8c0@thomasnotebook> References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> <20000204230137.B22123@cnri.reston.va.us> <049a01bf716a$863ab650$4500a8c0@thomasnotebook> <053e01bf7180$bdee8020$4500a8c0@thomasnotebook> Message-ID: <0Yyr1AARbxn4EwWP@jessikat.demon.co.uk> In article <053e01bf7180$bdee8020$4500a8c0@thomasnotebook>, Thomas Heller writes ... >Yes, you are right of course. ... being right doesn't help. The proper thing to do is somehow to get the right value for .pyds and or .dlls Greg does the higher level give any indication whether this is a python only thing or a normal dll? >> .... >> >> >From the MSDN docs [deleted] > >I know this stuff (or at least I can read it in my msdn). >The problem is: >The linker generates an .exp and .lib file for every python extension >built. Should msvccompiler.py delete these, or should install_ext >delete them? Or should install_ext only copy the .pyd files >instead of the whole tree? As I understand it the .exp files are only needed when we want to do complicated (2 stage) linking a dll imports from and exports to another. >> We could try and use /Z7 type debugging instead of /Zi >> >> /Z7 C 7.0- Compatible Produces an .OBJ file and an .EXE file containing >> line numbers and full symbolic debugging information for use with the >> debugger. The symbolic debugging information includes the names and >> types of variables, as well as functions and line numbers. >> >Is it true that /Zi to the compiler and /pdb:None to the linker >does the same as /Z7 to the compiler? I'm not sure... > You do cl -c -Z7 dingo.c .... link dingo.obj ...other.obj /incremental:no /pdb:NONE /debug /debugtype:cv /export:initdingo /out:dingo.pyd and no pdb file. I checked and it seems as though this results in files that can still be debugged provided you can find the .c files. >> Note that according to the standard we need to do stuff like using the >> _d versions of dll/pyd names for our link. >At least for python15.lib and pyton15_d.lib there is some >magic in config.h, so this is automatic. Works also for /D_DEBUG. >We should drop the line > self.add_library ( "python" + sys.version[0] + sys.version[2] ) >in msvccompiler.py. > >Thomas ... the python_d.lib needs to be used in the link phase. -- Robin Becker From gward@cnri.reston.va.us Mon Feb 7 22:15:01 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Mon, 7 Feb 2000 17:15:01 -0500 Subject: [Distutils] Re: Building PIL with Distutils In-Reply-To: <00bc01bf714b$d92c53b0$f29b12c2@secret.pythonware.com>; from fredrik@pythonware.com on Mon, Feb 07, 2000 at 10:15:15AM +0100 References: <20000204221734.A22037@cnri.reston.va.us> <00bc01bf714b$d92c53b0$f29b12c2@secret.pythonware.com> Message-ID: <20000207171501.A892@cnri.reston.va.us> On 07 February 2000, Fredrik Lundh said: > it's in the source distribution: ImConfig.h.win D'oh! It does help to use one's eyeballs. Anyways, all you Windows people, please try out the new snapshot with the PIL setup script. You'll have to manually rename ImConfig.h.win to ImConfig.h before running the setup script. > (don't have time to look at the new distutils kit right now, > but I'll make a serious attempt to add it to the upcoming > PIL 1.0.1 release -- maybe as soon as later this week. please > let me know if you think it's too soon...). Here's the deal: if you are one of the brave ones to adopt Distutils before it's part of Python 1.6 (ie. before 1.6 is released), then I'll give you really super-duper support. Just ask Paul Dubois or Andrew Kuchling. ;-) If you put it off, then you'll have to read the manual that I'll eventually write, just like everyone else. Of course, there may still be incompatible changes in the setup script between now and then. (I can guarantee there will be incompatible changes in the command-line UI and the extension API, but that doesn't worry me at this early stage.) Greg From gward@python.net Tue Feb 8 02:06:38 2000 From: gward@python.net (Greg Ward) Date: Mon, 7 Feb 2000 21:06:38 -0500 Subject: [Distutils] Compiling / Installing extensions In-Reply-To: <0Yyr1AARbxn4EwWP@jessikat.demon.co.uk>; from Robin Becker on Mon, Feb 07, 2000 at 07:02:41PM +0000 References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> <20000204230137.B22123@cnri.reston.va.us> <049a01bf716a$863ab650$4500a8c0@thomasnotebook> <053e01bf7180$bdee8020$4500a8c0@thomasnotebook> <0Yyr1AARbxn4EwWP@jessikat.demon.co.uk> Message-ID: <20000207210638.B838@beelzebub> On 07 February 2000, Robin Becker said: > being right doesn't help. The proper thing to do is somehow to get the > right value for .pyds and or .dlls > > Greg does the higher level give any indication whether this is a python > only thing or a normal dll? For now, the Distutils command classes only build DLLs (shared libraries or shared objects, whatever you want to call them) that are Python extensions. The compiler classes don't know that, though, and there's no guarantee that it'll stay that way in the future. As I've said, I favour hacking sysconfig.py. That means the PYD/DLL distinction is entirely filename based. Will that work? > You do > cl -c -Z7 dingo.c > .... > link dingo.obj ...other.obj /incremental:no /pdb:NONE /debug > /debugtype:cv /export:initdingo /out:dingo.pyd > > and no pdb file. I checked and it seems as though this results in files > that can still be debugged provided you can find the .c files. Oh, I forgot to mention: I strongly favour putting debug info in the executable or DLL or .so or whatever it's called. It's been that way in Unix since time immemorial, so IMHO it's good enough for Python extensions under Windows. ;-) Greg -- Greg Ward - Unix weenie gward@python.net http://starship.python.net/~gward/ This quote intentionally left blank. From gward@python.net Tue Feb 8 02:03:08 2000 From: gward@python.net (Greg Ward) Date: Mon, 7 Feb 2000 21:03:08 -0500 Subject: [Distutils] Compiling / Installing extensions In-Reply-To: <053e01bf7180$bdee8020$4500a8c0@thomasnotebook>; from Thomas Heller on Mon, Feb 07, 2000 at 04:33:53PM +0100 References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook><20000204230137.B22123@cnri.reston.va.us><049a01bf716a$863ab650$4500a8c0@thomasnotebook> <053e01bf7180$bdee8020$4500a8c0@thomasnotebook> Message-ID: <20000207210308.A838@beelzebub> On 07 February 2000, Thomas Heller said: > > In article <049a01bf716a$863ab650$4500a8c0@thomasnotebook>, Thomas > > Heller writes > > ... > > >> > > >Only one change neccessary, in sysconfig.py line 150: > > > g['SO'] = '.dll' > > >make this: > > > g['SO'] = '.pyd' > > > > > > > It's bad to do that here. This is supposed to be for general stuff. > > DLLs are called .dll; python extensions are called .pyd. > > The output extension is supposedly set in msvccompiler, but then > > build_extension doesn't use it. > Yes, you are right of course. Ooh, this is getting hairy. As it happens, I disagree with both of you -- or rather, I agree with Thomas' original position, which was to hack sysconfig.py. Now I have to convince Thomas back around to his original view. ;-) My reasoning: sysconfig.py is Python-specific (it exists solely to parse Python's config.h and Makefile), and therefore it's perfectly OK to put code in there that only applies when building Python extensions. Furthermore, the '_init_nt()' function is Windows-specific code in a Python-specific module: what *better* place to do the hacks needed to build Python extensions on Windows! If only all the {platform,language}-dependent stuff could be elegantly shoved in there. Any other opinions? Fred? You wrote sysconfig.py, was setting g['SO'] to '.dll' right or wrong? Should it be '.pyd' instead? > I know this stuff (or at least I can read it in my msdn). > The problem is: > The linker generates an .exp and .lib file for every python extension > built. Should msvccompiler.py delete these, or should install_ext > delete them? Or should install_ext only copy the .pyd files > instead of the whole tree? It should be the job of MSVCCompiler to know what the compiler and linker it drives do, including what temporary files they leave behind, how to convince them to leave as few temporary files as possible, and how to cleanup any that can't be suppressed. The whole point of the "build" tree is to make installation a trivial recursive copy; I really don't want to keep track of what files wind up in "build" that are supposed to be installed, because by definition they're *all* supposed to be installed. The fact that compiler turds wind up in the "build" tree is a bug in msvccompiler.py. (I could always blame it on Microsoft, but I expect they document it somewhere. Bastards.) Here's an idea I've been toying with: give the "build" tree two tasks, first to serve as an testing/installation staging area (its current task) and second as an area for temporary build files. Eg. after you build a module distribution with extensions, the "build" tree would look like -- build -- lib ------ .py files platlib -- .so files temp ----- .o, .a, and other temporary build by-products or, to allow using the same source tree to build multiple architectures: -- build -- lib ---------------- .py files platlib.linux-x86 -- .so files platlib.winnt-x86 -- .pyd files temp.linux-x86 ----- .o, .a, etc. temp.winnt-x86 ----- .obj, .lib, .exp, etc. This makes installation and testing only slightly less trivial, and now cleanup also becomes a snap: "rm -rf build/temp.*" (coded in Python and made available as a 'clean' command, of course). Key question for the Window guys: is it possible to make MSVC++ put .exp files in a separate directory from the .pyd files produced at the same time? If not, that doesn't invalidate the above scheme, it just makes it slightly less useful -- MSVCCompiler will still have to delete the .exp (and other compiler/linker turds) after creating a .pyd. > Is it true that /Zi to the compiler and /pdb:None to the linker > does the same as /Z7 to the compiler? I'm not sure... > > > Note that according to the standard we need to do stuff like using the > > _d versions of dll/pyd names for our link. > At least for python15.lib and pyton15_d.lib there is some > magic in config.h, so this is automatic. Works also for /D_DEBUG. Ugh. OK, here's a deal: I'll put in the infrastructure for supporting a "--debug" flag if you guys will figure out how to make it work with MSVC++. Fair enough? > We should drop the line > self.add_library ( "python" + sys.version[0] + sys.version[2] ) > in msvccompiler.py. Absoposidefinutely yes! I guess by my evolving standard, that should go in the MSVC-specific code in build_ext.py. (Am I going to live to regret that decision? sigh...) Greg -- Greg Ward - Linux weenie gward@python.net http://starship.python.net/~gward/ Think big -- pollute Lake Superior. From gward@python.net Tue Feb 8 01:33:19 2000 From: gward@python.net (Greg Ward) Date: Mon, 7 Feb 2000 20:33:19 -0500 Subject: [Distutils] Compiling / Installing extensions In-Reply-To: ; from Robin Becker on Mon, Feb 07, 2000 at 11:04:35AM +0000 References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> <20000204230137.B22123@cnri.reston.va.us> <9qtONDAat+m4Ewsg@jessikat.demon.co.uk> Message-ID: <20000207203319.A829@beelzebub> On 07 February 2000, Robin Becker said: > In my view it is wrong for the compiler specific stuff to be anywhere > other than the specific compiler file, but there you go. It's a tradeoff between kludging the Python-extension-specific, platform-neutral code or kludging the Python-extension-ignorant, platform-specific code. I prefer the former, because Python-extension-specific code has fewer applications outside of the Distutils. The application-neutral, platform-specific code (MSVCCompiler and UnixCCompiler) could be more broadly applicable, so I'm trying to keep it "pure". > so in build_ext.py line 225 should look like > > if self.compiler.compiler_type == 'msvc': > extra_args.append('/export:init%s'%extension_name) OK, except you mean line 238 in the current CVS version. ;-) At any rate, done. Any Windows people who feel a deep attachment for .def files better speak up now. More seriously, is there any reason a Python extension module needs to export any symbols other than "init%s" % extension_name ??? > What's the point of all the definitions in msvccompiler if they aren't > used? Hey, I said it was crufty code. But now that I look at it, I only see one class attribute ('_exe_ext') that's not used. That's not too bothersome. So what *are* you referring to? Greg -- Greg Ward - all-purpose geek gward@python.net http://starship.python.net/~gward/ Cheops' Law: Nothing *ever* gets built on schedule or within budget. From thomas.heller@ion-tof.com Tue Feb 8 10:21:34 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue, 8 Feb 2000 11:21:34 +0100 Subject: [Distutils] Compiling / Installing extensions References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook><20000204230137.B22123@cnri.reston.va.us><049a01bf716a$863ab650$4500a8c0@thomasnotebook> <053e01bf7180$bdee8020$4500a8c0@thomasnotebook> <20000207210308.A838@beelzebub> Message-ID: <01ab01bf721e$46972880$4500a8c0@thomasnotebook> > Key question for the Window guys: is it possible to make MSVC++ put .exp > files in a separate directory from the .pyd files produced at the same > time? If not, that doesn't invalidate the above scheme, it just makes > it slightly less useful -- MSVCCompiler will still have to delete the > .exp (and other compiler/linker turds) after creating a .pyd. Yes, the /implib:build\temp\mymodule.lib puts the .lib AND the .exp files in a certain directory. So build_ext (or whatever) can prune the temp-directory. > > Ugh. OK, here's a deal: I'll put in the infrastructure for supporting a > "--debug" flag if you guys will figure out how to make it work with > MSVC++. Fair enough? Fair enough. I'm waiting for your patch. > > > We should drop the line > > self.add_library ( "python" + sys.version[0] + sys.version[2] ) > > in msvccompiler.py. > > Absoposidefinutely yes! I guess by my evolving standard, that should go > in the MSVC-specific code in build_ext.py. It is NOT NEEDED at all! The correct pythonxxx.lib will automatically be included on windows. Thomas From thomas.heller@ion-tof.com Tue Feb 8 10:39:56 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue, 8 Feb 2000 11:39:56 +0100 Subject: [Distutils] Compiling / Installing extensions References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> <20000204230137.B22123@cnri.reston.va.us> <9qtONDAat+m4Ewsg@jessikat.demon.co.uk> <20000207203319.A829@beelzebub> Message-ID: <01cb01bf7220$d7d47c10$4500a8c0@thomasnotebook> > > so in build_ext.py line 225 should look like > > > > if self.compiler.compiler_type == 'msvc': > > extra_args.append('/export:init%s'%extension_name) > > OK, except you mean line 238 in the current CVS version. ;-) At any > rate, done. Any Windows people who feel a deep attachment for .def > files better speak up now. More seriously, is there any reason a Python > extension module needs to export any symbols other than > "init%s" % extension_name > ??? I suggest: if self.compiler.compiler_type == 'msvc': def_file = build_info.get ('def_file') if def_file is None: source_dir = os.path.dirname (sources[0]) ext_base = (string.split (extension_name, '.'))[-1] def_file = os.path.join (source_dir, "%s.def" % ext_base) if not os.path.exists (def_file): self.warn ("file '%s' not found: " % def_file + "might have problems building DLL") def_file = None if def_file is not None: extra_args.append ('/DEF:' + def_file) else: modname = string.split (extension_name, '.')[-1] extra_args.append('/export:init%s'%modname) split(extension_name) is needed for extensions living in package directories. If the developer bothers to write a def file, it should be used. Otherwise guess the exported symbol. Note: For normal python extensions, only this single export is used. This may be different for COM-modules for example... Thomas From robin@jessikat.demon.co.uk Tue Feb 8 10:37:03 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Tue, 8 Feb 2000 10:37:03 +0000 Subject: [Distutils] Compiling / Installing extensions In-Reply-To: <20000207210638.B838@beelzebub> References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> <20000204230137.B22123@cnri.reston.va.us> <049a01bf716a$863ab650$4500a8c0@thomasnotebook> <053e01bf7180$bdee8020$4500a8c0@thomasnotebook> <0Yyr1AARbxn4EwWP@jessikat.demon.co.uk> <20000207210638.B838@beelzebub> Message-ID: The Numeric package itself has _numpy which exports lots of things. I don't seriously object to changing sysconfig.py. Go ahead. Since the .pyd only gets done in sysconfig it follows that _shared_lib_ext = '.pyd' in msvccompiler doesn't get used so then it follows that shared_library_filename isn't being used properly. I say go for the -Z7 debugging mode. I would point out that this is essentially a legacy mode for M$ as it represents C7.0 (the CodeView era) and as with all legacy things it will be withdrawn as and when Bill sees fit. I can only find one output location option /out: and it seems that as soon as you specify one export that .lib and .exp files get created alongside the output file. So the new logic should be that the compiler shoves its .obj files into temp and the linker intially does likewise. A final stage copies wanted files .pyd .lib into the platlib.winnt.x86 or whatever. can we have a keep list which defaults to .pyd, but can be added to by an option. Then _numpy could add .lib As a final note; if you want to have a single temp area, we need different object extensions eg .obj and .dobj for optimised and debug versions. -- Robin Becker From thomas.heller@ion-tof.com Tue Feb 8 11:38:08 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue, 8 Feb 2000 12:38:08 +0100 Subject: [Distutils] Compiling / Installing extensions References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook><20000204230137.B22123@cnri.reston.va.us><049a01bf716a$863ab650$4500a8c0@thomasnotebook><053e01bf7180$bdee8020$4500a8c0@thomasnotebook><0Yyr1AARbxn4EwWP@jessikat.demon.co.uk> <20000207210638.B838@beelzebub> Message-ID: <020101bf7228$f94d8a50$4500a8c0@thomasnotebook> [Robin] > The Numeric package itself has _numpy which exports lots of things. > > I don't seriously object to changing sysconfig.py. Go ahead. Since the > .pyd only gets done in sysconfig it follows that > _shared_lib_ext = '.pyd' in msvccompiler doesn't get used so then it > follows that shared_library_filename isn't being used properly. > > I say go for the -Z7 debugging mode. I would point out that this is > essentially a legacy mode for M$ as it represents C7.0 (the CodeView > era) and as with all legacy things it will be withdrawn as and when Bill > sees fit. We could hope this -Z7 would stay for a while. Even now you can generate mapfiles (wasn't this the symdeb era?). > > I can only find one output location option /out: and it seems that as > soon as you specify one export that .lib and .exp files get created > alongside the output file. So the new logic should be that the compiler > shoves its .obj files into temp and the linker intially does likewise. > A final stage copies wanted files .pyd .lib into the platlib.winnt.x86 > or whatever. See my previous post about the /implib: linker flag. > > can we have a keep list which defaults to .pyd, but can be added to by > an option. Then _numpy could add .lib > > As a final note; if you want to have a single temp area, we need > different object extensions eg .obj and .dobj for optimised and debug > versions. The usual way (also used for core python) is to have separate directories: temp-debug and temp-release. And to have separate pyd filenames: module.pyd and module_d.pyd. From robin@jessikat.demon.co.uk Tue Feb 8 11:59:14 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Tue, 8 Feb 2000 11:59:14 +0000 Subject: [Distutils] Compiling / Installing extensions In-Reply-To: <020101bf7228$f94d8a50$4500a8c0@thomasnotebook> References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> <20000204230137.B22123@cnri.reston.va.us> <049a01bf716a$863ab650$4500a8c0@thomasnotebook> <053e01bf7180$bdee8020$4500a8c0@thomasnotebook> <0Yyr1AARbxn4EwWP@jessikat.demon.co.uk> <20000207210638.B838@beelzebub> <020101bf7228$f94d8a50$4500a8c0@thomasnotebook> Message-ID: In article <020101bf7228$f94d8a50$4500a8c0@thomasnotebook>, Thomas Heller writes >[Robin] >> The Numeric package itself has _numpy which exports lots of things. >> >> I don't seriously object to changing sysconfig.py. Go ahead. Since the >> .pyd only gets done in sysconfig it follows that >> _shared_lib_ext = '.pyd' in msvccompiler doesn't get used so then it >> follows that shared_library_filename isn't being used properly. >> >> I say go for the -Z7 debugging mode. I would point out that this is >> essentially a legacy mode for M$ as it represents C7.0 (the CodeView >> era) and as with all legacy things it will be withdrawn as and when Bill >> sees fit. >We could hope this -Z7 would stay for a while. Even now you can >generate mapfiles (wasn't this the symdeb era?). >> >> I can only find one output location option /out: and it seems that as >> soon as you specify one export that .lib and .exp files get created >> alongside the output file. So the new logic should be that the compiler >> shoves its .obj files into temp and the linker intially does likewise. >> A final stage copies wanted files .pyd .lib into the platlib.winnt.x86 >> or whatever. >See my previous post about the /implib: linker flag. > >> >> can we have a keep list which defaults to .pyd, but can be added to by >> an option. Then _numpy could add .lib >> >> As a final note; if you want to have a single temp area, we need >> different object extensions eg .obj and .dobj for optimised and debug >> versions. >The usual way (also used for core python) is to have separate directories: >temp-debug and temp-release. And to have separate pyd filenames: >module.pyd and module_d.pyd. > .... agreed! Will you do a version after pulling the CVS code? -- Robin Becker From thomas.heller@ion-tof.com Tue Feb 8 12:13:06 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue, 8 Feb 2000 13:13:06 +0100 Subject: [Distutils] Compiling / Installing extensions References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook><20000204230137.B22123@cnri.reston.va.us><049a01bf716a$863ab650$4500a8c0@thomasnotebook><053e01bf7180$bdee8020$4500a8c0@thomasnotebook><0Yyr1AARbxn4EwWP@jessikat.demon.co.uk> <20000207210638.B838@beelzebub><020101bf7228$f94d8a50$4500a8c0@thomasnotebook> Message-ID: <021901bf722d$dc025b60$4500a8c0@thomasnotebook> > .... > agreed! Will you do a version after pulling the CVS code? Yes. I'm only waiting for greg's code with the debug flag. Thomas From gward@python.net Wed Feb 9 01:20:01 2000 From: gward@python.net (Greg Ward) Date: Tue, 8 Feb 2000 20:20:01 -0500 Subject: [Distutils] Compiling / Installing extensions In-Reply-To: <01cb01bf7220$d7d47c10$4500a8c0@thomasnotebook>; from Thomas Heller on Tue, Feb 08, 2000 at 11:39:56AM +0100 References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> <20000204230137.B22123@cnri.reston.va.us> <9qtONDAat+m4Ewsg@jessikat.demon.co.uk> <20000207203319.A829@beelzebub> <01cb01bf7220$d7d47c10$4500a8c0@thomasnotebook> Message-ID: <20000208202001.A487@beelzebub> On 08 February 2000, Thomas Heller said: > I suggest: > if self.compiler.compiler_type == 'msvc': > def_file = build_info.get ('def_file') > if def_file is None: > source_dir = os.path.dirname (sources[0]) > ext_base = (string.split (extension_name, '.'))[-1] > def_file = os.path.join (source_dir, "%s.def" % ext_base) > if not os.path.exists (def_file): > self.warn ("file '%s' not found: " % def_file + > "might have problems building DLL") > def_file = None > > if def_file is not None: > extra_args.append ('/DEF:' + def_file) > else: > modname = string.split (extension_name, '.')[-1] > extra_args.append('/export:init%s'%modname) > > split(extension_name) is needed for extensions living in package > directories. D'oh! Thanks, fixed it (not in CVS yet though). > If the developer bothers to write a def file, it should be used. > Otherwise guess the exported symbol. > > Note: > For normal python extensions, only this single export is used. > This may be different for COM-modules for example... Oh bother. OK, if it's really needed. ;-( A question for the crowd (especially Windows extension developers) before I give in: have you ever needed to define a .def file to link your extensions? Or, more bluntly, is Thomas correct, that a COM-aware extension might need to export more symbols than 'initfoo'? No offense Thomas, I just want to make sure that this ugly hack really is needed. Greg -- Greg Ward - programmer-at-large gward@python.net http://starship.python.net/~gward/ It has just been discovered that research causes cancer in rats. From gward@python.net Wed Feb 9 02:27:26 2000 From: gward@python.net (Greg Ward) Date: Tue, 8 Feb 2000 21:27:26 -0500 Subject: [Distutils] Compiling / Installing extensions In-Reply-To: <01ab01bf721e$46972880$4500a8c0@thomasnotebook>; from Thomas Heller on Tue, Feb 08, 2000 at 11:21:34AM +0100 References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook><20000204230137.B22123@cnri.reston.va.us><049a01bf716a$863ab650$4500a8c0@thomasnotebook> <053e01bf7180$bdee8020$4500a8c0@thomasnotebook> <20000207210308.A838@beelzebub> <01ab01bf721e$46972880$4500a8c0@thomasnotebook> Message-ID: <20000208212726.A524@beelzebub> On 08 February 2000, Thomas Heller said: > Yes, the /implib:build\temp\mymodule.lib > puts the .lib AND the .exp files in a certain directory. > So build_ext (or whatever) can prune the temp-directory. OK, noted. Let's wait to put it in until after I've changed the "build" commands to build to that temporary directory. Oh, I think you said something about dealing with name conflicts between debugging and non-debugging versions. For temp files, I think the obvious solution is to make "debugging" another "architecture", e.g. build ---- temp.linux-x86 temp.linux-x86-debug temp.winnt-x86 temp.winnt-x86-debug etc. It's not so clear what to do for the final product (.pyd or .so files)... but now that I think of it, if I'm going to allow multi-architecture builds in the same place (someone tell me this is a useful feature please!), then I should carry through and make the "build/platlib" directory name the platform, eg. "build/platlib.linux-x86". Keep on thinking of "debugging mode" as an aspect of the "architecture", and problem solved. Seem reasonable? > > Ugh. OK, here's a deal: I'll put in the infrastructure for supporting a > > "--debug" flag if you guys will figure out how to make it work with > > MSVC++. Fair enough? > Fair enough. I'm waiting for your patch. Done. It's in CVS now. (I take it from earlier comments that this is enough for you, and a code snapshot is not necessary.) > > > We should drop the line > > > self.add_library ( "python" + sys.version[0] + sys.version[2] ) > > > in msvccompiler.py. > > > > Absoposidefinutely yes! I guess by my evolving standard, that should go > > in the MSVC-specific code in build_ext.py. > It is NOT NEEDED at all! > The correct pythonxxx.lib will automatically be included on windows. Cool! That's good news. I hate to keep sounding disbelieving of you, but if there is a "25 words or less" explanation of why that works, would you mind filling me in? (Mainly I'm curious -- if you tell me not to get worried about it, it's complicated but it just works, then I'll probably trust you. But I *do* like to know what's going on behind the scenes...) Greg -- Greg Ward - all-purpose geek gward@python.net http://starship.python.net/~gward/ A man wrapped up in himself makes a very small package. From gward@python.net Wed Feb 9 02:38:31 2000 From: gward@python.net (Greg Ward) Date: Tue, 8 Feb 2000 21:38:31 -0500 Subject: [Distutils] Release mechanics and other bureaucracy Message-ID: <20000208213831.B524@beelzebub> Hi all -- those of you who were at IPC8 (or follow python-dev) are probably aware of the suddenly accelerated release schedule for Python 1.6. The major consequence of this for me (and this SIG) is that Distutils must be in full working order *very* soon now, or I will have a big Dutch guy named Guido out for my code (yipes!). Consequently, I have to spend as much time as possible writing a) code, b) developer documentation, c) user documentation, and d) test suites. You'll note that doing releases doesn't fall anywhere on that list. This means that no matter how many bugs are fixed, or however many new features are added, there probably won't be another official Distutils release until 1.0, which will be the run-up to inclusion in Python 1.6. (Paul Dubois tried to convince me -- if I understood properly -- that I should call that version Distutils 1.6 instead of 0.2. It strikes me that both are equally silly, so I've decided to split the difference and call it Distutils 1.0.) I expect that Python 1.6 will ship with Distutils 1.0.x, where x is the number of tries it takes before we get it right. Now, I do have a very simple release mechanism that I've been using since before Distutil 0.1 was released: code snapshots. This is done with a little P**l script that throws all the current code into a tarball and puts it on the Distutils-SIG web page. No documentation is included, no files are cvs-tagged, I don't even ensure that everything is checked in. This was mainly intended as a convenience for people who don't have CVS but want to stay on top of things. So, two questions for the audience: would you like to see versions 0.2, 0.3 etc. that gradually incorporate the new features. (It is *very* unlikely that any amount of howling will sway me on this; it took me one full day to release 0.1.3, what with checkins and doc updates and all, and I'm not eager to throw away many more days to do that. I'm just curious to hear how many people will be put off by this.) And, more importantly, would you like to see periodic code snapshots -- or are those of you who actually have the code and work on it just as happy using CVS? Please reply to me personally (gward@python.net); no need to clutter the sig with meta-issues. Greg -- Greg Ward - all-purpose geek gward@python.net http://starship.python.net/~gward/ Know thyself. If you need help, call the CIA. From robin@jessikat.demon.co.uk Wed Feb 9 09:52:50 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Wed, 9 Feb 2000 09:52:50 +0000 Subject: [Distutils] Compiling / Installing extensions In-Reply-To: <20000208202001.A487@beelzebub> References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook> <20000204230137.B22123@cnri.reston.va.us> <9qtONDAat+m4Ewsg@jessikat.demon.co.uk> <20000207203319.A829@beelzebub> <01cb01bf7220$d7d47c10$4500a8c0@thomasnotebook> <20000208202001.A487@beelzebub> Message-ID: In article <20000208202001.A487@beelzebub>, Greg Ward writes ... > >A question for the crowd (especially Windows extension developers) >before I give in: have you ever needed to define a .def file to link >your extensions? Or, more bluntly, is Thomas correct, that a >COM-aware extension might need to export more symbols than 'initfoo'? >No offense Thomas, I just want to make sure that this ugly hack really >is needed. > > Greg It may be convenient to just provide a list of the exported symbols. The _numpy extension has quite a few things it wants to export. A disadavantage is that adding a new exportable requires editing two files. The .def may get forgotten. It won't matter till it's acutally needed. In the _numpy case it may be sometime before the C api actually gets used. -- Robin Becker From thomas.heller@ion-tof.com Wed Feb 9 10:33:12 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Wed, 9 Feb 2000 11:33:12 +0100 Subject: [Distutils] Compiling / Installing extensions References: <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook><20000204230137.B22123@cnri.reston.va.us><049a01bf716a$863ab650$4500a8c0@thomasnotebook> <053e01bf7180$bdee8020$4500a8c0@thomasnotebook> <20000207210308.A838@beelzebub> <01ab01bf721e$46972880$4500a8c0@thomasnotebook> <20000208212726.A524@beelzebub> Message-ID: <017c01bf72e9$1207f100$4500a8c0@thomasnotebook> > > > > We should drop the line > > > > self.add_library ( "python" + sys.version[0] + sys.version[2] ) > > > > in msvccompiler.py. > > > > > > Absoposidefinutely yes! I guess by my evolving standard, that should go > > > in the MSVC-specific code in build_ext.py. > > It is NOT NEEDED at all! > > The correct pythonxxx.lib will automatically be included on windows. > > Cool! That's good news. I hate to keep sounding disbelieving of you, > but if there is a "25 words or less" explanation of why that works, > would you mind filling me in? (Mainly I'm curious -- if you tell me not > to get worried about it, it's complicated but it just works, then I'll > probably trust you. But I *do* like to know what's going on behind the > scenes...) > MSVC has a pragma which allow to specify in the C-source file libraries to link with. Looking at config.h (from 1.5.2): /* So nobody needs to specify the .lib in their Makefile any more */ #ifdef _DEBUG #pragma comment(lib,"python15_d.lib") #else #pragma comment(lib,"python15.lib") Quoting from MSDN: #pragma comment( comment-type [, commentstring] ) Places a comment record into an object file or executable file. The comment-type is one of five predefined identifiers, described below, that specify the type of comment record. The optional commentstring is a string literal that provides additional information for some comment types. Because commentstring is a string literal, it obeys all the rules for string literals with respect to escape characters, embedded quotation marks ("), and concatenation. [...] lib Places a library-search record in the object file. This comment type must be accompanied by a commentstring parameter containing the name (and possibly the path) of the library that you want the linker to search. Since the library name precedes the default library-search records in the object file, the linker searches for this library just as if you had named it on the command line. You can place multiple library-search records in the same source file; each record appears in the object file in the same order in which it is encountered in the source file. [End of quote] From thomas.heller@ion-tof.com Wed Feb 9 11:05:07 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Wed, 9 Feb 2000 12:05:07 +0100 Subject: [Distutils] patch for msvccompiler.py and build_ext.py, debug flag Message-ID: <01a901bf72ed$89a41230$4500a8c0@thomasnotebook> This patch against the current CVS version implements the --debug flag. Also I've included the def-file kludge again, because I still think if a def-file is present, it should be used, but I would still like to hear opinions from more windows people here. Other notes: - The python15.lib is no longer set in msvccompiler - We can still argue about the debug-flags to be used I've tested this (with and without def-files, with and without the debug flag) on numpy-15.2. It builds nicely in normal and debug mode, and testwin.bat completes successfully. Thomas Heller PS: I'm still new to diff. If I've done somthing wrong, please complain! diff -cr c:\distutils\distutils/command/build_ext.py distutils/command/build_ext.py *** c:\distutils\distutils/command/build_ext.py Wed Feb 9 09:10:32 2000 --- distutils/command/build_ext.py Wed Feb 9 11:54:04 2000 *************** *** 241,249 **** libraries = build_info.get ('libraries') library_dirs = build_info.get ('library_dirs') extra_args = build_info.get ('extra_link_args') or [] if self.compiler.compiler_type == 'msvc': ! mod_name = string.split (extension_name, '.')[-1] ! extra_args.append ('/export:init%s' % mod_name) ext_filename = self.extension_filename \ (extension_name, self.package) --- 241,263 ---- libraries = build_info.get ('libraries') library_dirs = build_info.get ('library_dirs') extra_args = build_info.get ('extra_link_args') or [] + if self.compiler.compiler_type == 'msvc': ! def_file = build_info.get ('def_file') ! if def_file is None: ! source_dir = os.path.dirname (sources[0]) ! ext_base = (string.split (extension_name, '.'))[-1] ! def_file = os.path.join (source_dir, "%s.def" % ext_base) ! if not os.path.exists (def_file): ! ## self.warn ("file '%s' not found: " % def_file + ! ## "might have problems building DLL") ! def_file = None ! ! if def_file is not None: ! extra_args.append ('/DEF:' + def_file) ! else: ! modname = string.split (extension_name, '.')[-1] ! extra_args.append('/export:init%s'%modname) ext_filename = self.extension_filename \ (extension_name, self.package) diff -cr c:\distutils\distutils/msvccompiler.py distutils/msvccompiler.py *** c:\distutils\distutils/msvccompiler.py Wed Feb 9 09:10:32 2000 --- distutils/msvccompiler.py Wed Feb 9 11:47:16 2000 *************** *** 114,123 **** CCompiler.__init__ (self, verbose, dry_run, force) - # XXX This is a nasty dependency to add on something otherwise - # pretty clean. move it to build_ext under an nt specific part. - # shared libraries need to link against python15.lib - self.add_library ( "python" + sys.version[0] + sys.version[2] ) self.add_library_dir( os.path.join( sys.exec_prefix, 'libs' ) ) self.cc = _find_exe("cl.exe") --- 114,119 ---- *************** *** 133,140 **** --- 129,142 ---- os.environ['path'] = string.join(path,';') self.preprocess_options = None self.compile_options = [ '/nologo', '/Ox', '/MD' ] + self.compile_options_debug = [ + '/nologo', '/Od', '/MDd', '/Z7', '/D_DEBUG' + ] self.ldflags_shared = ['/DLL', '/nologo', '/INCREMENTAL:NO'] + self.ldflags_shared_debug = [ + '/DLL', '/nologo', '/INCREMENTAL:no', '/pdb:None', '/DEBUG' + ] self.ldflags_static = [ '/nologo'] *************** *** 176,181 **** --- 178,188 ---- base_pp_opts.append('/c') + if debug: + compile_options = self.compile_options_debug + else: + compile_options = self.compile_options + for srcFile in sources: base,ext = os.path.splitext(srcFile) objFile = base + ".obj" *************** *** 188,198 **** inputOpt = fileOpt + srcFile outputOpt = "/Fo" + objFile ! cc_args = self.compile_options + \ base_pp_opts + \ [outputOpt, inputOpt] ! if debug: ! pass # XXX what goes here? if extra_preargs: cc_args[:0] = extra_preargs if extra_postargs: --- 195,204 ---- inputOpt = fileOpt + srcFile outputOpt = "/Fo" + objFile ! cc_args = compile_options + \ base_pp_opts + \ [outputOpt, inputOpt] ! if extra_preargs: cc_args[:0] = extra_preargs if extra_postargs: *************** *** 250,257 **** # XXX should we sanity check the library name? (eg. no # slashes) ! self.link_shared_object (objects, ! self.shared_library_name(output_libname)) def link_shared_object (self, objects, --- 256,270 ---- # XXX should we sanity check the library name? (eg. no # slashes) ! self.link_shared_object (objects=objects, ! output_libname=\ ! self.shared_library_name(output_libname), ! output_dir=output_dir, ! libraries=libraries, ! library_dirs=library_dirs, ! debug=debug, ! extra_preargs=extra_preargs, ! extra_postargs=extra_postargs) def link_shared_object (self, objects, *************** *** 259,264 **** --- 272,278 ---- output_dir=None, libraries=None, library_dirs=None, + debug=0, extra_preargs=None, extra_postargs=None): """Link a bunch of stuff together to create a shared object *************** *** 273,282 **** self.library_dirs + library_dirs, self.libraries + libraries) - ld_args = self.ldflags_shared + lib_opts + \ - objects + ['/OUT:' + output_filename] if debug: ! pass # XXX what goes here? if extra_preargs: ld_args[:0] = extra_preargs if extra_postargs: --- 287,304 ---- self.library_dirs + library_dirs, self.libraries + libraries) if debug: ! ldflags = self.ldflags_shared_debug ! basename, ext = os.path.splitext (output_filename) ! #XXX not sure this belongs here ! # extensions in debug_mode are named 'module_d.pyd' ! output_filename = basename + '_d' + ext ! else: ! ldflags = self.ldflags_shared ! ! ld_args = ldflags + lib_opts + \ ! objects + ['/OUT:' + output_filename] ! if extra_preargs: ld_args[:0] = extra_preargs if extra_postargs: From gward@python.net Thu Feb 10 02:18:33 2000 From: gward@python.net (Greg Ward) Date: Wed, 9 Feb 2000 21:18:33 -0500 Subject: [Distutils] patch for msvccompiler.py and build_ext.py, debug flag In-Reply-To: <01a901bf72ed$89a41230$4500a8c0@thomasnotebook>; from Thomas Heller on Wed, Feb 09, 2000 at 12:05:07PM +0100 References: <01a901bf72ed$89a41230$4500a8c0@thomasnotebook> Message-ID: <20000209211833.A463@beelzebub> On 09 February 2000, Thomas Heller said: > This patch against the current CVS version implements the --debug flag. > Also I've included the def-file kludge again, because I still > think if a def-file is present, it should be used, but I would > still like to hear opinions from more windows people here. OK, I buy the arguments you and Robin have put forth since I took the .def file kludge out -- it's back in again. Sigh. > - The python15.lib is no longer set in msvccompiler Fine; thanks for the explanation of how that works, BTW. > - We can still argue about the debug-flags to be used Argue away. I'll sit back, marvel at the never-ending stream of weird command-line options, and accept patches that seem to be the consensus of the Windows/MSVC++ crowd. > I've tested this (with and without def-files, with and without > the debug flag) on numpy-15.2. It builds nicely in normal > and debug mode, and testwin.bat completes successfully. Gee, a *tested* patch even! I should try testing my code sometime. ;-) BTW, at least one bug slipped past: the "if MSVC" block in build_ext.py was improperly indented; I've fixed it to: if self.compiler.compiler_type == 'msvc': def_file = build_info.get ('def_file') if def_file is None: source_dir = os.path.dirname (sources[0]) ext_base = (string.split (extension_name, '.'))[-1] def_file = os.path.join (source_dir, "%s.def" % ext_base) if not os.path.exists (def_file): def_file = None if def_file is not None: extra_args.append ('/DEF:' + def_file) else: modname = string.split (extension_name, '.')[-1] extra_args.append('/export:init%s'%modname) i.e. *everything* MSVC-specific is under that top if statement! Also I deleted the commented out warning about the missing .def file. > PS: I'm still new to diff. If I've done somthing wrong, please complain! There were some problems. Specifically: > diff -cr c:\distutils\distutils/command/build_ext.py > distutils/command/build_ext.py Not-too-surprisingly, `patch' on my Linux system doesn't know how to interpret "c:\..." -- I had to tell it which file to patch manually. Unfortunately, the obvious way to work when changing several files and doing "diff -r" is to keep the pristine sources in one place, and hack on them in another place -- which leads to absolute pathnames in "diff" output. I don't know how to get around that. There was a more serious problem in the msvccompiler.py patch: a long line got wrapped around: > --- 256,270 ---- > > # XXX should we sanity check the library name? (eg. no > # slashes) > ! self.link_shared_object (objects=objects, > ! output_libname=\ > ! > self.shared_library_name(output_libname), > ! output_dir=output_dir, > ! libraries=libraries, > ! library_dirs=library_dirs, > ! debug=debug, > ! extra_preargs=extra_preargs, > ! extra_postargs=extra_postargs) Easy to see how it happened, but it made `patch' complain of a malformed patch. Turns out I had to patch that "hunk", and one other, manually; I'm not sure why. Oh well, the other 5 hunks in the patch succeeded. Actually, I'm kinda new to the receiving patches end of things as well. I've certainly *generated* a number of patches in the past, but haven't had much software that was popular with the sort of people (ie. Real Hackers) who generate patches. Hey, what would life be without new experiences... Anyways, your changes are in CVS now. Please try it out and make sure I got the patches right -- thanks! Greg -- Greg War - just another Perl hacker gward@python.net http://starship.python.net/~gward/ No problem is so formidable that you can't just walk away from it. From gward@python.net Thu Feb 10 02:50:10 2000 From: gward@python.net (Greg Ward) Date: Wed, 9 Feb 2000 21:50:10 -0500 Subject: [Distutils] Stylistic comments Message-ID: <20000209215010.A611@beelzebub> Hi all (especially Robin and Thomas!) -- I was just looking through the new registry-grovelling code in msvccompiler.py, making some stylistic changes, when I noticed the following in '_find_exe()' (comments added by me): try: for p in string.split(os.environ['Path'],';'): fn=os.path.join(os.path.abspath(p),exe) if os.path.isfile(fn): return fn # XXX BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD !!!!!!!!!!!!!!!! except: # XXX WHAT'S BEING CAUGHT HERE?!?!? pass What is this 'except' clause supposed to catch? Other changes: I have renamed '_devstudio_versions()' to 'get_devstudio_versions()' because it's more descriptive and I don't see a big need for the leading underscore; likewise '_msvc_get_paths()' is now 'get_msvc_paths()'. I've changed the 'get_devstudio_versions()' docstring to: """Get list of devstudio versions from the Windows registry. Return a list of strings (???) containing version numbers; the list will be empty if we were unable to access the registry (eg. couldn't import a registry-access module) or the appropriate registry keys weren't found. (XXX is this correct???)""" Input, please? I'd like a similarly explicit and useful docstring for 'get_msvc_paths()', but don't understand it well enough to write one: could someone supply me one please? Oh yeah, I'm mystified by the purpose for '_find_SET()' and '_do_SET()'. Could I get docstrings for those please? Better parameter names (full words, not single characters!) might help. Also, the real reason I went off on this tangent was because we get an even *worse* traceback than before now on machines without MSVC++ (eg. my Linux box being told to use an MSVCCompiler object, which used to work just fine in dry-run mode). In particular, it blows up because 'get_devstudio_versions()' returns None, but '_find_exe()' expects a sequence. My inclination is to fix 'get_devstudio_versions()' to always return a list, empty if the imports failed. Ditto for 'get_msvc_paths()' for consistency. Anyone see a problem with this? (I.e., are we ever going to need to distinguish "couldn't find anything in the registry" from "couldn't access the registry at all"?) Thanks -- Greg -- Greg Ward - Linux geek gward@python.net http://starship.python.net/~gward/ "He's dead, Jim. You get his tricorder and I'll grab his wallet." From robin@jessikat.demon.co.uk Thu Feb 10 11:20:13 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Thu, 10 Feb 2000 11:20:13 +0000 Subject: [Distutils] Stylistic comments In-Reply-To: <20000209215010.A611@beelzebub> References: <20000209215010.A611@beelzebub> Message-ID: In article <20000209215010.A611@beelzebub>, Greg Ward writes >Hi all (especially Robin and Thomas!) -- > >I was just looking through the new registry-grovelling code in >msvccompiler.py, making some stylistic changes, when I noticed the >following in '_find_exe()' (comments added by me): > > try: > for p in string.split(os.environ['Path'],';'): > fn=os.path.join(os.path.abspath(p),exe) > if os.path.isfile(fn): > return fn > # XXX BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD !!!!!!!!!!!!!!!! > except: # XXX WHAT'S BEING CAUGHT HERE?!?!? > pass > >What is this 'except' clause supposed to catch? > well os.environ['Path'] may not be there and then it would be a keyError >Other changes: I have renamed '_devstudio_versions()' to >'get_devstudio_versions()' because it's more descriptive and I don't see >a big need for the leading underscore; likewise '_msvc_get_paths()' is >now 'get_msvc_paths()'. > >I've changed the 'get_devstudio_versions()' docstring to: > > """Get list of devstudio versions from the Windows registry. Return a > list of strings (???) containing version numbers; the list will > be empty if we were unable to access the registry (eg. couldn't > import a registry-access module) or the appropriate registry > keys weren't found. (XXX is this correct???)""" > >Input, please? Correct on the doc. I put underscore to stop these being implicitly available outside of the module. > >I'd like a similarly explicit and useful docstring for >'get_msvc_paths()', but don't understand it well enough to write one: >could someone supply me one please? > >Oh yeah, I'm mystified by the purpose for '_find_SET()' and '_do_SET()'. >Could I get docstrings for those please? Better parameter names (full >words, not single characters!) might help. _find_SET(name,version_number) '''' looks up in the registry and returns a list of values suitable for use in a SET command eg SET name=value. Normally the value will be a ';' separated list similar to a path list. name is the name of an MSVC path and version_number is a version_number of an MSVC installation. '''' _do_SET(name,version_number) '''' sets os.environ[name] to an MSVC path type value obtained from _find_SET. This is equivalent to a SET command prior to exceution of spawned commands. '''' > >Also, the real reason I went off on this tangent was because we get an >even *worse* traceback than before now on machines without MSVC++ >(eg. my Linux box being told to use an MSVCCompiler object, which used >to work just fine in dry-run mode). In particular, it blows up because >'get_devstudio_versions()' returns None, but '_find_exe()' expects a >sequence. My inclination is to fix 'get_devstudio_versions()' to always >return a list, empty if the imports failed. Ditto for I thought we had that None fixed ages ago. They should return []. What to do if they do is open. >'get_msvc_paths()' for consistency. Anyone see a problem with this? >(I.e., are we ever going to need to distinguish "couldn't find anything >in the registry" from "couldn't access the registry at all"?) > >Thanks -- > > Greg -- Robin Becker From thomas.heller@ion-tof.com Thu Feb 10 19:38:45 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Thu, 10 Feb 2000 20:38:45 +0100 Subject: [Distutils] Stylistic comments References: <20000209215010.A611@beelzebub> Message-ID: <037801bf73fe$72d28570$4500a8c0@thomasnotebook> From: Greg Ward > Hi all (especially Robin and Thomas!) -- > > Also, the real reason I went off on this tangent was because we get an > even *worse* traceback than before now on machines without MSVC++ > (eg. my Linux box being told to use an MSVCCompiler object, which used > to work just fine in dry-run mode). In particular, it blows up because > 'get_devstudio_versions()' returns None, but '_find_exe()' expects a > sequence. My inclination is to fix 'get_devstudio_versions()' to always > return a list, empty if the imports failed. Ditto for > 'get_msvc_paths()' for consistency. Anyone see a problem with this? As Robin already said, this had been fixed some time ago, but it somehow made it back in my version. I have to admit that the patch I sent last night was not the latest one Robin and I had agreed on. It was the newest one I had on my harddrive though... Attached is another msvccompiler.py (zipped), and hopefully Robin can agree on this one. I've already included the doc-strings Robin has supplied. BTW: What is the preferred format posting sourcecode to the list? AFAIK, zipped attachments are not so nice in the archive, unpacked source-code has the white-space problem in Outlook Express, which I'm using here... As soon as you extend the distutils framework to use temp-directories for building, I will submit a patch for msvc again. Thomas From thomas.heller@ion-tof.com Thu Feb 10 19:44:33 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Thu, 10 Feb 2000 20:44:33 +0100 Subject: [Distutils] Stylistic comments References: <20000209215010.A611@beelzebub> <037801bf73fe$72d28570$4500a8c0@thomasnotebook> Message-ID: <038901bf73ff$41524a70$4500a8c0@thomasnotebook> This is a multi-part message in MIME format. ------=_NextPart_000_0386_01BF7407.A2A10D40 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit > Attached is another msvccompiler.py (zipped), > and hopefully Robin can agree on this one. > I've already included the doc-strings Robin has > supplied. Sorry, attachment was missing. Thomas Heller ------=_NextPart_000_0386_01BF7407.A2A10D40 Content-Type: application/x-zip-compressed; name="msvccompiler.zip" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="msvccompiler.zip" UEsDBBQAAAAIAHWlSijfZWvhwQ0AAMc0AAAPAAAAbXN2Y2NvbXBpbGVyLnB57Rprb9tG8rsB/4c5 +QJSkUo/+owKIUgdtzWi2IHtJL2zDYIiVxJriuRxSdv69zezD5JLUrLc67Uo0L1eRHNnZ+e18+L2 er0g5HmRhxF3fD9ZpmHEst2d3Z3jJM69MObw/vLT8bGaGIIXQ7hMI7ZkOJ2HSQzJDPIFA2/K88zz cyiBwY88znd3ZkkmIN6HfpbwZJbDp5AXXgSXeRGEidPr9WjD3Z098DPm5SyAw1evXu0ffLd/+GoI H1iWrRA2iSIC2d1x3cznYeC6MIbeP0+DESz5fUm7k66G93DoHB7B0cHBwf7B0f7hARwcjb4+Gn11 BPMHLwvg5DGFf4pdkZkkyyHh5SNf1Z7zLIznyEKWLKESFFKUZBwU0MvWfEmMBrnZ3QEaNUHOWeym GUuzxGecu0lKwuTyfRRO9QspmYDNcCJ3A3bPhdDce5Zxmge7PyIIwo6C/InlECEdpJUSGEpgQSfp 4nMYB8kDh4zNETpbOQAXLC+yGDxFKpR4pBA4+NIi8FkjhLhYTvHxe4FTwD+EUQRTViJhyzRfQTiD B4b/ZQyK2JtGDPIEPJ84F0s1GWCzuYMbFVEQW7mSXonLK+G+UGuXSVBErA/KwrwUxZlmIdpQhfKO rbjYGjGWqGZJEQfS8uhPhBuVc1ppD2H85ZGXht0TKA05wR59luZwKuZOyDJqqDIp1OtbraN3aLTW JR4CtEN2c1OeiZubt1pdloScICStk+RmMPU4gzAGW2/v/Pzu5F/u8eTN5eXJpXtxfn41rDYuhwk9 OT9+M3Hfvzn++fTs5Gnw448XFydnV+7Hy5OLp6EJ6rJf494UK407ZErL1blg8/OUxe/Y6uTRJvaG 7/omeIjgB+arhwWeHzgctalp76ZH2tj1BO0Wd7XvhmG/ewUabHp9cEvytg6Pvvzq62++/e6Vhd4v QGRxktPEZM1uNCYO2iKLAzvt2EBZTEkRaxhNfUzRI961p0gwIQzgsJraBmsqHDI9TRyO5mr39V8Z ozPN9AtltxPplte6HtM7kRt2Uy9f4BT9DLWfGFvfOAfWENLIy9GUl2Pr8btvLG0pym15HY4rCDPm 50kWMkQZxn5UBGyIcFM68bRF/884w/WTSYaCdIzHFlJl1VaJt2BNwmnmZSt1puklLpYe1SnQRDIh KdSkBW/DjFv9yk3YT/iJm5sX/ObGgkFNx9YPRRgFcLniOVve3FC4SWIM1Qj4QQkfHz8T82C/4H1E V4nY6sOLMlThsJX6hlpviri/vVE5fos3stPh/TDvd/ikT15UsM1eyTQdRDIWRrXBFX2qDI6nUZjb 90Pre2vNDjRIvfek208bsNJAcu6JAMsCvWQyEmlCGBds89rSQd5voGSN86PR4QD1+B/c6/N8aNtJ Vh5Q+kbyjO4Mcy2XPTIb/1+6RFemTto+SeYpCbCBxaIfa90qsTIeJ9whMOfXJIxt/Qfm4/SLNjLE fWtCRq1pmJDP0HztWdwfaWZmsfZye0EokjAiH8L8e7JoFBD6aMoAaX2DdMPIcAsW34dZEl9bH4iH W2F1/1/Ca0rBpcgCFiA5hhOOh4VywkWSMjD1cnlyZcfekg27JYzRJUqSOw6FYNHIVSkZkLvxWvC6 pzPMgRdhLhJdFE/JQiF9pge4K56T5ZJQsLn4k4gYi8UOnKGv9aJoJfYT73ReXUvPUZzAWeplomIS 2/NwGUZeJpJrGWvotaM1ioN2gVAm3eIZKcaSjmo8uYAoMkVB4F7jXYmvthxrxRyJFkVhFZWVOhpm LSTeNOpSMUFSqWWN5SN+znIONTMj8FvBep2ffJVqESZTKl9QVlQFlRyUZuDA1QJZxf/Yf4oQl2DY lJKsawuri0QIGA3ML1QBXCJDy32gHRQ0r+SQjhv21uaewCidqJ2RJntjdcbEgUmVH6f/iVLbKNXB Lh/7UEkNy3ossvEsyBX5wsurgp6L8j7OWTbzfFGjtWr248GgCr0ex8M1EzKdSmOtti/7AWIjXeTT Kl0cu0I3GD7IMCw9KyzAxTIzxwLf5iyadYV6kt404Wx80DkbZCs3K+I1s3gkfVxZFc40Ssqdxu56 q6HGOpQI+vXVBOl4QUClO+V7LmauNhj+jRoLDlkNlf2z8HEIlDJyC/pQc3LV0/1ZsUTpbMy9NSxF YgRvRCqFgX6oloE9WITzBUNHodDUUdAQXPg+Po1rYavnR0R3bygw9TuWRGF8Zy6hN+sX6RMucubN ECrp74YSOXZ1rspg2YbszsaeDl1pZ+iqDxmzuos9lU5QSrkmB6mSCT3au1f5mzz2VFmZGRyLOGsg 36tVUVSvioZHM4R1a38MWuPrVV0puHUK2k0thD/DKqQBp9xADegarP04iZJ5gkq09s8fxc/7t+Ln 85cW3G5GgedkWswJkUm4iTVQWAONln7+/a34eeu+Pfnh40+Wuf62xWMUzCJvzl2+wOosoB1x7WQi cNT2Oj07vjh5jyXLm8no7Nxqkm9iWU/9E5hj+SoNpiMSs2Skm43u7amR65vyv5VhhWD34Isv4HOS 3aFTX7J8kQQc3zxvaEz2skDvg2lMGXJk6ODFVIQJxku35vroSXIW88o4HL80AddP0455n5h3cM6q lOYm018JlCKNg89KKuSnytf4rF8rZVALVs8GUVTOClkZs+TDKln98ssv4ohZSZGnRU5xwAJK0lB0 MpcK53GSVRFTi1QczZBrPEIc/wA4hV9JZIgKc3ABlJD8sMjieIQpRXkIMc2pxd040Dg+xuFjreM8 RRw8WTK0Eaxbqf+aFyLLvIupFVwFeJFsaiQL715kAkvvjhEJD2gJjuPIafkvhWx1FNdHbJ4UGDR5 11QlqjEZcBfI0qM8ZO20ihKEYj2QOF/dKQHqMvMoLnvZfD0CBZWgEWgwM4fAMCwJJT3TdMMpq8mq e6QW1clfs9QAGdeaUTTQqpmf/4jyb89RJ8PFw4I+kiZvTLTdXyGkFh1F7kDR3ZlLtYdYapA7MKjv r6NOR1Fr37ea2Y3QXUMk7QCyPihUKzti5XaIqjXVEyUQPPNJ8iKNkDbeQC96SdJb6IRQJBr4ylaL G3kD6lOgHMs22wB65LiMUEsD5UJoaWOi1/CYXdkOojxPiYze/pXfCO4samEzPOxT+NI2dTEeappH AA060NJq8CscgMb1Y9LDdwMthSZe33fp+CFoU2+DpnVXwzgFG+CuS1KGJQO33XKvHEaHbBSR16MD St8M6HW4lFtZj8wR2gjANhf0m+QJ9YlKFOxrldTdItMKTdvWtO/QB7C0vxqoquNr0PUMQUc9jsHN Q0AmvwqjK5PhjZxaEM5m9BUul18hn45YlHdQzEIEeAgE01UkRR8lS60QYwrUqi7N3x5gHK9iXMFZ 4ABcJiPxHoldIU9i6mXMWMCCl0PqYvpeLCf1N0xMpMPcX7DgNQbjGUjYYaPYffPhVHZodLT0F148 Z2WcpAhJGXMtf9hQ26ohZb3R7arIiehET+FpyE0xVo9SrltCrjaHXT02hF89tgvDDeiN4bjkZU1Y reZbQbnO3MbVq3pQ1lPVk/qYTwCNb/sqzFY0DCp6nhNsDTIGBlVboum94IQFq/Xe/uT0hw9vrn4e veA9w7VEgfa7XdXDoOJzjXtV5ozTWCydf7waWeTjpVlSfBANLkMDXUGfCuY2bjmkH3qgltY8QXEu 0N28NjBuctuKv80u+yl3rZBs46rrbhqrl6q6Jl+tha2cWb05Jv1IWaX87Ufa43f2I9Ku+IKupdA9 Fu7FYb5CH8/8O3XzRdAvOuqv5R2WOKmv51jNLVg99Jbq1qqUWgJ7C20ZOCpLECIkEmxTnf1tcNX0 Wj1us7DS8rN8l6Hy53osaQLi323ATXMw/nrGcm0frSPdtUYfW/q3eWi1pp86t7DV0QVoOtHtoLc5 wfCcQwzPPMew1VGG555mWHug9XSv15uQn/Uwu4z9hbziVswwYU3mmK0y8XlH3kREGNXik5ow9iV5 Y0r5vvDpO9sdo36+4ZrtvjXUPWByE1LyLRz6mxx7xKrQD/OI+mEpPuO2Hi87SRrUqj4q0fhrJDl/ bDqzPq3apq+gMptWolPr0poLqLiU3/S6q3ywGypsOI09Ci/UpOeFKE1QD1MWJXTnklKYBjBURTkV 64Ied5kETBQ2hD4AS16MdAMnXQWNNnCDFtVkEI8DsNyA0jLcolrU0THZKKLupFEv+d0SRUOPf8XE zmiw/6jVscTCMaIbFs/qtdcTRFcWn7XmjS2IGJbeBivmmD2c6Hmoe0cp/VLQ7YZio+dVAj6j61Wu bx6EZqKVZ2GK/pkuGTTunrw2VzapLlsZqoFmsltnx2hsVAjqAm2JRH0Ylu2+6n0jyKgL1eXV6ERj qmQGfpJljKdJHBBjdK3A8xcGazxlfjijWCC3KxebYUCxoVp3HQbQoHWoQPXHkX6dYSNRqTxFN9vr uTZCZxXqWjx7fyTDje87Bt861DQ51on0ek5lEazrgPWsIvAaZus1RCenWKNjSQ4vbE1OyZH5TapL k7+dManCP4mxDlW1lYV5gco1NFf4ps6R3qbsbPTQISBMFzITEb7tQGQkKZVECdikUFxMqAPWCOQl dtOz4pSI6gjRzsMEjrFxrQRsqtM20dTwsbVbdMKhykYqLem64aBYVhB1YjvvHJwnKSf26hcHiZ2X 6L9fkgO0iLFGQqL2kFcE5NRel+yEdNWXUeOq0+7OfwFQSwECFAAUAAAACAB1pUoo32Vr4cENAADH NAAADwAAAAAAAAABACAAtoEAAAAAbXN2Y2NvbXBpbGVyLnB5UEsFBgAAAAABAAEAPQAAAO4NAAAA AA== ------=_NextPart_000_0386_01BF7407.A2A10D40-- From DavidA@ActiveState.com Thu Feb 10 19:50:41 2000 From: DavidA@ActiveState.com (David Ascher) Date: Thu, 10 Feb 2000 11:50:41 -0800 Subject: [Distutils] install w/o build, spaces in directory names Message-ID: <002301bf7400$1c3a2d60$c355cfc0@ski.org> 1) While building Numerical on NT last night w/ the latest Distutils CVS snapshot, I found that it refused to build because the directory names had spaces in them which were not quoted. In other words, the code being executed when something like cl .... -IC:\Program Files\Python... which broke. To correct it, I patched ccompiler.py and msvccompiler.py to turn those kinds of things into cl .... -I"C:\Program Files\Python..." ... but Greg says that that will break on Unix -- I'm surprised by that claim, since IIRC quoting directory names is OK, but it's been a while since I really dealt with Unix seriously. FWIW, so far the problem occurs for -I in ccompiler.py and /LIBPATH in msvccompiler.py. The patch is listed at the end of this message. 2) Again on Windows, if I do python setup.py build python setup.py install the second line causes the code to be recompiled, even though the compilation is not needed. Greg assures me that that is not the case on Unix. Anyone have the time to tackle this? --david ascher diff -c distutils//distutils/ccompiler.py C:\Program Files\Python\distutils/ccompiler.py *** distutils//distutils/ccompiler.py Wed Feb 09 02:16:14 2000 --- C:\Program Files\Python\distutils/ccompiler.py Thu Feb 10 19:44:37 2000 *************** *** 563,569 **** pp_opts.append ("-D%s=%s" % macro) for dir in include_dirs: ! pp_opts.append ("-I%s" % dir) return pp_opts --- 563,569 ---- pp_opts.append ("-D%s=%s" % macro) for dir in include_dirs: ! pp_opts.append ('-I"%s"' % dir) return pp_opts diff -c distutils//distutils/msvccompiler.py C:\Program Files\Python\distutils/msvccompiler.py *** distutils//distutils/msvccompiler.py Thu Feb 10 02:52:42 2000 --- C:\Program Files\Python\distutils/msvccompiler.py Thu Feb 10 19:45:05 2000 *************** *** 369,375 **** def library_dir_option (self, dir): ! return "/LIBPATH:" + dir def library_option (self, lib): return self.library_filename (lib) --- 369,375 ---- def library_dir_option (self, dir): ! return "/LIBPATH:" + '"' + dir + '"' def library_option (self, lib): return self.library_filename (lib) From robin@jessikat.demon.co.uk Thu Feb 10 20:01:51 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Thu, 10 Feb 2000 20:01:51 +0000 Subject: [Distutils] install w/o build, spaces in directory names In-Reply-To: <002301bf7400$1c3a2d60$c355cfc0@ski.org> References: <002301bf7400$1c3a2d60$c355cfc0@ski.org> Message-ID: That Bill; he broke everything with spaces in paths. -- Robin Becker From mmuller@enduden.com Fri Feb 11 02:24:12 2000 From: mmuller@enduden.com (Michael Muller) Date: Thu, 10 Feb 2000 21:24:12 -0500 Subject: [Distutils] Package-meta-info-ala-ConfigParser patch Message-ID: <38A372CC.CADC3C36@enduden.com> This is a multi-part message in MIME format. --------------692BE47D545CC81215FB9849 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Finally managed to scrape together enough spare time to complete this... This is my original meta-info patch with the pkginfo module altered to read and write ConfigParser style files. It was more difficult to do than I expected, largely because it needs to deal with some complex data structures - I think I might as well have done a complete parser from scratch. However, I must admit, the RFC-822 style is even more readable than the original python data structures. Enjoy. ============================================================================= michaelMuller = mmuller@enduden.com | http://www.cloud9.net/~proteus ----------------------------------------------------------------------------- Government, like dress, is the badge of lost innocence; the palaces of kings are built on the ruins of the bowers of paradise. - Thomas Paine ============================================================================= --------------692BE47D545CC81215FB9849 Content-Type: application/x-gzip; name="Distutils-0.1.3-pkginfo-patch-2.gz" Content-Transfer-Encoding: base64 Content-Disposition: inline; filename="Distutils-0.1.3-pkginfo-patch-2.gz" H4sICLVuozgAA0Rpc3R1dGlscy0wLjEuMy1wa2dpbmZvLXBhdGNoLTIA1Tz9c9u2kj/XfwUq NyMpohUrdr70xte+afJucpPk5dL00hnbp6FISOITRbIEaVvX9n+/3cUHAZCyZadznVMmokgu FovdxX4BcJwsFuyo/FCz14mo6ipJxdHxeDI+eRLr+ydRvtmEWfwkyUQVpum42PrAR0VYRSse 39ro4Ojo6F69fPNTnbH/CDN2cswmJ9OT0+nklD09Pj4+GI1GDyLhm8+rmv2Dz9nkGBBNAekz hfGHH9jR01fHwXM2wssr9sMPB8z+CJ4uxmWdzQrOSzboK6SzYtsfNpDJQgIiEWUyBzrybMxv qtkmj+uUi+l+SKEFYB1psObXISt5lJcxq1acKegQO4GbRd7AOYgbvAiE5LYJBu6tZosk5V0U RiUPKz4zMGwwPIjvrzg4rAcpj2p4PwVSjf5kJdJYbUWaTI+Pp09fNIr0InjBRviFSjSblZFI 4tmMnbHed2/jKXNRBVdsMpYkPTmePEEyXyoyl9chiPrNTcG+66HQFmW+YYa0MSgCKMGmyMuK /SjJPDjyYPBbw0R5sZ1VJecHo32gAviZXfGyIsELJCBKQyHYW0n/m5uKDVS/Q1AbGvrJy+AZ G8H35IUzhQ5ZUVdVki0ZKpAAHpAKl8lyVbEijNbhkgNBJUsEC1PQt3jL4jzj7HrFM3bNbUzz OknjcfMkryuJ9Eypqx4BG9A9wc8AeSDfa/7DE5hk9uRa8orISoEzLF+4tBKaACbgEqQCzwVC AHnwrmR5ybK8snFdcxZGVQ0dbZGnCadJu2lAFBnwXNGNvXbS7VFZ8iINI94QhTCsKPkiuWHX SbWyrYP1rrtrR8h3Mixo2u4wUJqHts1hahYx+h875guIkGyucpZUNo3QTnFmkWSxbyOVLXPg x4B71uAddNIK/w5bmvwge0ZdPsigqZb3tGiq1TdfYGyvecROJmzyCo3PMRiNV89fPdCkabS3 Osfj4JiNJsHzCU7rUa/Xs+0Q4Rvb+FBjlW0gwcpXJHc92xGs3JD3GgM+bLGHgRup+1yAcm4F tjpkVbmlXuSrkC3TfB6mrFesl9hLj0n3G6C7y8CqBEZLY74I6xTQZWAVAc1UKorCpNofjPhN xIuKvaXHb8oyLxWgS3Gr3cHIUbS38NAymWpW5wXyAM3X+aAPTY8ktWf9gPUL/PoQbjgaG825 qoQfaEslIA69Frw/vCR8EikMDFxPkiXVbEazOiAyh9NmJihCxl1gXihhG4zzy2bKN305M09h onlt99jC5T0YKVNAmBvcglczJaiZYhb10EINzFORFuD+QDJ1qURMYEzCdCeeAsTlt4JAqgUH GvulTCq0W55O0xgY6Lb9Rmt4YydBFfE9CK5kouBRskAXEWqZxmEVzkMBqgoQYBjHlqVb+KO1 6MJPo8f2B8RM2gmRyMBrP2xDF0nDSZhnYxXAnnstL92WO+eJ/SnDBIZk7NR7QkTAAbtogyt+ J9lVmCaxy2pFYv+R6PfYI58tDS6eCp9J9gDNfNUvuxxaUYM1KrbViqJtK/QGt8ijKgemAxzE L4mwTYuNKs0j2cSazEZlRMs/k+9WkwSoIA+oFNdJQMBEWC1sp6hnjen4DAwnBfvjf+VJNnA8 e3+m+NCde6Bd0NTY+Y2r1jJXAD3+KMeHJk+OJ4vIUs3rDKInht4gXziTx5kdjXNohQRGcmO7 j0GGVvKMyBzj72CXKvkfCH+E5A21Vbd7N3diTzvisBnT0CwTuJkatAyPBr6cAhprwPalQXYO lM9zNBn3axaX2xmYuIfldMX2YREQtbtn/ENtOhK6Z1+T0EmkrXzu5GkT+5xiPncavKSM5kA7 ebCKEILANMiWf3Vq1gRFrSzt47adpD2bBDDO0bOnwdOnXpb2PlxzJtDSQf6yCq8462E2UPVo wpIJ/ChNoHIIoP6lqA5cjbJKGjKXkFWS7jRhYcJDXt4R/rejf5v21/WmYDyrwBwjfyr4KbSN ITIaS42GyC+h2KgawEG1Cqu+tOhzHtbVFlECZzD+Cn283w7/pLT0qGOy3u6eVBIF1ILoSju9 GjwkZY0TDEaQ5mHLL3WkrP9/M9Z23rgjbWyGM/gn+WDk0xBT2wIDPjSDoFVwiWzYX375ha34 9ltkbhRm/YpUs8xTIwd4gT59k/wPVzL5G1NKVxc2KqOzV1YYgU+iuixB2fXMTED1S2BghRMQ 1CHlIcxQKkyd0sR/+SIA4+lXOHd9evNtxY/kMFHpHwkMtRbB/Vsboh8JJtZJUfCYUA27UXV7 MZ+5yTJDY0tiN3xcECOvQRsSDDuAxXGOvIfgBJV7De8gjLZRXfM+mQ09SkJFwFImAdUuorxO Y0RUJNEaJAjcBanEuY2JX/EMY3TsFCYR9jVm7G9HA9cAIu0ULhYhWG64phCzTBXBFBaV/CrJ a0HFE7Jlol4uuahIO7x5K4enxmVsN+gikl9sVYSJN/O8WgUIsUR1KvN6ubJxYc9guYq6sowg Dl1wLgcMnLrC1hYJyO4F1snMzPGCbaBpcX50Mr1kZ2esD3Ok35EXdMxEkupshsHvVVgmoczZ FufTo5PLe4Q57c8AicgxcsYJ2x8OjS85xHQPK9xWAtqiQOa3Mj8T9WKRRG6iC9neJ17VJWT1 oakmogflN3DTlEHJ9oWEiGUqy+/jXd8ZmyydQWPwaUJW+1lf99uHSQ7S+bXmqDmAwICJoUw7 LUzNjRQyVR0uXUkS4hvsQ3fh55fgZ88k0SMF3RK3TjNowALc6KYYdshcUTEOwRjAIAmsgSqJ hxqoo3L3cbtn0FryfcNUgrwzMCUoPxRtosZ9Q1GJ5tbC26vTl7gqhZfJZB+jbacC7N/OwPxf 8fRua+2mAlb1wytLBwzlimWiUpwdYxFuAwZzLc4m3gR4lwgZmdixWGjHYIATLONbUHlE1kcj KDhYWuzRURVRz3WzhAsq4/V1v24zJGeep0nE6KWr/y3FQjHYZfdmZM24zHIZ8mKTX3G1BCZ5 IcoIGgjsHdlMTGAWE95DAxg0NSk51nlo9mvx4MQGnh+hyVmk4VISvI9Kq1R9P61ugO9Q7Abw a4vMFqZ96sonJy/ahWWrjPpa1Wk7qj+gQp/Rj+qEBBePBHH2WhbomvqCVVVgoWiWd0K7hmMD oTkMm07NWhBCHIxIn0m0JoYVaMTJkoduZRAnQZhkKHxVQyJ3Mk+RLlAtLMFSFVgXDsNKVli4 8OokY+THe519AXROYWRHWZ1B1gePSpgfKwxcqJQoS1QYIi5CWZE5GF3n5VqOoWG+KliqzBKf ObX3kuvkN4AMN1sky49hKXipqvKUxVbbgpua+GPENuMiCgv+6Qb8B9g+FTsPyv5F9ftF9vvF BeZ0BugzseeM/cbgfX8K33AJ8CJv4Ip3GdxhsfcP6gGcBS+/iHYXgwsxGv4+GD+G63dD6ggi 45iXHdRkAEMQwJm44/1P9FZX92f/WUM4+I7fKAPYo3s5uZMSMsaU3/By3LPryrPZdQeNqlsJ AOang1OD8/++yC4fD7/roxDG74d+tdor58/BP/OyVSqXj9GN0w8fCehBBDnjlxXOIFAt3iqA b3QOKEcy3qAJGFi4h06WBbbTc/8uGdbd+WYMscBgOL30qYIo7AsIpLv43yZ56FKrhNlB6a10 VvmaY3FwM15CtFwMhg8cBn466tAafbNcYTkoetnBhXegGrdJRKrO1470+C8YqplTP4HdSPlP ZGKmzsKWiHArhTQ+0mplOeObOY9j8FkZv8bRSw8AL6oyVBldiZlwjD979jpD79pojJqiDavB TyeLrZpJV2Fqs1vTDUZugK/QPUhyP8Mj6t5bzZAkUzkLW0jbNcSU6GiiyK38No09U9LErjwq yc8pIgG8RSk8G0uYjsboLm8dIDy1xIKah5Hd1JWGznHkEEWnWJiUC/o3SyauLLSEHigTWt+t XJkgtSgRTxnV6I4tK4LJkKN39lxHTws0bTAEANTtBBc7FmKsyEPQroSn3W3HRFFAk3vKGYho U+YI//z40hqSdH8waHR4VX/nWM8n08udWCWS4a7XxIe9Va5zBIobxluPRZEmVaPLd3AR01uj vug+7lbfgPEwWsliaQI/VPAmUKP/Cs3cS/dc9/Yn6d9DjIwyc7TAiED3tjgKgS1mI8DXvPhc F2bJWwuwwmd2kK0lqlbzpDsI2c+f3vlhmNIInPtaOwYmBhOWNdhlHfZWAEf4KDIayV4awFNo n3LJUPbtGXu6XxMySipKU4PTFkqZg3vjsVli45rshet2W/cgp0bW6wKwpJAVDLsgFHm3NNZq 4mLwOSebaGoaY7qPbkMOgAqkUwXPfJF0b8Y6wCUR93FEfqFUbp74rzCt5ZaJQZ8uWEUWqKMR 5WJAC9bTeBZtcUUM01q3otjxwe4kMOM3WKjgcYBlXtZ7JHp99ujOcqs7ouYX4j0zkxouxUCN lcLYYfeyTBdDjJT+D7liDMjXs6aBUkiV3bELsOBewJBNvPE1WxQcprT8gILrsPtWj6rYOlCP PCwdUxQ/c9DstT0EHR02eAOUtGuqXyeR72pjeISLZ8DLDRBCBRFVusC6iWAD6XM3tai0QUe/ uytmH2IJI2QtBxC2XYCqo7iYdOey1+skTZkIq0QstmoXj1aZoHEhuHZX8luQRWHG5pzlc6SG x37qv49Xmc1i5eoQQru9r/c5KJI9XI4bfKwBtQo+xhhZiIFv6+/2Eev9wpE2Ms0H42z+nLj6 Hr5mkZv6WqB3LNzCC8+79HcGx32F1PU6LowC6UrFLdZYngnpu49foqTs7I7wWiu2n7yr2j7m EJYpBbIQ5W9/+HaNOvO4hc/QsOErcKnuy5inO964d4d6HZhvCiy4YZNunXLcEIB1KROOKslq fluPVNNzPDpha3Uqa3+NH8MFUC32rlXQ9pNDtaSrz0XQ0i4uB6JDCsjHcWfrvNW7kk9HP/hB OZ0rYi6b6pHRKtIahaJjj2g3uSUnp2bGLZ18J3UoEATfQZ706p0OnTRM+fA3JHK1F5DhZml/ NuGn0WBHBbD7DuhuvcaPCYe1EDs4sA/dPo07/K5FyCNxkT0SGG5omQTM07nm16HSi5QvqqMc LKfs1fEN3crxQKVQthVba+OjIwHJiTe0NxgDDl0m7/U+IadiucySbY2MaE1Fj5xja7MeITFa Gz+ntqkDoB8JAGICUaE96DptwH6UG2bSLRN1gasTQENRz3HZ0AxIL89A5lhagjE/MFjxtxRb cUyDpzMk06A6sMvqzdza/kYrUy07qQ4l1Wm6Pfq1DlO5ZRwXLWXkpPYYqDDH2hAGI/C7Lytv WckAoAR3UAuvYhkNJVzodTc8scWsqA7QOitSnh3WBIHC/FonpZJ+6zwjEZeYxTvqjLM13/pm XS7IWf2rta45d9bfiEMBPbrCNEGvibnYsBEqubAjSTqopol1BaZCQr1t1/PRMr7DLVg0lCrZ QPhAAeSw2c6PKzth5XNcPk1w/UuHr8CCvxslgDg1xtUOU3DF8LpM5rgZ1Q0kSBRuCNywVUqD nuiBJcLq3MWFRd1mRV1+4zIl+fo+crg/ZYN+P/Aj2qG/mVpnVQT//fff79OE9BkbnF8GTR26 CxI1GAF/+yMwWUgXnGK9wdmUf4Z/+AGUt6r2+PH62o4cbMN7HaZrvbWKlAe0FksIoPXLmnZy dQfX6+sd8aQf6mzMJl0SX1LJOZjyZZg268a7Q2qS2XgVihlQtiMuJxf2d41LWvD18F50mbmm LY0hjba7IbV30Hi+vjyfXN4e97cy/7dYKMXEXKaQc9x/QW7zriQdP4N10O4f59bw1rELXmED pRwo0B1OGQDNmTNlh8j5pam94C+5UGnjZunPDtWxxHpregYyt+nsYmh7KLLO5WVL/8yafKnr XNSuo35yu4W8BV1JKlrjx7E4pNhbCsDw9aGXvmXZ1baLfP4vHlW7NvlYeda52kpyiQtuHgs9 /qGLabEPRAU2bumyZnda9wjriBCn+SAtzXILirsRumTThhHWcZBk5zkStd/obBIwtcXr7NgO wwiV6AyWPHvfOcv1wSFrc5fc9yfb0DarzRoDla6jLpo4c3xFk2h2o1n9YzdyB5Z7lGnXGRo6 CuRiWOge7WN/4HUq1kN2g/vpBU0/XlOcRYquqT2x84Jn3UcBrZDbph3hB+ZJwPrXfkZgozfb mVpnpMd6OhpcNFlp514eevpB56K6ToVhVu8ohN682nmMyzvcqO37MsGtz9hHE960+nLDs9Tb ym79KQC93a/B9VYFL9KpwI8PdJZX8LCMVvokptxcRXrhqS5Iz6fGrvGo82PO+bi2TjWH5AJm qZaXvcmCUbV6l0dkXrYSqWdVOrtUzXb2ZGTf7LFNBIp9QOi6TLqq3erbrsKYnbH+nJFNrlB9 Qe1s402clRLSpe8O0uRdtFD2G0bomHP7RhcGDLBMLxcWVshVNnY1ifwl2NCmEmfp6E4n2D4T S3jPER6zXMSFFllmt4YaMPl0JlvJAaHtSECdd3WG9yGXx0PUiVazK6zpV58wNoUEIMSEYzKO dlZFbeF8MllTwmEa9RC6R2iSzMmfx33jtFVqHhZFuh1YzArYYBjI7vc7/EeGfK8dpwryju2m CsrZR306ffZievryPvuoNRrvEN/p9NT6oywnz09wHzVeTk7USb5DZh3Kok3P/oZn2t67c6fz PXY5J/4uZyPflDbIyyTWGD0Kk+KY9oSqgz9JafpQu6bNFmiDy94KLXbuhZbf7QMA8tudKbKy QDYKkeLZLOSKNntS/+Ell9o+gGsGigUXyEzbWmyE+Y9k95nvHiGjQTh/MQbDKnn6SApGdaLV nA79IBzRbG8PLKNZh6lVWOwFMDxar5hKLrexsPiMGsy6rKx/jMHA2VbXsdiakZ3o0FOBruxx XqKjI1e2+tfIU+uZCgW8ve63OYk7+nZsjT6yod4cNr3TPFPnn95CHEl74s3B+Z+z5OZI/TmE iIHxyDiWuPKSNGHF0+Lbg9ZOfL0HXx7WPXkRnLDR6fPT4GlzyG1B+pRhWGZmkbVTfyGjKEY1 fOBV7yLrqVNpi3GU4lb9gQmrnPOHVvfu37uA2fbvFA81C5GqhBHo85By1ou8LiNuWQzv7KPZ Ww5pKHDB+5sDCvrOMB2PfemuaGOyhUwOBGWqziMJXvAyBPQamcj1yVJa2uShSNKtOdZJf6dJ U0kFJ9kO0MxSLletYbaB8sOTofF9wLjzIwUzvcQFfgnhzlysQMP3SL1sQjlR3dU6poUI/HZb 61HZx1KxD/9AKra0haF0exMWoCjhZh6H7AZigk11hjtz0CpCTBKfQbMpXKHTm3N4Oe08qSZ1 BZ8fXEAcq7etQoRBglB/fOrgfwFHBl0ZoE8AAA== --------------692BE47D545CC81215FB9849-- From gward@python.net Fri Feb 11 02:19:11 2000 From: gward@python.net (Greg Ward) Date: Thu, 10 Feb 2000 21:19:11 -0500 Subject: [Distutils] install w/o build, spaces in directory names In-Reply-To: <002301bf7400$1c3a2d60$c355cfc0@ski.org>; from David Ascher on Thu, Feb 10, 2000 at 11:50:41AM -0800 References: <002301bf7400$1c3a2d60$c355cfc0@ski.org> Message-ID: <20000210211911.A486@beelzebub> On 10 February 2000, David Ascher said: > 1) While building Numerical on NT last night w/ the latest Distutils CVS > snapshot, I found that it refused to build because the directory names had > spaces in them which were not quoted. In other words, the code being > executed when something like > > cl .... -IC:\Program Files\Python... > > which broke. To correct it, I patched ccompiler.py and msvccompiler.py to > turn those kinds of things into > > cl .... -I"C:\Program Files\Python..." ... > > but Greg says that that will break on Unix -- I'm surprised by that claim, > since IIRC quoting directory names is OK, but it's been a while since I > really dealt with Unix seriously. FWIW, so far the problem occurs for -I in > ccompiler.py and /LIBPATH in msvccompiler.py. The patch is listed at the > end of this message. Here's the explanation that I didn't give David this morning in private email (I was at work, where I'm not paid to hack on Distutils; and I figured this deserved a public airing if it's a portability problem): in UnixCCompiler, I do stuff like this several times (ie. any time I ask the compiler or linker or archiver to compile, link, or archive something): self.spawn ([self.cc] + cc_args + [source, '-o', object] + extra_postargs) MSVCCompiler has similar code: cc_args = compile_options + \ base_pp_opts + \ [outputOpt, inputOpt] self.spawn ([self.cc] + cc_args) (The two could be a lot more similar if I went crusading through the code to enforce uniformity, which I'll do as soon as that next shipment of round tuits arrives.) 'self.spawn()' in both cases is a convenience method provided by the base class, CCompiler; it's just a wrapper around 'distutils.spawn.spawn()', which in turn is a wrapper around either '_spawn_posix()' or '_spawn_nt()'. Both of these have one required argument, which is a *list of command-line arguments*. '_spawn_posix()' forks and, in the child, 'exec()'s that list of command-line arguments. Key point: no shell is ever involved, so no quoting need be done; indeed, no quoting *can* be done, as the command-line must be split up into a (Python) list of arguments before you can do anything with it. This is a feature, as anyone who has ever done shell or Tcl programming can attest. (For those who have not, both of those languages have very fuzzy notions of the difference between strings and lists, and rather weird quoting rules. Compared to quoting strings and lists in shell or Tcl, Perl is a model of clarity. Python is a smidge clearer still than Perl, but not quite as flexible. But I digress.) Bottom line: on Unix, the above 'spawn()' call resolves to: pid = fork () if pid == 0 exec (self.cc, [self.cc] + cc_args + [source, '-o', object] + extra_postargs) No shell, no quoting. (The duplication of 'self.cc' is deliberate -- it's an obscure feature of 'exec()' that Python faithfully duplicates that allows parents to lie about the name of their children. My 'spawn()' interface doesn't expose this feature.) The picture on Windows is a bit different, since of course there's no such thing as fork-and-exec. '_spawn_nt()' just uses 'spawnv()', so the above code from MSVCCompiler resolves to: executable = search_path (cmd[0]) # search_path not shown here spawnv (os.P_WAIT, executable, cmd) I have long suspected that there are some command-line quoting rules in the version of DOS underlying current versions of Windows, but I have never been able to discern them. If someone would care to offer up an explanation to this confused Unix hacker (or a pointer to an explanation), I'd be glad to hear it. Anyways, on reflection it looks like the place to make the fix is in spawn.py, specifically in '_spawn_nt()'. In particular, somewhere before the call to 'os.spawnv()' should be a loop like this: for i in range (len (cmd)): if needs_quoting (cmd[i]): cmd[i] = quote (cmd[i]) I'm sure the implementation will need to be a bit more elaborate than this. For instance, how do you quote a string that already has quotes in it? I can't fathom a guess, since I don't know the quoting rules on Windows. I hope someone can contribute a patch. For inspiration, here is the equivalent for Unix in Perl. No doubt similar logic, but different details, will apply. =item shellquote (WORDLIST) Performs the opposite of the F module, namely it joins an array of words together, with some sub-strings quoted in order to escape shell meta-characters. WORDLIST should just be a list of substrings, not a list reference. This is useful for turning a list of arguments (such as C<@ARGV>, or something you're about to pass to Perl's C) into a string that looks like what you might type to the shell. The exact rules are as follows: if a word contains no metacharacters and is not empty, it is untouched. If it contains both single and double quotes (C<'> and C<">), all meta-characters are escaped with a backslash, and no quotes are added. If it contains just single quotes, it is encased in double quotes. Otherwise---that is, if it is empty or contains meta-characters other than C<'>---it is encased in single quotes. The list of shell meta-characters is taken from the Perl source code (C, in doio.c), and thus is specific to the Bourne shell: $ & * ( ) { } [ ] ' " ; \ | ? < > ~ ` \n (plus whitespace). For example, if C<@ARGV> is C<("foo", "*.bla")>, then C will return C<"foo '*.bla'">---thus turning a simple list of arguments into a string that could be given to the shell to re-generate that list of arguments. =cut sub shellquote { my (@words) = @_; local $_; for (@words) { # This list of shell metacharacters was taken from the Perl source # (do_exec(), in doio.c). It is, in slightly more readable form: # # $ & * ( ) { } [ ] ' " ; \ | ? < > ~ ` \n # # (plus whitespace). This totally screws up cperl-mode's idea of # the syntax, unfortunately, so don't expect indenting to work # at all in the rest of this function. if ($_ eq "" || /[\s\$\&\*\(\)\{\}\[\]\'\";\\\|\?<>~`\n]/) { # If the word has both " and ' in it, then just backslash all # metacharacters; # if it has just ' then encase it in ""; # otherwise encase it in '' SUBST: { (s/([\s\$\&\*\(\)\{\}\[\]\'\";\\\|\?<>~`\n])/\\$1/g, last SUBST) if (/\"/) && (/\'/); ($_ = qq/"$_"/, last SUBST) if (/\'/); $_ = qq/'$_'/; } } } join (" ", @words); } ...of course, the version in Python for 'spawn()' would return the modified list of strings, rather than joining them together into one big string. (The purpose of my Perl function was to print out a script's command-line arguments in a way that could be replicated in future: if you just print out join (" ", @ARGV) -- or join (sys.argv, ' '), choose yer poison -- then you lose vital information. > 2) Again on Windows, if I do > > python setup.py build > python setup.py install > > the second line causes the code to be recompiled, even though the > compilation is not needed. Greg assures me that that is not the case on > Unix. Anyone have the time to tackle this? Aughh! Again David, I apologize for brushing you off this morning without looking at the code; as I do so now, I see that there is *no* provision in MSVCCompiler for doing the kind of timestamp checking that is done in UnixCCompiler to avoid redundant compilation. So, to correct my arrogant assertion: that feature was not there in the beginning, it isn't there now, and it won't be until someone writes the 10 or 15 lines of needed code. This shouldn't be too hard, but I'm off for a weekend skiing in West Virginia tomorrow (woo-hoo!) so I won't be able to do it until Monday. Anyways, it looks like the priorities for Windows portability are: * fix '_spawn_nt()' to quote command-line arguments that need quoting (bad bug, breaks builds) * finish the job of incorporating Robin and Thomas' registry- grovelling patch (this comes first only because we're sort of in the middle of it) (bad omission, totally prevents builds in certain cases) * fix MSVCCompiler so it doesn't do redundant compilation/linking (minor bug, just wastes time) (but potentially lots of time -- NumPy is a lot of code to compile!) And again, I'm going to be away for three days, and I do not have the required knowledge to do the quoting fix anyways. As usual, Patches happily accepted! Thanks for the bug reports -- Greg -- Greg Ward - just another /P(erl|ython)/ hacker gward@python.net http://starship.python.net/~gward/ BE ALERT!!!! (The world needs more lerts ...) From gward@python.net Fri Feb 11 02:56:13 2000 From: gward@python.net (Greg Ward) Date: Thu, 10 Feb 2000 21:56:13 -0500 Subject: [Distutils] Stylistic comments In-Reply-To: <037801bf73fe$72d28570$4500a8c0@thomasnotebook>; from Thomas Heller on Thu, Feb 10, 2000 at 08:38:45PM +0100 References: <20000209215010.A611@beelzebub> <037801bf73fe$72d28570$4500a8c0@thomasnotebook> Message-ID: <20000210215613.B486@beelzebub> On 10 February 2000, Thomas Heller said: > As Robin already said, this had been fixed some time ago, but it > somehow made it back in my version. > I have to admit that the patch I sent last night was not the latest > one Robin and I had agreed on. It was the newest one I had on > my harddrive though... Oops! Oh well, it happens to the best of us. > Attached is another msvccompiler.py (zipped), > and hopefully Robin can agree on this one. > I've already included the doc-strings Robin has > supplied. Merging in the changes now. (And let me just say that ediff -- a super-duper-ultra-mega-diff with file-merge capabilities for [X]Emacs -- is *way* cool.) One question: you took away the bare 'except' in '_find_exe()', but now don't catch KeyError from the 'os.environ' lookup at all. Bad thing? OK, I've merged in the changes. Actually, it turns out that I accepted all the diffs, so I am just using the version of msvccompiler.py that you sent me -- good. I made one additional change: added a docstring for 'find_exe()' (renamed from '_find_exe()' because I don't buy the need for underscores -- nobody is doing "from msvccompiler import *"): def find_exe (exe, version_number): """Try to find an MSVC executable program 'exe' (from version 'version_number' of MSVC) in several places: first, one of the MSVC program search paths from the registry; next, the directories in the PATH environment variable. If any of those work, return an absolute path that is known to exist. If none of them work, just return the original program name, 'exe'.""" Correct? There, I've checked in those changes -- rev. 1.13 of msvccompiler.py. And finally, I can see no reason for the '_find_SET()' function to exist at all -- both calls to it can be changed to 'get_msvc_paths()'. Anyone have anything to see in defence of '_find_SET()'? ...OK, I've removed '_find_SET()' and made some other stylistic surgery, now checked in as rev. 1.14. This is *completely* untested, as usual -- please try it out and make sure I didn't screw anything up. ;-) > BTW: What is the preferred format posting sourcecode to the list? > AFAIK, zipped attachments are not so nice in the archive, > unpacked source-code has the white-space problem in > Outlook Express, which I'm using here... For cases like this -- well-understood, thoroughly-discussed changes -- go ahead and mail me the code directly (gward@python.net). The easiest way for me to deal with it is if the code is directly attached, not in a zip file or what-have-you. If you're using a mail client that is incapable of dealing with "text/plain" (and this wouldn't surprise me coming from Redmond), I don't know what to tell you. Attaching a zip file works, but it's a bit cumbersome on both ends. > As soon as you extend the distutils framework to use > temp-directories for building, I will submit a patch for msvc > again. That'll be a little while -- it's lower priority than fixing the "dist" and "install" commands, and definitely lower priority than writing the "bdist" command. As long as building on Windows is *working*, I'm not going to get *too* uptight about compiler turds being left in the wrong place (although it is something I want to fix). Happy hacking all -- pray for snow (at least in West Virginia!) -- Greg -- Greg Ward - just another /P(erl|ython)/ hacker gward@python.net http://starship.python.net/~gward/ Eschew obfuscation! From thomas.heller@ion-tof.com Fri Feb 11 13:07:11 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 11 Feb 2000 14:07:11 +0100 Subject: [Distutils] install w/o build, spaces in directory names References: <002301bf7400$1c3a2d60$c355cfc0@ski.org> Message-ID: <01b301bf7490$e951d470$4500a8c0@thomasnotebook> > 1) While building Numerical on NT last night w/ the latest Distutils CVS > snapshot, I found that it refused to build because the directory names had > spaces in them which were not quoted. In other words, the code being > executed when something like > > cl .... -IC:\Program Files\Python... > > which broke. To correct it, I patched ccompiler.py and msvccompiler.py to > turn those kinds of things into > > cl .... -I"C:\Program Files\Python..." ... > > but Greg says that that will break on Unix I tried quoting the parameters for _spawn_nt in the following way, which worked for me (after changing some paths to include spaces, which I usually avoid). Not sure if cmd[0] should also be quoted: *** distutils\spawn.py Fri Feb 11 13:40:10 2000 --- c:\python.org\distutils\distutils\spawn.py Mon Jan 17 22:57:56 2000 *************** *** 42,60 **** # spawn () - def _quote_if_needed (arg): - """Obscure quoting command line arguments on NT. - Simply quote every argument which contains blanks""" - if string.find (arg, ' ') == -1: - return arg - return '"%s"' % arg - def _spawn_nt ( cmd, search_path=1, verbose=0, dry_run=0): executable = cmd[0] - cmd[1:] = map (_quote_if_needed, cmd[1:]) if search_path: paths = string.split( os.environ['PATH'], os.pathsep) base,ext = os.path.splitext(executable) --- 42,52 ---- Thomas Heller From thomas.heller@ion-tof.com Fri Feb 11 13:20:23 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 11 Feb 2000 14:20:23 +0100 Subject: [Distutils] install w/o build, spaces in directory names References: <002301bf7400$1c3a2d60$c355cfc0@ski.org> Message-ID: <01b901bf7492$c14c6600$4500a8c0@thomasnotebook> From: David Ascher Sent: Thursday, February 10, 2000 8:50 PM > 2) Again on Windows, if I do > > python setup.py build > python setup.py install > > the second line causes the code to be recompiled, even though the > compilation is not needed. Greg assures me that that is not the case on > Unix. Anyone have the time to tackle this? I have patched msvccompiler.py to do this (patch is at the end). The problem is that this is currently not safe because of the debug flag recently introduced. If you do setup.py build_ext setup.py install everything will be fine (non-debug extensions will be compiled, linked and installed), if after that you do setup.py build_ext --debug setup.py install the debug extensions mod_d.pyd will be linked from obj-files compiled with the wrong flags. The only thing you can do currently is to force a rebuild with: setup.py build_ext --debug --force setup.py install This should either be fixed by: - compiling the debug and non-debug object-files into different directories (temp-release, temp-debug) - naming the object files differently (mod.obj vs mod_d.obj). Thomas Heller *** c:\python.org\distutils\distutils\msvccompiler.py Fri Feb 11 11:56:18 2000 --- distutils\msvccompiler.py Fri Feb 11 12:08:54 2000 *************** *** 14,19 **** --- 14,21 ---- from distutils.errors import * from distutils.ccompiler import \ CCompiler, gen_preprocess_options, gen_lib_options + from util import move_file, newer_pairwise, newer_group + from copy import copy def get_devstudio_versions (): *************** *** 222,239 **** compile_options = self.compile_options_debug else: compile_options = self.compile_options - - for srcFile in sources: - base,ext = os.path.splitext(srcFile) - objFile = base + ".obj" if ext in self._c_extensions: fileOpt = "/Tc" elif ext in self._cpp_extensions: fileOpt = "/Tp" ! inputOpt = fileOpt + srcFile ! outputOpt = "/Fo" + objFile cc_args = compile_options + \ base_pp_opts + \ --- 223,261 ---- compile_options = self.compile_options_debug else: compile_options = self.compile_options + # So we can mangle 'sources' without hurting the caller's data + orig_sources = sources + sources = copy (sources) + + # Get the list of expected output (object) files and drop files we + # don't have to recompile. (Simplistic check -- we just compare the + # source and object file, no deep dependency checking involving + # header files. Hmmm.) + objects = self.object_filenames (sources, output_dir=output_dir) + + if not self.force: + skipped = newer_pairwise (sources, objects) + for skipped_pair in skipped: + self.announce ("skipping %s (%s up-to-date)" % skipped_pair) + + # Build list of (source,object) tuples for convenience + srcobj = [] + for i in range (len (sources)): + srcobj.append ((sources[i], objects[i])) + + # Compile all source files that weren't eliminated by + # 'newer_pairwise()'. + + for (source,object) in srcobj: + base, ext = os.path.splitext (source) if ext in self._c_extensions: fileOpt = "/Tc" elif ext in self._cpp_extensions: fileOpt = "/Tp" ! inputOpt = fileOpt + source ! outputOpt = "/Fo" + object cc_args = compile_options + \ base_pp_opts + \ *************** *** 245,252 **** cc_args.extend (extra_postargs) self.spawn ([self.cc] + cc_args) ! objectFiles.append( objFile ) ! return objectFiles # XXX the signature of this method is different from CCompiler and --- 267,277 ---- cc_args.extend (extra_postargs) self.spawn ([self.cc] + cc_args) ! # Have to re-fetch list of object filenames, because we want to ! # return *all* of them, including those that weren't recompiled on ! # this call! ! return self.object_filenames (orig_sources, output_dir) ! ## return self.object_filenames (orig_sources) # XXX the signature of this method is different from CCompiler and *************** *** 344,352 **** if extra_postargs: ld_args.extend (extra_postargs) ! self.spawn ( [ self.link ] + ld_args ) # -- Filename mangling methods ------------------------------------- def _change_extensions( self, filenames, newExtension ): --- 369,391 ---- if extra_postargs: ld_args.extend (extra_postargs) ! # If any of the input object files are newer than the output shared ! # object, relink. Again, this is a simplistic dependency check: ! # doesn't look at any of the libraries we might be linking with. + if not self.force: + if self.dry_run: + newer = newer_group (objects, output_filename, missing='newer') + else: + newer = newer_group (objects, output_filename) + if self.force or newer: + self.spawn ( [ self.link ] + ld_args ) + else: + self.announce ("skipping %s (up-to-date)" % output_filename) + + + # -- Filename mangling methods ------------------------------------- def _change_extensions( self, filenames, newExtension ): *************** *** 359,367 **** return object_filenames ! def object_filenames (self, source_filenames): """Return the list of object filenames corresponding to each specified source filename.""" return self._change_extensions( source_filenames, self._obj_ext ) def shared_object_filename (self, source_filename): --- 398,407 ---- return object_filenames ! def object_filenames (self, source_filenames, output_dir=None): """Return the list of object filenames corresponding to each specified source filename.""" + #XXX output_dir currently ignored return self._change_extensions( source_filenames, self._obj_ext ) def shared_object_filename (self, source_filename): From thomas.heller@ion-tof.com Fri Feb 11 13:43:46 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 11 Feb 2000 14:43:46 +0100 Subject: [Distutils] install w/o build, spaces in directory names References: <002301bf7400$1c3a2d60$c355cfc0@ski.org> <20000210211911.A486@beelzebub> Message-ID: <01e901bf7496$05481630$4500a8c0@thomasnotebook> [explanation why quoting arguments breaks distutils on unix] > > 'self.spawn()' in both cases is a convenience method provided by the > base class, CCompiler; it's just a wrapper around > 'distutils.spawn.spawn()', which in turn is a wrapper around either > '_spawn_posix()' or '_spawn_nt()'. Both of these have one required > argument, which is a *list of command-line arguments*. '_spawn_posix()' > forks and, in the child, 'exec()'s that list of command-line arguments. > Key point: no shell is ever involved, so no quoting need be done; > indeed, no quoting *can* be done, as the command-line must be split up > into a (Python) list of arguments before you can do anything with it. Neither is a shell involved on NT, so the shell quoting rules do not matter. From a little investigation, it seems that MSVC can use -I"c:\Program files" as well as "-Ic:\Program Files", and this makes the whole business somewhat easier: We do not need to quote the directory, we can quote the whole parameter if it contains spaces. See my other post which contains a patch to spawn.py. Don't know however what happens if the argument already contains quotes... Thomas Heller From robin@jessikat.demon.co.uk Fri Feb 11 13:23:52 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Fri, 11 Feb 2000 13:23:52 +0000 Subject: [Distutils] install w/o build, spaces in directory names In-Reply-To: <01b901bf7492$c14c6600$4500a8c0@thomasnotebook> References: <002301bf7400$1c3a2d60$c355cfc0@ski.org> <01b901bf7492$c14c6600$4500a8c0@thomasnotebook> Message-ID: In article <01b901bf7492$c14c6600$4500a8c0@thomasnotebook>, Thomas Heller writes > ... > >This should either be fixed by: >- compiling the debug and non-debug object-files into different > directories (temp-release, temp-debug) >- naming the object files differently (mod.obj vs mod_d.obj). > >Thomas Heller > ... I thought Greg wanted the dual directory solution. -- Robin Becker From thomas.heller@ion-tof.com Fri Feb 11 15:53:17 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 11 Feb 2000 16:53:17 +0100 Subject: [Distutils] install w/o build, spaces in directory names References: <002301bf7400$1c3a2d60$c355cfc0@ski.org><01b901bf7492$c14c6600$4500a8c0@thomasnotebook> Message-ID: <028201bf74a8$1d27ab50$4500a8c0@thomasnotebook> > >This should either be fixed by: > >- compiling the debug and non-debug object-files into different > > directories (temp-release, temp-debug) > >- naming the object files differently (mod.obj vs mod_d.obj). > > > >Thomas Heller > > > ... > > I thought Greg wanted the dual directory solution. > Yes, but he writes: >> As soon as you extend the distutils framework to use >> temp-directories for building, I will submit a patch for msvc >> again. > >That'll be a little while -- it's lower priority than fixing the "dist" >and "install" commands, and definitely lower priority than writing the >"bdist" command. As long as building on Windows is *working*, I'm not >going to get *too* uptight about compiler turds being left in the wrong >place (although it is something I want to fix). Thomas From gward@python.net Mon Feb 14 23:49:27 2000 From: gward@python.net (Greg Ward) Date: Mon, 14 Feb 2000 18:49:27 -0500 Subject: [Distutils] install w/o build, spaces in directory names In-Reply-To: <01b301bf7490$e951d470$4500a8c0@thomasnotebook>; from Thomas Heller on Fri, Feb 11, 2000 at 02:07:11PM +0100 References: <002301bf7400$1c3a2d60$c355cfc0@ski.org> <01b301bf7490$e951d470$4500a8c0@thomasnotebook> Message-ID: <20000214184927.A800@beelzebub> On 11 February 2000, Thomas Heller said: > I tried quoting the parameters for _spawn_nt in the following way, > which worked for me (after changing some paths to include spaces, > which I usually avoid). Not sure if cmd[0] should also be quoted: Well, this patch looks better than nothing. But I have a couple questions for the Windows crowd: * when I use 'os.spawnv()', just how many levels of software are involved (C library? kernel? the child program being run?) in parsing the command line -- i.e. in splitting the Python-list-of-strings up into the argv[] seen by the child program? In other words, whose rules do we have to work with here to properly handle filenames with spaces in them? * regardless of who's involved, what *are* the rules? Ie. what characters in a command-line argument will cause that argument to need quoting? What are the allowed quote characters? How do you deal with arguments that contain the quote characters (ie. what if I had a directory named Program "Files" ? * is any of this standard or documented, or does it vary from application to application? I'll wait to hear some feedback on this before I accept Thomas' patch -- I'm just superstitious that it can't be *this* simple: def _quote_if_needed (arg): """Obscure quoting command line arguments on NT. Simply quote every argument which contains blanks""" if string.find (arg, ' ') == -1: return arg return '"%s"' % arg (BTW Thomas, your patch was backwards. This is easy to workaround -- I just have to run "patch -R" -- but it's easier to read patches if you say "diff " rather than "diff ", which is what I suspect you did.) Thanks -- Greg -- Greg Ward - just another Python hacker gward@python.net http://starship.python.net/~gward/ Disclaimer: All rights reserved. Void where prohibited. Limit 1 per customer. From gward@python.net Tue Feb 15 00:20:59 2000 From: gward@python.net (Greg Ward) Date: Mon, 14 Feb 2000 19:20:59 -0500 Subject: [Distutils] Should I rename the "dist" command? Message-ID: <20000214192059.A837@beelzebub> Hi all -- it just occurred to me that since the "dist" command is only responsible for creating *source* distributions, it might make sense to rename it to "sdist". That way, there would be two parallel, complementary "make a distribution" commands: sdist - make a source distribution bdist - make a "built" distribution (frequently a binary distribution) (Well, there will be two parallel, complementary commands anyways: this just makes the names consistent.) Also, it reduces the small potential for confusion between the Distribution class and the Dist class. (The former is the grand central station of the Distutils; the latter is just another command class that happens to implement the "dist" command.) I don't see any big problems with this; it just means that the developers currently using Distutils would have to adjust their habits by one letter. Any thoughts? Greg -- Greg Ward - Linux weenie gward@python.net http://starship.python.net/~gward/ No problem is so formidable that you can't just walk away from it. From gstein@lyra.org Tue Feb 15 00:38:33 2000 From: gstein@lyra.org (Greg Stein) Date: Mon, 14 Feb 2000 16:38:33 -0800 (PST) Subject: [Distutils] Should I rename the "dist" command? In-Reply-To: <20000214192059.A837@beelzebub> Message-ID: "pack" "package" "assemble" "create" "makedist" optional flag: --prebuild (or whatever) RPM uses the term "build", but we've reserved that for something else. Cheers, -g On Mon, 14 Feb 2000, Greg Ward wrote: > Hi all -- > > it just occurred to me that since the "dist" command is only responsible > for creating *source* distributions, it might make sense to rename it to > "sdist". That way, there would be two parallel, complementary "make a > distribution" commands: > > sdist - make a source distribution > bdist - make a "built" distribution (frequently a binary > distribution) > > (Well, there will be two parallel, complementary commands anyways: this > just makes the names consistent.) > > Also, it reduces the small potential for confusion between the > Distribution class and the Dist class. (The former is the grand central > station of the Distutils; the latter is just another command class that > happens to implement the "dist" command.) > > I don't see any big problems with this; it just means that the > developers currently using Distutils would have to adjust their habits > by one letter. Any thoughts? > > Greg > -- > Greg Ward - Linux weenie gward@python.net > http://starship.python.net/~gward/ > No problem is so formidable that you can't just walk away from it. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG@python.org > http://www.python.org/mailman/listinfo/distutils-sig > -- Greg Stein, http://www.lyra.org/ From thomas.heller@ion-tof.com Wed Feb 16 18:27:00 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Wed, 16 Feb 2000 19:27:00 +0100 Subject: [Distutils] install w/o build, spaces in directory names References: <002301bf7400$1c3a2d60$c355cfc0@ski.org> <01b301bf7490$e951d470$4500a8c0@thomasnotebook> <20000214184927.A800@beelzebub> Message-ID: <022601bf78ab$6a770930$4500a8c0@thomasnotebook> Greg said: > On 11 February 2000, Thomas Heller said: > > I tried quoting the parameters for _spawn_nt in the following way, > > which worked for me (after changing some paths to include spaces, > > which I usually avoid). Not sure if cmd[0] should also be quoted: > > Well, this patch looks better than nothing. But I have a couple > questions for the Windows crowd: > > * when I use 'os.spawnv()', just how many levels of software are > involved (C library? kernel? the child program being run?) in > parsing the command line -- i.e. in splitting the > Python-list-of-strings up into the argv[] seen by the child program? > In other words, whose rules do we have to work with here to properly > handle filenames with spaces in them? Eventually os.spawnv() calls the win32 API funtion CreateProcess. This is somewhat comparable to a unix-kernel call. Before, the argument list is concatenated into in single string, with all the arguments separated by blanks. This explains at least, that arguments containing blanks _must_ be quoted somehow, otherwise there would be no way to separate them again. > > * regardless of who's involved, what *are* the rules? Ie. what > characters in a command-line argument will cause that argument to > need quoting? What are the allowed quote characters? How > do you deal with arguments that contain the quote characters (ie. > what if I had a directory named > Program "Files" > ? The following characters are not allowed in filenames and directory names: \ / : * ? " < > | This is not only imposed by the GUI or the shell (cmd.exe resp. command.com), you get a windows error: >>> open ("\"", "a") Traceback (innermost last): File "", line 1, in ? IOError: [Errno 22] Invalid argument: '"' > > * is any of this standard or documented, or does it vary from > application to application? It is the responsibility of the application or the C-startup code in the runtime library. Assume the following program: #include int main (int argc, char **argv) { while (argc--) printf ("%s\n", *argv++); } compiled into test.exe. Playing with this gives some interesting insights about the way the commandline is parsed in the startup code of the runtime library: C:\work\orion\src\Fpanel\Python\ctypes>python Python 1.5.2 (#0, Apr 13 1999, 10:51:12) [MSC 32 bit (Intel)] on win32 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>> import os >>> # First, a simple one >>> os.spawnv (os.P_WAIT, "test.exe", ["a", "b", "c"]) a b c 0 #<-- This is the return code of os.spawnv(). >>> # Would you have expected the following? >>> os.spawnv (os.P_WAIT, "test.exe", ["a b c", "d", "e"]) a b c d e 0 >>> # Quoted an argument >>> os.spawnv (os.P_WAIT, "test.exe", ["\"a b c\"", "d", "e"]) a b c d e 0 >>> #Incompletely quoted argument >>> os.spawnv (os.P_WAIT, "test.exe", ["\"a b c", "d", "e"]) a b c d e 0 >>> os.spawnv (os.P_WAIT, "test.exe", ["a b c\"", "d", "e"]) a b c d e 0 >>> os.spawnv (os.P_WAIT, "test.exe", ["a b c\" \"d e"]) a b c d e 0 Interesting, isn't it? It seems that -I"c:\Program files" can be used as well as "-Ic:\Program Files", as well as -Ic:\Program" "files So these rules are trivial, assuming that VC uses this standard argument parsing, my patch should work (except that every arg should be _quote_if_needed(). cmd = map (_quote_if_needed, cmd) > > (BTW Thomas, your patch was backwards. This is easy to workaround -- I > just have to run "patch -R" -- but it's easier to read patches if you > say "diff " rather than "diff > ", which is what I suspect you did.) Thanks for that, I noted it shortly after the mail was out :-) Thomas From jlp@apti.com Thu Feb 17 17:06:28 2000 From: jlp@apti.com (James L. Preston) Date: Thu, 17 Feb 2000 17:06:28 +0000 Subject: [Distutils] Installing Numeric and linking against libpython.so Message-ID: <38AC2A94.5A6F1128@apti.com> I am trying to upgrade my Numeric, etc. and I seem to have a problem with the distutils setup. Everything appears to install fine, but when I import Numeric, I get the message >> import Numeric Traceback (innermost last): File "", line 1, in ? File "/usr/local/lib/python1.5/site-packages/Numeric/Numeric.py", line 76, in ? import multiarray ImportError: ld.so.1: python: fatal: relocation error: file /usr/local/lib/python1.5/site-packages/Numeric/multiarray.so: symbol PySequence_Length: referenced symbol not found >>> I believe that this is the result of the multiarray.so, not being linked with libpython. I have tried to understand where I can put this option in the setup.py, etc. but I must say that I am a bit baffled. I am using Solaris 2.6 on an UltraSPARC. I am not a big shared library expert, but I think that on Solaris, it is necessary to tell one .so file where to find the functions from the other .so files that it may be calling, if you know what I mean. Any suggestions. Thanks, jim preston jlp@apti.com From gward@cnri.reston.va.us Thu Feb 17 19:17:53 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Thu, 17 Feb 2000 14:17:53 -0500 Subject: [Distutils] Re: Where is Imaging.h? In-Reply-To: <14508.11387.761765.788240@weyr.cnri.reston.va.us>; from fdrake@acm.org on Thu, Feb 17, 2000 at 12:14:35PM -0500 References: <14508.11387.761765.788240@weyr.cnri.reston.va.us> Message-ID: <20000217141753.C10739@cnri.reston.va.us> On 17 February 2000, Fred L. Drake, Jr. said: > I think this would be a good idea. There are actually two Python > include directories, one for $prefix and one for $exec_prefix; > extensions would have to "do the right thing" here. > Greg Ward, I presume you've already thought of this and distutils > supports it? ;-) Nope, I blew it and had to hack the NumPy setup script to install the C header files. Both Paul Dubois and I totally forgot about this need until a NumPy user pointed it out. Oops. You're right though, Distutils will need to be fixed to do this properly for other module distributions than NumPy. For the record, NumPy installs its headers to $prefix/include/python1.5/Numeric which I guess makes sense. C headers aren't normally platform-specific, except for those of the config.h variety -- and in fact I believe that most of Python's own headers are installed to $prefix, but config.h goes to $exec_prefix. Wouldn't it be nice if we could just say "Extensions with platform-specific headers are EVIL". Doubt it though. ;-( Greg From mona@blinky.ucsd.edu Thu Feb 17 22:44:38 2000 From: mona@blinky.ucsd.edu (Mona Wong) Date: Thu, 17 Feb 2000 14:44:38 -0800 (PST) Subject: [Distutils] problem compiling on SGI IRIX 6.5 Message-ID: <200002172244.OAA22983@blinky.ucsd.edu> Hi: I've just downloaded Distutils 0.1.3 and tried to compile it on my SGI IRIX 6.5 machine. I am using python 1.5.1. Here is what I got: proteus:/p12/src/Distutils-0.1.3) python setup.py install running install Traceback (innermost last): File "setup.py", line 13, in ? setup (name = "Distutils", File "distutils/core.py", line 102, in setup raise SystemExit, "error: %s: %s" % (exc.filename, exc.strerror) AttributeError: filename What is wrong and how do I fix this? By the way, I just signed up to be on this list but have not received the confirmation yet so please reply back to me personally to be sure that I am getting your responses. Cheers, Mona From cjensen@bioeng.ucsd.edu Thu Feb 17 22:57:19 2000 From: cjensen@bioeng.ucsd.edu (Curtis Jensen) Date: Thu, 17 Feb 2000 14:57:19 -0800 Subject: [Distutils] problem compiling on SGI IRIX 6.5 References: <200002172244.OAA22983@blinky.ucsd.edu> Message-ID: <38AC7CCF.F410432C@be-research.ucsd.edu> Mona Wong wrote: > > Hi: > > I've just downloaded Distutils 0.1.3 and tried to compile it on my > SGI IRIX 6.5 machine. I am using python 1.5.1. > > Here is what I got: > > proteus:/p12/src/Distutils-0.1.3) python setup.py install > running install > Traceback (innermost last): > File "setup.py", line 13, in ? > setup (name = "Distutils", > File "distutils/core.py", line 102, in setup > raise SystemExit, "error: %s: %s" % (exc.filename, exc.strerror) > AttributeError: filename > > What is wrong and how do I fix this? > > By the way, I just signed up to be on this list but have not received > the confirmation yet so please reply back to me personally to be sure that I > am getting your responses. > > Cheers, > > Mona > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG@python.org > http://www.python.org/mailman/listinfo/distutils-sig I use IRIX 6.5 also. I'm not sure why you're getting your error, but I seem to remember that there was some issue with Python 1.5.1 and Distutils. If you upgrade to Python 1.5.2, some bugs get fixed. Try that. -- Curtis Jensen cjensen@be-research.ucsd.edu http://www-bioeng.ucsd.edu/~cjensen/ FAX (425) 740-1451 From gward@cnri.reston.va.us Fri Feb 18 00:09:37 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Thu, 17 Feb 2000 19:09:37 -0500 Subject: [Distutils] First incompatible change: 'dist' command is dead Message-ID: <20000217190935.A11406@cnri.reston.va.us> Hi all -- those of you following the Distutils CVS archive should be aware that I've just checked in the first incompatible change: the "dist" command is gone, replaced by "sdist". Like its predecessor, "sdist" exists to create a source distribution. Unlike its predecessor, there is now an explicit MANIFEST.in/MANIFEST model, meaning that any developers using Distutils now will have to rewrite their old MANIFEST files as a MANIFEST.in. The syntax is *not* what I proposed last week here; I think I've come up with something better. (I.e. I implemented my proposal and it didn't work too well. I came up with something new and a went ahead and implemented before I could poke any holes in it.) For now, the documentation for the MANIFEST.in (aka "the manifest template file") syntax consist of the Distutils manifest template and the 'read_template()' method in distutils/command/sdist.py. Use the Source, Luke! ...but seriously, I'll try to write up some notes on how to write manifest templates soon. It's a small but important feature. Greg From gward@cnri.reston.va.us Fri Feb 18 00:43:11 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Thu, 17 Feb 2000 19:43:11 -0500 Subject: [Distutils] Second incompatible change: command class renamings Message-ID: <20000217194311.A11645@cnri.reston.va.us> Hi again -- I've just commited a second batch of incompatible changes: the renaming of several command class attributes and methods has been done. In particular, options -> user_options (class attribute) set_default_options -> initialize_options (method) set_final_options -> finalize_options (method) I posted on Jan 30 about the reasons for doing this -- if you're curious, I'm sure you can find it in the archive. This will break code for anyone who's defining their own command classes, in particular the NumPy setup script. This just means NumPy users should stick to 0.1.3 and not follow the CVS archive, which I expect they're doing anyways. It does mean the NumPy setup script will have to change for the next release of Distutils -- which I expect to be 1.0, bundled with Python 1.6. Oh, I also bumped the version in the Distutils setup script to "0.2pre", not because there is going to be a release 0.2, but just to remind everyone of these incompatible changes. And I cvs-tagged the pre-incompatibility versions of everything with "pre_0_2_breakage", just in case you need the absolute latest code that still works with the NumPy setup script. Greg From gward@python.net Fri Feb 18 03:44:22 2000 From: gward@python.net (Greg Ward) Date: Thu, 17 Feb 2000 22:44:22 -0500 Subject: [Distutils] Installing Numeric and linking against libpython.so In-Reply-To: <38AC2A94.5A6F1128@apti.com>; from James L. Preston on Thu, Feb 17, 2000 at 05:06:28PM +0000 References: <38AC2A94.5A6F1128@apti.com> Message-ID: <20000217224422.A1559@beelzebub> On 17 February 2000, James L. Preston said: > ImportError: ld.so.1: python: fatal: relocation error: file > /usr/local/lib/python1.5/site-packages/Numeric/multiarray.so: symbol > PySequence_Length: referenced symbol not found > >>> > > I believe that this is the result of the multiarray.so, not being linked > with libpython. What version of Python is this? (And, for that matter, which version of NumPy and Distutils? I'm assuming 15.2 and 0.1.3, but you never know...) How was your libpython built -- is it static or dynamic? (Look in $exec_prefix/lib/python1.5/config to see -- you're looking for either libpython.a or libpython.so.) Are you able to import any of the other NumPy extensions? Have you run "ldd" on them to see if they were linked against libpython.so (assuming libpython.so is what you have)? Have you run "nm" or "elfdump" to see if you can figure out what symbols from libpython are expected by the extensions? Did you try hacking (just cut and paste it) the link command used by Distutils to create multiarray.so to add "-L$exec_prefix/lib/python1.5/config -lpython" (where $exec_prefix is Python's prefix, which of course you'll have to interpolate manually), and see if that helps? Has anyone else seen anything like this? I've built NumPy 15.2 on Solaris 2.6 with Distutils 0.1.3 a number of times, and for the most part It Just Works. My libpython is static, though -- I should probably keep a couple of different python builds around to play with... sigh... Greg -- Greg Ward - just another Python hacker gward@python.net http://starship.python.net/~gward/ Jesus Saves -- but Moses gets the rebound, he shoots, he SCORES! From gward@python.net Fri Feb 18 03:54:48 2000 From: gward@python.net (Greg Ward) Date: Thu, 17 Feb 2000 22:54:48 -0500 Subject: [Distutils] problem compiling on SGI IRIX 6.5 In-Reply-To: <200002172244.OAA22983@blinky.ucsd.edu>; from Mona Wong on Thu, Feb 17, 2000 at 02:44:38PM -0800 References: <200002172244.OAA22983@blinky.ucsd.edu> Message-ID: <20000217225448.B1559@beelzebub> On 17 February 2000, Mona Wong said: > I've just downloaded Distutils 0.1.3 and tried to compile it on my > SGI IRIX 6.5 machine. I am using python 1.5.1. ^^^^^ There are other known problems with Distutils on Python 1.5.1, so fixing this might just reveal other problems. Nevertheless.. > proteus:/p12/src/Distutils-0.1.3) python setup.py install > running install > Traceback (innermost last): > File "setup.py", line 13, in ? > setup (name = "Distutils", > File "distutils/core.py", line 102, in setup > raise SystemExit, "error: %s: %s" % (exc.filename, exc.strerror) > AttributeError: filename > > What is wrong and how do I fix this? Oddly enough, the line *right* before where this exception was raised is the following: # is this 1.5.2-specific? 1.5-specific? I think you may have answered my question: IOError objects only have 'filename' (and 'strerror'?) attributes since 1.5.2. Grumble. Here's a quick, dirty, completely-off-the-top-of-my-head patch that might fix *this* problem. If you fix whatever is causing the IOError, though, you'll probably blow up on an 'apply()' somewhere where I pass a list instead of a tuple, another 1.5.2 feature. Grumble^2. *** core.py.orig Thu Feb 17 19:36:20 2000 --- core.py Thu Feb 17 22:48:51 2000 *************** *** 99,106 **** except KeyboardInterrupt: raise SystemExit, "interrupted" except IOError, exc: ! # is this 1.5.2-specific? 1.5-specific? ! raise SystemExit, "error: %s: %s" % (exc.filename, exc.strerror) # setup () --- 99,110 ---- except KeyboardInterrupt: raise SystemExit, "interrupted" except IOError, exc: ! # arg, try to work with Python pre-1.5.2 ! if hasattr (exc, 'filename') and hasattr (exc, 'strerror'): ! raise SystemExit, \ ! "error: %s: %s" % (exc.filename, exc.strerror) ! else: ! raise SystemExit, str (exc) # setup () Note that this patch is relative to the current CVS version, so it might not even apply. Oh well, it's worth a shot. Greg -- Greg Ward - Linux weenie gward@python.net http://starship.python.net/~gward/ Beware of altruism. It is based on self-deception, the root of all evil. From gward@python.net Wed Feb 23 02:09:19 2000 From: gward@python.net (Greg Ward) Date: Tue, 22 Feb 2000 21:09:19 -0500 Subject: [Distutils] Installing Numeric and linking against libpython.so In-Reply-To: <38B15A1C.AC6284F2@apti.com>; from James L. Preston on Mon, Feb 21, 2000 at 03:30:36PM +0000 References: <38B15A1C.AC6284F2@apti.com> Message-ID: <20000222210919.A558@beelzebub> [Jim -- this discussion deserves to be on the SIG, so I'm cc'ing it there; SIG readers -- if it looks like you missed a message you did: Jim replied straight to me, but I'm putting this thread back on the SIG.] [Jim Preston explains the problem in the first place] > I am trying to upgrade my Numeric, etc. and I seem to have a problem > with the distutils > setup. Everything appears to install fine, but when I import Numeric, I > get the message > >> import Numeric > Traceback (innermost last): > File "", line 1, in ? > File "/usr/local/lib/python1.5/site-packages/Numeric/Numeric.py", line > 76, in ? > import multiarray > ImportError: ld.so.1: python: fatal: relocation error: file > /usr/local/lib/python1.5/site-packages/Numeric/multiarray.so: symbol > PySequence_Length: referenced symbol not found > >>> [me, bugging Jim for more info] > What version of Python is this? (And, for that matter, which version of > NumPy and Distutils? I'm assuming 15.2 and 0.1.3, but you never > know...) [Jim coughs up the details] > Python 1.5.2 (latest, AFAIK), Numeric 15.2 and distutils 0.1.3. [me again] > How was your libpython built -- is it static or dynamic? (Look in > $exec_prefix/lib/python1.5/config to see -- you're looking for either > libpython.a or libpython.so.) [Jim] > RIght. The default configuration was static, I tried to create libpython.so > to try and get the linkage going, but no luck there either. [me] > Are you able to import any of the other NumPy extensions? Have you run > "ldd" on them to see if they were linked against libpython.so (assuming > libpython.so is what you have)? Have you run "nm" or "elfdump" to see > if you can figure out what symbols from libpython are expected by the > extensions? [Jim] > ldd tells me nothing, and nm tells me that all of the Py_Sequence functions, > etc. are undefined. [me] > Did you try hacking (just cut and paste it) the link command used by > Distutils to create multiarray.so to add > "-L$exec_prefix/lib/python1.5/config -lpython" (where $exec_prefix is > Python's prefix, which of course you'll have to interpolate manually), > and see if that helps? [Jim] > No change. OK, I'm stumped. For those of you who missed his first post, Jim is running Solaris 2.6, which just happens to be what I have on my machine it work. So I can swear up and down that NumPy 15.2 builds just fine with Distutils 0.1.3 under Python 1.5.2 and Solaris 2.6. Something seems to be wrong with multiarray.so (one of the extensions installed by NumPy into site-packages/Numeric) in Jim's installation, but for me it works great: $ python Python 1.5.2 (#3, Feb 11 2000, 11:14:47) [GCC 2.8.1] on sunos5 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>> import multiarray >>> dir(multiarray) ['__doc__', '__file__', '__name__', '__version__', 'argmax', 'argsort', 'array', 'arraytype', 'binarysearch', 'choose', 'concatenate', 'cross_correlate', 'error', 'fromstring', 'innerproduct', 'repeat', 'reshape', 'set_string_function', 'sort', 'take', 'transpose', 'zeros'] See, It Just Works (TM). IOW, when my Python loads multiarray.so, it has no problem finding symbols like 'PySequence_Length'. My copy of multiarray.so can't possibly have all of Python linked into it -- it's only 70k. Anyways, that would be so colossally stupid it doesn't even bear thinking about. "ldd" on multiarray.so reveals that it depends on the usual Solaris shared libs -- libc.so, libelf.so, libucb.so, libsocket.so, and a few others. All of them are in /usr/lib, so clearly nothing Python-specific there. But Jim gives one last hint: > I keep thinking that there is some old stuff hanging around from earlier > python releases that is messing this up, but I have tried quite hard to get > rid of everything and build a clean installation and I am still having the same > problems. OK, have you done this, with a *fresh*, brand-new Python 1.5.2 source distribution: $ ./configure --prefix=/tmp/usr $ [edit the setup file, you know the routine] $ make $ make install and then unpack a fresh, brand-new Distutils 0.1.3 distribution and install it to the temporary Python installation $ /tmp/usr/bin/python setup.py install # in the Distutils-0.1.3 directory and then unpack a fresh, brand-new NumPy 15.2 distribution ... $ /tmp/usr/bin/python setup.py install # in the Numerical-15.2 directory I hate having to say "reinstall Python is the answer", but I don't see what could be going wrong except for a screwed up Python installation. Please try this and let us know if it works or not -- either way, I want to know! [Jim asks a relevant question] > I am not a shared library guru, by any means. How is it supposed to > work, anyway? Is it the case that the multiarraymodule should be able > to call functions from libpython,a without any such linkage specified, > simply because it is being imported into a program (python) that is > already linked against libpython.a ? I must say that I am quite > baffled here. That's a good question, and I'm not entirely sure how it works. My gut-level intuition says that because python is the binary that's loading multiarray.so into memory, all of Python's API is already there -- so it Just Works. (At least, for *me* it just works.) IOW, if you tried to load multiarray.so into Tcl or Perl or another C program, then of course it wouldn't be able to resolve PySequence_Length. But I would like to hear from someone who really understands this what's going on. I'm curious too! > I am using gcc-2.95.1, with gnu ld, etc. Hmmm, there's one difference between Jim's setup and mine -- we're still using gcc 2.8.1, and the linker used to create shared objects is the system linker -- /usr/ucb/ld. This comes from Python's Makefile, ie. this is how Python was built, so that's how Distutils builds extensions. If you build Python with a different linker, Distutils will build extensions with that linker. This might be key! Greg -- Greg Ward - all-purpose geek gward@python.net http://starship.python.net/~gward/ God is real, unless declared integer. From brian@garage.co.jp Wed Feb 23 10:17:07 2000 From: brian@garage.co.jp (Brian Takashi Hooper) Date: Wed, 23 Feb 2000 19:17:07 +0900 Subject: [Distutils] Convention for installing binary data files Message-ID: <38B3B3A33C1.01AEBRIAN@smtp.garage.co.jp> Hi there, I'm a newcomer to distutils, so sorry in advance if this is a question that is too obvious for everyone... I was wondering what the convention is for including binary data files with the installation of a module. Like say I had a table called table.data, that I wanted to have be installed with everything else on python setup.py install - should the data file be part of install_py, or install_ext, or something else? We'd like to use distutils to package up a current project, but weren't sure about this point, and couldn't find it in the docs... --Brian Hooper From gward@cnri.reston.va.us Wed Feb 23 14:50:01 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Wed, 23 Feb 2000 09:50:01 -0500 Subject: [Distutils] Distutils patch for Python 1.5.1 compatibility Message-ID: <20000223095001.A507@cnri.reston.va.us> --J2SCkAp4GZ/dPZZf Content-Type: text/plain; charset=us-ascii Hi all -- I've gotten enough bug reports about Distutils not working with Python 1.5.1 to finally do something about it. The result is the attached patch. Please give it a try, *especially* if you use Python 1.5.1 (or older, I suppose). If this works, I'll release it as Distutils 0.1.4 so the NumPy and PyFort people can get their jobs done without stumbling across my bugs. Oh yeah, you can apply the patch either in the Distutils source tree (right in Distutils-0.1.3, wherever you put it) or to the installed version, from the "site-packages" directory. In either case, apply it with "patch -p1". Eg. if Distutils is in the normal Unix "site-packages", and the patch is in /tmp/distutils-apply.patch, you can patch the installed version of Distutils with: cd /usr/local/lib/python1.5/site-packages patch -p1 < /tmp/distutils-apply.patch And then try building and installing NumPy, or PIL (using the setup script included with Distutils, that is), or something generated by PyFort -- whatever. Please let me know how it goes! Thanks -- Greg --J2SCkAp4GZ/dPZZf Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="distutils-apply.patch" diff -cr ../Distutils-0.1.3/distutils/command/build_ext.py ./distutils/command/build_ext.py *** ../Distutils-0.1.3/distutils/command/build_ext.py Sun Jan 30 13:34:12 2000 --- ./distutils/command/build_ext.py Tue Feb 22 12:35:18 2000 *************** *** 254,259 **** if package: ext_name = package + '.' + ext_name ext_path = string.split (ext_name, '.') ! return apply (os.path.join, ext_path) + sysconfig.SO # class BuildExt --- 254,259 ---- if package: ext_name = package + '.' + ext_name ext_path = string.split (ext_name, '.') ! return apply (os.path.join, tuple (ext_path)) + sysconfig.SO # class BuildExt diff -cr ../Distutils-0.1.3/distutils/command/build_py.py ./distutils/command/build_py.py *** ../Distutils-0.1.3/distutils/command/build_py.py Sun Jan 30 13:34:13 2000 --- ./distutils/command/build_py.py Tue Feb 22 12:35:18 2000 *************** *** 102,118 **** if type (package) is StringType: path = string.split (package, '.') ! elif type (package) in (TupleType, ListType): ! path = list (package) ! else: raise TypeError, "'package' must be a string, list, or tuple" if not self.package_dir: if path: ! return apply (os.path.join, path) else: return '' else: tail = [] while path: try: --- 102,119 ---- if type (package) is StringType: path = string.split (package, '.') ! elif type (package) not in (TupleType, ListType): raise TypeError, "'package' must be a string, list, or tuple" if not self.package_dir: if path: ! return apply (os.path.join, tuple (path)) else: return '' else: + if type (path) is not ListType: # must be mutable! + path = list (path) + tail = [] while path: try: *************** *** 122,133 **** del path[-1] else: tail.insert (0, pdir) ! return apply (os.path.join, tail) else: # arg! everything failed, we might as well have not even # looked in package_dir -- oh well if tail: ! return apply (os.path.join, tail) else: return '' --- 123,134 ---- del path[-1] else: tail.insert (0, pdir) ! return apply (os.path.join, tuple (tail)) else: # arg! everything failed, we might as well have not even # looked in package_dir -- oh well if tail: ! return apply (os.path.join, tuple (tail)) else: return '' *************** *** 263,269 **** outfile_path = list (package) outfile_path.append (module + ".py") outfile_path.insert (0, self.build_dir) ! outfile = apply (os.path.join, outfile_path) dir = os.path.dirname (outfile) self.mkpath (dir) --- 264,270 ---- outfile_path = list (package) outfile_path.append (module + ".py") outfile_path.insert (0, self.build_dir) ! outfile = apply (os.path.join, tuple (outfile_path)) dir = os.path.dirname (outfile) self.mkpath (dir) diff -cr ../Distutils-0.1.3/distutils/command/dist.py ./distutils/command/dist.py *** ../Distutils-0.1.3/distutils/command/dist.py Sun Jan 30 15:22:27 2000 --- ./distutils/command/dist.py Tue Feb 22 12:34:37 2000 *************** *** 346,352 **** words = string.split (pattern) assert words # must have something! if os.name != 'posix': ! words[0] = apply (os.path.join, string.split (words[0], '/')) # First word is a directory, possibly with include/exclude # patterns making up the rest of the line: it's a recursive --- 346,353 ---- words = string.split (pattern) assert words # must have something! if os.name != 'posix': ! words[0] = apply (os.path.join, ! tuple (string.split (words[0], '/'))) # First word is a directory, possibly with include/exclude # patterns making up the rest of the line: it's a recursive diff -cr ../Distutils-0.1.3/distutils/command/install.py ./distutils/command/install.py *** ../Distutils-0.1.3/distutils/command/install.py Sun Jan 30 13:34:14 2000 --- ./distutils/command/install.py Tue Feb 22 12:35:15 2000 *************** *** 202,208 **** # local form if string.find (extra_dirs, '/') and os.name != "posix": extra_dirs = string.split (extra_dirs, '/') ! extra_dirs = apply (os.path.join, extra_dirs) else: path_file = None extra_dirs = '' --- 202,208 ---- # local form if string.find (extra_dirs, '/') and os.name != "posix": extra_dirs = string.split (extra_dirs, '/') ! extra_dirs = apply (os.path.join, tuple (extra_dirs)) else: path_file = None extra_dirs = '' diff -cr ../Distutils-0.1.3/distutils/core.py ./distutils/core.py *** ../Distutils-0.1.3/distutils/core.py Sun Jan 30 13:30:32 2000 --- ./distutils/core.py Tue Feb 22 12:38:15 2000 *************** *** 904,909 **** --- 904,912 ---- a message to print if the verbosity level is high enough, and an optional verbosity threshold.""" + if type (args) is not TupleType: + raise TypeError, "'args' must be a tuple of command arguments" + # Generate a message if we weren't passed one if msg is None: msg = "%s %s" % (func.__name__, `args`) diff -cr ../Distutils-0.1.3/distutils/util.py ./distutils/util.py *** ../Distutils-0.1.3/distutils/util.py Sun Jan 30 14:57:48 2000 --- ./distutils/util.py Tue Feb 22 12:38:38 2000 *************** *** 170,175 **** --- 170,178 ---- 'args', but only if it needs to: i.e. if 'dst' does not exist or 'src' is newer than 'dst'.""" + if type (args) is not TupleType: + raise TypeError, "'args' must be a tuple of command arguments" + if newer (src, dst): if verbose and update_message: print update_message --J2SCkAp4GZ/dPZZf-- From fdrake@acm.org Wed Feb 23 17:24:36 2000 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Wed, 23 Feb 2000 12:24:36 -0500 (EST) Subject: [Distutils] Re: [XML-SIG] small installation problem In-Reply-To: <38B41518.3548D87B@trema.com> References: <38B41518.3548D87B@trema.com> Message-ID: <14516.6100.748263.302872@weyr.cnri.reston.va.us> Harri, This is (in part) a distutils issue. The distutils package should always create this directory if it doesn't exist. Raw Python installations should not create it, since adding it to the search path would slow down the module search. Greg, I don't know if you're reading the XML-SIG list, so I'm adding the distutils list to the list of recipients. Harri Pasanen writes: > I installed PyXML-0.5.3 on Solaris 2.7 following the README > instructions. > > > python setup.py install > > failed at first, because of missing > /usr/local/lib/python1.5/site-packages/ directory. > That directory does not appear the be created when Python 1.5.2 is > installed from the tar-ball. > > After manually creating the directory, the install went through without > complaints. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From est@hyperreal.org Wed Feb 23 20:06:22 2000 From: est@hyperreal.org (est@hyperreal.org) Date: Wed, 23 Feb 2000 12:06:22 -0800 (PST) Subject: [Distutils] a patch :) Message-ID: <20000223200622.16738.qmail@hyperreal.org> Could be old news, but I had to do this to 0.1.3: --- command/dist.py~ Sun Jan 30 15:22:27 2000 +++ command/dist.py Wed Feb 23 13:07:50 2000 @@ -223,6 +223,7 @@ for fn in standards: if type (fn) is TupleType: alts = fn + got_it = 0 for fn in alts: if os.path.exists (fn): got_it = 1 From est@hyperreal.org Wed Feb 23 20:18:26 2000 From: est@hyperreal.org (est@hyperreal.org) Date: Wed, 23 Feb 2000 12:18:26 -0800 (PST) Subject: [Distutils] distutils and the development process Message-ID: <20000223201826.27843.qmail@hyperreal.org> Greetings, disutils people! I'd like to be able to use distutil's build process while I'm developing and debugging code. This has proven difficult so far because of the use of the build directory. I've ended up just using Makefile-driven in-place builds during the development cycle. Does anyone have any tips or tricks in this area? I'm also curious how people like to run test scripts. One special case is post-install hello-world-type example programs for the installer to play with. To keep mine from being confused by the package source directory, I end up adding the following to them: # Ugh..make sure we look only in system directories for the package. # This is only important when executing a script in the package # distribution directory. import sys sys.path = sys.path[1:] There's got to be a better way. In fact, I often end up commenting out that assignment during development so I can test my in-place Makefile-driven builds! Anyhow, I feel like I must be doing things wrong and I'd appreciate opinions on these topics. :) Eric From gward@python.net Thu Feb 24 02:53:56 2000 From: gward@python.net (Greg Ward) Date: Wed, 23 Feb 2000 21:53:56 -0500 Subject: [Distutils] Re: [XML-SIG] small installation problem In-Reply-To: <14516.6100.748263.302872@weyr.cnri.reston.va.us>; from Fred L. Drake, Jr. on Wed, Feb 23, 2000 at 12:24:36PM -0500 References: <38B41518.3548D87B@trema.com> <14516.6100.748263.302872@weyr.cnri.reston.va.us> Message-ID: <20000223215356.A3815@beelzebub> On 23 February 2000, Fred L. Drake, Jr. said: > This is (in part) a distutils issue. The distutils package should > always create this directory if it doesn't exist. Fred is correct -- Distutils should create any directories it needs to install files. (In fact, it creates any directories it needs to do anything.) In fact, I've just tested this: both my current development version (on Linux) and the 0.1.3 release (Solaris 2.6) work peachy keen. If I remove or rename my site-packages directory, Distutils recreates it *as long as I have permission to write in $prefix/lib/python1.5*. Harri Pasanen writes: > python setup.py install > > failed at first, because of missing > /usr/local/lib/python1.5/site-packages/ directory. > That directory does not appear the be created when Python 1.5.2 is > installed from the tar-ball. Since I can't reproduce the bug, I'm going to need more information. Could you supply an exact transcript of the session where Distutils failed to create the site-packages directory? (I'm guessing there's a traceback that will reveal useful information.) Also, what version of Distutils did you use? Greg -- Greg Ward - Linux bigot gward@python.net http://starship.python.net/~gward/ Whatever became of eternal truth? From gward@python.net Thu Feb 24 03:01:49 2000 From: gward@python.net (Greg Ward) Date: Wed, 23 Feb 2000 22:01:49 -0500 Subject: [Distutils] Convention for installing binary data files In-Reply-To: <38B3B3A33C1.01AEBRIAN@smtp.garage.co.jp>; from Brian Takashi Hooper on Wed, Feb 23, 2000 at 07:17:07PM +0900 References: <38B3B3A33C1.01AEBRIAN@smtp.garage.co.jp> Message-ID: <20000223220149.B3815@beelzebub> On 23 February 2000, Brian Takashi Hooper said: > I was wondering what the convention is for including binary data files > with the installation of a module. Like say I had a table called > table.data, that I wanted to have be installed with everything else on > python setup.py install - should the data file be part of install_py, or > install_ext, or something else? Convention? There is no convention! Distutils currently handles the installation of precisely two types of files: pure Python modules and Python extension modules. Anything else and you're on your own. The good news is, the system is extensible: it's not too hard to augment the standard "install" command to handle whatever type of file interests you. See the NumPy setup script distributed with recent versions of NumPy for an example; visit this charming URL to see it directly: http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/Numerical/setup.py?rev=1.8&content-type=text/x-cvsweb-markup&cvsroot=numpy Anyways, what is the nature of this data file? Is it platform-neutral? Does it have anything to do with Python, or is it just an artifact of your project? Under Unix, /usr/share/ or /usr/lib/ would probably be good places for it (depending on whether it's platform-neutral or not). > We'd like to use distutils to package up a current project, but weren't > sure about this point, and couldn't find it in the docs... Docs? What docs? ;-) (Don't worry, I'm getting to that!) Greg -- Greg Ward - Unix bigot gward@python.net http://starship.python.net/~gward/ If you can read this, thank a programmer. From gward@python.net Thu Feb 24 03:24:44 2000 From: gward@python.net (Greg Ward) Date: Wed, 23 Feb 2000 22:24:44 -0500 Subject: [Distutils] distutils and the development process In-Reply-To: <20000223201826.27843.qmail@hyperreal.org>; from est@hyperreal.org on Wed, Feb 23, 2000 at 12:18:26PM -0800 References: <20000223201826.27843.qmail@hyperreal.org> Message-ID: <20000223222444.D3815@beelzebub> On 23 February 2000, est@hyperreal.org said: > I'd like to be able to use distutil's build process while I'm > developing and debugging code. This has proven difficult so far > because of the use of the build directory. I've ended up just using > Makefile-driven in-place builds during the development cycle. Does > anyone have any tips or tricks in this area? As they say in the trade, "That's not a bug -- that's a feature!" But seriously: what *about* the build directory is causing you problems? My current thinking is to put *everything* somewhere in the build directory, leaving the source tree pristine. I.e., .o or .obj files would go in build/tmp., along with the surprise compiler turds that Visual C++ leaves behind. (When you think about it, .o and .obj files are really just compiler turds too -- but we all expect them from years of experience.) The idea of the build directory is that you just do this: export PYTHONPATH=build/lib:build/platlib while you're in the develop-test-debug cycle for extension modules. Just make sure you stay in the root of the source tree. You can work this way too for pure Python modules if you *want*, but there's no good reason to. If your source tree is laid out in a sensible way (like Distutils, of course, or PyXML), then you don't have to do anything special -- eg. when I'm hacking on Distutils, I just let Python find distutils.core as distutils/core.py under the Distutils source root. No problem. > I'm also curious how people like to run test scripts. One special > case is post-install hello-world-type example programs for the > installer to play with. To keep mine from being confused by the > package source directory, I end up adding the following to them: That sounds like a "demo script" to me, but that's just quibbling over semantics. I simply don't understand what you mean by "confused by the package source directory". Please explain. > # Ugh..make sure we look only in system directories for the package. > # This is only important when executing a script in the package > # distribution directory. > import sys > sys.path = sys.path[1:] Ugh, barf, blechh indeed. Why *ever* do you need to do this? Greg -- Greg Ward - Linux weenie gward@python.net http://starship.python.net/~gward/ The NSA. We care: we listen to our customers. From gward@python.net Thu Feb 24 03:30:18 2000 From: gward@python.net (Greg Ward) Date: Wed, 23 Feb 2000 22:30:18 -0500 Subject: [Distutils] Documentation: "Installing Python Modules" Message-ID: <20000223223018.A3646@beelzebub> Hi all -- well, I finally sat down and started spewing Distutils documentation. My plan is to write two howto-style manuals (maybe a bit bigger than "Documenting Python", but along those lines) to be included in the 1.6 manual set. (Clever of me to announce this intention in public before mentioning it to Fred or Guido, just to make it harder for them to tell me to get stuffed. ;-) The two manuals are tentatively entitled: Installing Python Modules Distributing Python Modules So far, I have written one section of the "Installing" manual, entitled "Custom Installation (Unix)". This one is kind of important: it explains how to use the Distutils to install modules to any non-standard place, with lots of commentary on good practices and multi-platform installations. I want feedback on this on several axes: * Distutils terminology: eg. does "install-lib" mean what you think it should mean, or do you have a better idea? * Distutils semantics: eg. is the behaviour of "install-lib" sensible? * documentation organization * documentation wording * spelling, typos, typesetting, etc. The docs are available at http://www.python.org/sigs/distutils-sig/documentation.html Currently only LaTeX source and PDF available -- HTML soon. -- Greg Ward - programmer-at-large gward@python.net http://starship.python.net/~gward/ If it can't be expressed in figures, it is not science--it is opinion. From brian@garage.co.jp Thu Feb 24 03:32:32 2000 From: brian@garage.co.jp (Brian Takashi Hooper) Date: Thu, 24 Feb 2000 12:32:32 +0900 Subject: [Distutils] Convention for installing binary data files In-Reply-To: <20000223220149.B3815@beelzebub> References: <38B3B3A33C1.01AEBRIAN@smtp.garage.co.jp> <20000223220149.B3815@beelzebub> Message-ID: <38B4A6502FB.01B4BRIAN@smtp.garage.co.jp> Hi Greg, thanks for the quick response! On Wed, 23 Feb 2000 22:01:49 -0500 "Greg Ward" wrote: > On 23 February 2000, Brian Takashi Hooper said: > > I was wondering what the convention is for including binary data files > > with the installation of a module. Like say I had a table called > > table.data, that I wanted to have be installed with everything else on > > python setup.py install - should the data file be part of install_py, or > > install_ext, or something else? > > Convention? There is no convention! Distutils currently handles the > installation of precisely two types of files: pure Python modules and > Python extension modules. Anything else and you're on your own. Are there any plans to include additional hooks for other stuff? Another one that I can think of is documentation - is that typically built as part of the install_ext portion...? What are other people doing with this? I suppose adding the data file install to 'install' would be all right - it seems like it would be nicer though to be able to have install still just be the sum of its component targets (install_py and install_ext)... (or is that not the way it works?) > > The good news is, the system is extensible: it's not too hard to augment > the standard "install" command to handle whatever type of file interests > you. See the NumPy setup script distributed with recent versions of > NumPy for an example; visit this charming URL to see it directly: > > http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/Numerical/setup.py?rev=1.8&content-type=text/x-cvsweb-markup&cvsroot=numpy > > Anyways, what is the nature of this data file? Is it platform-neutral? > Does it have anything to do with Python, or is it just an artifact of > your project? Under Unix, /usr/share/ or > /usr/lib/ would probably be good places for it (depending > on whether it's platform-neutral or not). It is a platform-neutral data file - specifically, a character mapping table (this is for a Japanese encoding package that a colleague of mine is currently working on)... I guess this would be /usr/lib or /usr/local/lib then? Thanks, --Brian Hoopeer From gward@python.net Thu Feb 24 04:06:18 2000 From: gward@python.net (Greg Ward) Date: Wed, 23 Feb 2000 23:06:18 -0500 Subject: [Distutils] Convention for installing binary data files In-Reply-To: <38B4A6502FB.01B4BRIAN@smtp.garage.co.jp>; from Brian Takashi Hooper on Thu, Feb 24, 2000 at 12:32:32PM +0900 References: <38B3B3A33C1.01AEBRIAN@smtp.garage.co.jp> <20000223220149.B3815@beelzebub> <38B4A6502FB.01B4BRIAN@smtp.garage.co.jp> Message-ID: <20000223230618.A3947@beelzebub> On 24 February 2000, Brian Takashi Hooper said: > Are there any plans to include additional hooks for other stuff? > Another one that I can think of is documentation - is that typically > built as part of the install_ext portion...? What are other people > doing with this? My long-term plan is to have specific support for installing certain types of documentation. Anything more concrete than that will have to wait for some standard module documentation system to emerge from the doc-sig. (In other words, don't hold your breath!) However, you've given me an interesting idea: the ability to register certain commands as "sub-install" commands that would then be automatically called by the "install" command itself. Currently, if you want to add some funky new kind of installation, you can either: * subclass the standard "install" command, adding capability to install your custom stuff -- that's the approach I took in adding header installation to NumPy * register your customized version of the "install" command or: * define a new command "my_install" (or whatever) that just does whatever it takes to install your custom stuff * subclass the standard "install" command; the only change is to add a call to your command -- run_peer('my_install') * register both "my_install" and the customized version of "install" However, if we add a way to add arbitrary "sub-install" commands, that simplifies to: * define a new custom "sub-install" command * register it as a "sub-install" command and Bob's your uncle. The same would have to apply to "build" commands -- anything you install (code, documentation, whatever), you usually have to be able to build. > It is a platform-neutral data file - specifically, a character mapping > table (this is for a Japanese encoding package that a colleague of mine > is currently working on)... I guess this would be /usr/lib or > /usr/local/lib then? No, if you follow the GNU coding standards, platform-neutral stuff goes in /usr/share (or /usr/local/share). /usr/lib (/usr/local/share) is for platform-specific stuff like C libraries. (Now, let's count the number of major packages that actually respect this. You can probably leave your socks on...) Greg -- Greg Ward - Linux weenie gward@python.net http://starship.python.net/~gward/ If you can read this, thank a programmer. From est@hyperreal.org Thu Feb 24 14:42:06 2000 From: est@hyperreal.org (est@hyperreal.org) Date: Thu, 24 Feb 2000 06:42:06 -0800 (PST) Subject: [Distutils] distutils and the development process In-Reply-To: <20000223222444.D3815@beelzebub> from Greg Ward at "Feb 23, 2000 10:24:44 pm" Message-ID: <20000224144207.14447.qmail@hyperreal.org> Greg Ward discourseth: > > The idea of the build directory is that you just do this: > > export PYTHONPATH=build/lib:build/platlib > > while you're in the develop-test-debug cycle for extension modules. Sounds good. :) > > I'm also curious how people like to run test scripts. One special > > case is post-install hello-world-type example programs for the > > installer to play with. To keep mine from being confused by the > > package source directory, I end up adding the following to them: > > That sounds like a "demo script" to me, but that's just quibbling over > semantics. I simply don't understand what you mean by "confused by the > package source directory". Please explain. OK. A recipient of my MIDI stream parser package (called `MIDI') installs via distutils. They're still in the distribution directory and want to run a demo script to see that things are working. The script does `import MIDI'.. Traceback (innermost last): File "./test.py", line 9, in ? import MIDI File "./MIDI/__init__.py", line 1, in ? In = __import__('MIDI.In').In.MidiIn File "./MIDI/In.py", line 27, in ? _midi = __import__('MIDI._In') ImportError: No module named _In The script went to the source directory (MIDI) and, of course, didn't find the extension module. > > # Ugh..make sure we look only in system directories for the package. > > # This is only important when executing a script in the package > > # distribution directory. > > import sys > > sys.path = sys.path[1:] > > Ugh, barf, blechh indeed. Why *ever* do you need to do this? Well, it makes the script work. As I said, I'm sure there must be a better way. :) Eric From gward@python.net Fri Feb 25 01:21:31 2000 From: gward@python.net (Greg Ward) Date: Thu, 24 Feb 2000 20:21:31 -0500 Subject: [Distutils] distutils and the development process In-Reply-To: <20000224144207.14447.qmail@hyperreal.org>; from est@hyperreal.org on Thu, Feb 24, 2000 at 06:42:06AM -0800 References: <20000223222444.D3815@beelzebub> <20000224144207.14447.qmail@hyperreal.org> Message-ID: <20000224202131.A452@beelzebub> On 24 February 2000, est@hyperreal.org said: > OK. A recipient of my MIDI stream parser package (called `MIDI') > installs via distutils. They're still in the distribution directory > and want to run a demo script to see that things are working. The > script does `import MIDI'.. > > Traceback (innermost last): > File "./test.py", line 9, in ? > import MIDI > File "./MIDI/__init__.py", line 1, in ? > In = __import__('MIDI.In').In.MidiIn > File "./MIDI/In.py", line 27, in ? > _midi = __import__('MIDI._In') > ImportError: No module named _In > > The script went to the source directory (MIDI) and, of course, didn't > find the extension module. Ah-ha! I see the problem. It's because Python always puts the directory of the script being executed *first* in sys.path. Allow me to demonstrate: I'm doing this in the Distutils source directory, because it's handy for me. You can do it in any Distutilized package laid out in the canonical fashion that has been built, i.e. module "foo.bar" is in both "foo/bar.py" *and* "build/lib/foo/bar.py", and you want the built version -- "build/lib/foo/bar.py". (This is irrelevant with pure Python modules -- in fact, you usually want the unbuilt one, because you don't want to go through the pointless exercise of "building" -- copying files around -- just because you changed one line of Python. But with extensions, it's essential, since Distutils builds them in the build tree rather than in the source tree!) Anyways. Here's my test setup: $ ls -lF # partial display! drwxr-xr-x 3 greg users 1024 Feb 16 22:17 build/ drwxr-xr-x 3 greg users 1024 Feb 17 19:25 distutils/ -rwxr-xr-x 1 greg users 748 Feb 17 19:01 setup.py* -rw-r--r-- 1 greg users 71 Feb 24 20:05 which.py "distutils" is where my development copy is, "build/lib/distutils" is the built version of that. Again, the distinction only matters for extensions, but just play along. My test script is which.py: $ cat which.py import sys, pprint, distutils pprint.pprint (sys.path) print distutils Let's try to force Python to find the "build/lib/distutils" version of the code: $ export PYTHONPATH=build/lib (I didn't bother to specify build/platlib here, but it's a good habit to supply both just in case there are extensions present.) Let's run it twice, two different ways: $ python which.py ['', 'build/lib', '/usr/local/python/lib/python1.5/', '/usr/local/python/lib/python1.5/plat-linux2', '/usr/local/python/lib/python1.5/lib-tk', '/usr/local/bin/../python.i86-linux/lib/python1.5/lib-dynload'] $ python ./which.py ['.', 'build/lib', '/usr/local/python/lib/python1.5/', '/usr/local/python/lib/python1.5/plat-linux2', '/usr/local/python/lib/python1.5/lib-tk', '/usr/local/bin/../python.i86-linux/lib/python1.5/lib-dynload'] Spot the difference? Yep, that's right: Python uses dirname(scriptname) as the very first element of sys.path. Apart from sys.path[0] and the first element of the path to the distutils __init__.py, everything else is the same. Most importantly, my PYTHONPATH comes *after* dirname(scriptname) in sys.path. In other words, the problem suggests the solution: put your test script in another directory! I humbly recommend "test" for test scripts, "example" for example scripts, "demo" for demo scripts, etc. If I move my which.py and run it again: $ mv which.py test/ $ python test/which.py ['test', 'build/lib', '/usr/local/python/lib/python1.5/', '/usr/local/python/lib/python1.5/plat-linux2', '/usr/local/python/lib/python1.5/lib-tk', '/usr/local/bin/../python.i86-linux/lib/python1.5/lib-dynload'] ...well, isn't that interesting -- It Just Works! Python finds my "build/lib" version of Distutils as expected. Problem solved, hooray. Looks like this little anectode ought to be written up for the mythical "Distributing Python Modules" manual. Greg -- Greg Ward - all-purpose geek gward@python.net http://starship.python.net/~gward/ Disclaimer: All rights reserved. Void where prohibited. Limit 1 per customer. From gward@python.net Fri Feb 25 02:34:36 2000 From: gward@python.net (Greg Ward) Date: Thu, 24 Feb 2000 21:34:36 -0500 Subject: [Distutils] Installation scheme: third kick-at-the-can Message-ID: <20000224213436.A578@beelzebub> Hi all -- well, you can pretty much ignore the fragment of "Installing Python Modules" that I announced yesterday. I had a brief chat with Guido this afternoon, and he pretty much torpedoed the lovely, baroque scheme that Fred Drake and I cooked up and which became said document. Sigh. Anyways, here's a rough outline of the New New Installation Scheme (whatever you may say, you can't accuse me of inflexible thinking...): * as before, a standard installation (to $prefix/lib/python1.X/site-packages on Unix, or C:\Program Files\Python on Windows) is painless(TM): python setup.py install * for a custom installation where you don't care about supporting multiple platforms, install-lib is the option normally used: python setup.py install --install-lib=/home/greg/lib/python python setup.py install --install-lib=/tmp/pylib python setup.py install --install-lib=d:\Temp\Python\Lib In most circumstances, top-level modules are installed right in the 'install-lib' directory. * if you do care about a multi-platform installation, you should supply both 'install-lib' and 'install-platlib': the former will be used for pure Python module distributions, the latter for module distributions containing extensions: python setup.py install --install-lib=~/lib/python \ --install-platlib=~/lib/python.linux-x86 A subtle wrinkle: for any given module distribution, only one of these is necessary. I don't think installers should have to be aware of the implementation details of the software they're installing, though, so the safe thing to do is supply both options -- and the likely way to do this will be through a config file. Not sure how the config file will look, though. * the only reason to supply 'prefix' and 'exec-prefix' is to install a module distribution to a different Python installation than the one expected by the current interpreter. Say you have a Python installation in /usr and another one in /usr/local: /usr/bin/python setup.py --prefix=/usr/local This will install modules to /usr/local/lib/python1.X/site-packages. (This could be one solution to the "problem" of Linux distributions shipping Python in /usr, but wanting to keep third-party modules in /usr/local -- just create a dummy /usr/local/lib/python1.X/site-packages and you're off to the races.) * 'install-path' is still there, and its purpose is still to supply an extra sub-path to append to 'install-lib', so that non-package-ized module distributions will still get a directory of their own. If 'install-path' used and 'install-lib' is not -- i.e. we're installing to a site-packages directory under some prefix or exec-prefix (or directly in prefix/exec-prefix on Windows) -- then Distutils will also create a .pth file pointing to the 'install-path' directory. There are a few tricky issues with 'install-path', though: + should installers be allowed to set it, or should it be a developer-only option? + if the installer supplied 'install-lib' and/or 'install-platlib', do we still respect 'install-path', i.e. do we still append 'install-path' to 'install-lib'? We certainly don't create a .pth file in that case (Python will ignore it). + do we allow installers to supply *both* 'install-lib' and 'install-path'? I vote YES on all three, but I'd like to hear who agrees with me and who is a hopelessly misguided heretic. >grin< * default values: prefix: sys.prefix exec-prefix: if installer supplied prefix: prefix else: sys.exec_prefix install-lib: $prefix/lib/python1.X/site-packages install-platlib: if installer supplied install-lib: install-lib else: $exec-prefix/lib/python1.X/site-packages * installation rules if install-path not supplied: modules from pure Python distribution go to install-lib modules from distribution with extensions go to install-platlib * installation rules if install-path supplied: modules from pure Python distribution go to install-lib + install-path modules from distribution with extensions go to install-platlib + install-path Note that the "install-path not supplied" case is just a special case of "install-path supplied", where install-path is an empty string. But it's worth documenting it separately, because it's the usual case. I think I like this scheme better than the other two I've tried. Please let me know what you think: I'll rewrite the "Custom Installation" section and then rewrite the code to match the docs Real Soon Now. Greg -- Greg Ward - Linux geek gward@python.net http://starship.python.net/~gward/ When the ship lifts, all bills are paid. No regrets. From est@hyperreal.org Fri Feb 25 14:35:21 2000 From: est@hyperreal.org (est@hyperreal.org) Date: Fri, 25 Feb 2000 06:35:21 -0800 (PST) Subject: [Distutils] distutils and the development process In-Reply-To: <20000224202131.A452@beelzebub> from Greg Ward at "Feb 24, 2000 08:21:31 pm" Message-ID: <20000225143521.6034.qmail@hyperreal.org> OK, I just put export "PYTHONPATH=build/lib:build/plat-lib" in my .profile. I think I prefer to have to explicitly turn it *off*. :) Greg Ward discourseth: > In other words, the problem suggests the solution: put your test script > in another directory! I humbly recommend "test" for test scripts, > "example" for example scripts, "demo" for demo scripts, etc. If I move > my which.py and run it again: Yes, this solves my problems. I feel stupid. :) Eric From est@hyperreal.org Fri Feb 25 15:01:57 2000 From: est@hyperreal.org (est@hyperreal.org) Date: Fri, 25 Feb 2000 07:01:57 -0800 (PST) Subject: [Distutils] distutils and the development process In-Reply-To: <20000225143521.6034.qmail@hyperreal.org> from "est@hyperreal.org" at "Feb 25, 2000 06:35:21 am" Message-ID: <20000225150157.17660.qmail@hyperreal.org> est@hyperreal.org discourseth: > OK, I just put export "PYTHONPATH=build/lib:build/plat-lib" in my > .profile. I think I prefer to have to explicitly turn it *off*. :) Except that that should be `platlib', not `plat-lib'. :O Eric From est@hyperreal.org Fri Feb 25 17:04:30 2000 From: est@hyperreal.org (est@hyperreal.org) Date: Fri, 25 Feb 2000 09:04:30 -0800 (PST) Subject: [Distutils] distutils and the development process In-Reply-To: <20000225143521.6034.qmail@hyperreal.org> from "est@hyperreal.org" at "Feb 25, 2000 06:35:21 am" Message-ID: <20000225170430.25450.qmail@hyperreal.org> est@hyperreal.org discourseth: > > Yes, this solves my problems. I feel stupid. :) I take that back. I have two major continued problems even when I do, "export PYTHONPATH=build/lib:build/platlib". 1) When developing in the top-level directory of a distribution, the current directory still comes first in sys.path, so python still thinks the package source directory (with its __init__.py) is where the package lives. 2) Even when executing an example script in a sub-directory, python gets the idea that the package is in build/lib and doesn't find it's extension submodules. If I copy those over to the build/lib subdirectory they are found. I see the PyXML distribution does not pretty things to control where the extension modules are built, probably for these reasons. Does everyone who has extension modules in their packages have to do that? A final note, even if these problems didn't exist, I'd really prefer not to have to do "python setup.py build" before I test any change to a .py file. I don't want the `compile' back in the edit/compile/debug cycle for .py files. If I think of good strategies for dealing with these problems, I'll post them. In the meantime, it's back to Makefiles for me. :) Eric From gward@python.net Sat Feb 26 01:00:49 2000 From: gward@python.net (Greg Ward) Date: Fri, 25 Feb 2000 20:00:49 -0500 Subject: [Distutils] distutils and the development process In-Reply-To: <20000225170430.25450.qmail@hyperreal.org>; from est@hyperreal.org on Fri, Feb 25, 2000 at 09:04:30AM -0800 References: <20000225143521.6034.qmail@hyperreal.org> <20000225170430.25450.qmail@hyperreal.org> Message-ID: <20000225200049.A425@beelzebub> On 25 February 2000, est@hyperreal.org said: > I have two major continued problems even when I do, "export > PYTHONPATH=build/lib:build/platlib". > > 1) When developing in the top-level directory of a distribution, the > current directory still comes first in sys.path, so python still > thinks the package source directory (with its __init__.py) is where > the package lives. Yeah, I see the problem -- putting test scripts in a test directory is not a full solution; what about interactive sessions? And while it's good practice to put test scripts in a test directory, I don't want to *force* people to do that. > 2) Even when executing an example script in a sub-directory, python > gets the idea that the package is in build/lib and doesn't find it's > extension submodules. If I copy those over to the build/lib > subdirectory they are found. Oops. This is handled properly on installation, since build/lib and build/platlib are merged at that point. I've been wondering for a while why I decided always to build pure Python modules to build/lib, and extensions to build/platlib. You have discovered a very good reason *not* to do so: it breaks the idea of the build tree as a pseudo-installation for testing purposes. > If I think of good strategies for dealing with these problems, I'll > post them. In the meantime, it's back to Makefiles for me. :) I see a couple of solutions: first, put all modules in either build/lib or build/platlib, i.e. do the merge at build-time rather than install-time. Second, have some sort of "build in-place" option that lets developers, well, build in-place. That is, .py files in the source tree wouldn't be touched (they're already in-place!), and extension modules would be put into the source tree. (I still favour putting compiler turds in build/temp.plat -- anyone foresee problems with that?) I'm still struggling with the Right Way to install modules, so I won't be able to work on fixing the build stuff right away. Keep bugging me though! Greg -- Greg Ward - Linux bigot gward@python.net http://starship.python.net/~gward/ A man wrapped up in himself makes a very small package.