From gward@cnri.reston.va.us Sun Sep 12 21:41:27 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Sun, 12 Sep 1999 16:41:27 -0400 Subject: [Distutils] New snapshot with MSVCCompiler integrated Message-ID: <19990912164125.A731@cnri.reston.va.us> Hi all -- this code has been in the CVS archive for a few days, but I've slacked off on making a snapshot and announcing it to the list because I knew there were problems with it (there still are). But, it's probably worth announcing anyways that I've integrated Perry Stoll's patch for NT and his MSVCCompiler class. Hooray! There's also an example setup.py that successfully builds Numeric Python, *but* it's not in the source tree right now. Oops. It doesn't work on Unix because UnixCCompiler and MSVCCompiler (currently) deal differently with source files in other directories. I'm fixing that now in my current copy of the code, but it's not ready yet. I'll post the sample setup.py to the list separately -- it's worth discussing. Anyways, the current snapshot is http://www.python.org/sigs/distutils-sig/distutils-19990907.tar.gz ...help yourself if you enjoy mucking with buggy code. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gward@cnri.reston.va.us Sun Sep 12 21:59:39 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Sun, 12 Sep 1999 16:59:39 -0400 Subject: [Distutils] Sample setup.py for Numeric Python Message-ID: <19990912165938.A795@cnri.reston.va.us> Hi again -- [cc'd to Paul Dubois: you said you weren't following the distutils sig anymore, but this directly concerns NumPy and I'd like to get your input!] here's that sample setup.py for NumPy. See below for discussion (and questions!). ------------------------------------------------------------------------ #!/usr/bin/env python # Setup script example for building the Numeric extension to Python. # This does sucessfully compile all the .dlls. Nothing happens # with the .py files currently. # Move this file to the Numerical directory of the LLNL numpy # distribution and run as: # python numpysetup.py --verbose build_ext # # created 1999/08 Perry Stoll __rcsid__ = "$Id: numpysetup.py,v 1.1 1999/09/12 20:42:48 gward Exp $" from distutils.core import setup setup (name = "numerical", version = "0.01", description = "Numerical Extension to Python", url = "http://www.python.org/sigs/matrix-sig/", ext_modules = [ ( '_numpy', { 'sources' : [ 'Src/_numpymodule.c', 'Src/arrayobject.c', 'Src/ufuncobject.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/numpy.def' } ), ( 'multiarray', { 'sources' : ['Src/multiarraymodule.c'], 'include_dirs' : ['./Include'], 'def_file': 'Src/multiarray.def' } ), ( 'umath', { 'sources': ['Src/umathmodule.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/umath.def' } ), ( 'fftpack', { 'sources': ['Src/fftpackmodule.c', 'Src/fftpack.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/fftpack.def' } ), ( 'lapack_lite', { 'sources' : [ 'Src/lapack_litemodule.c', 'Src/dlapack_lite.c', 'Src/zlapack_lite.c', 'Src/blas_lite.c', 'Src/f2c_lite.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/lapack_lite.def' } ), ( 'ranlib', { 'sources': ['Src/ranlibmodule.c', 'Src/ranlib.c', 'Src/com.c', 'Src/linpack.c', ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/ranlib.def' } ), ] ) ------------------------------------------------------------------------ First, what d'you think? Too clunky and verbose? Too much information for each extension? I kind of think so, but I'm not sure how to reduce it elegantly. Right now, the internal data structures needed to compile a module are pretty obviously exposed: is this a good thing? Or should there be some more compact form for setup.py that will be expanded later into the full glory we see above? I've already made one small step towards reducing the amount of cruft by factoring 'include_dirs' out and supplying it directly as a parameter to 'setup()'. (But that needs code not in the CVS archive yet, so I've left the sample setup.py the same for now.) The next thing I'd like to do is get that damn "def_file" out of there. To support it in MSVCCompiler, there's already an ugly hack that unnecessarily affects both the UnixCCompiler and CCompiler classes, and I want to get rid of that. (I refer to passing the 'build_info' dictionary into the compiler classes, if you're familiar with the code -- that dictionary is part of the Distutils extension-building system, and should not propagate into the more general compiler classes.) But I don't want to give these weird "def file" things standing on the order of source files, object files, libraries, etc., because they seem to me to be a bizarre artifact of one particular compiler, rather than something present in a wide range of C/C++ compilers. Based on the NumPy model, it seems like there's a not-too-kludgy way to handle this problem. Namely: if building extension "foo": if file "foo.def" found in same directory as "foo.c" add "/def:foo.def" to MSVC command line this will of course require some platform-specific code in the build_ext command class, but I figured that was coming eventually, so why put it off? ;-) To make this hack work with NumPy, one change would be necessary: rename Src/numpy.def to Src/_numpy.def to match Src/_numpy.c, which implements the _numpy module. Would this be too much to ask of NumPy? (Paul?) What about other module distributions that support MSVC++ and thus ship with "def" files? Could they be made to accomodate this scheme? Thanks for your feedback -- Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From s341625@student.uq.edu.au Mon Sep 13 03:02:43 1999 From: s341625@student.uq.edu.au (Anthony Pfrunder) Date: Mon, 13 Sep 1999 12:02:43 +1000 (GMT+1000) Subject: [Distutils] Sample setup.py for Numeric Python In-Reply-To: <19990912165938.A795@cnri.reston.va.us> Message-ID: Hi, Thanks for the snapshot. RE the proposed script file format - yes, I think it is too verbose. My suggestion is to allow importing and exporting various types of makefiles (ie Setup, dsw/dsp, unix makefiles). This way, people can choose the way to build their extensions. Give me a couple of weeks and I'll port the code for this to the current setup format (assuming it doesn't change). The other suggestion is to add *aquisition* to the system. This is a concept invented / named by the DigiCool Zope team. I have found this is a fantasic way to handle platform dependances. I have used this in my Zope build system to abstract out the def problem [Greg, do you still want the Zope build system? I don't have full access to the net so I cannot use CVS but I can email it to you]. Basically I use a make file style format where platform deps are handled by loadable makefiles which override or modify rules. For example, MyExtension: SomeDirectory$/Setup $(PLATFORM)$/make.dsw \ MyExtensionMetaRule file.c file1.c MyExtension.pyd Here, MyExtension imports from a Setup file, a MSVC workspace file and builds the resulting MetaRule (which is modified by the previous 2 imports), the two c sources and finally generates the pyd. As each c file is compiled the resulting object files are added to the global variable OBJS. Then, if the generation of the pyd uses a compiler needing def files (or any others...) it is generated as a temp file, used and then deleted. The OBJS variable is then reset. This is what I mean by aquisition. A few bits 'n pieces to add... We need to accomedate people using diverse platforms. This implies using verbose os.path.join for all paths OR by supplying a platform to generic converter. This converter converts platform file specs to $/ format. The $/ is replaced with os.path.sep upon load hence autoconverting to the right format. Other metachars include the $(..) format for variables. The above system has been tested on a subset of Mark Hammond's win32 extensions, the Python Imaging Extensions (including numeric py i think) and other misc stuff. The other big thing is that I have converted the entire Python build system into this format so once I write an exporter you can build python cross-platform using dist-utils. I have removed the need for SED 'n friends and merged freeze and Gordon's freezer so that if you add pyc/pyo's to the source line it will either freeze it into the exe or the currently active MPZ file. The only thing missing now is a cross-platform point 'n click extension installer! I think WxPython should do the job ;) Cheers, Anthony Pfrunder Computer Systems Engineering University of Queensland From mal@lemburg.com Mon Sep 13 11:41:25 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 12:41:25 +0200 Subject: [Distutils] Sample setup.py for Numeric Python References: <19990912165938.A795@cnri.reston.va.us> Message-ID: <37DCD4D5.60304DA@lemburg.com> Greg Ward wrote: > > [sample setup.py] > ext_modules = [ ( '_numpy', { 'sources' : [ 'Src/_numpymodule.c', > 'Src/arrayobject.c', > 'Src/ufuncobject.c' > ], > 'include_dirs' : ['./Include'], > 'def_file' : 'Src/numpy.def' } > ), > > [/] > > First, what d'you think? Too clunky and verbose? Too much information > for each extension? I kind of think so, but I'm not sure how to reduce > it elegantly. Right now, the internal data structures needed to compile > a module are pretty obviously exposed: is this a good thing? Or should > there be some more compact form for setup.py that will be expanded later > into the full glory we see above? I'd suggest using a class based approach for the ext_modules data,e.g. class ExtModule: def_file_template = 'Src/%s.def' def get_includedirs(self): return ['./Include'] def get_def_file(self): return self.def_file_template % self.name # etc. class _numpy(ExtModule): name = 'numpy' sources = ('_numpymodule.c','arraymodule.c',...) setup() would then be passed a list of instances (or classes which it then instantiates): setup(..., ext_modules = [_numpy(),...], ...) Advantages are greater incapsulation and more flexibility due to the fact that naming schemes used in the external module can be incoporated into the classes. > I've already made one small step towards reducing the amount of cruft by > factoring 'include_dirs' out and supplying it directly as a parameter to > 'setup()'. (But that needs code not in the CVS archive yet, so I've > left the sample setup.py the same for now.) > > The next thing I'd like to do is get that damn "def_file" out of there. AFAIK, you don't need the def-files anymore. All you have to do is use a tag on every function you wish to export in the source code. Python already works this way and so do all my extensions (the needed macros are in the file mxh.h). -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gward@cnri.reston.va.us Mon Sep 13 14:45:23 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Mon, 13 Sep 1999 09:45:23 -0400 Subject: [Distutils] Sample setup.py for Numeric Python In-Reply-To: <000501befd7a$771ed7a0$3c810b18@plstn1.sfba.home.com>; from Paul F. Dubois on Sun, Sep 12, 1999 at 04:56:43PM -0700 References: <19990912165938.A795@cnri.reston.va.us> <000501befd7a$771ed7a0$3c810b18@plstn1.sfba.home.com> Message-ID: <19990913094522.A1086@cnri.reston.va.us> On 12 September 1999, Paul F. Dubois said: > I'm eager to try it but please (a) send it as an attachment and (b) send me > also as an attachment the archive for distutils that goes with it. There are a few more, umm, "issues" to be resolved before I go pushing the code into your mailbox. (Ie. anyone is welcome to download buggy code, but it's not yet stable enough that I'm going to go foisting it off on people. Even if they do ask for it. ;-) I need to cook up some simpler test cases than NumPy and see how they work, etc. etc. I'll toss the code your way when that's done. > The syntax required of the user isn't that bad. However, I favor using a > little language parser so that the input is less > computer-sciency: > > version 2.0 > "Numerical extension to Python" > external module _numpy > sources Src/_numpymodule.c Src/arrayobject.c > includes Include > defs numpy.def > end Hmmm... I've been thinking something along those lines might be needed, but why invent another little language? Might it be better just to parse the current "Setup" file? After all, it's got the list of source files, plus -I, -L, and -l options. If I extend it to allow -D and -U options, then interpret all that in a compiler/platform-neutral way, that could solve most of the problem. The .def files are still troublesome, but Marc-Andre just posted to the distutils-sig that they may not be needed anymore. Not sure of the details. > The implementation can be to create the data structure in question and > invoke your existing software, > so I'm not talking about starting over. Exactly -- that's why I'm making the current code public; the front-end may change, but the guts are reasonably solid at this point. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From da@ski.org Mon Sep 13 18:56:26 1999 From: da@ski.org (David Ascher) Date: Mon, 13 Sep 1999 10:56:26 -0700 (Pacific Daylight Time) Subject: [Distutils] Sample setup.py for Numeric Python In-Reply-To: <37DCD4D5.60304DA@lemburg.com> Message-ID: On Mon, 13 Sep 1999, M.-A. Lemburg wrote: > AFAIK, you don't need the def-files anymore. All you have to do > is use a tag on every function you wish to export in the source > code. Python already works this way and so do all my extensions > (the needed macros are in the file mxh.h). But that requires modifying the source, which is IMHO unacceptable. There is an alternative which is to specify the exports w/ a command line switch. --david From mal@lemburg.com Mon Sep 13 19:35:43 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 20:35:43 +0200 Subject: [Distutils] Sample setup.py for Numeric Python References: Message-ID: <37DD43FF.53523BA5@lemburg.com> David Ascher wrote: > > On Mon, 13 Sep 1999, M.-A. Lemburg wrote: > > > AFAIK, you don't need the def-files anymore. All you have to do > > is use a tag on every function you wish to export in the source > > code. Python already works this way and so do all my extensions > > (the needed macros are in the file mxh.h). > > But that requires modifying the source, which is IMHO unacceptable. Why is that unacceptable ? After all, package authors do have access to the code and know their way around ;-) Better drop the old def-file stuff and move to more modern ways of intergrating the information into the code... > There is an alternative which is to specify the exports w/ a command line > switch. ...and perhaps add this as backup solution. Note that the /EXPORT:symbol you need for every symbol may well fill up the available command line argument memory (not sure how much there is on WinXX). -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From da@ski.org Tue Sep 14 04:58:02 1999 From: da@ski.org (David Ascher) Date: Mon, 13 Sep 1999 20:58:02 -0700 (Pacific Daylight Time) Subject: [Distutils] Sample setup.py for Numeric Python In-Reply-To: <37DD43FF.53523BA5@lemburg.com> Message-ID: > Why is that unacceptable ? After all, package authors do have > access to the code and know their way around ;-) Better drop > the old def-file stuff and move to more modern ways of > intergrating the information into the code... Well, what strikes me as unacceptable is: 1) it makes creating setup.py files for existing code impossible w/o modifying said code. This will make migration more difficult. 2) it introduces platform-dependencies related to compilers in the code, instead of in compiler-spefific metafiles, which seems wrong. > ...and perhaps add this as backup solution. Note that the /EXPORT:symbol > you need for every symbol may well fill up the available command > line argument memory (not sure how much there is on WinXX). I've never run into it, if it exists. --david From kernr@ncifcrf.gov Tue Sep 14 07:19:38 1999 From: kernr@ncifcrf.gov (Robert Kern) Date: Tue, 14 Sep 1999 02:19:38 -0400 Subject: [Distutils] DEF Files Message-ID: <37DDE8FA.DB02F1D4@mail.ncifcrf.gov> I may be missing something, but is it permissible for distutils to generate a temporary DEF file that exists only until the end of the compilation? If the initialization function of our extension is 'initfoo' then the DEF file is as follows: """EXPORTS initfoo """ Leading whitespace is optional. I don't believe that anything else is necessary. Since the mapping from module name to function name is one-to-one, distutils already has the required information at hand. FWIW, I agree with David that requiring modifying the source of an extension is not a good thing. It only adds a headache for people trying to port from *NIX to Windows (and we have enough to deal with, believe me), and it pollutes the code with weird MS extensions. -- Robert Kern | ----------------------|"In the fields of Hell where the grass grows high This space | Are the graves of dreams allowed to die." intentionally | - Richard Harter left blank. | From s341625@student.uq.edu.au Tue Sep 14 10:38:28 1999 From: s341625@student.uq.edu.au (Anthony Pfrunder) Date: Tue, 14 Sep 1999 19:38:28 +1000 (GMT+1000) Subject: [Distutils] DEF Files In-Reply-To: <37DDE8FA.DB02F1D4@mail.ncifcrf.gov> Message-ID: On Tue, 14 Sep 1999, Robert Kern wrote: > I may be missing something, but is it permissible for distutils to > generate a temporary DEF file that exists only until the end of the > compilation? That's what I currently do for cygwin - msvc uses the /exports version instead (btw how do you tell it via command line options not to generate big fat files that are a waste of space?? (ie precompiled headers ;) > FWIW, I agree with David that requiring modifying the source of an > extension is not a good thing. It only adds a headache for people trying to port from *NIX to > Windows (and we have enough to deal with, > believe me), and it pollutes the code with weird MS extensions. ... and weird unix only utilites such as sed [sorry, my Windows side speaking] Isn't the point of distutils to provide a cross-platform compile system? Cheers, Anthony Pfrunder Computer Systems Engineering University of Queensland From mal@lemburg.com Tue Sep 14 11:13:41 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 14 Sep 1999 12:13:41 +0200 Subject: [Distutils] DEF Files References: Message-ID: <37DE1FD5.314E07A9@lemburg.com> Anthony Pfrunder wrote: > > On Tue, 14 Sep 1999, Robert Kern wrote: > > > FWIW, I agree with David that requiring modifying the source of an > > extension is not a good thing. It only adds a headache for people trying to port from *NIX to > > Windows (and we have enough to deal with, > > believe me), and it pollutes the code with weird MS extensions. > ... and weird unix only utilites such as sed [sorry, my Windows side speaking] I don't really get your points here... you can stick all those weird symbols into macros and then forget about them. Writing... #include "mxh.h" MX_EXPORT(void) initmxODBC(void) { } ...isn't all that bad and makes the intention pretty clear. Here's the contents of mxh.h: /* Macro to "mark" a symbol for DLL export */ #if defined(_MSC_VER) && _MSC_VER > 850 /* MS VC++ 2.0 and up */ # ifdef __cplusplus # define MX_EXPORT(type) extern "C" type __declspec(dllexport) # else # define MX_EXPORT(type) extern type __declspec(dllexport) # endif #elif defined(__WATCOMC__) # define MX_EXPORT(type) extern type __export #else # define MX_EXPORT(type) extern type #endif /* Macro to "mark" a symbol for DLL import */ #if defined(__BORLANDC__) # define MX_IMPORT(type) extern type __import #elif defined(_MSC_VER) && _MSC_VER > 850 /* MS VC++ 2.0 and up */ # ifdef __cplusplus # define MX_IMPORT(type) extern "C" type __declspec(dllimport) # else # define MX_IMPORT(type) extern type __declspec(dllimport) # endif #else # define MX_IMPORT(type) extern type #endif BTW, additions to support more compilers are always welcome. The above works for pretty much all Unix, Mac and Windows compilers people have used to compile my extensions, so I guess its pretty generic. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 108 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From Fred L. Drake, Jr." References: <37DE1FD5.314E07A9@lemburg.com> Message-ID: <14302.19072.817477.658966@weyr.cnri.reston.va.us> M.-A. Lemburg writes: > I don't really get your points here... you can stick all those > weird symbols into macros and then forget about them. In fact, when you: #include "Python.h" you get "the right stuff" defined as DL_IMPORT and DL_EXPORT. (Why these aren't called Py_DL_IMPORT and Py_DL_EXPORT, I have no idea; I think they should be.) -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From mal@lemburg.com Tue Sep 14 14:57:21 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 14 Sep 1999 15:57:21 +0200 Subject: [Distutils] DEF Files References: <37DE1FD5.314E07A9@lemburg.com> <14302.19072.817477.658966@weyr.cnri.reston.va.us> Message-ID: <37DE5441.5E6C79A1@lemburg.com> Fred L. Drake, Jr. wrote: > > M.-A. Lemburg writes: > > I don't really get your points here... you can stick all those > > weird symbols into macros and then forget about them. > > In fact, when you: > > #include "Python.h" > > you get "the right stuff" defined as DL_IMPORT and DL_EXPORT. (Why > these aren't called Py_DL_IMPORT and Py_DL_EXPORT, I have no idea; I > think they should be.) Interesting. The DL_EXPORT macro must be new. I hacked up my own because previously there was no explicit way to say "export this symbol", only "import this symbol" via DL_IMPORT (and export it under some other conditions). Perhaps I should move back to using the Python macros instead of mine. And yes, DL_IMPORT/EXPORT should have the Py prefix. Those names just sound like they could attract trouble ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 108 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From skip@mojam.com (Skip Montanaro) Tue Sep 14 16:34:48 1999 From: skip@mojam.com (Skip Montanaro) (Skip Montanaro) Date: Tue, 14 Sep 1999 10:34:48 -0500 Subject: [Distutils] relative package imports, version numbering, yadda yadda Message-ID: <199909141534.KAA03648@dolphin.mojam.com> I haven't been paying real close attention to the relative package import thread on python-dev, but some way or other I strongly believe something has to be done to improve the situation I now find myself in or more and more people are going to start getting bitten by the same sort of problem. Allow me to 'splain. I use a web server based on Zope's ZServer (which is itself based on medusa) sitting behind an Apache web server as a long-running process to handle what many people traditionally used CGI for. It's a big performance boost over CGI. Unfortunately, I have recently been experiencing frequent server hangs. So far I've been unable to figure out what the cause is. I do notice occasional tracebacks like: Unhandled exception in thread: Traceback (innermost last): File "ZServer/PubCore/ZServerPublisher.py", line 97, in __init__ File "/home/skip/src/Zope/ZServer/HTTPResponse.py", line 209, in _finish self.stdout.close() File "/home/skip/src/Zope/ZServer/HTTPResponse.py", line 235, in close self._channel.push(CallbackProducer(self._channel.done)) File "/home/skip/src/Zope/ZServer/HTTPServer.py", line 307, in push if send: self.initiate_send() File "/usr/lib/python1.5/asynchat.py", line 199, in initiate_send _push_mode = 0 File "/usr/lib/python1.5/asynchat.py", line 191, in refill_buffer self.ac_in_buffer = '' File "/usr/lib/python1.5/asynchat.py", line 253, in pop # this could maybe be made faster with a computed regex? IndexError: list index out of range and also notice that the server can pile up a huge number of connections in the ESTABLISHED state, at which point the whole mess grinds to a halt with not much computation or network traffic happening. A separate shell script uses netstat to detect when a large number of sockets have piled up and restarts the server. Brutal, but crudely effective. (When in doubt, treat the symptoms...) Today it just dawned on me looking at the above traceback that ZServer is getting the wrong version of asynchat.py (and presumably of asyncore.py as well). It's using the version delivered with Python 1.5.2 distribution instead of the version that was delivered with the version of Zope I'm using (1.something). So I started looking around at the versions and dates of various copies of asynchat.py. Here's what I found: source version number:date owner Python 1.5.2 1.2:1999/06/18 guido Python CVS 1.2:1999/06/18 guido Zope 1.? 1.7:1999/04/09 amos Zope 2.0 1.9:1999/07/19 amos Medusa 990902 2.24:1999/07/07 rushing What's apparently been happening is that people have picked up asyncore and asynchat at various time and stuck them in their own CVS repositories without somehow freezing the Id string of the version they originally got from Sam Rushing. It's not clear what the differences are until you compare the actual files. It turns out that the Zope 2.0 and Medusa versions have no content differences, only wildly different version numbers. The Medusa and Python CVS versions only have one difference: if index > 0: # don't bother reporting the empty string (source of subtle bugs) self.collect_incoming_data (self.ac_in_buffer[:index]) which *may* be what's causing my problems (note the IndexError in my traceback). Somewhere along the way I think we need to apply more restraint to our packaging and numbering of Python modules. I don't know what form that restraint will eventually take, but at this point I'm willing to replace Greg Ward's s/must/should/ with s/should/must/. a-nightmare-indeed!-i don't think-this-is-what-Sam-meant-ly y'rs Skip Montanaro | http://www.mojam.com/ skip@mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From da@ski.org Tue Sep 14 17:18:58 1999 From: da@ski.org (David Ascher) Date: Tue, 14 Sep 1999 09:18:58 -0700 (Pacific Daylight Time) Subject: [Distutils] DEF Files In-Reply-To: <14302.19072.817477.658966@weyr.cnri.reston.va.us> Message-ID: > M.-A. Lemburg writes: > > I don't really get your points here... you can stick all those > > weird symbols into macros and then forget about them. > > In fact, when you: > > #include "Python.h" > > you get "the right stuff" defined as DL_IMPORT and DL_EXPORT. (Why > these aren't called Py_DL_IMPORT and Py_DL_EXPORT, I have no idea; I > think they should be.) Good point, but then I think the docuemntation should emphasize this (it may have been doing this in recent versions, I don't know.). --david From jim@interet.com Tue Sep 14 18:59:03 1999 From: jim@interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 13:59:03 -0400 Subject: [Distutils] Re: [Python-Dev] relative package imports, version numbering, yadda yadda References: <199909141534.KAA03648@dolphin.mojam.com> Message-ID: <37DE8CE7.F211F6FA@interet.com> Skip Montanaro wrote: > So I started looking around at the versions and dates of various copies of > asynchat.py. >... > What's apparently been happening is that people have picked up asyncore and > asynchat at various time and stuck them in their own CVS repositories > without somehow freezing the Id string of the version they originally got > from Sam Rushing. It's not clear what the differences are until you compare > the actual files. It turns out that the Zope 2.0 and Medusa versions have Yes, I have had this happen too. I am unwilling and unable to risk this sort of problem at a customer's site. So I ship a complete app with no external dependencies. Also crude but effective. Jim Ahlstrom From gstein@lyra.org Tue Sep 14 19:55:47 1999 From: gstein@lyra.org (Greg Stein) Date: Tue, 14 Sep 1999 11:55:47 -0700 (PDT) Subject: [Distutils] relative package imports, version , numbering, yadda yadda In-Reply-To: <37DE8CE7.F211F6FA@interet.com> Message-ID: On Tue, 14 Sep 1999, James C. Ahlstrom wrote: > Skip Montanaro wrote: > > So I started looking around at the versions and dates of various copies of > > asynchat.py. > >... > > What's apparently been happening is that people have picked up asyncore and > > asynchat at various time and stuck them in their own CVS repositories > > without somehow freezing the Id string of the version they originally got > > from Sam Rushing. It's not clear what the differences are until you compare > > the actual files. It turns out that the Zope 2.0 and Medusa versions have > > Yes, I have had this happen too. I am unwilling and unable to risk > this sort of problem at a customer's site. So I ship a complete > app with no external dependencies. Also crude but effective. We did the same with the Python-based apps/servers at Microsoft. Merchant Server was a big frozen app (based on the non-sensical requirement to hide the fact that Python was used). In Site Server 2.0 and 3.0, we used a mini-install -- just the Lib files we needed plus our extensions. In the Site Server (non-frozen) case, we did use the registry, but built the interpreter with a custom "version". SS20 and SS30. In the registry, this meant we used Python/PythonCore/SS20 (I think that's the layout). Worked great, no complaints. Oh, and the DLLs we put into the system directory (pretty necessary for COM) were named SS20.DLL to prevent conflicts. I'm not sure who said it, but I agree with the following statement: * ship your app as a complete black box, or ship your app with dependencies on modules/packages [at the top level] This monkeying around with "mx.foo" working where mx is at the top level or is embedded is just scary. That said, personally, I would just do something like the following at the startup of my app: ZopeImporter("zopedir").install() [where ZopeImporter is a imputil.Importer subclass] The importer would just Do The Right Thing for all imports, and only defer to the Python library for things that weren't shipped with Zope (the empty set?) Cheers, -g -- Greg Stein, http://www.lyra.org/ From guido@CNRI.Reston.VA.US Tue Sep 14 21:09:07 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 14 Sep 1999 16:09:07 -0400 Subject: [Distutils] Re: [Python-Dev] relative package imports, version numbering, yadda yadda In-Reply-To: Your message of "Tue, 14 Sep 1999 10:34:48 CDT." <199909141534.KAA03648@dolphin.mojam.com> References: <199909141534.KAA03648@dolphin.mojam.com> Message-ID: <199909142009.QAA15534@eric.cnri.reston.va.us> > So I started looking around at the versions and dates of various copies of > asynchat.py. Here's what I found: > > source version number:date owner > Python 1.5.2 1.2:1999/06/18 guido > Python CVS 1.2:1999/06/18 guido > Zope 1.? 1.7:1999/04/09 amos > Zope 2.0 1.9:1999/07/19 amos > Medusa 990902 2.24:1999/07/07 rushing > > What's apparently been happening is that people have picked up asyncore and > asynchat at various time and stuck them in their own CVS repositories > without somehow freezing the Id string of the version they originally got > from Sam Rushing. It's not clear what the differences are until you compare > the actual files. It turns out that the Zope 2.0 and Medusa versions have > no content differences, only wildly different version numbers. The Medusa > and Python CVS versions only have one difference: > > if index > 0: > # don't bother reporting the empty string (source of subtle bugs) > self.collect_incoming_data (self.ac_in_buffer[:index]) > > which *may* be what's causing my problems (note the IndexError in my > traceback). My bad. I despise putting RCS identifiers in code I release, but I don't always freeze them before incorporating other people's code in my CVS tree. I will fix this if I can and I promise to try not to repeat this mistake in the future. --Guido van Rossum (home page: http://www.python.org/~guido/) From kernr@ncifcrf.gov Tue Sep 14 22:32:08 1999 From: kernr@ncifcrf.gov (Robert Kern) Date: Tue, 14 Sep 1999 17:32:08 -0400 Subject: [Distutils] DEF Files References: <37DE1FD5.314E07A9@lemburg.com> Message-ID: <37DEBED8.553F47D8@mail.ncifcrf.gov> "M.-A. Lemburg" wrote: > > Anthony Pfrunder wrote: > > > > On Tue, 14 Sep 1999, Robert Kern wrote: > > > > > FWIW, I agree with David that requiring modifying the source of an > > > extension is not a good thing. It only adds a headache for people trying to port from *NIX to > > > Windows (and we have enough to deal with, > > > believe me), and it pollutes the code with weird MS extensions. > > ... and weird unix only utilites such as sed [sorry, my Windows side speaking] > > I don't really get your points here... you can stick all those > weird symbols into macros and then forget about them. For modules that will be written in the future, yes, we can insist that all extensions use a macro like yours. Python's code uses DL_IMPORT to do the same thing. But since we can generate the DEF files so easily, we don't have to require any source code changes for already-existing modules. > Writing... > > #include "mxh.h" > MX_EXPORT(void) > initmxODBC(void) > { > } > > ...isn't all that bad and makes the intention pretty clear. Here's > the contents of mxh.h: > > /* Macro to "mark" a symbol for DLL export */ > > #if defined(_MSC_VER) && _MSC_VER > 850 /* MS VC++ 2.0 and up */ > # ifdef __cplusplus > # define MX_EXPORT(type) extern "C" type __declspec(dllexport) > # else > # define MX_EXPORT(type) extern type __declspec(dllexport) > # endif > #elif defined(__WATCOMC__) > # define MX_EXPORT(type) extern type __export > #else > # define MX_EXPORT(type) extern type > #endif > > /* Macro to "mark" a symbol for DLL import */ > > #if defined(__BORLANDC__) > # define MX_IMPORT(type) extern type __import > #elif defined(_MSC_VER) && _MSC_VER > 850 /* MS VC++ 2.0 and up */ > # ifdef __cplusplus > # define MX_IMPORT(type) extern "C" type __declspec(dllimport) > # else > # define MX_IMPORT(type) extern type __declspec(dllimport) > # endif > #else > # define MX_IMPORT(type) extern type > #endif > > BTW, additions to support more compilers are always welcome. You can add "|| defined(__MINGW32__)" to the lines with _MSC_VER. > The above works for pretty much all Unix, Mac and Windows compilers > people have used to compile my extensions, so I guess its pretty > generic. > > -- > Marc-Andre Lemburg -- Robert Kern | ----------------------|"In the fields of Hell where the grass grows high This space | Are the graves of dreams allowed to die." intentionally | - Richard Harter left blank. | From s341625@student.uq.edu.au Tue Sep 14 23:52:27 1999 From: s341625@student.uq.edu.au (Anthony Pfrunder) Date: Wed, 15 Sep 1999 08:52:27 +1000 (GMT+1000) Subject: [Distutils] DEF Files In-Reply-To: <37DE1FD5.314E07A9@lemburg.com> Message-ID: On Tue, 14 Sep 1999, M.-A. Lemburg wrote: > Anthony Pfrunder wrote: > > > > On Tue, 14 Sep 1999, Robert Kern wrote: > > > > > FWIW, I agree with David that requiring modifying the source of an > > > extension is not a good thing. It only adds a headache for people trying to port from *NIX to > > > Windows (and we have enough to deal with, > > > believe me), and it pollutes the code with weird MS extensions. > > ... and weird unix only utilites such as sed [sorry, my Windows side speaking] > > I don't really get your points here... you can stick all those > weird symbols into macros and then forget about them. I was refering to the current process for building python. At the moment, non-unix platforms use a totally different build system to that of unix. This means that when new modules are added several build systems need to be updated instead of one. distutil should provide a mechanism so that only one build system should be updated. Also, this build system should use python as much as possible and not rely on platform specific extensions such as SED or /export: except in platform specific meta-files. The approach I use for handling source mods is to create a generic macro which is then included from the appropiate compiler meta-file. The disadvantage of lumping all the #defines in one place is that it makes it "look" more complex as it has elements not relevate to the users compilation platform. These source mods (compiler-metafiles) are then applied using a context diff, compiled and then removed. This means that the user is looking at the generic version of the source with no platform specific stuff to add complexity. If they want a platform specific version they say "make source-version-$(PLATFORM)" to get this (for debugging etc). Cheers, Anthony Pfrunder Computer Systems Engineering University of Queensland From mhammond@skippinet.com.au Wed Sep 15 00:15:57 1999 From: mhammond@skippinet.com.au (Mark Hammond) Date: Wed, 15 Sep 1999 09:15:57 +1000 Subject: [Distutils] DEF Files In-Reply-To: <37DE5441.5E6C79A1@lemburg.com> Message-ID: <002b01beff07$1a14fe70$0801a8c0@bobcat> [Marc writes] > Interesting. The DL_EXPORT macro must be new. I hacked up my > own because previously there was no explicit way to say "export > this symbol", only "import this symbol" via DL_IMPORT (and export > it under some other conditions). It has been around for as long as I have :-) In fact, I prompted Guido to include it when I first tackled the Windows port - he had similar issues on other OS' even then, so did not object (actually - there may have already been part of the scheme in place even then) > Perhaps I should move back to using the Python macros instead > of mine. Except they serve a slightly different purpose - they indicate the linkage for Python function - ie, when building Python DL_EXPORT is defined as "__declspec(dlexport)", otherwise defined as "__declspec(dlimport)". Thus, when building mxTools, you are not building Python itself, so this will resolve to "dlimport", which is clearly not what you want. For my extensions I simply use my own macros with the same intent - as you are already doing, and sa you will probably need to continue to do. Mark. From mal@lemburg.com Wed Sep 15 09:29:37 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 15 Sep 1999 10:29:37 +0200 Subject: [Distutils] DEF Files References: <002b01beff07$1a14fe70$0801a8c0@bobcat> Message-ID: <37DF58F1.2DFA7988@lemburg.com> Mark Hammond wrote: > > [Marc writes] > > > Interesting. The DL_EXPORT macro must be new. I hacked up my > > own because previously there was no explicit way to say "export > > this symbol", only "import this symbol" via DL_IMPORT (and export > > it under some other conditions). > > It has been around for as long as I have :-) In fact, I prompted > Guido to include it when I first tackled the Windows port - he had > similar issues on other OS' even then, so did not object (actually - > there may have already been part of the scheme in place even then) Well, I just did a grep over Python 1.5 and 1.5.2: the DL_EXPORT macro is not defined in 1.5, only in 1.5.2. There is some machinery in place to get the semantics you describe below. In 1.5.2 there is a new DL_EXPORT macro defined in Include/Python.h but it also uses the DL_IMPORT semantics... which are useless for me since I am not building Python, but my own modules. > > Perhaps I should move back to using the Python macros instead > > of mine. > > Except they serve a slightly different purpose - they indicate the > linkage for Python function - ie, when building Python DL_EXPORT is > defined as "__declspec(dlexport)", otherwise defined as > "__declspec(dlimport)". Thus, when building mxTools, you are not > building Python itself, so this will resolve to "dlimport", which is > clearly not what you want. > > For my extensions I simply use my own macros with the same intent - as > you are already doing, and sa you will probably need to continue to > do. Ok. Thanks for the insight, Mark. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 107 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal@lemburg.com Wed Sep 15 09:33:10 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 15 Sep 1999 10:33:10 +0200 Subject: [Distutils] DEF Files References: <37DE1FD5.314E07A9@lemburg.com> <37DEBED8.553F47D8@mail.ncifcrf.gov> Message-ID: <37DF59C6.2B304095@lemburg.com> Robert Kern wrote: > > "M.-A. Lemburg" wrote: > > > > Anthony Pfrunder wrote: > > > > > > On Tue, 14 Sep 1999, Robert Kern wrote: > > > > > > > FWIW, I agree with David that requiring modifying the source of an > > > > extension is not a good thing. It only adds a headache for people trying to port from *NIX to > > > > Windows (and we have enough to deal with, > > > > believe me), and it pollutes the code with weird MS extensions. > > > ... and weird unix only utilites such as sed [sorry, my Windows side speaking] > > > > I don't really get your points here... you can stick all those > > weird symbols into macros and then forget about them. > > For modules that will be written in the future, yes, we can insist that > all extensions use a macro like yours. Python's code uses DL_IMPORT > to do the same thing. But since we can generate the DEF files so > easily, we don't have to require any source code changes for > already-existing modules. Ok -- was just a thought. > > [...] > > BTW, additions to support more compilers are always welcome. > > You can add "|| defined(__MINGW32__)" to the lines with _MSC_VER. Thanks, Robert. BTW, I've looked at the new BeOS port included in the standard distribution. Anyone know details on what to use there (it seems to use the same symbols as MSVC). > > The above works for pretty much all Unix, Mac and Windows compilers > > people have used to compile my extensions, so I guess its pretty > > generic. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 107 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From guido@CNRI.Reston.VA.US Thu Sep 16 19:53:52 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Thu, 16 Sep 1999 14:53:52 -0400 Subject: [Distutils] Reminder: Python Conference papers due soon! Message-ID: <199909161853.OAA01738@eric.cnri.reston.va.us> The deadline for paper submissions for the next Python conference is nearing! Make sure that if you are considering submitting a paper, the program chair sees your paper by SEPTEMBER 30. Papers can be on any Python, JPython or Zope related subject. Papers will be reviewed and the best papers will be selected for presentation. You will hear about selection by October 22. For more information, see the official call for papers: http://www.python.org/workshops/2000-01/cfp.html ** We are planning a significant Zope presence at this conference! ** The conference will be held from January 24-27 in Alexandria, VA (just across the Potomac from DC). Make Python the first conference in the new millennium you visit! For more info, see the conference home page: http://www.python.org/workshops/2000-01/ Send all email questions about papers to: ipc8papers@cs.uchicago.edu --Guido van Rossum (home page: http://www.python.org/~guido/) (And sorry for the broad spamming.) From gward@cnri.reston.va.us Tue Sep 21 21:49:32 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 21 Sep 1999 16:49:32 -0400 Subject: [Distutils] Can now build & install real distributions Message-ID: <19990921164932.A4301@cnri.reston.va.us> Well, the last bit of discussion seems to have died down again here, so I guess it's time I kicked off another one. This time, it's good news: the Distutils are now able to build and install three non-trivial module distributions (Numeric Python, mxDateTime, and PIL). Four, if you count Distutils itself (easier because it's pure Python, but I still had to worry about packages and stuff to get it right). WHAT HAS CHANGED AND HOW IT WORKS --------------------------------- [this is long -- if you're not interested, just skip down to "TRYING IT OUT"] I should explain what's different, since the last time I made an announcement here it was about the ability to build Numeric Python. That was with the example setup.py that Perry Stoll contributed; at that time, building only worked properly on Windows, because my UnixCCcompiler class didn't deal well with source files in another directory. That's been fixed. Also, Perry's setup.py didn't deal with the pure Python modules in Numeric -- now fixed (tiny change to the setup.py, big changes behind the scenes in the Distutils build_py command). Dealing with pure Python modules actually got a lot more complex inside Distutils, but a lot simpler for module developers: in particular, the setup.py for Distutils itself no longer has to explicitly list all the modules to install. Rather, it just lists the packages to install from/to. To explain, let me dissect the four existing setup.py files. First, Distutils' own -- it's easiest because I laid out the source tree based on someone else's wisdom and experience (thanks Fred!). Here's the relevant bit from the Distutils setup script: setup ([...meta-data omitted...] packages = ['distutils', 'distutils.command'], ) Yep, that's all it takes: one line to say, "Please build and install all pure Python modules in the 'distutils' and 'distutils.command' packages." So how does it find those modules? Easy, it does the most sensible thing: package 'distutils' is assumed to be in the directory 'distutils' under the distribution directory (ie. where setup.py lives). So, saying "packages = ['foo']" means "Please install foo/*.py, oh and while you're at it make sure I included foo/__init__.py." But what if you don't follow that layout? What if you install to a package, but the source files for that package are right in the distribution directory alongside setup.py? No problem: just use the 'package_dir' option. For example, mxDateTime works this way, so in the example setup.py file for it (examples/mxdatetime_setup.py in the Distutils distribution), you'll see: setup (# ... packages = ['DateTime', 'DateTime.Examples', 'DateTime.mxDateTime'], package_dir = {'DateTime': ''}, # ... ) Together, those two options mean: "please install ./*.py, ./Examples/*.py, and ./mxDateTime/*.py -- oh, and by the way, make sure I included ./__init__.py, ./Examples/__init__.py, and ./mxDateTime/__init__.py." (Oops! I just realized that this means Distutils will install setup.py, which it probably shouldn't do. D'oh! Well, that's another argument against organizing your source this way. ;-) BTW, if you're really pathological, you can supply an entry in 'package_dir' for each package you install. But note that normally Distutils will do the Right Thing: when the package_dir for "DateTime" is '' (the current directory), it looks for sub-packages of "DateTime" in subdirectories of the current directory. Thus, more complicated directory layouts are certainly possible, but involve a bit of effort in your setup.py. (My subtle way of discouraging such complexity. ;-) Another way to lay things out that is essentially the same is used by Numeric Python: the .py files live in "Lib". The difference is that Numeric is not package-ized, so we have to deal with giving it a directory and a .pth file. For example, my first pass at supporting Numeric's pure Python modules looked like this setup (# ... packages = [''], package_dir = {'': 'Lib'}, # ... ) which of course dumped a whole bunch of stuff (Numeric's Lib/*.py) right in my "site-packages" directory when I installed it. Oops. The fix is to add an "install_path" option: setup (# ... packages = [''], package_dir = {'': 'Lib'}, install_path = 'Numeric', # ... ) this then interposes "Numeric" between 'install_site_platlib' (err, I think that's what the installation directory for module distributions that include extensions is called... anyways, it's just the "site-packages" dir under exec-prefix) and where files get installed, *and* creates Numeric.pth in 'install_site_platlib'. If for some reason you want your .pth file to have a different name from the directory it refers to, no problem: install_path = ('foo','bar/baz/qux') creates foo.pth which contains "bar/baz/qux", as does install_path = "foo,bar/baz/qux" (The string/tuple duality is needed because 'install_path' can also be specified on the command line, eg. ./setup.py install --install-path=foo,bar/baz/qux has the same effect as the two previous examples.) Finally, PIL uses the same layout as Distutils: it provides a package "PIL" in the directory "./PIL" under the distribution directory. Thus: setup (# ... packages = ['PIL'], # ... ) Err, this assumes you're installing PIL in package form, which is what my current setup.py for PIL does. Unfortunately, it installs PIL's extension module at top-level. I think Fredrik will have to deal with this problem -- it boils down to PIL not being 100% packagized yet. To install PIL as a collection of top-level modules in their own directory with a .pth file, the way Numeric is done: setup (# ... packages = [''], package_dir = { '': 'PIL' }, install_path = 'PIL' # ... ) (Another oops: this will install PIL/__init__.py, which means having an __init__.py on a directory in sys.path. Not sure what the consequences of this are, but it just sounds like a bad idea. Again, it boils down to PIL being not-quite-fully packagized.) HOW EXTENSION MODULES ARE DEALT WITH ------------------------------------ Basically the same. Extension building and installing now knows about packages, but it still uses the same moderately clunky "you provide the low-level data structure that drives Distutils" mode. For example, the Numeric setup.py contains: ext_modules = [('_numpy', { 'sources' : ['Src/_numpymodule.c', 'Src/arrayobject.c', 'Src/ufuncobject.c'], 'def_file' : 'Src/numpy.def' } ) # ... I would dearly love to get rid of that "def_file" thing by requiring that .def files be named after their respective modules, ie. "Src/_numpy.def" here instead of "Src/numpy.def". Or just auto-generate them if that'll work. Another example: here's how mxDateTime's extension is defined: ext_modules = [('DateTime.mxDateTime.mxDateTime', { 'sources': ['mxDateTime/mxDateTime.c'], 'include_dirs': ['mxDateTime'], 'macros': [('HAVE_STRFTIME', None), ('HAVE_STRPTIME', None), ('HAVE_TIMEGM', None)], 'def_file': 'mxDateTime/mxDateTime.def' } )] Note that currently we'll require users to edit setup.py to comment out the HAVE_* macros that don't apply to their platform. Yechh! This *will* have to change, I'm just not sure how yet. The all-singing all-dancing solution is to reimplement (bits of) Autoconf in Python. On the bright side, note that the name of the extension is fully packagized: this tells Distutils where to put the .so (or .pyd) file. PIL's setup.py has a similar problem: ext_modules = \ [('_imaging', { 'sources': ['_imaging.c', 'decode.c', 'encode.c', 'map.c', 'display.c', 'outline.c', 'path.c'], # This must include the directories with the JPEG, # zlib, and Tcl/Tk header files (if installed) 'include_dirs': ['libImaging', '/usr/local/include], # Keep this for Tcl/Tk support 'macros': [('WITH_TKINTER', None)], # This must include the directories with the JPEG, zlib, # and Tcl/Tk libraries (whatever's available) 'library_dirs': ['libImaging', '/usr/local/lib'], # And this, of course, lists which of those external # libraries to link against (plus libImaging, which *must* # be included!) 'libraries': ['Imaging', 'jpeg', 'z', 'tcl8.0', 'tk8.0'] } )] At least here, I've commented what users would have to change. ;-) Still, this is not viable for the long term, and again this could be dealt with in an Autoconf-ish way -- but it would have to be a cross-platform Autoconf! TRYING IT OUT ------------- Intrigued? Interested? Want to play with it? First, download the current snapshot: http://www.python.org/sigs/distutils-sig/distutils-19990921.tar.gz Unpack it and dive in. Note there is no documentation, not so much as a README -- that's what this message is for. ;-) First, you want to install the Distutils proper: python setup.py -v install If *this* doesn't work, then we have real problems. ;-) -v (verbose) is optional but *highly* recommended while developing/ debugging. (If you're doing this, you're developing/debugging -- thanks for the help!) If you wish to peruse the source, distutils/core.py is the place to start. distutils/command/build_py.py implements the build_py command for building pure Python modules; figuring out the rest of the commands is left as an exercise for the reader. If you want to try out one of the example setup.py scripts, they're in the examples subdirectory. For instance, if you have unpacked Distutils to /tmp/Distutils, and the LLNL distribution to /tmp/LLNLDistribution11, then: cd /tmp/LLNLDistribution11/Numerical ln -s /tmp/distutils/examples/numpy_setup.py setup.py (or local equivalent, for non-Unix geeks) and then build and install Numeric: python ./setup.py -v install and ba-da-boom: you have one brand-spanking new Numeric Python installation. (If you have one already, you should rename it out of the way first.) Well, it takes a while with those enormous C source files in Numeric, but you get the idea. Similar shenanigans will get you an mxDateTime installation, or a partially package-ized PIL installation. (The problem with PIL is that its extension module isn't in a package, so it winds up in your site-packages directory. Yuck. I blame Fredrik, I'm sure he'll blame me. ;-) WHAT DOESN'T WORK YET --------------------- Arg, I'll save this for later. I need to get back to my day job. I'm sure if you try it out, you'll find plenty o' problems. (Especially Windows people -- I have made no attempt to keep MSVCCompiler up-to-date with some minor changes in UnixCCompiler, so I'm sure things will break. Send those patches in, please!) Oh, as soon as this *does* work on both Unix and Windows, I'm going to write up a spot of documentation and throw it out the door as Distutils 0.1. So let's get those patches rolling in... Enjoy -- Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gward@cnri.reston.va.us Thu Sep 23 14:46:48 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Thu, 23 Sep 1999 09:46:48 -0400 Subject: [Distutils] The "dist" command Message-ID: <19990923094647.A9390@cnri.reston.va.us> I realized last night that I can't honestly release Distutils 0.1 until it can distribute itself, i.e. until there's a "dist" command to create a source distribution. So I sat down and did a brain dump on what's needed to create source distributions. If you enjoy reading braindumps, please read this one and post your reactions. Possible modes of operation: - require an explicit manifest that lists every single file (presumably along with a way to auto-generate the manifest) - require an explicit manifest, but allow it to have globs or filename patterns of some kind (and also have auto-generation) - allow an explict manifest, but automatically augment it at runtime with the source files mentioned in 'packages', 'py_modules', and 'ext_modules' (and any other such things that might come along) I'm liking the third way. Possible gotchas: - redundant specification: 'packages' includes 'foo' and manifest includes 'foo/*.py' - obvious conflict: 'packages' includes 'foo' and manifest includes '! foo/*.py' (can't imagine why you'd want this) - subtle conflict: 'packages' includes 'foo' and manifest includes '! foo/bar.py' (this could well be desired: eg. exclude an experimental module from distribution) Syntax for the manifest file: - each line is by default a Unix-style glob specifying files to include - if the line starts with '!', then it's an exclude pattern - if the line just names a directory in the distribution, that directory is included recursively - if the first word of a line names a directory, then the remaining words are include/exclude patterns that are applied recursively under that directory example (ignoring auto-augmentation!): distutils/*.py distutils/command/*.py ! distutils/bleeding_edge.py examples/*.py examples/README smarter way (that *will* include distutils/command/bleeding_edge.py!) distutils *.py ! distutils/bleeding_edge.py examples !*~ !*.py[co] test !*~ !*.py[co] README setup.py The actual Distutils manifest (don't need to mention source files, README, setup.py -- they're automatically distributed!): examples !*~ !*.py[co] test !*~ !*.py[co] The algorithm that will make it work: files = stuff from 'packages', 'py_modules', 'ext_modules', plus README, setup.py, ... ? foreach spec in manifest file: if simple-include-spec: # "distutils/*.py" files.append (glob (spec)) elif simple-exclude-spec: # "! distutils/foo*" xfiles = glob (spec) remove all xfiles from files elif dir-spec: # "examples" (just a directory name) dir_specs = rest-of-words-on-line if dir_specs: for dspec in dir_specs: if include-spec: dir_files.append (recursive_glob (dir, dspec)) elif exclude-spec: xfiles = recursive_glob (dir, dspec) remove all xfiles from dir_files else: dir_files = recursive_glob (dir, '*') Other things we need to look for in creating a source distribution: - make sure there's a README - make sure the distribution meta-info is supplied and non-empty (*must* have name, version, ((author and author_email) or (maintainer and maintainer_email)), url Frills: - make sure the setup script is called "setup.py" - make sure the README refers to "setup.py" (ie. has a line matching /^\s*python\s+setup\.py/) A crazy idea that conflicts with having/requiring 'version' in setup.py: - make sure there's a version number in the "main file" (main file is __init__.py of first package, or the first module if no packages, or the first extension module if no pure Python modules) - XXX how do we look for __version__ in an extension module? - XXX do we import and look for __version__? or just scan source for /^__version__\s*=\s*"[^"]+"/ ? - what about 'version_from' as an alternative to 'version' -- then we know just where to search for the version -- no guessing about what the "main file" is If you have comments, better get 'em in quick -- I've already started implementing this. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From Fred L. Drake, Jr." References: <19990923094647.A9390@cnri.reston.va.us> Message-ID: <14314.25009.145143.624472@weyr.cnri.reston.va.us> Greg Ward writes: > I realized last night that I can't honestly release Distutils 0.1 until > it can distribute itself, i.e. until there's a "dist" command to create > a source distribution. So I sat down and did a brain dump on what's > - allow an explict manifest, but automatically augment it at runtime > with the source files mentioned in 'packages', 'py_modules', and > 'ext_modules' (and any other such things that might come along) Another possibility: If a manifest exists, it must be complete; a file MANIFEST should be considered an explicit manifest if present at the root of the package and default behavior is used. If there is no manifest file, the default should be something like (using your syntax, or trying to): * !*~ !*.bak !RCS !CVS (Presuming, of course, that a directory that matches an exclusion rule will not be recursed into.) > I'm liking the third way. Possible gotchas: > - redundant specification: 'packages' includes 'foo' and manifest > includes 'foo/*.py' Duplicate inclusions are not a problem as long as the file is only included once in the resulting file list. > - obvious conflict: 'packages' includes 'foo' and manifest > includes '! foo/*.py' (can't imagine why you'd want this) Perhaps a warning should be issued, but there's no real need to protect against operator error. The resulting dist would be broken because it was package improperly. The warning should be enough to get the operator to fix the file selection. > - subtle conflict: 'packages' includes 'foo' and manifest > includes '! foo/bar.py' (this could well be desired: eg. exclude > an experimental module from distribution) No conflict here unless the manifest says "! foo/__init__.py"; in that case, use a warning as discussed above. > Other things we need to look for in creating a source distribution: > - make sure there's a README Warnings would be sufficient if anything at all is needed; things like README and INSTALL are largely policy rather than requirements at this level; do you want to have to consider each possible name for README? I'd certainly consider calling it README.txt if it were reasonable to distribute the package to non-Unix systems. Some people call it READ.ME, and others README.1st; how many flavors have to be automatically detected? Let's shoot for as few special cases as possible. > - make sure the distribution meta-info is supplied and non-empty > (*must* have name, version, ((author and author_email) or > (maintainer and maintainer_email)), url Issue warnings, not errors; these are social engineering issues, which are better handled by warnings. > Frills: > - make sure the setup script is called "setup.py" > - make sure the README refers to "setup.py" (ie. has a line matching > /^\s*python\s+setup\.py/) The line: Run 'python setup.py build install' to install this package. would be reasonable; perhaps just /python\s+setup.py/? Hmm, no, because then wordwrap would cause the match to fail. Might have to enable . to match \n in that case (yes, I know there's a flag for that in re). > A crazy idea that conflicts with having/requiring 'version' in setup.py: Version should be explicitly specified somewhere. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From gward@cnri.reston.va.us Thu Sep 23 20:59:38 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Thu, 23 Sep 1999 15:59:38 -0400 Subject: [Distutils] The "dist" command In-Reply-To: <14314.25009.145143.624472@weyr.cnri.reston.va.us>; from Fred L. Drake, Jr. on Thu, Sep 23, 1999 at 01:21:53PM -0400 References: <19990923094647.A9390@cnri.reston.va.us> <14314.25009.145143.624472@weyr.cnri.reston.va.us> Message-ID: <19990923155938.A11455@cnri.reston.va.us> On 23 September 1999, Fred L. Drake, Jr. said: > Another possibility: If a manifest exists, it must be complete; a > file MANIFEST should be considered an explicit manifest if present at > the root of the package and default behavior is used. I'm not sure what you mean by "complete" -- if the defaults are sensible (get all source files mentioned in or implied by 'packages', 'py_modules', and 'extensions', and toss in README and setup.py) then the generated distribution will be "complete". (Minus documentation, test suite, etc.) > If there is no > manifest file, the default should be something like (using your > syntax, or trying to): > > * !*~ !*.bak !RCS !CVS > > (Presuming, of course, that a directory that matches an exclusion rule > will not be recursed into.) Hmm, that's a good idea. If you were to state the rule explicitly under my currently proposed syntax, I think it would actually be . * !*~ !*.bak !RCS !CVS -- ie. the first "word" ('.' in this case) has to be a directory, under which the following include/exclude patterns apply recursively. Unfortunately, I don't think this will work under my proposed algorithm. Will have to think about it more tonight. > > - obvious conflict: 'packages' includes 'foo' and manifest > > includes '! foo/*.py' (can't imagine why you'd want this) > > Perhaps a warning should be issued, but there's no real need to > protect against operator error. The resulting dist would be broken > because it was package improperly. The warning should be enough to > get the operator to fix the file selection. I'm inclined not to worry about it because the distribution would be broken -- you'll find out soon enough. > > - subtle conflict: 'packages' includes 'foo' and manifest > > includes '! foo/bar.py' (this could well be desired: eg. exclude > > an experimental module from distribution) > > No conflict here unless the manifest says "! foo/__init__.py"; in > that case, use a warning as discussed above. Ditto -- broken distribution, you'll get a warning when you try to build it (gee, is that build-time check for __init__.py implemented? well, there's *supposed* to be one!). > > Other things we need to look for in creating a source distribution: > > - make sure there's a README > > Warnings would be sufficient if anything at all is needed; things > like README and INSTALL are largely policy rather than requirements at > this level; do you want to have to consider each possible name for > README? I'd certainly consider calling it README.txt if it were > reasonable to distribute the package to non-Unix systems. Some people > call it READ.ME, and others README.1st; how many flavors have to be > automatically detected? Let's shoot for as few special cases as > possible. Fred, I know I'm preaching to the choir with you, but anyone who doubts the value of standards in these sort of things should spend a few minutes at http://search.cpan.org/ This is a new web site created by Perl wizard Graham Barr. Turns out he knows about more than just networking and Perl internals; this is a *slick* site. Calling your readme file "README" is only the beginning. I can give a bit of leeway for README.txt (not to be friendly to Windows users, but to Emacs users ;-), but anyone who tries READ.ME or README.1ST or whatever should be ... ummm ... treated harshly. While I'm on the pulpit, Eric Raymond's "Software Release Practice HOWTO" is also recommended reading for the unconverted: http://metalab.unc.edu/mdw/HOWTO/Software-Release-Practice-HOWTO.html > The line: > > Run 'python setup.py build install' to install this package. Actually, it looks like "python setup.py install" is looking to be the preferred way of doing things: if you supply both 'build' and 'install' commands, *and* you want to build in a funny directory (ie. other than ./build), then you have to supply a --build-base option to both commands. Icky, but I can't think of a nice way around it. (Anyone with a better idea is welcome to read the source code *before* shooting from the hip.) However, if you specify *just* the install command, you only need to give it a --build-dir, and it does the right thing. In other words, ./setup.py install --build-base=/tmp/pybuild works, as does ./setup.py build --build-base=/tmp/pybuild \ install --build-base=/tmp/pybuild but, sadly, ./setup.py build install --build-base=/tmp/pybuild does not. (It builds in ./build, and then attempts to install from /tmp/pybuild.) Anyways, that's beside the point. This sort of check is a frill, but it's the sort of frill I enjoy doing because it's so damn *easy* with /^P/ languages. > > A crazy idea that conflicts with having/requiring 'version' in setup.py: > > Version should be explicitly specified somewhere. And I'm inclined to say, "It must be explicitly specified in setup.py" rather than in any particular source file. The only problem with this is that you can't check what version of module foo you have installed after the fact *unless* there's a database of installed modules. Since this is an near-essential feature anyways, I don't see that as such a big drawback. Anyways, if you want to know what version of module foo is installed *without* a module database, then either the __version__ has to be in every bloody module from a distribution, or "the system" has to know what distribution a module belongs to and what file is that distribution's "main file" (ie. where the __version__ is)... and that leads back to a module database again. Anyone have a problem with requiring the version number be specified in setup.py? That means the __version__ attribute in modules has no real meaning, since dependency checking would have to be done through the module database. Probably just as well, since the __version__ attribute doesn't seem to be consistently used. Oh yeah, I completely agree with you that all those checks -- README exists, distribution meta-data supplied, etc. -- should only result in warnings. They're not fatal errors, but they are poor manners -- and polite warnings are a good way to deal with poor manners. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From Fred L. Drake, Jr." References: <19990923094647.A9390@cnri.reston.va.us> <14314.25009.145143.624472@weyr.cnri.reston.va.us> <19990923155938.A11455@cnri.reston.va.us> Message-ID: <14314.35327.460578.961004@weyr.cnri.reston.va.us> Greg Ward writes: > I'm not sure what you mean by "complete" -- if the defaults are sensible > (get all source files mentioned in or implied by 'packages', I mean that if a manifest file is used, the defaults should not be. This should be an either/or decision for the developer. > -- ie. the first "word" ('.' in this case) has to be a directory, under > which the following include/exclude patterns apply recursively. > Unfortunately, I don't think this will work under my proposed > algorithm. Will have to think about it more tonight. Yeah, that's right: just fix the algorithm! ;-) > commands, *and* you want to build in a funny directory (ie. other than > ./build), then you have to supply a --build-base option to both > commands. Icky, but I can't think of a nice way around it. (Anyone > with a better idea is welcome to read the source code *before* shooting > from the hip.) That's totally bogus; I'll take a look at the code as soon as I have time (around March 2020, I think ;). I said: > Version should be explicitly specified somewhere. And Greg said: > And I'm inclined to say, "It must be explicitly specified in setup.py" > rather than in any particular source file. The only problem with this That's about right. I don't really care where it is, as long as we're *not* doing heuristic extraction from sources or documentation. > Anyone have a problem with requiring the version number be specified in > setup.py? That means the __version__ attribute in modules has no real > meaning, since dependency checking would have to be done through the Package version and module version are two different things; don't confuse them. The package database, once it exists, should only contain package information. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From mal@lemburg.com Fri Sep 24 10:32:13 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 24 Sep 1999 11:32:13 +0200 Subject: [Distutils] The "dist" command References: <19990923094647.A9390@cnri.reston.va.us> Message-ID: <37EB451D.499C0DDD@lemburg.com> Greg Ward wrote: > > > Possible modes of operation: > - require an explicit manifest that lists every single file (presumably > along with a way to auto-generate the manifest) > - require an explicit manifest, but allow it to have globs or > filename patterns of some kind (and also have auto-generation) > - allow an explict manifest, but automatically augment it at runtime > with the source files mentioned in 'packages', 'py_modules', and > 'ext_modules' (and any other such things that might come along) > > I'm liking the third way. Possible gotchas: > - redundant specification: 'packages' includes 'foo' and manifest > includes 'foo/*.py' > - obvious conflict: 'packages' includes 'foo' and manifest > includes '! foo/*.py' (can't imagine why you'd want this) > - subtle conflict: 'packages' includes 'foo' and manifest > includes '! foo/bar.py' (this could well be desired: eg. exclude > an experimental module from distribution) > > Syntax for the manifest file: > - each line is by default a Unix-style glob specifying files to include > - if the line starts with '!', then it's an exclude pattern > - if the line just names a directory in the distribution, that directory > is included recursively > - if the first word of a line names a directory, then the remaining > words are include/exclude patterns that are applied recursively > under that directory In my packaging tool I also have the notion of directory local manifest files, ie. the local manifest can override the global patterns to e.g. include files that would normally be excluded. Would be nice if you could integrate something along those lines too, since it makes life a little easier. Also, I have two possibilities for choosing the manifest: one for source distribution and one for binary distribution. Something like it in distutils would be nice too. > Other things we need to look for in creating a source distribution: > - make sure there's a README + maybe LICENSE to give people a chance to find this information quickly. > - make sure the distribution meta-info is supplied and non-empty > (*must* have name, version, ((author and author_email) or > (maintainer and maintainer_email)), url > > Frills: > - make sure the setup script is called "setup.py" > - make sure the README refers to "setup.py" (ie. has a line matching > /^\s*python\s+setup\.py/) As option maybe, but please don't build this into the default dist command. Note that on Unix you usually have a file INSTALL that describes the installation. > A crazy idea that conflicts with having/requiring 'version' in setup.py: > - make sure there's a version number in the "main file" (main file > is __init__.py of first package, or the first module if no packages, > or the first extension module if no pure Python modules) > - XXX how do we look for __version__ in an extension module? > - XXX do we import and look for __version__? or just scan source for > /^__version__\s*=\s*"[^"]+"/ ? > - what about 'version_from' as an alternative to 'version' -- then > we know just where to search for the version -- no guessing about > what the "main file" is Why would you do this ? I thought you need to include the version in the meta information (which should be enough) ? Note that it would also be nice to have "dist" produce a meta information Python module that it automatically includes in the archive. Which options for the archive format do you consider ? I'm currently using .zip, .tar.gz, .tgz, .tar.bz2 and .tz2. Just in case you have missed my current tool set: http://starship.skyport.net/~lemburg/pkgtools-0.1.0.zip I'm really looking forward to using distutils... as soon as I have finished moving all the stuff under the 'mx' superpackage and hacking up the PathImporter for imputil that is... -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 100 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From Fred L. Drake, Jr." References: <19990923094647.A9390@cnri.reston.va.us> <37EB451D.499C0DDD@lemburg.com> Message-ID: <14315.42418.81182.616212@weyr.cnri.reston.va.us> M.-A. Lemburg writes: > In my packaging tool I also have the notion of directory local > manifest files, ie. the local manifest can override the global > patterns to e.g. include files that would normally be excluded. Hmm. Thinking about using manifest files which are fully explicit, perhaps the syntax should be reconsidered. Perhaps each line should start with a known token, from the set "include", "file", "dir". "include" could be used to include a subdirectory's local manifest. All file/directory names should be resolved relative to the manifest file that contains the name, and should never refer to parent directories. For example: include tools/MANIFEST file README INSTALL COPYING file setup.py dir docs > Also, I have two possibilities for choosing the manifest: > one for source distribution and one for binary distribution. > Something like it in distutils would be nice too. This is cool; as long as the manifest can be specified from the command line it should be easy to adjust the defaulting of the filename: if MANIFEST.SRC is present, it is used for the source dist, otherwise use MANIFEST, or the default rules. The same for MANIFEST.BIN. (Or whatever names they have.) > Note that it would also be nice to have "dist" produce a meta > information Python module that it automatically includes in the What would this contain and how would it be used? > Which options for the archive format do you consider ? > I'm currently using .zip, .tar.gz, .tgz, .tar.bz2 and .tz2. Seems like all can be supported and selected by configuration and command line option. > Just in case you have missed my current tool set: > > http://starship.skyport.net/~lemburg/pkgtools-0.1.0.zip Hmm, no README, INSTALL, or LICENSE.... I'll see if I can figure it out! ;-) Thanks for the pointer. Greg, perhaps a pointer on the Distutils-SIG page on python.org would be useful? -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From gmcm@hypernet.com Sat Sep 25 03:07:27 1999 From: gmcm@hypernet.com (Gordon McMillan) Date: Fri, 24 Sep 1999 22:07:27 -0400 Subject: [Distutils] The "dist" command Message-ID: <1273897450-7159209@hypernet.com> Greg wrote: Possible modes of operation: - require an explicit manifest that lists every single file (presumably along with a way to auto-generate the manifest) - require an explicit manifest, but allow it to have globs or filename patterns of some kind (and also have auto-generation) - allow an explict manifest, but automatically augment it at runtime with the source files mentioned in 'packages', 'py_modules', and 'ext_modules' (and any other such things that might come along) Since you can't possibly anticipate the preferences of every sicko out there, I'd suggest a 2 phase approach. Use specifiers / filters to create an explicit manifest. Let the programmer edit the manifest, then go on to phase 2. - Gordon From mal@lemburg.com Sun Sep 26 18:16:59 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Sun, 26 Sep 1999 19:16:59 +0200 Subject: [Distutils] The "dist" command References: <19990923094647.A9390@cnri.reston.va.us> <37EB451D.499C0DDD@lemburg.com> <14315.42418.81182.616212@weyr.cnri.reston.va.us> Message-ID: <37EE550B.3C9F23B6@lemburg.com> Fred L. Drake, Jr. wrote: > > M.-A. Lemburg writes: > > In my packaging tool I also have the notion of directory local > > manifest files, ie. the local manifest can override the global > > patterns to e.g. include files that would normally be excluded. > > Hmm. Thinking about using manifest files which are fully explicit, > perhaps the syntax should be reconsidered. Perhaps each line should > start with a known token, from the set "include", "file", "dir". > "include" could be used to include a subdirectory's local manifest. > All file/directory names should be resolved relative to the manifest > file that contains the name, and should never refer to parent > directories. For example: > > include tools/MANIFEST > file README INSTALL COPYING > file setup.py > dir docs Please also add 'exclude' to the list (presuming you mean patterns here and not single files). Here are my defaults: # Source distribution SOURCE_INCLUDE = [ '\.py','\.c','\.h','\.cc','\.cpp','\.cxx','\.html','\.zip', '\.gz', 'Setup.in','Makefile.pre.in','\.sit','\.bat','\.ps','\.pdf', '\.pyd','\.def','README.*','\.pkg','\.patch','\.htm','\.i','\.swig', '','\.gif','\.jpg','\.hqx','LICENSE.*'] SOURCE_EXCLUDE = [ 'private.*','config.c','~','orig.*','\.pkgfiles'] # Binary distribution BINARY_INCLUDE = [ '\.pyo','\.so','\.lib','\.pyc','\.html','\.htm','\.gif','\.jpg', '\.dll','\.exe','\.pyd','README.*','\.sit','\.bat','\.ps','\.pdf', '\.zip','\.gz','','','\.hqx','LICENSE.*' ] BINARY_EXCLUDE = [ 'private.*','~','orig.*' ] The lists are passed to a filter function which does the following: """ The patterns in include and exclude are matched using re.search (a $ is appended to the pattern prior to processing it). First the path must match at least one pattern in include and then not match any pattern in exclude. To enhance the file search, additional qualifiers may be given. These are prepended to the pattern and must be enclosed in angular brackets (<>). The following qualifiers are implemented: only matches if the path is user executable only matches if the path is a directory synonym for After successful qualifier test, the qualifier is removed from the pattern and normal pattern matching takes place. """ > > Also, I have two possibilities for choosing the manifest: > > one for source distribution and one for binary distribution. > > Something like it in distutils would be nice too. > > This is cool; as long as the manifest can be specified from the > command line it should be easy to adjust the defaulting of the > filename: if MANIFEST.SRC is present, it is used for the source dist, > otherwise use MANIFEST, or the default rules. The same for > MANIFEST.BIN. (Or whatever names they have.) That should work... using an option to specify the MANIFEST extension I mean. It would even allow other standard classifications as well, e.g. for specific platforms, customers, editions, etc. > > Note that it would also be nice to have "dist" produce a meta > > information Python module that it automatically includes in the > > What would this contain and how would it be used? I guess all the information used as setup information for distutils. This would be useful for a rpm like tool that allows you to query the installed packages using this meta information. > > Which options for the archive format do you consider ? > > I'm currently using .zip, .tar.gz, .tgz, .tar.bz2 and .tz2. > > Seems like all can be supported and selected by configuration and > command line option. > > > Just in case you have missed my current tool set: > > > > http://starship.skyport.net/~lemburg/pkgtools-0.1.0.zip > > Hmm, no README, INSTALL, or LICENSE.... I'll see if I can figure it > out! ;-) Thanks for the pointer. Things should be pretty obvious. Just run the tools with -h and they'll tell you a whole lot about themselves :-) Can't remember all those options myself... Note that I've been using the pkgcreate script for quite a while now, so it fits the task for my purposes rather well, YMMV. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 98 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From Fred L. Drake, Jr." References: <19990923094647.A9390@cnri.reston.va.us> <37EB451D.499C0DDD@lemburg.com> <14315.42418.81182.616212@weyr.cnri.reston.va.us> <37EE550B.3C9F23B6@lemburg.com> Message-ID: <14319.33215.649221.874474@weyr.cnri.reston.va.us> M.-A. Lemburg writes: > Please also add 'exclude' to the list (presuming you mean patterns > here and not single files). That sounds good to me. Regarding a distutils-generated metadata module, I asked: > What would this contain and how would it be used? And Marc-Andre replied: > I guess all the information used as setup information for > distutils. This would be useful for a rpm like tool that allows > you to query the installed packages using this meta information. Sounds like the package database. I'd rather see something like this: import distutils.info pkgname = distutils.info.get_module_package(__name__) pkginfo = distutils.info.get_package_info(pkgname) print "This is %s, version %s." % (pkgname, pkginfo.version) (or perhaps all queries return a package info object, and the name should be retrieved from that). -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From gward@cnri.reston.va.us Mon Sep 27 20:50:39 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Mon, 27 Sep 1999 15:50:39 -0400 Subject: [Distutils] The "dist" command In-Reply-To: <1273897450-7159209@hypernet.com>; from Gordon McMillan on Fri, Sep 24, 1999 at 10:07:27PM -0400 References: <1273897450-7159209@hypernet.com> Message-ID: <19990927155038.A14802@cnri.reston.va.us> On 24 September 1999, Gordon McMillan said: > Greg wrote: > > Possible modes of operation: > - require an explicit manifest that lists every single file (presumably > along with a way to auto-generate the manifest) > - require an explicit manifest, but allow it to have globs or filename > patterns of some kind (and also have auto-generation) > - allow an explict manifest, but automatically augment it at runtime > with the source files mentioned in 'packages', 'py_modules', and > 'ext_modules' (and any other such things that might come along) > > Since you can't possibly anticipate the preferences of every sicko out > there, I'd suggest a 2 phase approach. Use specifiers / filters to create > an explicit manifest. Let the programmer edit the manifest, then go on > to phase 2. Gordon, Fred, Marc-André: thanks for your ideas. They're all, ummm, interesting. However, I'm trying to balance simplicity and utility here, and I think my original manifest syntax does an adequate job of that. At least it works fine for generating a source distribution for Distutils 0.1. The MANIFEST file is simplicity itself: # # List of files to ship in the Distutils source distribution (apart from # the standard stuff -- Python source files, README and setup script, # test scripts). # # GPW 1999/09/23 # # $Id$ # examples * !*~ !*.py[co] Gee, I haven't even checked it in yet. Oh well, I've got *tons* of changes from the weekend still to checkin. And then I finish writing the USAGE file, which should probably be added to the MANIFEST. (Hmmm: perhaps the "list of default files" should include /^[A-Z_]+(\.txt)?$/, to handle README and USAGE and LICENCE and TODO and INSTALL and all those conventional all-uppercase filenames...) So yes, I'm ignoring Fred's idea and using the "default" files (all source files implied by 'py_modules', 'ext_modules', and 'packages', plus README and setup.py) even when there's a MANIFEST. And I'm ignoring Marc-André's idea: there's only one MANIFEST. I don't need anything fancy for *my* little module distribution, so it's hard for me to predict what more complex distributions might throw in. Oh yeah, I'm ignoring Gordon's suggestion too: if you want to list every file explictly, you're welcome to do so! If you want automated help in doing so, may I suggest the 'find(1)' command? >smirk< Oh, one comment on the idea of separate MANIFESTs for source and binary distributions: no. The binary distribution should be built from the 'build' directory; the whole point of that directory is to make installation and the generation of binary (err, built) distributions trivial. The information about what goes in the built distribution should come from an extra 'build' phase, eg. 'build_doc'. Wow, I never thought there'd come a day when I would refuse complexity and featuritis. This Python thing must be getting to me. Better go spend some time hacking Perl to remind myself of the joys of unbridled complexity. ;-) Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From Fred L. Drake, Jr." References: <1273897450-7159209@hypernet.com> <19990927155038.A14802@cnri.reston.va.us> Message-ID: <14319.59155.838976.411998@weyr.cnri.reston.va.us> Greg Ward writes: > explictly, you're welcome to do so! If you want automated help in doing > so, may I suggest the 'find(1)' command? >smirk< I don't think there's any equivalent for Windows without buying 3rd party addons (not an acceptable alternative). It would be easy to implement using os.path.walk(); I'm sure an interested Windows-bound reader of this group could contribute such a beast. > Oh, one comment on the idea of separate MANIFESTs for source and binary > distributions: no. The binary distribution should be built from the > 'build' directory; the whole point of that directory is to make > installation and the generation of binary (err, built) distributions It's too early to say, probably: those of us dragging our feet need more experience with what you already have before we can say that it's absolutely required or you can say that it absolutely isn't. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From gward@cnri.reston.va.us Mon Sep 27 22:58:34 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Mon, 27 Sep 1999 17:58:34 -0400 Subject: [Distutils] The "dist" command In-Reply-To: <14319.59155.838976.411998@weyr.cnri.reston.va.us>; from Fred L. Drake, Jr. on Mon, Sep 27, 1999 at 05:52:19PM -0400 References: <1273897450-7159209@hypernet.com> <19990927155038.A14802@cnri.reston.va.us> <14319.59155.838976.411998@weyr.cnri.reston.va.us> Message-ID: <19990927175833.H14105@cnri.reston.va.us> On 27 September 1999, Fred L. Drake, Jr. said: > > Greg Ward writes: > > explictly, you're welcome to do so! If you want automated help in doing > > so, may I suggest the 'find(1)' command? >smirk< > > I don't think there's any equivalent for Windows without buying 3rd > party addons (not an acceptable alternative). It would be easy to > implement using os.path.walk(); I'm sure an interested Windows-bound > reader of this group could contribute such a beast. Hey, I was smirking when I said that. The whole point of having some wildcard support in the Distutils MANIFEST format is that people don't need to use 'find', or a separate "make_manifest" script to generate it: they just type a few lines and it works. You and Marc-André have come up with some clever ideas that will allow people to type even fewer lines in some circumstances, at the expense of more complexity in the code. > It's too early to say, probably: those of us dragging our feet need > more experience with what you already have before we can say that it's > absolutely required or you can say that it absolutely isn't. Exactly -- I claim that the current MANIFEST syntax is excellent for generating a source distribution of the Distutils. It would also work quite well for the other software I've released in the past, whether it's a C library, a MATLAB toolbox, or Perl module distributions. It's certainly better than the MANIFEST format used by Perl's MakeMaker (all files explicitly listed, with the ExtUtils::MakeManifest module to help you generate it the first time -- good luck maintaining it). I hope it fits the style of other developers, but we'll have to wait and see on that. Hyper-defensively yours, Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From mikael@pobox.com Mon Sep 27 23:27:56 1999 From: mikael@pobox.com (Mikael Lyngvig) Date: Tue, 28 Sep 1999 00:27:56 +0200 Subject: [Distutils] The "dist" command In-Reply-To: <19990927175833.H14105@cnri.reston.va.us> Message-ID: <199909272224.SAA17543@python.org> Hi Greg, [Sorry for jumping in rather abruptly; I've been busy moving to and from the UK recently...] > Hey, I was smirking when I said that. The whole point of having some > wildcard support in the Distutils MANIFEST format is that people don't > need to use 'find', or a separate "make_manifest" script to generate it: > they just type a few lines and it works. You and Marc-André have come > up with some clever ideas that will allow people to type even fewer > lines in some circumstances, at the expense of more complexity in the > code. My two pennies (with a bit of luck I'll comment on something you recognize): Manifest files should be as explicit as possible. I.e., wildcards should only be allowed in a preprocessor tool for generating the initial manifest file. The reason for this is simply that you don't want to accidentally ship your boss' latest hate mail to you, because you accidentally saved it in the source directory :) (so, I prefer the 'make_manifest' approach) As for 'find', I am still, extremely slowly, working on Python replacements for the most common Unix/GNU utilities. It would be kind of nifty to have a 'find' module written so generically, using keyword parameters, that you could use if from your own Python scripts: User = os.environ["USER"] Edits = posixcmd.find("/src", perm="-u+w", name="*.c", type="f", user=User) if Edits == []: print "You can go home now, %s. See you tomorrow." % User else: print "You need to check in the following files %s:" % User for Edit in Edits: print "\t%s" % Edit -- Mikael From gward@cnri.reston.va.us Mon Sep 27 23:34:25 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Mon, 27 Sep 1999 18:34:25 -0400 Subject: [Distutils] The "dist" command In-Reply-To: <199909272224.SAA13053@cnri.reston.va.us>; from Mikael Lyngvig on Tue, Sep 28, 1999 at 12:27:56AM +0200 References: <19990927175833.H14105@cnri.reston.va.us> <199909272224.SAA13053@cnri.reston.va.us> Message-ID: <19990927183424.I14105@cnri.reston.va.us> On 28 September 1999, Mikael Lyngvig said: > Manifest files should be as explicit as possible. I.e., wildcards should > only be allowed in a preprocessor tool for generating the initial manifest > file. The reason for this is simply that you don't want to accidentally > ship your boss' latest hate mail to you, because you accidentally saved it > in the source directory :) (so, I prefer the 'make_manifest' approach) Hmmm, good point. Looks like manifest files are shaping up to be one of those personal/religious issues that will just have to allow different ways of using them. You'll never find me using a two-phase manifest as you and Gordon advocate, but I have no problem using the machinery in distutils.command.dist to write a "make_manifest" script for those who like this modus operandi. (Err, I should say you'll never find me working this way until *I* accidentally ship the boss' latest hate mail... ;-) > As for 'find', I am still, extremely slowly, working on Python replacements > for the most common Unix/GNU utilities. It would be kind of nifty to have a > 'find' module written so generically, using keyword parameters, that you > could use if from your own Python scripts: > > User = os.environ["USER"] > Edits = posixcmd.find("/src", perm="-u+w", name="*.c", type="f", > user=User) *BZZT* *BZZT* Off-topic alert! But I can't resist... 1) What's wrong with os.path.walk()? (Apart from the fact that it's buried in os.path where it would never occur to anyone in a million years to look for it.) 2) How would you handle find's ability to combine options booleanologically? Won't work with this syntax -- that's why you need something like os.path.walk, to which you supply a test function. 3) If you used CVS, then you would just need to say "cvs status" to find out which files aren't checked in. (Or better yet, my "cvstatus" script, which gives a much more readable version of this. ;-) Off topicologically yours, Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From mikael@pobox.com Tue Sep 28 02:24:06 1999 From: mikael@pobox.com (Mikael Lyngvig) Date: Tue, 28 Sep 1999 03:24:06 +0200 Subject: [Distutils] The "dist" command In-Reply-To: <19990927183424.I14105@cnri.reston.va.us> Message-ID: <199909280121.VAA19466@python.org> Hi Greg, > *BZZT* *BZZT* Off-topic alert! But I can't resist... I'll try to sneak this one through and pray I don't get a severe beating. > 1) What's wrong with os.path.walk()? (Apart from the fact that it's > buried in os.path where it would never occur to anyone in a million > years to look for it.) Nothing as such, I use it all the time. The proposal was mostly related to my "dream" that one day most of the POSIX commands will be implemented in Python, so that a full POSIX command set is available as soon as Python has been ported. This might actually serve as a great marketing gimmick for Python itself. >2) How would you handle find's ability to combine options booleanologically? > Won't work with this syntax -- > that's why you need something like os.path.walk, to which you supply a > test function. Good point. You'd have to use a list of options instead, then. Pity, I liked the looks of the keyword parameter solution :) >3) If you used CVS, then you would just need to say "cvs > status" to find out which files aren't checked in. (Or better yet, my > "cvstatus" script, which gives a much more readable version of this. ;-) Professionally, I've recently (and briefly, phew) been using SCCS... :) (and created a Python script, vcscheck (supposedly SCCS and RCS), which did something similar to "cvs status") Privately, I've been using RCS so far, but just decided a few weeks ago that I'd switch to CVS. Is your cvstatus script available anywhere? That's one of the very first issues I'd have addresses myself for CVS. -- Mikael From pas@scansoft.com Tue Sep 28 06:32:03 1999 From: pas@scansoft.com (Perry Stoll) Date: Mon, 27 Sep 1999 22:32:03 -0700 Subject: [Distutils] The "dist" command Message-ID: <008c01bf0972$ddec5d00$c410cb97@pstoll.bellatlantic.net> -----Original Message----- From: Fred L. Drake, Jr. >Greg Ward writes: > > explictly, you're welcome to do so! If you want automated help in doing > > so, may I suggest the 'find(1)' command? >smirk< > > I don't think there's any equivalent for Windows without buying 3rd >party addons (not an acceptable alternative Fred, Ignoring the fact that Greg was being facetious (I hope...). There is a free replacement for Win32 called, oddly enough, find. From the cygnus win32 GNU utilties: http://sourceware.cygnus.com/cygwin/ Makes a unix-weenie feel right at home on win32. -Perry From s341625@student.uq.edu.au Tue Sep 28 04:27:28 1999 From: s341625@student.uq.edu.au (Anthony Pfrunder) Date: Tue, 28 Sep 1999 13:27:28 +1000 (GMT+1000) Subject: [Distutils] The "dist" command In-Reply-To: <008c01bf0972$ddec5d00$c410cb97@pstoll.bellatlantic.net> Message-ID: On Mon, 27 Sep 1999, Perry Stoll wrote: [snip] > Fred, > > Ignoring the fact that Greg was being facetious (I hope...). There is a free > replacement for Win32 called, oddly enough, find. From the cygnus win32 GNU > utilties: > http://sourceware.cygnus.com/cygwin/ > > Makes a unix-weenie feel right at home on win32. Extreme off-topic... Cygwin was benchtested by an Australian computer magazine and it verified that GCC (DJGPP) generates smaller files quicker and in less space than Visual C. [Now I know why python compiles so quickly compared to VC project file;)] Seriously, Cygwin is basicaly unix for NT Cheers, Anthony Pfrunder Computer Systems Engineering University of Queensland From da@ski.org Tue Sep 28 05:01:49 1999 From: da@ski.org (David Ascher) Date: Mon, 27 Sep 1999 21:01:49 -0700 (Pacific Daylight Time) Subject: [Distutils] The "dist" command In-Reply-To: <199909272224.SAA17543@python.org> Message-ID: On Tue, 28 Sep 1999, Mikael Lyngvig wrote: > As for 'find', I am still, extremely slowly, working on Python replacements > for the most common Unix/GNU utilities. Interesting project. I don't know if you know that TomC has a similar project for Perl: http://language.perl.com/ppt/ I like the notion of a module w/ functions as opposed to just getopt parsing. --david From mal@lemburg.com Mon Sep 27 23:18:06 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 28 Sep 1999 00:18:06 +0200 Subject: [Distutils] The "dist" command References: <1273897450-7159209@hypernet.com> <19990927155038.A14802@cnri.reston.va.us> Message-ID: <37EFED1E.2564E685@lemburg.com> Greg Ward wrote: > > Gordon, Fred, Marc-André: > > thanks for your ideas. They're all, ummm, interesting. However, I'm > trying to balance simplicity and utility here, and I think my original > manifest syntax does an adequate job of that. At least it works fine > for generating a source distribution for Distutils 0.1. The MANIFEST > file is simplicity itself: > > # > # List of files to ship in the Distutils source distribution (apart from > # the standard stuff -- Python source files, README and setup script, > # test scripts). > # > # GPW 1999/09/23 > # > # $Id$ > # > > examples * !*~ !*.py[co] > > Gee, I haven't even checked it in yet. Oh well, I've got *tons* of > changes from the weekend still to checkin. And then I finish writing > the USAGE file, which should probably be added to the MANIFEST. (Hmmm: > perhaps the "list of default files" should include /^[A-Z_]+(\.txt)?$/, > to handle README and USAGE and LICENCE and TODO and INSTALL and all > those conventional all-uppercase filenames...) > > So yes, I'm ignoring Fred's idea and using the "default" files (all > source files implied by 'py_modules', 'ext_modules', and 'packages', > plus README and setup.py) even when there's a MANIFEST. And I'm > ignoring Marc-André's idea: there's only one MANIFEST. I don't need > anything fancy for *my* little module distribution, so it's hard for me > to predict what more complex distributions might throw in. Oh yeah, I'm > ignoring Gordon's suggestion too: if you want to list every file > explictly, you're welcome to do so! If you want automated help in doing > so, may I suggest the 'find(1)' command? >smirk< > > Oh, one comment on the idea of separate MANIFESTs for source and binary > distributions: no. The binary distribution should be built from the > 'build' directory; the whole point of that directory is to make > installation and the generation of binary (err, built) distributions > trivial. The information about what goes in the built distribution > should come from an extra 'build' phase, eg. 'build_doc'. How does that work ? I usually let my pkgcreate run over the same package tree twice: once for source distribution and then again for a binary one -- pretty easy I'd say and without *any* overhead on my part since the default file inclusion/exclusion sets are chosen by sinlge option on the command line. > Wow, I never thought there'd come a day when I would refuse complexity > and featuritis. This Python thing must be getting to me. Better go > spend some time hacking Perl to remind myself of the joys of unbridled > complexity. ;-) Greg, what we proposed here is not featuritis, it's plain experience. All of the mentioned things are needed one way or another. Anyway, I guess that a simple tool to create the MANIFEST file will probably already do the trick for all of us. Perhaps you could add an option to choose the filename for it... e.g. -M . Or have the dist tool look for a MANIFEST.py file and let it return the list of files to process via a get_files() API. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 98 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mikael@pobox.com Tue Sep 28 09:41:24 1999 From: mikael@pobox.com (Mikael Lyngvig) Date: Tue, 28 Sep 1999 10:41:24 +0200 Subject: [Distutils] The "dist" command In-Reply-To: Message-ID: <199909280838.EAA29826@python.org> > Interesting project. And shouldn't be that hard given Python already has built-in regular expressions, etc. >I don't know if you know that TomC has a similar project for Perl: http://language.perl.com/ppt/ I almost see it as a question of personal integrity to NOT know what the Perl dudes are doing :) Shame on me! Perl is probably the greatest competitor to Python, so one should now - and try to give Python something better. > I like the notion of a module w/ functions as opposed to just getopt parsing. Yes, but Greg pointed out a severe short-coming (and I found one more: keyword args don't preserve the order of the options), so it would have to be a list of options, or even a simple command-line string, so there's not much difference. -- Mikael From gward@cnri.reston.va.us Tue Sep 28 12:55:42 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 28 Sep 1999 07:55:42 -0400 Subject: [Distutils] The "dist" command In-Reply-To: <37EFED1E.2564E685@lemburg.com>; from M.-A. Lemburg on Tue, Sep 28, 1999 at 12:18:06AM +0200 References: <1273897450-7159209@hypernet.com> <19990927155038.A14802@cnri.reston.va.us> <37EFED1E.2564E685@lemburg.com> Message-ID: <19990928075542.A15284@cnri.reston.va.us> On 28 September 1999, M.-A. Lemburg said: > Greg, what we proposed here is not featuritis, it's plain > experience. All of the mentioned things are needed one way > or another. > > Anyway, I guess that a simple tool to create the > MANIFEST file will probably already do the trick for all of > us. Perhaps you could add an option to choose the filename for it... > e.g. -M . Or have the dist tool look for > a MANIFEST.py file and let it return the list of files to > process via a get_files() API. Already there; from distutils/distutils/command/dist.py: options = [('formats=', 'f', "formats for source distribution (tar, ztar, gztar, or zip)"), ('manifest=', 'm', "name of manifest file"), ] thus: python setup.py -v dist --manifest MyManifestFile or python setup.py -v dist -m MyManifestFile will do what you want. (Well, it *should*, I haven't tested it...) Please note, I have never said all of these ideas that have circulated are baseless or silly or unneeded. I just want some real developers to get real experience using the Distutils on real module distributions before we make a final decision about which all-singing, all-dancing solution is needed for manifest files. There's more than one way to do it, and I don't want to implement (or support or maintain) all of them. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From Fred L. Drake, Jr." References: <008c01bf0972$ddec5d00$c410cb97@pstoll.bellatlantic.net> Message-ID: <14320.49271.599024.584925@weyr.cnri.reston.va.us> Perry Stoll writes: > Ignoring the fact that Greg was being facetious (I hope...). There is a free > replacement for Win32 called, oddly enough, find. From the cygnus win32 GNU > utilties: > http://sourceware.cygnus.com/cygwin/ Perry, Actually, I was already aware of that, and the MKS Toolkit as well (which includes a non-free version of find). However, these are both third-party addons, even if one is free. Since the use of find in this case is quite simple, I think implementing a "manifest" command in distutils would be worth the 15-20 lines of code it would take. I'd hate for us to recommend users to go install YASP ("yet another software package") just to be able to get a list of files and directories! -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From Fred L. Drake, Jr." References: <199909280838.EAA29826@python.org> Message-ID: <14320.52827.126401.592172@weyr.cnri.reston.va.us> Mikael Lyngvig writes: > Yes, but Greg pointed out a severe short-coming (and I found one more: > keyword args don't preserve the order of the options), so it would have to > be a list of options, or even a simple command-line string, so there's not > much difference. Mikael, Still, a lot of the common cases can be handled by convenience functions that either do the right thing simply or build the more complicated paramters to the general implementation. find, in particular, is subject to the ordering problem, but most commands are not. Sounds like a cool thing to have handy! -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From pas@scansoft.com Tue Sep 28 16:18:59 1999 From: pas@scansoft.com (Perry A. Stoll) Date: Tue, 28 Sep 1999 11:18:59 -0400 Subject: [Distutils] The "dist" command Message-ID: <01bf09c4$c9fd6490$3e3e79cf@nara.scansoft.com> Fred, I'm all for a Python-only approach. As for avoiding YASP, oye, tell me about it. We have sooo many for our build environment, it's just a mess. I just wanted to make sure people didn't take away the message "there is no free find replacment for Win32". -Perry -----Original Message----- From: Fred L. Drake, Jr. To: Perry Stoll Cc: distutils-sig@python.org Date: Tuesday, September 28, 1999 9:14 AM Subject: Re: [Distutils] The "dist" command > >Perry Stoll writes: > > Ignoring the fact that Greg was being facetious (I hope...). There is a free > > replacement for Win32 called, oddly enough, find. From the cygnus win32 GNU > > utilties: > > http://sourceware.cygnus.com/cygwin/ > >Perry, > Actually, I was already aware of that, and the MKS Toolkit as well >(which includes a non-free version of find). However, these are both >third-party addons, even if one is free. Since the use of find in >this case is quite simple, I think implementing a "manifest" command >in distutils would be worth the 15-20 lines of code it would take. > I'd hate for us to recommend users to go install YASP ("yet another >software package") just to be able to get a list of files and >directories! > From gward@cnri.reston.va.us Wed Sep 29 14:27:26 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Wed, 29 Sep 1999 09:27:26 -0400 Subject: [Distutils] ANNOUNCE: Distutils 0.1 Message-ID: <19990929092725.A24474@cnri.reston.va.us> Python Module Distribution Utilities release 0.1 September 29, 1999 The Python Module Distribution Utilities, or Distutils for short, are a collection of modules that aid in the development, distribution, and installation of Python modules. (It is intended that ultimately the Distutils will grow up into a system for distributing and installing whole Python applications, but for now their scope is limited to module distributions.) The Distutils will be a standard part of Python 1.6. This release is intended to provide a wider audience an early opportunity to try out the system, improve its portability, shake out the bugs, make suggestions, and so forth. Do not expect perfection: if you're trying out this release, it should be because you want to help debug and develop, not because you need a stable, working system immediately. You can download the Distutils from the Distutils SIG home page: http://www.python.org/sigs/distutils-sig/ Any feedback, comments, bug reports, success reports, etc. should be reported to the Distutils SIG at distutils-sig@python.org If you're not yet a member of this mailing list, you can join at http://www.python.org/mailman/listinfo/distutils-sig Reports of minor bugs (patches are especially welcome!) can be sent directly to me (Greg Ward) at gward@python.net. Anything that expects a longer response than "Thanks for the bug report/fix!" should be directed to the SIG. From amos@aracnet.com Wed Sep 29 19:57:03 1999 From: amos@aracnet.com (Amos Latteier) Date: Wed, 29 Sep 1999 11:57:03 -0700 Subject: [Distutils] distutils 0.1 problem and fix Message-ID: <3.0.5.32.19990929115703.009df100@mail.aracnet.com> Three cheers for distutils 0.1! Unfortunately it fails to install itself on my NT box. D:\TEMP\Distutils-0.1>python setup.py -v install running install running build running build_py not copying distutils\ccompiler.py (output up-to-date) not copying distutils\core.py (output up-to-date) not copying distutils\errors.py (output up-to-date) not copying distutils\fancy_getopt.py (output up-to-date) not copying distutils\msvccompiler.py (output up-to-date) not copying distutils\spawn.py (output up-to-date) not copying distutils\sysconfig.py (output up-to-date) not copying distutils\text_file.py (output up-to-date) not copying distutils\unixccompiler.py (output up-to-date) not copying distutils\util.py (output up-to-date) not copying distutils\version.py (output up-to-date) not copying distutils\__init__.py (output up-to-date) not copying distutils\command\build.py (output up-to-date) not copying distutils\command\build_ext.py (output up-to-date) not copying distutils\command\build_py.py (output up-to-date) not copying distutils\command\dist.py (output up-to-date) not copying distutils\command\install.py (output up-to-date) not copying distutils\command\install_ext.py (output up-to-date) not copying distutils\command\install_py.py (output up-to-date) not copying distutils\command\__init__.py (output up-to-date) running install_py creating D:\Python1.5.2 Traceback (innermost last): File "setup.py", line 22, in ? packages = ['distutils', 'distutils.command'], File "distutils\core.py", line 87, in setup dist.run_commands () File "distutils\core.py", line 377, in run_commands self.run_command (cmd) File "distutils\core.py", line 426, in run_command cmd_obj.run () File "distutils\command\install.py", line 287, in run self.run_peer ('install_py') File "distutils\core.py", line 710, in run_peer self.distribution.run_command (command) File "distutils\core.py", line 426, in run_command cmd_obj.run () File "distutils\command\install_py.py", line 51, in run outfiles = self.copy_tree (self.build_dir, self.install_dir) File "distutils\core.py", line 773, in copy_tree self.distribution.dry_run) File "distutils\util.py", line 318, in copy_tree mkpath (dst, verbose=verbose) File "distutils\util.py", line 74, in mkpath raise DistutilsFileError, "%s: %s" % (head, errstr) distutils.errors.DistutilsFileError: D:\Python1.5.2: File exists To fix this I changed line 317 of distutils/util.py to: if not dry_run and not os.path.exists(os.path.normpath(dst)): This checks to see if the destination directory already exists before creating it. Note: I had to use normpath since for me dst was 'D:\\Python1.5.2\\' which os.path.exists claims doesn't exist, while 'D:\\Python1.5.2' does exist. -Amos