From gward@cnri.reston.va.us Thu Apr 1 17:17:58 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Thu, 1 Apr 1999 12:17:58 -0500 Subject: [Distutils] Compiler abstractiom model In-Reply-To: <37029E32.6D137D76@lyra.org>; from Greg Stein on Wed, Mar 31, 1999 at 02:14:10PM -0800 References: <001f01be7b39$cc993ea0$f4160218@c1004579-c.plstn1.sfba.home.com> <19990331135016.A8894@cnri.reston.va.us> <37029E32.6D137D76@lyra.org> Message-ID: <19990401121757.A9181@cnri.reston.va.us> Quoth Greg Stein, on 31 March 1999: > You can also save yourself a lot of time by building something that > seems reasonable and waiting for the feedback from people who need it > and/or feel it is important. Umm, that *is* how I'm attacking the problem! I just want to make sure that the design is open enough that people can attack that last 5% (or 10% or 20% or whatever) without running up against the brick wall of a short-sighted design. That's the biggest pitfall of OO: sure it leads to extensible and reusable designs, but only if you carefully build extensibility and reusability into those designs. That's what I'm trying to do with Distutils, and the point of this whole discussion is to get the feedback I need to see if the design is indeed extensible and reusable enough! I think we're in agreement here, except that I'm not even shooting for a 95% job. IMHO getting 90% or 80% of the way there would be enough to build on. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gward@cnri.reston.va.us Sun Apr 4 04:35:47 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Sat, 3 Apr 1999 22:35:47 -0500 Subject: [Distutils] Fixed "-n" bug Message-ID: <19990403223547.A1644@cnri.reston.va.us> Hi all -- I've just fixed the bug that Michel Sanner inadvertently pointed out -- the "-n" ("--dry-run") flag was universally ignored. This was solved by adding several methods to the Command base class through which all "external actions" (anything that affects the outside world, most particularly the filesystem) should be channelled. Now the verbose and dry-run flags can be respected in a consistently painless way by *all* command classes. In addition to "cvs update", you can now download a snapshot tarball: http://www.python.org/sigs/distutils-sig/distutils-19990403.tar.gz I make no promises about the longevity or reliability of that URL -- it'll probably go away the next time I make a snapshot. Let me know if you like the snapshot, but would prefer a zip file to a tarball; under duress, I could hack the 'mksnap' script to make both. ;-) Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From dalke@bioreason.com Wed Apr 7 08:13:38 1999 From: dalke@bioreason.com (Andrew Dalke) Date: Wed, 07 Apr 1999 00:13:38 -0700 Subject: [Distutils] (final?) automake related questions Message-ID: <370B05A2.509602A8@bioreason.com> As I've been going on about, I've got the changes made so that GNU automake supports Python. Because it is the FSF, I had to sign paperwork to disclaim copyright interest in our contributions before they would accept it. I mailed them in today. Thus, I've been touching up my changes before submitting them. I wanted to make sure I was using a sensible (or at least reasonable) set of names before I did that. Here are the names for which I am uncertain: o PYTHON_SITE This is the value sys.prefix + "/python" + sys.version[:3] + "/site-packages" It is the location where platform independent modules should be installed. It can be overridden on the configure line with the parameter --with-python-site=DIR o PYTHON_SITE_PACKAGE automake defines the variable PACKAGE, which I am taking also as the module name (nothing unusual about that, I hope). o PYTHON_SITE_EXEC This is the value sys.exec_prefix + "/python" + sys.version[:3] + "/site-packages" It can be overridded on the configure line with the parameter --with-python-site-exec=DIR. I played around with a few names. These seem to make sense to me (eg, PYTHON_SITE_EXEC is better than PYTHON_SITE_SHARED which I had). I cannot find if the standard Python libraries refer to these directories in some other fashion. The closest I can find is "site.py" which combines the first and last directory names directly into "sitedirs". *** question 1: Do these names make sense to others? o I added a new m4 macro for "aclocal" called "AM_CHECK_PYTHON". This is the command which goes in "configure.in" to tell autoconf that it needs to generate Python information. The macro takes one optional parameter. If the value is "classes" the Python files in the package are installed directly in PYTHON_SITE. If the value is "module" then they are installed into PYTHON_SITE_PACKAGE. The installation directory name is stored in the variable PYTHON_SITE_INSTALL. Eg: If you have the file "Vector.py" in the autoconf package "vecmath" with the PYTHON_SITE set to /usr/local/lib/python1.5/site-packages AM_CHECK_PYTHON(module) puts it in /usr/local/lib/python1.5/site-packages/vecmath/Vector.py AM_CHECK_PYTHON(classes) puts it in /usr/local/lib/python1.5/site-packages/Vector.py *** question 2: Should it be "classes" or something else? Since there's no existing python command-line program to compile a *single* python program (as compared to compileall which compiles, well, all :), I'm using the following in the Makefile (the value of PYTHON is the full location of the python executable and was found during the configure step): PYTHON_COMPILE_CMD = "-c 'import py_compile, sys; \ py_compile.compile(sys.argv[1])' PYTHON_OCOMPILE = $(PYTHON) $(PYTHON_COMPILE_CMD) PYTHON_OCOMPILE = $(PYTHON) -c $(PYTHON_COMPILE_CMD) .py.pyc: @echo "Compiling python file $< " @$(PYTHON_COMPILE) $< .py.pyo: @echo "Compiling python file $< (optimized)" @$(PYTHON_OCOMPILE) $< The idea is that I hope distutils will have a command-line way to do this, so I can replace all that with: .py.pyc: @$(PYTHON) $pythondir/distutils/compile.pr $< .py.pyo: @$(PYTHON) -O $pythondir/distutils/compile.py $< (I hide the actual compile with @ because otherwise you just see the full "import .... argv[1])" which is hard to read. By having a program to byte compile a file directly I can get rid of the noise.) *** question 3: Can distutils provide this sort of service? *** question 4: Anyone interested in getting the latest CVS distribution of automake (after I get the changes in) to play^U test things out? To help out, I'm also working on documentation so people can use it without reading the full automake documentation (which has a strong emphasis on building C programs). Not suprisingly, this email will be a hefty part of said documentation :) Andrew dalke@bioreason.com P.S. Here's the list of other definitions created by configure.in. I cannot see how there's any real problem with these names, but who knows .... PYTHON = full name of the python executable PYTHON_PREFIX = sys.prefix PYTHON_EXEC_PREFIX = sys.exec_prefix PYTHON_VERSION = sys.version PYTHON_PLATFORM = sys.platform pythondir = sys.prefix + "/python" + sys.version[:3] From dalke@bioreason.com Wed Apr 7 08:40:27 1999 From: dalke@bioreason.com (Andrew Dalke) Date: Wed, 07 Apr 1999 00:40:27 -0700 Subject: [Distutils] (final?) automake related questions References: <370B05A2.509602A8@bioreason.com> Message-ID: <370B0BEB.222E8CAE@bioreason.com> Minor nit: > PYTHON_OCOMPILE = $(PYTHON) $(PYTHON_COMPILE_CMD) > PYTHON_OCOMPILE = $(PYTHON) -c $(PYTHON_COMPILE_CMD) should be > PYTHON_COMPILE = $(PYTHON) $(PYTHON_COMPILE_CMD) > PYTHON_OCOMPILE = $(PYTHON) -c $(PYTHON_COMPILE_CMD) The problem came from trying to reduce the number of deletions because the original text was escaped for use in perl double quotes; did a copy&paste w/o deleting everything. Andrew From gstein@lyra.org Wed Apr 7 08:49:12 1999 From: gstein@lyra.org (Greg Stein) Date: Wed, 07 Apr 1999 00:49:12 -0700 Subject: [Distutils] (final?) automake related questions References: <370B05A2.509602A8@bioreason.com> <370B0BEB.222E8CAE@bioreason.com> Message-ID: <370B0DF8.43B655BB@lyra.org> Andrew Dalke wrote: > > Minor nit: > > > PYTHON_OCOMPILE = $(PYTHON) $(PYTHON_COMPILE_CMD) > > PYTHON_OCOMPILE = $(PYTHON) -c $(PYTHON_COMPILE_CMD) > > should be > > > PYTHON_COMPILE = $(PYTHON) $(PYTHON_COMPILE_CMD) > > PYTHON_OCOMPILE = $(PYTHON) -c $(PYTHON_COMPILE_CMD) Shouldn't that be: PYTHON_OCOMPILE = $(PYTHON) -O $(PYTHON_COMPILE_CMD) ?? -g -- Greg Stein, http://www.lyra.org/ From dalke@bioreason.com Wed Apr 7 18:10:10 1999 From: dalke@bioreason.com (Andrew Dalke) Date: Wed, 07 Apr 1999 10:10:10 -0700 Subject: [Distutils] (final?) automake related questions References: <370B05A2.509602A8@bioreason.com> <370B0BEB.222E8CAE@bioreason.com> <370B0DF8.43B655BB@lyra.org> Message-ID: <370B9172.120E1C@bioreason.com> > Shouldn't that be: > > PYTHON_OCOMPILE = $(PYTHON) -O $(PYTHON_COMPILE_CMD) An imprompt debugging hikau: Nits, nits, everywhere Infestation of my docs Time for some caffiene. (It really is the documentation. The code is: &define_variable('PYTHON_COMPILE', "\$(PYTHON) \$(PYTHON_COMPILE_CMD)"); &define_variable('PYTHON_OCOMPILE', "\$(PYTHON) -O \$(PYTHON_COMPILE_CMD)") ) Andrew dalke@bioreason.com From gward@cnri.reston.va.us Wed Apr 7 19:34:14 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Wed, 7 Apr 1999 14:34:14 -0400 Subject: [Distutils] (final?) automake related questions In-Reply-To: <370B05A2.509602A8@bioreason.com>; from Andrew Dalke on Wed, Apr 07, 1999 at 12:13:38AM -0700 References: <370B05A2.509602A8@bioreason.com> Message-ID: <19990407143414.A3584@cnri.reston.va.us> Quoth Andrew Dalke, on 07 April 1999: > Here are the names for which I am uncertain: > > o PYTHON_SITE > > This is the value > sys.prefix + "/python" + sys.version[:3] + "/site-packages" [...] > o PYTHON_SITE_EXEC > > This is the value > sys.exec_prefix + "/python" + sys.version[:3] + "/site-packages" In the Distutils 'install' command (probably the part that's most analogous to what you're doing), I called these install_site_lib install_site_platlib I don't claim Automake and Distutils should use identical terms, but we should settle on one or the other of "exec" and "plat". I like exec because most (all?) of the Python stuff under exec_prefix isn't executable, but it is platform-specific (shared objects for the most part). Also, I like having "lib" in the name just to be explicit about what's getting installed there. (You might have a different installation directory for scripts.) So I think I would rename your PYTHON_SITE and PYTHON_SITE_EXEC to PYTHON_SITE_LIB and PYTHON_SITE_PLATLIB, give or take a few underscores. And I think you would rename my install_site_lib and install_site_platlib to install_site and install_site_exec. I notice that both schemes have no notion of /usr/local/lib/site-python -- this is a feature, right? (Ie. has that directory been replaced by /usr/local/lib/python1.5/site-packages? or does it still live on?) > o PYTHON_SITE_PACKAGE > > automake defines the variable PACKAGE, which I am taking also > as the module name (nothing unusual about that, I hope). Umm, are you talking about "package" in the Python sense of a special directory full of modules, or in the general sense of a discrete chunk of software to distribute? (The latter is called a "distribution" in the Distutils lexicon, because of the conflict. This seems to be common usage in the Perl world as well, since there's also an existing meaning for "package" over there.) Anyways, whichever it is, it's not necessarily the same as the "module name". Both packages (Python meaning) and packages (general meaning) may have many modules! In Distutils, the Python package may be implied by the layout of files in the source distribution, or explicitly specified in setup.py. The name of the distribution must be specified in setup.py. > *** question 3: Can distutils provide this sort of service? I assume you mean the one-line compilation business. I would argue against Distutils providing a one-line compiler for use in Makefiles: if you have Distutils, you don't *need* a Makefile! (At least not for your Python components.) > Here's the list of other definitions created by configure.in. > I cannot see how there's any real problem with these names, but > who knows .... [...] > pythondir = sys.prefix + "/python" + sys.version[:3] Again, my inclination would be to make explicit the fact that this is the library directory, eg. "pythonlibdir". Greg P.S. Regarding your snippet of Perl: > (It really is the documentation. The code is: > > &define_variable('PYTHON_COMPILE', > "\$(PYTHON) \$(PYTHON_COMPILE_CMD)"); > &define_variable('PYTHON_OCOMPILE', > "\$(PYTHON) -O \$(PYTHON_COMPILE_CMD)") > ) You can singly-quote those strings, and thus not have to escape the dollar signs. I tend to reserve double-quotes for when I need them, ie. when I actually want to interpolate Perl variables into a string. -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-262-5376 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From dalke@bioreason.com Wed Apr 7 22:35:41 1999 From: dalke@bioreason.com (Andrew Dalke) Date: Wed, 07 Apr 1999 14:35:41 -0700 Subject: [Distutils] (final?) automake related questions References: <370B05A2.509602A8@bioreason.com> <19990407143414.A3584@cnri.reston.va.us> Message-ID: <370BCFAD.4CCBA7BE@bioreason.com> Thanks for the comments, Greg. > I like exec because most (all?) of the Python stuff under > exec_prefix isn't executable, but it is platform-specific (shared > objects for the most part). Did you mean you "like" or "don't like"? I personally do not like it for that reason. Still, that's the way Python does it which is why I (eventually) chose that name. > install_site_lib > install_site_platlib I would shy away from using 'lib'. Automake expects 'lib_' as a special prefix (for files which are installed into the standard library directory). Also, 'LIB' is the prefix for the suffixes (like 'LIBS', 'LIBTOOL', 'LIBADD', ...) related to the construction of static and shared libraries. I worry that those two uses (variables for Python libraries and "real" libraries) should have names which are more distinct. This is important for us because we're going to use the same system for all of our code (C, C++, Python) so I want to make things pretty clear. > And I think you would rename my install_site_lib and > install_site_platlib to install_site and install_site_exec. Yep :) Though if it makes sense, I think people can live with the differences. However, do you distinguish between files which are placed under Python's lib/ directory and files which are placed, say, in /usr/local/lib/ ? I could make up an example, if you want, though since I cannot think of anything but contrived ones, I don't think it's an urgent need. Still, it may be useful that such an option is available. > I notice that both schemes have no notion of > /usr/local/lib/site-python -- this is a feature, right? Yes, based on Fred Drake's comment in this list that this directory is to be considered deprecated. Though for the automake mods it can be overridden with ./configure --with-python-site=/usr/local/lib/site-python This is important for us because we install all of our internally developed packages in /br/stable/lib/python1.5/ while externally developed ones are in the usual /usr/local/lib/python1.5/site-packages. > Umm, are you talking about "package" in the Python sense of a > special directory full of modules, or in the general sense of a > discrete chunk of software to distribute? Indeed, I am confused by its definition in the Python world. However, in this case I use "PACKAGE" because that's the word specifically used by automake, and automake uses it in the general sense of the world. PYTHON_SITE_PACKAGE is the combination of (my) "PYTHON_SITE" directory and (automake's) "PACKAGE" name. Also, someone can override PYTHON_SITE_PACKAGE in the configure.in file to be anywhere they want it to be. My code just sets the default and uses that variable wherever it might be useful. > Anyways, whichever it is, it's not necessarily the same as the > "module name". Both packages (Python meaning) and packages > (general meaning) may have many modules! Yes, just like in C you can have a single source distribution which includes several programs, each of which installs to different places. Automake supports this; you can override almost everything as long as you are cleaverer in how you use it. I think what I've added is able to support having a distribution with several Python module distributions. Though I haven't tested it, because we're operating under the "one distribution - one python package" development philosophy. > The name of the distribution must be specified in setup.py. The automake way lets you define/override information in the configure.in file(s) > I assume you mean the one-line compilation business. I would > argue against Distutils providing a one-line compiler for use > in Makefiles: if you have Distutils, you don't *need* a Makefile! > (At least not for your Python components.) It was either that or petition for an option to 'compileall' to take a list of files to compile rather than a directory name. We (Bioreason) still need a Makefile since it drives things not part of the Makefile, like tagging everything in CVS with the current build tag, updating the build number, removing .pyo and .pyc files with "make clean", driving the building of the documentation from LaTex (or pod), running the regression tests, etc. > > pythondir = sys.prefix + "/python" + sys.version[:3] > > Again, my inclination would be to make explicit the fact that this > is the library directory, eg. "pythonlibdir". This variable name also is an automake-ism. The existance of a "pythondir" tells automake that targets like "python-clean" exist. It also says that "python_PYTHON" contains the list of Python files. > You can singly-quote those [Perl] strings Yeah. There again I was sticking with the existing automake code, which uses double quotes for the equivalent section related to emacs lisp compilation. Perhaps it was a hobgoblin consistancy. Andrew dalke@bioreason.com From dalke@bioreason.com Thu Apr 8 00:29:49 1999 From: dalke@bioreason.com (Andrew Dalke) Date: Wed, 07 Apr 1999 16:29:49 -0700 Subject: [Distutils] (final?) automake related questions References: <370B05A2.509602A8@bioreason.com> <19990407143414.A3584@cnri.reston.va.us> Message-ID: <370BEA6D.E4805475@bioreason.com> Greg said: > Umm, are you talking about "package" in the Python sense of a special > directory full of modules, or in the general sense of a discrete chunk > of software to distribute? To which I responded: > Indeed, I am confused by its definition in the Python world. Quick clarification question, then. What do you call a single file which is placed in python1.5/site-packages/something.py and referenced via 'import something' and what do you call a set of files contained in a directory named 'something_else', placed in python1.5/site-packages/something_else/* and contains the __init__.py to tell Python that this importable via import something_else Is something == 'module' and something_else == 'package' ? If so, I want to change my name for the first (currently is 'classes', for lack of a better word) and the second from 'module' to 'package'. Andrew dalke@bioreason.com From flight@mathi.uni-heidelberg.de Sun Apr 11 15:18:28 1999 From: flight@mathi.uni-heidelberg.de (Gregor Hoffleit) Date: Sun, 11 Apr 1999 16:18:28 +0200 Subject: [Distutils] Some code to play with In-Reply-To: <19990322101021.A489@cnri.reston.va.us>; from Greg Ward on Mon, Mar 22, 1999 at 10:10:22AM -0500 References: <19990322101021.A489@cnri.reston.va.us> Message-ID: <19990411161828.A7807@freefly.hoffleit.org> On Mon, Mar 22, 1999 at 10:10:22AM -0500, Greg Ward wrote: > Go crazy. No documentation yet -- please read the code! Start with > distutils/core.py which, as its name implies, is the start of > everything. I went nuts with docstrings in that module last night, so > hopefully there's just enough there that you can figure things out in > the absence of the "Distutils Implementation Notes" document that I'd > like to write. Just a quick question: Does the package support something like autoconf's DESTDIR ? I.e. installing the built package in a directory different from the actual target ? I'm not sure if that's in the scope of `build --dir' or `install --prefix'. DESTDIR is necessary for building Debian packages: Debian's packaging tools work on files in `pwd`/debian/tmp. Generally, something like "./configure --prefix /usr; make; make DESTDIR=`pwd`/debian/tmp install" is used. Note that this is different from setting prefix to `pwd`/debian/tmp! Gregor From gward@cnri.reston.va.us Wed Apr 14 14:48:21 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Wed, 14 Apr 1999 09:48:21 -0400 Subject: [Distutils] Some code to play with In-Reply-To: <19990411161828.A7807@freefly.hoffleit.org>; from Gregor Hoffleit on Sun, Apr 11, 1999 at 04:18:28PM +0200 References: <19990322101021.A489@cnri.reston.va.us> <19990411161828.A7807@freefly.hoffleit.org> Message-ID: <19990414094820.A1816@cnri.reston.va.us> Quoth Gregor Hoffleit, on 11 April 1999: > Just a quick question: Does the package support something like > autoconf's DESTDIR ? I.e. installing the built package in a directory > different from the actual target ? I'm not sure if that's in the scope > of `build --dir' or `install --prefix'. The installation directories are up to the 'install' command. The most convenient way to set them is by setting the prefix, but you can (or will be able to) set every target directory individually. RTFS for the options that control this -- I haven't documented anything yet because I'm not sure the option names are final. > DESTDIR is necessary for building Debian packages: Debian's packaging > tools work on files in `pwd`/debian/tmp. Generally, something like > "./configure --prefix /usr; make; make DESTDIR=`pwd`/debian/tmp > install" is used. Note that this is different from setting prefix to > `pwd`/debian/tmp! In general, that's true -- but as the Distutils currently stand, the prefix is used *solely* to generate the installation directories. So it doesn't (yet) matter which one you set. I'm not sure if that will remain true forever, though. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gward@cnri.reston.va.us Wed Apr 14 16:24:58 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Wed, 14 Apr 1999 11:24:58 -0400 Subject: [Distutils] (final?) automake related questions In-Reply-To: <370BCFAD.4CCBA7BE@bioreason.com>; from Andrew Dalke on Wed, Apr 07, 1999 at 02:35:41PM -0700 References: <370B05A2.509602A8@bioreason.com> <19990407143414.A3584@cnri.reston.va.us> <370BCFAD.4CCBA7BE@bioreason.com> Message-ID: <19990414112457.B1816@cnri.reston.va.us> [I mis-speak myself] > I like exec because most (all?) of the Python stuff under > exec_prefix isn't executable, but it is platform-specific (shared > objects for the most part). [Andrew catches me] > Did you mean you "like" or "don't like"? I personally do not > like it for that reason. Still, that's the way Python does it > which is why I (eventually) chose that name. Ooops! I *don't like* using "execlib" for "platform-specific library directory". I now see that that usage descends from Autoconf's "exec prefix" notion, which is pretty well established by now. However, in switching from the Autoconf point-of-view ("let's build this Python thing, which is basically a C program with some ancillary files, some of which are platform-specific and some not") to the Distutils POV ("let's build this bundle of modules to add to the Python library; some of the files to install are pure Python modules, and some are platform-specific extension modules"), a change of terminology is in order. The convention in Python-land seems to be "plat" for "platform" (as opposed to "arch" for "architecture" -- six of one, half a dozen of the other!), so I went with "install_platlib" and "install_site_platlib" as the directories to which platform-specific files will be installed. (Most distributions will install to install_site*; only distributions that are part of the standard library will use the "install_*" directories. The idea here is that some distribution -- say, the Distutils itself -- will be part of the standard library, but might be released more frequently than Python itself.) > I would shy away from using 'lib'. Automake expects 'lib_' as > a special prefix (for files which are installed into the standard > library directory). Also, 'LIB' is the prefix for the suffixes > (like 'LIBS', 'LIBTOOL', 'LIBADD', ...) related to the > construction of static and shared libraries. I worry that > those two uses (variables for Python libraries and "real" > libraries) should have names which are more distinct. > > This is important for us because we're going to use the same > system for all of our code (C, C++, Python) so I want to make > things pretty clear. That makes sense for you in the Automake context. If Automake/Autoconf think that "lib" is a platform-specific C library (libfoo.a, libfoo.so), then it makes sense for you to stick to that narrow definition. However, we all know perfectly well that foo.py is a Python library (or part of one), and Python is just as good as C -- so why come up with some other word than library? So I think your scheme makes sense for Automake, but I would still consider changing "exec" to "plat" at some point if it makes sense and Automake doesn't whine. And I think my scheme makes sense for a Python-specific build/distribute/install mechanism, so I'm gonna stick with what I've got. > However, do you distinguish between files which are placed > under Python's lib/ directory and files which are placed, say, > in /usr/local/lib/ ? I could make up an example, if you > want, though since I cannot think of anything but contrived > ones, I don't think it's an urgent need. Still, it may be > useful that such an option is available. That's quite possible for a Python application that has configuration files and such. Note that the Distutils "prefix" is still (eg.) /usr/local -- it's just Python's prefix. So there's no problem in a particular application (or even module distribution, although it's probably bad practice for a programming library to have config files) using that to put its stuff in /usr/local/lib. If there's a demand for it, Distutils could probably have direct support for installing "ancillary files", but I don't see that being an issue until we truly support Python *applications* (as opposed to libraries). Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From Fred L. Drake, Jr." References: <370B05A2.509602A8@bioreason.com> <19990407143414.A3584@cnri.reston.va.us> <370BCFAD.4CCBA7BE@bioreason.com> <19990414112457.B1816@cnri.reston.va.us> Message-ID: <14100.46301.702423.532804@weyr.cnri.reston.va.us> Greg Ward writes: > particular application (or even module distribution, although it's > probably bad practice for a programming library to have config files) > using that to put its stuff in /usr/local/lib. If there's a demand for > it, Distutils could probably have direct support for installing > "ancillary files", but I don't see that being an issue until we truly > support Python *applications* (as opposed to libraries). There are too many examples of libraries needing supplemental data to ignore this. A common example would be a library that provides character set re-encoding; the code could be fairly simple, and specific encodings could be loaded on demand from tables associated with a library. Similar considerations may be applicable to many other functions; the tables may contain font information for a layout engine, etc. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From gward@cnri.reston.va.us Wed Apr 14 16:37:31 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Wed, 14 Apr 1999 11:37:31 -0400 Subject: [Distutils] (final?) automake related questions In-Reply-To: <370BEA6D.E4805475@bioreason.com>; from Andrew Dalke on Wed, Apr 07, 1999 at 04:29:49PM -0700 References: <370B05A2.509602A8@bioreason.com> <19990407143414.A3584@cnri.reston.va.us> <370BEA6D.E4805475@bioreason.com> Message-ID: <19990414113730.C1816@cnri.reston.va.us> Quoth Andrew Dalke, on 07 April 1999: > Quick clarification question, then. Quick answer: while still_confused: read_page ('http://www.python.org/doc/essays/packages.html') Longer answer: > What do you call a single file which is placed in > python1.5/site-packages/something.py > > and referenced via 'import something' I call that a module. But that's not the only kind of module! > and what do you call a set of files contained in a directory > named 'something_else', placed in > > python1.5/site-packages/something_else/* > > and contains the __init__.py to tell Python that this importable via > > import something_else I call that a package. But remember, a package is also a module! > Is something == 'module' and something_else == 'package' ? So yes, this is correct. > If so, I want to change my name for the first (currently is 'classes', > for lack of a better word) and the second from 'module' to 'package'. Calling a module "classes" is ugly and doesn't tell the whole story (a module can also provide functions and global variables!). Calling a package a module is *very* confusing! I definitely think you should make this change. Here's my take on packages and modules in Python: A Python source file ending in ".py" and living in a directory on sys.path is a "module". But this isn't the only kind of module. A directory living on sys.path containing a file called "__init__.py" is a special kind of module called a "package". A platform-specific dynamically loaded file on sys.path might also be a module, usually called an "extension module". In general, modules can provide any kind of Python object, where something "provided by" a module may be directly pulled into another namespace using "from module import name", or accessed via the module as "module.name". Usually, packages only provide other modules, but they can provide classes, functions, variables, etc. Usually, "pure Python" modules provide classes, functions, and variables, but they can provide other modules (eg. 'os' provides a module called 'path'). And usually, extension modules only provide functions and variables -- my understanding is that they cannot provided classes, and I have no idea what the other hard-and-fast limitations on extension modules are (I really have to play around more there!). Assuming all that is correct, I hope that clears things up! (And if anything up there is *not* correct, somebody tell me soon before I hard-code my misunderstandings in the Distutils...) Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From amos@aracnet.com Fri Apr 16 01:48:32 1999 From: amos@aracnet.com (Amos Latteier) Date: Thu, 15 Apr 1999 17:48:32 -0700 Subject: [Distutils] PATCH: sysconfig and NT Message-ID: <3.0.5.32.19990415174832.0121a950@mail.aracnet.com> Here's a patch that seems to get sysconfig minimally working under NT for me. Has anyone else done better? I noticed that distutils installs itself under 'lib\site-packages' which does not seem to be a convention under NT, at least the directory does not exist by default, and is not in the Python path by default. It seems that Windows packages normally go in the top level Python directory--at least that's where the win32 packages install themselves, and it *is* in the Python path by default... Anyway here's the simple-minded patch. -Amos RCS file: /projects/cvsroot/distutils/distutils/sysconfig.py,v retrieving revision 1.5 diff -r1.5 sysconfig.py 130a131,139 > def _init_nt(): > """Initialize the module as appropriate for NT""" > g=globals() > # load config.h, though I don't know how useful this is > parse_config_h(open( > os.path.join(sys.exec_prefix, "include", "config.h")), g) > # set basic install directories > g['LIBDEST']=os.path.join(sys.exec_prefix, "Lib") > g['BINLIBDEST']=os.path.join(sys.exec_prefix, "Lib") 132c141 < --- > 141a151 > del _init_nt From dalke@bioreason.com Fri Apr 16 02:18:22 1999 From: dalke@bioreason.com (Andrew Dalke) Date: Thu, 15 Apr 1999 19:18:22 -0600 Subject: [Distutils] PATCH: sysconfig and NT References: <3.0.5.32.19990415174832.0121a950@mail.aracnet.com> Message-ID: <37168FDE.E7CEFF8C@bioreason.com> Amos Latteier said: > I noticed that distutils installs itself under 'lib\site-packages' > which does not seem to be a convention under NT, at least the > directory does not exist by default, and is not in the Python path > by default. It directory is added by site.py, which is imported when you start python (unless you use "python -S"). Quoting from it: | This will append site-specific paths to to the module search path. On | Unix, it starts with sys.prefix and sys.exec_prefix (if different) and | appends lib/python/site-packages as well as lib/site-python. | On other platforms (mainly Mac and Windows), it uses just sys.prefix | (and sys.exec_prefix, if different, but this is unlikely). The | resulting directories, if they exist, are appended to sys.path, and | also inspected for path configuration files. Regarding the fact that the directory doesn't exist, I just checked the unix install and it doesn't seem to create that directory either. Rather, I just did: make -n install | grep site and didn't find anything. Also, I see my "site-packages" directory has the wrong unix group and creation time, so it must have been created manually. Anyway, lib's "site-packages" is the official place for installing Python packages. from your patch: > g['LIBDEST']=os.path.join(sys.exec_prefix, "Lib") > g['BINLIBDEST']=os.path.join(sys.exec_prefix, "Lib") Is that really "Lib" or just "lib"? Granted, it is NT which means I know nearly diddly about it. Andrew dalke@bioreason.com From dalke@bioreason.com Fri Apr 16 10:50:27 1999 From: dalke@bioreason.com (Andrew Dalke) Date: Fri, 16 Apr 1999 03:50:27 -0600 Subject: [Distutils] (final?) automake related questions References: <370B05A2.509602A8@bioreason.com> <19990407143414.A3584@cnri.reston.va.us> <370BCFAD.4CCBA7BE@bioreason.com> <19990414112457.B1816@cnri.reston.va.us> Message-ID: <371707E3.5053CDE0@bioreason.com> Greg said: > So I think your scheme makes sense for Automake, but I would > still consider changing "exec" to "plat" at some point if it makes > sense and Automake doesn't whine. And I think my scheme makes sense > for a Python-specific build/distribute/install mechanism, so I'm > gonna stick with what I've got. The automake people suggested an even better alternative; don't provide those options at all from the configure script (which gets rid of the --with-python-site-exec). And, the "plat" extension means nothing (that I know of) in automake, so I can change that. That nomenclature does make more sense. > > Is something == 'module' and something_else == 'package' ? > > So yes, this is correct. Also now on my change list. (I've got to wait, I assume, until the automake people get told by the FSF that I've sent in the right copyright disclaimer papers; though it's been over a week since I mailed them.) > Assuming all that is correct, I hope that clears things up! (And > if anything up there is *not* correct, somebody tell me soon before > I hard-code my misunderstandings in the Distutils...) Sounds good to me. And besides, even if it is wrong, by documenting and implement it one way, sometimes things become right even if originally wrong! Andrew dalke@bioreason.com From amos@aracnet.com Fri Apr 16 22:19:22 1999 From: amos@aracnet.com (Amos Latteier) Date: Fri, 16 Apr 1999 14:19:22 -0700 Subject: [Distutils] PATCH: sysconfig and NT In-Reply-To: <37168FDE.E7CEFF8C@bioreason.com> References: <3.0.5.32.19990415174832.0121a950@mail.aracnet.com> Message-ID: <3.0.5.32.19990416141922.01198da0@mail.aracnet.com> At 07:18 PM 4/15/99 -0600, Andrew Dalke wrote: >Amos Latteier said: >> I noticed that distutils installs itself under 'lib\site-packages' >> which does not seem to be a convention under NT, at least the >> directory does not exist by default, and is not in the Python path >> by default. > >It directory is added by site.py, which is imported when you >start python (unless you use >"python -S"). Quoting from it: >| This will append site-specific paths to to the module search path. On >| Unix, it starts with sys.prefix and sys.exec_prefix (if different) and >| appends lib/python/site-packages as well as lib/site-python. >| On other platforms (mainly Mac and Windows), it uses just sys.prefix >| (and sys.exec_prefix, if different, but this is unlikely). The >| resulting directories, if they exist, are appended to sys.path, and >| also inspected for path configuration files. Exactly. On Windows, the site directory is just sys.exec_prefix, rather than sys.exex_prefix/lib/python/site-packages. So now we need additional fixes to distutils to handle this platform difference. >Anyway, lib's "site-packages" is the official place for installing >Python packages. On Unix ;-) >from your patch: >> g['LIBDEST']=os.path.join(sys.exec_prefix, "Lib") >> g['BINLIBDEST']=os.path.join(sys.exec_prefix, "Lib") > >Is that really "Lib" or just "lib"? Granted, it is NT which means >I know nearly diddly about it. I guess it should be 'lib'. I don't know. I just used 'Lib' cause that's what the some other Python path directories (set in the registry) call it... OK, here's two patches, one to handle the site-package platform differences, another repeat of my last one, only this time with 'lib', not 'Lib' RCS file: /projects/cvsroot/distutils/distutils/sysconfig.py,v retrieving revision 1.5 diff -r1.5 sysconfig.py 131,132c131,140 < < --- > def _init_nt(): > """Initialize the module as appropriate for NT""" > g=globals() > # load config.h, though I don't know how useful this is > parse_config_h(open( > os.path.join(sys.exec_prefix, "include", "config.h")), g) > # set basic install directories > g['LIBDEST']=os.path.join(sys.exec_prefix, "lib") > g['BINLIBDEST']=os.path.join(sys.exec_prefix, "lib") > 141a150 > del _init_nt RCS file: /projects/cvsroot/distutils/distutils/command/install.py,v retrieving revision 1.1 diff -r1.1 install.py 110d109 < 112,113c111,115 < self.install_site_lib = \ < os.path.join (self.install_lib, 'site-packages') --- > if os.sep == '/': > self.install_site_lib = \ > os.path.join (self.install_lib, 'site-packages') > else: > self.install_site_lib = self.exec_prefix 118,120c120,124 < self.install_site_platlib = \ < os.path.join (self.install_lib, 'site-packages') < --- > if os.sep == '/': > self.install_site_platlib = \ > os.path.join (self.install_lib, 'site-packages') > else: > self.install_site_platlib = self.exec_prefix With these patches distutils installs itself correctly on NT for me. -Amos From timd@macquarie.com.au Mon Apr 19 05:49:27 1999 From: timd@macquarie.com.au (Timothy Docker) Date: Mon, 19 Apr 1999 14:49:27 +1000 (EST) Subject: [Distutils] Package versioning Message-ID: <14106.45806.186668.178220@qad16> Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd@macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1 From gward@cnri.reston.va.us Wed Apr 21 14:32:03 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Wed, 21 Apr 1999 09:32:03 -0400 Subject: [Distutils] Package versioning In-Reply-To: <14106.45806.186668.178220@qad16>; from Timothy Docker on Mon, Apr 19, 1999 at 02:49:27PM +1000 References: <14106.45806.186668.178220@qad16> Message-ID: <19990421093202.A6191@cnri.reston.va.us> On 19 April 1999, Timothy Docker said: > I like the look of the proposed api, but have one question. Will this > support an installed system that has multiple versions of the same > package installed simultaneously? If not, then this would seem to be a > significant limitation, especially when dependencies between packages > are considered. That's not something I've thought about, and it didn't come up in the "heated discussion" (not *quite* a flamefest) about version numbers that kicked off this sig back in December. (Although just about everything else relating to software versioning did come up: see http://www.python.org/pipermail/distutils-sig/1998-December/000016.html and ensuing posts for the whole discussion. *Please* don't bring it up again on the mailing list; I'm going to try to put off the continuation of that discussion until I have actual code for dependency checking.) > Assuming it does, then how will this be achieved? I am presently > managing this with a messy arrangement of symlinks. A package is > installed with its version number in it's name, and a separate > directory is created for an application with links from the > unversioned package name to the versioned one. Then I just set the > pythonpath to this directory. That's about the only thing I can think of. But that scheme isn't entirely compatible with the library organization I had in mind for the Distutils, ie. installed packages and modules go right under site-packages, rather than having a separate directory for each distribution. (The trouble with this is that you either have to add to your PYTHONPATH for every installed distribution, or have lots of .pth files scattered around. The former is annoying and error-prone, and the latter is a nasty kludge that is neatly avoided by using the site-packages directory.) So your scheme could be made to work for distributions that put all of their modules under a common base package -- Distutils itself, for example, or the XML library. Distributions that use multiple base packages could probably be made to work with some difficulty. But it couldn't handle distributions that don't use the Python package system at all, because modules from those distributions will be dumped straight into site-packages. (Moral: use packages! [unless supporting Python 1.4 is essential for you]) > I'm sure there is a better solution that this, and I'm not sure that > this would work under windows anyway (does windows have symlinks?). If there is a better solution, it hasn't occured to me in the five minutes I've been writing this post. Nor do I know if it would work on !Unix. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From da@ski.org Wed Apr 21 22:32:23 1999 From: da@ski.org (David Ascher) Date: Wed, 21 Apr 1999 14:32:23 -0700 (Pacific Daylight Time) Subject: [Distutils] Package versioning In-Reply-To: <19990421093202.A6191@cnri.reston.va.us> Message-ID: On Wed, 21 Apr 1999, Greg Ward wrote: > On 19 April 1999, Timothy Docker said: > > Assuming it does, then how will this be achieved? I am presently > > managing this with a messy arrangement of symlinks. > > That's about the only thing I can think of. Symlinks aren't portable. I use a package which imports a subpackage based on code performed in the toplevel __init__.py file. So, for example, my Snow directory (in my sys.path): Snow/ __init__.py mknewversion.py Snow_1999_3_14 Snow_1999_3_16 ... The code in __init__.py is a bit ugly but looks something like: import sys, os, imp, string _dir = __path__[0] _listdir = os.listdir(_dir) _thisdir = os.path.basename(_dir) _instdirs = filter(lambda x: x[:len(_thisdir)+1] == _thisdir+'_', _listd def datesort(f1, f2): parts1 = string.split(f1, '_') parts1[1:] = map(int, parts1[1:]) parts2 = string.split(f2, '_') parts2[1:] = map(int, parts2[1:]) return cmp(parts1, parts2) _instdirs.sort(datesort) _latest = _instdirs[-1] _file, _pathname, _description = imp.find_module(_latest, __path__) _module = imp.load_module(_latest, _file, _pathname, _description) globals().update(_module.__dict__) This is similar to Pmw's scheme. mknewversion.py is a similar-looking script which makes copies of the current version, comes up with the directory name based on the date and existing versions, and does some platform-specific things (add symbolic links for the programmer's sake, not the user's). It's not ideal, but it's portable and customizable. --david