From jjl at pobox.com Tue Nov 1 00:25:59 2005 From: jjl at pobox.com (John J Lee) Date: Mon, 31 Oct 2005 23:25:59 +0000 (UTC) Subject: [Distutils] setuptools: Missing unmanaged module warning; Failed SVN cleanup; Wrong version in .egg name In-Reply-To: <5.1.1.6.0.20051031155551.01fa6578@mail.telecommunity.com> References: <5.1.1.6.0.20051031155551.01fa6578@mail.telecommunity.com> Message-ID: On Mon, 31 Oct 2005, Phillip J. Eby wrote: > At 08:37 PM 10/31/2005 +0000, John J Lee wrote: > >svn: Unknown hostname 'svn.eby-sarna.com' > > This is a DNS problem that I just fixed (I think), please try again. (Can't try from here, will try in the morning.) [...] > >3. It said: > > > >Installed c:\python24\lib\site-packages\clientform-0.2.1a-py2.4.egg > > > >But I asked for version "dev", and the Cheese shop entry I uploaded for > >testing purposes (just to get the SVN URL into the long-description) was > >different again: "0.2.1a-pre1"! Why don't I get an egg with version > >"dev"? > > Because when building from source, the version number is controlled by the > source setup.py. Ah, of course. I eagerly anticipate the day when .require() installs at the start of sys.path. Do I understand correctly that, when this happens, we will in practice be able to do pkg_resources.require("smth==1.1") import smth and be confident that we do indeed have module smth version 1.1? Sorry if this is a stupid question... stupid-is-as-stupid-does ly y'rs, John From pje at telecommunity.com Tue Nov 1 02:05:40 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 31 Oct 2005 20:05:40 -0500 Subject: [Distutils] setuptools: Missing unmanaged module warning; Failed SVN cleanup; Wrong version in .egg name In-Reply-To: References: <5.1.1.6.0.20051031155551.01fa6578@mail.telecommunity.com> <5.1.1.6.0.20051031155551.01fa6578@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051031200257.02139880@mail.telecommunity.com> At 11:25 PM 10/31/2005 +0000, John J Lee wrote: >I eagerly anticipate the day when .require() installs at the start of >sys.path. It's not going to do that, but as of 0.6a6 eggs are placed before the directory that *contains* them on sys.path. So, if you require an egg that's in site-packages, it will be added just before site-packages in sys.path, thus preventing unmanaged packages from interfering. > Do I understand correctly that, when this happens, we will in >practice be able to do > >pkg_resources.require("smth==1.1") >import smth > > >and be confident that we do indeed have module smth version 1.1? That's the case now; it's just that you can always get a DistributionNotFound (if it's not available) or a VersionConflict (if another version has already been activated). From jjl at pobox.com Tue Nov 1 02:50:37 2005 From: jjl at pobox.com (John J Lee) Date: Tue, 1 Nov 2005 01:50:37 +0000 (UTC) Subject: [Distutils] setuptools: Missing unmanaged module warning; Failed SVN cleanup; Wrong version in .egg name In-Reply-To: <5.1.1.6.0.20051031200257.02139880@mail.telecommunity.com> References: <5.1.1.6.0.20051031155551.01fa6578@mail.telecommunity.com> <5.1.1.6.0.20051031155551.01fa6578@mail.telecommunity.com> <5.1.1.6.0.20051031200257.02139880@mail.telecommunity.com> Message-ID: On Mon, 31 Oct 2005, Phillip J. Eby wrote: > At 11:25 PM 10/31/2005 +0000, John J Lee wrote: > > >I eagerly anticipate the day when .require() installs at the start of > >sys.path. > > It's not going to do that, but as of 0.6a6 eggs are placed before the > directory that *contains* them on sys.path. So, if you require an egg > that's in site-packages, it will be added just before site-packages in > sys.path, thus preventing unmanaged packages from interfering. That does not prevent unmanaged packages from interfering. For example, I happen to have an SVN copy of module ClientForm installed fairly early on in sys.path on a Windows machine at work (from PYTHONPATH or a .pth or what else I don't recall offhand), and that, unsurprisingly, is what I get when I "import ClientForm" from an interactive prompt. setuptools inserts its sys.path entry *after* the place where my ad-hoc sys.path entry happens to have ended up, so even immediately after a pkg_resources.require("ClientForm==0.2.1a"), I still get my old SVN working copy picked up when I "import ClientForm". I don't always get a conflict warning even here on this linux box (setuptools 0.6a7): dev[0]$ python Python 2.3.4 (#1, Feb 19 2005, 23:24:45) [GCC 3.3.2 20031218 (Gentoo Linux 3.3.2-r5, propolice-3.3-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import mechanize >>> mechanize.__file__ '/home/john/lib/python/mechanize/__init__.pyc' >>> dev[0]$ easy_install -d . -m mechanize Searching for mechanize Reading http://www.python.org/pypi/mechanize/ Reading http://wwwsearch.sourceforge.net/mechanize/ Best match: mechanize 0.0.9a Downloading http://wwwsearch.sourceforge.net/mechanize/src/mechanize-0.0.9a.tar.gz Processing mechanize-0.0.9a.tar.gz Running mechanize-0.0.9a/setup.py -q bdist_egg --dist-dir /tmp/easy_install-T4qRVN/mechanize-0.0.9a/egg-dist-tmp-nGc2hg zip_safe flag not set; analyzing archive contents... Installed /hda/usr/local/buf/comp/dev/mechanize-0.0.9a-py2.3.egg Because this distribution was installed --multi-version or --install-dir, before you can import modules from this package in an application, you will need to 'import pkg_resources' and then use a 'require()' call similar to one of these examples, in order to select the desired version: pkg_resources.require("mechanize") # latest installed version pkg_resources.require("mechanize==0.0.9a") # this exact version pkg_resources.require("mechanize>=0.0.9a") # this version or higher Note also that the installation directory must be on sys.path at runtime for this to work. (e.g. by being the application's script directory, by being on PYTHONPATH, or by being added to sys.path by your code.) Processing dependencies for mechanize dev[0]$ No conflict warning! OTOH, I DO get a conflict warning for: dev[0]$ easy_install -d . -m ClientForm But again, NO conflict warning for: dev[0]$ easy_install -d . -m 'mechanize==dev' > > Do I understand correctly that, when this happens, we will in > >practice be able to do > > > >pkg_resources.require("smth==1.1") > >import smth > > > > > >and be confident that we do indeed have module smth version 1.1? > > That's the case now; it's just that you can always get a > DistributionNotFound (if it's not available) or a VersionConflict (if > another version has already been activated). ... or just, silently, some entirely different version of the module that I installed before I'd even heard of eggs. What am I doing wrong? I'm also scratching my head over why easy_install does not oes not install dependencies for Cheese Shop-registered module mechanize, but that's probably just a job for tomorrow and some coffee :-) easy_install 'mechanize==dev' --> grabs and installs mechanize from SVN, but not its dependencies, despite my (committed!) setup.py requesting otherwise... INSTALL_REQUIRES = [ "ClientForm>=0.2.1", "ClientCookie>=1.0.3", "pullparser>=0.0.4"] NAME = "mechanize" [...SNIP...] setuptools.setup( name = NAME, version = VERSION, license = LICENSE, platforms = PLATFORMS, classifiers = [c for c in CLASSIFIERS.split("\n") if c], install_requires = INSTALL_REQUIRES, zip_safe = ZIP_SAFE, test_suite = "test", author = "John J. Lee", author_email = "jjl at pobox.com", description = doclines[0], long_description = "\n".join(doclines[2:]), url = "http://wwwsearch.sourceforge.net/%s/" % NAME, download_url = ("http://wwwsearch.sourceforge.net/%s/src/" "%s-%s.tar.gz" % (NAME, NAME, VERSION)), py_modules = py_modules, packages = packages, ) BTW, this work of yours looks really promising, which is why I'm doing this in the first place, of course. John From jjl at pobox.com Tue Nov 1 12:19:56 2005 From: jjl at pobox.com (John J Lee) Date: Tue, 1 Nov 2005 11:19:56 +0000 (GMT Standard Time) Subject: [Distutils] setuptools: Missing unmanaged module warning; Failed SVN cleanup; Wrong version in .egg name In-Reply-To: <5.1.1.6.0.20051031155551.01fa6578@mail.telecommunity.com> References: <5.1.1.6.0.20051031155551.01fa6578@mail.telecommunity.com> Message-ID: On Mon, 31 Oct 2005, Phillip J. Eby wrote: > At 08:37 PM 10/31/2005 +0000, John J Lee wrote: >> svn: Unknown hostname 'svn.eby-sarna.com' > > This is a DNS problem that I just fixed (I think), please try again. > > >> Problems: >> >> 1. Failure to remove temporary SVN working copy on XP SP2. I get the >> same issue with my own manual checkouts, and use rmdir /q /s working_copy >> to get rid of them. > > Check and see if this happens when you're *not* getting the 'Unknown > hostname' bit. If so, then I'd guess there's a problem with how SVN is > setting the permissions. I'd need more info about this to debug. (I use > cygwin subversion and don't have any problems with this on Win2K.) [...] The DNS problem is gone, but it still fails to remove the checkout. That's no surprise to me, even though I'm ignorant of exactly how Windows handles file permissions: Everybody here uses non-cygwin SVN on Windows, and wev'e all found we have to use rmdir /q /s to kill off SVN working copies (or use Windows explorer) -- eg. win32-compiled rm -rf from SF project Unxutils does NOT do the trick. If I apply the following hack, the problem with removing the temp working copy goes away, but I still don't get dependencies downloaded or installed: --- setuptools\command\easy_install.py~ Mon Oct 31 18:09:31 2005 +++ setuptools\command\easy_install.py Tue Nov 01 11:12:53 2005 @@ -318,7 +318,8 @@ finally: if os.path.exists(tmpdir): - shutil.rmtree(tmpdir) + #shutil.rmtree(tmpdir) + os.system('rmdir /q /s %s' % tmpdir) Before applying patch: """ C:\jjlee>easy_install -d . -m "mechanize==dev" Searching for mechanize==dev Reading http://www.python.org/pypi/mechanize/ Reading http://wwwsearch.sourceforge.net/mechanize/ Best match: mechanize dev Downloading http://codespeak.net/svn/wwwsearch/mechanize/trunk#egg=mechanize-dev Doing subversion checkout from http://codespeak.net/svn/wwwsearch/mechanize/trunk to c:\docume~1\johnle~1\locals~1\temp\easy_install---5g_f\trunk Processing trunk Running setup.py -q bdist_egg --dist-dir c:\docume~1\johnle~1\locals~1\temp\easy_install---5g_f\trunk\egg-dist-tmp-ruuebc Installed c:\jjlee\mechanize-0.0.10a-py2.4.egg Because this distribution was installed --multi-version or --install-dir, before you can import modules from this package in an application, you will need to 'import pkg_resources' and then use a 'require()' call similar to one of these examples, in order to select the desired version: pkg_resources.require("mechanize") # latest installed version pkg_resources.require("mechanize==0.0.10a") # this exact version pkg_resources.require("mechanize>=0.0.10a") # this version or higher error: c:\docume~1\johnle~1\locals~1\temp\easy_install---5g_f\trunk\.svn\dir-prop-base: Permission denied C:\jjlee> """ After applying patch: """ Searching for mechanize==dev Reading http://www.python.org/pypi/mechanize/ Reading http://wwwsearch.sourceforge.net/mechanize/ Best match: mechanize dev Downloading http://codespeak.net/svn/wwwsearch/mechanize/trunk#egg=mechanize-dev Doing subversion checkout from http://codespeak.net/svn/wwwsearch/mechanize/trunk to c:\docume~1\johnle~1\locals~1\temp\easy_install-soaort\trunk Processing trunk Running setup.py -q bdist_egg --dist-dir c:\docume~1\johnle~1\locals~1\temp\easy_install-soaort\trunk\egg-dist-tmp-rqzfts Adding mechanize 0.0.10a to easy-install.pth file Installed c:\python24\lib\site-packages\mechanize-0.0.10a-py2.4.egg """ Note no dependencies downloaded, contrary to the setup.py's install_requires declaration (included in my earlier post here)! Any ideas? John From jjl at pobox.com Tue Nov 1 12:51:53 2005 From: jjl at pobox.com (John J Lee) Date: Tue, 1 Nov 2005 11:51:53 +0000 (GMT Standard Time) Subject: [Distutils] setuptools: Missing unmanaged module warning; Failed SVN cleanup; Wrong version in .egg name In-Reply-To: References: <5.1.1.6.0.20051031155551.01fa6578@mail.telecommunity.com> Message-ID: On Tue, 1 Nov 2005, John J Lee wrote: [...] > --- setuptools\command\easy_install.py~ Mon Oct 31 18:09:31 2005 > +++ setuptools\command\easy_install.py Tue Nov 01 11:12:53 2005 > @@ -318,7 +318,8 @@ > > finally: > if os.path.exists(tmpdir): > - shutil.rmtree(tmpdir) > + #shutil.rmtree(tmpdir) > + os.system('rmdir /q /s %s' % tmpdir) [...] A colleague suggests that setuptools should use svn export to avoid creating the permissions issue in the first place. Does that sound reasonable? John From pje at telecommunity.com Tue Nov 1 18:11:00 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 01 Nov 2005 12:11:00 -0500 Subject: [Distutils] setuptools: Missing unmanaged module warning; Failed SVN cleanup; Wrong version in .egg name In-Reply-To: References: <5.1.1.6.0.20051031155551.01fa6578@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051101120814.01fae658@mail.telecommunity.com> At 11:51 AM 11/1/2005 +0000, John J Lee wrote: >On Tue, 1 Nov 2005, John J Lee wrote: >[...] > > --- setuptools\command\easy_install.py~ Mon Oct 31 18:09:31 2005 > > +++ setuptools\command\easy_install.py Tue Nov 01 11:12:53 2005 > > @@ -318,7 +318,8 @@ > > > > finally: > > if os.path.exists(tmpdir): > > - shutil.rmtree(tmpdir) > > + #shutil.rmtree(tmpdir) > > + os.system('rmdir /q /s %s' % tmpdir) >[...] > >A colleague suggests that setuptools should use svn export to avoid >creating the permissions issue in the first place. Does that sound >reasonable? It would then not work for revision tagged builds, which is one of the big selling points for having subversion checkouts in the first place. It would also not work if you were planning to do development work on the code, which is another one of the big selling points of the feature. I wonder what the permission problem is, and if perhaps it could be fixed by easy_install doing an os.walk() and chmod'ing the files and/or directories before the rmtree? From jjl at pobox.com Tue Nov 1 19:41:52 2005 From: jjl at pobox.com (John J Lee) Date: Tue, 1 Nov 2005 18:41:52 +0000 (GMT Standard Time) Subject: [Distutils] setuptools: Missing unmanaged module warning; Failed SVN cleanup; Wrong version in .egg name In-Reply-To: <5.1.1.6.0.20051101120814.01fae658@mail.telecommunity.com> References: <5.1.1.6.0.20051031155551.01fa6578@mail.telecommunity.com> <5.1.1.6.0.20051101120814.01fae658@mail.telecommunity.com> Message-ID: On Tue, 1 Nov 2005, Phillip J. Eby wrote: [...] > I wonder what the permission problem is, and if perhaps it could be fixed by > easy_install doing an os.walk() and chmod'ing the files and/or directories > before the rmtree? I'll give it a try, just a minute... John From jjl at pobox.com Tue Nov 1 20:02:05 2005 From: jjl at pobox.com (John J Lee) Date: Tue, 1 Nov 2005 19:02:05 +0000 (GMT Standard Time) Subject: [Distutils] setuptools: Missing unmanaged module warning; Failed SVN cleanup; Wrong version in .egg name In-Reply-To: References: <5.1.1.6.0.20051031155551.01fa6578@mail.telecommunity.com> <5.1.1.6.0.20051101120814.01fae658@mail.telecommunity.com> Message-ID: On Tue, 1 Nov 2005, John J Lee wrote: > On Tue, 1 Nov 2005, Phillip J. Eby wrote: > [...] >> I wonder what the permission problem is, and if perhaps it could be fixed by >> easy_install doing an os.walk() and chmod'ing the files and/or directories >> before the rmtree? > > I'll give it a try, just a minute... Yes, that works -- see attached patch, verified to work on my XP SP2 machine with native win32 SVN 1.2.3. I'm still wondering why the advertised package dependency resolution doesn't seem to work for me, though -- multi-version or not (mechanize's dependencies ClientCookie, ClientForm and pullparser are not fetched): C:\jjlee\code>easy_install -d . -m "mechanize==dev" Searching for mechanize==dev Reading http://www.python.org/pypi/mechanize/ Reading http://wwwsearch.sourceforge.net/mechanize/ Best match: mechanize dev Downloading http://codespeak.net/svn/wwwsearch/mechanize/trunk#egg=mechanize-dev Doing subversion checkout from http://codespeak.net/svn/wwwsearch/mechanize/trunk to c:\docume~1\johnle~1\locals~1\temp\ easy_install-6b1iw5\trunk Processing trunk Running setup.py -q bdist_egg --dist-dir c:\docume~1\johnle~1\locals~1\temp\easy_install-6b1iw5\trunk\egg-dist-tmp-q7jmh m Installed c:\jjlee\code\mechanize-0.0.10a-py2.4.egg Because this distribution was installed --multi-version or --install-dir, before you can import modules from this package in an application, you will need to 'import pkg_resources' and then use a 'require()' call similar to one of these examples, in order to select the desired version: pkg_resources.require("mechanize") # latest installed version pkg_resources.require("mechanize==0.0.10a") # this exact version pkg_resources.require("mechanize>=0.0.10a") # this version or higher C:\jjlee\code>easy_install "mechanize==dev" Searching for mechanize==dev Reading http://www.python.org/pypi/mechanize/ Reading http://wwwsearch.sourceforge.net/mechanize/ Best match: mechanize dev Downloading http://codespeak.net/svn/wwwsearch/mechanize/trunk#egg=mechanize-dev Doing subversion checkout from http://codespeak.net/svn/wwwsearch/mechanize/trunk to c:\docume~1\johnle~1\locals~1\temp\ easy_install-pxmwtr\trunk Processing trunk Running setup.py -q bdist_egg --dist-dir c:\docume~1\johnle~1\locals~1\temp\easy_install-pxmwtr\trunk\egg-dist-tmp-ad3px 3 Adding mechanize 0.0.10a to easy-install.pth file Installed c:\python24\lib\site-packages\mechanize-0.0.10a-py2.4.egg C:\jjlee\code> John -------------- next part -------------- --- c:\Python24\Lib\site-packages\setuptools-0.6a6-py2.4.egg\setuptools\command\easy_install.py~ Tue Nov 01 18:56:30 2005 +++ c:\Python24\Lib\site-packages\setuptools-0.6a6-py2.4.egg\setuptools\command\easy_install.py Tue Nov 01 18:53:55 2005 @@ -10,7 +10,7 @@ __ http://peak.telecommunity.com/DevCenter/EasyInstall """ -import sys, os.path, zipimport, shutil, tempfile, zipfile, re +import sys, os.path, zipimport, shutil, tempfile, zipfile, re, stat from glob import glob from setuptools import Command from setuptools.sandbox import run_setup @@ -318,6 +318,9 @@ finally: if os.path.exists(tmpdir): + for dirpath, subdirs, fns in os.walk(tmpdir): + for fn in fns: + os.chmod(os.path.join(dirpath, fn), stat.S_IWRITE) shutil.rmtree(tmpdir) From pje at telecommunity.com Tue Nov 1 20:44:38 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 01 Nov 2005 14:44:38 -0500 Subject: [Distutils] setuptools: Missing unmanaged module warning; Failed SVN cleanup; Wrong version in .egg name In-Reply-To: References: <5.1.1.6.0.20051031155551.01fa6578@mail.telecommunity.com> <5.1.1.6.0.20051101120814.01fae658@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051101143756.01fae618@mail.telecommunity.com> At 07:02 PM 11/1/2005 +0000, John J Lee wrote: >Yes, that works -- see attached patch, verified to work on my XP SP2 >machine with native win32 SVN 1.2.3. Interestingly, I *am* able to reproduce your problem, but only with your packages, though I honestly haven't tried many others. Anyway, I'll work on getting a variant of your patch into 0.6a8. >I'm still wondering why the advertised package dependency resolution >doesn't seem to work for me, though -- multi-version or not (mechanize's >dependencies ClientCookie, ClientForm and pullparser are not fetched): Hm. The '==dev' part is the problem. Doing a bit of research, I see that easy_install doesn't process dependencies for a package when its name and version don't match what's requested on the command line. I've unfortunately forgotten *why* it does this, though. The workaround for the moment is to request 'mechanize>=dev' or just 'mechanize', both of which will match any 'mechanize' version. Then the dependencies get processed. I'll have to look into it some more to see if I can figure out why I did what I did; I think it may actually not be necessary any more to have that restriction, if in fact it ever was. From jjl at pobox.com Tue Nov 1 21:11:43 2005 From: jjl at pobox.com (John J Lee) Date: Tue, 1 Nov 2005 20:11:43 +0000 (GMT Standard Time) Subject: [Distutils] setuptools: Missing unmanaged module warning; Failed SVN cleanup; Wrong version in .egg name In-Reply-To: <5.1.1.6.0.20051101143756.01fae618@mail.telecommunity.com> References: <5.1.1.6.0.20051031155551.01fa6578@mail.telecommunity.com> <5.1.1.6.0.20051101120814.01fae658@mail.telecommunity.com> <5.1.1.6.0.20051101143756.01fae618@mail.telecommunity.com> Message-ID: On Tue, 1 Nov 2005, Phillip J. Eby wrote: > At 07:02 PM 11/1/2005 +0000, John J Lee wrote: >> Yes, that works -- see attached patch, verified to work on my XP SP2 >> machine with native win32 SVN 1.2.3. > > Interestingly, I *am* able to reproduce your problem, but only with your > packages, though I honestly haven't tried many others. Anyway, I'll work on > getting a variant of your patch into 0.6a8. How odd. FWIW, I wonder if the open source ReportLab toolkit also triggers this problem (since I see the same SVN working copy un-remove-ability with that in my day-to-day work). I can't try on this box as no MSVC installed and we don't have a setuptools setup.py for it yet... >> I'm still wondering why the advertised package dependency resolution >> doesn't seem to work for me, though -- multi-version or not (mechanize's >> dependencies ClientCookie, ClientForm and pullparser are not fetched): > > Hm. The '==dev' part is the problem. Doing a bit of research, I see that > easy_install doesn't process dependencies for a package when its name and > version don't match what's requested on the command line. I've unfortunately > forgotten *why* it does this, though. The workaround for the moment is to > request 'mechanize>=dev' or just 'mechanize', both of which will match any > 'mechanize' version. Then the dependencies get processed. That's not a workaround for me, since as it happens what drove me to support setuptools was the ability to grab the SVN version + deps and set up sys.path. (Not that the other stuff isn't great too, of course!-) > I'll have to look into it some more to see if I can figure out why I did what > I did; I think it may actually not be necessary any more to have that > restriction, if in fact it ever was. Great. John From pje at telecommunity.com Tue Nov 1 21:21:10 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 01 Nov 2005 15:21:10 -0500 Subject: [Distutils] setuptools: Missing unmanaged module warning; Failed SVN cleanup; Wrong version in .egg name In-Reply-To: References: <5.1.1.6.0.20051101143756.01fae618@mail.telecommunity.com> <5.1.1.6.0.20051031155551.01fa6578@mail.telecommunity.com> <5.1.1.6.0.20051101120814.01fae658@mail.telecommunity.com> <5.1.1.6.0.20051101143756.01fae618@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051101151801.02e9d548@mail.telecommunity.com> At 08:11 PM 11/1/2005 +0000, John J Lee wrote: >On Tue, 1 Nov 2005, Phillip J. Eby wrote: > > > At 07:02 PM 11/1/2005 +0000, John J Lee wrote: > >> Yes, that works -- see attached patch, verified to work on my XP SP2 > >> machine with native win32 SVN 1.2.3. > > > > Interestingly, I *am* able to reproduce your problem, but only with your > > packages, though I honestly haven't tried many others. Anyway, I'll > work on > > getting a variant of your patch into 0.6a8. > >How odd. FWIW, I wonder if the open source ReportLab toolkit also >triggers this problem (since I see the same SVN working copy >un-remove-ability with that in my day-to-day work). I can't try on this >box as no MSVC installed and we don't have a setuptools setup.py for it >yet... > > > >> I'm still wondering why the advertised package dependency resolution > >> doesn't seem to work for me, though -- multi-version or not (mechanize's > >> dependencies ClientCookie, ClientForm and pullparser are not fetched): > > > > Hm. The '==dev' part is the problem. Doing a bit of research, I see that > > easy_install doesn't process dependencies for a package when its name and > > version don't match what's requested on the command line. I've > unfortunately > > forgotten *why* it does this, though. The workaround for the moment is to > > request 'mechanize>=dev' or just 'mechanize', both of which will match any > > 'mechanize' version. Then the dependencies get processed. > >That's not a workaround for me, since as it happens what drove me to >support setuptools was the ability to grab the SVN version + deps and set >up sys.path. Well, if you run 'easy_install mechanize' after first running 'easy_install mechanize==dev', it will keep the existing installed 'mechanize' egg, and then download ClientForm. i.e., it's just a two step update. Actually, you can do 'easy_install mechanize==dev mechanize' on one line and it should do the same thing. :) But either approach is of course just a workaround until I figure out why I restricted dependency processing to things that match the specified version requested. From strawman at astraw.com Wed Nov 2 10:36:52 2005 From: strawman at astraw.com (Andrew Straw) Date: Wed, 2 Nov 2005 01:36:52 -0800 (PST) Subject: [Distutils] ez_setup 0.6a7 crashes on Python2.3, AMD64, Debian linux Message-ID: <47172.72.129.116.92.1130924212.squirrel@webmail.astraw.com> Hi, on a just-installed Debian sarge AMD64 installation, I have the following behavior, which looks similar to the issue discussed here: http://mail.python.org/pipermail/python-dev/2005-September/056955.html (Like the author of that message, I have also downloaded setuptools from CVS, and also get an OverflowError.) Any idea why setuptools doesn't seem to be running on my platform? $ uname -a Linux hdmg 2.6.14 #1 SMP Tue Nov 1 23:42:27 PST 2005 x86_64 GNU/Linux $ python Python 2.3.5 (#2, Sep 9 2005, 21:37:55) [GCC 3.3.5 (Debian 1:3.3.5-13)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import ez_setup >>> ez_setup.use_setuptools() --------------------------------------------------------------------------- This script requires setuptools version 0.6a7 to run (even to display help). I will attempt to download it for you (from http://cheeseshop.python.org/packages/2.3/s/setuptools/), but you may need to enable firewall access for this script first. I will start the download in 15 seconds. --------------------------------------------------------------------------- Downloading http://cheeseshop.python.org/packages/2.3/s/setuptools/setuptools-0.6a7-py2.3.egg Traceback (most recent call last): File "", line 1, in ? File "ez_setup.py", line 80, in use_setuptools import setuptools; setuptools.bootstrap_install_from = egg OverflowError: signed integer is greater than maximum From dangoor at gmail.com Wed Nov 2 17:20:45 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Wed, 2 Nov 2005 11:20:45 -0500 Subject: [Distutils] Fwd: [TurboGears] Error installing cElementTree ! In-Reply-To: <1130889062.304195.120340@z14g2000cwz.googlegroups.com> References: <1130889062.304195.120340@z14g2000cwz.googlegroups.com> Message-ID: <3f085ecd0511020820g50ef5780u108dec0f50fb7bda@mail.gmail.com> Has any recent change to setuptools affected a call to "swig_sources"? Kevin ---------- Forwarded message ---------- From: narm Date: Nov 1, 2005 6:51 PM Subject: [TurboGears] Error installing cElementTree ! To: TurboGears Hi all, I am trying to install on Fedora Core 4 (Python 2.4.1) and I get the following error (after easy_install -f http://www.turbogears.org/download/index.html --script-dir /usr/local/bin TurboGears): ... ... Running cElementTree-1.0.2-20050302/setup.py -q bdist_egg --dist-dir /tmp/easy_install-iN8sd5/cElementTree-1.0.2-20050302/egg-dist-tmp-3C0jrn Traceback (most recent call last): File "/usr/local/bin/easy_install", line 7, in ? sys.exit( File "/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/setuptools/command/easy_install.py", line 1181, in main setup(script_args = ['-q','easy_install', '-v']+argv, **kw) File "/usr/lib/python2.4/distutils/core.py", line 149, in setup dist.run_commands() ... ... TypeError: swig_sources() takes exactly 2 arguments (3 given) Any help, Thank you narm -- Kevin Dangoor Author of the Zesty News RSS newsreader email: kid at blazingthings.com company: http://www.BlazingThings.com blog: http://www.BlueSkyOnMars.com From ianb at colorstudy.com Wed Nov 2 17:39:16 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 02 Nov 2005 10:39:16 -0600 Subject: [Distutils] setuptools-0.6a7 In-Reply-To: <20051102163606.GA15044@phd.pp.ru> References: <20051102163606.GA15044@phd.pp.ru> Message-ID: <4368EBB4.3020300@colorstudy.com> Oleg Broytmann wrote: > Hello! > > When I test a new patch I set PYTHONPATH=. and run > py.test sqlobject/test/test_something.py > > Until today it works pretty good. Unfortunately, setuptools-0.6a7 has > got an additional intelligence and now spits this warning: > > /usr/local/lib/python2.4/site-packages/setuptools-0.6a7-py2.4.egg/pkg_resources.py:1943: UserWarning: Module sqlobject was already imported from /home/phd/work/SQLObject/SQLObject-trunk-index_get/sqlobject/__init__.py, but /usr/local/lib/python2.4/site-packages/SQLObject-0.8dev_r1185-py2.4.egg is being added to sys.path > warn( > > (I don't know what this 'warn(' means.) > > How can I stop setuptools-0.6a7 from being so clever? or at least how to > eliminate this warning? I think this is happening because sqlobject/conftest.py does require('SQLObject'), and apparently that's loading the installed egg instead of the code already on sys.path. I'm not sure what the proper resolution on this is. How can conftest.py make sure that the containing files are made active, and not any other installed SQLObject code? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Wed Nov 2 17:54:59 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 02 Nov 2005 11:54:59 -0500 Subject: [Distutils] ez_setup 0.6a7 crashes on Python2.3, AMD64, Debian linux In-Reply-To: <47172.72.129.116.92.1130924212.squirrel@webmail.astraw.com > Message-ID: <5.1.1.6.0.20051102114957.01fe8738@mail.telecommunity.com> At 01:36 AM 11/2/2005 -0800, Andrew Straw wrote: >Hi, on a just-installed Debian sarge AMD64 installation, I have the >following behavior, which looks similar to the issue discussed here: > >http://mail.python.org/pipermail/python-dev/2005-September/056955.html > >(Like the author of that message, I have also downloaded setuptools from >CVS, and also get an OverflowError.) > >Any idea why setuptools doesn't seem to be running on my platform? You need Python 2.4 if you're using a 64-bit platform; Python 2.3's "zipimport" module has a bug that wasn't fixed until Python 2.4 was released. See this message for more details: http://mail.python.org/pipermail/python-dev/2005-September/056818.html I did, however, forget to add this to the setuptools documentation as I promised in that message, so I've just added it to my local copy so I won't forget again. :) From ianb at imagescape.com Wed Nov 2 19:17:00 2005 From: ianb at imagescape.com (Ian Bicking) Date: Wed, 02 Nov 2005 12:17:00 -0600 Subject: [Distutils] setuptools: Bug in requirement loading Message-ID: <4369029C.1000302@imagescape.com> This seems odd to me, but the requirement loading just seems all off in pkg_resources. This is with 0.6a7. In resolve() we have this code: req = requirements.pop(0) # process dependencies breadth-firs if req in processed: # Ignore cyclic or redundant dependencies continue dist = best.get(req.key) if dist is None: # Find the best distribution and add it to the map dist = self.by_key.get(req.key) if dist is None: if env is None: env = Environment(self.entries) dist = best[req.key] = env.best_match(req, self, instal if dist is None: raise DistributionNotFound(req) # XXX put more inf to_activate.append(dist) elif dist not in req: # Oops, the "best" so far conflicts with a dependency raise VersionConflict(dist,req) # XXX put more info here requirements.extend(dist.requires(req.extras)[::-1]) processed[req] = True (got a little truncated, but ignoring that.) If "dist is None", i.e., no egg for the requirement is yet loaded, the egg is loaded with "self.by_key.get(req.key)". req.key doesn't have any version information, it's just the package name. How does the proper version get loaded? There's also no test that "dist in req" for that branch. So I'm currently doing "require('SQLObject>=0.8dev')" and getting back SQLObject 0.7b1, with no error. 0.7b1 is the default egg, which is why I guess it is in self.by_key. setuptools 0.6a5 acted the same way, so it's not a (recent) regression. If I change the second "if dist is None" to "if dist is None or dist not in req" then at least I get a version conflict. But I thought setuptools 0.6a7 was able to load an egg besides the default version. Actually, I'm now confused how anything but the default version can be loaded. I'm having a hard time figuring out how I didn't notice this before. Ian From pje at telecommunity.com Wed Nov 2 20:22:11 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 02 Nov 2005 14:22:11 -0500 Subject: [Distutils] Fwd: [TurboGears] Error installing cElementTree ! In-Reply-To: <3f085ecd0511020820g50ef5780u108dec0f50fb7bda@mail.gmail.co m> References: <1130889062.304195.120340@z14g2000cwz.googlegroups.com> <1130889062.304195.120340@z14g2000cwz.googlegroups.com> Message-ID: <5.1.1.6.0.20051102142102.01febdf0@mail.telecommunity.com> At 11:20 AM 11/2/2005 -0500, Kevin Dangoor wrote: >Has any recent change to setuptools affected a call to "swig_sources"? > >Kevin No. If you search the TurboGears archives via Google, someone else reported this problem in September, and the issue turned out to be missing files in a distribution on turbogears.org. You might check to see if this is the same thing. >---------- Forwarded message ---------- >From: narm >Date: Nov 1, 2005 6:51 PM >Subject: [TurboGears] Error installing cElementTree ! >To: TurboGears > > > >Hi all, >I am trying to install on Fedora Core 4 (Python 2.4.1) and I get the >following error (after easy_install -f >http://www.turbogears.org/download/index.html --script-dir >/usr/local/bin TurboGears): > >... >... >Running cElementTree-1.0.2-20050302/setup.py -q bdist_egg --dist-dir >/tmp/easy_install-iN8sd5/cElementTree-1.0.2-20050302/egg-dist-tmp-3C0jrn >Traceback (most recent call last): > File "/usr/local/bin/easy_install", line 7, in ? > sys.exit( >File >"/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/setuptools/command/easy_install.py", >line 1181, in main > setup(script_args = ['-q','easy_install', '-v']+argv, **kw) > File "/usr/lib/python2.4/distutils/core.py", line 149, in setup > dist.run_commands() > >... >... >TypeError: swig_sources() takes exactly 2 arguments (3 given) > > >Any help, Thank you >narm From pje at telecommunity.com Wed Nov 2 20:26:03 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 02 Nov 2005 14:26:03 -0500 Subject: [Distutils] setuptools-0.6a7 In-Reply-To: <4368EBB4.3020300@colorstudy.com> References: <20051102163606.GA15044@phd.pp.ru> <20051102163606.GA15044@phd.pp.ru> Message-ID: <5.1.1.6.0.20051102142236.0309cb78@mail.telecommunity.com> At 10:39 AM 11/2/2005 -0600, Ian Bicking wrote: >Oleg Broytmann wrote: > > Hello! > > > > When I test a new patch I set PYTHONPATH=. and run > > py.test sqlobject/test/test_something.py > > > > Until today it works pretty good. Unfortunately, setuptools-0.6a7 has > > got an additional intelligence and now spits this warning: > > > > > /usr/local/lib/python2.4/site-packages/setuptools-0.6a7-py2.4.egg/pkg_resources.py:1943: > UserWarning: Module sqlobject was already imported from > /home/phd/work/SQLObject/SQLObject-trunk-index_get/sqlobject/__init__.py, > but > /usr/local/lib/python2.4/site-packages/SQLObject-0.8dev_r1185-py2.4.egg > is being added to sys.path > > warn( > > > > (I don't know what this 'warn(' means.) > > > > How can I stop setuptools-0.6a7 from being so clever? or at least how to > > eliminate this warning? Try running 'setup.py egg_info' before trying to run the tests. Most likely, the problem is because your checkout doesn't contain any egg metadata, so the runtime system can't tell that '.' is actually an egg, or what version it has. From dangoor at gmail.com Wed Nov 2 20:36:26 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Wed, 2 Nov 2005 14:36:26 -0500 Subject: [Distutils] Fwd: [TurboGears] Error installing cElementTree ! In-Reply-To: <5.1.1.6.0.20051102142102.01febdf0@mail.telecommunity.com> References: <1130889062.304195.120340@z14g2000cwz.googlegroups.com> <5.1.1.6.0.20051102142102.01febdf0@mail.telecommunity.com> Message-ID: <3f085ecd0511021136k294272ban2a676e1341d4a944@mail.gmail.com> On 11/2/05, Phillip J. Eby wrote: > At 11:20 AM 11/2/2005 -0500, Kevin Dangoor wrote: > >Has any recent change to setuptools affected a call to "swig_sources"? > > > >Kevin > > No. If you search the TurboGears archives via Google, someone else > reported this problem in September, and the issue turned out to be missing > files in a distribution on turbogears.org. You might check to see if this > is the same thing. It doesn't look like it. The cElementTree tarball that's up there now has all of the .h files. Kevin From pje at telecommunity.com Wed Nov 2 20:38:09 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 02 Nov 2005 14:38:09 -0500 Subject: [Distutils] setuptools: Bug in requirement loading In-Reply-To: <4369029C.1000302@imagescape.com> Message-ID: <5.1.1.6.0.20051102142623.01fda570@mail.telecommunity.com> At 12:17 PM 11/2/2005 -0600, Ian Bicking wrote: >(got a little truncated, but ignoring that.) If "dist is None", i.e., >no egg for the requirement is yet loaded, the egg is loaded with >"self.by_key.get(req.key)". req.key doesn't have any version >information, it's just the package name. Eep. That is indeed a bug. >If I change the second "if dist is None" to "if dist is None or dist not >in req" then at least I get a version conflict. But I thought >setuptools 0.6a7 was able to load an egg besides the default version. It is able to, as long as the egg is one required by the __main__ program. There is special startup logic that checks for VersionConflict on application start, and then fudges things to work around it. It is not, however, an all-purpose facility to allow loading non-default versions. That would involve more complex refactoring and isn't something I'll attempt in a minor release. (Indeed, I suspect that supporting it properly would require a rethink of the entire WorkingSet API and implementation, and perhaps a fair bit of hooking into Python's import machinery besides.) >Actually, I'm now confused how anything but the default version can be >loaded. Yeah, that's the bug all right. >I'm having a hard time figuring out how I didn't notice this before. Me too! I'm guessing this was introduced when I made some changes to resolve() to avoid unnecessary directory scans. resolve() was previously not even looking to see if the working set already contained a matching distribution. Anyway, the simplest fix should be to change the 'elif dist not in req' to 'if dist not in req'. From phd at phd.pp.ru Wed Nov 2 20:45:04 2005 From: phd at phd.pp.ru (Oleg Broytmann) Date: Wed, 2 Nov 2005 22:45:04 +0300 Subject: [Distutils] setuptools-0.6a7 In-Reply-To: <5.1.1.6.0.20051102142236.0309cb78@mail.telecommunity.com> References: <20051102163606.GA15044@phd.pp.ru> <20051102163606.GA15044@phd.pp.ru> <5.1.1.6.0.20051102142236.0309cb78@mail.telecommunity.com> Message-ID: <20051102194504.GA24563@phd.pp.ru> On Wed, Nov 02, 2005 at 02:26:03PM -0500, Phillip J. Eby wrote: > Try running 'setup.py egg_info' before trying to run the tests. Yes, this helps. Thank you! Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From narm at go.com.jo Wed Nov 2 21:01:28 2005 From: narm at go.com.jo (Basem Narmok) Date: Wed, 02 Nov 2005 22:01:28 +0200 Subject: [Distutils] Fwd: [TurboGears] Error installing cElementTree ! In-Reply-To: <3f085ecd0511021136k294272ban2a676e1341d4a944@mail.gmail.com> References: <1130889062.304195.120340@z14g2000cwz.googlegroups.com> <5.1.1.6.0.20051102142102.01febdf0@mail.telecommunity.com> <3f085ecd0511021136k294272ban2a676e1341d4a944@mail.gmail.com> Message-ID: <43691B18.4040407@go.com.jo> Kevin Dangoor wrote: >On 11/2/05, Phillip J. Eby wrote: > > >>At 11:20 AM 11/2/2005 -0500, Kevin Dangoor wrote: >> >> >>>Has any recent change to setuptools affected a call to "swig_sources"? >>> >>>Kevin >>> >>> >>No. If you search the TurboGears archives via Google, someone else >>reported this problem in September, and the issue turned out to be missing >>files in a distribution on turbogears.org. You might check to see if this >>is the same thing. >> >> > >It doesn't look like it. The cElementTree tarball that's up there now >has all of the .h files. > >Kevin > > > Hi all, This is all of the traceback: easy_install -f http://www.turbogears.org/download/index.html --script-dir /usr/local/bin TurboGears Reading http://www.turbogears.org/download/index.html Searching for TurboGears Best match: TurboGears 0.8a4 Downloading http://www.turbogears.org/download/eggs/TurboGears-0.8a4-py2.4.egg Processing TurboGears-0.8a4-py2.4.egg removing '/usr/lib/python2.4/site-packages/TurboGears-0.8a4-py2.4.egg' (and everything under it) creating /usr/lib/python2.4/site-packages/TurboGears-0.8a4-py2.4.egg Extracting TurboGears-0.8a4-py2.4.egg to /usr/lib/python2.4/site-packages TurboGears 0.8a4 is already the active version in easy-install.pth Installing tg-admin script to /usr/local/bin Installed /usr/lib/python2.4/site-packages/TurboGears-0.8a4-py2.4.egg Processing dependencies for TurboGears Searching for cElementTree>=1.0.2 Best match: cElementTree 1.0.2-20050302 Downloading http://www.turbogears.org/download/eggs/cElementTree-1.0.2-20050302.tar.gz Processing cElementTree-1.0.2-20050302.tar.gz Running cElementTree-1.0.2-20050302/setup.py -q bdist_egg --dist-dir /tmp/easy_install-XzDGfh/cElementTree-1.0.2-20050302/egg-dist-tmp-scXJsf Traceback (most recent call last): File "/usr/local/bin/easy_install", line 7, in ? sys.exit( File "/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/setuptools/command/easy_install.py", line 1181, in main setup(script_args = ['-q','easy_install', '-v']+argv, **kw) File "/usr/lib/python2.4/distutils/core.py", line 149, in setup dist.run_commands() File "/usr/lib/python2.4/distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/usr/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/setuptools/command/easy_install.py", line 230, in run self.easy_install(spec, not self.no_deps) File "/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/setuptools/command/easy_install.py", line 317, in easy_install return self.install_item(spec, download, tmpdir, deps) File "/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/setuptools/command/easy_install.py", line 340, in install_item self.process_distribution(spec, dist, deps) File "/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/setuptools/command/easy_install.py", line 397, in process_distribution WorkingSet([]).resolve( File "/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/pkg_resources.py", line 481, in resolve dist = best[req.key] = env.best_match(req, self, installer) File "/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/pkg_resources.py", line 635, in best_match return self.obtain(req, installer) # try and download/install File "/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/pkg_resources.py", line 647, in obtain return installer(requirement) File "/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/setuptools/command/easy_install.py", line 317, in easy_install return self.install_item(spec, download, tmpdir, deps) File "/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/setuptools/command/easy_install.py", line 338, in install_item dists = self.install_eggs(spec, download, tmpdir) File "/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/setuptools/command/easy_install.py", line 573, in install_eggs return self.build_and_install(setup_script, setup_base) File "/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/setuptools/command/easy_install.py", line 848, in build_and_install self.run_setup(setup_script, setup_base, args) File "/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/setuptools/command/easy_install.py", line 837, in run_setup run_setup(setup_script, args) File "/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/setuptools/sandbox.py", line 26, in run_setup DirectorySandbox(setup_dir).run( File "/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/setuptools/sandbox.py", line 63, in run return func() File "/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/setuptools/sandbox.py", line 29, in {'__file__':setup_script, '__name__':'__main__'} File "setup.py", line 93, in ? File "/usr/lib/python2.4/distutils/core.py", line 149, in setup dist.run_commands() File "/usr/lib/python2.4/distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/usr/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/setuptools/command/bdist_egg.py", line 174, in run cmd = self.call_command('install_lib', warn_dir=0) File "/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/setuptools/command/bdist_egg.py", line 161, in call_command self.run_command(cmdname) File "/usr/lib/python2.4/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/usr/lib/python2.4/distutils/command/install_lib.py", line 88, in run self.build() File "/usr/lib/python2.4/distutils/command/install_lib.py", line 110, in build self.run_command('build_ext') File "/usr/lib/python2.4/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/usr/lib/python2.4/site-packages/setuptools-0.6a6-py2.4.egg/setuptools/command/build_ext.py", line 16, in run _build_ext.run(self) File "/usr/lib/python2.4/distutils/command/build_ext.py", line 279, in run self.build_extensions() File "/usr/lib/python2.4/distutils/command/build_ext.py", line 405, in build_extensions self.build_extension(ext) File "/usr/lib/python2.4/distutils/command/build_ext.py", line 442, in build_extension sources = self.swig_sources(sources, ext) TypeError: swig_sources() takes exactly 2 arguments (3 given) From pje at telecommunity.com Wed Nov 2 21:39:48 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 02 Nov 2005 15:39:48 -0500 Subject: [Distutils] Fwd: [TurboGears] Error installing cElementTree ! Message-ID: <5.1.1.6.0.20051102153933.02f580a0@mail.telecommunity.com> At 02:36 PM 11/2/2005 -0500, Kevin Dangoor wrote: >On 11/2/05, Phillip J. Eby wrote: > > At 11:20 AM 11/2/2005 -0500, Kevin Dangoor wrote: > > >Has any recent change to setuptools affected a call to "swig_sources"? > > > > > >Kevin > > > > No. If you search the TurboGears archives via Google, someone else > > reported this problem in September, and the issue turned out to be missing > > files in a distribution on turbogears.org. You might check to see if this > > is the same thing. > >It doesn't look like it. The cElementTree tarball that's up there now >has all of the .h files. Okay, after a bunch of Googling for swig_sources, I think the issue is that Pyrex is installed, but not a version that's compatible with Python 2.4. See this patch: https://www.redhat.com/archives/fedora-cvs-commits/2005-September/msg01305.html It looks like the user needs to either uninstall the Pyrex RPM, or install the updated version that contains this patch. See also this URL, that suggests what version might be appropriate: http://people.ubuntulinux.org/~scott/patches/pyrex/pyrex_0.9.3-3ubuntu1.patch So, basically Pyrex's distutils support is broken for Python 2.4 without the patch to swig_sources. :( Last, but not least, see also: http://bugzilla.ubuntu.com/show_bug.cgi?id=4852 which is probably a good canonical location to point people to. However, now that I know the source of the problem, I can add a monkeypatch to the next version of setuptools to would work around the issue, if detected. From simon at arrowtheory.com Thu Nov 3 12:17:21 2005 From: simon at arrowtheory.com (Simon Burton) Date: Thu, 3 Nov 2005 11:17:21 +0000 Subject: [Distutils] -O3 brings the compiler to it's knees Message-ID: <20051103111721.0a4c3bcb.simon@arrowtheory.com> Is there a way to disable -O3 builds ? I have a big (100k+ lines) extension module, and -O3 compile is, at best useless and terribly slow to build, and at worst gobbles up all memory and kills the compile. Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com From simon at arrowtheory.com Thu Nov 3 12:52:58 2005 From: simon at arrowtheory.com (Simon Burton) Date: Thu, 3 Nov 2005 11:52:58 +0000 Subject: [Distutils] -O3 brings the compiler to it's knees In-Reply-To: <20051103111721.0a4c3bcb.simon@arrowtheory.com> References: <20051103111721.0a4c3bcb.simon@arrowtheory.com> Message-ID: <20051103115258.3d093dd3.simon@arrowtheory.com> On Thu, 3 Nov 2005 11:17:21 +0000 Simon Burton wrote: > > Is there a way to disable -O3 builds ? I have a big (100k+ lines) extension > module, and -O3 compile is, at best useless and terribly slow to build, and at worst > gobbles up all memory and kills the compile. > > Simon. OK, i've set OPT in the sysconfig vars dictionary to " -w ". Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com From pje at telecommunity.com Thu Nov 3 05:07:13 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 02 Nov 2005 23:07:13 -0500 Subject: [Distutils] New setuptools in subversion Message-ID: <5.1.1.6.0.20051102225818.0205ace0@mail.telecommunity.com> Since the Python CVS has been migrated to Subversion, it's now possible to get fresh setuptools checkouts using EasyInstall. That is, you can use: easy_install setuptools==dev to request the development version of setuptools from Subversion. The current in-development version is 0.6a8dev, which means it is an unreleased version of 0.6a8, and will be considered obsolete when the real 0.6a8 is released. Some new "features" of the dev version include: * Bug fixes for all the Subversion-related problems reported recently by John J. Lee * Fix for the version conflict problem reported today by Ian Bicking * Fix for the extension building problem w/Pyrex and Python 2.4 reported by Kevin Dangoor So if you've been experiencing any of the above issues, you might want to give it a try. I still have a long list of to-do's for the official 0.6a8 release, so it probably won't be out before the weekend. In any case, it's always helpful to get some third-party testing before making an official release. Thanks. From pje at telecommunity.com Thu Nov 3 16:47:50 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 03 Nov 2005 10:47:50 -0500 Subject: [Distutils] New setuptools in subversion In-Reply-To: <3f085ecd0511030616w1efb480eh388623fd5b4a3404@mail.gmail.co m> References: <5.1.1.6.0.20051102225818.0205ace0@mail.telecommunity.com> <5.1.1.6.0.20051102225818.0205ace0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051103104513.0205c3d8@mail.telecommunity.com> At 09:16 AM 11/3/2005 -0500, Kevin Dangoor wrote: >On 11/2/05, Phillip J. Eby wrote: > > Since the Python CVS has been migrated to Subversion, it's now possible to > > get fresh setuptools checkouts using EasyInstall. That is, you can use: > > > > easy_install setuptools==dev > >How do you set this up? I'd like to do something similar for >TurboGears, since the 0.9 release is a bit bigger than the 0.8 >release. Just include an #egg link in your PyPI long_description. Take a look at the setuptools setup.py and setuptools.txt (which is where setup.py reads the long description from). Alternately, you can just put the #egg link on the download page, since in your case the download page is linked from PyPI. EasyInstall scans the home page and download pages of a project for useful links when you request it. Note, by the way, that this means as long as you have links on your download page for all the dependency projects, they should be satisfied from there as well, without a need to explicitly reference the download page. From ianb at colorstudy.com Thu Nov 3 23:09:21 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 03 Nov 2005 16:09:21 -0600 Subject: [Distutils] find_package_data in setuptools Message-ID: <436A8A91.90609@colorstudy.com> I haven't really used the find_package_data that I emails distutils-sig about before (http://svn.pythonpaste.org/Paste/trunk/paste/util/finddata.py; just haven't gotten around to it, really. But clearly something like that should go in setuptools. And it's harder to maintain separately because you can't import it from another package (though I suppose svn:externals would work). TurboGears has been using the function; has it been working well there? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From delza at livingcode.org Fri Nov 4 00:46:51 2005 From: delza at livingcode.org (Dethe Elza) Date: Thu, 3 Nov 2005 15:46:51 -0800 Subject: [Distutils] (no subject) Message-ID: <55286AD0-5D84-4DF1-9615-CD1B7EC881AD@livingcode.org> Hi folks, Over on the vpython-users mailing list I've been trying to get help with VPython's complicated autoconf-based setup, which turns out to be fairly broken for OS X. They realize that the current setup is too complex and are looking at other options. The following is an exerpt from the ongoing discussion: > On Wed, 2005-10-19 at 21:06 -0700, Dethe Elza wrote: > >> On 19-Oct-05, at 4:57 PM, Jonathan Brandmeyer wrote: >> >>> >>> Are you (or anyone else out there) familiar with Scons? >>> >> >> What is wrong with using distutils[1], the standard method for >> building Python extensions? The new Setuptools[2] package makes it >> even easier, and supports EasyInstall[3] and Python Eggs[4]. >> >> [1] http://docs.python.org/lib/module-distutils.html >> [2] http://peak.telecommunity.com/DevCenter/setuptools >> [3] http://peak.telecommunity.com/DevCenter/EasyInstall >> [4] http://peak.telecommunity.com/DevCenter/PythonEggs >> > > But it doesn't properly support C++, or the use of a compiler other > than > the one that built Python itself. Those features are absolutely > required for VPython. > > -Jonathan I'm not a distutils/setuptools expert (though I'm willing and eager to become one rather than struggle with autoconf any further), but can anyone tell me if Jonathan's statement is even true? I'm willing to create a parallel setuptools-based builder just for OS X use, since the same compiler problem isn't an issue on OS X, but can I use distutils with C++ code? Thanks! --Dethe From pje at telecommunity.com Fri Nov 4 01:08:00 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 03 Nov 2005 19:08:00 -0500 Subject: [Distutils] (no subject) In-Reply-To: <55286AD0-5D84-4DF1-9615-CD1B7EC881AD@livingcode.org> Message-ID: <5.1.1.6.0.20051103190112.0205b630@mail.telecommunity.com> At 03:46 PM 11/3/2005 -0800, Dethe Elza wrote: > > But it doesn't properly support C++, or the use of a compiler other > > than > > the one that built Python itself. Those features are absolutely > > required for VPython. > > > > -Jonathan > >I'm not a distutils/setuptools expert (though I'm willing and eager >to become one rather than struggle with autoconf any further), but >can anyone tell me if Jonathan's statement is even true? > >I'm willing to create a parallel setuptools-based builder just for OS >X use, since the same compiler problem isn't an issue on OS X, but >can I use distutils with C++ code? As far as I know, yes. I don't know what is meant by "properly support C++", though. If the meaning is "support static initializers when Python was not built with a C++ compiler and runtime", then that statement might be correct, but if that is the issue then no other method of building (short of rebuilding the Python executable) would fix the problem anyway, and distutils is neither the problem nor the solution in that case. As for the "use of a compiler other than the one that built Python itself", that's not correct either. For example, it is common to build Python extensions for Windows using distutils and the MinGW compiler, even though Python itself is built with Visual C++. Also, it's possible to get the distutils to build using the freeware version of Visual C++ as well. Alas, I am not an OS X or C++ expert, so I'm unlikely to be able to give you much guidance, other than pointing you to the distutils docs: http://docs.python.org/ext/building.html As far as I'm aware, it should be sufficient to list the C++ (.cpp) files in 'sources', and to specify the various libraries, defines, etc. as shown on the page above. (Creating a useful cross-platform setup.py will of course require conditional code to change the libraries, macros, or other values as appropriate.) From delza at livingcode.org Fri Nov 4 01:22:35 2005 From: delza at livingcode.org (Dethe Elza) Date: Thu, 3 Nov 2005 16:22:35 -0800 Subject: [Distutils] (no subject) In-Reply-To: <5.1.1.6.0.20051103190112.0205b630@mail.telecommunity.com> References: <5.1.1.6.0.20051103190112.0205b630@mail.telecommunity.com> Message-ID: <676D8087-1CF8-42CA-B13C-A94EEC40F268@livingcode.org> Thanks, Phillip. You've confirmed what I thought, the next step is for me to dig in and try it. --Dethe "Computers are beyond dumb, they're mind-numbingly stupid. They're hostile, rigid, capricious, and unforgiving. They're impossibly demanding and they never learn anything." -- John R. Levine From pobrien at orbtech.com Fri Nov 4 01:29:13 2005 From: pobrien at orbtech.com (Patrick K. O'Brien) Date: Thu, 03 Nov 2005 18:29:13 -0600 Subject: [Distutils] (no subject) In-Reply-To: <5.1.1.6.0.20051103190112.0205b630@mail.telecommunity.com> References: <5.1.1.6.0.20051103190112.0205b630@mail.telecommunity.com> Message-ID: <436AAB59.4020900@orbtech.com> Phillip J. Eby wrote: > > As for the "use of a compiler other than the one that built Python itself", > that's not correct either. For example, it is common to build Python > extensions for Windows using distutils and the MinGW compiler, even though > Python itself is built with Visual C++. Also, it's possible to get the > distutils to build using the freeware version of Visual C++ as well. Unfortunately there is a good deal of outdated information on the web that lists all sorts of hoops one needs to jump through to support a compiler such as MinGW. Or rather, used to have to jump through. Things are much easier now. Take Qt for example. Most people still refer to a webpage that lists lots of steps that are no longer needed, at least not with Python 2.4. -- Patrick K. O'Brien Orbtech http://www.orbtech.com Schevo http://www.schevo.org From pje at telecommunity.com Fri Nov 4 01:56:31 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 03 Nov 2005 19:56:31 -0500 Subject: [Distutils] (no subject) In-Reply-To: <5.1.1.6.0.20051103190112.0205b630@mail.telecommunity.com> References: <55286AD0-5D84-4DF1-9615-CD1B7EC881AD@livingcode.org> Message-ID: <5.1.1.6.0.20051103193225.01fad4b0@mail.telecommunity.com> At 07:08 PM 11/3/2005 -0500, Phillip J. Eby wrote: >As far as I know, yes. I don't know what is meant by "properly support >C++", though. If the meaning is "support static initializers when Python >was not built with a C++ compiler and runtime", then that statement might >be correct, but if that is the issue then no other method of building >(short of rebuilding the Python executable) would fix the problem anyway, >and distutils is neither the problem nor the solution in that case. A follow up on this... it appears that "proper support for C++" was implemented in the distutils about 2 years ago, and has been distributed with Python since version 2.3: http://svn.python.org/view/python/trunk/Lib/distutils/command/build_ext.py?rev=29513&view=markup If I understand this link correctly, no special actions are needed to build C++ extension modules with the distutils; just list 'em and go. Of course, if you're still using 2.2 (like Google), you're probably out of luck on this issue. From delza at livingcode.org Fri Nov 4 05:33:58 2005 From: delza at livingcode.org (Dethe Elza) Date: Thu, 3 Nov 2005 20:33:58 -0800 Subject: [Distutils] Compiling VPython In-Reply-To: <5.1.1.6.0.20051103193225.01fad4b0@mail.telecommunity.com> References: <55286AD0-5D84-4DF1-9615-CD1B7EC881AD@livingcode.org> <5.1.1.6.0.20051103193225.01fad4b0@mail.telecommunity.com> Message-ID: <48656413-64C9-408B-9A8C-5E2CD3165260@livingcode.org> On 3-Nov-05, at 4:56 PM, Phillip J. Eby wrote: > If I understand this link correctly, no special actions are needed > to build C++ extension modules with the distutils; just list 'em > and go. Excellent, I'll try it and let you know. > Of course, if you're still using 2.2 (like Google), you're probably > out of luck on this issue. No, using 2.4 is the goal. --Dethe "Well I've wrestled with reality for thirty-five years now, doctor, and I'm happy to state I've finally won out over it." -- Elwood P. Dowd, Harvey From ianb at colorstudy.com Fri Nov 4 20:09:54 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 04 Nov 2005 13:09:54 -0600 Subject: [Distutils] setuptools: .egg-info/ and PKG-INFO In-Reply-To: <5.1.1.6.0.20051028211034.02c47680@mail.telecommunity.com> References: <5.1.1.6.0.20051028211034.02c47680@mail.telecommunity.com> Message-ID: <436BB202.4070407@colorstudy.com> Sorry, I somehow missed this response... Phillip J. Eby wrote: > At 04:30 PM 10/25/2005 -0500, Ian Bicking wrote: > >> I'm trying to figure out how I should deal with the .egg-info directory, >> and PKG-INFO. >> >> Right now PKG-INFO is being generated from setup.py. So I don't want to >> put it in my source control repository, since it is derivative. But I'm >> maintaining by hand some other files in the .egg-info directory, usually >> custom package metadata that we consume internally, so I need those >> files (and .egg-info) in the repository, and hence in the checkout. >> >> However, without PKG-INFO lots of things break, as pkg_resources gives a >> ValueError during some operations (e.g.: ValueError: ("Missing >> 'Version:' header and/or PKG-INFO file", ProjectName [unknown version] >> (/usr/home/ianb/co/IscapeLib/tests/appsetup/output/Proj-05))) -- this >> happens anytime I iterate over entry points, including when setuptools >> iterates over entry points, which means lots of setup.py commands are >> broken. They are broken simply because the incomplete .egg-info >> directory is on the path, even though I don't need direct access to it. >> I can run setup.py egg_info (creating PKG-INFO) for some reason (that >> I have not yet determined), but not my own custom setup.py extensions. >> This includes stuff like "python setup.py egg_info my_custom_command", >> so I have to write PKG-INFO in a completely separate command. > > > 'setup.py develop' should create the info and put the directory on the > path for you. Likewise, so does 'setup.py test'. Which other commands > are giving you problems? Well, like I said, any custom command. These just inherit from setuptools.Command, implementing the standard initialize_options/finalize_options/run methods. I even tried having my custom command recursively call "egg_info" (with get_finalized_command), but because the error is signaled before I get as far as .run() (where I put the egg_info call) it doesn't help. Maybe I could use another method that is invoked earlier. Though now I can't reproduce it; maybe this was fixed in 0.6a7? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Fri Nov 4 21:15:43 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 04 Nov 2005 15:15:43 -0500 Subject: [Distutils] setuptools: .egg-info/ and PKG-INFO In-Reply-To: <436BB202.4070407@colorstudy.com> References: <5.1.1.6.0.20051028211034.02c47680@mail.telecommunity.com> <5.1.1.6.0.20051028211034.02c47680@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051104150825.01fa64b0@mail.telecommunity.com> At 01:09 PM 11/4/2005 -0600, Ian Bicking wrote: >Well, like I said, any custom command. These just inherit from >setuptools.Command, implementing the standard >initialize_options/finalize_options/run methods. I even tried having my >custom command recursively call "egg_info" (with get_finalized_command), >but because the error is signaled before I get as far as .run() (where I >put the egg_info call) it doesn't help. Maybe I could use another method >that is invoked earlier. > >Though now I can't reproduce it; maybe this was fixed in 0.6a7? Dunno. As I mentioned on the Turbogears list, the problem isn't going to happen if your source code is in a subdirectory of the one containing setup.py. Perhaps you've changed layouts? In any case, as I mentioned on the Turbogears list, I'm going to try to put in some code to detect whether there is a "broken egg" in the working set and monkeypatch its 'version' attribute to work around this bootstrapping problem. That should make it more robust and eliminate the need to include any of the .egg-info files in Subversion, at least for most normal projects. (Setuptools is an oddity in that it defines entry points that it then uses while building itself! Its .egg-info/entry_points.txt file is therefore not optional. But the need for this sort of self-bootstrapping setup process should be extremely rare.) From ianb at colorstudy.com Fri Nov 4 21:19:38 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 04 Nov 2005 14:19:38 -0600 Subject: [Distutils] setuptools: .egg-info/ and PKG-INFO In-Reply-To: <5.1.1.6.0.20051104150825.01fa64b0@mail.telecommunity.com> References: <5.1.1.6.0.20051028211034.02c47680@mail.telecommunity.com> <5.1.1.6.0.20051028211034.02c47680@mail.telecommunity.com> <5.1.1.6.0.20051104150825.01fa64b0@mail.telecommunity.com> Message-ID: <436BC25A.6090201@colorstudy.com> Phillip J. Eby wrote: > At 01:09 PM 11/4/2005 -0600, Ian Bicking wrote: > >> Well, like I said, any custom command. These just inherit from >> setuptools.Command, implementing the standard >> initialize_options/finalize_options/run methods. I even tried having >> my custom command recursively call "egg_info" (with >> get_finalized_command), but because the error is signaled before I get >> as far as .run() (where I put the egg_info call) it doesn't help. >> Maybe I could use another method that is invoked earlier. >> >> Though now I can't reproduce it; maybe this was fixed in 0.6a7? > > > Dunno. As I mentioned on the Turbogears list, the problem isn't going > to happen if your source code is in a subdirectory of the one containing > setup.py. Perhaps you've changed layouts? > > In any case, as I mentioned on the Turbogears list, I'm going to try to > put in some code to detect whether there is a "broken egg" in the > working set and monkeypatch its 'version' attribute to work around this > bootstrapping problem. That should make it more robust and eliminate > the need to include any of the .egg-info files in Subversion, at least > for most normal projects. (Setuptools is an oddity in that it defines > entry points that it then uses while building itself! Its > .egg-info/entry_points.txt file is therefore not optional. But the need > for this sort of self-bootstrapping setup process should be extremely > rare.) FYI, I'm already putting my own custom files in the .egg-info/ directory. And sqlobject-admin now looks for a sqlobject.txt file in the egg metadata, which it uses to find the SQLObject subclasses in the project (and a couple other things). These could all possibly be moved to entry points, but treating them as simple configuration files has so far seemed easier to manage. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Fri Nov 4 21:31:56 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 04 Nov 2005 15:31:56 -0500 Subject: [Distutils] setuptools: .egg-info/ and PKG-INFO In-Reply-To: <436BC25A.6090201@colorstudy.com> References: <5.1.1.6.0.20051104150825.01fa64b0@mail.telecommunity.com> <5.1.1.6.0.20051028211034.02c47680@mail.telecommunity.com> <5.1.1.6.0.20051028211034.02c47680@mail.telecommunity.com> <5.1.1.6.0.20051104150825.01fa64b0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051104152946.01fdc640@mail.telecommunity.com> At 02:19 PM 11/4/2005 -0600, Ian Bicking wrote: >FYI, I'm already putting my own custom files in the .egg-info/ >directory. And sqlobject-admin now looks for a sqlobject.txt file in the >egg metadata, which it uses to find the SQLObject subclasses in the >project (and a couple other things). These could all possibly be moved to >entry points, but treating them as simple configuration files has so far >seemed easier to manage. Not everything has to be an entry point. But if it's configuration that's easier to express in Python code, or if you prefer to put everything in the setup script (as I usually do), then you can define an "egg_info writer" entry point and a "setup keyword" entry point in a parent project (e.g. SQLObject itself), and then projects using SQLObject can do setup_requires="SQLObject" and then define the other metadata using keyword arguments. Most of the existing .egg-info files are written in exactly this way, including install_requires and entry_points. From robin at jessikat.fsnet.co.uk Sat Nov 5 13:02:24 2005 From: robin at jessikat.fsnet.co.uk (Robin Becker) Date: Sat, 05 Nov 2005 12:02:24 +0000 Subject: [Distutils] pysqlite egg problem Message-ID: <436C9F50.9000604@jessikat.fsnet.co.uk> I'm trying to follow the Turbogears wiki20, but I came across a problem with pysqlite. First off I am a complete newbie to the easy_setup and egg stuff and also have zero knowledge of pysqlite, but assuming that the filename I chose didn't have to exist I found that TG's tg-admin sql create failed. I assumed I needed to get some sort of database set up first so tried this to learn about pyslite C:\Tmp\wiki20>python Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from pysqlite2 import dbapi2 as sqlite Traceback (most recent call last): File "", line 1, in ? File "c:\python\lib\site-packages\pysqlite-2.0.6-py2.4-win32.egg\pysqlite2\dbapi2.py", line 32, in ? File "c:\python\lib\site-packages\pysqlite-2.0.6-py2.4-win32.egg\pysqlite2\_sqlite.py", line 7, in ? File "c:\python\lib\site-packages\pysqlite-2.0.6-py2.4-win32.egg\pysqlite2\_sqlite.py", line 6, in __bootstrap__ ImportError: DLL load failed: The specified module could not be found. >>> I checked in the egg and the dll is certainly there as lib/site-packages/pysqlite2/sqlite3.dll I have quite a few non egg packages installed so perhaps I have some wrong version of something on the path. What should I be looking for? -- Robin Becker From pje at telecommunity.com Sat Nov 5 17:28:54 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 05 Nov 2005 11:28:54 -0500 Subject: [Distutils] pysqlite egg problem In-Reply-To: <436C9F50.9000604@jessikat.fsnet.co.uk> Message-ID: <5.1.1.6.0.20051105110223.0201cd78@mail.telecommunity.com> At 12:02 PM 11/5/2005 +0000, Robin Becker wrote: >I'm trying to follow the Turbogears wiki20, but I came across a problem >with pysqlite. > >First off I am a complete newbie to the easy_setup and egg stuff and >also have zero knowledge of pysqlite, but assuming that the filename I >chose didn't have to exist I found that TG's tg-admin sql create failed. > >I assumed I needed to get some sort of database set up first so tried >this to learn about pyslite > >C:\Tmp\wiki20>python >Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] on >win32 >Type "help", "copyright", "credits" or "license" for more information. > >>> from pysqlite2 import dbapi2 as sqlite >Traceback (most recent call last): > File "", line 1, in ? > File >"c:\python\lib\site-packages\pysqlite-2.0.6-py2.4-win32.egg\pysqlite2\dbapi2.py", > >line 32, in ? > File >"c:\python\lib\site-packages\pysqlite-2.0.6-py2.4-win32.egg\pysqlite2\_sqlite.py", > >line 7, in ? > File >"c:\python\lib\site-packages\pysqlite-2.0.6-py2.4-win32.egg\pysqlite2\_sqlite.py", > >line 6, in __bootstrap__ >ImportError: DLL load failed: The specified module could not be found. > >>> > >I checked in the egg and the dll is certainly there as >lib/site-packages/pysqlite2/sqlite3.dll That's the problem: it should just be pysqlite2/sqlite3.dll; the lib/site-packages is extraneous. It appears that the way that pysqlite chooses to install the DLL is as a data file with an absolute path. Unfortunately, setuptools has limited ability to virtualize such paths, and the way pysqlite's setup script is setting that path is non-standard, as it's using backslash path separators rather than slashes. I'll have to look into whether there's some kind of workaround I can do. The other issue is that either the DLL needs to be listed in the egg's "eager resources", or else the whole egg needs to be marked as not zip-safe (i.e. it must be installed uncompressed). I suppose I should also look at automatically detecting .dll and .so files that are listed as data files, as that's what's happening here. (That is, the sqlite3.dll file is listed as a data file.) As a workaround for now, run this: easy_install -UZ pysqlite which will "upgrade" (reinstall) the egg in unzipped form. Then, move the sqlite3.dll file to the pysqlite2 subdirectory under the .egg directory. That should get you working for now. From david at handysoftware.com Tue Nov 8 16:21:36 2005 From: david at handysoftware.com (David Handy) Date: Tue, 8 Nov 2005 10:21:36 -0500 Subject: [Distutils] setuptools 0.6a8dev using non-existing shutil._build_cmdtuple function Message-ID: <20051108152136.GA1599@arno2> I got the following stack trace after installing setuptools 0.6a8dev and running "python setup.py install --home=%HOME%" on my project: Traceback (most recent call last): File "setup.py", line 42, in ? main() File "setup.py", line 37, in main license = "Internal Use Only", File "C:\Python24\lib\distutils\core.py", line 149, in setup dist.run_commands() File "C:\Python24\lib\distutils\dist.py", line 946, in run_commands self.run_command(cmd) File "C:\Python24\lib\distutils\dist.py", line 966, in run_command cmd_obj.run() File "c:\python24\lib\site-packages\setuptools-0.6a8dev_r41396-py2.4.egg\setuptools\command\install.py", line 50, in run cmd.run() File "c:\python24\lib\site-packages\setuptools-0.6a8dev_r41396-py2.4.egg\setuptools\command\easy_install.py", line 230, in run self.easy_install(spec, not self.no_deps) File "c:\python24\lib\site-packages\setuptools-0.6a8dev_r41396-py2.4.egg\setuptools\command\easy_install.py", line 321, in easy_install smart_rmtree(tmpdir) File "c:\python24\lib\site-packages\setuptools-0.6a8dev_r41396-py2.4.egg\setuptools\command\easy_install.py", line 1193, in smart_rmtree shutil._build_cmdtuple(path, cmdtuples) AttributeError: 'module' object has no attribute '_build_cmdtuple' -- David Handy Computer Programming is Fun! Beginning Computer Programming with Python http://www.handysoftware.com/cpif/ From pje at telecommunity.com Tue Nov 8 17:02:04 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 08 Nov 2005 11:02:04 -0500 Subject: [Distutils] setuptools 0.6a8dev using non-existing shutil._build_cmdtuple function In-Reply-To: <20051108152136.GA1599@arno2> Message-ID: <5.1.1.6.0.20051108110102.04be1df8@mail.telecommunity.com> At 10:21 AM 11/8/2005 -0500, David Handy wrote: >I got the following stack trace after installing setuptools 0.6a8dev and >running "python setup.py install --home=%HOME%" on my project: I found this problem a few days ago; update to 0.6a8dev-r41402 to get the fix. (The problem was that I was relying on shutil internals that changed in Python 2.4.) From david at handysoftware.com Tue Nov 8 21:10:31 2005 From: david at handysoftware.com (David Handy) Date: Tue, 8 Nov 2005 15:10:31 -0500 Subject: [Distutils] eggs end up at end of sys.path Message-ID: <20051108201031.GC1599@arno2> First, I want to say that I am very glad that setuptools/eggs are being developed - it is a great thing for Python. I have been lurking on this list for a while. The two features that convinced me it was time to start using setuptools were automatic generation of .exe starter programs from console_scripts entry points, and this feature: """ Activated distributions are now inserted in sys.path (and the working set) just before the directory that contains them, instead of at the end. This allows e.g. eggs in site-packages to override unmanaged modules in the same location, and allows eggs found earlier on sys.path to override ones found later. """ However, after I do a pkg_resources.require(), my egg always ends up at the end of sys.path. I exected the egg file to show up before the directory that contained it, in this case a non-site-packages directory in my PYTHONPATH. This is on setuptools versions 0.6a7 and 0.6a8dev-r41402. (Hopefully this is helpful feedback. If not let me know and I'll go back to lurking.) -- David Handy Computer Programming is Fun! Beginning Computer Programming with Python http://www.handysoftware.com/cpif/ From pje at telecommunity.com Tue Nov 8 22:11:31 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 08 Nov 2005 16:11:31 -0500 Subject: [Distutils] eggs end up at end of sys.path In-Reply-To: <20051108201031.GC1599@arno2> Message-ID: <5.1.1.6.0.20051108160611.01ee81b0@mail.telecommunity.com> At 03:10 PM 11/8/2005 -0500, David Handy wrote: >I have been lurking on this list for a while. The two features that >convinced me it was time to start using setuptools were automatic generation >of .exe starter programs from console_scripts entry points, and this >feature: > >""" >Activated distributions are now inserted in sys.path (and the working set) >just before the directory that contains them, instead of at the end. This >allows e.g. eggs in site-packages to override unmanaged modules in the same >location, and allows eggs found earlier on sys.path to override ones found >later. >""" > >However, after I do a pkg_resources.require(), my egg always ends up at the >end of sys.path. I exected the egg file to show up before the directory that >contained it, in this case a non-site-packages directory in my PYTHONPATH. >This is on setuptools versions 0.6a7 and 0.6a8dev-r41402. Could you provide me with your steps to reproduce, so I can try to find out what's wrong? One possibly important thing: the code that finds what sys.path entry contains the egg is currently case-sensitive and separator-sensitive. It's possible that if your PYTHONPATH uses lowercase and slash separators and the egg is the other way around, or vice versa, the container checking may be failing. You can verify this by printing sys.path after the require() and see if the paths are the same. From david at handysoftware.com Wed Nov 9 03:41:48 2005 From: david at handysoftware.com (David Handy) Date: Tue, 8 Nov 2005 21:41:48 -0500 Subject: [Distutils] eggs end up at end of sys.path In-Reply-To: <5.1.1.6.0.20051108160611.01ee81b0@mail.telecommunity.com> References: <20051108201031.GC1599@arno2> <5.1.1.6.0.20051108160611.01ee81b0@mail.telecommunity.com> Message-ID: <20051109024148.GA4612@arno2> On Tue, Nov 08, 2005 at 04:11:31PM -0500, Phillip J. Eby wrote: > At 03:10 PM 11/8/2005 -0500, David Handy wrote: > >""" > >Activated distributions are now inserted in sys.path (and the working set) > >just before the directory that contains them, instead of at the end. This > >allows e.g. eggs in site-packages to override unmanaged modules in the same > >location, and allows eggs found earlier on sys.path to override ones found > >later. > >""" > > > >However, after I do a pkg_resources.require(), my egg always ends up at the > >end of sys.path. I exected the egg file to show up before the directory > >that > >contained it, in this case a non-site-packages directory in my PYTHONPATH. > >This is on setuptools versions 0.6a7 and 0.6a8dev-r41402. > > Could you provide me with your steps to reproduce, so I can try to find out > what's wrong? > > One possibly important thing: the code that finds what sys.path entry > contains the egg is currently case-sensitive and separator-sensitive. It's > possible that if your PYTHONPATH uses lowercase and slash separators and > the egg is the other way around, or vice versa, the container checking may > be failing. You can verify this by printing sys.path after the require() > and see if the paths are the same. I created a fairly minimal project which I have attached as MyProject-0.1.zip which demonstrates the problem. On Linux, it puts the egg in the proper place in sys.path (ahead of the PYTHONPATH directory), but on Windows it puts the egg file on the end. See the README.txt file for example output. (I have seen this problem on both Win98 and WinXP.) -- David Handy Computer Programming is Fun! Beginning Computer Programming with Python http://www.handysoftware.com/cpif/ -------------- next part -------------- A non-text attachment was scrubbed... Name: MyProject-0.1.zip Type: application/zip Size: 5105 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20051108/7f955906/MyProject-0.1.zip From pje at telecommunity.com Wed Nov 9 04:04:19 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 08 Nov 2005 22:04:19 -0500 Subject: [Distutils] eggs end up at end of sys.path In-Reply-To: <20051109024148.GA4612@arno2> References: <5.1.1.6.0.20051108160611.01ee81b0@mail.telecommunity.com> <20051108201031.GC1599@arno2> <5.1.1.6.0.20051108160611.01ee81b0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051108215745.01f2a658@mail.telecommunity.com> At 09:41 PM 11/8/2005 -0500, David Handy wrote: >I created a fairly minimal project which I have attached as >MyProject-0.1.zip which demonstrates the problem. On Linux, it puts the egg >in the proper place in sys.path (ahead of the PYTHONPATH directory), but on >Windows it puts the egg file on the end. See the README.txt file for >example output. (I have seen this problem on both Win98 and WinXP.) Based on the contents of README.txt, it's clearly a case-sensitivity issue, as 'c:\\home\\lib\\python\\myproject-0.1-py2.4.egg' doesn't start with 'C:\\home\\lib\\python' (lowercase vs. uppercase C:). I believe the correct fix therefore is for Distribution.insert_on() to normalize the sys.path entries before comparison. Please give revision 41408 a try. ("easy_install setuptools==dev" should upgrade you.) From ashleywalsh at gmail.com Wed Nov 9 04:55:44 2005 From: ashleywalsh at gmail.com (Ashley Walsh) Date: Wed, 9 Nov 2005 13:55:44 +1000 Subject: [Distutils] finding my own version with setuptools Message-ID: Hi, Is there an easy way for a module to uses its __name__ to identify the Distribution it comes from? I had a look through pkg_resources but I couldn't see a straight forward way to do it, so I thought I'd ask. The reason is that I want a console script log its version. Cheers, Ashley From pje at telecommunity.com Wed Nov 9 05:08:49 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 08 Nov 2005 23:08:49 -0500 Subject: [Distutils] finding my own version with setuptools In-Reply-To: Message-ID: <5.1.1.6.0.20051108230145.01f3ea48@mail.telecommunity.com> At 01:55 PM 11/9/2005 +1000, Ashley Walsh wrote: >Hi, > >Is there an easy way for a module to uses its __name__ to identify >the Distribution it comes from? I had a look through pkg_resources >but I couldn't see a straight forward way to do it, so I thought I'd >ask. > >The reason is that I want a console script log its version. Well, if you know the *project* name, you can just do: pkg_resources.get_distribution("ProjectName").version to look up the distribution for "ProjectName" and get its version. However, if you don't know the project name, then figuring out the distribution is a lot more work. However, since it's presumably you writing the code, you should probably be aware of what your name for the project is. :) From ashleywalsh at gmail.com Wed Nov 9 05:16:59 2005 From: ashleywalsh at gmail.com (Ashley Walsh) Date: Wed, 9 Nov 2005 14:16:59 +1000 Subject: [Distutils] finding my own version with setuptools In-Reply-To: <5.1.1.6.0.20051108230145.01f3ea48@mail.telecommunity.com> References: <5.1.1.6.0.20051108230145.01f3ea48@mail.telecommunity.com> Message-ID: <0812244E-E02E-40CF-9F37-6D9CEA6A39FC@gmail.com> Thats what I ended up doing. Thanks, Ashley On 09/11/2005, at 2:08 PM, Phillip J. Eby wrote: > At 01:55 PM 11/9/2005 +1000, Ashley Walsh wrote: >> Hi, >> >> Is there an easy way for a module to uses its __name__ to identify >> the Distribution it comes from? I had a look through pkg_resources >> but I couldn't see a straight forward way to do it, so I thought I'd >> ask. >> >> The reason is that I want a console script log its version. > > Well, if you know the *project* name, you can just do: > > pkg_resources.get_distribution("ProjectName").version > > to look up the distribution for "ProjectName" and get its version. > > However, if you don't know the project name, then figuring out the > distribution is a lot more work. However, since it's presumably > you writing the code, you should probably be aware of what your > name for the project is. :) > From david at handysoftware.com Wed Nov 9 05:27:57 2005 From: david at handysoftware.com (David Handy) Date: Tue, 8 Nov 2005 23:27:57 -0500 Subject: [Distutils] eggs end up at end of sys.path In-Reply-To: <5.1.1.6.0.20051108215745.01f2a658@mail.telecommunity.com> References: <5.1.1.6.0.20051108160611.01ee81b0@mail.telecommunity.com> <20051108201031.GC1599@arno2> <5.1.1.6.0.20051108160611.01ee81b0@mail.telecommunity.com> <5.1.1.6.0.20051108215745.01f2a658@mail.telecommunity.com> Message-ID: <20051109042757.GA4843@arno2> On Tue, Nov 08, 2005 at 10:04:19PM -0500, Phillip J. Eby wrote: > At 09:41 PM 11/8/2005 -0500, David Handy wrote: > >I created a fairly minimal project which I have attached as > >MyProject-0.1.zip which demonstrates the problem. On Linux, it puts the egg > >in the proper place in sys.path (ahead of the PYTHONPATH directory), but on > >Windows it puts the egg file on the end. See the README.txt file for > >example output. (I have seen this problem on both Win98 and WinXP.) > > Based on the contents of README.txt, it's clearly a case-sensitivity issue, > as 'c:\\home\\lib\\python\\myproject-0.1-py2.4.egg' doesn't start with > 'C:\\home\\lib\\python' (lowercase vs. uppercase C:). > > I believe the correct fix therefore is for Distribution.insert_on() to > normalize the sys.path entries before comparison. Please give revision > 41408 a try. ("easy_install setuptools==dev" should upgrade you.) I ended up grabbing revision 41411. It worked great, thanks! Now the egg file appears before its containing directory, on both Linux and Windows. I'm going to have a lot of fun with setuptools; it meets some packaging/deployment needs that I have right now. Thanks again, -- David Handy Computer Programming is Fun! Beginning Computer Programming with Python http://www.handysoftware.com/cpif/ From ianb at colorstudy.com Wed Nov 9 17:29:44 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 09 Nov 2005 10:29:44 -0600 Subject: [Distutils] finding my own version with setuptools In-Reply-To: <5.1.1.6.0.20051108230145.01f3ea48@mail.telecommunity.com> References: <5.1.1.6.0.20051108230145.01f3ea48@mail.telecommunity.com> Message-ID: <437223F8.8050403@colorstudy.com> Phillip J. Eby wrote: > At 01:55 PM 11/9/2005 +1000, Ashley Walsh wrote: > >>Hi, >> >>Is there an easy way for a module to uses its __name__ to identify >>the Distribution it comes from? I had a look through pkg_resources >>but I couldn't see a straight forward way to do it, so I thought I'd >>ask. >> >>The reason is that I want a console script log its version. > > > Well, if you know the *project* name, you can just do: > > pkg_resources.get_distribution("ProjectName").version Incidentally, I use ('%s from %s' % (dist, dist.location)) for my version strings, as I find it useful to know where on disk it is coming from, in addition to what version. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ashleywalsh at gmail.com Thu Nov 10 01:03:29 2005 From: ashleywalsh at gmail.com (Ashley Walsh) Date: Thu, 10 Nov 2005 10:03:29 +1000 Subject: [Distutils] finding my own version with setuptools In-Reply-To: <437223F8.8050403@colorstudy.com> References: <5.1.1.6.0.20051108230145.01f3ea48@mail.telecommunity.com> <437223F8.8050403@colorstudy.com> Message-ID: <1F9E42DC-2EF9-4F04-A2C2-9F0077048C64@gmail.com> On 10/11/2005, at 2:29 AM, Ian Bicking wrote: > Phillip J. Eby wrote: >> At 01:55 PM 11/9/2005 +1000, Ashley Walsh wrote: >>> Hi, >>> >>> Is there an easy way for a module to uses its __name__ to identify >>> the Distribution it comes from? I had a look through pkg_resources >>> but I couldn't see a straight forward way to do it, so I thought I'd >>> ask. >>> >>> The reason is that I want a console script log its version. >> Well, if you know the *project* name, you can just do: >> pkg_resources.get_distribution("ProjectName").version > > Incidentally, I use ('%s from %s' % (dist, dist.location)) for my > version strings, as I find it useful to know where on disk it is > coming from, in addition to what version. > > > -- > Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org Yup, I'm also logging repr(dist) ('setuptools 0.6a7 (/Library/Python/ 2.3/site-packages/setuptools-0.6a7-py2.3.egg)' ) of both the Environment and the working_set (via add_activation_listener). Ashley From ben at groovie.org Thu Nov 10 19:15:13 2005 From: ben at groovie.org (Ben Bangert) Date: Thu, 10 Nov 2005 10:15:13 -0800 Subject: [Distutils] Distutils has no module named util, or is it a package problem? Message-ID: I'm getting the following very odd behavior when downloading and installing from subversion for Paste lately. The error makes it tough to know if this is a Paste problem, a local problem (I've had it on two different OSX machines now), or a setuptools problem. To replicate, sudo easy_install -U -D http://svn.pythonpaste.org/Paste/trunk/ Downloading http://svn.pythonpaste.org/Paste/trunk/ Doing subversion checkout from http://svn.pythonpaste.org/Paste/ trunk/ to /tmp/easy_install-SIB1qx/trunk Processing trunk Running setup.py -q bdist_egg --dist-dir /tmp/easy_install-SIB1qx/ trunk/egg-dist-tmp-m-NhjS Traceback (most recent call last): File "/usr/local/bin/easy_install", line 7, in ? sys.exit( File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a7-py2.4.egg/setuptools/command/ easy_install.py", line 1181, in main setup(script_args = ['-q','easy_install', '-v']+argv, **kw) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/core.py", line 149, in setup dist.run_commands() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a7-py2.4.egg/setuptools/command/ easy_install.py", line 230, in run self.easy_install(spec, not self.no_deps) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a7-py2.4.egg/setuptools/command/ easy_install.py", line 317, in easy_install return self.install_item(spec, download, tmpdir, deps) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a7-py2.4.egg/setuptools/command/ easy_install.py", line 343, in install_item self.process_distribution(spec, dists[0], deps, "Using") File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a7-py2.4.egg/setuptools/command/ easy_install.py", line 397, in process_distribution WorkingSet([]).resolve( File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a7-py2.4.egg/pkg_resources.py", line 481, in resolve dist = best[req.key] = env.best_match(req, self, installer) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a7-py2.4.egg/pkg_resources.py", line 635, in best_match return self.obtain(req, installer) # try and download/install File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a7-py2.4.egg/pkg_resources.py", line 647, in obtain return installer(requirement) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a7-py2.4.egg/setuptools/command/ easy_install.py", line 317, in easy_install return self.install_item(spec, download, tmpdir, deps) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a7-py2.4.egg/setuptools/command/ easy_install.py", line 338, in install_item dists = self.install_eggs(spec, download, tmpdir) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a7-py2.4.egg/setuptools/command/ easy_install.py", line 573, in install_eggs return self.build_and_install(setup_script, setup_base) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a7-py2.4.egg/setuptools/command/ easy_install.py", line 848, in build_and_install self.run_setup(setup_script, setup_base, args) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a7-py2.4.egg/setuptools/command/ easy_install.py", line 837, in run_setup run_setup(setup_script, args) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a7-py2.4.egg/setuptools/ sandbox.py", line 26, in run_setup DirectorySandbox(setup_dir).run( File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a7-py2.4.egg/setuptools/ sandbox.py", line 63, in run return func() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a7-py2.4.egg/setuptools/ sandbox.py", line 29, in {'__file__':setup_script, '__name__':'__main__'} File "setup.py", line 12, in ? ImportError: No module named util Thanks, Ben From pje at telecommunity.com Thu Nov 10 20:01:05 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 10 Nov 2005 14:01:05 -0500 Subject: [Distutils] Distutils has no module named util, or is it a package problem? In-Reply-To: Message-ID: <5.1.1.6.0.20051110135913.01f56da0@mail.telecommunity.com> At 10:15 AM 11/10/2005 -0800, Ben Bangert wrote: >I'm getting the following very odd behavior when downloading and >installing from subversion for Paste lately. The error makes it tough >to know if this is a Paste problem, a local problem (I've had it on >two different OSX machines now), or a setuptools problem. Check whether there is a distutils/util.py in your /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ directory. From ianb at colorstudy.com Thu Nov 10 22:27:49 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 10 Nov 2005 15:27:49 -0600 Subject: [Distutils] Distutils has no module named util, or is it a package problem? In-Reply-To: <5.1.1.6.0.20051110135913.01f56da0@mail.telecommunity.com> References: <5.1.1.6.0.20051110135913.01f56da0@mail.telecommunity.com> Message-ID: <4373BB55.4070609@colorstudy.com> Phillip J. Eby wrote: > At 10:15 AM 11/10/2005 -0800, Ben Bangert wrote: > >>I'm getting the following very odd behavior when downloading and >>installing from subversion for Paste lately. The error makes it tough >>to know if this is a Paste problem, a local problem (I've had it on >>two different OSX machines now), or a setuptools problem. > > > Check whether there is a distutils/util.py in your > /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ directory. I realize now what the problem is. I included this in the setup.py: import sys, os sys.path.insert(0, os.path.dirname(__file__)) from paste.util import finddata This would actually work... except it doesn't work when doing the sandbox install, and the exception comes during the sandbox phase. I'm not sure why a sandbox installation is happening (zip_safe is explicitly set to False)? Here's the end of the traceback: File "/usr/home/ianb/co/setuptools/setuptools/command/easy_install.py", line 837, in run_setup run_setup(setup_script, args) File "/usr/home/ianb/co/setuptools/setuptools/sandbox.py", line 27, in run_setup DirectorySandbox(setup_dir).run( File "/usr/home/ianb/co/setuptools/setuptools/sandbox.py", line 64, in run return func() File "/usr/home/ianb/co/setuptools/setuptools/sandbox.py", line 30, in {'__file__':setup_script, '__name__':'__main__'} File "/home/ianb/src/py-dist/setup.py", line 12, in ? ImportError: No module named util -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Thu Nov 10 22:59:47 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 10 Nov 2005 16:59:47 -0500 Subject: [Distutils] Distutils has no module named util, or is it a package problem? In-Reply-To: <4373BB55.4070609@colorstudy.com> References: <5.1.1.6.0.20051110135913.01f56da0@mail.telecommunity.com> <5.1.1.6.0.20051110135913.01f56da0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051110165631.01f5f040@mail.telecommunity.com> At 03:27 PM 11/10/2005 -0600, Ian Bicking wrote: >Phillip J. Eby wrote: >>At 10:15 AM 11/10/2005 -0800, Ben Bangert wrote: >> >>>I'm getting the following very odd behavior when downloading and >>>installing from subversion for Paste lately. The error makes it tough >>>to know if this is a Paste problem, a local problem (I've had it on >>>two different OSX machines now), or a setuptools problem. >> >>Check whether there is a distutils/util.py in your >>/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ directory. > >I realize now what the problem is. I included this in the setup.py: > >import sys, os >sys.path.insert(0, os.path.dirname(__file__)) >from paste.util import finddata > >This would actually work... I don't understand what you're trying to do; the setup script's directory should already *be* on sys.path. >except it doesn't work when doing the sandbox install, and the exception >comes during the sandbox phase. I'm not sure why a sandbox installation >is happening (zip_safe is explicitly set to False)? easy_install runs all setup scripts in a sandbox, since it has no way of knowing whether a given script is "safe" until it actually runs it in the sandbox. Easier to ask forgiveness than permission, you might say. :) From ianb at colorstudy.com Thu Nov 10 23:05:51 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 10 Nov 2005 16:05:51 -0600 Subject: [Distutils] Distutils has no module named util, or is it a package problem? In-Reply-To: <5.1.1.6.0.20051110165631.01f5f040@mail.telecommunity.com> References: <5.1.1.6.0.20051110135913.01f56da0@mail.telecommunity.com> <5.1.1.6.0.20051110135913.01f56da0@mail.telecommunity.com> <5.1.1.6.0.20051110165631.01f5f040@mail.telecommunity.com> Message-ID: <4373C43F.5000106@colorstudy.com> Phillip J. Eby wrote: > At 03:27 PM 11/10/2005 -0600, Ian Bicking wrote: > >> Phillip J. Eby wrote: >> >>> At 10:15 AM 11/10/2005 -0800, Ben Bangert wrote: >>> >>>> I'm getting the following very odd behavior when downloading and >>>> installing from subversion for Paste lately. The error makes it tough >>>> to know if this is a Paste problem, a local problem (I've had it on >>>> two different OSX machines now), or a setuptools problem. >>> >>> >>> Check whether there is a distutils/util.py in your >>> /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ >>> directory. >> >> >> I realize now what the problem is. I included this in the setup.py: >> >> import sys, os >> sys.path.insert(0, os.path.dirname(__file__)) >> from paste.util import finddata >> >> This would actually work... > > > I don't understand what you're trying to do; the setup script's > directory should already *be* on sys.path. I wasn't sure, so I did it to be safe. It shouldn't effect anything. >> except it doesn't work when doing the sandbox install, and the >> exception comes during the sandbox phase. I'm not sure why a sandbox >> installation is happening (zip_safe is explicitly set to False)? > > > easy_install runs all setup scripts in a sandbox, since it has no way of > knowing whether a given script is "safe" until it actually runs it in > the sandbox. Easier to ask forgiveness than permission, you might say. :) But apparently that import doesn't work when in the sandbox. Why test it when setup() says explicitly that it isn't safe? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ben at groovie.org Thu Nov 10 23:20:58 2005 From: ben at groovie.org (Ben Bangert) Date: Thu, 10 Nov 2005 14:20:58 -0800 Subject: [Distutils] Distutils has no module named util, or is it a package problem? In-Reply-To: <4373C43F.5000106@colorstudy.com> References: <5.1.1.6.0.20051110135913.01f56da0@mail.telecommunity.com> <5.1.1.6.0.20051110135913.01f56da0@mail.telecommunity.com> <5.1.1.6.0.20051110165631.01f5f040@mail.telecommunity.com> <4373C43F.5000106@colorstudy.com> Message-ID: On Nov 10, 2005, at 2:05 PM, Ian Bicking wrote: > I wasn't sure, so I did it to be safe. It shouldn't effect anything. > >>> except it doesn't work when doing the sandbox install, and the >>> exception comes during the sandbox phase. I'm not sure why a >>> sandbox installation is happening (zip_safe is explicitly set to >>> False)? >> easy_install runs all setup scripts in a sandbox, since it has no >> way of knowing whether a given script is "safe" until it actually >> runs it in the sandbox. Easier to ask forgiveness than >> permission, you might say. :) > > But apparently that import doesn't work when in the sandbox. Why > test it when setup() says explicitly that it isn't safe? Even if it doesn't 'test' it, it would still need a sandbox to stick the svn checkout in while it builds and installs it, no? (Not that I have any clue how setuptools works, but this is just a guess) - Ben From ianb at colorstudy.com Thu Nov 10 23:23:55 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 10 Nov 2005 16:23:55 -0600 Subject: [Distutils] Distutils has no module named util, or is it a package problem? In-Reply-To: References: <5.1.1.6.0.20051110135913.01f56da0@mail.telecommunity.com> <5.1.1.6.0.20051110135913.01f56da0@mail.telecommunity.com> <5.1.1.6.0.20051110165631.01f5f040@mail.telecommunity.com> <4373C43F.5000106@colorstudy.com> Message-ID: <4373C87B.3000102@colorstudy.com> Ben Bangert wrote: > On Nov 10, 2005, at 2:05 PM, Ian Bicking wrote: > >> I wasn't sure, so I did it to be safe. It shouldn't effect anything. >> >>>> except it doesn't work when doing the sandbox install, and the >>>> exception comes during the sandbox phase. I'm not sure why a >>>> sandbox installation is happening (zip_safe is explicitly set to >>>> False)? >>> >>> easy_install runs all setup scripts in a sandbox, since it has no >>> way of knowing whether a given script is "safe" until it actually >>> runs it in the sandbox. Easier to ask forgiveness than permission, >>> you might say. :) >> >> >> But apparently that import doesn't work when in the sandbox. Why >> test it when setup() says explicitly that it isn't safe? > > > Even if it doesn't 'test' it, it would still need a sandbox to stick > the svn checkout in while it builds and installs it, no? (Not that I > have any clue how setuptools works, but this is just a guess) The sandbox in this case is when setuptools runs setup.py with fake file routines, to see if the setup.py file writes things to weird locations. Now that I think about it, this isn't for zip-safe testing, but to test if setuptools can properly wrap this. I can understand why to do this when running a distutils setup.py file, but could this be surpressed for a setup.py file which imports from setuptools? I think it should be presumed that it is safe in that case. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ben at groovie.org Thu Nov 10 23:52:21 2005 From: ben at groovie.org (Ben Bangert) Date: Thu, 10 Nov 2005 14:52:21 -0800 Subject: [Distutils] setuptools using -D on package requirements? Message-ID: <5FEAEE5A-CDC7-42E5-A019-12D269FC3880@groovie.org> Is there a reason I can't have setuptools automatically upgrade/ delete existing/conflicting package versions that are a 'requirement' for the package I want to install? For example, if I have the 0.3 of PasteScript installed, and run my own package which requires (and has the URL for) the svn dev version of PasteScript, here's what I get: % sudo python ez_setup.py -D -f http://pylons.groovie.org/ Pylons Reading http://pylons.groovie.org/ Searching for Pylons Best match: Pylons 0.1dev-r87 Processing Pylons-0.1dev_r87-py2.4.egg Pylons 0.1dev-r87 is already the active version in easy-install.pth Using /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/Pylons-0.1dev_r87-py2.4.egg Processing dependencies for Pylons error: Installed distribution Paste 0.3 conflicts with requirement Paste>=0.4dev-r362 I've tried -U as well, and there doesn't seem to be anyway to tell setuptools to do this. Perhaps a 'Upgrade requirements if necessary' option or something? Thanks, Ben From pje at telecommunity.com Fri Nov 11 00:05:44 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 10 Nov 2005 18:05:44 -0500 Subject: [Distutils] Distutils has no module named util, or is it a package problem? In-Reply-To: <4373C87B.3000102@colorstudy.com> References: <5.1.1.6.0.20051110135913.01f56da0@mail.telecommunity.com> <5.1.1.6.0.20051110135913.01f56da0@mail.telecommunity.com> <5.1.1.6.0.20051110165631.01f5f040@mail.telecommunity.com> <4373C43F.5000106@colorstudy.com> Message-ID: <5.1.1.6.0.20051110180116.01f5e220@mail.telecommunity.com> At 04:23 PM 11/10/2005 -0600, Ian Bicking wrote: >The sandbox in this case is when setuptools runs setup.py with fake file >routines, to see if the setup.py file writes things to weird >locations. Now that I think about it, this isn't for zip-safe testing, >but to test if setuptools can properly wrap this. > >I can understand why to do this when running a distutils setup.py file, >but could this be surpressed for a setup.py file which imports from >setuptools? I think it should be presumed that it is safe in that case. This is a red herring. Just because a package uses setuptools, doesn't mean it's safe. The author might have simply taken an older script and changed it to import setuptools. That doesn't fix any issues like custom data installation commands, or code in the body of setup.py does any installation. So, it doesn't make any sense to scan setup.py for setuptools and then run it without sandboxing, and in any case I doubt the sandbox has anything to do with this problem. A simple test should suffice: change the setuptools.sandbox to replace the DirectorySandbox().run() call with a straight execfile. If the problem remains, then it's not the sandboxing that's the problem. From ianb at colorstudy.com Fri Nov 11 00:17:09 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 10 Nov 2005 17:17:09 -0600 Subject: [Distutils] Distutils has no module named util, or is it a package problem? In-Reply-To: <5.1.1.6.0.20051110180116.01f5e220@mail.telecommunity.com> References: <5.1.1.6.0.20051110135913.01f56da0@mail.telecommunity.com> <5.1.1.6.0.20051110135913.01f56da0@mail.telecommunity.com> <5.1.1.6.0.20051110165631.01f5f040@mail.telecommunity.com> <4373C43F.5000106@colorstudy.com> <5.1.1.6.0.20051110180116.01f5e220@mail.telecommunity.com> Message-ID: <4373D4F5.1040708@colorstudy.com> Phillip J. Eby wrote: > At 04:23 PM 11/10/2005 -0600, Ian Bicking wrote: > >> The sandbox in this case is when setuptools runs setup.py with fake >> file routines, to see if the setup.py file writes things to weird >> locations. Now that I think about it, this isn't for zip-safe >> testing, but to test if setuptools can properly wrap this. >> >> I can understand why to do this when running a distutils setup.py >> file, but could this be surpressed for a setup.py file which imports >> from setuptools? I think it should be presumed that it is safe in >> that case. > > > This is a red herring. Just because a package uses setuptools, doesn't > mean it's safe. The author might have simply taken an older script and > changed it to import setuptools. That doesn't fix any issues like > custom data installation commands, or code in the body of setup.py does > any installation. Yes, it is a red herring. After putting some print statements in my setup.py file, I realized that the problem is namespace packages. When I "import paste" it is importing another namespace package (PasteWebKit, but I don't know why that one specifically). I'm guessing the module is loaded because it is an egg, and provides an entry point, and entry points are being scanned. Or maybe just because it is a namespace package, and I don't understand how they work. Well, I do understand that they cause me constant problems, and it seems like namespace packages that aren't installed multi-version are highly problematic. I'm not sure exactly how to do this, except maybe to put paste/util on the path itself. I suppose that would work well enough. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Fri Nov 11 00:26:34 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 10 Nov 2005 18:26:34 -0500 Subject: [Distutils] Distutils has no module named util, or is it a package problem? In-Reply-To: <4373D4F5.1040708@colorstudy.com> References: <5.1.1.6.0.20051110180116.01f5e220@mail.telecommunity.com> <5.1.1.6.0.20051110135913.01f56da0@mail.telecommunity.com> <5.1.1.6.0.20051110135913.01f56da0@mail.telecommunity.com> <5.1.1.6.0.20051110165631.01f5f040@mail.telecommunity.com> <4373C43F.5000106@colorstudy.com> <5.1.1.6.0.20051110180116.01f5e220@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051110182253.01f7f250@mail.telecommunity.com> At 05:17 PM 11/10/2005 -0600, Ian Bicking wrote: >Or maybe just because it is a namespace package, and I don't understand >how they work. Well, I do understand that they cause me constant >problems, and it seems like namespace packages that aren't installed >multi-version are highly problematic. I personally try to never import a package from within the package's setup script. It's way too likely to have weird results, even without namespace packages being involved. However, if you want to make the package being installed become part of the appropriate namespace package(s), you can use: pkg_resources.fixup_namespace_packages(source_dir) in order to tell pkg_resources that source_dir may contain package snippets relevant to already-declared namespace packages. See pkg_resources.txt for details. From ianb at colorstudy.com Fri Nov 11 00:29:59 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 10 Nov 2005 17:29:59 -0600 Subject: [Distutils] Distutils has no module named util, or is it a package problem? In-Reply-To: <5.1.1.6.0.20051110182253.01f7f250@mail.telecommunity.com> References: <5.1.1.6.0.20051110180116.01f5e220@mail.telecommunity.com> <5.1.1.6.0.20051110135913.01f56da0@mail.telecommunity.com> <5.1.1.6.0.20051110135913.01f56da0@mail.telecommunity.com> <5.1.1.6.0.20051110165631.01f5f040@mail.telecommunity.com> <4373C43F.5000106@colorstudy.com> <5.1.1.6.0.20051110180116.01f5e220@mail.telecommunity.com> <5.1.1.6.0.20051110182253.01f7f250@mail.telecommunity.com> Message-ID: <4373D7F7.9040504@colorstudy.com> Phillip J. Eby wrote: > At 05:17 PM 11/10/2005 -0600, Ian Bicking wrote: > >> Or maybe just because it is a namespace package, and I don't >> understand how they work. Well, I do understand that they cause me >> constant problems, and it seems like namespace packages that aren't >> installed multi-version are highly problematic. > > > I personally try to never import a package from within the package's > setup script. It's way too likely to have weird results, even without > namespace packages being involved. > > However, if you want to make the package being installed become part of > the appropriate namespace package(s), you can use: > > pkg_resources.fixup_namespace_packages(source_dir) > > in order to tell pkg_resources that source_dir may contain package > snippets relevant to already-declared namespace packages. See > pkg_resources.txt for details. I added paste/util to sys.path, and just run "import finddata", and that works fine. It's expedient and doesn't depend on anything too fancy at the module-level. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From Richard at artsalliancemedia.com Fri Nov 11 17:51:27 2005 From: Richard at artsalliancemedia.com (Richard Cooper) Date: Fri, 11 Nov 2005 16:51:27 -0000 Subject: [Distutils] __init__.py files missing from my eggs Message-ID: Hi, I'm developing an application and a number of plugins for that app. The app is frozen using py2exe and I want to distribute the plugins as eggs. Unfortunately I'm having some difficulty. Either I'm not building the egg correctly or I'm not importing it correctly (or possibly both). I'm sure I'm doing something stupid, so if someone could hit me with the cluestick I would appreciate it. My package structure looks roughly like this (All of the directories contain __init__.py files which I've omitted): My/ Project/ app/ setup.py (Builds the app) ... plugin/ setup.py (Builds all the plugins) Plugin1/ ... Plugin2/ ... Plugin3/ ... In the plugin's setup.py I'm doing the following for each plugin: setup( name = pluginName, packages = ['My.Project.plugin.'+pluginName], namespace_packages = ['My.Project.plugin'], package_dir = {'':'../../..'}, package_data = ... ) This is producing an egg file containing the following structure: My/ Project/ plugin/ Plugin1/ ... Which is what I expected except that there is no __init__.py(c|o)? files in "My" or "Project" or "plugin". It looks like if I want to use the generated egg as a plugin then the __init__ files are required. In my main app I'm calling: pkg_resources.working_set.add_entry(r'path\to\my\Plugin1.egg') x = pkg_resources.resource_string('My.Project.plugin.Plugin1', someResource) # Where someResource exists in Plugin1.egg. Which results in the following error: File "c:\progra~1\python23\lib\site-packages\setuptools-0.6a7-py2.3.egg\pkg_r esources.py", line 688, in resource_string return get_provider(package_or_requirement).get_resource_string( File "c:\progra~1\python23\lib\site-packages\setuptools-0.6a7-py2.3.egg\pkg_r esources.py", line 119, in get_provider __import__(moduleOrReq) ImportError: No module named Plugin1 If I manually edit Plugin1.egg to include all the __init__.py files then the same code in the main app works as expected. Does anyone know where I'm going wrong? How are other people creating namespace packages? Thanks, Richard From pje at telecommunity.com Fri Nov 11 18:16:45 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 11 Nov 2005 12:16:45 -0500 Subject: [Distutils] __init__.py files missing from my eggs In-Reply-To: Message-ID: <5.1.1.6.0.20051111120809.02008590@mail.telecommunity.com> At 04:51 PM 11/11/2005 +0000, Richard Cooper wrote: >In the plugin's setup.py I'm doing the following for each plugin: > >setup( > name = pluginName, > packages = ['My.Project.plugin.'+pluginName], > namespace_packages = ['My.Project.plugin'], > package_dir = {'':'../../..'}, > package_data = ... >) > >This is producing an egg file containing the following structure: >My/ > Project/ > plugin/ > Plugin1/ > ... > >Which is what I expected except that there is no __init__.py(c|o)? files >in "My" or "Project" or "plugin". I would guess that your problem is that you need to do: packages = [ 'My', 'My.Project', 'My.Project.plugin', 'MyProject.plugin.'+pluginName ] Because it is otherwise not including those packages (and their __init__.py files). On an unrelated note, I would caution you that your unconventional project layout is likely to lead to other problems, such as an inability to create usable source distributions. Neither setuptools nor the distutils are designed to work with setup.py in any directory other than the distribution root of the project. What you probably want is a layout like this: app/ setup.py My/ Project/ app Plugin1/ setup.py My/ Project/ plugin1/ etc. Then, each project can have a source distribution built, be managed by easy_install, etc. Each project can also use 'find_packages()' to get the list of packages to install, without having to manually list them or the package directory names. If you are working on these projects at the same time, you need only run "setup.py develop" in each one to set up a sys.path structure that merges all the packages into one. Note, by the way, that you don't need namespace packages to have plugins for an application. Plugins can be in any package structure you like, as eggs' "entry points" system can be used by plugins to advertise the functions they provide, and the importing can be done automatically for you. From Richard at artsalliancemedia.com Fri Nov 11 19:31:52 2005 From: Richard at artsalliancemedia.com (Richard Cooper) Date: Fri, 11 Nov 2005 18:31:52 -0000 Subject: [Distutils] __init__.py files missing from my eggs Message-ID: From: Phillip J. Eby [mailto:pje at telecommunity.com] > At 04:51 PM 11/11/2005 +0000, Richard Cooper wrote: > >In the plugin's setup.py I'm doing the following for each plugin: > > > >setup( > > name = pluginName, > > packages = ['My.Project.plugin.'+pluginName], > > namespace_packages = ['My.Project.plugin'], > > package_dir = {'':'../../..'}, > > package_data = ... > >) > I would guess that your problem is that you need to do: > > packages = [ > 'My', 'My.Project', 'My.Project.plugin', > 'MyProject.plugin.'+pluginName > ] > > Because it is otherwise not including those packages (and > their __init__.py files). That pulls in too much. There are files in those packages other than the __init__ files which I don't want included in the plugin. > On an unrelated note, I would caution you that your > unconventional project layout is likely to lead to other > problems, such as an inability to create usable source > distributions. Neither setuptools nor the distutils are > designed to work with setup.py in any directory other than > the distribution root of the project. Yeah. I thought that might be part of the problem. I've had no other problems so far though. > What you probably want is a layout like this: > > app/ > setup.py > My/ > Project/ > app > Plugin1/ > setup.py > My/ > Project/ > plugin1/ > > etc. My knee-jerk reaction is 'Ewww!' Is this what PEAK (or ant other large setuptools based project) does? How do you handle My.Project.utils (for example) and the half dozen or so other packages at different levels of the hierarchy which are used in multiple "products"? > Then, each project can have a source distribution built, be > managed by easy_install, etc. Each project can also use > 'find_packages()' to get the list of packages to install, > without having to manually list them or the package directory > names. My needs are a bit specialised, this is a commercial app so source distribution, easy_install, etc will never be on the cards. Distribution is strictly "frozen" products (py2exe and eggs) only. > If you are working on these projects at the same > time, you need only run "setup.py develop" in each one to set > up a sys.path structure that merges all the packages into one. We would probably end up with 20 or so "projects" if we did things that way as opposed to the single code tree we have now. Given that, "setup.py develop"*20 seems like more work than our current approach which is "add the root of the code tree to PYTHONPATH" > Note, by the way, that you don't need namespace packages to > have plugins for an application. Plugins can be in any > package structure you like, as eggs' "entry points" system > can be used by plugins to advertise the functions they > provide, and the importing can be done automatically for you. Yeah I know. The entry point system is the main reason I started playing with setuptools. However I still wanted the plugins we produce to all live in the My.project.plugins package. It looks like my problem is that I'm mingling separate products in the same code tree and setuptools/distutils doesn't like that. Fair enough. I don't think I'm quite ready to drink the "one tree per product" kool-aid just yet. Firstly, it seems weird to me to split up the source like that and secondly, my colleagues would probably kill me ;-) So what I will probably end up doing is hacking up setuptools to insert the __init__.py files I need. Which begs the question - Is this: a) A dirty, dirty hack I should never speak of again OR b) Not a bad idea and potentially useful to other people. I have a feeling I know the answer to that ;-) Thanks for the help. Richard From pje at telecommunity.com Fri Nov 11 21:00:46 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 11 Nov 2005 15:00:46 -0500 Subject: [Distutils] __init__.py files missing from my eggs In-Reply-To: Message-ID: <5.1.1.6.0.20051111141005.02005f58@mail.telecommunity.com> At 06:31 PM 11/11/2005 +0000, Richard Cooper wrote: >From: Phillip J. Eby [mailto:pje at telecommunity.com] > > What you probably want is a layout like this: > > > > app/ > > setup.py > > My/ > > Project/ > > app > > Plugin1/ > > setup.py > > My/ > > Project/ > > plugin1/ > > > > etc. > >My knee-jerk reaction is 'Ewww!' Is this what PEAK (or ant other large >setuptools based project) does? Yes, actually. More precisely, it's what PEAK *will* do, since I haven't actually started breaking out individual projects yet. >How do you handle My.Project.utils (for >example) and the half dozen or so other packages at different levels of >the hierarchy which are used in multiple "products"? You make them individual projects. For example, I have a module, peak.util.imports, that will be spun off as its own project. Its layout will be: setup.py peak/ __init__.py util/ __init__.py imports.py The __init__.py files will be empty, of course, since as namespace packages they don't need any contents. Any project (including PEAK itself) that needs to use this module will simply depend on this project as a whole. My original plan for setuptools was to use an approach that could section out parts of a large distribution like this. However, I quickly realized that this was counterproductive to the Python "ecosystem"; it would be more beneficial to have lots of small projects that can be used by others (without buying into the entire PEAK system), than it would be for me to keep using the same "all-in-one" source tree layout that I'm currently using for PEAK. I suspect that you may like Zope's "zpkg" system better for your purposes, since it is designed to extract subsets of a larger distribution and automatically generate setup scripts for them. I don't know if you can use it to create eggs, but it might be possible to modify it to do so. >My needs are a bit specialised, this is a commercial app so source >distribution, easy_install, etc will never be on the cards. Distribution >is strictly "frozen" products (py2exe and eggs) only. easy_install is useful as an in-house tool for automatically building your projects. For example, you could create a setup.py that lists all your plugin project names as requirements, and a setup.cfg that lists their subversion URLs, and you could run easy_install against that directory to check out and build all the latest plugin eggs, dropping them in a distribution directory. Your users don't ever have to see easy_install, for it to be useful in your project. Of course, you may not be using Subversion, etc., and this may be moot, but I'm not psychic and don't know anything about what you're doing besides what you tell me. :) > > If you are working on these projects at the same > > time, you need only run "setup.py develop" in each one to set > > up a sys.path structure that merges all the packages into one. > >We would probably end up with 20 or so "projects" if we did things that >way as opposed to the single code tree we have now. Given that, >"setup.py develop"*20 seems like more work than our current approach >which is "add the root of the code tree to PYTHONPATH" Does that mean you have 19 plugins? Perhaps I'm misunderstanding something about your intentions. I assume that your projects would be one for the app, and one for each plugin, so that you can create a py2exe for the app and an egg for each plugin. If that's the case, then you would only have 20 projects if you had 19 plugins. Also, if the plugins don't depend on each other, then to work on any one plugin you only run "develop" twice: once for the app and once for the plugin. Also, this may not be especially relevant to your project at the moment, but "develop" also makes sure that your environment is synchronized with any external requirements. For example, in a multi-developer scenario where developer Aaron changes a dependency on the FooBar package from 1.1 to 1.2, then when developer Bob updates his source checkout and runs "develop", it will find/fetch/build/install FooBar 1.2, thereby helping to reduce the pain of adding or removing dependencies. You could probably quite rightly say that this doesn't affect your project because you don't have external dependencies, but a significant part of *why* few projects have external dependencies is because they're a costly pain -- if you're not using setuptools. I point this out because setuptools is intended primarily to enable a new way of developing projects, and enabling people to do things that were previously unthinkable. If you aren't trying to do what was impossible before, then certainly setuptools will not appear to help your current situation much. :) > > Note, by the way, that you don't need namespace packages to > > have plugins for an application. Plugins can be in any > > package structure you like, as eggs' "entry points" system > > can be used by plugins to advertise the functions they > > provide, and the importing can be done automatically for you. > >Yeah I know. The entry point system is the main reason I started playing >with setuptools. However I still wanted the plugins we produce to all >live in the My.project.plugins package. Note that giving each plugin its own top-level package would completely eliminate any duplication of directory structure between plugins, if that's your concern. But note that in any case, all that you'd be duplicating are empty directories and zero-length __init__.py files. >It looks like my problem is that I'm mingling separate products in the >same code tree and setuptools/distutils doesn't like that. Fair enough. >I don't think I'm quite ready to drink the "one tree per product" >kool-aid just yet. Firstly, it seems weird to me to split up the source >like that and secondly, my colleagues would probably kill me ;-) Here's another possible layout, that will work nicely for eggs, but won't work for "setup.py develop": TheProject/ MyApp/ __init__.py setup.py *all app code here* Plugin1/ __init__.py setup.py *plugin code here* The idea here is that you have a MyApp top-level package, and a top-level package for each plugin. Each setup script is *inside* the corresponding package directory. Adding TheProject to PYTHONPATH makes everything importable, just like you have now. But each project corresponds to a top level package, and uses package_dir = {'MyApp':'.'}, or {'Plugin1':'.'} etc. so that each project directory is also a package directory. This compromise layout preserves the single-checkout+PYTHONPATH approach, and should allow you to generate eggs without any special hacks, at the cost of flattening your hierarchy a bit, and making it impossible to use setuptools' "develop" or "test" commands (which need the package directories to be somewhere *under* the directory containing setup.py). Personally, I find that deeply nested foo.bar.baz hierarchies are a Java-ism; Python doesn't need them and is usually better off without them; "flat is better than nested", and all that. But I can also understand that reorganizing your packages just to support eggs might be considered too costly. Just make sure you factor in the cost of *maintaining* a hack to do this, since the reorganization cost would be a one-time thing, but the hack would be something you'd have to maintain forever. >So what I will probably end up doing is hacking up setuptools to insert >the __init__.py files I need. Which begs the question - Is this: > >a) A dirty, dirty hack I should never speak of again OR >b) Not a bad idea and potentially useful to other people. Yes. :) It's both, really. Consider Chandler, for example, which is currently very similar to your project. Actually, it's worse, because it doesn't even have a setup script yet. In the next release cycle, we'll be moving to eggs and we will have to do some source tree splitting, and there will be some pain, and questioning, and maybe even some griping about it. *But*, we spent a lot of time during the current release cycle doing quite a lot of *flattening* of the source tree, turning deeply nested modules like "osaf.contentmodel.contacts.Contacts" into "osaf.pim.contacts", and consolidating APIs so that most code can now do: from osaf import pim aContact = pim.Contact() instead of the former: import osaf.pim.contacts.Contacts as Contacts aContact = Contacts.Contact() Flat really *is* better than nested. Much better. And as a bonus, anything that we choose to split into separate eggs will not have as much duplication. In the general case, the 'osaf' top-level package will be the only thing that gets duplicated, and since it's a namespace package, the __init__.py will be an empty file anyway. So, I would encourage you to urge those you share the hack with, to consider all of the issues I've brought out here, and use it only as a crutch to allow a project to get past a temporary issue with its layout, not as a cane to perpetually lean on. Putting on my software development manager hat, I would tell you flatly that the time to manage the directory structure duplication and use of "develop" is nothing compared to the time saved by being able to intelligently manage external dependencies. And if you don't have any external dependencies, it's only because they cost too much before. Setuptools changes the cost equation, and therefore changes what the ideal development patterns are. The reality may be that in your particular environment you don't have the authority or reputation to lead such a change, but we shouldn't let that stop other people from getting that awareness. Okay, I'll get off the soapbox now. :) Hack away, just make sure to point people to this thread and especially this message. One other idea, by the way... Have you tried adding: py_modules = ['My.__init__', 'My.App.__init__', ...] to the setup() arguments? It's just a guess on my part, but I think it might actually work, without doing any hacking on distutils or setuptools. Just a thought. Happy hacking, either way. :) From pje at telecommunity.com Fri Nov 11 21:26:45 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 11 Nov 2005 15:26:45 -0500 Subject: [Distutils] setuptools using -D on package requirements? In-Reply-To: <5FEAEE5A-CDC7-42E5-A019-12D269FC3880@groovie.org> Message-ID: <5.1.1.6.0.20051111151428.020946c0@mail.telecommunity.com> At 02:52 PM 11/10/2005 -0800, Ben Bangert wrote: >Is there a reason I can't have setuptools automatically upgrade/ >delete existing/conflicting package versions that are a 'requirement' >for the package I want to install? > >For example, if I have the 0.3 of PasteScript installed, and run my >own package which requires (and has the URL for) the svn dev version >of PasteScript, here's what I get: > >% sudo python ez_setup.py -D -f http://pylons.groovie.org/ Pylons > >Reading http://pylons.groovie.org/ >Searching for Pylons >Best match: Pylons 0.1dev-r87 >Processing Pylons-0.1dev_r87-py2.4.egg >Pylons 0.1dev-r87 is already the active version in easy-install.pth > >Using /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/ >python2.4/site-packages/Pylons-0.1dev_r87-py2.4.egg >Processing dependencies for Pylons >error: Installed distribution Paste 0.3 conflicts with requirement >Paste>=0.4dev-r362 That's weird. What version of setuptools are you using? The current version uses a "breadth-first" dependency resolution algorithm that should give Pylons' dependencies precedence over those of other packages. I tried doing your command above (adding -vv to get more detailed info), but I get this error instead: No local packages or download links found for Paste>=0.4dev-r3631 error: Could not find distribution for Requirement.parse('Paste>=0.4dev-r3631') Presumably this is because you've changed your requirements but not your links page. Anyway, it shows that my local setuptools is doing the right thing, because it's not reporting a conflict with Paste 0.3 (which I installed first). Are you using at least setuptools 0.6a6? If not, please try installing the development version of setuptools, by running: easy_install setuptools==dev And then try your easy_install again. From ben at groovie.org Mon Nov 14 20:24:09 2005 From: ben at groovie.org (Ben Bangert) Date: Mon, 14 Nov 2005 11:24:09 -0800 Subject: [Distutils] setuptools using -D on package requirements? In-Reply-To: <5.1.1.6.0.20051111151428.020946c0@mail.telecommunity.com> References: <5.1.1.6.0.20051111151428.020946c0@mail.telecommunity.com> Message-ID: On Nov 11, 2005, at 12:26 PM, Phillip J. Eby wrote: > That's weird. What version of setuptools are you using? The > current version uses a "breadth-first" dependency resolution > algorithm that should give Pylons' dependencies precedence over > those of other packages. > > I tried doing your command above (adding -vv to get more detailed > info), but I get this error instead: > > No local packages or download links found for Paste>=0.4dev-r3631 > error: Could not find distribution for Requirement.parse > ('Paste>=0.4dev-r3631') > > Presumably this is because you've changed your requirements but not > your links page. Anyway, it shows that my local setuptools is > doing the right thing, because it's not reporting a conflict with > Paste 0.3 (which I installed first). Are you using at least > setuptools 0.6a6? If not, please try installing the development > version of setuptools, by running: > > easy_install setuptools==dev > > And then try your easy_install again. Hmm, well, I ran the command again earlier today, and it updated all the requisite packages as well with no problem. So I guess this probably wasn't a problem after all and just user error. :) - Ben From daishi at egcrc.net Wed Nov 16 00:32:15 2005 From: daishi at egcrc.net (daishi@egcrc.net) Date: Tue, 15 Nov 2005 15:32:15 -0800 Subject: [Distutils] (b)dist-dir questions Message-ID: <2c3d38012b57487465a07b365ab3b07d@egcrc.net> Hi, I hope this is the correct forum for user questions concerning setuptools/eggs, and apologize in advance if not. My problem is likely due to misunderstanding or obtuseness on my part, for which I also apologize. I am trying to build a package using setuptools while maintaining a pristine source tree. For the standard distutils setup, I do this by invoking: python setup.py build --build-base= install Now for setups which generate eggs using setuptools this still creates a `dist' directory in the source root. Looking at the setuptools documentation, I tried: python setup.py build --build-base= \ bdist_egg --bdist-dir= install and also --dist-dir= instead of --bdist-dir, but these both seem to place the "final" egg in , and no longer install into the invoked python's site-packages directory. I am wondering what the proper incantation might be to use as the work/temporary egg creation directory (i.e., the equivalent of the default ./dist), while still in the end installing the resulting egg into site-packages. I am using: setuptools-0.6a8dev_r41411-py2.4.egg for my setuptools. Thanks in advance, Daishi From pje at telecommunity.com Wed Nov 16 00:44:19 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 15 Nov 2005 18:44:19 -0500 Subject: [Distutils] (b)dist-dir questions In-Reply-To: <2c3d38012b57487465a07b365ab3b07d@egcrc.net> Message-ID: <5.1.1.6.0.20051115184149.01f13c98@mail.telecommunity.com> At 03:32 PM 11/15/2005 -0800, daishi at egcrc.net wrote: >I am trying to build a package using setuptools while >maintaining a pristine source tree. For the standard >distutils setup, I do this by invoking: > python setup.py build --build-base= install > >Now for setups which generate eggs using setuptools >this still creates a `dist' directory in the source >root. Looking at the setuptools documentation, I >tried: > python setup.py build --build-base= \ > bdist_egg --bdist-dir= install >and also --dist-dir= instead of --bdist-dir, >but these both seem to place the "final" egg in >, and no longer install into the invoked >python's site-packages directory. Replace the 'install' part of your command line with 'easy_install projectname-projversion-pyversion.egg', or if is an empty directory, then 'easy_install /*.egg' will also work. From pje at telecommunity.com Wed Nov 16 02:07:17 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 15 Nov 2005 20:07:17 -0500 Subject: [Distutils] (b)dist-dir questions In-Reply-To: <72b6087b38cfec924e2e39ffc461b95b@egcrc.net> References: <5.1.1.6.0.20051115184149.01f13c98@mail.telecommunity.com> <5.1.1.6.0.20051115184149.01f13c98@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051115200501.01f33a20@mail.telecommunity.com> At 04:30 PM 11/15/2005 -0800, daishi at egcrc.net wrote: >On Nov 15, 2005, at 3:44 PM, Phillip J. Eby wrote: >>Replace the 'install' part of your command line with 'easy_install >>projectname-projversion-pyversion.egg', or if is an empty >>directory, then 'easy_install /*.egg' will also work. > >Right; of course. Thanks. > >Is there a way to query a setuptools-based >setup.py to ask what the output egg filename >would be, so that I could do something like: > >EGG=`python setup.py egg_name` >easy_install $DIST/$EGG Try `python setup.py --name` and `python setup.py `--version`. Unfortunately, these won't be in the escaped format used to create filenames. If the name or version contain "illegal" characters they will be replaced with '_' in the name and version actually used for the filename. Rather than deal with all that, you may find it easier to just use an empty dist directory and *.egg. From ashleywalsh at gmail.com Wed Nov 16 03:18:34 2005 From: ashleywalsh at gmail.com (Ashley Walsh) Date: Wed, 16 Nov 2005 12:18:34 +1000 Subject: [Distutils] EntryPoint.require() when dist=None Message-ID: EntryPoint.require() fails when extras==() and dist==None because it still attempt to call dist.requires(). Should the method be changed to somthting like:: def require(self, env=None, installer=None): if self.extras and not self.dist: raise UnknownExtra("Can't require() without a distribution", self) if self.dist: map(working_set.add, working_set.resolve(self.dist.requires (self.extras),env,installer)) Cheers, Ashley From pje at telecommunity.com Wed Nov 16 04:27:44 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 15 Nov 2005 22:27:44 -0500 Subject: [Distutils] EntryPoint.require() when dist=None In-Reply-To: Message-ID: <5.1.1.6.0.20051115222716.01f30008@mail.telecommunity.com> At 12:18 PM 11/16/2005 +1000, Ashley Walsh wrote: >EntryPoint.require() fails when extras==() and dist==None because it >still attempt to call dist.requires(). Should the method be changed >to somthting like:: > > def require(self, env=None, installer=None): > if self.extras and not self.dist: > raise UnknownExtra("Can't require() without a >distribution", self) > if self.dist: > map(working_set.add, > working_set.resolve(self.dist.requires >(self.extras),env,installer)) Probably. :) I'll have a look at it. From pje at telecommunity.com Wed Nov 16 04:26:48 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 15 Nov 2005 22:26:48 -0500 Subject: [Distutils] MANIFEST destiny :) Message-ID: <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> I originally added CVS and Subversion support to setuptools in order to get past the pain of the distutils' MANIFEST system. I used to use MANIFEST.in, but it was a royal pain to get right, and I pretty much always forgot to add stuff to it. The most common problem when I shipped a source distribution was that the MANIFEST was screwed up, such that a CVS checkout worked fine but a source distribution would break. Ugh. So, the CVS/Subversion support for setuptools automatically makes your MANIFEST include anything under revision control, whether you have a MANIFEST.in or not. If you don't have a MANIFEST.in, the MANIFEST is built every time you run an sdist, so it's always up-to-date. Ah, bliss! But all is not happy in MANIFESTville. It turns out that the bdist_rpm command expects to build an sdist, for reasons impenetrable to me. If you build an RPM from a source checkout, everything is fine because setuptools can auto-discover your files and build the MANIFEST for the new sdist. But if you build an RPM *from* an sdist, it's a no-go. In addition, many folks have been asking for this autodetection to cover package data files as well. Why, they reasonably ask, must I specify each and every file to be included in a package, when the system already knows what files I have in revision control, or which is covered by my MANIFEST.in? The reason I've been avoiding adding this feature, however, is because of the first issue; when you make an sdist, you lose that additional metadata, so it would become impossible to build *any* binary from an sdist, not just RPMs. Until recently, that issue seemed insurmountable. So today, after looking over the issue a bit, I think I have a plan for dealing with MANIFEST: * Change the MANIFEST format to be platform-independent (currently it contains OS-specific path separators) * Always, always, always build MANIFEST, and always include both the MANIFEST file and MANIFEST.in (if present) in the source distribution. * Disable all the options that allow user control over MANIFEST generation, including pruning, defaults, changing the filenames, etc. * Use the MANIFEST data (along with revision control info) not only for producing source distributions, but also to determine what files should be considered "package data", if the user passes an 'include_package_data=True' keyword to setup(). The net result would be a single source for what constitutes "the distribution contents", in the sense of files that are not directly part of the distutils build process. For files that are built automatically in some way but should be included in source distributions or as package data, you would still have to put them in MANIFEST.in. But anything that was under CVS or Subversion would be handled automatically, and you wouldn't have to duplicate data between MANIFEST.in and setup(package_data={...}). I'm also thinking that most of the MANIFEST logic could and should move to the Distribution class, since the data will be used by multiple commands. Thus, the sdist command could just ask the Distribution for the MANIFEST and get it, as would the commands that copy package data files to the build directory. I suspect the most controversial parts of this idea are: * Disabling all user control of MANIFEST * Forcibly including MANIFEST and MANIFEST.in in source distributions * Making MANIFEST be always platform-independent When Googling the issues around MANIFEST, I noticed that the idea of having MANIFEST or MANIFEST.in included automatically has been repeatedly shot down here over the years. However, if I followed the logic put forth on those occasions, I would never have implemented revision control support in the first place, so I guess if I'm in for a penny, I might as well be in for a pound, as they say. I couldn't find any argument one way or the other about the manifest-generation options, nor any reasons why MANIFEST needs to remain platform-specific, so I presume the options are just YAGNI and the format was just an implementation accident. Likewise, as far as I can tell there is no reason for *not* regenerating a MANIFEST whenever you need one, so the current behavior of only building one when MANIFEST.in changes or you use --force-manifest, seems like a premature optimization. Or maybe it wasn't an excessive optimization when the distutils were created, but it's not as if it's going to save you much time compared to the actual archive building process today. I'm thinking that basically --force-manifest would become a no-op in setuptools, in the sense that you won't be able to *stop* the MANIFEST from being built every single time. --manifest-only would still be possible. --manifest and --template would have to be rejected, however, because the standard name is needed for MANIFEST to be re-read when you build stuff from the produced sdist. --no-defaults would be ignored, except for a warning. If you don't want the defaults, you can always start your MANIFEST.in with an exclude pattern to exclude absolutely everything already included. There shouldn't be two ways to do the same thing, especially not one that you can use on the command line to mess things up in a non-repeatable fashion! Likewise --no-prune, because that's a similar recipe for disaster. A lot of these ideas are potential backward compatibility problems, so we'll have to see how they play out in setuptools before considering them for addition to the distutils. My guess, however, is that most prolific Python developers want to spend their time writing code, not writing and debugging MANIFEST.in files, and that fact has been responsible for a lot of setuptools uptake so far. I've been seeing a lot of projects that use setuptools for no apparent reason other than it makes writing the setup script a little easier, due to find_packages(), package_data, and the lack of need for a MANIFEST when source control is involved. These are qualities I'd like to extend further, even at the cost of some flexibility. Heck, most of the distutils' flaws lie in their extreme versatility. You can tell each individual command that it's using different build or distribution directories, for example, and in the process completely foul up your builds. What's more, every distutils tutorial may well end up giving people different instructions as to the "best" way to lay out a project directory. If there's ever a "distutils 2", it needs to become dictator-ware and tell you exactly what the One Obvious Way is. If everything *had* to be a particular way, then changing how the distutils work would actually be possible, whereas now, it's bloody hard to even figure out which of the nine billion ways to do it are actually in use. Okay, off the soapbox now. :) Does anybody see any issues with this that I'm missing, with respect to using the MANIFEST/FileList machinery to control sdist and package data, or my implementation plans for doing so? Thanks. From ianb at colorstudy.com Wed Nov 16 05:17:32 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 15 Nov 2005 22:17:32 -0600 Subject: [Distutils] [TurboGears] hardcoded paths in tg-admin.exe and easy_install.exe? In-Reply-To: <1132114166.686802.158620@o13g2000cwo.googlegroups.com> References: <1132114166.686802.158620@o13g2000cwo.googlegroups.com> Message-ID: <437AB2DC.2050106@colorstudy.com> rtilley at vt.edu wrote: > Perhaps this has been mentioned already. So, forgive me if it has. > > I install Python into 'C:/Program Files/Python24' as I do not like to > clutter up the C:/ portion of my file system. After installing > turbogears and trying to begin the tutorial, I get the below error when > I do this 'tg-admin quickstart': > > Program: can't open file 'Files\Python24\python.exe' : [Error 2] No > such file or directory > > The same thing happens when I try to run easy_install.exe > > The .py versions of tg-admin and easy_install work fine... it's just > the .exe versions that crash. I have named this the 'hard-coded python > path bug'. > > Is this a know issue that will be fixed? > > Also, the instructions about adding 'C:\Python24' and > 'C:\Python24\Scripts' to the PATH variable should be adjusted to > wherever the user has actually installed Python ;) Sounds like a bug in .exe generator that it doesn't like spaces. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From mal at egenix.com Wed Nov 16 11:49:47 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 16 Nov 2005 11:49:47 +0100 Subject: [Distutils] MANIFEST destiny :) In-Reply-To: <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> References: <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> Message-ID: <437B0ECB.5030802@egenix.com> Hi Phillip, you asked for it, so I'm giving you some flaming ;-) ... Phillip J. Eby wrote: > I originally added CVS and Subversion support to setuptools in order to get > past the pain of the distutils' MANIFEST system. I used to use > MANIFEST.in, but it was a royal pain to get right, and I pretty much always > forgot to add stuff to it. The most common problem when I shipped a source > distribution was that the MANIFEST was screwed up, such that a CVS checkout > worked fine but a source distribution would break. Ugh. > > So, the CVS/Subversion support for setuptools automatically makes your > MANIFEST include anything under revision control, whether you have a > MANIFEST.in or not. If you don't have a MANIFEST.in, the MANIFEST is built > every time you run an sdist, so it's always up-to-date. Ah, bliss! > > But all is not happy in MANIFESTville. It turns out that the bdist_rpm > command expects to build an sdist, for reasons impenetrable to me. If you > build an RPM from a source checkout, everything is fine because setuptools > can auto-discover your files and build the MANIFEST for the new sdist. But > if you build an RPM *from* an sdist, it's a no-go. I don't understand what you mean with "no-go" - the current system works just fine if you include the MANIFEST file in the sdist. > In addition, many folks have been asking for this autodetection to cover > package data files as well. Why, they reasonably ask, must I specify each > and every file to be included in a package, when the system already knows > what files I have in revision control, or which is covered by my MANIFEST.in? > > The reason I've been avoiding adding this feature, however, is because of > the first issue; when you make an sdist, you lose that additional metadata, > so it would become impossible to build *any* binary from an sdist, not just > RPMs. Until recently, that issue seemed insurmountable. That's simply not true. You have to include the MANIFEST file in the sdist and then everything is fine. > So today, after looking over the issue a bit, I think I have a plan for > dealing with MANIFEST: > > * Change the MANIFEST format to be platform-independent (currently it > contains OS-specific path separators) -0.5 You are missing an important point: MANIFEST files can be build using tools outside distutils and external package building tools may require these to be platform dependent. Distutils itself is happy with posix style separators on all platforms. > * Always, always, always build MANIFEST, and always include both the > MANIFEST file and MANIFEST.in (if present) in the source distribution. -1 on always building MANIFEST. This would miss the point of managing MANIFEST files independently of your package files, e.g. using Makefiles or other tools dealing with file dependencies, checkouts, etc. > * Disable all the options that allow user control over MANIFEST generation, > including pruning, defaults, changing the filenames, etc. -1 Again, you are forgetting that MANIFEST files serve a purpose and are external to the distutils process for a reason. You are free to have distutils build your MANIFEST files from MANIFEST.in files, have distutils command auto-generate them, or use external programs triggered by Makefiles or similar distribution building processes to generate them. In your world, everything is done within distutils, so it's understandable that you'd like to get rid of the external nature of MANIFEST files, but please keep in mind that these features are being used and removing the logic would seriously break things for packagers relying on other mechanisms to build their MANIFEST files. Simply overwriting the MANIFEST file everytime you run the sdist command would break such use. > * Use the MANIFEST data (along with revision control info) not only for > producing source distributions, but also to determine what files should be > considered "package data", if the user passes an > 'include_package_data=True' keyword to setup(). Isn't that already the case ? I mean you can put anything you like into MANIFEST and it will be included in the sdist. > The net result would be a single source for what constitutes "the > distribution contents", in the sense of files that are not directly part of > the distutils build process. For files that are built automatically in > some way but should be included in source distributions or as package data, > you would still have to put them in MANIFEST.in. But anything that was > under CVS or Subversion would be handled automatically, and you wouldn't > have to duplicate data between MANIFEST.in and setup(package_data={...}). Again, not everybody is using distribution processes built around CVS or Subversion. Left aside that there are quite a few other SCM tools out there, you also have the case where you create distributions from plain directories (which is what MANIFEST.in and MANIFEST are targetting). > I'm also thinking that most of the MANIFEST logic could and should move to > the Distribution class, since the data will be used by multiple > commands. Thus, the sdist command could just ask the Distribution for the > MANIFEST and get it, as would the commands that copy package data files to > the build directory. Wait: MANIFEST defines what goes into the sdist - not an arbitrary (binary) distribution. > I suspect the most controversial parts of this idea are: > > * Disabling all user control of MANIFEST > * Forcibly including MANIFEST and MANIFEST.in in source distributions > * Making MANIFEST be always platform-independent > > When Googling the issues around MANIFEST, I noticed that the idea of having > MANIFEST or MANIFEST.in included automatically has been repeatedly shot > down here over the years. However, if I followed the logic put forth on > those occasions, I would never have implemented revision control support in > the first place, so I guess if I'm in for a penny, I might as well be in > for a pound, as they say. I'm not sure what you mean by "having MANIFEST[.in] included". It would certainly make sense to have the MANIFEST[.in] files automatically be added as default in sdist.py and I'd be +1 on that (even though it never was an issue for me as I always include them in the MANIFEST file). > I couldn't find any argument one way or the other about the > manifest-generation options, nor any reasons why MANIFEST needs to remain > platform-specific, so I presume the options are just YAGNI and the format > was just an implementation accident. See above. > Likewise, as far as I can tell there is no reason for *not* regenerating a > MANIFEST whenever you need one, so the current behavior of only building > one when MANIFEST.in changes or you use --force-manifest, seems like a > premature optimization. Or maybe it wasn't an excessive optimization when > the distutils were created, but it's not as if it's going to save you much > time compared to the actual archive building process today. See above. > I'm thinking that basically --force-manifest would become a no-op in > setuptools, in the sense that you won't be able to *stop* the MANIFEST from > being built every single time. --manifest-only would still be > possible. --manifest and --template would have to be rejected, however, > because the standard name is needed for MANIFEST to be re-read when you > build stuff from the produced sdist. > > --no-defaults would be ignored, except for a warning. If you don't want > the defaults, you can always start your MANIFEST.in with an exclude pattern > to exclude absolutely everything already included. There shouldn't be two > ways to do the same thing, especially not one that you can use on the > command line to mess things up in a non-repeatable fashion! Likewise > --no-prune, because that's a similar recipe for disaster. These options are meant for people who don't have a MANIFEST.in file to begin with or just quickly want to build an sdist with parts of the whole distribution or an extended version (e.g. for testing or upgrading). > A lot of these ideas are potential backward compatibility problems, so > we'll have to see how they play out in setuptools before considering them > for addition to the distutils. My guess, however, is that most prolific > Python developers want to spend their time writing code, not writing and > debugging MANIFEST.in files, and that fact has been responsible for a lot > of setuptools uptake so far. I've been seeing a lot of projects that use > setuptools for no apparent reason other than it makes writing the setup > script a little easier, due to find_packages(), package_data, and the lack > of need for a MANIFEST when source control is involved. These are > qualities I'd like to extend further, even at the cost of some flexibility. > > Heck, most of the distutils' flaws lie in their extreme versatility. That comment is just silly: distutils is so powerful because of its versatility. You wouldn't have been able to build setuptools without this versatility. Just because you don't like some of this flexibility doesn't mean that distutils is broken in some way. > You can tell each individual command that it's using different build or > distribution directories, for example, and in the process completely foul > up your builds. What's more, every distutils tutorial may well end up > giving people different instructions as to the "best" way to lay out a > project directory. If there's ever a "distutils 2", it needs to become > dictator-ware and tell you exactly what the One Obvious Way is. If > everything *had* to be a particular way, then changing how the distutils > work would actually be possible, whereas now, it's bloody hard to even > figure out which of the nine billion ways to do it are actually in use. That's your point of view - I've never had a hard time adjusting distutils to whatever I wanted it to do. After you get used to the way things are handled in distutils, extending it is often enough really easy and would be much harder in your One Obvious Way to do it (unless you had a time-machine, zoom to 2042 and then take all possibly ways to build distributions into account on your way back to 2005 ;-). You are free to develop setuptools into your own little vision of distutils 2 - and that's one of distutils strengths ! > Okay, off the soapbox now. :) Does anybody see any issues with this that > I'm missing, with respect to using the MANIFEST/FileList machinery to > control sdist and package data, or my implementation plans for doing > so? Thanks. I think I gave you some more hints as to why MANIFEST[.in] works the way it does. Adding these files as defaults to the set of sdist files sounds like a good idea (I don't remember discussions about this, so maybe wrong). Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 16 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From ianb at colorstudy.com Wed Nov 16 17:29:33 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 16 Nov 2005 10:29:33 -0600 Subject: [Distutils] MANIFEST destiny :) In-Reply-To: <437B0ECB.5030802@egenix.com> References: <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> <437B0ECB.5030802@egenix.com> Message-ID: <437B5E6D.70805@colorstudy.com> M.-A. Lemburg wrote: >>* Always, always, always build MANIFEST, and always include both the >>MANIFEST file and MANIFEST.in (if present) in the source distribution. > > > -1 on always building MANIFEST. > > This would miss the point of managing MANIFEST files > independently of your package files, e.g. using > Makefiles or other tools dealing with file dependencies, > checkouts, etc. How about always build it, automatically, if some particular option is not passed to setup()? This would only apply to packages using setuptools, not distutils, and there's other things that existing distutils packages might have to tweak to move to setuptools. Maybe it would be as simple as a keyword argument that refers to a routine to build the MANIFEST file; and if you build MANIFEST by hand, then you just make the function a no-op. Also, disable all command-line options that control when/if/how MANIFEST is generated; these seem peculiar to me. MANIFEST generation seems like package metadata, not something particular to your build, and command-line options seem like a bad way to control that. It's a little fuzzy, though, since command-line options and setup.cfg are equivalent, and setup.cfg options often feel like package data. So... I don't know what the middle ground is there. Either way, these aren't things that seem like they should be regularly tweaked. I don't know, I don't have any packages with complex (or even simple) build processes, so I only know that the current system is mysterious to me, and almost but doesn't always work without my intervention. I think Phillip also mentioned using MANIFEST for package_data? At least insofar as the packages and the MANIFEST intersect, this seems like a good idea. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Wed Nov 16 18:53:40 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 16 Nov 2005 12:53:40 -0500 Subject: [Distutils] MANIFEST destiny :) In-Reply-To: <437B0ECB.5030802@egenix.com> References: <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051116114559.01f47bd8@mail.telecommunity.com> At 11:49 AM 11/16/2005 +0100, M.-A. Lemburg wrote: >I don't understand what you mean with "no-go" - the current >system works just fine if you include the MANIFEST file >in the sdist. But it's not included, and you have to know to include it - and people who previously requested here that MANIFEST and MANIFEST.in be included in the manifest were shot down under the claim that including these defaults would be "too much magic". > > The reason I've been avoiding adding this feature, however, is because of > > the first issue; when you make an sdist, you lose that additional > metadata, > > so it would become impossible to build *any* binary from an sdist, not > just > > RPMs. Until recently, that issue seemed insurmountable. > >That's simply not true. You have to include the MANIFEST file >in the sdist and then everything is fine. I'm speaking about the context of setuptools, where the typical user has no MANIFEST.in, because they're using Subversion or CVS and therefore don't need one. > > * Change the MANIFEST format to be platform-independent (currently it > > contains OS-specific path separators) > >-0.5 > >You are missing an important point: MANIFEST files can be >build using tools outside distutils and external package >building tools may require these to be platform dependent. One reason I posted to find out what specific uses people actually had for such things. Do the uses actually exist? What are they used for? One important point: on POSIX-y platforms (i.e. virtually everything but Windows), there's no difference between distutils paths and system paths. So, what external tools build or read MANIFEST files on Windows, or any other platform that doesn't accept '/' as a separator? Also, '/' *is* a valid path separator on Windows, so if your MANIFEST processing is done in Python, a platform-independent format won't have any effect. (Indeed, the only reason to actually *change* the format is so that sdist files built *on Windows* would be usable on other platforms! Other platforms would never see a difference.) I know that in principle, somebody somewhere may have some tool that would break if MANIFEST changed it's format. My question is, who and how many? If there's some widely-used tool that does this, then it's reasonable to take it into account. If it's just a theoretical possibility, it's not worth much concern. If there are a handful of people with a hard-to-change setup, it's somewhere in between. > > * Always, always, always build MANIFEST, and always include both the > > MANIFEST file and MANIFEST.in (if present) in the source distribution. > >-1 on always building MANIFEST. > >This would miss the point of managing MANIFEST files >independently of your package files, e.g. using >Makefiles or other tools dealing with file dependencies, >checkouts, etc. Please point me to some examples of this, especially one that can't simply generate a MANIFEST.in instead. > > * Disable all the options that allow user control over MANIFEST > generation, > > including pruning, defaults, changing the filenames, etc. > >-1 > >Again, you are forgetting that MANIFEST files serve a purpose >and are external to the distutils process for a reason. You >are free to have distutils build your MANIFEST files from >MANIFEST.in files, have distutils command auto-generate them, >or use external programs triggered by Makefiles or similar >distribution building processes to generate them. Examples? You keep saying that people *can* do these things, but that's not anything like the same as saying that any significant number of people *actually* do them. Frankly, most people I've encountered who are doing Python software development don't know how to get a basic setup.py to work right, and feel the distutils are way too complicated, underdocumented, or just plain broken, because they don't feel like they can control them. OSAF, for example, has some developers who are very smart about creating build processes. Smart enough to be *able* to do the kinds of things you're describing. But they sure as heck don't use the distutils to *actually* do them, because from their perspective the distutils is a big pile of broken undocumentedness. If the distutils are so frustrating to such smart developers, there's something wrong. Thus, I find these super-custom processes you're talking about highly implausible, because the only people who could implement them are the people with a strong knowledge of the distutils -- an incredibly rare breed of person, in other words. Most people just want this stuff to work, and they don't want to have to learn *how* it works. They have better things to do with their time. They want a build *tool*, not a build library or build framework. Most people don't know MANIFEST exists, until they get bitten by the need to have one, or by it being out of date. Hell, look at how many packages on PyPI and the Vaults aren't even packaged with distutils at all! For many people, it's clearly easier to just tarball your source directory than to have to learn about this MANIFEST stuff. >In your world, everything is done within distutils, so it's >understandable that you'd like to get rid of the external >nature of MANIFEST files, but please keep in mind that >these features are being used and removing the logic would >seriously break things for packagers relying on other >mechanisms to build their MANIFEST files. Please point me to these developers, and show me one that couldn't just spend a few minutes making their tools generate a MANIFEST.in instead. I'm suggesting that we present a very small number of highly-capable people with a truly minor inconvenience, in order to make an extremely large group of people happier by taking away something that invariably bites them. > > * Use the MANIFEST data (along with revision control info) not only for > > producing source distributions, but also to determine what files should be > > considered "package data", if the user passes an > > 'include_package_data=True' keyword to setup(). > >Isn't that already the case ? I mean you can put anything >you like into MANIFEST and it will be included in the sdist. I'm talking about package data - a feature that was pioneered in setuptools and added to the distutils in Python 2.4. The ability to specify data files that are *installed* in a package directory. Specifically, I'm suggesting that users be able to replace the package_data setup keyword with a simple include_package_data flag, so that the MANIFEST data can be used to determine what data files to install. This has nothing to do with sdist; I'm talking about being able to unify the distutils' idea of what files are part of the distribution, because for the simple cases that are 90% of projects, that's a very useful thing to have. For every SciPy and mxODBC and Twisted, there are easily a hundred packages without anything like their level of complex build needs. I'm talking about streamlining things for the simple packages, but there isn't anything in what I'm proposing that keeps anybody from doing complex things. I'm just saying we should remove *multiple* ways to do the *same* complex thing. We should pick One Obvious Way to *customize*, replacing multiple hooks with single hooks that allow the same degree of customizability. For example, if you have tools that generate something, right now you could choose to generate either MANIFEST or MANIFEST.in. I'm suggesting that we should choose for you, and say it's MANIFEST.in that you should generate, since it's the more expressive and flexible of the two formats. Similarly, you can currently choose what filenames MANIFEST and MANIFEST.in have, and I'm suggesting that those be their only names. If you need other files you certainly have the option of copying, renaming, and using other existing file manipulation tools. These additional degrees of built-in "freedom" just give you *meaningless* choices. They don't make any *real* difference to your ability to get the job done. Instead, they raise a *barrier* to creating tools. If somebody creates a tool to do something with manifest files, you can't use it unless you and they have already agreed to make the *same* choices about these superfluous options, or else the tool maker has to support all possible options - and sometimes, a useful tool that supports all the options just isn't *possible*. See, it wouldn't matter if the arbitrary choice was that you have to generate MANIFEST instead of MANIFEST.in. You'd still have the same flexibility. The important thing in these arbitrary choices is that *we pick one*. That's why we have a BDFL for the language - we all argue for a particular surface syntax, and then he picks one, and we all move on. The distutils needs a BDFL to pick between all the mutually incompatible but semantically indistinguishable surface syntaxes for how to build and package something. >Again, not everybody is using distribution processes >built around CVS or Subversion. Left aside that there >are quite a few other SCM tools out there, you also have >the case where you create distributions from plain >directories (which is what MANIFEST.in and MANIFEST >are targetting). The source control is a supplement to the "add_defaults" of the sdist process, not a replacement for MANIFEST.in. Some users need to add non-source-controlled files, for example. But *simple things should be simple*, and treating source control info as part of the defaults makes them work simply for most people. And if you don't want the defaults, there should be only one way to turn them off - by excluding files in MANIFEST.in, not by a command line option. > > I'm also thinking that most of the MANIFEST logic could and should move to > > the Distribution class, since the data will be used by multiple > > commands. Thus, the sdist command could just ask the Distribution for the > > MANIFEST and get it, as would the commands that copy package data files to > > the build directory. > >Wait: MANIFEST defines what goes into the sdist - not an >arbitrary (binary) distribution. I'm talking about the include_package_data option. >It would certainly make sense to have the MANIFEST[.in] files >automatically be added as default in sdist.py and I'd be >+1 on that (even though it never was an issue for me as I >always include them in the MANIFEST file). Interesting; I could've sworn that you were one of the people who told somebody it would be "too much magic" to include this. But whatever the case, I'm glad you don't oppose it now. > > --no-defaults would be ignored, except for a warning. If you don't want > > the defaults, you can always start your MANIFEST.in with an exclude > pattern > > to exclude absolutely everything already included. There shouldn't be two > > ways to do the same thing, especially not one that you can use on the > > command line to mess things up in a non-repeatable fashion! Likewise > > --no-prune, because that's a similar recipe for disaster. > >These options are meant for people who don't have a >MANIFEST.in file to begin with or just quickly want to >build an sdist with parts of the whole distribution or >an extended version (e.g. for testing or upgrading). Sure - and there are plenty of things I can leave some room for play in. For example, I could simply revert to old behaviors when you use any non-default options. I could make a separate MANIFEST.setuptools file, etc. But these things add complexity, so I want to know who *actually* needs them, and can't trivially work around their absence. I'd rather briefly inconvenience distutils mavens like you, than continue to stump and frustrate the hundreds of people who just don't get why it's all so damn complicated. > > Heck, most of the distutils' flaws lie in their extreme versatility. > >That comment is just silly: distutils is so powerful because >of its versatility. You wouldn't have been able to build setuptools >without this versatility. You're confusing a well-factored framework with user-level versatility. Using variables instead of hardcoding filenames internally is a very good idea. Exposing those variables for users to change (in the absence of concrete use cases), however, is just bad UI design and a lack of social awareness. >That's your point of view - I've never had a hard time >adjusting distutils to whatever I wanted it to do. After >you get used to the way things are handled in distutils, >extending it is often enough really easy and would be >much harder in your One Obvious Way to do it (unless >you had a time-machine, zoom to 2042 and then take >all possibly ways to build distributions into account >on your way back to 2005 ;-). And your point of view is missing the part where everybody else isn't a distutils expert like you or I, and unlike you or I, has *no interest whatsoever in becoming one*. Simple things should be simple, and distributing most packages shouldn't be rocket science. In particular, there should be a gentle learning curve from "distribute one module" to "complex distribution with autogenerated bits not in source control". And, the path for *how* you do those things should be laid out. There are plenty of things that *are* Obvious use cases, but for which the Distutils Way is not obvious. It's always *possible* to customize via subclassing, and I'm not suggesting that be disallowed. But it shouldn't be necessary for the Obvious Way, and should be *required* for any deviation from the Obvious Way. If you're going to deviate, you should be *aware* that you're on your own, and parting ways with the larger community. You should be aware that you are potentially isolating yourself from the use of community tools based on that Way. Currently, you can never be sure, because there *is* no Way. Everybody has their own, and the result is chaos. Ironically, although the Perl community's language philosophy is "more than one way to do it", their build and distribution philosophy seems to be that there's not merely one obvious way to do it, there's *exactly* one way to do it. And *that* is the real reason why Perl has always been ahead of Python in readily-available libraries. The Perl distribution culture reflects the idea that build tools are for sharing software with the community, not a framework for creating private build systems. From robin at jessikat.fsnet.co.uk Wed Nov 16 18:55:26 2005 From: robin at jessikat.fsnet.co.uk (Robin Becker) Date: Wed, 16 Nov 2005 17:55:26 +0000 Subject: [Distutils] subway setup.py failure Message-ID: <437B728E.6010602@chamonix.reportlab.co.uk> I'm getting this strangness from the subway setup script. Installed C:\Python\Lib\site-packages\FormEncode-0.3-py2.4.egg Processing dependencies for FormEncode>=0.2.2 Searching for CherryPy>=2.1.0 Reading http://www.python.org/pypi/CherryPy/ Reading http://www.cherrypy.org Reading http://sourceforge.net/project/showfiles.php?group_id=56099 Best match: CherryPy 2.1.0-rc2 Downloading http://prdownloads.sourceforge.net/cherrypy/CherryPy-2.1.0-rc2.tar.gz?download error: Unexpected HTML page found at http://prdownloads.sourceforge.net/cherrypy/CherryPy-2.1.0-rc2.tar.gz?download It seems that easy_install isn't picking up from SF right now and in addition it hasn't noticed that the latest version is now 2.1.0. I tried a manual installation of the cherrypy as follows C:\Tmp\subway_svn\trunk>\python\Scripts\easy_install "CherryPy==2.1.0" Searching for CherryPy==2.1.0 Reading http://www.python.org/pypi/CherryPy/ Reading http://www.cherrypy.org Reading http://sourceforge.net/project/showfiles.php?group_id=56099 Best match: CherryPy 2.1.0 Downloading http://prdownloads.sourceforge.net/cherrypy/CherryPy-2.1.0.win32.exe?download error: Unexpected HTML page found at http://prdownloads.sourceforge.net/cherrypy/CherryPy-2.1.0.win32.exe?download an it seems I have to try an exact download path to get things right eg C:\Tmp\subway_svn\trunk>\python\Scripts\easy_install http://heanet.dl.sourceforge.net/sourceforge/cherrypy/CherryPy-2.1. 0.tar.gz Downloading http://heanet.dl.sourceforge.net/sourceforge/cherrypy/CherryPy-2.1.0.tar.gz Processing CherryPy-2.1.0.tar.gz Running CherryPy-2.1.0\setup.py -q bdist_egg --dist-dir c:\temp\easy_install-uu1h8w\CherryPy-2.1.0\egg-dist-tmp-jkracy zip_safe flag not set; analyzing archive contents... cherrypy._cphttptools: module references __file__ cherrypy.lib.autoreload: module references __file__ cherrypy.lib.covercp: module references __file__ cherrypy.lib.cptools: module references __file__ cherrypy.lib.profiler: module references __file__ cherrypy.test.test: module references __file__ cherrypy.test.test_core: module references __file__ cherrypy.test.test_static_filter: module references __file__ cherrypy.tutorial.tut09_files: module references __file__ cherrypy.tutorial.tut10_http_errors: module references __file__ Adding CherryPy 2.1.0 to easy-install.pth file Installed C:\Python\Lib\site-packages\CherryPy-2.1.0-py2.4.egg Processing dependencies for CherryPy==2.1.0 -- Robin Becker From ianb at colorstudy.com Wed Nov 16 18:57:40 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 16 Nov 2005 11:57:40 -0600 Subject: [Distutils] subway setup.py failure In-Reply-To: <437B728E.6010602@chamonix.reportlab.co.uk> References: <437B728E.6010602@chamonix.reportlab.co.uk> Message-ID: <437B7314.2050003@colorstudy.com> Robin Becker wrote: > I'm getting this strangness from the subway setup script. SourceForge just changed there download pages, I think. After years of non-action, now they change. Blast you SourceForge! -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From dangoor at gmail.com Wed Nov 16 19:02:16 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Wed, 16 Nov 2005 13:02:16 -0500 Subject: [Distutils] subway setup.py failure In-Reply-To: <437B728E.6010602@chamonix.reportlab.co.uk> References: <437B728E.6010602@chamonix.reportlab.co.uk> Message-ID: <3f085ecd0511161002i2a3b89f3uf67035ecb84699a@mail.gmail.com> setuptools thinks that "-something" after a version number means that this is a "patchlevel" version. "2.1.0rc2" would look like a prelease, "2.1.0-rc2" looks like a post-release patch level, and thus appears higher than "2.1.0". We had to explicitly exclude the "-beta" and "-rc" versions. Kevin On 11/16/05, Robin Becker wrote: > I'm getting this strangness from the subway setup script. > > Installed C:\Python\Lib\site-packages\FormEncode-0.3-py2.4.egg > Processing dependencies for FormEncode>=0.2.2 > Searching for CherryPy>=2.1.0 > Reading http://www.python.org/pypi/CherryPy/ > Reading http://www.cherrypy.org > Reading http://sourceforge.net/project/showfiles.php?group_id=56099 > Best match: CherryPy 2.1.0-rc2 > Downloading > http://prdownloads.sourceforge.net/cherrypy/CherryPy-2.1.0-rc2.tar.gz?download > error: Unexpected HTML page found at > http://prdownloads.sourceforge.net/cherrypy/CherryPy-2.1.0-rc2.tar.gz?download > > > It seems that easy_install isn't picking up from SF right now and in addition it > hasn't noticed that the latest version is now 2.1.0. > > I tried a manual installation of the cherrypy as follows > > C:\Tmp\subway_svn\trunk>\python\Scripts\easy_install "CherryPy==2.1.0" > Searching for CherryPy==2.1.0 > Reading http://www.python.org/pypi/CherryPy/ > Reading http://www.cherrypy.org > Reading http://sourceforge.net/project/showfiles.php?group_id=56099 > Best match: CherryPy 2.1.0 > Downloading > http://prdownloads.sourceforge.net/cherrypy/CherryPy-2.1.0.win32.exe?download > error: Unexpected HTML page found at > http://prdownloads.sourceforge.net/cherrypy/CherryPy-2.1.0.win32.exe?download > > an it seems I have to try an exact download path to get things right eg > > C:\Tmp\subway_svn\trunk>\python\Scripts\easy_install > http://heanet.dl.sourceforge.net/sourceforge/cherrypy/CherryPy-2.1. > 0.tar.gz > Downloading > http://heanet.dl.sourceforge.net/sourceforge/cherrypy/CherryPy-2.1.0.tar.gz > Processing CherryPy-2.1.0.tar.gz > Running CherryPy-2.1.0\setup.py -q bdist_egg --dist-dir > c:\temp\easy_install-uu1h8w\CherryPy-2.1.0\egg-dist-tmp-jkracy > zip_safe flag not set; analyzing archive contents... > cherrypy._cphttptools: module references __file__ > cherrypy.lib.autoreload: module references __file__ > cherrypy.lib.covercp: module references __file__ > cherrypy.lib.cptools: module references __file__ > cherrypy.lib.profiler: module references __file__ > cherrypy.test.test: module references __file__ > cherrypy.test.test_core: module references __file__ > cherrypy.test.test_static_filter: module references __file__ > cherrypy.tutorial.tut09_files: module references __file__ > cherrypy.tutorial.tut10_http_errors: module references __file__ > Adding CherryPy 2.1.0 to easy-install.pth file > > Installed C:\Python\Lib\site-packages\CherryPy-2.1.0-py2.4.egg > Processing dependencies for CherryPy==2.1.0 > -- > Robin Becker > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Kevin Dangoor Author of the Zesty News RSS newsreader email: kid at blazingthings.com company: http://www.BlazingThings.com blog: http://www.BlueSkyOnMars.com From mal at egenix.com Wed Nov 16 19:03:41 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 16 Nov 2005 19:03:41 +0100 Subject: [Distutils] MANIFEST destiny :) In-Reply-To: <437B5E6D.70805@colorstudy.com> References: <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> <437B0ECB.5030802@egenix.com> <437B5E6D.70805@colorstudy.com> Message-ID: <437B747D.506@egenix.com> Ian Bicking wrote: > M.-A. Lemburg wrote: > >>> * Always, always, always build MANIFEST, and always include both the >>> MANIFEST file and MANIFEST.in (if present) in the source distribution. >> >> >> >> -1 on always building MANIFEST. >> >> This would miss the point of managing MANIFEST files >> independently of your package files, e.g. using >> Makefiles or other tools dealing with file dependencies, >> checkouts, etc. > > > How about always build it, automatically, if some particular option is > not passed to setup()? This would only apply to packages using > setuptools, not distutils, and there's other things that existing > distutils packages might have to tweak to move to setuptools. Maybe it > would be as simple as a keyword argument that refers to a routine to > build the MANIFEST file; and if you build MANIFEST by hand, then you > just make the function a no-op. Why can't Philipp just add the explicit build command (--force-manifest) to his setuptools ? > Also, disable all command-line options that control when/if/how MANIFEST > is generated; these seem peculiar to me. MANIFEST generation seems like > package metadata, not something particular to your build, and > command-line options seem like a bad way to control that. It's a little > fuzzy, though, since command-line options and setup.cfg are equivalent, > and setup.cfg options often feel like package data. So... I don't know > what the middle ground is there. Either way, these aren't things that > seem like they should be regularly tweaked. I don't know, I don't have > any packages with complex (or even simple) build processes, so I only > know that the current system is mysterious to me, and almost but doesn't > always work without my intervention. Philipp can do the same to his setuptools - he'd just have to subclass the sdist command. > I think Phillip also mentioned using MANIFEST for package_data? At > least insofar as the packages and the MANIFEST intersect, this seems > like a good idea. Again, this depends on the distribution format. It may make a lot of sense for setuptools, but there are other formats where the source distribution file list and the files to be included in a binary distribution are two completely separate things. For example, bdist_rpm actually installs the package to a temporary directory and then takes the list of installed files as basis for the binary distribution file list. Since the installation can depend on other external factors (e.g. libs being available or not), the file list is not necessarily static, nor does it have to match the file layout you have in MANIFEST (since the install commands may move files to different target directories). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 16 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From pje at telecommunity.com Wed Nov 16 19:06:17 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 16 Nov 2005 13:06:17 -0500 Subject: [Distutils] MANIFEST destiny :) In-Reply-To: <437B5E6D.70805@colorstudy.com> References: <437B0ECB.5030802@egenix.com> <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> <437B0ECB.5030802@egenix.com> Message-ID: <5.1.1.6.0.20051116125407.0210e200@mail.telecommunity.com> At 10:29 AM 11/16/2005 -0600, Ian Bicking wrote: >M.-A. Lemburg wrote: >>>* Always, always, always build MANIFEST, and always include both the >>>MANIFEST file and MANIFEST.in (if present) in the source distribution. >> >>-1 on always building MANIFEST. >>This would miss the point of managing MANIFEST files >>independently of your package files, e.g. using >>Makefiles or other tools dealing with file dependencies, >>checkouts, etc. > >How about always build it, automatically, if some particular option is not >passed to setup()? This would only apply to packages using setuptools, >not distutils, and there's other things that existing distutils packages >might have to tweak to move to setuptools. Especially in cases where they have complex custom builds. People with stuff as sophisticated as SciPy, mx* stuff, Twisted, Zope, are not people I expect to move to setuptools any time soon. (I've heard rumors about Zope using it for Products, but reasonably speaking I think it's going to take some time before the Zope core could be moved to it, probably by adapting the zpkg tools to generate setuptools-based setup scripts.) My thinking here is more about the long term, and mostly setuptools is "disruptive technology" anyway, because it's shifting the balance of distribution power towards smaller and simpler tools. One of the reasons complex packages are dominant today is because they reduce the number of things you have to install. Setuptools makes the number of things you have to install a lot less relevant, so in the "new world", smaller packages will tend to dominate. In order to effect that shift, it's got to become a lot easier for a complete distutils novice to turn out a package. > Maybe it would be as simple as a keyword argument that refers to a > routine to build the MANIFEST file; and if you build MANIFEST by hand, > then you just make the function a no-op. Now *that's* an interesting idea. I'll have to give that one some thought. Thanks! >Also, disable all command-line options that control when/if/how MANIFEST >is generated; these seem peculiar to me. MANIFEST generation seems like >package metadata, not something particular to your build, and command-line >options seem like a bad way to control that. It's a little fuzzy, though, >since command-line options and setup.cfg are equivalent, and setup.cfg >options often feel like package data. So... I don't know what the middle >ground is there. Either way, these aren't things that seem like they >should be regularly tweaked. I don't know, I don't have any packages with >complex (or even simple) build processes, so I only know that the current >system is mysterious to me, and almost but doesn't always work without my >intervention. Marc's explanation of these options is actually pretty reasonable as a projected use case. If there are any *actual* field uses, I'll consider trying to support them. Of course, if those use cases are ones where there's no reasonable likelihood of the project moving to setuptools, it doesn't make sense to support them in setuptools. It would be more a matter of, if some o the relevant setuptools stuff makes it back into the distutils, it would need that additional degree of support. >I think Phillip also mentioned using MANIFEST for package_data? At least >insofar as the packages and the MANIFEST intersect, this seems like a good >idea. Yeah, it would make your find_package_data() less necessary. The reason I haven't put it in yet, is because I've been pondering the issue of how to integrate these notions of "the distribution contents". While I can envision scenarios where you'd want certain data files to get built on the target user's machine, there's still the option of manually or programmatically specifying those files in package_data. (That is, I intend to make include_package_data be supplemental to the existing package_data option, not a replacement for it, except in the sense that for simple cases you won't need package_data any more.) From pje at telecommunity.com Wed Nov 16 19:15:59 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 16 Nov 2005 13:15:59 -0500 Subject: [Distutils] subway setup.py failure In-Reply-To: <437B728E.6010602@chamonix.reportlab.co.uk> Message-ID: <5.1.1.6.0.20051116130703.03219f90@mail.telecommunity.com> At 05:55 PM 11/16/2005 +0000, Robin Becker wrote: >I'm getting this strangness from the subway setup script. > >Installed C:\Python\Lib\site-packages\FormEncode-0.3-py2.4.egg >Processing dependencies for FormEncode>=0.2.2 >Searching for CherryPy>=2.1.0 >Reading http://www.python.org/pypi/CherryPy/ >Reading http://www.cherrypy.org >Reading http://sourceforge.net/project/showfiles.php?group_id=56099 >Best match: CherryPy 2.1.0-rc2 >Downloading >http://prdownloads.sourceforge.net/cherrypy/CherryPy-2.1.0-rc2.tar.gz?download >error: Unexpected HTML page found at >http://prdownloads.sourceforge.net/cherrypy/CherryPy-2.1.0-rc2.tar.gz?download > > >It seems that easy_install isn't picking up from SF right now and in >addition it >hasn't noticed that the latest version is now 2.1.0. Setuptools reads '2.1.0-rc2' as "2.1.0 with patch/branch rc2", not "2.1.0 release candidate 2" (which is spelled '2.1.0rc2' or '2.1.0.rc2'). In other words 2.1.0rc2 < 2.1.0 < 2.1.0-rc2. (The most popular use of '-' in version numbers is to separate a patch level or distro-specific branch/port, and such versions are *after* the version that they amend.) As for the SF issue, the code in setuptools/package_index.py is: # Check for a SourceForge header elif sf_url: if re.search(r'^', line, re.I): continue # skip first line elif re.search(r'Select a Mirror for File:',line): You can change this as a temporary workaround: # Check for a SourceForge header elif sf_url: if True: I'll try to get a more robust fix out promptly, and I'll try to make it resilient against future skin changes by SourceForge. From pje at telecommunity.com Wed Nov 16 19:19:54 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 16 Nov 2005 13:19:54 -0500 Subject: [Distutils] [TurboGears] hardcoded paths in tg-admin.exe and easy_install.exe? In-Reply-To: <437AB2DC.2050106@colorstudy.com> References: <1132114166.686802.158620@o13g2000cwo.googlegroups.com> <1132114166.686802.158620@o13g2000cwo.googlegroups.com> Message-ID: <5.1.1.6.0.20051116012849.01f2ef98@mail.telecommunity.com> At 10:17 PM 11/15/2005 -0600, Ian Bicking wrote: >rtilley at vt.edu wrote: > > Perhaps this has been mentioned already. So, forgive me if it has. > > > > I install Python into 'C:/Program Files/Python24' as I do not like to > > clutter up the C:/ portion of my file system. After installing > > turbogears and trying to begin the tutorial, I get the below error when > > I do this 'tg-admin quickstart': > > > > Program: can't open file 'Files\Python24\python.exe' : [Error 2] No > > such file or directory > > > > The same thing happens when I try to run easy_install.exe > > > > The .py versions of tg-admin and easy_install work fine... it's just > > the .exe versions that crash. I have named this the 'hard-coded python > > path bug'. Actually, it's not hardcoding that's the issue. It's that something somewhere is using spaces to parse the #! line. I'll look into this, as I'm not sure whether it's in the setuptools #! line code, or in the .exe's #! parser. > > Also, the instructions about adding 'C:\Python24' and > > 'C:\Python24\Scripts' to the PATH variable should be adjusted to > > wherever the user has actually installed Python ;) I make the assumption that somebody smart enough to be able to change the location should be smart enough to figure out how to adjust the instructions. :) No sense making the directions more complicated for everybody else. From pje at telecommunity.com Wed Nov 16 19:42:39 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 16 Nov 2005 13:42:39 -0500 Subject: [Distutils] subway setup.py failure In-Reply-To: <437B7314.2050003@colorstudy.com> References: <437B728E.6010602@chamonix.reportlab.co.uk> <437B728E.6010602@chamonix.reportlab.co.uk> Message-ID: <5.1.1.6.0.20051116132855.01f1c7e8@mail.telecommunity.com> At 11:57 AM 11/16/2005 -0600, Ian Bicking wrote: >Robin Becker wrote: > > I'm getting this strangness from the subway setup script. > >SourceForge just changed there download pages, I think. After years of >non-action, now they change. Blast you SourceForge! Here's a patch that fixes it, and should also be compatible with the old way if they change it back. As you can see, they significantly changed the original page format, and also the HTML syntax of their "use_mirror" URLs. Index: setuptools/package_index.py =================================================================== --- setuptools/package_index.py (revision 41401) +++ setuptools/package_index.py (working copy) @@ -463,16 +463,14 @@ return self._download_svn(url, filename) # Check for a SourceForge header elif sf_url: - if re.search(r'^<HTML><HEAD>', line, re.I): - continue # skip first line - elif re.search(r'<TITLE>Select a Mirror for File:',line): - # Sourceforge mirror page - page = file.read() - file.close() + page = ''.join(list(file)) + if '?use_mirror=' in page: + file.close(); os.unlink(filename) return self._download_sourceforge(url, page, tmpdir) break # not an index page file.close() + os.unlink(filename) raise DistutilsError("Unexpected HTML page found at "+url) def _download_svn(self, url, filename): @@ -490,12 +488,14 @@ def warn(self, msg, *args): log.warn(msg, *args) + + def _download_sourceforge(self, source_url, sf_page, tmpdir): """Download package from randomly-selected SourceForge mirror""" self.debug("Processing SourceForge mirror page") - mirror_regex = re.compile(r'HREF=(/.*?\?use_mirror=[^>]*)') + mirror_regex = re.compile(r'HREF="?(/.*?\?use_mirror=[^">]*)', re.I) urls = [m.group(1) for m in mirror_regex.finditer(sf_page)] if not urls: raise DistutilsError( From pobrien at orbtech.com Wed Nov 16 20:00:02 2005 From: pobrien at orbtech.com (Patrick K. O'Brien) Date: Wed, 16 Nov 2005 13:00:02 -0600 Subject: [Distutils] MANIFEST destiny :) In-Reply-To: <5.1.1.6.0.20051116114559.01f47bd8@mail.telecommunity.com> References: <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> <5.1.1.6.0.20051116114559.01f47bd8@mail.telecommunity.com> Message-ID: <437B81B2.8040901@orbtech.com> Phillip J. Eby wrote: <major snippage> > And your point of view is missing the part where everybody else isn't a > distutils expert like you or I, and unlike you or I, has *no interest > whatsoever in becoming one*. Simple things should be simple, and > distributing most packages shouldn't be rocket science. In particular, > there should be a gentle learning curve from "distribute one module" to > "complex distribution with autogenerated bits not in source control". And, > the path for *how* you do those things should be laid out. +1 to everything you said in this post, the above snippet being a pretty good synopsis. And thank you, Phillip, for championing this effort to make Python package distribution simpler and easier for those of us who really don't care how it all works, just that it do so painlessly. > Ironically, although the Perl community's language philosophy is "more than > one way to do it", their build and distribution philosophy seems to be that > there's not merely one obvious way to do it, there's *exactly* one way to > do it. And *that* is the real reason why Perl has always been ahead of > Python in readily-available libraries. The Perl distribution culture > reflects the idea that build tools are for sharing software with the > community, not a framework for creating private build systems. Sadly, I think there is a lot of truth in this observation. -- Patrick K. O'Brien Orbtech http://www.orbtech.com Schevo http://www.schevo.org From pje at telecommunity.com Wed Nov 16 20:11:12 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 16 Nov 2005 14:11:12 -0500 Subject: [Distutils] [TurboGears] hardcoded paths in tg-admin.exe and easy_install.exe? In-Reply-To: <5.1.1.6.0.20051116012849.01f2ef98@mail.telecommunity.com> References: <437AB2DC.2050106@colorstudy.com> <1132114166.686802.158620@o13g2000cwo.googlegroups.com> <1132114166.686802.158620@o13g2000cwo.googlegroups.com> Message-ID: <5.1.1.6.0.20051116135718.02088e58@mail.telecommunity.com> At 01:19 PM 11/16/2005 -0500, Phillip J. Eby wrote: >At 10:17 PM 11/15/2005 -0600, Ian Bicking wrote: > >rtilley at vt.edu wrote: > > > Perhaps this has been mentioned already. So, forgive me if it has. > > > > > > I install Python into 'C:/Program Files/Python24' as I do not like to > > > clutter up the C:/ portion of my file system. After installing > > > turbogears and trying to begin the tutorial, I get the below error when > > > I do this 'tg-admin quickstart': > > > > > > Program: can't open file 'Files\Python24\python.exe' : [Error 2] No > > > such file or directory > > > > > > The same thing happens when I try to run easy_install.exe > > > > > > The .py versions of tg-admin and easy_install work fine... it's just > > > the .exe versions that crash. I have named this the 'hard-coded python > > > path bug'. > >Actually, it's not hardcoding that's the issue. It's that something >somewhere is using spaces to parse the #! line. I'll look into this, as >I'm not sure whether it's in the setuptools #! line code, or in the .exe's >#! parser. Okay, it's more subtle than that. The problem is that something about how Python is spawned by the launcher, is causing Python to think that the space in its filename means that the part after the space is an argument. (I'd like to have an argument, please!) So I'm not quite sure how to fix it, exactly, particularly because I don't know if it's Python that's the problem, or the way the launcher is invoking Python. :( From cmlenz at gmx.de Wed Nov 16 20:21:55 2005 From: cmlenz at gmx.de (Christopher Lenz) Date: Wed, 16 Nov 2005 20:21:55 +0100 Subject: [Distutils] MANIFEST destiny :) In-Reply-To: <5.1.1.6.0.20051116114559.01f47bd8@mail.telecommunity.com> References: <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> <5.1.1.6.0.20051116114559.01f47bd8@mail.telecommunity.com> Message-ID: <F8262AA1-40D4-418A-80E1-ECC2722D9660@gmx.de> Am 16.11.2005 um 18:53 schrieb Phillip J. Eby: >> Again, not everybody is using distribution processes >> built around CVS or Subversion. Left aside that there >> are quite a few other SCM tools out there, you also have >> the case where you create distributions from plain >> directories (which is what MANIFEST.in and MANIFEST >> are targetting). > > The source control is a supplement to the "add_defaults" of the sdist > process, not a replacement for MANIFEST.in. Some users need to add > non-source-controlled files, for example. But *simple things > should be > simple*, and treating source control info as part of the defaults > makes > them work simply for most people. And if you don't want the defaults, > there should be only one way to turn them off - by excluding files in > MANIFEST.in, not by a command line option. I'm not sure I understand this proposal completely, but I don't think it's a good idea to have a project's build/setup process rely on having version control meta-data around. Wouldn't that mean that the built package would be incorrect when the setup is run from an `svn export`ed (or otherwise cleaned up) copy? Cheers, Chris -- Christopher Lenz cmlenz at gmx.de http://www.cmlenz.net/ From pje at telecommunity.com Wed Nov 16 20:24:51 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 16 Nov 2005 14:24:51 -0500 Subject: [Distutils] Setuptools 0.6a8 update to fix SourceForge problems Message-ID: <5.1.1.6.0.20051116141925.031d2a38@mail.telecommunity.com> FYI, SourceForge changed their download pages such that older versions of setuptools can no longer recognize them and follow the links to the "real" download pages. I'm releasing 0.6a8 early in order to address these issues. You can update your setuptools with: easy_install -U setuptools If you have an 'ez_setup' subdirectory in a Subversion-based project, you can use 'svn update ez_setup' to ensure you're requiring the latest version. Unfortunately, there are still outstanding bugs, specifically the "space in python.exe filename" problem, and the issue with creating RPMs from source distributions that use the source-control-driven MANIFEST feature. However, these bugs are also in older versions, so they aren't new to 0.6a8. And 0.6a8 also fixes a lot of previous issues with doing Subversion-based development, so it's a good upgrade to get anyway. Sorry for the inconvenience; the original SourceForge processing code was a user contribution and I didn't really vet it for robustness in the face of SourceForge changes. The new parsing code could still be broken by future SF changes, but it's now less likely. From pje at telecommunity.com Wed Nov 16 20:39:56 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 16 Nov 2005 14:39:56 -0500 Subject: [Distutils] MANIFEST destiny :) In-Reply-To: <F8262AA1-40D4-418A-80E1-ECC2722D9660@gmx.de> References: <5.1.1.6.0.20051116114559.01f47bd8@mail.telecommunity.com> <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> <5.1.1.6.0.20051116114559.01f47bd8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051116143321.03b59050@mail.telecommunity.com> At 08:21 PM 11/16/2005 +0100, Christopher Lenz wrote: >Am 16.11.2005 um 18:53 schrieb Phillip J. Eby: > >> Again, not everybody is using distribution processes > >> built around CVS or Subversion. Left aside that there > >> are quite a few other SCM tools out there, you also have > >> the case where you create distributions from plain > >> directories (which is what MANIFEST.in and MANIFEST > >> are targetting). > > > > The source control is a supplement to the "add_defaults" of the sdist > > process, not a replacement for MANIFEST.in. Some users need to add > > non-source-controlled files, for example. But *simple things > > should be > > simple*, and treating source control info as part of the defaults > > makes > > them work simply for most people. And if you don't want the defaults, > > there should be only one way to turn them off - by excluding files in > > MANIFEST.in, not by a command line option. > >I'm not sure I understand this proposal completely, but I don't think >it's a good idea to have a project's build/setup process rely on >having version control meta-data around. Wouldn't that mean that the >built package would be incorrect when the setup is run from an `svn >export`ed (or otherwise cleaned up) copy? The MANIFEST file would be generated from that data, and then source distributions would contain the MANIFEST, which would then be read in place of the source control metadata when it's not available. There's a similar feature in setuptools now for tagging releases with SVN revision numbers; if you don't have the metadata available, setuptools looks in PKG-INFO to get the revision number that the source release was built from. You're right, however, in that 'svn export' doesn't work with either scheme, in that setuptools assumes the One Obvious Way to distribute source is by running "sdist" on your code base, not by making an export from your revision control. If you don't have that option (or don't use CVS or SVN), you don't get to use setuptools' convenience features, and have to do things the distutils way, with an explicit MANIFEST.in file. From bkc at murkworks.com Wed Nov 16 21:19:08 2005 From: bkc at murkworks.com (Brad Clements) Date: Wed, 16 Nov 2005 15:19:08 -0500 Subject: [Distutils] easy_install -U setuptools fails on Windows (workaround) In-Reply-To: <5.1.1.6.0.20051116141925.031d2a38@mail.telecommunity.com> Message-ID: <437B4DEC.26723.FA16656@bkc.murkworks.com> On 16 Nov 2005 at 14:24, Phillip J. Eby wrote: > these issues. You can update your setuptools with: > > easy_install -U setuptools > Except on windows, the easy_install binary is in-use: E:\>easy_install -U setuptools Searching for setuptools Reading http://www.python.org/pypi/setuptools/ Reading http://peak.telecommunity.com/DevCenter/setuptools Best match: setuptools 0.6a8 Downloading http://cheeseshop.python.org/packages/2.4/s/setuptools/setuptools-0.6a8-py2.4.egg#md5=799018f2894f14c9f8bcb2b34e69b391 Processing setuptools-0.6a8-py2.4.egg creating d:\python24\lib\site-packages\setuptools-0.6a8-py2.4.egg Extracting setuptools-0.6a8-py2.4.egg to d:\python24\lib\site-packages Removing setuptools 0.6a6 from easy-install.pth file Adding setuptools 0.6a8 to easy-install.pth file Installing easy_install-script.py script to d:\python24\Scripts Installing easy_install.exe script to d:\python24\Scripts error: d:\python24\Scripts\easy_install.exe: Permission denied (now copy easy_install .exe and .py to a temp location) E:\>f:\temp\easy_install.exe -U setuptools Searching for setuptools Reading http://www.python.org/pypi/setuptools/ Reading http://peak.telecommunity.com/DevCenter/setuptools Best match: setuptools 0.6a8 Downloading http://cheeseshop.python.org/packages/2.4/s/setuptools/setuptools-0. 6a8-py2.4.egg#md5=799018f2894f14c9f8bcb2b34e69b391 Processing setuptools-0.6a8-py2.4.egg removing 'd:\python24\lib\site-packages\setuptools-0.6a8-py2.4.egg' (and everything under it) creating d:\python24\lib\site-packages\setuptools-0.6a8-py2.4.egg Extracting setuptools-0.6a8-py2.4.egg to d:\python24\lib\site-packages setuptools 0.6a8 is already the active version in easy-install.pth Installing easy_install-script.py script to d:\python24\Scripts Installing easy_install.exe script to d:\python24\Scripts Installed d:\python24\lib\site-packages\setuptools-0.6a8-py2.4.egg Processing dependencies for setuptools -- Brad Clements, bkc at murkworks.com (315)268-1000 http://www.murkworks.com AOL-IM or SKYPE: BKClements From mal at egenix.com Thu Nov 17 00:19:07 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 17 Nov 2005 00:19:07 +0100 Subject: [Distutils] MANIFEST destiny :) In-Reply-To: <5.1.1.6.0.20051116114559.01f47bd8@mail.telecommunity.com> References: <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> <5.1.1.6.0.20051116114559.01f47bd8@mail.telecommunity.com> Message-ID: <437BBE6B.3000609@egenix.com> Hi Phillip, In general, I think you are having a different focus here than what distutils is trying to be and that's perfectly OK - you can implement all these nice strategies and automated decisions into your setuptools. I just don't see a benefit in stripping down the framework distutils itself. If people choose setuptools as front-end to distutils that's a perfectly good choice and one I'd like to encourage. Note that distutils would benefit a lot from more support for e.g. InnoSetup, NSIS, native packages for Solaris, HP-UX, Debian. There are a few projects out there trying to add this support, but few have stepped forward to suggest integration with the core framework. The issues around MANIFEST[.in] that you present are really minor compared to not being able to build e.g. Debian packages out of the box and without too much user interaction. Phillip J. Eby wrote: > At 11:49 AM 11/16/2005 +0100, M.-A. Lemburg wrote: > >> I don't understand what you mean with "no-go" - the current >> system works just fine if you include the MANIFEST file >> in the sdist. > > > But it's not included, and you have to know to include it - and people > who previously requested here that MANIFEST and MANIFEST.in be included > in the manifest were shot down under the claim that including these > defaults would be "too much magic". I don't see that as a big problem. Why not include it per default like README and the others ?! >> > The reason I've been avoiding adding this feature, however, is >> because of >> > the first issue; when you make an sdist, you lose that additional >> metadata, >> > so it would become impossible to build *any* binary from an sdist, >> not just >> > RPMs. Until recently, that issue seemed insurmountable. >> >> That's simply not true. You have to include the MANIFEST file >> in the sdist and then everything is fine. > > > I'm speaking about the context of setuptools, where the typical user has > no MANIFEST.in, because they're using Subversion or CVS and therefore > don't need one. Right, and that's a different context than the one needed for the core framework distutils itself. >> > * Change the MANIFEST format to be platform-independent (currently it >> > contains OS-specific path separators) >> >> -0.5 >> >> You are missing an important point: MANIFEST files can be >> build using tools outside distutils and external package >> building tools may require these to be platform dependent. > > > One reason I posted to find out what specific uses people actually had > for such things. Do the uses actually exist? What are they used for? eGenix for one uses its own file selection mechanism (mostly for historical reasons because we had our own packaging system before we switched to distutils). The MANIFEST.in format is not everybody's favorite, so I expect others to use more common tools such as e.g. Unix find, sed or even just a plain text editor. We also manage the MANIFEST files using Makefiles which take care of the build process, do the checkouts, copies, rsyncs, etc. needed for remote builds. > One important point: on POSIX-y platforms (i.e. virtually everything but > Windows), there's no difference between distutils paths and system > paths. So, what external tools build or read MANIFEST files on Windows, > or any other platform that doesn't accept '/' as a separator? > > Also, '/' *is* a valid path separator on Windows, so if your MANIFEST > processing is done in Python, a platform-independent format won't have > any effect. (Indeed, the only reason to actually *change* the format is > so that sdist files built *on Windows* would be usable on other > platforms! Other platforms would never see a difference.) > > I know that in principle, somebody somewhere may have some tool that > would break if MANIFEST changed it's format. My question is, who and > how many? If there's some widely-used tool that does this, then it's > reasonable to take it into account. If it's just a theoretical > possibility, it's not worth much concern. If there are a handful of > people with a hard-to-change setup, it's somewhere in between. The question should not be: how many setups can I break ? It should be: what do we gain by using a single format, e.g. the posix one and how can we avoid breakage ? Note that distutils knows how to transform posix file names into platform dependent ones. >> > * Always, always, always build MANIFEST, and always include both the >> > MANIFEST file and MANIFEST.in (if present) in the source distribution. >> >> -1 on always building MANIFEST. >> >> This would miss the point of managing MANIFEST files >> independently of your package files, e.g. using >> Makefiles or other tools dealing with file dependencies, >> checkouts, etc. > > > Please point me to some examples of this, especially one that can't > simply generate a MANIFEST.in instead. See above. Tools like "find" are simply much more complete in terms of file selection. It is also sometimes necessary to massage the paths a bit, using e.g. sed. >> > * Disable all the options that allow user control over MANIFEST >> generation, >> > including pruning, defaults, changing the filenames, etc. >> >> -1 >> >> Again, you are forgetting that MANIFEST files serve a purpose >> and are external to the distutils process for a reason. You >> are free to have distutils build your MANIFEST files from >> MANIFEST.in files, have distutils command auto-generate them, >> or use external programs triggered by Makefiles or similar >> distribution building processes to generate them. > > > Examples? You keep saying that people *can* do these things, but that's > not anything like the same as saying that any significant number of > people *actually* do them. Frankly, most people I've encountered who > are doing Python software development don't know how to get a basic > setup.py to work right, and feel the distutils are way too complicated, > underdocumented, or just plain broken, because they don't feel like they > can control them. It's underdocumented, yes, but getting a simple setup.py to work is really not all that complicated - and this is underlined by the fact that most Python packages nowadays are distributed as distutils-based packages. > OSAF, for example, has some developers who are very smart about creating > build processes. Smart enough to be *able* to do the kinds of things > you're describing. But they sure as heck don't use the distutils to > *actually* do them, because from their perspective the distutils is a > big pile of broken undocumentedness. If the distutils are so > frustrating to such smart developers, there's something wrong. The code itself is well documented and easy to read. It should be well within range of every average Python programmer. Furthermore, you only need to dig into distutils if you plan to extend or modify its default functionality in some way. The casual user does not have to read the sources. > Thus, I find these super-custom processes you're talking about highly > implausible, because the only people who could implement them are the > people with a strong knowledge of the distutils -- an incredibly rare > breed of person, in other words. I was only talking about special ways to build the MANIFEST files, not "super-custom" processes. No idea where you got that impression from. > Most people just want this stuff to > work, and they don't want to have to learn *how* it works. They have > better things to do with their time. They want a build *tool*, not a > build library or build framework. distutils does work for these people. The many existing packages using distutils is proof enough, IMHO. Of course, you can always do better if you have more specific requirements such as your CVS/Subversion integration. Those people should then use your setuptools front-end. I don't see that as a problem. > Most people don't know MANIFEST exists, until they get bitten by the > need to have one, or by it being out of date. Hell, look at how many > packages on PyPI and the Vaults aren't even packaged with distutils at > all! Not that many... :-) > For many people, it's clearly easier to just tarball your source > directory than to have to learn about this MANIFEST stuff. I agree that this feature is underdocumented, but changing the framework won't help with this: documentation patches is what we *really* need ! BTW, not many people need to have these MANIFEST files at all - distutils uses a built-in file finder based on these defaults: - README or README.txt - setup.py - test/test*.py - all pure Python modules mentioned in setup script - all C sources listed as part of extensions or C libraries in the setup script (doesn't catch C headers!) >> In your world, everything is done within distutils, so it's >> understandable that you'd like to get rid of the external >> nature of MANIFEST files, but please keep in mind that >> these features are being used and removing the logic would >> seriously break things for packagers relying on other >> mechanisms to build their MANIFEST files. > > > Please point me to these developers, and show me one that couldn't just > spend a few minutes making their tools generate a MANIFEST.in instead. > I'm suggesting that we present a very small number of highly-capable > people with a truly minor inconvenience, in order to make an extremely > large group of people happier by taking away something that invariably > bites them. See above. Carelessly overwriting hand-edited or otherwise generated files in a build process is simply bad design. If a MANIFEST file exists it should be left untouched. If no such file exists, but there's a MANIFEST.in exists, it should be rebuilt. If there's not MANIFEST.in, use a set of sane defaults determined by introspection of the setup.py details. This is what distutils does. >> > * Use the MANIFEST data (along with revision control info) not only for >> > producing source distributions, but also to determine what files >> should be >> > considered "package data", if the user passes an >> > 'include_package_data=True' keyword to setup(). >> >> Isn't that already the case ? I mean you can put anything >> you like into MANIFEST and it will be included in the sdist. > > > I'm talking about package data - a feature that was pioneered in > setuptools and added to the distutils in Python 2.4. The ability to > specify data files that are *installed* in a package directory. > > Specifically, I'm suggesting that users be able to replace the > package_data setup keyword with a simple include_package_data flag, so > that the MANIFEST data can be used to determine what data files to > install. This has nothing to do with sdist; I'm talking about being > able to unify the distutils' idea of what files are part of the > distribution, because for the simple cases that are 90% of projects, > that's a very useful thing to have. Perhaps you should then enhance the sdist way of finding suitable defaults - it currently does not take package_data files into account. MANIFEST is only used for source code distrbutions. I don't see how you can use it for anything else. See e.g. the way bdist_rpm works: it actually installs the package to find out which files are actually installed and then records all the files copied during that process - that's a very smart, future proof and flexible design. > For every SciPy and mxODBC and Twisted, there are easily a hundred > packages without anything like their level of complex build needs. I'm > talking about streamlining things for the simple packages, but there > isn't anything in what I'm proposing that keeps anybody from doing > complex things. I'm just saying we should remove *multiple* ways to do > the *same* complex thing. We should pick One Obvious Way to > *customize*, replacing multiple hooks with single hooks that allow the > same degree of customizability. I don't buy this: on one hand you are talking about simple packages (which don't need the MANIFEST files in the first place), on the other about hooks to adjust distutils' build process, something I'd group under more complex setups. > For example, if you have tools that generate something, right now you > could choose to generate either MANIFEST or MANIFEST.in. I'm suggesting > that we should choose for you, and say it's MANIFEST.in that you should > generate, since it's the more expressive and flexible of the two > formats. Similarly, you can currently choose what filenames MANIFEST > and MANIFEST.in have, and I'm suggesting that those be their only > names. If you need other files you certainly have the option of > copying, renaming, and using other existing file manipulation tools. > > These additional degrees of built-in "freedom" just give you > *meaningless* choices. They don't make any *real* difference to your > ability to get the job done. Instead, they raise a *barrier* to > creating tools. If somebody creates a tool to do something with > manifest files, you can't use it unless you and they have already agreed > to make the *same* choices about these superfluous options, or else the > tool maker has to support all possible options - and sometimes, a useful > tool that supports all the options just isn't *possible*. So your point is to make your life as setuptools author easier ? Why don't you just disable all these options in your setuptools front-end and hard-code the MANIFEST file names ? > See, it wouldn't matter if the arbitrary choice was that you have to > generate MANIFEST instead of MANIFEST.in. You'd still have the same > flexibility. The important thing in these arbitrary choices is that *we > pick one*. That's why we have a BDFL for the language - we all argue > for a particular surface syntax, and then he picks one, and we all move > on. The distutils needs a BDFL to pick between all the mutually > incompatible but semantically indistinguishable surface syntaxes for how > to build and package something. distutils is a loosly coupled framework of components. In such a framework, a basic design principle is to be able to decouple and recouple existing components. The only way to implement this is by making the components suitably independent and this is what was done in distutils. Note that adding user options to change certain assumptions or defaults does not count towards having "multiple ways to get something done" - it just gives the user a possiblity to adapt the framework to a particular need and on a case-by-case basis. Also note that it's not hard for setuptools or any other front-end to access these user options - just ask the component for them. >> Again, not everybody is using distribution processes >> built around CVS or Subversion. Left aside that there >> are quite a few other SCM tools out there, you also have >> the case where you create distributions from plain >> directories (which is what MANIFEST.in and MANIFEST >> are targetting). > > > The source control is a supplement to the "add_defaults" of the sdist > process, not a replacement for MANIFEST.in. Some users need to add > non-source-controlled files, for example. But *simple things should be > simple*, and treating source control info as part of the defaults makes > them work simply for most people. And if you don't want the defaults, > there should be only one way to turn them off - by excluding files in > MANIFEST.in, not by a command line option. If you don't want the defaults added, you are requesting a change in the way distutils works. Such a change should be done using the command line switch --no-defaults (or added to setup.cfg). MANIFEST.in OTOH is really only needed in case you plan to add non-standard files to your source distribution. You are not changing the way distutils itself works - just tell it to add a few more things that you might need or that you might not want in the distribution. >> > I'm also thinking that most of the MANIFEST logic could and should >> move to >> > the Distribution class, since the data will be used by multiple >> > commands. Thus, the sdist command could just ask the Distribution >> for the >> > MANIFEST and get it, as would the commands that copy package data >> files to >> > the build directory. >> >> Wait: MANIFEST defines what goes into the sdist - not an >> arbitrary (binary) distribution. > > > I'm talking about the include_package_data option. > > >> It would certainly make sense to have the MANIFEST[.in] files >> automatically be added as default in sdist.py and I'd be >> +1 on that (even though it never was an issue for me as I >> always include them in the MANIFEST file). > > > Interesting; I could've sworn that you were one of the people who told > somebody it would be "too much magic" to include this. But whatever the > case, I'm glad you don't oppose it now. > >> > --no-defaults would be ignored, except for a warning. If you don't >> want >> > the defaults, you can always start your MANIFEST.in with an exclude >> pattern >> > to exclude absolutely everything already included. There shouldn't >> be two >> > ways to do the same thing, especially not one that you can use on the >> > command line to mess things up in a non-repeatable fashion! Likewise >> > --no-prune, because that's a similar recipe for disaster. >> >> These options are meant for people who don't have a >> MANIFEST.in file to begin with or just quickly want to >> build an sdist with parts of the whole distribution or >> an extended version (e.g. for testing or upgrading). > > > Sure - and there are plenty of things I can leave some room for play > in. For example, I could simply revert to old behaviors when you use > any non-default options. I could make a separate MANIFEST.setuptools > file, etc. But these things add complexity, so I want to know who > *actually* needs them, and can't trivially work around their absence. > I'd rather briefly inconvenience distutils mavens like you, than > continue to stump and frustrate the hundreds of people who just don't > get why it's all so damn complicated. Look, nobody stops you from removing all these features in your front-end. distutils lets you do all this and that's what so great about it. My point is that you shouldn't try to strip down distutils itself just because you think it's hard work to support all these features in setuptools. It's not needed to strip down distutils for this reason as you can easily disable these options for anyone using your setuptools. As a result, both users of setuptools and straight distutils are happy. Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 16 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From pje at telecommunity.com Thu Nov 17 01:28:52 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 16 Nov 2005 19:28:52 -0500 Subject: [Distutils] MANIFEST destiny :) In-Reply-To: <437BBE6B.3000609@egenix.com> References: <5.1.1.6.0.20051116114559.01f47bd8@mail.telecommunity.com> <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> <5.1.1.6.0.20051116114559.01f47bd8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051116185051.01fd13d8@mail.telecommunity.com> At 12:19 AM 11/17/2005 +0100, M.-A. Lemburg wrote: > > Thus, I find these super-custom processes you're talking about highly > > implausible, because the only people who could implement them are the > > people with a strong knowledge of the distutils -- an incredibly rare > > breed of person, in other words. > >I was only talking about special ways to build the MANIFEST >files, not "super-custom" processes. No idea where you >got that impression from. Anything that involves a user knowing about MANIFEST at *all* (let alone changing its name) is a "super custom" process. Which unfortunately means that even for a fairly basic sdist, it's an issue. >I agree that this feature is underdocumented, but changing >the framework won't help with this: documentation patches >is what we *really* need ! That's certainly part of it; but it's certainly not all of it. Mostly, it's the difference between being a tool and a framework. The distutils provides all the mechanism you like, and none of the policy. That was a brilliant political move, as it dodges all the infighting that otherwise would have stopped it getting off the ground. However, there's no longer any reason *not* to have One Official Python Way To Do It, and to streamline the tools around that philosophy. One minor example of the problems with the distutils, is that their flexibility and inconsistent documentation causes people to create wildly different project source layouts, just because they can. You call that flexibility, I call that bad design for the developers who don't want to have to think about all these options (i.e., most of them). >BTW, not many people need to have these MANIFEST files >at all - distutils uses a built-in file finder based on these >defaults: > > - README or README.txt > - setup.py > - test/test*.py > - all pure Python modules mentioned in setup script > - all C sources listed as part of extensions or C libraries > in the setup script (doesn't catch C headers!) It also leaves out documentation and package data, for starters. >Carelessly overwriting hand-edited or otherwise generated >files in a build process is simply bad design. Not if it's a file *owned* by the distutils. If MANIFEST is assumed to be some sort of possibly user-generated file, the distutils should've picked a different name for *its* manifest file, and maybe provided the ability to read other files to generate its manifest. This is my point: there should be a manifest file that is wholly owned and controlled by the distutils with *no user-serviceable parts inside*. Having a file that is *partly* owned by the distutils and partly by the user is a recipe for disaster and the real bad design in this case. >MANIFEST is only used for source code distrbutions. I don't >see how you can use it for anything else. Because if you have package data files as part of your *source*, then the MANIFEST has to list them in order to have a valid source distribution. So, for some projects, it is equally valid to say, "be sure to install any package data files listed in the MANIFEST", and that is what the 'install_package_data' option would do. (Or maybe 'use_manifest_package_data' would be a better name, but you get the general idea.) >I don't buy this: on one hand you are talking about simple >packages (which don't need the MANIFEST files in the >first place), Not true! You run into a problem the moment you have any documentation besides README, or if you have any package data files. >on the other about hooks to adjust distutils' >build process, something I'd group under more complex >setups. My point is that a well-designed system that offers customization hooks should not offer you *meaningless* choices between different ways to customize the same thing. >So your point is to make your life as setuptools author >easier ? No, if that was my point I'd have just done whatever the heck I wanted and not bothered posting to the list. :) My point is to get the *community* better tools, and part of that is lowering the entry barriers to creating them. If it was only me I cared about, I'd not have embarked on the entire venture to begin with. Sheesh. >Why don't you just disable all these options in your >setuptools front-end and hard-code the MANIFEST file >names ? I plan to! The point of my posting was to find out whether I was going to be introducing any meaningful barriers to adoption of setuptools. For example, if there were some bdist* command out there in widespread use that would break as a result of these changes. Yes, I ranted a little about making any "distutils 2" less flexible, but my practical present-day assumption is that setuptools actually modifying the distutils is still a long time coming. I don't even plan to *propose* that idea myself, certainly not any time soon. >distutils is a loosly coupled framework of components. >In such a framework, a basic design principle is to >be able to decouple and recouple existing components. >The only way to implement this is by making the components >suitably independent and this is what was done in >distutils. You're confusing a usability issue with a technical design issue. One calls for TMTOWTDI (There's More Than One Way To Do It), the other for TSBO-APOO-OWTDI (There Should Be One - And Preferably Only One - Obvious Way To Do It). >My point is that you shouldn't try to strip down distutils >itself just because you think it's hard work to support >all these features in setuptools. It's not needed to strip >down distutils for this reason as you can easily disable >these options for anyone using your setuptools. > >As a result, both users of setuptools and straight >distutils are happy. This was always my plan; the question was to find out whether there were any holes in that plan, for *setuptools*. I'm sorry if that wasn't clear; I only talked about removing these things in distutils under a hypothetical "distutils 2" scenario. *None* of these changes were being proposed for a "pure" distutils today. That doesn't mean I don't think that the distutils design is too flexible in terms of offering meaningless choices, or more precisely, offering lots of choices to either shoot yourself in the foot. :) It just means that I realistically understand that today's distutils are unlikely to be changeable. Rather, I'm looking at the eventual transmutation of setuptools into "distutils 2", and want to make sure that late adopters moving to it in the distant future don't run into issues that haven't been accomodated in some way. From pobrien at orbtech.com Thu Nov 17 04:45:02 2005 From: pobrien at orbtech.com (Patrick K. O'Brien) Date: Wed, 16 Nov 2005 21:45:02 -0600 Subject: [Distutils] MANIFEST destiny :) In-Reply-To: <437BBE6B.3000609@egenix.com> References: <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> <5.1.1.6.0.20051116114559.01f47bd8@mail.telecommunity.com> <437BBE6B.3000609@egenix.com> Message-ID: <437BFCBE.7070708@orbtech.com> M.-A. Lemburg wrote: > Hi Phillip, > > In general, I think you are having a different focus here > than what distutils is trying to be and that's perfectly > OK - you can implement all these nice strategies and > automated decisions into your setuptools. <major snippage> > Look, nobody stops you from removing all these features > in your front-end. distutils lets you do all this and that's > what so great about it. > > My point is that you shouldn't try to strip down distutils > itself just because you think it's hard work to support > all these features in setuptools. It's not needed to strip > down distutils for this reason as you can easily disable > these options for anyone using your setuptools. > > As a result, both users of setuptools and straight > distutils are happy. Since I commented on Phillip's post, I figured I'd better comment on this one as well. In a nutshell, this has been one of the more passionate yet even-handed and rational threads I've seen in some time. I'm impressed by the quality of this dialog and am hopeful that good things will come from it. Thank you both, for what it's worth. :-) -- Patrick K. O'Brien Orbtech http://www.orbtech.com Schevo http://www.schevo.org From p.f.moore at gmail.com Thu Nov 17 11:40:27 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 17 Nov 2005 10:40:27 +0000 Subject: [Distutils] MANIFEST destiny :) In-Reply-To: <5.1.1.6.0.20051116185051.01fd13d8@mail.telecommunity.com> References: <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> <5.1.1.6.0.20051116114559.01f47bd8@mail.telecommunity.com> <437BBE6B.3000609@egenix.com> <5.1.1.6.0.20051116185051.01fd13d8@mail.telecommunity.com> Message-ID: <79990c6b0511170240h637b7d44iea7c149856d72124@mail.gmail.com> On 11/17/05, Phillip J. Eby <pje at telecommunity.com> wrote: > That's certainly part of it; but it's certainly not all of it. Mostly, > it's the difference between being a tool and a framework. The distutils > provides all the mechanism you like, and none of the policy. That was a > brilliant political move, as it dodges all the infighting that otherwise > would have stopped it getting off the ground. However, there's no longer > any reason *not* to have One Official Python Way To Do It, and to > streamline the tools around that philosophy. This seems to me to be a good point. However, wouldn't a relatively simple way of addressing it be just to *document* the One Official Way (ideally, in the Python documentation, but maybe hosted elsewhere as well for users of current and older versions)? You're right in saying (elsewhere, I think - I can't find the exact quote now) that the distutils documentation is a bit lacking in "how should I do it" help like this. You could add a footnote somewhere that setuptools is limited to support of projects following this approach. Paul. From robin at jessikat.fsnet.co.uk Thu Nov 17 18:54:04 2005 From: robin at jessikat.fsnet.co.uk (Robin Becker) Date: Thu, 17 Nov 2005 17:54:04 +0000 Subject: [Distutils] Mac OS 10.4 Message-ID: <437CC3BC.9020202@chamonix.reportlab.co.uk> Can some knowledgeable person tell me if there are any nasty things to watch out for when building extensions for the standard installed python for Mac OS 10.4 (I believe that's the number corresponding to 'Tiger'). I have built my own Python versions in the past and that worked fine. I just wondered if there are any gotchas relating to the mach loader etc etc. -- Robin Becker From bob at redivi.com Thu Nov 17 19:04:34 2005 From: bob at redivi.com (Bob Ippolito) Date: Thu, 17 Nov 2005 10:04:34 -0800 Subject: [Distutils] Mac OS 10.4 In-Reply-To: <437CC3BC.9020202@chamonix.reportlab.co.uk> References: <437CC3BC.9020202@chamonix.reportlab.co.uk> Message-ID: <446B805B-884E-44D0-8ABD-D5CAF9CC108E@redivi.com> On Nov 17, 2005, at 9:54 AM, Robin Becker wrote: > Can some knowledgeable person tell me if there are any nasty things > to watch out > for when building extensions for the standard installed python for > Mac OS 10.4 > (I believe that's the number corresponding to 'Tiger'). The extensions you build may not be compatible with previous versions of OS X, and you may need to use GCC 3.3 to compile some extensions (gcc_select makes it easy to do that). -bob From pje at telecommunity.com Thu Nov 17 19:11:09 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 17 Nov 2005 13:11:09 -0500 Subject: [Distutils] MANIFEST destiny :) In-Reply-To: <79990c6b0511170240h637b7d44iea7c149856d72124@mail.gmail.co m> References: <5.1.1.6.0.20051116185051.01fd13d8@mail.telecommunity.com> <5.1.1.6.0.20051115194823.01f2e1f0@mail.telecommunity.com> <5.1.1.6.0.20051116114559.01f47bd8@mail.telecommunity.com> <437BBE6B.3000609@egenix.com> <5.1.1.6.0.20051116185051.01fd13d8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051117125400.01f586f8@mail.telecommunity.com> At 10:40 AM 11/17/2005 +0000, Paul Moore wrote: >This seems to me to be a good point. However, wouldn't a relatively >simple way of addressing it be just to *document* the One Official Way >(ideally, in the Python documentation, but maybe hosted elsewhere as >well for users of current and older versions)? I'm doing this with the setuptools documentation to some extent now, although I'm not quite so prescriptive as I imply we should be in the larger and long term sense. Mainly that's because right now setuptools adoption depends on early adopters among existing package distributors, and it doesn't make sense for me to dictate e.g. their directory layout, as long as it's halfway sane. :) >You could add a footnote somewhere that setuptools is limited to >support of projects following this approach. Actually, I can't. Setuptools needs to be 100% backward-compatible with existing distributions, at least to the point of building eggs for them. Otherwise, I can't get the critical mass of users needed to bootstrap this thing. From pje at telecommunity.com Thu Nov 17 19:21:10 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 17 Nov 2005 13:21:10 -0500 Subject: [Distutils] easy_install -U setuptools fails on Windows (workaround) In-Reply-To: <437B4DEC.26723.FA16656@bkc.murkworks.com> References: <5.1.1.6.0.20051116141925.031d2a38@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051117131144.03295e10@mail.telecommunity.com> At 03:19 PM 11/16/2005 -0500, Brad Clements wrote: >On 16 Nov 2005 at 14:24, Phillip J. Eby wrote: > > > these issues. You can update your setuptools with: > > > > easy_install -U setuptools > > > >Except on windows, the easy_install binary is in-use: Argh. Curse you Windows, and your non-Posix filesystem that won't allow replacing open files! :( Thanks for pointing this out; I'll have to tell people to use ez_setup.py on Windows for upgrades. Alternately, I could try having easy_install detect that an .exe targeted for upgrade is in use, and if the .exe is the same as the one it would have installed, it can ignore it. Hm. On the other hand, it would then mean that it would sometimes work, and sometimes not. The real trouble is that the launcher uses spawnv() to run Python, which means that the original easy_install.exe is technically still in use while Python runs. I have to do that in order to avoid returning to the shell prompt before the Python script runs, which produces weird effects (to say the least). For some reason, an execv() of Python on Windows seems to create an independent process instead of replacing the current process. So, I only use execv() to run GUI (.pyw) scripts on Windows. If somebody with more Windows-fu than me can suggest a way to do a more exec-like exec, that unloads the launcher.exe and runs Python without dropping back to a shell prompt *before* the Python script runs, I could make easy_install safely self-upgrading on Windows. From bkc at murkworks.com Thu Nov 17 19:52:21 2005 From: bkc at murkworks.com (Brad Clements) Date: Thu, 17 Nov 2005 13:52:21 -0500 Subject: [Distutils] easy_install -U setuptools fails on Windows (workaround) In-Reply-To: <5.1.1.6.0.20051117131144.03295e10@mail.telecommunity.com> References: <437B4DEC.26723.FA16656@bkc.murkworks.com> Message-ID: <437C8B15.27919.147852F3@bkc.murkworks.com> On 17 Nov 2005 at 13:21, Phillip J. Eby wrote: > If somebody with more Windows-fu than me can suggest a way to do a > more exec-like exec, that unloads the launcher.exe and runs Python > without dropping back to a shell prompt *before* the Python script > runs, I could make easy_install safely self-upgrading on Windows. No-fu here, sorry. In olden times, I think this kind of stuff was done via .bat files and %if errorlevel, plus creating temp copies of the .exe for subsequent respawn. I don't think it's worth the time, an end-user warning should be enough for this rare occurance (with self-upgrading easy_install) .. oth, I think if the install renames the target file first, then creates the new .exe with the correct name, that will work. I'm not sure if you can delete a renamed .exe that's still in-use. -- Brad Clements, bkc at murkworks.com (315)268-1000 http://www.murkworks.com AOL-IM or SKYPE: BKClements From pje at telecommunity.com Thu Nov 17 20:11:36 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 17 Nov 2005 14:11:36 -0500 Subject: [Distutils] easy_install -U setuptools fails on Windows (workaround) In-Reply-To: <437C8B15.27919.147852F3@bkc.murkworks.com> References: <5.1.1.6.0.20051117131144.03295e10@mail.telecommunity.com> <437B4DEC.26723.FA16656@bkc.murkworks.com> Message-ID: <5.1.1.6.0.20051117140927.03014b98@mail.telecommunity.com> At 01:52 PM 11/17/2005 -0500, Brad Clements wrote: >oth, I think if the install renames the target file first, then creates >the new >.exe with the correct name, that will work. > >I'm not sure if you can delete a renamed .exe that's still in-use. Even if you could, the rename isn't going to work on older (non-NT) versions of Windows, and even on newer versions I gather that renaming an in-use .exe is harder than it sounds. Maybe what I need is a way to build an executable that doesn't need to keep its .exe open, or perhaps use a .com launcher instead. :) From lstep at adelux.fr Thu Nov 17 18:11:50 2005 From: lstep at adelux.fr (Luc Stepniewski) Date: Thu, 17 Nov 2005 17:11:50 +0000 (UTC) Subject: [Distutils] Question about usage of pje's setuptools Message-ID: <loom.20051117T175452-822@post.gmane.org> I have made a program that I want to distribute as an egg. The problem is that I need to put some files in directories like /etc/init.d and some configurations in a newly created directory in /etc. I know there is the "entry_points" + "console_scripts" stuff that I can use in my setup.py, but this is only for putting executables in /usr/bin. I know that setuptools is pretty new, and I understand the reasons why the sandbox refuses to create files outside of it, but if so, how do people develop big applications, that require configuration files, init files (at boot I mean), etc.? I did not find any example of big programs using setuptools that need global installation. Thanks for your help, Luc From pje at telecommunity.com Thu Nov 17 22:44:28 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 17 Nov 2005 16:44:28 -0500 Subject: [Distutils] Question about usage of pje's setuptools In-Reply-To: <loom.20051117T175452-822@post.gmane.org> Message-ID: <5.1.1.6.0.20051117164033.01f7fe08@mail.telecommunity.com> At 05:11 PM 11/17/2005 +0000, Luc Stepniewski wrote: >I have made a program that I want to distribute as an egg. The problem is >that >I need to put some files in directories like /etc/init.d and some >configurations in a newly created directory in /etc. >I know there is the "entry_points" + "console_scripts" stuff that I can >use in >my setup.py, but this is only for putting executables in /usr/bin. > >I know that setuptools is pretty new, and I understand the reasons why the >sandbox refuses to create files outside of it, but if so, how do people >develop >big applications, that require configuration files, init files (at boot I >mean), etc.? The common approach to this is to define console scripts that perform installation and uninstallation of configuration files, creation of project files, etc. At some point, I'm thinking it might be useful to provide some install-time entry point facility for doing these things as part of installation, but in the meantime the recommended approach is to provide console scripts that the user runs. (One advantage to separate scripts is that you can offer either an interactive setup or provide task-specific command-line options, since it's rare that you have an application whose /etc configuration can be 100% automated, due to differences in setup from one machine to the next.) From simon at arrowtheory.com Fri Nov 18 00:27:52 2005 From: simon at arrowtheory.com (Simon Burton) Date: Fri, 18 Nov 2005 10:27:52 +1100 Subject: [Distutils] extra_compile/link_args : split string before exec Message-ID: <20051118102752.5365a80d.simon@arrowtheory.com> HI, i just found a nasty bug in my distutils: passing in multiple arguments to a single element in the extra_link_args list. Those multiple arguments are not .split() by distutils before exec'ing gcc and so ld was completely ignoring some essential libs. I would suggest either adding a .split() into distutils, or at least documenting this. Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com From robin at jessikat.fsnet.co.uk Fri Nov 18 13:54:09 2005 From: robin at jessikat.fsnet.co.uk (Robin Becker) Date: Fri, 18 Nov 2005 12:54:09 +0000 Subject: [Distutils] Mac OS 10.4 In-Reply-To: <446B805B-884E-44D0-8ABD-D5CAF9CC108E@redivi.com> References: <437CC3BC.9020202@chamonix.reportlab.co.uk> <446B805B-884E-44D0-8ABD-D5CAF9CC108E@redivi.com> Message-ID: <437DCEF1.6030101@chamonix.reportlab.co.uk> Bob Ippolito wrote: > > On Nov 17, 2005, at 9:54 AM, Robin Becker wrote: > ..... > > The extensions you build may not be compatible with previous versions > of OS X, and you may need to use GCC 3.3 to compile some extensions > (gcc_select makes it easy to do that). > .... Thanks for the info. So is it impossible to compile dylibs etc for earlier versions of darwin? Or can that also be achieved using some special environment? -- Robin Becker From bob at redivi.com Fri Nov 18 19:22:01 2005 From: bob at redivi.com (Bob Ippolito) Date: Fri, 18 Nov 2005 10:22:01 -0800 Subject: [Distutils] Mac OS 10.4 In-Reply-To: <437DCEF1.6030101@chamonix.reportlab.co.uk> References: <437CC3BC.9020202@chamonix.reportlab.co.uk> <446B805B-884E-44D0-8ABD-D5CAF9CC108E@redivi.com> <437DCEF1.6030101@chamonix.reportlab.co.uk> Message-ID: <B396E607-6FBE-4E44-913A-1843EEA28700@redivi.com> On Nov 18, 2005, at 4:54 AM, Robin Becker wrote: > Bob Ippolito wrote: >> On Nov 17, 2005, at 9:54 AM, Robin Becker wrote: > ..... >> The extensions you build may not be compatible with previous >> versions of OS X, and you may need to use GCC 3.3 to compile some >> extensions (gcc_select makes it easy to do that). > > .... > Thanks for the info. > So is it impossible to compile dylibs etc for earlier versions of > darwin? Or can that also be achieved using some special environment? It can be achieved using SDKs, but it's hard to orchestrate. My blog host's MySQL server is borked right now, but here's some info out of google's cache: http://66.102.7.104/search?q=cache:gBdCPTJVro4J:bob.pythonmac.org/ archives/category/macosx/universal-binaries/+universal +binaries&hl=en&client=safari For the archives, here's the actual relevant URL: http://bob.pythonmac.org/archives/category/macosx/universal-binaries/ -bob From vishnubob at gmail.com Fri Nov 18 23:27:24 2005 From: vishnubob at gmail.com (Vishnu Bob) Date: Fri, 18 Nov 2005 17:27:24 -0500 Subject: [Distutils] C++, swig, and distutils Message-ID: <437E554C.5040204@gmail.com> Hi, I have a two comments about the usability of C++, swig, and distutils. I'm using python 2.4.2. Comment 1: First, build_ext.py:525 tests the list "swig_opts". If the string "-c++" is in the swig_opts list, build_ext.py will build the wrapper with a ".cpp" extension instead of the default ".c" if self.swig_cpp or ('-c++' in self.swig_opts): target_ext = '.cpp' else: target_ext = '.c' However, "swig_cpp" is a command level option, and it's impossible to set via the Extension() arguments. This gets a little weird, because of build_ext.py:550: # Do not override commandline arguments if not self.swig_opts: for o in extension.swig_opts: swig_cmd.append(o) This is below the "-c++" string test described above. So, if you specify "-c++" like this: Extension( 'package.module', ['src/module.i'], swig_opts=['-c++'], language='c++' ) build_ext.py will include "-c++" into the swig command, but the output file will /still/ have a ".c" extension. In order to workaround this without specifying a command line argument or building a setup.cfg file, you have to specify an "options" keyword argument in the call to setup(): options={'build_ext':{'swig_opts':'-c++'}} Comment 2: In spite of workaround described above, build_ext.py:464 compiles the swig generated ".cpp" wrapper with "gcc". Luckily, gcc sees the ".cpp" extension and switches to g++, but that's totally sloppy. I've tried to coax distutils to use its C++ swig strategies with three different variables, and it's still assuming C. thanks, -g From gh at ghaering.de Sun Nov 20 23:46:24 2005 From: gh at ghaering.de (=?ISO-8859-1?Q?Gerhard_H=E4ring?=) Date: Sun, 20 Nov 2005 23:46:24 +0100 Subject: [Distutils] setuptools, eager_resources, and DLLs Message-ID: <4380FCC0.4000905@ghaering.de> Hello, I want to make pysqlite installable as an egg. The normal pysqlite install works like this (win32): pysqlite2/__init__.py pysqlite2/_sqlite.pyd pysqlite2/sqlite3.dll I read that with eager_resources, setuptools can install shared libraries too. But I have no idea how to use this feature if the extension module is located in a package. Any ideas? -- Gerhard From pje at telecommunity.com Mon Nov 21 01:48:07 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 20 Nov 2005 19:48:07 -0500 Subject: [Distutils] setuptools, eager_resources, and DLLs In-Reply-To: <4380FCC0.4000905@ghaering.de> Message-ID: <5.1.1.6.0.20051120194552.01fa9fe8@mail.telecommunity.com> At 11:46 PM 11/20/2005 +0100, Gerhard H?ring wrote: >Hello, > >I want to make pysqlite installable as an egg. > >The normal pysqlite install works like this (win32): > >pysqlite2/__init__.py >pysqlite2/_sqlite.pyd >pysqlite2/sqlite3.dll > >I read that with eager_resources, setuptools can install shared >libraries too. But I have no idea how to use this feature if the >extension module is located in a package. The latest release of setuptools will automatially detect that there is a .dll present, and add it to the native_libraries.txt file for you. However, if you want to be explicit, just add eager_resources=['pysqlite2/sqlite3.dll'] to your setup() arguments. I actually added the DLL detection specifically because of pysqlite. :) Mainly, it made me realize that I ought to just detect dll, so, pyd, and dylib files automatically, so that eager_resources is only needed for *data* files that may be used by C extensions. From tanner at real-time.com Mon Nov 21 23:11:53 2005 From: tanner at real-time.com (Bob Tanner) Date: Mon, 21 Nov 2005 16:11:53 -0600 Subject: [Distutils] Overiding the site-packages/<module> directory name? Message-ID: <200511211611.54842.tanner@real-time.com> Is there a way to override the site-packages/<module> directory name? I'm trying to make a debian package for FormEncode-0.4, by default "/usr/bin/python2.4 setup.py install --no-compile -O0" creates a directory structure llike: `-- usr |-- lib | `-- python2.4 | `-- site-packages | `-- FormEncode-0.4-py2.4.egg | |-- EGG-INFO | `-- formencode | |-- javascript | `-- util Reading python-policy 1.4, I believe a 3rd party module should be just /usr/lib/python<X>.<Y>/site-packages/<module-dir> I would like to see `-- usr |-- lib | `-- python2.4 | `-- site-packages | `-- formencode | |-- javascript | `-- util I'm working around the "problem" by installing a formencode.pth pointing to the /usr/lib/python2.4/site-packages/FormEncode-0.4-py2.4.egg directory. I've played around with FormEncode's setup.py and the site-packages path seems to be based on the setup(name=xxx, version=yyy), but I could not see any way to change these values without really messing up all the other tools that depend on these values. -- Bob Tanner <tanner at real-time.com> | Phone : (952)943-8700 http://www.real-time.com, Minnesota, Linux | Fax : (952)943-8500 Key fingerprint = AB15 0BDF BCDE 4369 5B42 1973 7CF1 A709 2CC1 B288 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20051121/0e475843/attachment.pgp From pje at telecommunity.com Mon Nov 21 23:20:54 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 21 Nov 2005 17:20:54 -0500 Subject: [Distutils] Overiding the site-packages/<module> directory name? In-Reply-To: <200511211611.54842.tanner@real-time.com> Message-ID: <5.1.1.6.0.20051121171756.02fb4290@mail.telecommunity.com> At 04:11 PM 11/21/2005 -0600, Bob Tanner wrote: >Is there a way to override the site-packages/<module> directory name? > >I'm trying to make a debian package for FormEncode-0.4, by default >"/usr/bin/python2.4 setup.py install --no-compile -O0" creates a directory >structure llike: > >`-- usr > |-- lib > | `-- python2.4 > | `-- site-packages > | `-- FormEncode-0.4-py2.4.egg > | |-- EGG-INFO > | `-- formencode > | |-- javascript > | `-- util > >Reading python-policy 1.4, I believe a 3rd party module should be just > > /usr/lib/python<X>.<Y>/site-packages/<module-dir> > >I would like to see > >`-- usr > |-- lib > | `-- python2.4 > | `-- site-packages > | `-- formencode > | |-- javascript > | `-- util > >I'm working around the "problem" by installing a formencode.pth pointing to >the /usr/lib/python2.4/site-packages/FormEncode-0.4-py2.4.egg directory. This isn't a "problem", it's just that the Debian policy isn't up-to-date. Python eggs install this way, and many packages (e.g. TurboGears) require the new structure. >I've played around with FormEncode's setup.py and the site-packages path >seems >to be based on the setup(name=xxx, version=yyy), but I could not see any way >to change these values without really messing up all the other tools that >depend on these values. Or all the tools that depend on the directory containing version information, and expect a certain layout within that directory structure. Do not attempt to change .egg layouts, as any package that has bothered to make itself be laid out this way almost certainly has non-trivial dependencies on it being laid out this way. Note also that in many cases, the package will be a single .egg *file*, (analagous to a Java .jar file) rather than a directory, and files are preferable to directories in most cases as they make Python import processing faster. From tanner at real-time.com Mon Nov 21 23:44:21 2005 From: tanner at real-time.com (Bob Tanner) Date: Mon, 21 Nov 2005 16:44:21 -0600 Subject: [Distutils] Overiding the site-packages/<module> directory name? References: <200511211611.54842.tanner@real-time.com> <5.1.1.6.0.20051121171756.02fb4290@mail.telecommunity.com> Message-ID: <dltik8$ppu$1@sea.gmane.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Phillip J. Eby wrote: >>`-- usr >> |-- lib >> | `-- python2.4 >> | `-- site-packages >> | `-- formencode >> | |-- javascript >> | `-- util >> >>I'm working around the "problem" by installing a formencode.pth pointing >>to the /usr/lib/python2.4/site-packages/FormEncode-0.4-py2.4.egg >>directory. > This isn't a "problem", it's just that the Debian policy isn't > up-to-date. Python eggs install this way, and many packages (e.g. > TurboGears) require the new structure. Then we have a problem in the Debian world. :-( Here is a snippet of an email I received from the Debian powers. I also believe this individual is one of the authors of the python-policy for Debian. I've X'd the name/email to protect the innocent. ===================== snip ================================== > On Monday 21 November 2005 09:09 am, XXXXXXX XXXXXX wrote: > > $ dpkg -L python2.4-formencode > > [...] > > /usr/lib/python2.4/site-packages/FormEncode-0.4-py2.4.egg > > /usr/share/doc/python2.4-formencode/copyright > > /usr/share/doc/python2.4-formencode/changelog.Debian.gz > > > > ??? Please don't do that in a Debian package. > > First question, why? > Because you can't us it with a simple "use formencode". "pydoc" doesn't work with eggs either. > If it break Debian policies, please point me to the appropriate > section(s) and documents. I wasn't able to find any related to python > module packaging. > zless /usr/share/doc/python/python-policy.txt.gz >Lastly that is the default installation method when using cdbs. Which I think > just delegates it to the setup.py (ie authors choice?) > Probably. .egg packages might be a nice generic way to ship Python stuff to random OS installations, but I don't think they should be used in standard Python packages that are shipped in a Linux distribution like Debian. ===================== snip ================================== To comply, I added a zip_safe = False to the setup() of FormEncode, then moved /usr/lib/python2.4/site-packages/FormEncode-0.4-py2.4.egg/formencode TO /usr/lib/python2.4/site-packages/formencode Are you saying the comply-hack- above will potentially break other python packages (like Turbogears)? - -- Bob Tanner <tanner at real-time.com> | Phone : (952)943-8700 http://www.real-time.com, Minnesota, Linux | Fax : (952)943-8500 Key fingerprint = AB15 0BDF BCDE 4369 5B42 1973 7CF1 A709 2CC1 B288 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (GNU/Linux) iD8DBQFDgk3GfPGnCSzBsogRAgjzAJ9RPFF+Ae5uxWjDwNHfHgXGjl01TACeM/Hy 6yJukvolhuOLRbqNpwpxhCc= =iu6Y -----END PGP SIGNATURE----- From ianb at colorstudy.com Tue Nov 22 00:20:56 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 21 Nov 2005 17:20:56 -0600 Subject: [Distutils] Overiding the site-packages/<module> directory name? In-Reply-To: <dltik8$ppu$1@sea.gmane.org> References: <200511211611.54842.tanner@real-time.com> <5.1.1.6.0.20051121171756.02fb4290@mail.telecommunity.com> <dltik8$ppu$1@sea.gmane.org> Message-ID: <43825658.6050007@colorstudy.com> Bob Tanner wrote: >>This isn't a "problem", it's just that the Debian policy isn't >>up-to-date. Python eggs install this way, and many packages (e.g. >>TurboGears) require the new structure. > > > Then we have a problem in the Debian world. :-( Here is a snippet of an > email I received from the Debian powers. I also believe this individual is > one of the authors of the python-policy for Debian. I've X'd the name/email > to protect the innocent. I do think we need to resolve these issues with Linux distributions. I'd encourage the X in question to join this list, and we can discuss how this stuff should work. Some of what Python Eggs do are redundant with a Debian package (though that redundancy can be used to positive effect) and some parts of Eggs offer features that don't fit into Debian packaging very well (like multiple installations of different versions of the same library, which requires versions in the package names themselves in Debian). Also, .pth file management is a bit of an issue, and is something that should be addressed by Debian policy, as it's somewhere where packages should cooperate. There are performance issues with every package having its own .pth file. > ===================== snip ================================== > >>On Monday 21 November 2005 09:09 am, XXXXXXX XXXXXX wrote: >> >>>$ dpkg -L python2.4-formencode >>>[...] >>>/usr/lib/python2.4/site-packages/FormEncode-0.4-py2.4.egg >>>/usr/share/doc/python2.4-formencode/copyright >>>/usr/share/doc/python2.4-formencode/changelog.Debian.gz >>> >>>??? Please don't do that in a Debian package. >> >>First question, why? >> > > Because you can't us it with a simple "use formencode". > "pydoc" doesn't work with eggs either. I think there's a good argument that Python isn't quite ready for installing zip files, in that lots of tools don't work that well with them right now. This can be addressed in code through distutils settings -- you can force all files to be unzipped globally in distutils.cfg, and perhaps Debian could hook into that (if they don't want to create a distutils.cfg file, which is reasonable as that can be useful for user configuration). >>If it break Debian policies, please point me to the appropriate >>section(s) and documents. I wasn't able to find any related to python >>module packaging. >> > > zless /usr/share/doc/python/python-policy.txt.gz > > >>Lastly that is the default installation method when using cdbs. Which I > > think > >>just delegates it to the setup.py (ie authors choice?) >> > > Probably. .egg packages might be a nice generic way to ship Python stuff > to random OS installations, but I don't think they should be used in > standard Python packages that are shipped in a Linux distribution like > Debian. > ===================== snip ================================== > > To comply, I added a zip_safe = False to the setup() of FormEncode, then > moved > > /usr/lib/python2.4/site-packages/FormEncode-0.4-py2.4.egg/formencode > > TO > > /usr/lib/python2.4/site-packages/formencode > > Are you saying the comply-hack- above will potentially break other python > packages (like Turbogears)? You lose metadata that TurboGears really wants to have; this isn't just dependency information, but also plugin information (which I don't think is being used in 0.7, but will be for 0.8). Python packages can't be expected to read Debian package metadata. Also more generally, I think Debian needs to play more nicely with user-installed packages, and if you rely purely on Debian metadata for detecting installed packages, you lose useful information. As of a few months ago I've moved to Ubuntu, so I don't know what the exact situation is on Debian, but after learning more about distutils I've realized that Debian has been missing functionality that would make user-installed code better adhere to Debian policy (like installing into /usr/local/lib/pythonX.Y). So I think Debian policy really needs some fixing on these matters anyway. FWIW, Ubuntu sponsored a project for making Eggs work better there: http://easy-deb.sourceforge.net/ -- it doesn't mean they'll necessarily use that work (I have no idea what their plans are, and Google Summer of Code work doesn't imply anything except interest). I'd like it to be possible to do "python setup.py bdist_deb" and get a Debian package; I think this should be very doable if we get some participation from people with Debian packaging experience. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Tue Nov 22 01:01:11 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 21 Nov 2005 19:01:11 -0500 Subject: [Distutils] Overiding the site-packages/<module> directory name? In-Reply-To: <dltik8$ppu$1@sea.gmane.org> References: <200511211611.54842.tanner@real-time.com> <5.1.1.6.0.20051121171756.02fb4290@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051121182230.02fba448@mail.telecommunity.com> At 04:44 PM 11/21/2005 -0600, Bob Tanner wrote: > On Monday 21 November 2005 09:09 am, XXXXXXX XXXXXX wrote: > > > $ dpkg -L python2.4-formencode > > > [...] > > > /usr/lib/python2.4/site-packages/FormEncode-0.4-py2.4.egg > > > /usr/share/doc/python2.4-formencode/copyright > > > /usr/share/doc/python2.4-formencode/changelog.Debian.gz > > > > > > ??? Please don't do that in a Debian package. > > > > First question, why? > > >Because you can't us it with a simple "use formencode". This is only true if you don't include an updated .pth file. Properly packaged eggs should include a .pth file, or better, code to update a single vendor-eggs.pth file. Normally, this would be done as a postinstall and uninstall modification to a vendor-eggs.pth file in site-packages. (I'm assuming "use formencode" should be "import formencode", btw.) I was not aware of the pydoc problem, however it's only a problem with *packages*, not modules, and then only if the package is zipped. For example, with the wsgiref package, "pydoc wsgiref" fails but "pydoc wsgiref.handlers" works fine. The issue is in fact a pydoc bug; it was not updated properly for PEP 302, introduced in Python 2.3. It should allow for the fact that __path__ may be empty, or that the strings contained within it may not be local filesystem directories, and for that matter, that the whole thing may be in a zipfile. A workaround is to install eggs always unzipped, which you can do with the -Z option to easy_install. I will endeavor to create a patch to ensure that future releases of Python will fix the pydoc problem - hopefully including maintenance releases of 2.3 and 2.4, although I don't know if there will be any additional 2.3 maintenance releases. > >Lastly that is the default installation method when using cdbs. Which I >think > > just delegates it to the setup.py (ie authors choice?) > > >Probably. .egg packages might be a nice generic way to ship Python stuff >to random OS installations, but I don't think they should be used in >standard Python packages that are shipped in a Linux distribution like >Debian. That's also not correct. Egg packaging is not merely a method of transporting packages, but also a method of providing additional metadata and runtime plugin capabilities, while allowing platform-independent runtime introspection and verification of package versions. What's being said here is like saying that Debian can only install Java .jar files if they are unpacked and all non-code files deleted. >Are you saying the comply-hack- above will potentially break other python >packages (like Turbogears)? It will force them to install a non-Debian version, since you are stripping all the data that allows the other packages to determine whether an appropriate version of the package is installed. In essence, you are relegating your packaging to "legacy" status, in the sense that code which uses the pkg_resources library to manage its dependencies will be unable to make use of it, so from the other packages' viewpoint, it might as well not exist. The purpose of this packaging metadata is to allow universal dependency management for Python software authors - they can depend on particular versions without having to commit to a particular vendor packaging system, simply by listing the desired eggs in their setup script. To offer a constructive suggestion, I would propose that Debian egg packages use 'pyegg' instead of 'python' in their naming, and that the name be derived from the PyPI name, not the name of the contained module. This would then hopefully allow 1:1 automated translation from egg dependencies to Debian packages. The thing is, the packages that need eggs are rapidly growing, and they can potentially depend on egg versions of *any* pre-existing PyPI package. In the long run, this means that it is most expedient to simply package *all* Python packages as eggs, or at any rate have a tool that can automatically translate an egg to a Debian package. From hawk78_it at yahoo.it Tue Nov 22 01:24:49 2005 From: hawk78_it at yahoo.it (Vincenzo Di Massa) Date: Tue, 22 Nov 2005 01:24:49 +0100 Subject: [Distutils] Overiding the site-packages/<module> directory name? In-Reply-To: <5.1.1.6.0.20051121182230.02fba448@mail.telecommunity.com> References: <200511211611.54842.tanner@real-time.com> <5.1.1.6.0.20051121171756.02fb4290@mail.telecommunity.com> <5.1.1.6.0.20051121182230.02fba448@mail.telecommunity.com> Message-ID: <200511220124.49823.hawk78_it@yahoo.it> Alle 01:01, marted? 22 novembre 2005, Phillip J. Eby ha scritto: > or at any rate have a tool that can automatically > translate an egg to a Debian package. That tool could be easy-deb http://easy-deb.sf.net ___________________________________ Yahoo! Mail: gratis 1GB per i messaggi e allegati da 10MB http://mail.yahoo.it From pje at telecommunity.com Tue Nov 22 01:37:38 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 21 Nov 2005 19:37:38 -0500 Subject: [Distutils] Overiding the site-packages/<module> directory name? In-Reply-To: <200511220124.49823.hawk78_it@yahoo.it> References: <5.1.1.6.0.20051121182230.02fba448@mail.telecommunity.com> <200511211611.54842.tanner@real-time.com> <5.1.1.6.0.20051121171756.02fb4290@mail.telecommunity.com> <5.1.1.6.0.20051121182230.02fba448@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051121193452.02f2c288@mail.telecommunity.com> At 01:24 AM 11/22/2005 +0100, Vincenzo Di Massa wrote: >Alle 01:01, marted? 22 novembre 2005, Phillip J. Eby ha scritto: > > or at any rate have a tool that can automatically > > translate an egg to a Debian package. >That tool could be easy-deb >http://easy-deb.sf.net I'll be honest; I haven't really investigated easy-deb to find out the answers to at least these questions: 1. Does it automatically handle egg-specified dependencies, translating them to python-pypi-foo dependencies? 2. Does it maintain a single .pth file for all installed eggs? 3. Does it honor the "zip safety" flags, and install as either a zipfile or a directory accordingly? From pythonhead at gentoo.org Tue Nov 22 02:27:34 2005 From: pythonhead at gentoo.org (Rob Cakebread) Date: Mon, 21 Nov 2005 17:27:34 -0800 Subject: [Distutils] Finding only activated eggs in Environment() Message-ID: <43827406.4020109@gentoo.org> Hi, I'm using pkg_resources.Environment() to get a list of all eggs. Besides making a list of activated eggs with pkg_resources.working_set and comparing it with pkg_resources.Environment, is there a way to check if an individual egg is activated? Has the setuptools cvs repository moved to svn with Python? I've found the API docs on cvs.sf.net, but it seems a couple versions behind. Thanks, Rob -- Rob Cakebread Gentoo Linux Developer Public Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x96BA679B Key fingerprint = 5E1A 57A0 0FA6 939D 3258 8369 81C5 A17B 96BA 679B From pje at telecommunity.com Tue Nov 22 03:03:28 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 21 Nov 2005 21:03:28 -0500 Subject: [Distutils] Finding only activated eggs in Environment() In-Reply-To: <43827406.4020109@gentoo.org> Message-ID: <5.1.1.6.0.20051121205851.01f7a7a0@mail.telecommunity.com> At 05:27 PM 11/21/2005 -0800, Rob Cakebread wrote: >I'm using pkg_resources.Environment() to get a list of all >eggs. > >Besides making a list of activated eggs with >pkg_resources.working_set >and comparing it with pkg_resources.Environment, is there >a way to check if an individual egg is activated? if someDist in working_set: # a distribution w/same location/version/pyversion/etc. # is in there Note that distribution objects are "value" objects in the sense that you can have dist1==dist2 be true, and dist1 is dist2 be false. So, even if the 'dist' object came from a different source, the 'in' test will still return True if the dist refers to the same location/version/pyversion/platform/etc. >Has the setuptools cvs repository moved to svn with Python? Yes. For checkouts, use: http://svn.python.org/projects/sandbox/trunk/setuptools For browsing, you can also use: http://svn.python.org/view/sandbox/trunk/setuptools From tanner at real-time.com Tue Nov 22 08:04:02 2005 From: tanner at real-time.com (Bob Tanner) Date: Tue, 22 Nov 2005 01:04:02 -0600 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <4382B786.2080705@v.loewis.de> References: <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> Message-ID: <200511220104.03160.tanner@real-time.com> On Tuesday 22 November 2005 12:15 am, Martin v. L?wis wrote: > I don't think Debian should use the egg structure. It apparently relies > on building a long sys.path (even though through only a single .pth > file); I'm not sure of how .eggs are implemented, but I'm going to cross-post this info to the python-distutils mailing list. See below for additional comments. > this adds additional costs to all import statements on startup. > It gets worse if these are zipfiles, because then each import statement > will have to look into each zipfile (until the import is resolved). The is the opposite of what I was told by upstream development over on distutils, snippet from Phillip J. Eby <pje at telecommunity.com>: <snip> Note also that in many cases, the package will be a single .egg file, (analagous to a Java .jar file) rather than a directory, and files are preferable to directories in most cases as they make Python import processing faster. <snip> ( Rest of the comments debian-python specific) -- Bob Tanner <tanner at real-time.com> | Phone : (952)943-8700 http://www.real-time.com, Minnesota, Linux | Fax : (952)943-8500 Key fingerprint = AB15 0BDF BCDE 4369 5B42 1973 7CF1 A709 2CC1 B288 From pythonhead at gentoo.org Tue Nov 22 09:10:14 2005 From: pythonhead at gentoo.org (Rob Cakebread) Date: Tue, 22 Nov 2005 00:10:14 -0800 Subject: [Distutils] Finding only activated eggs in Environment() In-Reply-To: <5.1.1.6.0.20051121205851.01f7a7a0@mail.telecommunity.com> References: <5.1.1.6.0.20051121205851.01f7a7a0@mail.telecommunity.com> Message-ID: <4382D266.9030609@gentoo.org> Phillip J. Eby wrote: > At 05:27 PM 11/21/2005 -0800, Rob Cakebread wrote: > >> I'm using pkg_resources.Environment() to get a list of all >> eggs. >> >> Besides making a list of activated eggs with >> pkg_resources.working_set >> and comparing it with pkg_resources.Environment, is there >> a way to check if an individual egg is activated? > > > if someDist in working_set: > # a distribution w/same location/version/pyversion/etc. > # is in there > > Note that distribution objects are "value" objects in the sense that you > can have dist1==dist2 be true, and dist1 is dist2 be false. So, even if > the 'dist' object came from a different source, the 'in' test will still > return True if the dist refers to the same > location/version/pyversion/platform/etc. > > Thanks, thats what I needed. I've been working on "yolk"[1] again and fixed it so it lists all versions of an egg, not just the latest. (moved from the Peak mailing list) If anyone is interested, I started a utility to list all eggs, only activated eggs, or only non-activated eggs with some of the metadata (version, desc, license etc.) I just added a -d option so you can list dependencies of an egg. I'm working on making it show what version of each dependency is actually installed. [1] http://eggs.gentooexperimental.org/file/trunk/tools/yolk.py From ben at groovie.org Wed Nov 16 22:55:00 2005 From: ben at groovie.org (Ben Bangert) Date: Wed, 16 Nov 2005 13:55:00 -0800 Subject: [Distutils] Recent upgrade uses security restrictions? Message-ID: <47A377FA-F170-492D-84A2-6453D1FD4304@groovie.org> I ran the setuptools upgrade of Kid to 0.7.1 this morning with no problems. When I ran it just now using the latest setuptools, I get this error which I've never seen before: Best match: kid 0.7.1 Downloading http://lesscode.org/dist/kid/0.7.1/kid-0.7.1.tar.gz Processing kid-0.7.1.tar.gz Running kid-0.7.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install- RBM6QG/kid-0.7.1/egg-dist-tmp-egfsVX error: SandboxViolation: mkdir('/Users/ben/.python-eggs', 511) {} The package setup script has attempted to modify files on your system that are not within the EasyInstall build area, and has been aborted. This package cannot be safely installed by EasyInstall, and may not support alternate installation locations even if you run its setup script by hand. Please inform the package's author and the EasyInstall maintainers to find out if a fix or workaround is available. Is this a new restriction that kicked in? Thanks, Ben From pje at telecommunity.com Tue Nov 22 16:35:10 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 10:35:10 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <200511220104.03160.tanner@real-time.com> References: <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> Message-ID: <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> At 01:04 AM 11/22/2005 -0600, Bob Tanner wrote: >On Tuesday 22 November 2005 12:15 am, Martin v. L?wis wrote: > > I don't think Debian should use the egg structure. It apparently relies > > on building a long sys.path (even though through only a single .pth > > file); > >I'm not sure of how .eggs are implemented, but I'm going to cross-post this >info to the python-distutils mailing list. See below for additional comments. > > > this adds additional costs to all import statements on startup. > > It gets worse if these are zipfiles, because then each import statement > > will have to look into each zipfile (until the import is resolved). > >The is the opposite of what I was told by upstream development over on >distutils, snippet from Phillip J. Eby <pje at telecommunity.com>: > ><snip> >Note also that in many cases, the package will be a single .egg file, >(analagous to a Java .jar file) rather than a directory, and files are >preferable to directories in most cases as they make Python import >processing faster. ><snip> Yes, it's true, zipfile import processing is faster than normal import processing; it is in fact one of the reasons zipfile imports were added to Python, because the zip directories are cached. A zipfile import lookup is a single dictionary lookup, whereas a directory import lookup requires multiple stat() calls. For all practical purposes, zipfiles added to sys.path are free after the initial directory read operation. Note that the need for a .pth is a limitation caused by the requirement to have packages importable at startup. Packages installed in "multi-version" or "deactivated" mode are only added to sys.path upon request and have no impact on startup time. Relatively few eggs *need* to be installed with a .pth file; we are simply in a transitional period where people still expect "installed" packages to be importable without an additional require() operation. Finally, I think it's important to note that what Debian should or should not use isn't really relevant to Debian's users, who will quite simply need eggs for many packages. If Debian doesn't provide them, the users will be forced to obtain them elsewhere. Over time, the number of packages that users need in egg form will continue to increase, and there will be an increasing number of users wanting to know why Debian can't provide them. It's perfectly reasonable not to redo existing Debian packages to use eggs, but for some packages, *not* using eggs is simply not an option. From pje at telecommunity.com Tue Nov 22 16:38:49 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 10:38:49 -0500 Subject: [Distutils] Recent upgrade uses security restrictions? In-Reply-To: <47A377FA-F170-492D-84A2-6453D1FD4304@groovie.org> Message-ID: <5.1.1.6.0.20051122103538.03ca4840@mail.telecommunity.com> At 01:55 PM 11/16/2005 -0800, Ben Bangert wrote: >Is this a new restriction that kicked in? No, the restriction has been in effect for many months now. It would appear, however, that the setup script you're using is using an egg containing a C library or some other file that needs to be extracted to a temporary location, and that's what's tripping it. That's an interaction I didn't think about, so I'll take a look at the sandbox code and see if I can make it so that egg resource extraction doesn't trigger sandboxing exceptions. In the meantime, this can be worked around by ensuring that any packages this setup script is using are installed unzipped, or alternately that the needed libraries or files are extracted beforehand. From ianb at colorstudy.com Tue Nov 22 17:19:12 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 22 Nov 2005 10:19:12 -0600 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> References: <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> Message-ID: <43834500.2090009@colorstudy.com> Phillip J. Eby wrote: >><snip> >>Note also that in many cases, the package will be a single .egg file, >>(analagous to a Java .jar file) rather than a directory, and files are >>preferable to directories in most cases as they make Python import >>processing faster. >><snip> > > > Yes, it's true, zipfile import processing is faster than normal import > processing; it is in fact one of the reasons zipfile imports were added to > Python, because the zip directories are cached. A zipfile import lookup is > a single dictionary lookup, whereas a directory import lookup requires > multiple stat() calls. For all practical purposes, zipfiles added to > sys.path are free after the initial directory read operation. Maybe an easier way to understand this (at least my impression) is that zip files are treated as read-only. Any directory on sys.path gets scanned everytime a new module is imported. And you never know if someone added something, so you do it all over again each time. A zip file is scanned only once. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Tue Nov 22 17:27:33 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 22 Nov 2005 10:27:33 -0600 Subject: [Distutils] Finding only activated eggs in Environment() In-Reply-To: <5.1.1.6.0.20051121205851.01f7a7a0@mail.telecommunity.com> References: <5.1.1.6.0.20051121205851.01f7a7a0@mail.telecommunity.com> Message-ID: <438346F5.2030609@colorstudy.com> Phillip J. Eby wrote: > At 05:27 PM 11/21/2005 -0800, Rob Cakebread wrote: > >>I'm using pkg_resources.Environment() to get a list of all >>eggs. >> >>Besides making a list of activated eggs with >>pkg_resources.working_set >>and comparing it with pkg_resources.Environment, is there >>a way to check if an individual egg is activated? > > > if someDist in working_set: > # a distribution w/same location/version/pyversion/etc. > # is in there Relatedly, how do I get all the distributions, not just the active ones? I added a command to Paste Script for display information about entry points, but it only gets the active distributions. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Tue Nov 22 18:09:10 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 12:09:10 -0500 Subject: [Distutils] Finding only activated eggs in Environment() In-Reply-To: <438346F5.2030609@colorstudy.com> References: <5.1.1.6.0.20051121205851.01f7a7a0@mail.telecommunity.com> <5.1.1.6.0.20051121205851.01f7a7a0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051122120650.01fc4810@mail.telecommunity.com> At 10:27 AM 11/22/2005 -0600, Ian Bicking wrote: >Phillip J. Eby wrote: >>At 05:27 PM 11/21/2005 -0800, Rob Cakebread wrote: >> >>>I'm using pkg_resources.Environment() to get a list of all >>>eggs. >>> >>>Besides making a list of activated eggs with >>>pkg_resources.working_set >>>and comparing it with pkg_resources.Environment, is there >>>a way to check if an individual egg is activated? >> >>if someDist in working_set: >> # a distribution w/same location/version/pyversion/etc. >> # is in there > >Relatedly, how do I get all the distributions, not just the active >ones? I added a command to Paste Script for display information about >entry points, but it only gets the active distributions. # scan all sys.path dirs for all eggs (slow) env = pkg_resources.Environment() for project_name in env: # iterate over distributions for this project # in *descending* version order, newest to oldest for dist in env[project_name]: # etc. Since an enviroment can contain multiple versions of the same egg, you'll probably want to limit yourself to the first (i.e. highest-versioned) distribution for each project, if your intent is to list available entry points. From mal at egenix.com Tue Nov 22 18:33:12 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 22 Nov 2005 18:33:12 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> References: <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> Message-ID: <43835658.30606@egenix.com> Phillip J. Eby wrote: > At 01:04 AM 11/22/2005 -0600, Bob Tanner wrote: > >>On Tuesday 22 November 2005 12:15 am, Martin v. L?wis wrote: >> >>>I don't think Debian should use the egg structure. It apparently relies >>>on building a long sys.path (even though through only a single .pth >>>file); >> >>I'm not sure of how .eggs are implemented, but I'm going to cross-post this >>info to the python-distutils mailing list. See below for additional comments. >> >> >>>this adds additional costs to all import statements on startup. >>>It gets worse if these are zipfiles, because then each import statement >>>will have to look into each zipfile (until the import is resolved). >> >>The is the opposite of what I was told by upstream development over on >>distutils, snippet from Phillip J. Eby <pje at telecommunity.com>: >> >><snip> >>Note also that in many cases, the package will be a single .egg file, >>(analagous to a Java .jar file) rather than a directory, and files are >>preferable to directories in most cases as they make Python import >>processing faster. >><snip> > > > Yes, it's true, zipfile import processing is faster than normal import > processing; Only after *all* ZIP files on sys.path have been scanned for their contents. The more you add to sys.path, the longer Python takes to startup. What's worse is that the slow-down affects the whole Python installation - each and every application using Python will have to scan all these ZIP files in case it tries to import a non-existing module or one which it finds late on sys.path. > it is in fact one of the reasons zipfile imports were added to > Python, because the zip directories are cached. A zipfile import lookup is > a single dictionary lookup, whereas a directory import lookup requires > multiple stat() calls. For all practical purposes, zipfiles added to > sys.path are free after the initial directory read operation. They are "free" for long running applications only where this caching makes sense. > Note that the need for a .pth is a limitation caused by the requirement to > have packages importable at startup. Packages installed in "multi-version" > or "deactivated" mode are only added to sys.path upon request and have no > impact on startup time. Relatively few eggs *need* to be installed with a > .pth file; we are simply in a transitional period where people still expect > "installed" packages to be importable without an additional require() > operation. > > Finally, I think it's important to note that what Debian should or should > not use isn't really relevant to Debian's users, who will quite simply need > eggs for many packages. If Debian doesn't provide them, the users will be > forced to obtain them elsewhere. Over time, the number of packages that > users need in egg form will continue to increase, and there will be an > increasing number of users wanting to know why Debian can't provide > them. It's perfectly reasonable not to redo existing Debian packages to > use eggs, but for some packages, *not* using eggs is simply not an option. Why should "eggs" be the only way to install a package ? Doesn't the standard "python setup.py install" work with eggified packages anymore (meaning that the package is installed as normal site-packages package) ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 22 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From ianb at colorstudy.com Tue Nov 22 18:46:48 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 22 Nov 2005 11:46:48 -0600 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <43835658.30606@egenix.com> References: <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> Message-ID: <43835988.3060106@colorstudy.com> M.-A. Lemburg wrote: >>Finally, I think it's important to note that what Debian should or should >>not use isn't really relevant to Debian's users, who will quite simply need >>eggs for many packages. If Debian doesn't provide them, the users will be >>forced to obtain them elsewhere. Over time, the number of packages that >>users need in egg form will continue to increase, and there will be an >>increasing number of users wanting to know why Debian can't provide >>them. It's perfectly reasonable not to redo existing Debian packages to >>use eggs, but for some packages, *not* using eggs is simply not an option. > > > Why should "eggs" be the only way to install a package ? > > Doesn't the standard "python setup.py install" work with > eggified packages anymore (meaning that the package is > installed as normal site-packages package) ? Eggs give room for package metadata that doesn't exist otherwise. Putting dependencies aside, this is functionality that simply doesn't exist with the standard distutils installation. In the case of FormEncode, it doesn't make use of any egg features (except that other packages may want to depend on it using setuptools). In the case of other frameworks -- including TurboGears (which I think is the ultimate packaging goal here) -- the Egg metadata really is important, it's not just used for dependencies. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From martin at v.loewis.de Tue Nov 22 18:53:28 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 22 Nov 2005 18:53:28 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> References: <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> Message-ID: <43835B18.6010604@v.loewis.de> Phillip J. Eby wrote: > Yes, it's true, zipfile import processing is faster than normal import > processing; it is in fact one of the reasons zipfile imports were added > to Python, because the zip directories are cached. A zipfile import > lookup is a single dictionary lookup, whereas a directory import lookup > requires multiple stat() calls. For all practical purposes, zipfiles > added to sys.path are free after the initial directory read operation. OTOH, it does add an overhead on startup, as it will have to read the TOC of all zipfiles on sys.path, atleast if the module you are looking for is in the last zipfile on the path. It then also adds memory overhead, as the TOC of all files is cached in memory. > Note that the need for a .pth is a limitation caused by the requirement > to have packages importable at startup. Packages installed in > "multi-version" or "deactivated" mode are only added to sys.path upon > request and have no impact on startup time. Relatively few eggs *need* > to be installed with a .pth file; we are simply in a transitional period > where people still expect "installed" packages to be importable without > an additional require() operation. People reasonable will have this expectation for a Debian package. If you install a Debian package with some library, you expect the library to be usable right away. > Finally, I think it's important to note that what Debian should or > should not use isn't really relevant to Debian's users, who will quite > simply need eggs for many packages. If Debian doesn't provide them, the > users will be forced to obtain them elsewhere. Debian should provide the packages, but not as eggs. For a Debian user, eggs do not add advantages, and for a Debian Developer, they only add additional hassle. > Over time, the number of > packages that users need in egg form will continue to increase, and > there will be an increasing number of users wanting to know why Debian > can't provide them. It's perfectly reasonable not to redo existing > Debian packages to use eggs, but for some packages, *not* using eggs is > simply not an option. Debian developers should work with upstream authors to keep a distutils-based setup.py operational. Regards, Martin From martin at v.loewis.de Tue Nov 22 18:57:05 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 22 Nov 2005 18:57:05 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <43834500.2090009@colorstudy.com> References: <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43834500.2090009@colorstudy.com> Message-ID: <43835BF1.9080008@v.loewis.de> Ian Bicking wrote: > Maybe an easier way to understand this (at least my impression) is that > zip files are treated as read-only. Any directory on sys.path gets > scanned everytime a new module is imported. And you never know if > someone added something, so you do it all over again each time. A zip > file is scanned only once. The difference is that a directory inside site-python is not scanned *at all* if the application looks for some other package. Only the top-level site-python directory is read, and it is not scanned, but instead a lookup operation directly asks for the subdirectory with the package name. If you have many zipfiles on sys.path, all applications will suffer from having to read the TOC of all those zipfiles, even if they need none of them. OTOH, if you had packages inside site-python, the contents of the unused packages is simply ignored. Regards, Martin From mal at egenix.com Tue Nov 22 19:32:15 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 22 Nov 2005 19:32:15 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <43835988.3060106@colorstudy.com> References: <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> Message-ID: <4383642F.8030804@egenix.com> Ian Bicking wrote: > M.-A. Lemburg wrote: > >>> Finally, I think it's important to note that what Debian should or >>> should not use isn't really relevant to Debian's users, who will >>> quite simply need eggs for many packages. If Debian doesn't provide >>> them, the users will be forced to obtain them elsewhere. Over time, >>> the number of packages that users need in egg form will continue to >>> increase, and there will be an increasing number of users wanting to >>> know why Debian can't provide them. It's perfectly reasonable not to >>> redo existing Debian packages to use eggs, but for some packages, >>> *not* using eggs is simply not an option. >> >> >> >> Why should "eggs" be the only way to install a package ? >> >> Doesn't the standard "python setup.py install" work with >> eggified packages anymore (meaning that the package is >> installed as normal site-packages package) ? > > > Eggs give room for package metadata that doesn't exist otherwise. > Putting dependencies aside, this is functionality that simply doesn't > exist with the standard distutils installation. In the case of > FormEncode, it doesn't make use of any egg features (except that other > packages may want to depend on it using setuptools). In the case of > other frameworks -- including TurboGears (which I think is the ultimate > packaging goal here) -- the Egg metadata really is important, it's not > just used for dependencies. Understood, but wouldn't it be reasonably possible to also install this meta-data into a standard site-packages package directory ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 22 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From ben at groovie.org Tue Nov 22 20:51:03 2005 From: ben at groovie.org (Ben Bangert) Date: Tue, 22 Nov 2005 11:51:03 -0800 Subject: [Distutils] HTTP auth in svn repositories Message-ID: <6097FFA8-57AB-4988-AFD6-BA4A0341566B@groovie.org> A svn repository I was attempting to install from has a public login/ group, so I tried entering the egg link like so: http://guest:guest at svn.myghty.org/myghty/branches/ 99a_exception_save#egg.... However the parsing setuptools did resulted in the following error: httplib.InvalidURL: nonnumeric port: 'guest at svn.myghty.org' I thought using that http format was fairly standard for putting http basic auth, shouldn't setuptools honor it? Thanks, Ben From ben at groovie.org Tue Nov 22 20:55:22 2005 From: ben at groovie.org (Ben Bangert) Date: Tue, 22 Nov 2005 11:55:22 -0800 Subject: [Distutils] Version numbers with a letter at the end Message-ID: <B5EC7670-5B85-40BD-B314-633C6651AE1A@groovie.org> Does setuptools distinguish and recognize letters in a version? For example, the latest Myghty is: version='0.99a' When I up it to '0.99a.1', and setuptools does a pre-requisite check, it doesn't seem to recognize that 0.99a.1 is more recent than 0.99a. Also, this seems like it would cause problems with requirements that look for a dev version, as it'd be: "Myghty >= 0.99.adev-r1980" Is a numerical-only version number a requirement for version info under setuptools? - Ben From ianb at colorstudy.com Tue Nov 22 21:00:01 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 22 Nov 2005 14:00:01 -0600 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <4383642F.8030804@egenix.com> References: <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> Message-ID: <438378C1.4050401@colorstudy.com> M.-A. Lemburg wrote: >>Eggs give room for package metadata that doesn't exist otherwise. >>Putting dependencies aside, this is functionality that simply doesn't >>exist with the standard distutils installation. In the case of >>FormEncode, it doesn't make use of any egg features (except that other >>packages may want to depend on it using setuptools). In the case of >>other frameworks -- including TurboGears (which I think is the ultimate >>packaging goal here) -- the Egg metadata really is important, it's not >>just used for dependencies. > > > Understood, but wouldn't it be reasonably possible to > also install this meta-data into a standard site-packages > package directory ? An egg and Python packages don't map 1-to-1. An egg can contain multiple packages (which is fairly uncommon so far), but also a top-level package can exist in more than one egg (i.e., namespace packages, like zope.interfaces or paste.script). The metadata belongs to the egg, not to the package inside the egg. Also, some of the metadata is encoded in the directory name itself, like the version information. I think this makes it easier to do some scanning operations, without a single database of installed packages (and also respecting sys.path manipulation). That said, I think it would be nice if the transition was smoother. E.g., if a file "ElementTree-1.2.6.egg-provided" could point to an installed elementtree library (similar to the currently-supported .egg-link file, but also slightly different). And, perhaps, elementtree/ElementTree.egg-info could exist (with the same data as the current ElementTree-1.2.6/EGG-INFO), though I think the simpler case where extra metadata is disallowed would be easier. That would only work for situations when there's a 1-to-1 mapping from packages to eggs/projects, but that covers many situations, especially cases where we're currently seeing conflicts. You lose the ability to easily support multiple versions of a package with this, though that could probably be handled too. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Tue Nov 22 21:28:48 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 15:28:48 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <43835BF1.9080008@v.loewis.de> References: <43834500.2090009@colorstudy.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43834500.2090009@colorstudy.com> Message-ID: <5.1.1.6.0.20051122152125.01fc44a8@mail.telecommunity.com> At 06:57 PM 11/22/2005 +0100, Martin v. L?wis wrote: >Ian Bicking wrote: >>Maybe an easier way to understand this (at least my impression) is that >>zip files are treated as read-only. Any directory on sys.path gets >>scanned everytime a new module is imported. And you never know if >>someone added something, so you do it all over again each time. A zip >>file is scanned only once. > >The difference is that a directory inside site-python is not scanned >*at all* if the application looks for some other package. Only the >top-level site-python directory is read, and it is not scanned, but >instead a lookup operation directly asks for the subdirectory with >the package name. > >If you have many zipfiles on sys.path, all applications will suffer >from having to read the TOC of all those zipfiles, even if they need >none of them. OTOH, if you had packages inside site-python, the >contents of the unused packages is simply ignored. I'm sorry, but this is, shall we say, "fact challenged"? .pth files' contents are added to the *end* of sys.path. This means that stdlib imports and normal site-packages imports are satisfied *before* any hypothetical overhead from .pth entries, whether they're zipfiles or directories. If Python never reaches the .pth entries at runtime, it will not even read the zipfile TOCs, let alone attempting to stat() for contained packages. Please check your facts before spreading untruths like this - a simple reading of PEP 302 and site.py should make it clear to you that the only absolute startup overhead being added here is the reading of the .pth file itself - which I've been encouraging vendors to maintain as a single .pth file rather than having individual .pth files for each package. From pje at telecommunity.com Tue Nov 22 21:33:39 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 15:33:39 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <43835658.30606@egenix.com> References: <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> At 06:33 PM 11/22/2005 +0100, M.-A. Lemburg wrote: >Phillip J. Eby wrote: > > > > Yes, it's true, zipfile import processing is faster than normal import > > processing; > >Only after *all* ZIP files on sys.path have been scanned >for their contents. The more you add to sys.path, the longer >Python takes to startup. This is simply not true. If you don't believe PEP 302 and site.py, measure it for yourself. The *only* addition to startup is the time to actually read the .pth file and append the entries to the list. >What's worse is that the slow-down affects the whole Python >installation - each and every application using Python will >have to scan all these ZIP files in case it tries to import >a non-existing module or one which it finds late on sys.path. And how often do programs attempt to import non-existing modules along performance critical paths? Note by the way that "scan all these ZIP files" is a misleading term in any case - the files are not "scanned". They are opened, and a small amount of data is read from the end of the file. Nothing that I would consider "scanning" is involved. From tanner at real-time.com Tue Nov 22 21:32:27 2005 From: tanner at real-time.com (Bob Tanner) Date: Tue, 22 Nov 2005 14:32:27 -0600 Subject: [Distutils] formencode as .egg in Debian ?? References: <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> Message-ID: <dlvv8r$kgk$1@sea.gmane.org> Ian Bicking wrote: >>>Eggs give room for package metadata that doesn't exist otherwise. >>>Putting dependencies aside, this is functionality that simply doesn't >>>exist with the standard distutils installation. <snip> >> Understood, but wouldn't it be reasonably possible to >> also install this meta-data into a standard site-packages >> package directory ? <snip> > An egg and Python packages don't map 1-to-1. An egg can contain > multiple packages (which is fairly uncommon so far), but also a > top-level package can exist in more than one egg (i.e., namespace > packages, like zope.interfaces or paste.script). The metadata belongs > to the egg, not to the package inside the egg. I'd like to bring focus back to immediate problem at hand (and yes, I understand there is something much bigger involved). The ultimate goal is to debianize TurboGears, reading the above, and other posts using the legacy site-packages (non-egg) installation will "break" TurboGears? -- Bob Tanner <tanner at real-time.com> | Phone : (952)943-8700 http://www.real-time.com, Minnesota, Linux | Fax : (952)943-8500 Key fingerprint = AB15 0BDF BCDE 4369 5B42 1973 7CF1 A709 2CC1 B288 From pje at telecommunity.com Tue Nov 22 21:44:52 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 15:44:52 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <43835B18.6010604@v.loewis.de> References: <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> At 06:53 PM 11/22/2005 +0100, Martin v. L?wis wrote: >>Finally, I think it's important to note that what Debian should or should >>not use isn't really relevant to Debian's users, who will quite simply >>need eggs for many packages. If Debian doesn't provide them, the users >>will be forced to obtain them elsewhere. > >Debian should provide the packages, but not as eggs. For packages that only operate as eggs, and/or require their dependencies as eggs, you are stating a contradiction in terms. Eggs are not merely a distribution format, any more than Java .jar files are. >>Over time, the number of packages that users need in egg form will >>continue to increase, and there will be an increasing number of users >>wanting to know why Debian can't provide them. It's perfectly reasonable >>not to redo existing Debian packages to use eggs, but for some packages, >>*not* using eggs is simply not an option. > >Debian developers should work with upstream authors to keep a >distutils-based setup.py operational. It's perfectly operational; clearly the entire egg system is *well* within the Python runtime's intended operating parameters, as it uses only well-defined and published aspects of the Python language, API, stdlib, and build process. Note that it's perfectly valid for a distutils-based package to install itself in a subdirectory under site-packages, complete with its own .pth file - it's a *documented feature* of the distutils. Perhaps you have some other definition of "operational" in mind? As I've already stated, applying this same policy to Java libraries would be to demanding that all the .class files be extracted to the filesystem and any manifest files be deleted, before Debian would consent to package them. In other words, it would be silly and pointless, because the users would then ignore the packages in favor of actual jars, because then their applications would actually work. From ianb at colorstudy.com Tue Nov 22 21:50:00 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 22 Nov 2005 14:50:00 -0600 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <dlvv8r$kgk$1@sea.gmane.org> References: <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> <dlvv8r$kgk$1@sea.gmane.org> Message-ID: <43838478.5020500@colorstudy.com> Bob Tanner wrote: >>An egg and Python packages don't map 1-to-1. An egg can contain >>multiple packages (which is fairly uncommon so far), but also a >>top-level package can exist in more than one egg (i.e., namespace >>packages, like zope.interfaces or paste.script). The metadata belongs >>to the egg, not to the package inside the egg. > > > I'd like to bring focus back to immediate problem at hand (and yes, I > understand there is something much bigger involved). > > The ultimate goal is to debianize TurboGears, reading the above, and other > posts using the legacy site-packages (non-egg) installation will "break" > TurboGears? Well... not really, but TurboGears will think it is broken, because it will require packages that will be available (through some non-egg form), but it won't realize are available. ElementTree in particular; I don't know if the other packages TG uses are available as Debian packages currently. That's for 0.8. In 0.9 and ahead it will be more broken, because TG will both provide and consume egg metadata (entry_points, in particular). So while it would be reasonably easy to patch TurboGears now to be installed without eggs, that's only a short-term solution. However, it is also possible that you could patch Kid (which TG requires) to remove the requirement on ElementTree (adding that requirement to the Debian package metadata) and then you'd be fine into the future. In that model, all the new packages you do for TurboGears would be installed as eggs, but packages already available won't be installed as eggs. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Tue Nov 22 21:53:28 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 15:53:28 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <dlvv8r$kgk$1@sea.gmane.org> References: <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> Message-ID: <5.1.1.6.0.20051122154922.01fcd998@mail.telecommunity.com> At 02:32 PM 11/22/2005 -0600, Bob Tanner wrote: >The ultimate goal is to debianize TurboGears, reading the above, and other >posts using the legacy site-packages (non-egg) installation will "break" >TurboGears? If by that you mean, will you be able to create a Debian-packaged TurboGears without extensive patching to work around the issue, the answer is no, you will not. Right now, easy-deb looks like the best bet for people wanting a Debian-packaged TurboGears, in that my understanding from the author is that it does in fact Debianize eggs. From pje at telecommunity.com Tue Nov 22 21:55:01 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 15:55:01 -0500 Subject: [Distutils] HTTP auth in svn repositories In-Reply-To: <6097FFA8-57AB-4988-AFD6-BA4A0341566B@groovie.org> Message-ID: <5.1.1.6.0.20051122155353.03c897b8@mail.telecommunity.com> At 11:51 AM 11/22/2005 -0800, Ben Bangert wrote: >A svn repository I was attempting to install from has a public login/ >group, so I tried entering the egg link like so: >http://guest:guest at svn.myghty.org/myghty/branches/ >99a_exception_save#egg.... > >However the parsing setuptools did resulted in the following error: >httplib.InvalidURL: nonnumeric port: 'guest at svn.myghty.org' > >I thought using that http format was fairly standard for putting http >basic auth, shouldn't setuptools honor it? As you can see, it's httplib that has a problem with it, not setuptools. If anybody has suggestions for resolving the issue (or better, patches!), I'm wide open. From mal at egenix.com Tue Nov 22 21:54:38 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 22 Nov 2005 21:54:38 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <438378C1.4050401@colorstudy.com> References: <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> Message-ID: <4383858E.2080602@egenix.com> Ian Bicking wrote: > M.-A. Lemburg wrote: > >>> Eggs give room for package metadata that doesn't exist otherwise. >>> Putting dependencies aside, this is functionality that simply doesn't >>> exist with the standard distutils installation. In the case of >>> FormEncode, it doesn't make use of any egg features (except that other >>> packages may want to depend on it using setuptools). In the case of >>> other frameworks -- including TurboGears (which I think is the ultimate >>> packaging goal here) -- the Egg metadata really is important, it's not >>> just used for dependencies. >> >> >> >> Understood, but wouldn't it be reasonably possible to >> also install this meta-data into a standard site-packages >> package directory ? > > > An egg and Python packages don't map 1-to-1. An egg can contain > multiple packages (which is fairly uncommon so far), but also a > top-level package can exist in more than one egg (i.e., namespace > packages, like zope.interfaces or paste.script). The metadata belongs > to the egg, not to the package inside the egg. > > Also, some of the metadata is encoded in the directory name itself, like > the version information. I think this makes it easier to do some > scanning operations, without a single database of installed packages > (and also respecting sys.path manipulation). Well, yes, but all of this is only needed for the egg support. In order to keep compatibilty with the existing wide-spread approach to install packages in site-packages/ using "python setup.py install", it should be possible (and I believe this should be the default to not disrupt existing usage and documentation) to run "python setup.py install" with an eggified source distribution in addition to the command to install it as regular egg. Otherwise, we'll end up with completely confused users and two disjoint and incompatible installations mechanisms. > That said, I think it would be nice if the transition was smoother. > E.g., if a file "ElementTree-1.2.6.egg-provided" could point to an > installed elementtree library (similar to the currently-supported > .egg-link file, but also slightly different). And, perhaps, > elementtree/ElementTree.egg-info could exist (with the same data as the > current ElementTree-1.2.6/EGG-INFO), though I think the simpler case > where extra metadata is disallowed would be easier. That would only > work for situations when there's a 1-to-1 mapping from packages to > eggs/projects, but that covers many situations, especially cases where > we're currently seeing conflicts. You lose the ability to easily > support multiple versions of a package with this, though that could > probably be handled too. I'm not suggesting to port over all the features you get from using setuptools' eggs (even though I do believe that you can go a long way using a special egg import hook), but it should be possible to get a regular working installation using "python setup.py install". PS: I understand setuptools and eggs as feature set which adds functionality to distutils, not as competitive and disjoint all-in-one solution. The latter won't fly well with installations that require native installers to be used such as Debian's apt-get, rpm and all the others. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 22 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From pje at telecommunity.com Tue Nov 22 22:05:49 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 16:05:49 -0500 Subject: [Distutils] Version numbers with a letter at the end In-Reply-To: <B5EC7670-5B85-40BD-B314-633C6651AE1A@groovie.org> Message-ID: <5.1.1.6.0.20051122155525.01fcc188@mail.telecommunity.com> At 11:55 AM 11/22/2005 -0800, Ben Bangert wrote: >Does setuptools distinguish and recognize letters in a version? > >For example, the latest Myghty is: > version='0.99a' > >When I up it to '0.99a.1', and setuptools does a pre-requisite check, >it doesn't seem to recognize that 0.99a.1 is more recent than 0.99a. Really? I'm not able to reproduce this: >>> from pkg_resources import parse_version as pv >>> pv('0.99a.1') > pv('0.99a') True Are you sure those are the version numbers at issue? >Also, this seems like it would cause problems with requirements that >look for a dev version, as it'd be: >"Myghty >= 0.99.adev-r1980" If you have multiple pre-release tags like 'a' and 'dev', you should separate them with a '.' - setuptools reads a contiguous run of letters as a single pre- or post-release tag, so the above reads as the 'adev' prerelease, not the 'dev' prerelease of the 'a' prerelease. Compare: >>> pv("0.99.adev-r1980") ('00000000', '00000099', '*adev', '*final-', '*r', '00001980', '*final') >>> pv("0.99.a.dev-r1980") ('00000000', '00000099', '*a', '*dev', '*final-', '*r', '00001980', '*final') Also, you don't need a dot to separate a numeric run from an alpha run, although you can include it if you want: >>> pv("0.99.a.dev-r1980")== pv("0.99a.dev-r1980") True Last week, I added a new documentation section to the setuptools.txt manual explaining the versioning scheme in more detail, and including tips about defining your pre- and post-release tagging schemes. See: http://svn.python.org/projects/sandbox/trunk/setuptools/setuptools.txt Under the heading "Specifying Your Project's Version", which is not far from the top of the document. From mal at egenix.com Tue Nov 22 22:27:52 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 22 Nov 2005 22:27:52 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> References: <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> Message-ID: <43838D58.7000801@egenix.com> Phillip J. Eby wrote: > At 06:33 PM 11/22/2005 +0100, M.-A. Lemburg wrote: > >>Phillip J. Eby wrote: >> >>>Yes, it's true, zipfile import processing is faster than normal import >>>processing; >> >>Only after *all* ZIP files on sys.path have been scanned >>for their contents. The more you add to sys.path, the longer >>Python takes to startup. > > > This is simply not true. If you don't believe PEP 302 and site.py, measure > it for yourself. The *only* addition to startup is the time to actually > read the .pth file and append the entries to the list. > > >>What's worse is that the slow-down affects the whole Python >>installation - each and every application using Python will >>have to scan all these ZIP files in case it tries to import >>a non-existing module or one which it finds late on sys.path. > > > And how often do programs attempt to import non-existing modules along > performance critical paths? Every single time you fire up Python and the user has not installed a module called "sitecustomize" (which is deliberatly not shipped with Python), Python will scan the complete sys.path for this module... and that's just one example. It is rather common in Python code to test for the availability of a faster variant by trying an import (e.g. for XML parsers) and then falling back to some slower emulations. > Note by the way that "scan all these ZIP files" is a misleading term in any > case - the files are not "scanned". They are opened, and a small amount of > data is read from the end of the file. Nothing that I would consider > "scanning" is involved. The data read from the end of the file is the directory which is decoded using marshal functions. You normally call this scanning data. Like Martin said: you always have to read the whole ZIP directory - even if you're just interested in a single module with the file. Actually loading the module then requires decompressing the code which takes a whole lot longer than just reading a file from the file system. In summary, things get slower when importing from ZIP files; it really only makes sense for applications that have a long run time and where startup is not that important, e.g. Zope et al. The main argument for using ZIP imports is to easy distribution of complete pure-Python packages, not a performance gain. You'd use one of the freeze tools for that, e.g. mxCGIPython which creates a single file Python interpreter which has a really good startup time due to the fact that the Python lib is embedded into the Python executable as static data and then loaded on demand by the OS as needed. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 22 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From tanner at real-time.com Tue Nov 22 22:41:06 2005 From: tanner at real-time.com (Bob Tanner) Date: Tue, 22 Nov 2005 15:41:06 -0600 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <43838D58.7000801@egenix.com> References: <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> <43838D58.7000801@egenix.com> Message-ID: <200511221541.08761.tanner@real-time.com> On Tuesday 22 November 2005 03:27 pm, M.-A. Lemburg wrote: > In summary, things get slower when importing from ZIP files; > it really only makes sense for applications that have a long > run time and where startup is not that important, e.g. > Zope et al. I was going to stay out of this discussion :-) When I read the above, my knee-jerk reaction is: Where is the data to backup this statement? Follow up questions are: How much slower? We talking milliseconds, seconds, minutes? Yes, there are variables, here, but narrow them to a set number and compare zip vs directory? Previous post talked about memory consumption, again bytes, kilobytes, megabytes? -- Bob Tanner <tanner at real-time.com> | Phone : (952)943-8700 http://www.real-time.com, Minnesota, Linux | Fax : (952)943-8500 Key fingerprint = AB15 0BDF BCDE 4369 5B42 1973 7CF1 A709 2CC1 B288 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20051122/c670c7a1/attachment.pgp From pje at telecommunity.com Tue Nov 22 23:05:34 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 17:05:34 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <4383858E.2080602@egenix.com> References: <438378C1.4050401@colorstudy.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> Message-ID: <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> At 09:54 PM 11/22/2005 +0100, M.-A. Lemburg wrote: >In order to keep compatibilty with the existing wide-spread >approach to install packages in site-packages/ using "python setup.py >install", it should be possible (and I believe this should be the >default to not disrupt existing usage and documentation) to >run "python setup.py install" with an eggified source >distribution in addition to the command to install it as >regular egg. "python setup.py install" *is* the command to install it as an egg - and the layout is no different than what occurs with an 'extra_path' setup() today. The only difference of any consequence is that it maintains a *single* .pth file for all eggs, rather than one .pth file per egg, and that's an improvement by the standard you and Martin have put forth, where saving startup time is a good thing. >I'm not suggesting to port over all the features you >get from using setuptools' eggs (even though I do believe >that you can go a long way using a special egg import >hook), Eggs don't need an import hook, which is one reason why they're so useful - they use the existing, well-established platform provided by Python 2.3 and above. > but it should be possible to get a regular working >installation using "python setup.py install". You already do! It's just that *dependencies* also need to be installed as eggs, in order for the depending package to take advantage. Without that, the depending package can't guarantee what version of the dependency is in use, thereby increasing the developer's support load. (Kevin Dangoor, the TurboGears author, has previously mentioned that he couldn't imagine trying to support TurboGears without eggs, simply due to the overhead of debugging people's dependency problems on multiple platforms and packaging systems. Even the support overhead caused by him making mistakes in using setuptools or by bugs in setuptools itself, was still dwarfed by the number of trouble-free installations on Windows and Mac OS. And over the last few months, I believe we've also succeeded in stomping most of the issues that people had with getting solid non-root installations on their Linux distributions. So the reasons for developers to prefer their dependencies to be managed as eggs will only improve over time, as the egg system allows Python developers to control and introspect their dependencies, rather than keeping that information hidden behind diverse platform-specific packaging tools. So, the issue being surfaced here is that Python developers who want to use other people's code will want the other people's code packaged as eggs, even if their own package doesn't need to be an egg itself (other than to take advantage of the automatic dependency handling). That's why easy_install exists: it can build eggs for most PyPI packages, even if the package author never heard of Python eggs. >PS: I understand setuptools and eggs as feature set which >adds functionality to distutils, not as competitive >and disjoint all-in-one solution. It's not disjoint at all, since it's trivial to package the majority of plain ol' distutils packages as eggs. It isn't even competitive, in the sense that you are not forced to choose between one or the other. You can have a "legacy" version of a package installed in the traditional way, and an egg version too. Code that isn't using the dependency management facility will import the legacy version, and code that does will see the egg version. It's only "competitive" if you feel that there must be only one way to do it. (And if you do feel that way, then it also should be obvious that eggs are the superior solution, since they don't take away any capabilities of the old, only provide new ones.) >The latter won't fly >well with installations that require native installers >to be used such as Debian's apt-get, rpm and all >the others. Currently, there is an experimental Gentoo "ebuild", and there's easy_deb. I'm talking with some people about putting together an "easy_rpm" equivalent. I recently heard that the easy_deb project was backed by Ubuntu, so it sounds like there's a fair amount of interest in supporting it out there. Mainly, I see this as an issue of the packagers having to adapt to meet their users' changing needs. It's reasonable to expect the major vendors to lag behind in terms of egg support, so there will naturally be a transitional period where tools like easy_deb and the hypothetical easy_rpm will be necessary. (Setuptools' "bdist_rpm" command builds egg-based RPMs already, but a separate easy_rpm that can also rpm/eggify legacy packages would be less invasive.) At the same time, it's also not reasonable for vendors to expect that ignoring eggs will make them go away - the practical advantages are far too compelling. A developer who targets the egg runtime gets cross-platform dependency management and has the option of doing distributed development tracking subversion releases of their dependencies, and doesn't have to worry about the specifics of how those dependencies will get resolved. He or she also gets to treat nearly all of PyPI as effectively his or her "standard library". This is a major advantage over developers who do not do this, not only in developer effectivness, but also because a developer who depends exclusively on a specific packaging system will not have the same effective reach for their offering, or conversely will require a greater investment of effort to support various packaging systems. To top it all off, the egg approach opens the possibility for system packagers to make use of developer-provided dependencies, thereby making the packaging process for Python packages potentially more robust. The only packages for which additional dependency information is required are Python extensions that link to system libraries -- and even for those, I expect that we can find a packager-independent way to add a lot of that information to the egg metadata. Obviously, every individual distribution would like to have Python packages conform to their individual system. However, on the whole, it is clearly better for the Python developer to have practical dependency management that doesn't tie their efforts to a single platform, packaging system, or distribution. From bob at redivi.com Tue Nov 22 23:04:45 2005 From: bob at redivi.com (Bob Ippolito) Date: Tue, 22 Nov 2005 14:04:45 -0800 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <43838D58.7000801@egenix.com> References: <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> <43838D58.7000801@egenix.com> Message-ID: <881594FE-967B-4F18-B2E5-5E918EDB537C@redivi.com> On Nov 22, 2005, at 1:27 PM, M.-A. Lemburg wrote: > Phillip J. Eby wrote: >> Note by the way that "scan all these ZIP files" is a misleading >> term in any >> case - the files are not "scanned". They are opened, and a small >> amount of >> data is read from the end of the file. Nothing that I would consider >> "scanning" is involved. > > The data read from the end of the file is the directory > which is decoded using marshal functions. You normally > call this scanning data. > > Like Martin said: you always have to read the whole ZIP > directory - even if you're just interested in a single > module with the file. > > Actually loading the module then requires decompressing > the code which takes a whole lot longer than just reading > a file from the file system. Last I checked, CPUs and RAM are a lot faster than disk. Unless it's sitting in cache already, reading a zip should be way faster than reading an uncompressed file. On top of that, I don't think egg zips are compressed by default... -bob From martin at v.loewis.de Tue Nov 22 23:25:15 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 22 Nov 2005 23:25:15 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <43835658.30606@egenix.com> References: <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> Message-ID: <43839ACB.9020709@v.loewis.de> M.-A. Lemburg wrote: > Doesn't the standard "python setup.py install" work with > eggified packages anymore (meaning that the package is > installed as normal site-packages package) ? No. First, an eggified package tries to download ez_setup first, i.e. it won't do the distutils setup(), but the easy_install one. Then, "setup.py install" will not install into, say, site-packages/mxTools, but into site-packages/mxTools-2.0.3.egg/mxTools, and add, to easy_install.pth, the line /usr/lib/python2.3/site-packages/mxTools-2.0.3.egg (which in turn adds this directory to sys.path) If you edit setup.py to use distutils setup instead of easy_install setup, it will install as before, but warn that setup() is being passed unrecognized parameters. Regards, Martin From pje at telecommunity.com Tue Nov 22 23:30:28 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 17:30:28 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <438378C1.4050401@colorstudy.com> References: <4383642F.8030804@egenix.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> Message-ID: <5.1.1.6.0.20051122170625.01fcae60@mail.telecommunity.com> At 02:00 PM 11/22/2005 -0600, Ian Bicking wrote: >An egg and Python packages don't map 1-to-1. An egg can contain multiple >packages (which is fairly uncommon so far), but also a top-level package >can exist in more than one egg (i.e., namespace packages, like >zope.interfaces or paste.script). The metadata belongs to the egg, not to >the package inside the egg. In addition, an egg can contain various Python modules, and still have metadata even if no packages are involved. >That said, I think it would be nice if the transition was smoother. E.g., >if a file "ElementTree-1.2.6.egg-provided" could point to an installed >elementtree library (similar to the currently-supported .egg-link file, >but also slightly different). And, perhaps, >elementtree/ElementTree.egg-info could exist (with the same data as the >current ElementTree-1.2.6/EGG-INFO), though I think the simpler case where >extra metadata is disallowed would be easier. It's not necessary to create a new way to do this; you can simply create 'ElementTree.egg-info' in site-packages and put the PKG-INFO file in there (and any other egg-info files), and it's supported by the existing mechanisms just fine, as long as the project has no top-level resources (files, data directories, etc.), and does not participate in namespace packages. So, for practical purposes, this would be more of a way to upgrade legacy packages to be detectable by egg-based packages, than a way to install egg packages as non-eggs. However, it might be a workable compromise for getting many of the Debian-packaged TurboGears dependencies to be usable while still mostly conforming to the existing Debian policy. A few months ago, this approach wouldn't have worked due to possible conflicts with locally-installed eggs, but setuptools now has runtime conflict management that can smooth it over as long as you haven't imported any of the conflicting packages yet. > That would only work for situations when there's a 1-to-1 mapping from > packages to eggs/projects, but that covers many situations, especially > cases where we're currently seeing conflicts. You lose the ability to > easily support multiple versions of a package with this, though that > could probably be handled too. This approach won't support multiple simultaneous versions, but then neither do most system packaging tools, and if this is strictly a workaround for system packagers who don't want to move everything to eggs, then it works just fine for that. They will, however, have to be careful about namespace packages in setuptools-based packages, since the package directories will be shared by two separately-installed projects, and each package will want to include its own __init__.py files. From martin at v.loewis.de Tue Nov 22 23:41:06 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 22 Nov 2005 23:41:06 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051122152125.01fc44a8@mail.telecommunity.com> References: <43834500.2090009@colorstudy.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43834500.2090009@colorstudy.com> <5.1.1.6.0.20051122152125.01fc44a8@mail.telecommunity.com> Message-ID: <43839E82.3090809@v.loewis.de> Phillip J. Eby wrote: >> If you have many zipfiles on sys.path, all applications will suffer >> from having to read the TOC of all those zipfiles, even if they need >> none of them. OTOH, if you had packages inside site-python, the >> contents of the unused packages is simply ignored. > > > I'm sorry, but this is, shall we say, "fact challenged"? .pth files' > contents are added to the *end* of sys.path. This means that stdlib > imports and normal site-packages imports are satisfied *before* any > hypothetical overhead from .pth entries, whether they're zipfiles or > directories. Correct. I was not talking about stdlib imports. I was talking about imports satisfied from the end of sys.path, or imports resulting in ImportErrors. > If Python never reaches the .pth entries at runtime, it > will not even read the zipfile TOCs, let alone attempting to stat() for > contained packages. Correct. However, a false preposition can imply anything: Python *always* reaches the .pth entries atleast once, in a typical installation, while looking for sitecustomize. This will cause a load of all zipfiles on sys.path, before site.py is done. > Please check your facts before spreading untruths like this I did check: I have a file a.pth in site-packages, which refers to a.zip (in the same directory), and I have an empty Python file e.py. Running strace -o xxx python e.py shows, among others open("/usr/lib/python2.3/site-packages/a.zip", O_RDONLY|O_LARGEFILE) = 5 ... ead(5, "PK\3\4\n\0\0\0\0\0\202\274v3\265<\267\r\16\0\0\0\16\0\0"..., 132) = 132 So a.zip is read even though the program does not contain a single import statement. What is the untruth I'm spreading? Regards, Martin From pje at telecommunity.com Tue Nov 22 23:42:35 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 17:42:35 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <43838D58.7000801@egenix.com> References: <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051122173224.03d0bbf0@mail.telecommunity.com> At 10:27 PM 11/22/2005 +0100, M.-A. Lemburg wrote: >Every single time you fire up Python and the user has not >installed a module called "sitecustomize" (which is deliberatly >not shipped with Python), Python will scan the complete sys.path >for this module... and that's just one example. Hm, I did forget about sitecustomize in this case. >It is rather common in Python code to test for the availability >of a faster variant by trying an import (e.g. for XML parsers) >and then falling back to some slower emulations. But a failed import is hardly a fast operation even with only a handful of directories on sys.path, so I don't see where this is hurting anything. In fact, if it's "rather common", then even the zip directory read cost is amortized over a larger total number of operations. > > Note by the way that "scan all these ZIP files" is a misleading term in > any > > case - the files are not "scanned". They are opened, and a small > amount of > > data is read from the end of the file. Nothing that I would consider > > "scanning" is involved. > >The data read from the end of the file is the directory >which is decoded using marshal functions. You normally >call this scanning data. Well, *I* normally wouldn't call it scanning, I'd call it reading, decoding, or marshalling. Scanning would be *looking* for something, versus reading and decoding a contiguous block of data at a well-known location. But oh well. >In summary, things get slower when importing from ZIP files; >it really only makes sense for applications that have a long >run time and where startup is not that important, e.g. >Zope et al. I'm terribly curious what Python applications exist for whom: 1. Startup time is a consideration, that 2. Haven't already been refactored to a long-running process. The only thing that occurs to me as even a possibility would be some kind of frequently-used system administration utility, like if you were going to rewrite all the bash builtin commands as Python scripts. (And no, I'm not being sarcastic, I really *am* curious about what kind of tools this would be an actual issue for.) >The main argument for using ZIP imports is to easy >distribution of complete pure-Python packages, not >a performance gain. You'd use one of the freeze tools >for that, e.g. mxCGIPython which creates a single >file Python interpreter which has a really good >startup time due to the fact that the Python lib >is embedded into the Python executable as static data >and then loaded on demand by the OS as needed. Then why was the python##.zip entry added to sys.path in Python 2.3? My understanding was that it was added to allow Python to start faster by cutting down on extraneous stat() calls. From martin at v.loewis.de Tue Nov 22 23:45:50 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 22 Nov 2005 23:45:50 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> References: <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> Message-ID: <43839F9E.3060106@v.loewis.de> Phillip J. Eby wrote: > This is simply not true. If you don't believe PEP 302 and site.py, > measure it for yourself. The *only* addition to startup is the time to > actually read the .pth file and append the entries to the list. I did. strace shows that all zip files are loaded. > And how often do programs attempt to import non-existing modules along > performance critical paths? Every time. Atleast sitecustomize is imported in most programs (except those skipping site.py), and is not present in most installations. The standard library catches ImportError about 250 times, although fewer expect the failure in a typical installation. Regards, Martin From martin at v.loewis.de Tue Nov 22 23:56:12 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 22 Nov 2005 23:56:12 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> References: <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> Message-ID: <4383A20C.3040307@v.loewis.de> Phillip J. Eby wrote: >> Debian should provide the packages, but not as eggs. > > > For packages that only operate as eggs, and/or require their > dependencies as eggs, you are stating a contradiction in terms. Eggs > are not merely a distribution format, any more than Java .jar files are. So I should say "Debian should not provide eggs, period", since what Debian provides are packages, and eggs are not? >> Debian developers should work with upstream authors to keep a >> distutils-based setup.py operational. > > > It's perfectly operational; clearly the entire egg system is *well* > within the Python runtime's intended operating parameters, as it uses > only well-defined and published aspects of the Python language, API, > stdlib, and build process. I didn't say the egg system in inoperational. I said that distutils setup is not operational for, for example, FormEncode: this uses another packaging library in setup.py, not distutils setup. > Perhaps you have some other definition of "operational" in mind? I had "*distutils-based* setup.py" in mind. > As > I've already stated, applying this same policy to Java libraries would > be to demanding that all the .class files be extracted to the filesystem > and any manifest files be deleted, before Debian would consent to > package them. In other words, it would be silly and pointless, because > the users would then ignore the packages in favor of actual jars, > because then their applications would actually work. This is not the same. A java .jar file is deployed by putting it on disk. For an egg, an (apparently undocumented) number of additional steps is necessary, such as editing easy-install.pth. In Java, the drawback of course is that each user has to edit CLASSPATH to include all the jar files desired. easy_setup makes this unnecessary, but in a way unfriendly to dpkg (and I assume other Linux package formats). Regards, Martin From pje at telecommunity.com Tue Nov 22 23:59:00 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 17:59:00 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <43839ACB.9020709@v.loewis.de> References: <43835658.30606@egenix.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> Message-ID: <5.1.1.6.0.20051122174646.03af9820@mail.telecommunity.com> At 11:25 PM 11/22/2005 +0100, Martin v. L?wis wrote: >M.-A. Lemburg wrote: >>Doesn't the standard "python setup.py install" work with >>eggified packages anymore (meaning that the package is >>installed as normal site-packages package) ? > >No. First, an eggified package tries to download ez_setup >first, i.e. it won't do the distutils setup(), Please try to check these facts a little more closely. First of all, these packages do *not* download ez_setup. ez_setup is simply a tool to ensure that *setuptools* is installed. Quite reasonably, a package that uses setuptools in its setup script, requires that setuptools be present on the machine. ez_setup allows packages to do this without requiring the user to manually download and install setuptools. Instead, the user runs "python setup.py install", and the issue of downloading the setuptools egg is handled as a convenient byproduct. If a sufficiently-recent version of setuptools is already available, and the package has no other unsatisfied dependencies, no downloading occurs. >but the easy_install >one. Then, "setup.py install" will not install into, say, >site-packages/mxTools, but into >site-packages/mxTools-2.0.3.egg/mxTools, and add, to easy_install.pth, >the line > >/usr/lib/python2.3/site-packages/mxTools-2.0.3.egg > >(which in turn adds this directory to sys.path) Note that if a distutils project uses the extra_path argument, the exact same two things will happen, except that each project will get its own .pth file, or a collision will occur if two projects use the same .pth name. You can't reasonably claim that this layout is somehow not a distutils layout; packages like Numeric have been using extra_path installs for years. >If you edit setup.py to use distutils setup instead of >easy_install setup, it will install as before, but warn >that setup() is being passed unrecognized parameters. And depending on the package, a variety of features will then fail to operate correctly, either at build time, sdist time, or run time. From martin at v.loewis.de Wed Nov 23 00:00:37 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Nov 2005 00:00:37 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <43838D58.7000801@egenix.com> References: <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> <43838D58.7000801@egenix.com> Message-ID: <4383A315.2030503@v.loewis.de> M.-A. Lemburg wrote: > The main argument for using ZIP imports is to easy > distribution of complete pure-Python packages, not > a performance gain. Precisely. For this reason, python2x.zip is on sys.path, allowing you to include the entire library (subset) in a single file. It may also be of convenience to users and local admins which have to install the same package across different machines, when the system vendor does not provide a package, but it appears that eggs are currently unfriendly to system vendors which want to include eggified packages in their own packaging system. Regards, Martin From pje at telecommunity.com Wed Nov 23 00:03:19 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 18:03:19 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <881594FE-967B-4F18-B2E5-5E918EDB537C@redivi.com> References: <43838D58.7000801@egenix.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> <43838D58.7000801@egenix.com> Message-ID: <5.1.1.6.0.20051122180034.01fbd988@mail.telecommunity.com> At 02:04 PM 11/22/2005 -0800, Bob Ippolito wrote: >Last I checked, CPUs and RAM are a lot faster than disk. Unless it's >sitting in cache already, reading a zip should be way faster than >reading an uncompressed file. On top of that, I don't think egg zips >are compressed by default... Actually, they are. However, I also don't believe that decompression tends to be a very computationally intensive activity. It would be interesting, however, to see some actual comparisons of import times between compressed, uncompressed, and entirely unzipped eggs. From pje at telecommunity.com Wed Nov 23 00:12:21 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 18:12:21 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <43839E82.3090809@v.loewis.de> References: <5.1.1.6.0.20051122152125.01fc44a8@mail.telecommunity.com> <43834500.2090009@colorstudy.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43834500.2090009@colorstudy.com> <5.1.1.6.0.20051122152125.01fc44a8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051122180357.03ab66b0@mail.telecommunity.com> At 11:41 PM 11/22/2005 +0100, Martin v. L?wis wrote: >What is the untruth I'm spreading? Your statements implied that a higher performance cost would be paid by the mere existence of additional directories on sys.path. This is not true, since it is quite possible to run programs without having any failed imports, if a sitecustomize.py is present. In all honesty, I had forgotten about sitecustomize.py, and so thought this was much "more untrue" than it actually is. Due to the relative infrequent appearance of sitecustomize.py, your statement about this in fact true for the majority of installations, and I apologize for my statement. From martin at v.loewis.de Wed Nov 23 00:25:14 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Nov 2005 00:25:14 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <200511221541.08761.tanner@real-time.com> References: <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> <43838D58.7000801@egenix.com> <200511221541.08761.tanner@real-time.com> Message-ID: <4383A8DA.8070807@v.loewis.de> Bob Tanner wrote: > When I read the above, my knee-jerk reaction is: Where is the data to backup > this statement? One could show strace outputs, and compare the number of system calls. Compiling this into actual timing is difficult: you would have to trade stat calls for read calls, and you would have to take the operating system's disk and directory caching into account. > How much slower? Not noticably, for a small number of zip files, and non-trivial Python scripts. Likewise, the presumed speed advantage of zip files (for zip file directory hits) is just as unnoticable, due to OS caching (depending on the OS, of course, and sizes of directories). > We talking milliseconds, seconds, minutes? Yes, there are > variables, here, but narrow them to a set number and compare zip vs > directory? Milliseconds, either way. > Previous post talked about memory consumption, again bytes, kilobytes, > megabytes? This is easier to say. For FormEncode-0.4, which contains 51 files, the Zip directory is 3625 bytes. Now, this is stored in memory somehow: Each file ends up as a directory entry, e.g. 'formencode/fields.pyc': ('/tmp/FormEncode-0.4-py2.4.egg/formencode/fields.pyc', 8, 12603, 34753, 69084, -28392, 13172, -784586915) This makes 12 pointers (48 bytes) for pointers in containers per file (3 in the dictionary, 9 in the tuple), plus 7*12 byte for the integers (some shared, of course), plus 3*12 bytes for the string and tuple headers, plus the characters for the strings (including null bytes, 4146 bytes for FormEncode). Counting this all together, the in-memory usage is about 4300 bytes for FormEncode; if I had this in /usr/lib/python2.3/site-packages instead of /tmp, it would be larger. This is roughly 2% of the egg file size. Larger eggs likely have more files, so this might serve as an estimate. The memory overhead would occur for all eggs installed, for all Python scripts. Regards, Martin From martin at v.loewis.de Wed Nov 23 00:35:57 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Nov 2005 00:35:57 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051122173224.03d0bbf0@mail.telecommunity.com> References: <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> <5.1.1.6.0.20051122173224.03d0bbf0@mail.telecommunity.com> Message-ID: <4383AB5D.2040107@v.loewis.de> Phillip J. Eby wrote: > The only thing that occurs to me as even a possibility would be some > kind of frequently-used system administration utility, like if you were > going to rewrite all the bash builtin commands as Python scripts. This whole discussion is not about whether the start time actually matters - it is about whether it is a fact or not that eggs improve the startup. Some people said it does, others said it doesn't, and this is just the finding-of-facts phase. Anyway, > I'm terribly curious what Python applications exist for whom: > 1. Startup time is a consideration, that > 2. Haven't already been refactored to a long-running process. For this, CGI scripts come to mind. Many people use them, and they are often short-running, and they often get invoked frequently. > Then why was the python##.zip entry added to sys.path in Python 2.3? My > understanding was that it was added to allow Python to start faster by > cutting down on extraneous stat() calls. PEP 273 doesn't give much rationale: "Booting" ... "Just as there are default directories in sys.path, there must be one or more default zip archives too." IIRC, it was to simplify deployment, having the entire library in a single file. Regards, Martin From pje at telecommunity.com Wed Nov 23 00:47:47 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 18:47:47 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <4383A20C.3040307@v.loewis.de> References: <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> At 11:56 PM 11/22/2005 +0100, Martin v. L?wis wrote: >Phillip J. Eby wrote: >>>Debian should provide the packages, but not as eggs. >> >>For packages that only operate as eggs, and/or require their dependencies >>as eggs, you are stating a contradiction in terms. Eggs are not merely a >>distribution format, any more than Java .jar files are. > >So I should say > >"Debian should not provide eggs, period", since what Debian provides >are packages, and eggs are not? I don't understand you. >>>Debian developers should work with upstream authors to keep a >>>distutils-based setup.py operational. >> >>It's perfectly operational; clearly the entire egg system is *well* >>within the Python runtime's intended operating parameters, as it uses >>only well-defined and published aspects of the Python language, API, >>stdlib, and build process. > >I didn't say the egg system in inoperational. I said that distutils >setup is not operational for, for example, FormEncode: this uses >another packaging library in setup.py, not distutils setup. I still don't understand you. If a package subclasses a distutils command, is it no longer a distutils setup? What if it bundles a library module that includes a subclass of a distutils command? Where, precisely, do you draw the line between a "distutils setup" and something else? *Many* packages subclass distutils commands or use unusual arguments to the distutils setup() that cause things to be installed in unusual ways. I'm very familiar with this because easy_install tries to support as many of those quirky subclasses or arguments as is practical, so it seems to me that the definition of a "distutils setup" is nowhere near as clear cut as your statements here imply. >>As I've already stated, applying this same policy to Java libraries would >>be to demanding that all the .class files be extracted to the filesystem >>and any manifest files be deleted, before Debian would consent to package >>them. In other words, it would be silly and pointless, because the users >>would then ignore the packages in favor of actual jars, because then >>their applications would actually work. > >This is not the same. A java .jar file is deployed by putting it on disk. >For an egg, an (apparently undocumented) An egg must be on sys.path, if you want to use it without explicitly using the egg runtime. See "The Quick Guide To Python Eggs", in particular this passage from http://peak.telecommunity.com/DevCenter/PythonEggs#using-eggs : "If you have a pure-Python egg that doesn't use any in-package data files, and you don't mind manually placing it on sys.path or PYTHONPATH, you can use the egg without installing setuptools." >number of additional >steps is necessary, such as editing easy-install.pth. Nothing except performance considerations prevents you having a separate .pth file for each and every egg, just as nothing prevents distutils packages from being installed as directory+.pth today. Does Debian currently reject packages that use the extra_path argument to setup(), like Numeric? >In Java, the drawback of course is that each user has to edit >CLASSPATH to include all the jar files desired. easy_setup >makes this unnecessary, but in a way unfriendly to dpkg (and >I assume other Linux package formats). I don't understand you here. Are you saying that it's not possible for dpkg to do a post-install or uninstall operation like adding or removing a line from a file? In any case, if you look at the approach Ian Bicking suggested and I commented further on, you'll see that you *can* in fact bypass this whole issue by packaging the egg metadata in another form, that gets rid of the need for .egg files or directories, as well as .pth manipulation. That approach, however, is not significantly documented at this time (other than a post to the distutils-SIG earlier this year outlining the design), but I'd be more than happy to document it further, if it makes the need for the rest of this discussion go away. :) Here are the steps to create a "single-version" egg: 1. Build the egg 2. Unzip the egg directly into site-packages, but rename the EGG-INFO subdirectory in the process to ProjectName.egg-info, where ProjectName is the pkg_resources.safe_name() of the setup(name="...") argument. (Alternately, you can take 'filename.split("-")[0].replace("_","-")', where 'filename' is the os.path.basename of the egg.) 3. (optional) remove any .py/.pyc/.pyo files that have an adjacent C extension file of the same name, such as 'foo.py' and 'foo.pyd' or 'foo.so'. The .py/.pyc/.pyo are stubs created by setuptools to extract the C extension from a zipped egg at runtime, and are not needed by an extracted installation. (This step is optional because Python's import gives precedence to the C extensions over the .py files, so nothing bad will happen if you don't delete the files.) What this process will *not* do for you is address conflicts in top-level data files, nor will it allow you to deal with packages that are partly installed by one package, and partly by another. For example, Ian's Paste package has a 'paste' package that is split across multiple eggs, and I expect this to be a popular feature for PEAK and Zope in the future. The 'pkgutil' module added in Python 2.3 was added to support namespace packages, but if you install the parts of a namespace package in the same directory you're going to have to deal with the fact that they all will want to install the '__init__.py'. If you are using this "single-version egg" approach, you will probably need to create an extra Debian package to hold the __init__.py, and have the individual packages depend on this package. Of course, this creates additional work for package maintainers that wouldn't be present with setuptools' normal .egg file/directory distributions, and my assumption was that the maintainers would prefer to be able to ignore such issues and get the benefit of dependencies defined by the upstream developers. Eggs keep each project in its own little bubble, where it can't overwrite anything else and can be uninstalled without removing any overlapping parts. From jjl at pobox.com Wed Nov 23 00:51:59 2005 From: jjl at pobox.com (John J Lee) Date: Tue, 22 Nov 2005 23:51:59 +0000 (UTC) Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <4383A20C.3040307@v.loewis.de> References: <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <4383A20C.3040307@v.loewis.de> Message-ID: <Pine.LNX.4.58.0511222338170.6518@alice> On Tue, 22 Nov 2005, "Martin v. L?wis" wrote: > > As > > I've already stated, applying this same policy to Java libraries would > > be to demanding that all the .class files be extracted to the filesystem > > and any manifest files be deleted, before Debian would consent to > > package them. In other words, it would be silly and pointless, because > > the users would then ignore the packages in favor of actual jars, > > because then their applications would actually work. > > This is not the same. A java .jar file is deployed by putting it on > disk. For an egg, an (apparently undocumented) number of additional > steps is necessary, such as editing easy-install.pth. > > In Java, the drawback of course is that each user has to edit > CLASSPATH to include all the jar files desired. easy_setup > makes this unnecessary, but in a way unfriendly to dpkg (and > I assume other Linux package formats). Actually, I believe this is the eventual primary intended mode of operation of eggs. In that case, the necessary sys.path manipulation is handled either by setuptools' wrapper scripts / wrapper .exe files calling require(version), or by calling that function elsewhere. The difference from .jar files is that you have the *option* of doing a global install into site-packages, which adds an entry to the .pth file. Phillip's primary use case for the whole egg / setuptools thing is, IIUC, precisely the zero-install case, in particular to support installation of plug-ins by dropping them into a directory, without even requiring any "add it to a path" step. Can I ask you (Martin) a couple of questions? 1. Does the above affect your concern about reading many zip files? 2. I understand your concern about memory usage (though the above seems to make it a non-issue in practice, if used sensibly), but I must have missed the argument you made for setuptools and/or Python Eggs being problematic for distributors such as Debian and Gentoo. What specifically is/are the problem(s)? It seems at least two distributions are already actively moving towards use of Python Eggs, so it would be good to inform those distributors of any problem you see before they get too far. John From pje at telecommunity.com Wed Nov 23 00:57:13 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 18:57:13 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <4383AB5D.2040107@v.loewis.de> References: <5.1.1.6.0.20051122173224.03d0bbf0@mail.telecommunity.com> <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> <5.1.1.6.0.20051122173224.03d0bbf0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051122185054.0317bb18@mail.telecommunity.com> At 12:35 AM 11/23/2005 +0100, Martin v. L?wis wrote: >For this, CGI scripts come to mind. Many people use them, and they >are often short-running, and they often get invoked frequently. I find that surprising, since I only use CGI if I'm not concerned about the start time. It's not like there aren't dozens of "long-running process" solutions for Python web apps including mod_python, FastCGI, SCGI, Twisted, and even ReadyExec, to fit almost every conceivable need. And since the advent of WSGI, more frameworks can be used with more of those deployment options than ever before. I guess I can imagine the extra millisecond or two pushing a borderline case over from "making do with CGI" to "need an LRP solution", but I'd also think that anybody with that kind of a situation would already know it. I could be wrong of course, and I'm not arguing that it's *not* a possible issue for some people, I'm just saying why it didn't occur to me as a use case for needing faster startup. It also seems strange to me to think the import time (especially import time of non-existent modules) would be a significant factor compared to the actual request handling time. From jjl at pobox.com Wed Nov 23 01:00:44 2005 From: jjl at pobox.com (John J Lee) Date: Wed, 23 Nov 2005 00:00:44 +0000 (UTC) Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> References: <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> Message-ID: <Pine.LNX.4.58.0511222359000.6518@alice> On Tue, 22 Nov 2005, Phillip J. Eby wrote: > At 11:56 PM 11/22/2005 +0100, Martin v. L?wis wrote: > >Phillip J. Eby wrote: > >>>Debian should provide the packages, but not as eggs. > >> > >>For packages that only operate as eggs, and/or require their dependencies > >>as eggs, you are stating a contradiction in terms. Eggs are not merely a > >>distribution format, any more than Java .jar files are. > > > >So I should say > > > >"Debian should not provide eggs, period", since what Debian provides > >are packages, and eggs are not? > > I don't understand you. [...] I can only assume Martin is reading 'package' to mean 'a unit of software distribution' rather than 'a Python package'. John From jjl at pobox.com Wed Nov 23 01:15:45 2005 From: jjl at pobox.com (John J Lee) Date: Wed, 23 Nov 2005 00:15:45 +0000 (UTC) Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <Pine.LNX.4.58.0511222338170.6518@alice> References: <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <4383A20C.3040307@v.loewis.de> <Pine.LNX.4.58.0511222338170.6518@alice> Message-ID: <Pine.LNX.4.58.0511230012040.6518@alice> On Tue, 22 Nov 2005, John J Lee wrote: [...] > Actually, I believe this is the eventual primary intended mode of > operation of eggs. In that case, the necessary sys.path manipulation is > handled either by setuptools' wrapper scripts / wrapper .exe files calling > require(version), or by calling that function elsewhere. The difference > from .jar files is that you have the *option* of doing a global install > into site-packages, which adds an entry to the .pth file. [...] Forgot to add: eggs may also be installed without use of zip files. Given that, why are the properties of zip file imports necessarily relevant to people using eggs (whether they be users of software distribututions or people concerned about memory usage or startup time)? John From pje at telecommunity.com Wed Nov 23 01:21:18 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 19:21:18 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <Pine.LNX.4.58.0511222338170.6518@alice> References: <4383A20C.3040307@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <4383A20C.3040307@v.loewis.de> Message-ID: <5.1.1.6.0.20051122190635.031191a0@mail.telecommunity.com> At 11:51 PM 11/22/2005 +0000, John J Lee wrote: >1. Does the above affect your concern about reading many zip files? > >2. I understand your concern about memory usage (though the above seems to >make it a non-issue in practice, if used sensibly), but I must have missed >the argument you made for setuptools and/or Python Eggs being problematic >for distributors such as Debian and Gentoo. What specifically is/are the >problem(s)? It seems at least two distributions are already actively >moving towards use of Python Eggs, so it would be good to inform those >distributors of any problem you see before they get too far. Actually, the .egg-info approach that Ian reminded me of should alleviate *all* the concerns raised so far, although it requires more hands-on management of the package definitions. However, if the Debian packagers don't care about this (it's likely no significant change for the better or worse compared to what they have to do already), then they can take a pass altogether on the file/directory issues, at the cost of slowing down tools that actually *do* use eggs. The tradeoff is that .egg-info files have to have their PKG-INFO files opened and read in order to do dependency processing.... but actually, if they created full ProjectName-Version.egg-info directories, this issue could actually be bypassed as well. Hm. Yes, that's even better. I didn't think of this before because the normal use of .egg-info directories is for "packaging" a source tree as an egg, and you don't want to have to rename the directory every time the version changes. But for a system packager, this isn't an issue. So, I guess for packaging tools that can't support multiple versions being installed (or don't desire to), the .egg-info approach allows them to precisely preserve the status quo with respect to the implementation tradeoffs, without hurting anybody or breaking anything. Now that it's clear to me that we can skirt the performance issue for .egg-info directories, it seems reasonable to recommend this approach as a low-risk way for system vendors to support Python eggs, and to provide a way for them to expose metadata for their existing Python packages in a way that setuptools-based packages can use - just make a Project-Version.egg-info directory with a PKG-INFO in it, added to site-packages along with the actual package installation. Indeed, it seems like it would be reasonable to propose that perhaps the normal distutils install process could provide this additional metadata, which would then eliminate the need for any repackaging or upgrades to any tools that already use "setup.py install" to create their packages. This would address things like ElementTree, which isn't inherently egg-based, but which people would like to depend on without having to rely on a platform-specific packaging tool to resolve the dependency "out of band". From martin at v.loewis.de Wed Nov 23 01:21:59 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Nov 2005 01:21:59 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> References: <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> Message-ID: <4383B627.5000506@v.loewis.de> Phillip J. Eby wrote: >>>> Debian should provide the packages, but not as eggs. >>> >>> >>> For packages that only operate as eggs, and/or require their >>> dependencies as eggs, you are stating a contradiction in terms. Eggs >>> are not merely a distribution format, any more than Java .jar files are. >> >> >> So I should say >> >> "Debian should not provide eggs, period", since what Debian provides >> are packages, and eggs are not? > > > I don't understand you. This is getting difficult: I don't actually know what "a contradiction in terms" is. You seemed to be saying that eggs are not a distribution format. If so, Debian should not distribute them. If eggs are, in fact, a distribution format: what is the contradiction then? I would still claim that Debian should not distribute them, but instead distribute policy-conforming Debian packages instead. >> I didn't say the egg system in inoperational. I said that distutils >> setup is not operational for, for example, FormEncode: this uses >> another packaging library in setup.py, not distutils setup. > > > I still don't understand you. If a package subclasses a distutils > command, is it no longer a distutils setup? It is not a distutils setup because it does not invoke distutils.core.setup. Also, if it did so, and changed the install command to do something inherently different (like installing into a different target directory without the user asking for it), it is not a distutils setup anymore. > What if it bundles a > library module that includes a subclass of a distutils command? Where, > precisely, do you draw the line between a "distutils setup" and > something else? Extending distutils is fine. An extension is a feature that, if not invoked, has no effect. easy_setup changes install in a way that has an effect. > *Many* packages subclass distutils commands or use > unusual arguments to the distutils setup() that cause things to be > installed in unusual ways. Yes, but they preserve the default behaviour: packages get installed into subdirectories of site-packages if you invoke the install command. >> This is not the same. A java .jar file is deployed by putting it on >> disk. For an egg, an (apparently undocumented) > > > An egg must be on sys.path, if you want to use it without explicitly > using the egg runtime. See "The Quick Guide To Python Eggs", in > particular this passage from > http://peak.telecommunity.com/DevCenter/PythonEggs#using-eggs : > > "If you have a pure-Python egg that doesn't use any in-package data > files, and you don't mind manually placing it on sys.path or PYTHONPATH, > you can use the egg without installing setuptools." Right. However, easy_setup apparently edits a .pth file to achieve this effect. This is different from jarfiles, where the end-user is expected to put it on sys.path. While this would be possible to do for egg files as well (ie just drop it somewhere, and let the user edit PYTHONPATH), it violates Debian policy to do so: the packages should be immediately usable after the Debian package is installed. > Nothing except performance considerations prevents you having a separate > .pth file for each and every egg That is not true. Usability also suffers if sys.path becomes long. > just as nothing prevents distutils > packages from being installed as directory+.pth today. Does Debian > currently reject packages that use the extra_path argument to setup(), > like Numeric? No. The policy states "Install your modules into /usr/lib/pythonX.Y/site-packages/". This is somewhat imprecise, and it is not clear (to me) whether the python-numeric violates that policy or not. >> but in a way unfriendly to dpkg > I don't understand you here. Are you saying that it's not possible for > dpkg to do a post-install or uninstall operation like adding or removing > a line from a file? That is certainly possible - but currently, each maintainer would have to come up with his own solution. This is more tedious to do than it could be. > Here are the steps to create a "single-version" egg: > > 1. Build the egg > ... This is still tedious to do, but certainly fits with Debian conventions (policy or not) better than any other approach. > Of course, this creates additional work for package maintainers that > wouldn't be present with setuptools' normal .egg file/directory > distributions, and my assumption was that the maintainers would prefer > to be able to ignore such issues and get the benefit of dependencies > defined by the upstream developers. Eggs keep each project in its own > little bubble, where it can't overwrite anything else and can be > uninstalled without removing any overlapping parts. I don't see how the maintainer could use the dependency information in the egg files. Debian policy is that the .deb files need to define proper dependencies, so the maintainer has to lookup and edit the dependency information *anyway*. Using the egg package name is of limited, help, either, because Debian policy mandates a certain naming scheme for packages, giving the FormEncode package a name of python2.4-formencode. Regards, Martin From martin at v.loewis.de Wed Nov 23 01:40:33 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Nov 2005 01:40:33 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051122185054.0317bb18@mail.telecommunity.com> References: <5.1.1.6.0.20051122173224.03d0bbf0@mail.telecommunity.com> <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> <5.1.1.6.0.20051122173224.03d0bbf0@mail.telecommunity.com> <5.1.1.6.0.20051122185054.0317bb18@mail.telecommunity.com> Message-ID: <4383BA81.7090004@v.loewis.de> Phillip J. Eby wrote: > I find that surprising, since I only use CGI if I'm not concerned about > the start time. It's not like there aren't dozens of "long-running > process" solutions for Python web apps including mod_python, FastCGI, > SCGI, Twisted, and even ReadyExec, to fit almost every conceivable > need. And since the advent of WSGI, more frameworks can be used with > more of those deployment options than ever before. As an example, both MoinMoin and pypi (cheeseshop) ran as CGI scripts on python.org for quite some time. I'm not sure whether this is still the case case, but there were certainly many accesses to both, and they produced a significant load. Currently, viewcvs runs as a CGI script on svn.python.org. This is because I don't know what FastCGI, SCGI, Twisted, and ReadyExec are, and I have only heard of mod_python, and I don't have any time to learn any of these, and then find out how to adjust viewcvs to use them. I guess once the search engines find them, they will also contribute to the load. Also, the sf redirector (python.org/sf) is a CGI script. It runs fairly quickly these days, but it might not get invoked so often. Regards, Martin From jjl at pobox.com Wed Nov 23 01:50:19 2005 From: jjl at pobox.com (John J Lee) Date: Wed, 23 Nov 2005 00:50:19 +0000 (UTC) Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <4383B627.5000506@v.loewis.de> References: <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <4383B627.5000506@v.loewis.de> Message-ID: <Pine.LNX.4.58.0511230045560.6518@alice> On Wed, 23 Nov 2005, "Martin v. L?wis" wrote: [...] > I don't see how the maintainer could use the dependency information > in the egg files. The maintainers could use the dependency information in the egg files by writing a tool to automatically map from the egg dep info to whatever format their packaging system uses -- for example, .deb files. > Debian policy is that the .deb files need to > define proper dependencies, so the maintainer has to lookup > and edit the dependency information *anyway*. Using the egg > package name is of limited, help, either, because Debian policy > mandates a certain naming scheme for packages, giving the > FormEncode package a name of python2.4-formencode. Why does any of that block distributors from using egg dependency info? John From david at mantara.com Wed Nov 23 01:53:19 2005 From: david at mantara.com (David Arnold) Date: Wed, 23 Nov 2005 11:53:19 +1100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: Your message of "Tue, 22 Nov 2005 17:05:34 CDT." <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> Message-ID: <E1Eeisx-0008Py-00@server.0x1.org> -->"Phillip" == Phillip J Eby <pje at telecommunity.com> writes: Phillip> This is a major advantage over developers who do not do this, Phillip> not only in developer effectivness, but also because a Phillip> developer who depends exclusively on a specific packaging Phillip> system will not have the same effective reach for their Phillip> offering, or conversely will require a greater investment of Phillip> effort to support various packaging systems. <coming to this a little late> So, this would seem to imply that installation of eggs is similar to using PEAR or CPAN? If so, from the perspective of a Debian user, this is a major disadvantage. I'm used to having a single framework that manages my packages and their dependency relationships, and it works well. Adding a language-specific mechanism simply causes problems, with stray files installed into directories "owned" by a .deb package, versions of CPAN/PEAR-installed packages drifting out of date with the interpreter and standard library, and just the cognitive load of needing to deal with something other than apt-get. My experiences with CPAN/PEAR packages have been universally bad, and I now try very, very hard to use nothing except apt/dpkg. I understand that from a Python-only perspective eggs might have a bunch of ease-of-use advantages, but from my point of view I'd suggest it's better that the developer (or Debian packager) takes the trouble to make it work with dpkg so all Debian users get to maintain the consistency they're used to. My 2c, etc, d From pje at telecommunity.com Wed Nov 23 02:03:34 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 20:03:34 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <4383B627.5000506@v.loewis.de> References: <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051122192931.01f84530@mail.telecommunity.com> At 01:21 AM 11/23/2005 +0100, Martin v. L?wis wrote: >Phillip J. Eby wrote: >>>>>Debian should provide the packages, but not as eggs. >>>> >>>> >>>>For packages that only operate as eggs, and/or require their >>>>dependencies as eggs, you are stating a contradiction in terms. Eggs >>>>are not merely a distribution format, any more than Java .jar files are. >>> >>>So I should say >>> >>>"Debian should not provide eggs, period", since what Debian provides >>>are packages, and eggs are not? >> >>I don't understand you. > >This is getting difficult: I don't actually know what "a contradiction >in terms" is. You seemed to be saying that eggs are not a distribution >format. They are not a distribution format. There are in fact three physical formats that an egg can take (if we ignore .egg-link files, which are really needed only to work around the absence of symlinks on Windows). In principle, there could be many others. I suspect that part of the confusion stems that I prefer to use "package" to refer only to a Python package (thing you import), and not to refer to a distribution as a "package". However, Debian calls distributions "packages", so some confusion is perhaps inevitable. What's more, it appears that the Debian policy calls for the Debian package to be named for the contained Python package, regardless of whether that's the name of the distribution. An "egg" is a "distribution" of a "project" that is importable and can carry both standardized and individualized metadata that can be read by the pkg_resources module. There are various distribution *formats* in which an "egg" may be physically manifested, but the "egg" itself is a logical concept, not a physical one. It is therefore, as I said, "not merely a distribution format". Is that any clearer? The "contradiction in terms" was that I took your meaning of "package" to be the same as my term "project" - i.e., a functional collection of Python resources. Projects that *are* eggs, can't be provided "but not as eggs". They *are* eggs, so not providing them as eggs means not providing them at all. In contrast, projects that are not built with setuptools aren't inherently eggs, but you can certainly make eggs out of them. For these projects, you *do* have the choice to provide them "not as eggs", but then they are also of no use to the projects that need eggs. As we've already briefly discussed, in the simplest form a project can be made eggs just by adding an appropriately-named .egg-info/PKG-INFO file. > If so, Debian should not distribute them. This is what I don't understand, as it has nothing to do whether or not is a distribution format, at least not that I can see. My statement was that eggs are not merely a distribution format; they are a logical concept that can be physically packaged in various ways, and if it's necessary to invent yet another physical layout, well, we can do that too. >If eggs are, >in fact, a distribution format: what is the contradiction then? >I would still claim that Debian should not distribute them, but >instead distribute policy-conforming Debian packages instead. Which would be the same as saying you wouldn't distribute, say, setuptools itself. Setuptools is an egg, and can't function except as an egg, because it is more than a Python package. Again, an "egg" is some specific release of a project and its introspectable metadata. >>I still don't understand you. If a package subclasses a distutils >>command, is it no longer a distutils setup? > >It is not a distutils setup because it does not invoke >distutils.core.setup. Now I really don't understand you. Line 43 of setuptools/__init__.py reads: setup = distutils.core.setup So, how is it not invoking distutils.core.setup? >>What if it bundles a library module that includes a subclass of a >>distutils command? Where, precisely, do you draw the line between a >>"distutils setup" and something else? > >Extending distutils is fine. An extension is a feature that, if not >invoked, has no effect. easy_setup changes install in a way that >has an effect. So do all the packages that rework install_data to be more to their liking - and there are quite a lot of them, as I discovered when I began testing easy_install. >>Nothing except performance considerations prevents you having a separate >>.pth file for each and every egg > >That is not true. Usability also suffers if sys.path becomes long. How? I don't understand this. Someone using eggs rarely has reason to manually manipulate sys.path unless they are adding some kind of plugin directory to it. If they want to know what package version they are using, pkg_resources provides a superior API for querying it; I can say e.g. 'require("TurboGears")' and receive back a list of all the eggs that compose or are required by TurboGears, along with their locations. (Or conversely, receive a DistributionNotFound or VersionConflict error explaining what's missing or what was already imported that's a different version than the one needed.) > >> but in a way unfriendly to dpkg >>I don't understand you here. Are you saying that it's not possible for >>dpkg to do a post-install or uninstall operation like adding or removing >>a line from a file? > >That is certainly possible - but currently, each maintainer would have >to come up with his own solution. This is more tedious to do than it >could be. easy_deb implements this, so it seems to me it would be a simple matter of running easy_deb to produce the .deb from the .egg. (Caveat: I have not used easy_deb, but its author assures me that it is able to handle the .pth manipulation in a sane way.) >>Of course, this creates additional work for package maintainers that >>wouldn't be present with setuptools' normal .egg file/directory >>distributions, and my assumption was that the maintainers would prefer to >>be able to ignore such issues and get the benefit of dependencies defined >>by the upstream developers. Eggs keep each project in its own little >>bubble, where it can't overwrite anything else and can be uninstalled >>without removing any overlapping parts. > >I don't see how the maintainer could use the dependency information >in the egg files. Debian policy is that the .deb files need to >define proper dependencies, so the maintainer has to lookup >and edit the dependency information *anyway*. Using the egg >package name is of limited, help, either, because Debian policy >mandates a certain naming scheme for packages, giving the >FormEncode package a name of python2.4-formencode. What I would suggest here is having a namespace (e.g. pyegg2.4-whatever) for naming packages based on their PyPI names, so that there can be an automated relationship between setuptools dependencies and Debian ones. This doesn't work for existing Debian packages, of course, but it seems to me that they could in fact have the same contents as their pyegg cousin; both could simply use the .egg-info approach. (easy_deb uses python-pypi-whatever, which seems a bit long to me, but then, it's also implemented and my pyegg2.4 idea isn't.) Anyway, I don't see any obvious reasons why this can't be an automated process, even for the system library dependencies. easy_deb even has a simple configuration file that can augment the setuptools-style dependencies with explicit Debian dependencies. There's also nothing stopping us from defining a way to add Debian dependency information to setup(); in fact setuptools encourages this by offering an extensible system to allow distutils extensions to offer and validate new setup() keywords and use them to generate additional metadata in the egg. This would make it possible to push back Debian dependency information to upstream maintainers, if this were desired. From andrew at puzzling.org Wed Nov 23 02:06:50 2005 From: andrew at puzzling.org (Andrew Bennetts) Date: Wed, 23 Nov 2005 12:06:50 +1100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <4383AB5D.2040107@v.loewis.de> References: <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> <5.1.1.6.0.20051122173224.03d0bbf0@mail.telecommunity.com> <4383AB5D.2040107@v.loewis.de> Message-ID: <20051123010650.GB3073@home.puzzling.org> Martin v. L?wis wrote: [...] > > This whole discussion is not about whether the start time actually > matters - it is about whether it is a fact or not that eggs improve > the startup. Some people said it does, others said it doesn't, and this > is just the finding-of-facts phase. > > Anyway, > > > I'm terribly curious what Python applications exist for whom: > > 1. Startup time is a consideration, that > > 2. Haven't already been refactored to a long-running process. > > For this, CGI scripts come to mind. Many people use them, and they > are often short-running, and they often get invoked frequently. Another example would be bzr <http://bazaar.canonical.com/Bzr>; revision control command line tools (or command line tools in general, I suppose) feel much nicer to use when they respond immediately. It doesn't take many hundreds of milliseconds of startup delay for a tool to start feeling a little bit sluggish. -Andrew. From pje at telecommunity.com Wed Nov 23 02:12:32 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 20:12:32 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <4383BA81.7090004@v.loewis.de> References: <5.1.1.6.0.20051122185054.0317bb18@mail.telecommunity.com> <5.1.1.6.0.20051122173224.03d0bbf0@mail.telecommunity.com> <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> <5.1.1.6.0.20051122173224.03d0bbf0@mail.telecommunity.com> <5.1.1.6.0.20051122185054.0317bb18@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051122200819.02caa8a0@mail.telecommunity.com> At 01:40 AM 11/23/2005 +0100, Martin v. L?wis wrote: >Phillip J. Eby wrote: >>I find that surprising, since I only use CGI if I'm not concerned about >>the start time. It's not like there aren't dozens of "long-running >>process" solutions for Python web apps including mod_python, FastCGI, >>SCGI, Twisted, and even ReadyExec, to fit almost every conceivable >>need. And since the advent of WSGI, more frameworks can be used with >>more of those deployment options than ever before. > >As an example, both MoinMoin and pypi (cheeseshop) ran as CGI scripts >on python.org for quite some time. I'm not sure whether this is still >the case case, but there were certainly many accesses to both, and >they produced a significant load. > >Currently, viewcvs runs as a CGI script on svn.python.org. This is >because I don't know what FastCGI, SCGI, Twisted, and ReadyExec are, >and I have only heard of mod_python, and I don't have any time to >learn any of these, and then find out how to adjust viewcvs to >use them. I guess once the search engines find them, they will >also contribute to the load. > >Also, the sf redirector (python.org/sf) is a CGI script. It >runs fairly quickly these days, but it might not get invoked >so often. Sure, I run viewcvs and MoinMoin as CGI scripts as well, because they're low-volume use. My point was more that if a few milliseconds per request would make any meaningful difference to those applications' performance, I would have already migrated them to use one of the many long-running process options for Python web applications. That's why it didn't occur to me to consider that worth caring about. Again, I'm not saying it might not be an issue, just that I never even considered it because I've been doing long-running Python web apps with FastCGI for about 8-9 years now, so the idea of running something millisecond-critical as a CGI just isn't something that would occur to me. From pje at telecommunity.com Wed Nov 23 02:37:14 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Nov 2005 20:37:14 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <E1Eeisx-0008Py-00@server.0x1.org> References: <Your message of "Tue, 22 Nov 2005 17:05:34 CDT." <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051122201306.03a9a090@mail.telecommunity.com> At 11:53 AM 11/23/2005 +1100, David Arnold wrote: >-->"Phillip" == Phillip J Eby <pje at telecommunity.com> writes: > > Phillip> This is a major advantage over developers who do not do this, > Phillip> not only in developer effectivness, but also because a > Phillip> developer who depends exclusively on a specific packaging > Phillip> system will not have the same effective reach for their > Phillip> offering, or conversely will require a greater investment of > Phillip> effort to support various packaging systems. > ><coming to this a little late> > >So, this would seem to imply that installation of eggs is similar to >using PEAR or CPAN? Not at the level I think you mean. Apart from the .pth file, and any scripts, each egg has its *own*, 100% encapsulated file or directory, for example, which is quite different from at least CPAN. (I don't know anything about PEAR.) One thing in particular is significantly different: eggs have runtime discovery and introspection of metadata and dependencies. It would be more appropriate to compare them with Eclipse plugins or "OSGi bundles" for Java. (These are an enhanced jar format with dependency information, version info, platform metadata, etc.) >Adding a language-specific mechanism simply causes problems, with stray >files installed into directories "owned" by a .deb package, versions of >CPAN/PEAR-installed packages drifting out of date with the interpreter >and standard library, and just the cognitive load of needing to deal >with something other than apt-get. Eggs cannot overwrite each other's contents, or indeed anything else other than scripts (which can be directed to individual package directories ala Stow if you prefer). Eggs carry the Python version number they are built for. In addition, eggs do not have to be installed in system library directories; an application can simply dump all its eggs and its main script in a single directory, and then run from there without relying on system packages at all. The egg runtime identifies which versions of which eggs are needed to satisfy the dependencies when the script runs. > My experiences with CPAN/PEAR >packages have been universally bad, and I now try very, very hard to use >nothing except apt/dpkg. That's certainly understandable, but comparison with CPAN is definitely inappropriate, since many of the issues that can exist with CPAN (or the bare distutils in Python's case) simply cannot exist with .egg files and directories. For example, even if a user ran easy_install as root and installed a new version of a package, the older .egg would still remain on the system, and none of the files in the new egg would overwrite the older egg's files, since eggs are installed as either a zipfile or directory, named for the package/version/python-version/platform. (Note, by the way, that this means you can actualy install .eggs for multiple architectures in the same directory and get away with it.) Let's say the user installed SomeEgg-1.2, replacing the system-installed SomeEgg-1.1. If they want to put things back the way they were, they need only run "easy_install SomeEgg==1.1", which will find the still-untouched SomeEgg-1.1 sitting where it always was, and this will rewrite the .pth file to make SomeEgg-1.1 the active version again. Meanwhile, whatever program the user installed that needed SomeEgg-1.2 will most likely continue to work, as long as it's using the egg dependency machinery to get at it. Of course, the user who's installing some program that needs newer packages than are offered by the packaging vendor can simply designate another installation directory, and tell EasyInstall to put any new eggs and scripts there, instead of adding them to the main system. They are then nicely isolated from any system-level changes. >I understand that from a Python-only perspective eggs might have a bunch >of ease-of-use advantages, but from my point of view I'd suggest it's >better that the developer (or Debian packager) takes the trouble to make >it work with dpkg so all Debian users get to maintain the consistency >they're used to. Which is all very well and good, except there are plenty of packaging systems besides Debian, and platforms that don't even have anything resembling a packaging system. (And likely plenty of Python developers who've never heard of Debian, and a larger number who've never used it.) In any case, the current discussion is more about the issue of providing metadata so that Python developers can *tell* when the packaging system provides a usable version of something, without having to write tools for every packaging system in existence. Providing the .egg-info directory with Debian-installed packages is a reasonable solution for offering system-provided packages, but without using the .egg file/directory formats. (The irony here is that the solution perceived as more desirable here, is the one that *requires* the package maintainers to avoid inter-project conflicts, whereas the "default" format for eggs avoids such issues almost entirely!) As far as the benefits or lack thereof of Debian itself, I certainly can't say. However, having had to administer boxes with the RPM and pkgsrc systems, I would say that eggs are immensely superior for the Python developer, and I doubt that Debian has any features that would sway me on that point if I did use it. Just the ability to have SomeEgg-1.1 and SomeEgg-1.2 simultaneously available is a lifesaver, from my perspective. From tanner at real-time.com Wed Nov 23 07:17:48 2005 From: tanner at real-time.com (Bob Tanner) Date: Wed, 23 Nov 2005 00:17:48 -0600 Subject: [Distutils] .egg in Debian summary? References: <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <200511220104.03160.tanner@real-time.com> Message-ID: <dm11id$pdi$1@sea.gmane.org> Bob Tanner wrote: >> I don't think Debian should use the egg structure. It apparently relies >> on building a long sys.path (even though through only a single .pth >> file); > > I'm not sure of how .eggs are implemented, but I'm going to cross-post > this info to the python-distutils mailing list. Read and re-read the complete thread regarding .eggs in Debian and I cannot tell if any progress has been made. Still in the discussion/fact-finding stage? As "just a package maintainer" I was looking for the "options" to move forward. Looking at the thread, I think these are the options (skipping the pro's and con's for now): 1. Do nothing, go with the status quo as documented in the Debian python policy, which is no .egg's and unpackage everything into a sub-directory of site-packages. 2. Investigate easydeb <http://cheeseshop.python.org/pypi/easydeb/> 3. Using Phillip's .egg-info solution <http://permalink.gmane.org/gmane.comp.python.distutils.devel/2567> Any others? -- Bob Tanner <tanner at real-time.com> | Phone : (952)943-8700 http://www.real-time.com, Minnesota, Linux | Fax : (952)943-8500 Key fingerprint = AB15 0BDF BCDE 4369 5B42 1973 7CF1 A709 2CC1 B288 From wl at flexis.de Wed Nov 23 10:15:39 2005 From: wl at flexis.de (Wolfgang Langner) Date: Wed, 23 Nov 2005 10:15:39 +0100 Subject: [Distutils] Setuptools now in FreeBSD Ports Collection Message-ID: <dm1bvr$hme$1@sea.gmane.org> Hello, yesterday I discovered that setuptools are now in the FreeBSD ports collection: http://www.freshports.org/devel/py-setuptools/ (official ports index) http://www.freebsd.org/cgi/ports.cgi?query=setuptools&stype=all bye by Wolfgang From martin at v.loewis.de Wed Nov 23 11:08:43 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Nov 2005 11:08:43 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051122192931.01f84530@mail.telecommunity.com> References: <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <5.1.1.6.0.20051122192931.01f84530@mail.telecommunity.com> Message-ID: <43843FAB.6040006@v.loewis.de> As for terminology, you seem to suggest to use "distribution" where Debian uses "package". So "Debian package" would become "Debian distribution". This does not sound right, because "Debian distribution" is the entire collection of packages that is released e.g. on a DVD-ROM. I'll try to use "project" in your sense and "package" in the Python sense whenever I can. Phillip J. Eby wrote: > An "egg" is a "distribution" of a "project" that is importable and can > carry both standardized and individualized metadata that can be read by > the pkg_resources module. There are various distribution *formats* in > which an "egg" may be physically manifested, but the "egg" itself is a > logical concept, not a physical one. It is therefore, as I said, "not > merely a distribution format". Is that any clearer? Yes. When I said "an egg", I meant "a zipfile with a .egg extension, or a directory with a .egg extension". In response to # [...] who will quite simply need eggs for many packages. # If Debian doesn't provide them, the users will be forced to obtain # them elsewhere. I meant "Debian should provide the distributions, but not as .egg files"; it should provide the distribution as a deb file. So users are provided with the project, but in a form that is not one of the three forms an egg could have. > The "contradiction in terms" was that I took your meaning of "package" > to be the same as my term "project" - i.e., a functional collection of > Python resources. Projects that *are* eggs, can't be provided "but not > as eggs". They *are* eggs, so not providing them as eggs means not > providing them at all. I would expect that you can "unegg" a project. You can distribute the project as a collection of Python modules, not as a collection of Python resources. The Debian developer could (and I was suggesting he should) just ignore the entire egg structure, and distribute the code of the library only. >> If so, Debian should not distribute them. > > > This is what I don't understand, as it has nothing to do whether or not > is a distribution format, at least not that I can see. My statement was > that eggs are not merely a distribution format; they are a logical > concept that can be physically packaged in various ways, and if it's > necessary to invent yet another physical layout, well, we can do that too. Yes, but this logical concept is in the way of Debian packages/distributions (atleast if done naively by the Debian developer). This is what started the entire discussion: Matthias Urlichs complained that Bob Tanner included the egg structure in the formencode Debian package/distribution. The specific initial complaints where: - you can't use it with a simple "import formencode", - pydoc does not work on "eggs". I would add the complaint: - it increases sys.path for no good reason. > Which would be the same as saying you wouldn't distribute, say, > setuptools itself. Setuptools is an egg, and can't function except as > an egg, because it is more than a Python package. Again, an "egg" is > some specific release of a project and its introspectable metadata. I could rewrite setuptools to function as a regular Python package. After a shallow inspection, there aren't many places where it really needs the pkg_resources functionalities for itself - I could only identify the part that locates cli.exe. As this is used on Windows only, a Debian port of setuptools could simply ignore this code. >> It is not a distutils setup because it does not invoke >> distutils.core.setup. > > > Now I really don't understand you. Line 43 of setuptools/__init__.py > reads: > > setup = distutils.core.setup > > So, how is it not invoking distutils.core.setup? Ah, I didn't look so far. I noticed that when I replace from setuptools import setup with from distutils.core import setup I get warnings about package_data and extras_require, and assumed this means setup was a different function; instead, it really is the import that plays tricks here. >> Extending distutils is fine. An extension is a feature that, if not >> invoked, has no effect. easy_setup changes install in a way that >> has an effect. > > So do all the packages that rework install_data to be more to their > liking - and there are quite a lot of them, as I discovered when I began > testing easy_install. Right. It really isn't that much about what is and is not conforming; it more matters what the practical effects on the Debian developer are. If "setup.py install" just puts some files into some locations, and the files don't conflict with files in other Debian packages/distributions, the developer can easily package the entire thing. If "setup.py install" does other things, like editing an existing file, it is not so easy anymore. >> That is not true. Usability also suffers if sys.path becomes long. > > > How? I don't understand this. People will often inspect sys.path to understand where Python is looking for their code. They can do so manually if sys.path fits on one or two lines of terminal output. On my system, it is now four lines, primarily thanks to .pth files. This is getting unusable. >> That is certainly possible - but currently, each maintainer would have >> to come up with his own solution. This is more tedious to do than it >> could be. > > > easy_deb implements this, so it seems to me it would be a simple matter > of running easy_deb to produce the .deb from the .egg. (Caveat: I have > not used easy_deb, but its author assures me that it is able to handle > the .pth manipulation in a sane way.) I can't comment on this. Somebody probably should examine whether easy_deb complies with all policies, and this is what the developer should use (I'm worried though that I don't see a mentioning of dpkg-buildpackage in http://www.python.org/pypi/easydeb) > What I would suggest here is having a namespace (e.g. pyegg2.4-whatever) > for naming packages based on their PyPI names, so that there can be an > automated relationship between setuptools dependencies and Debian ones. That would be a policy change (I think). Whether it would be agreeable, I have no idea. > Anyway, I don't see any obvious reasons why this can't be an automated > process, even for the system library dependencies. easy_deb even has a > simple configuration file that can augment the setuptools-style > dependencies with explicit Debian dependencies. Debian policy currently seems to require that the dependencies are provided as plain text in a patch to the upstream sources(*). So the idea certainly is that dependencies are managed by the developer, not automatically. Regards, Martin (*) More precisely, it requires that after unpacking the source of the source package/distribution, the dependencies are listed in debian/control. From hawk78_it at yahoo.it Wed Nov 23 13:23:21 2005 From: hawk78_it at yahoo.it (Vincenzo Di Massa) Date: Wed, 23 Nov 2005 13:23:21 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <43843FAB.6040006@v.loewis.de> References: <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <5.1.1.6.0.20051122192931.01f84530@mail.telecommunity.com> <43843FAB.6040006@v.loewis.de> Message-ID: <200511231323.21251.hawk78_it@yahoo.it> Alle 11:08, mercoled? 23 novembre 2005, Martin v. L?wis ha scritto: > > easy_deb implements this, so it seems to me it would be a simple matter > > of running easy_deb to produce the .deb from the .egg. ?(Caveat: I have > > not used easy_deb, but its author assures me that it is able to handle > > the .pth manipulation in a sane way.) > > I can't comment on this. Somebody probably should examine whether > easy_deb complies with all policies, and this is what the developer > should use (I'm worried though that I don't see a mentioning > of dpkg-buildpackage in http://www.python.org/pypi/easydeb) easy-deb ( http://easy-deb.sf.net ) is just a tool that creates a debian source package. It is "more or less" like dh_make for autotools based projects. What easy-deb does is: 1) downloads the python module source (using setuptools) 2) reads the python module metadata (name, version, author, dependencies...) 3) converts dependencies from pypi names to debian package names (using an hand written mapping database which is just a text file) 4) populates the project ./debian directory with rules, control, changelog (to be improved) and some post-install scripts. Then the Debian developer is required to debuild or dpkg-buildpackage the sources to get the .deb as always. The ph file is handled this way: it stores pth files (one per package) inside a repository (a directory which is not on python path). When you install a new package, a new pth file is added to the repository. After the pakages' pth file are added to the repository, the repository is used to update a single pth file inside site-packages (the update is done by the post-install scripts). This way easy deb has just one pth file inside python path. Moreover easy-deb has cmdline tools that can disable/re-enable an egg by removing its pth file from the repository (the repository just contains symlinks to pth files like linux init scripts are just symlinks in /etc/rc?.d/) and re-updating the site-packages pth file. The script for updating the site-packages pth file us named update-pypi (to look like update-rc.d). The generated source debian pkg is generated from templates of rules, control, changelog... in which placeholders like %%VERSION%% are substituded by the actual metadata. If you change the templates easy-ded can create completely different packages without changing its code. > > What I would suggest here is having a namespace (e.g. pyegg2.4-whatever) > > for naming packages based on their PyPI names, so that there can be an > > automated relationship between setuptools dependencies and Debian ones. > > That would be a policy change (I think). Whether it would be agreeable, > I have no idea. > > > Anyway, I don't see any obvious reasons why this can't be an automated > > process, even for the system library dependencies. ?easy_deb even has a > > simple configuration file that can augment the setuptools-style > > dependencies with explicit Debian dependencies. > > Debian policy currently seems to require that the dependencies are > provided as plain text in a patch to the upstream sources(*). So the > idea certainly is that dependencies are managed by the developer, > not automatically. Using easy-deb dependencies are handled by the developer, but a "guessed and to look carefully at" list of dependencies is automatically available. In many circumstances the Debian developer will accept the easy-deb provided dependencyes making it really automatic. Vincenzo ___________________________________ Yahoo! Mail: gratis 1GB per i messaggi e allegati da 10MB http://mail.yahoo.it From p.f.moore at gmail.com Wed Nov 23 16:25:42 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 23 Nov 2005 15:25:42 +0000 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <43838D58.7000801@egenix.com> References: <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> <43838D58.7000801@egenix.com> Message-ID: <79990c6b0511230725x1d48544fjc227ba75ce73e1de@mail.gmail.com> On 11/22/05, M.-A. Lemburg <mal at egenix.com> wrote: > Actually loading the module then requires decompressing > the code which takes a whole lot longer than just reading > a file from the file system. > > In summary, things get slower when importing from ZIP files; > it really only makes sense for applications that have a long > run time and where startup is not that important, e.g. > Zope et al. It's a long time ago, and back prior to PEP 302, but I believe that the original version of the zipimport PEP (PEP 273 by Jim Ahlstrom) included some timings based on the original patch. While this did show slowdowns with compressed zip files, it showed distinct speedups with *un*compressed zip files. IIRC, this was explained as because a zipfile directory scan is faster than multiple filesystem stats. So someone once recommended that zipfiles on sys.path should be in uncompressed format, for performance reasons. That recommendation got lost a long time ago, though. Paul. From hawk78_it at yahoo.it Wed Nov 23 16:39:58 2005 From: hawk78_it at yahoo.it (Vincenzo Di Massa) Date: Wed, 23 Nov 2005 16:39:58 +0100 Subject: [Distutils] .egg in Debian summary? In-Reply-To: <dm11id$pdi$1@sea.gmane.org> References: <5bnlf-5nA-35@gated-at.bofh.it> <200511220104.03160.tanner@real-time.com> <dm11id$pdi$1@sea.gmane.org> Message-ID: <200511231639.59028.hawk78_it@yahoo.it> Alle 07:17, mercoled? 23 novembre 2005, Bob Tanner ha scritto: > 1. Do nothing, go with the status quo as documented in the Debian python > policy, which is no .egg's and unpackage everything into a sub-directory of > site-packages. > > 2. Investigate easydeb <http://cheeseshop.python.org/pypi/easydeb/> > > 3. Using Phillip's .egg-info solution > <http://permalink.gmane.org/gmane.comp.python.distutils.devel/2567> > 2 & 3 can be merged using easy-deb with the right template of debian/rules. even 1 & 1 can be merged using easy-deb to create a package that the debian developer will manually patch to remove/change egg functionality. I think setuptools could really help Debian developer as well as Fedora, Mandriva, Gentoo... by: 1) providing functionality to install files outside of egg dir/zip (for example one could install code to site-packages and data-files to /usr/share/ or somewhere else) 2) providing a "fakeroot" option to fool setup.py (for example I want setuptools to install the files in debian/tmp/usr/lib/pythonX.Y/site-packages and scripts in debian/tmp/usr/bin and everithing must work when it copied to /usr/{bin,lib}. 2) providing a paseable output with setup process data like: dependencies (meet and unmett), files installed, author, license, and so on All this will let people just "convert" the output of setuptools into an rpm specfile, debian dir, gentoo ebuild and so on. All that is needed is just a common metadata format that can be parsed by tools like easy-deb, easy-rmp easy-ebuild easy-(???). This way setuptools provides the information to distribution maintainers in a well documented, api independent way. Distribution maintainers can use this info to reorganize, repackage, invent new package formats, but without involving setuptools and python developers. About the pythonpath and pth files handlind: if a project A really wants to use project B eggs it can easily require them without the need for a pth file. If project C wants to use project B, but project C is not setuptools aware one could patch project C (just adding require() to it) instead of paching project B and project A removing egg support from them. Onother possible solution would be to have: python-A python-pypi-A python-B python-pypi-B python-C Where python-B) depends on python-pypi-A and just provides the pth file (handled like easy-deb does) python-pypi-B) contains the egg So project C will depend on python-B, which will install the pth. But if project B is required just by project A which is setuptools aware the dependency will be on python-pypi-B: the pth file will not be installed. Vincenzo ___________________________________ Yahoo! Mail: gratis 1GB per i messaggi e allegati da 10MB http://mail.yahoo.it From p.f.moore at gmail.com Wed Nov 23 17:53:40 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 23 Nov 2005 16:53:40 +0000 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051122201306.03a9a090@mail.telecommunity.com> References: <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <E1Eeisx-0008Py-00@server.0x1.org> <5.1.1.6.0.20051122201306.03a9a090@mail.telecommunity.com> Message-ID: <79990c6b0511230853i4cf7912cla54e1c1fea82b66d@mail.gmail.com> On 11/23/05, Phillip J. Eby <pje at telecommunity.com> wrote: > At 11:53 AM 11/23/2005 +1100, David Arnold wrote: > >-->"Phillip" == Phillip J Eby <pje at telecommunity.com> writes: > > > > Phillip> This is a major advantage over developers who do not do this, > > Phillip> not only in developer effectivness, but also because a > > Phillip> developer who depends exclusively on a specific packaging > > Phillip> system will not have the same effective reach for their > > Phillip> offering, or conversely will require a greater investment of > > Phillip> effort to support various packaging systems. > > > ><coming to this a little late> > > > >So, this would seem to imply that installation of eggs is similar to > >using PEAR or CPAN? > > Not at the level I think you mean. Apart from the .pth file, and any > scripts, each egg has its *own*, 100% encapsulated file or directory, for > example, which is quite different from at least CPAN. (I don't know > anything about PEAR.) Interesting. I would say that *something* in the easy_install/egg/setuptools area feels *exactly* like CPAN to me. Where I would like to use my system's standard packaging solution (I'm on Windows, so I mean the Windows Add/Remove Programs control panel applet here, as supported by bdist_wininst or more recently bdist_msi), I am being required to use a different mechanism. Like David, I don't like anything other than the "official" (ie, Windows installers in my case) mechanisms having access to the Python installation directory. If there was a way of building a Windows installer that installed packages in "egg" form, so I didn't have to use setup.py at all when installing, just double-click on the installer, that would suit me. This feels like what the Debian people want with their .deb format as well. The wording I'm using here is possibly not accurate - there seem to be a confusing mix of concepts and ideas going round, with no-one having a good understanding of all of them. My apologies - if there's a good glossary somewhere, which explains what to call things like * The executable I click on in Windows to install something * The .deb file that a Debian user downloads and installs * The set of files that end up in Python's site-packages - for a "normal" install (bdist_wininst, non-egg, whatever) - for an egg - that comprises the difference between the above two ( :-) ) * etc, etc then I'll be happy to restate my comments in those terms. (Assuming I understand the glossary - I've not managed to assimilate any of the previous attempts to establish clear terminology, unfortunately). Paul. From pje at telecommunity.com Wed Nov 23 18:18:07 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 23 Nov 2005 12:18:07 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <43843FAB.6040006@v.loewis.de> References: <5.1.1.6.0.20051122192931.01f84530@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <5.1.1.6.0.20051122192931.01f84530@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051123105311.01f95960@mail.telecommunity.com> At 11:08 AM 11/23/2005 +0100, Martin v. L?wis wrote: >As for terminology, you seem to suggest to use "distribution" where >Debian uses "package". So "Debian package" would become "Debian >distribution". No, I'm fine with "Debian package"; I was using "distribution" in the sense of "distutils distribution", such that you can have a "Debian package" of a "distutils distribution". The issue is that a "Python package" is not 1:1 with either a "Debian package" nor a "distutils distribution". An "egg" is a "distutils distribution" that may or may not contain "Python packages", but also contains "egg metadata" which is specific to the "distribution", not to any individual Python module or Python package contained within that distribution. >I'll try to use "project" in your sense and "package" in the >Python sense whenever I can. Great - and let's use "Debian package" to mean the thing that manages the installation of a project containing packages. :) >Phillip J. Eby wrote: >>An "egg" is a "distribution" of a "project" that is importable and can >>carry both standardized and individualized metadata that can be read by >>the pkg_resources module. There are various distribution *formats* in >>which an "egg" may be physically manifested, but the "egg" itself is a >>logical concept, not a physical one. It is therefore, as I said, "not >>merely a distribution format". Is that any clearer? > >Yes. When I said "an egg", I meant "a zipfile with a .egg extension, >or a directory with a .egg extension". In response to > ># [...] who will quite simply need eggs for many packages. ># If Debian doesn't provide them, the users will be forced to obtain ># them elsewhere. > >I meant > >"Debian should provide the distributions, but not as .egg files"; >it should provide the distribution as a deb file. So users are provided >with the project, but in a form that is not one of the three forms >an egg could have. I was referring to how the distribution is *installed*. You don't use things directly from a deb file, they have to be installed on the system. When you install an egg, you must use one of the three forms, or the system as a whole will not function. Eggs that depend on the egg will not be able to find it, nor use any plugins it contains. Eggs that define a plugin system of their own, will usually define self-plugins in their own metadata, as this is considered good style as well as being more convenient. Such eggs will not function without their *own* metadata installed. (Setuptools is an example of this, and I believe Trac 1.0 will be similar; some of the Paste projects may be using this already, too.) So, when I say it is a contradiction in terms to install an egg in a non-egg form, I mean that it is nonsensical to say that you have installed it, because it will be unusable (by other eggs), nonfunctional (by itself), or both. >>The "contradiction in terms" was that I took your meaning of "package" to >>be the same as my term "project" - i.e., a functional collection of >>Python resources. Projects that *are* eggs, can't be provided "but not >>as eggs". They *are* eggs, so not providing them as eggs means not >>providing them at all. > >I would expect that you can "unegg" a project. For projects that make use of eggs, you expect wrong. Try it with setuptools, and you will find that it is unable to even run its own tests, because the "test" command is registered via an entry point. Entry points are just one kind of project metadata that can be registered; other projects like Trac and SQLObject have their own kinds of metadata as well. None of this metadata is accessible without the EGG-INFO or .egg-info directory; removing it is like removing the JavaBean metadata or the deployment descriptors from Java jars, rendering the jar useless in many contexts, despite the fact that all the "code" remains. The only projects that can be "unegged", then, are ones that no egg project depends on, and which do not themselves depend on any eggs. The number of projects that are not depended on by other projects will be smaller and smaller over time, as will the number that do not depend on other eggs. Hm, that reminds me. One of the newer setuptools features for egg projects is automatic script generation using entry points. A developer can designate a function in some module as the implementation for a script, and a platform-appropriate script to invoke that function is automatically generated during installation. (In the case of Windows, an .exe is created alongside a .py or .pyw, on all other platforms it's a simple #!python script with no extension.) However, these generated scripts contain only a couple of lines that invoke the function via the project's entry point table - which is part of its egg metadata. So, if you remove the metadata, any scripts of this type that are installed by the project will fail to operate as well. Since there is no script in the original source, you would have to manually copy information from the project's setup.py in order to create scripts with equivalent functionality. In essence, trying to work around the absence of egg metadata is a bottomless pit, because over time there will be an ever-increasing amount of functionality in the field that is based on the use of metadata. >You can distribute the >project as a collection of Python modules, not as a collection of >Python resources. The Debian developer could (and I was suggesting >he should) just ignore the entire egg structure, and distribute >the code of the library only. Sure, just like you could delete the metadata files and directories from jar files, if you had some policy that required it. However, this wouldn't make any more sense than what you're proposing here. The projects would be unusable by other projects and/or nonfunctional in themselves, just like eggs. >>> If so, Debian should not distribute them. >> >>This is what I don't understand, as it has nothing to do whether or not >>is a distribution format, at least not that I can see. My statement was >>that eggs are not merely a distribution format; they are a logical >>concept that can be physically packaged in various ways, and if it's >>necessary to invent yet another physical layout, well, we can do that too. > >Yes, but this logical concept is in the way of Debian >packages/distributions (atleast if done naively by the Debian >developer). This is what started the entire discussion: Matthias >Urlichs complained that Bob Tanner included the egg structure >in the formencode Debian package/distribution. It's in the way of not changing the policy, sure. However, the policy's restriction in this case is not providing any functional benefit to anyone. Eggs, on the other hand, are a functional technical construct with actual usefulness in the field. To choose the policy over your users' needs in this case is like choosing to eat the restaurant's menu because the food in the pictures is more neatly arranged than the food on your plate. :) >The specific initial complaints where: >- you can't use it with a simple "import formencode", >- pydoc does not work on "eggs". These are both incorrect. First, if you install a .pth file (as easy_deb does, and any extra_path distutils distributions do), the first is moot. Second, pydoc works fine on all varieties of eggs, with a single exception: it does not work with zipped packages - the modules in the package can be documented, but not the parent package itself. This is a clear and obvious bug in pydoc (failure to update for PEP 302), and it is easily fixed. Nonetheless, it is trivially avoided by using either the unzipped or .egg-info installation formats. (Detail: PEP 302 specifically allows strings in a package __path__ to not be directories, and it also allows __path__ to be empty. pydoc assumes that it is non-empty and that its first element is a directory.) >I would add the complaint: >- it increases sys.path for no good reason. It is only true that it increases the length in the case of the two .egg forms, not the .egg-info form. The "no good reason" part is an interesting opinion, although in my view it is rather narrow-minded. Being able to support multi-version importing is a very good reason indeed, as is avoiding the need for a platform-specific package management tool in order to manage Python projects. Of course, you can safely ignore these points if you are looking at it strictly from the point of view of a package management tool that doesn't support installing multiple versions of things. You are blocked from these eminently "good reasons", however, by something that has nothing to do with eggs, so putting the "no good reason" on eggs is inappropriate. There are quite good reasons; you are simply blocked from taking advantage of them by the limitations of your chosen packaging tool. In any case, this complaint is moot in the case of the .egg-info form, since it does not affect the length of sys.path. >>Which would be the same as saying you wouldn't distribute, say, >>setuptools itself. Setuptools is an egg, and can't function except as an >>egg, because it is more than a Python package. Again, an "egg" is some >>specific release of a project and its introspectable metadata. > >I could rewrite setuptools to function as a regular Python package. >After a shallow inspection, there aren't many places where it really >needs the pkg_resources functionalities for itself - I could only >identify the part that locates cli.exe. As this is used on Windows >only, a Debian port of setuptools could simply ignore this code. Your "shallow inspection" is just that. Try this experiment. Delete the "setuptools.egg-info" directory, and then try to run "setup.py test" or "setup.py bdist_egg". After you figure out how to fix that, and install your setuptools in a "non-egg" form, I encourage you to try to build and install SQLObject and buildutils, or any other package that adds setup commands to setuptools, and see whether those commands work when the provider is lacking its metadata. For an encore, see if you can figure out how to get PasteDeploy configuration files to work - they're a format that allows users to deploy arbitrary WSGI applications as long as they're importable... and installed as an egg, with egg metadata. Eggs (and their metadata) exist because they provide functionality that is not practical to provide without them, and the scope of the deployed functionality that relies on the metadata is increasing rather quickly. >If "setup.py install" does other things, like editing an >existing file, it is not so easy anymore. I'm thinking that perhaps I should add an option like '--single-version-externally-managed' to the install command so that you can indicate that you are installing for the sake of an external package manager that will manage conflicts and uninstallation needs. This would then allow installation using the .egg-info form and no .pth files. The only issues remaining then are namespace packages and other inter-project overlaps, which of course you have to deal with now. (Example: the PyDispatcher and RuleDispatch projects both contain a 'dispatch' package, with unrelated contents.) >>>That is not true. Usability also suffers if sys.path becomes long. >> >>How? I don't understand this. > >People will often inspect sys.path to understand where Python >is looking for their code. As I pointed out, eggs give you much better information on this. For example: python -c "import pkg_resources; print pkg_resources.require('kid')" [kid 0.7a (c:\cygwin\home\pje\chandlerstuff\chandler\release\bin\lib\site-packages\kid-0.7a-py2.4.egg), elementtree 1.2.6 (c:\cygwin\home\pje\chandlerstuff\chandler\release\bin\lib\site-packages\elementtree-1.2.6-py2.4.egg)] I get the versions along with the paths, and the versions and paths of all dependencies. This information is not available in a cross-platform way without eggs. (And again, I mean the logical egg, not the .egg format; the above command would've listed any projects in .egg-info format as well as .egg files and directories.) >>What I would suggest here is having a namespace (e.g. pyegg2.4-whatever) >>for naming packages based on their PyPI names, so that there can be an >>automated relationship between setuptools dependencies and Debian ones. > >That would be a policy change (I think). Whether it would be agreeable, >I have no idea. I understand that, on both points. I was simply suggesting it would be useful, not trying to debate what the policy currently is. >>Anyway, I don't see any obvious reasons why this can't be an automated >>process, even for the system library dependencies. easy_deb even has a >>simple configuration file that can augment the setuptools-style >>dependencies with explicit Debian dependencies. > >Debian policy currently seems to require that the dependencies are >provided as plain text in a patch to the upstream sources(*). So the >idea certainly is that dependencies are managed by the developer, >not automatically. I'm only interested in what's helpful or useful to Debian developers and users, not what the current policy is. Policies tend to adapt to fit things that are useful, or else they become more of a drawback than a benefit. I mention these things because they may allow the process and policy to be improved, to everyone's benefit. If the policy doesn't change, however, then it should suffice to use .egg-info format to allow the distribution of egg projects as Debian packages conforming to the existing policy, assuming the policy does not prohibit including non-package directories in site-packages. The fact that .egg-info packaging may inconvenience packagers is a pain caused by the policy, however, not by eggs. I do intend, though, to update setuptools and easy_install to make using .egg-info form easier, and I will probably also fix it so that running e.g. bdist_rpm on a setuptools-based package will produce an .egg-info format egg wrapped in an RPM. I remain concerned about how such packages will work with namespace packages, since namespace packages mean that two different distributions may be supplying the same __init__.py files, and some package managers may not be able to deal with two system packages (e.g. Debian packages, RPMs, etc.) supplying the same file, even if it has identical contents in each system package. From pje at telecommunity.com Wed Nov 23 18:35:42 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 23 Nov 2005 12:35:42 -0500 Subject: [Distutils] .egg in Debian summary? In-Reply-To: <dm11id$pdi$1@sea.gmane.org> References: <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <200511220104.03160.tanner@real-time.com> Message-ID: <5.1.1.6.0.20051123121814.01f8c300@mail.telecommunity.com> At 12:17 AM 11/23/2005 -0600, Bob Tanner wrote: >Bob Tanner wrote: > > >> I don't think Debian should use the egg structure. It apparently relies > >> on building a long sys.path (even though through only a single .pth > >> file); > > > > I'm not sure of how .eggs are implemented, but I'm going to cross-post > > this info to the python-distutils mailing list. > >Read and re-read the complete thread regarding .eggs in Debian and I cannot >tell if any progress has been made. > >Still in the discussion/fact-finding stage? > >As "just a package maintainer" I was looking for the "options" to move >forward. Looking at the thread, I think these are the options (skipping the >pro's and con's for now): > >1. Do nothing, go with the status quo as documented in the Debian python >policy, which is no .egg's and unpackage everything into a sub-directory of >site-packages. > >2. Investigate easydeb <http://cheeseshop.python.org/pypi/easydeb/> > >3. Using Phillip's .egg-info solution ><http://permalink.gmane.org/gmane.comp.python.distutils.devel/2567> > >Any others? #1 is pointless if your goal is to get TurboGears packaged. As others have pointed out, future versions of TurboGears will be depending even more heavily on egg metadata than the current version does, and many of TurboGears' dependencies (e.g. SQLObject) already make use of egg metadata themselves, even if some only need the metadata for the sake of the projects that use them. #2 or something like it is the best long-term approach in my view, in that it supports a more automated packaging process, while avoiding inter-project name conflicts and properly supporting namespace packages. Also, under a 'pyegg' namespace, it could also be reasonable to install some projects such that 'require()' is necessary to use them. But there are a number of practical hurdles -- including policy decisions -- that keep this option from being a quick and easy choice. Practically speaking, I think #3 is the best compromise/transitional option, especially for packaging things like ElementTree, which is not packaged using setuptools, but which other projects need egg metadata for. This allows the existing Debian Python packages to be utilized by the egg system without duplication, since you need only add an empty .egg-info directory to the existing Debian packages. (Since the only metadata needed is the project name and version, which can be supplied from the .egg-info path.) I double-checked the implementation, and I'm going to have to actually change setuptools a bit to support this, because currently it doesn't parse the version from an .egg-info directory name. This is easy to fix, though, and I can put it out in 0.6a9, along with a --single-version-externally-managed option for the easy_install and install commands, and perhaps a bdist_rpm command using the same approach. These changes shouldn't take long, though. Until then, however, you would have to create an .egg-info/PKG-INFO file to be compatible with currently-released versions of setuptools. From pje at telecommunity.com Wed Nov 23 18:56:14 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 23 Nov 2005 12:56:14 -0500 Subject: [Distutils] .egg in Debian summary? In-Reply-To: <200511231639.59028.hawk78_it@yahoo.it> References: <dm11id$pdi$1@sea.gmane.org> <5bnlf-5nA-35@gated-at.bofh.it> <200511220104.03160.tanner@real-time.com> <dm11id$pdi$1@sea.gmane.org> Message-ID: <5.1.1.6.0.20051123123734.039adbb8@mail.telecommunity.com> At 04:39 PM 11/23/2005 +0100, Vincenzo Di Massa wrote: >I think setuptools could really help Debian developer as well as Fedora, >Mandriva, Gentoo... by: >1) providing functionality to install files outside of egg dir/zip (for >example one could install code to site-packages and data-files to /usr/share/ >or somewhere else) There will be some steps towards this in the 0.7 series, although initially that's going to be by providing user-executed post-installation tools, which would also be executable by packager-defined postinstallation steps. Since easy_install is not a package manager, however, I would prefer users be able to trust that running easy_install is only going to make a Python package and some scripts available, and not do anything to files elsewhere on their system. This position may evolve over time, since there are a lot of other features that (in my mind at least) need to come first before I can think deeply about the issue. (Related issues: header file installation, documentation, etc.) >2) providing a "fakeroot" option to fool setup.py (for example I want >setuptools to install the files in debian/tmp/usr/lib/pythonX.Y/site-packages >and scripts in debian/tmp/usr/bin and everithing must work when it copied >to /usr/{bin,lib}. There is in fact such an option now; as far as I know the --root option to "setup.py install" should do it, and I am pretty sure it's used by bdist_rpm. >2) providing a paseable output with setup process data like: dependencies >(meet and unmett), files installed, author, license, and so on Perhaps you could provide a sample of what you have in mind? A patch would be even better, although I would be using it mainly to understand what you'd like to have and not necessarily as a basis for implementing it. >All this will let people just "convert" the output of setuptools into an rpm >specfile, debian dir, gentoo ebuild and so on. >All that is needed is just a common metadata format that can be parsed by >tools like easy-deb, easy-rmp easy-ebuild easy-(???). Note that easy_install and install have --record options that write the names of all installed files or directories to a file for you. The remaining metadata you're asking about can already be queried via the pkg_resources API, since it's part of the resulting eggs. 'someEgg.get_metadata("PKG-INFO")' will get you the PKG-INFO file with all the standard distutils metadata, for example. The 'requires()' method gives information about dependencies. >This way setuptools provides the information to distribution maintainers in a >well documented, api independent way. Well, if you want API independence, you can directly look at the .egg-info or EGG-INFO directory contents, as the data is in plain-text files like requires.txt and PKG-INFO. PKG-INFO is a PEP-defined format, and the requires.txt format has been stable for maybe 7 months now. I don't anticipate any changes to it. > Distribution maintainers can use this >info to reorganize, repackage, invent new package formats, but without >involving setuptools and python developers. Fair enough, although you should keep in mind that there are only three valid installation layouts for egg-specific projects: .egg zipfile, .egg directory, and "old school" installation with an .egg-info directory installed alongside. Anything else, and it would involve changing the egg runtime to work with it. >About the pythonpath and pth files handlind: >if a project A really wants to use project B eggs it can easily require them >without the need for a pth file. >If project C wants to use project B, but project C is not setuptools aware >one >could patch project C (just adding require() to it) instead of paching >project B and project A removing egg support from them. Note that if C is an application rather than a library, one can add the requirements to setup.py and use easy_install to install it, which will create script wrappers that perform the necessary require() operations. So, you don't have to patch C very deeply in this case. In addition, the scripts will require() C itself, so you don't need to install C with a .pth unless it is a library that is needed by something else that is not packaged using this system. Basically, the whole thing is designed so that as long as a project is setuptools-based, it doesn't need explicit require() calls, as these are handled by setuptools. Only if you want a project to be *always* on sys.path do you need a .pth file. >Onother possible solution would be to have: >python-A >python-pypi-A >python-B >python-pypi-B >python-C > >Where >python-B) depends on python-pypi-A and just provides the pth file (handled >like easy-deb does) >python-pypi-B) contains the egg > >So project C will depend on python-B, which will install the pth. But if >project B is required just by project A which is setuptools aware the >dependency will be on python-pypi-B: the pth file will not be installed. So, you're saying that python-foo would install 'foo' such that it is always on sys.path, but python-pypi-foo simply installs 'foo' in such a way that is discoverable by the egg runtime. That's an interesting idea, I'll have to think a while about the consequences of that before I comment on it further. From pje at telecommunity.com Wed Nov 23 19:05:55 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 23 Nov 2005 13:05:55 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <79990c6b0511230853i4cf7912cla54e1c1fea82b66d@mail.gmail.co m> References: <5.1.1.6.0.20051122201306.03a9a090@mail.telecommunity.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <E1Eeisx-0008Py-00@server.0x1.org> <5.1.1.6.0.20051122201306.03a9a090@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051123125622.039b8008@mail.telecommunity.com> At 04:53 PM 11/23/2005 +0000, Paul Moore wrote: >Interesting. I would say that *something* in the >easy_install/egg/setuptools area feels *exactly* like CPAN to me. >Where I would like to use my system's standard packaging solution (I'm >on Windows, so I mean the Windows Add/Remove Programs control panel >applet here, as supported by bdist_wininst or more recently >bdist_msi), I am being required to use a different mechanism. Right, but this is an issue on a different metalevel. Nobody is proposing that Debian users install eggs directly into their /usr/lib/python2.x/site-packages directory. >Like David, I don't like anything other than the "official" (ie, >Windows installers in my case) mechanisms having access to the Python >installation directory. > >If there was a way of building a Windows installer that installed >packages in "egg" form, so I didn't have to use setup.py at all when >installing, just double-click on the installer, that would suit me. >This feels like what the Debian people want with their .deb format as >well. Right - this would hopefully be covered at some point using an .msi or .exe wrapper for eggs on Windows. So far, though, you're the only person I've heard from about wanting this on Windows, versus the number of people who want it for e.g. Debian. So I'm still considering Debian wrapper support as being more urgent than .msi wrapper support. My point to David was simply that egg packaging in the .egg form is more akin to Stow than to CPAN, so most of the flaws of CPAN are not applicable to them. >The wording I'm using here is possibly not accurate - there seem to be >a confusing mix of concepts and ideas going round, with no-one having >a good understanding of all of them. My apologies - if there's a good >glossary somewhere, which explains what to call things like > > * The executable I click on in Windows to install something > * The .deb file that a Debian user downloads and installs I would call these "system packages", to distinguish them from Python packages. You (and others) would like to ensure that any project you install is wrapped in a system package. > * The set of files that end up in Python's site-packages A "project distribution", or just a "project" (which is a less accurate term but more convenient). > - for a "normal" install (bdist_wininst, non-egg, whatever) This has never seemed to me to need a special term unto itself, so I don't have one. > - for an egg An egg. :) > - that comprises the difference between the above two ( :-) ) Egg metadata. An egg is just an installed project distribution that includes egg metadata, in one of the three possible layouts for putting that metadata alongside the rest of the project contents. From smurf at smurf.noris.de Wed Nov 23 20:12:08 2005 From: smurf at smurf.noris.de (Matthias Urlichs) Date: Wed, 23 Nov 2005 20:12:08 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051123105311.01f95960@mail.telecommunity.com> References: <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <5.1.1.6.0.20051122192931.01f84530@mail.telecommunity.com> <5.1.1.6.0.20051123105311.01f95960@mail.telecommunity.com> Message-ID: <20051123191208.GK20960@kiste.smurf.noris.de> Hi, Phillip J. Eby: > I'm thinking that perhaps I should add an option like > '--single-version-externally-managed' to the install command so that you > can indicate that you are installing for the sake of an external package > manager that will manage conflicts and uninstallation needs. This would > then allow installation using the .egg-info form and no .pth files. > You might shorten that option a bit. ;-) I agree that this would be a good option to have. > >People will often inspect sys.path to understand where Python > >is looking for their code. > > As I pointed out, eggs give you much better information on this. The .egg metadata does. That, as you say, is distinct from the idea of packaging the .egg as a zip file. Most likely, one that includes .pyc files which were byte-compiled with different file paths; That causes no problems whatsoever ... until you get obscure ideas like trying to step through the code with pdb, or opening it in your editor to insert an assertion or a printf, trying to figure out why your code breaks. :-/ > >Debian policy currently seems to require that the dependencies are > >provided as plain text in a patch to the upstream sources(*). So the > >idea certainly is that dependencies are managed by the developer, > >not automatically. > > I'm only interested in what's helpful or useful to Debian developers and > users, not what the current policy is. Policies tend to adapt to fit > things that are useful, or else they become more of a drawback than a > benefit. I mention these things because they may allow the process and > policy to be improved, to everyone's benefit. That's not exactly negotiable. Debian has a packaging format which resolves generic installation dependencies on its own. Therefore it cannot depend on Python-specific .egg metadata. Therefore we need a way to translate .egg metadata to Debian metadata. > I remain concerned about how such packages will work with namespace > packages, since namespace packages mean that two different distributions > may be supplying the same __init__.py files, and some package managers may > not be able to deal with two system packages (e.g. Debian packages, RPMs, > etc.) supplying the same file, even if it has identical contents in each > system package. > Debian packaging has a method to explicitly rename a different package's file if it conflicts with yours ("dpkg-divert"; it does _not_ depend on which package gets installed first). IMHO that's actually superior randomly executing only one of these files, since you are aware that there is a conflict (the second package simply doesn't install if you don't fix it), and thus can handle it intelligently. -- Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf at smurf.noris.de Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de - - A father doesn't destroy his children. -- Lt. Carolyn Palamas, "Who Mourns for Adonais?", stardate 3468.1. From pje at telecommunity.com Wed Nov 23 21:29:17 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 23 Nov 2005 15:29:17 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <20051123191208.GK20960@kiste.smurf.noris.de> References: <5.1.1.6.0.20051123105311.01f95960@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <5.1.1.6.0.20051122192931.01f84530@mail.telecommunity.com> <5.1.1.6.0.20051123105311.01f95960@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051123144125.01f9a9f8@mail.telecommunity.com> At 08:12 PM 11/23/2005 +0100, Matthias Urlichs wrote: >Hi, > >Phillip J. Eby: > > I'm thinking that perhaps I should add an option like > > '--single-version-externally-managed' to the install command so that you > > can indicate that you are installing for the sake of an external package > > manager that will manage conflicts and uninstallation needs. This would > > then allow installation using the .egg-info form and no .pth files. > > >You might shorten that option a bit. ;-) I agree that this would be a >good option to have. I try to use very long names for options that can have damaging effects if used indiscriminately. A project that's installed the "old-fashioned way" (which is what this does, apart from adding .egg-info) is hard to uninstall and may overwrite other projects' files. So, it is only safe to use if the files are being managed by some external package manager, and it further only works for a single installed version at a time. So the name is intended to advertise these facts, and to discourage people who are just reading the option list from trying it out to see what it does. :) > > >People will often inspect sys.path to understand where Python > > >is looking for their code. > > > > As I pointed out, eggs give you much better information on this. > >The .egg metadata does. That, as you say, is distinct from the idea of >packaging the .egg as a zip file. Most likely, one that includes .pyc >files which were byte-compiled with different file paths; That causes no >problems whatsoever ... until you get obscure ideas like trying to step >through the code with pdb, or opening it in your editor to insert an >assertion or a printf, trying to figure out why your code breaks. :-/ This is actually what the .egg-info mode was designed for. That is, doing development of the project. A setuptools-based project can run "setup.py develop" to add the project's source directory to sys.path, after generating an .egg-info directory in the project source if necessary. This allows you to do all your development right in your source checkout, and of course all the file paths are just fine, and the egg metadata is available at runtime. You can then deploy the project as an .egg file or directory. (Also, for the .egg directory format, note that easy_install recompiles the .pyc/.pyo files so their paths *do* point to the .egg contents instead of the original build paths. The issues with zipfiles and precompiled .pyc files are orthogonal to anything about setuptools, eggs, etc.; they will bite you in today's Python no matter what's in the zipfile or who precompiled the .pyc files. I do have some ideas for fixing both of these problems in future versions of Python, but they're rather off-topic for all the lists we are currently talking on.) >That's not exactly negotiable. Debian has a packaging format which >resolves generic installation dependencies on its own. Therefore it >cannot depend on Python-specific .egg metadata. Therefore we need a way >to translate .egg metadata to Debian metadata. Yes, that's precisely what I was suggesting would be helpful. As Vincenzo already mentioned, the egg metadata is a good starting point for defining the Debian metadata. I'm obviously not proposing changing Debian's metadata system. Well, maybe it wasn't *obvious* that I wasn't proposing that, but in any case I'm not. :) > > I remain concerned about how such packages will work with namespace > > packages, since namespace packages mean that two different distributions > > may be supplying the same __init__.py files, and some package managers may > > not be able to deal with two system packages (e.g. Debian packages, RPMs, > > etc.) supplying the same file, even if it has identical contents in each > > system package. > > >Debian packaging has a method to explicitly rename a different package's >file if it conflicts with yours ("dpkg-divert"; it does _not_ depend on >which package gets installed first). IMHO that's actually superior >randomly executing only one of these files, since you are aware that >there is a conflict (the second package simply doesn't install if you >don't fix it), and thus can handle it intelligently. The two kinds of possible conflicts are namespace packages, and project-level resources. A namespace package is more like a Java package than a traditional Python package. A Java package can be split across multiple directories or jar files; it doesn't have to be all in one place. Thus you can have lots of jars with org.apache.* classes in them. Python, however, requires packages to have an __init__.py file, and by default the entire package is assumed to be in the directory containing the __init__.py file. However, as of Python 2.3, the 'pkgutil' module was introduced in the Python standard library which allowed you to create a Java-style "namespace package", automatically combining package directories found on different parts of sys.path. So, if in one sys.path directory you had a 'zope.interface' package, and in another you had a 'zope.publisher' package, these would be combined, instead of the first one being treated as if it were all of 'zope.*', and the second being completely ignored. However, *each* of the subpackages needs its own zope/__init__.py file for this to work. So, the issue here is that if you install two projects that contain zope.* packages into the *same* directory (e.g. site-packages), then there will be two different zope/__init__.py files installed at the same location, even though they will have the same content (a short snippet of code to activate the namespace mechanism via the pkgutil module or via setuptools' pkg_resources module). To date, there are only a small number of these namespace packages in existence, but over time they will represent a fairly large number of *projects*. As I go through the breakup of the PEAK meta-project into separate components, I expect to have a dozen or so projects contributing to the peak.* and peak.util.* namespace packages. Ian Bicking's Paste meta-project has a paste.* namespace package spread out in two or three subprojects so far. There has been some off-and-on discussion about whether Zope 3 will move to eggs instead of their own zpkg tool (which has issues on Windows and Mac OS that eggs do not), and in that case they will likely have a couple dozen components in zope.* and zope.app.*. So, for the long-term solution of wrapping Python projects in Debian packages, the namespace issue needs to be addressed, because renaming each project's zope/__init__.py or whatever isn't going to work very well. There has to be one __init__.py file, or else such projects need to be installed in their own .egg directories or zipfiles to avoid collisions. The second collision issue with --single-version-externally-managed is top-level resource collisions. Some existing projects that are not egg-based manipulate their install_data operation in such a way that they create files or directories in site-packages directly, rather than inside their own package data structures. Setuptools neither encourages nor discourages this, because it doesn't cause any problems for any egg layout except the .egg-info one -- and the .egg-info one was originally designed to support development, not deployment. In the development scenario, any such files are isolated to the source tree, and for deployment the .egg file or directory keeps each projects' contents completely isolated. So, what I'm saying is that putting all projects in the same directory (as all "traditional" Python installations do) has some inherent limitations with respect to namespace packages and top-level resources, and these limitations are orthogonal to the question of egg metadata. The .egg formats were created to solve these problems (including clean upgrades, multi-version support, and uninstallation in scenarios where a package manager isn't usable), and so the other features that they enable will be increasingly popular as well. In other words, as people make more use of PyPI (because they now really *can*), more people will put things on PyPI, and the probability of package name conflicts will increase more rapidly. The natural response will be a desire to claim uber-project or organizational names (like paste.*, peak.*, zope.*, etc.) putting individual projects under sub-package names. (For example, someone has already argued that I should move RuleDispatch's 'dispatch' package to 'peak.dispatch' rather than keeping the top-level 'dispatch' name all to myself.) So, I'm just saying that using the --single-version-externally-managed approach requires that a package manager like Debian grow a way to handle these namespace packages safely and sanely. One possibility is to create dummy packages that contain only the __init__.py file for that namespace, and then have the real packages all depend on the dummy package, while omitting the __init__.py. So, perhaps each project containing a peak.util.* subpackage would depend on a 'python2.4-peak.util-namespace' package, which in turn would depend on a 'python2.4-peak-namespace' package. It's rather ugly, to say the least, but it would work as long as upstream developers never put anything in namespace __init__.py files except for the pkg_resources.declare_namespace() call. (By the way, since part of an egg's metadata lists what namespace packages the project contains code or data for, the generation of these dependencies can be automated as part of the egg-to-deb conversion process.) Or, of course, the .egg directory approach can also be used to bypass all collision issues, but this brings sys.path and .pth files back into the discussion. On the other hand, it can possibly be assumed that anything in a namespace package can be used only after a require() (either implicit or explicit), so maybe the .pth can be dropped for projects with namespace packages. These are possibilities worth considering, since they avoid the ugliness of creating dummy packages just to hold namespace __init__.py files. From martin at v.loewis.de Wed Nov 23 21:38:24 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Nov 2005 21:38:24 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <79990c6b0511230725x1d48544fjc227ba75ce73e1de@mail.gmail.com> References: <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> <43838D58.7000801@egenix.com> <79990c6b0511230725x1d48544fjc227ba75ce73e1de@mail.gmail.com> Message-ID: <4384D340.9080506@v.loewis.de> Paul Moore wrote: > It's a long time ago, and back prior to PEP 302, but I believe that > the original version of the zipimport PEP (PEP 273 by Jim Ahlstrom) > included some timings based on the original patch. While this did show > slowdowns with compressed zip files, it showed distinct speedups with > *un*compressed zip files. IIRC, this was explained as because a > zipfile directory scan is faster than multiple filesystem stats. This all is convincing *if* the files you are looking for actually are located in the zipfile - then reading from the zipfile might indeed be faster than directly reading from disk. If, however, the files are *not* in the zipfile, it's not so clear anymore what the consequences are. On the downside, you have the need to read the entire zip table-of-content (TOC), and you have the need to perform a dictionary lookup. On the up-side, you avoid a system call. Regards, Martin From martin at v.loewis.de Wed Nov 23 22:00:47 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Nov 2005 22:00:47 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051123105311.01f95960@mail.telecommunity.com> References: <5.1.1.6.0.20051122192931.01f84530@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <5.1.1.6.0.20051122192931.01f84530@mail.telecommunity.com> <5.1.1.6.0.20051123105311.01f95960@mail.telecommunity.com> Message-ID: <4384D87F.5080307@v.loewis.de> Phillip J. Eby wrote: > I was referring to how the distribution is *installed*. You don't use > things directly from a deb file, they have to be installed on the > system. When you install an egg, you must use one of the three forms, > or the system as a whole will not function. That depends on whether the "system" (pkg_resources, I assume) is used at all. If the project is just a Python library, you can install it as a Python package in site-python, not as an egg. > Eggs that depend on the egg > will not be able to find it, nor use any plugins it contains. Not sure what an egg plugin is, so I cannot comment on that. As for other eggs finding the one: In Debian, there normally shouldn't be any need to, since there will be also a Debian package providing the other project, and then a plain "import" will be sufficient to find the Python package. Of course, any usage of the pkg_resource API would break. One way to deal with that is to encourage upstream authors to have a fallback mode where they can work without pkg_resource; another is to provide a fallback implementation of pkg_resource. > So, when I say it is a contradiction in terms to install an egg in a > non-egg form, I mean that it is nonsensical to say that you have > installed it, because it will be unusable (by other eggs), nonfunctional > (by itself), or both. That makes me not like the egg infrastructure: too many subtle dependencies, and you are too much forced into using the structures that the setuptools authors came up with. Of course, the pragmatic view is just to bite the bitter pill (is this the idiom?) and find some strategy that makes pkg_resource work, without any of the drawbacks of setuptools. >> I would expect that you can "unegg" a project. > > > For projects that make use of eggs, you expect wrong. Try it with > setuptools, and you will find that it is unable to even run its own > tests, because the "test" command is registered via an entry point. I would have to rewrite the code, of course. I do all registration that needs to be done in __init__.py > Entry points are just one kind of project metadata that can be > registered; other projects like Trac and SQLObject have their own kinds > of metadata as well. None of this metadata is accessible without the > EGG-INFO or .egg-info directory; removing it is like removing the > JavaBean metadata or the deployment descriptors from Java jars, > rendering the jar useless in many contexts, despite the fact that all > the "code" remains. Sure, *just* removing it would be wrong. I have to replace it with Python code. > The only projects that can be "unegged", then, are ones that no egg > project depends on, and which do not themselves depend on any eggs. The > number of projects that are not depended on by other projects will be > smaller and smaller over time, as will the number that do not depend on > other eggs. Define "depends on". If this is "imports", I don't see a problem with unegging the package. If the dependent package is installed, the import statement will just succeed right away. > In essence, trying to work around the absence of egg metadata is a > bottomless pit, because over time there will be an ever-increasing > amount of functionality in the field that is based on the use of metadata. That is really sad. >> I would add the complaint: >> - it increases sys.path for no good reason. > > > It is only true that it increases the length in the case of the two .egg > forms, not the .egg-info form. Ok, then I think this is what Debian should use. > The "no good reason" part is an interesting opinion, although in my view > it is rather narrow-minded. Being able to support multi-version > importing is a very good reason indeed, as is avoiding the need for a > platform-specific package management tool in order to manage Python > projects. I don't see why multi-version support necessarily requires to increase sys.path. In the case of eggs, version dependencies are expressed explicitly in the code (through require() calls), so that essentially replace the standard Python import search algorithm. Because of that, you could have a default version inside site-packages, and additional versions elsewhere, only found when require() is called. Regards, Martin From martin at v.loewis.de Wed Nov 23 22:07:42 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Nov 2005 22:07:42 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051123144125.01f9a9f8@mail.telecommunity.com> References: <5.1.1.6.0.20051123105311.01f95960@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <5.1.1.6.0.20051122192931.01f84530@mail.telecommunity.com> <5.1.1.6.0.20051123105311.01f95960@mail.telecommunity.com> <5.1.1.6.0.20051123144125.01f9a9f8@mail.telecommunity.com> Message-ID: <4384DA1E.4050702@v.loewis.de> Phillip J. Eby wrote: > I try to use very long names for options that can have damaging effects > if used indiscriminately. A project that's installed the "old-fashioned > way" (which is what this does, apart from adding .egg-info) is hard to > uninstall and may overwrite other projects' files. So, it is only safe > to use if the files are being managed by some external package manager, > and it further only works for a single installed version at a time. So > the name is intended to advertise these facts, and to discourage people > who are just reading the option list from trying it out to see what it > does. :) And that is indeed a fine principle. The option would only occur inside debian/rules (which is the Makefile to build the deb file), so users would never need to type it manually (except for the DD creating debian/rules). My request would be that the option is stable in its meaning, e.g. if there is a need to extend/change its semantics, a different option is introduced (unless it is certain that existing Debian packages would continue to build correctly under the new meaning). Regards, Martin From pje at telecommunity.com Thu Nov 24 00:06:24 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 23 Nov 2005 18:06:24 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <4384D87F.5080307@v.loewis.de> References: <5.1.1.6.0.20051123105311.01f95960@mail.telecommunity.com> <5.1.1.6.0.20051122192931.01f84530@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <5.1.1.6.0.20051122192931.01f84530@mail.telecommunity.com> <5.1.1.6.0.20051123105311.01f95960@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051123161333.01f9b7e8@mail.telecommunity.com> At 10:00 PM 11/23/2005 +0100, Martin v. L?wis wrote: >Phillip J. Eby wrote: >>I was referring to how the distribution is *installed*. You don't use >>things directly from a deb file, they have to be installed on the >>system. When you install an egg, you must use one of the three forms, or >>the system as a whole will not function. > >That depends on whether the "system" (pkg_resources, I assume) is used >at all. If the project is just a Python library, you can install it >as a Python package in site-python, not as an egg. > >>Eggs that depend on the egg will not be able to find it, nor use any >>plugins it contains. > >Not sure what an egg plugin is, so I cannot comment on that. >As for other eggs finding the one: In Debian, there normally shouldn't >be any need to, since there will be also a Debian package providing >the other project, and then a plain "import" will be sufficient to >find the Python package. No, it won't, because... oh never mind. I'll explain again below. What you seem to keep missing, though, is that eggs and their metadata are a *feature*, not a bug. The rapid uptake of setuptools by developers trying to build more powerful frameworks and platforms for Python is sufficient evidence that they provide useful features that Python developers desire to have, precisely because they can be used to wrap non-setuptools based pacakges without code changes and without reinventing wheels - either the wheels provided by setuptools, or the wheels provided by other projects when wrapped by setuptools. Removing the metadata gives them neither option. >Of course, any usage of the pkg_resource API would break. One way >to deal with that is to encourage upstream authors to have a fallback >mode where they can work without pkg_resource; another is to provide >a fallback implementation of pkg_resource. Yes, and while we're at it, let's encourage developers to have fallbacks so their code can run on Python 1.5.2. Heck, why stop there? Anything that requires features introduced after Python 1.0 would obviously only be an impossible attempt to improve upon perfection. For that matter, let's not have any dependencies on other packages at all! Clearly it would be better for everybody to write their own modules and not use something written by some random person on the Internet. :) All joking aside, one of the central points of having setuptools in the first place is that it allows people to avoid duplicating code. Code like, say, the pkg_resources module. This is another example of what I'm calling a contradiction in terms, because I keep saying that the purpose of all this is to allow X, and then you propose, "well, do it without X", and I say, "but X is the whole point! Doing it without X isn't actually doing 'it' because X is what 'it' is." And then you say, "Ah, but what if you do it with Y?", and so we go round the loop again. >>So, when I say it is a contradiction in terms to install an egg in a >>non-egg form, I mean that it is nonsensical to say that you have >>installed it, because it will be unusable (by other eggs), nonfunctional >>(by itself), or both. > >That makes me not like the egg infrastructure: too many subtle >dependencies, and you are too much forced into using the structures >that the setuptools authors came up with. [boggle] Um, what is Debian but a collection of subtle dependencies forced into the structures that its authors came up with? Perhaps your point here is just too subtle for me. :) >Of course, the pragmatic view is just to bite the bitter pill (is >this the idiom?) The idioms are to "bite the bullet", or "swallow the bitter pill". The former is from the one-time medical practice of biting on a bullet to avoid screaming during procedures performed without anesthetic. The latter of course is also a medical idiom, in the sense that a medicine may be bitter but nonetheless good for one's health. :) In any case, both idioms imply a desire to get an unpleasant but beneficial task over with, so mixing them is quite understandable, albeit odd-sounding. :) > and find some strategy that makes pkg_resource >work, without any of the drawbacks of setuptools. Just as I'm trying to help find a way to make Debian be able to provide something useful for setuptools-based projects, despite the drawbacks of the current Debian arrangements. ;) The degree of negativity from the Debian side at the outset of this conversation (virtually all of it from you) has not been conducive to making this happen. As a simple matter of practicality, I can't afford to leave your comments unanswered, not because I feel any need to convince you personally of anything, but because I don't want to leave anyone else with the impression that your portrayal of these so-called "drawbacks" is a fair one. Otherwise, I would have just ignored your comments and focused on working with the people who seem more interested in finding solutions than finding ways to declare a non-existence of the problem. As it is, I feel forced to spend time replying to your comments point-by-point, that I could otherwise spend on actually helping to resolve the issues. If I were to adopt your tone, I would be calling Debian a fragile and broken system that is unable to deal well with simple matters like editing a file upon installation, or having multiple versions of a package installed at the same time. Sure, the limitation might exist, but is it fair to call Debian fragile or broken because of it? Not a bit! I've therefore been very careful to describe any such tradeoffs that Debian makes in neutral terms rather than categorically pejorative ones. I would prefer if you would extend me the same courtesy of not describing every design tradeoff I make as being a "non-standard", "drawback", "for no good reason". (Even though I have referred to the existing Debian policy as "outdated", I meant it only in the sense that it does not deal explicitly with the issue of eggs, which is a neutral statement, not a judgment of the condition. It would be stupid and unreasonable for me to imply that Debian's policy must be updated to include eggs, as setuptools is alpha software that is very much still in development. Which is why it isn't me who approached the Debian developers about this, as opposed to the other way around. However, once contacted about the matter, I'm certainly going to point out that ignoring the existence of eggs and their likely rapid increase in popularity (e.g. TurboGears claims 40,000 eggs served) is also unreasonable.) >>>I would expect that you can "unegg" a project. >> >>For projects that make use of eggs, you expect wrong. Try it with >>setuptools, and you will find that it is unable to even run its own >>tests, because the "test" command is registered via an entry point. > >I would have to rewrite the code, of course. I do all registration >that needs to be done in __init__.py That registration can't be done until a package is imported, so even if you did the significant patching this would require, your effort will fail as soon as you bring extensions into the picture, such as buildutils or SQLObject, as I already explained. >>Entry points are just one kind of project metadata that can be >>registered; other projects like Trac and SQLObject have their own kinds >>of metadata as well. None of this metadata is accessible without the >>EGG-INFO or .egg-info directory; removing it is like removing the >>JavaBean metadata or the deployment descriptors from Java jars, rendering >>the jar useless in many contexts, despite the fact that all the "code" remains. > >Sure, *just* removing it would be wrong. I have to replace it with >Python code. Which will *never be imported* and will therefore never execute, because the project it needs to *plug into* won't know it exists. A project "foo" that extends the functionality of project "bar" can't be statically known about by project "bar". The dependency is that foo requires bar, but bar must be able to "discover" at runtime that foo exists. The idea is that project "bar" can be extensible by other projects, by providing entry point groups that other projects can add themselves to (via published metadata). These other projects do not need to be imported; they are found by their metadata, which describes them as offering entry points in the "bar"-supplied entry point groups. Thus, new projects like "foo" can hook in to the infrastructure provided by "bar". For example, SQLObject and buildutils are project "foo" with respect to setuptools; setuptools doesn't depend on them, or know about their existence a priori. But their mere presence on sys.path (or more precisely, the presence of egg metadata in well-defined locations relative to sys.path entries) is enough to allow setuptools to find them. The "Trac" web-based project management application is an example of project "bar" - it offers a sophisticated plugin capability to allow people to customize its database, web interface, and so on. The mere existence of a plugin project on sys.path, or its presence in the Trac plugins directory, is sufficient to allow that project's code to be *dynamically imported* on an as-needed basis whenever a particular notification hook is invoked. These things are not practical without some kind of metadata. You cannot simply replace the metadata with code, because the code has to be imported, which means that you would have to import every module and package on sys.path in order to be sure you found all the metadata. >>The only projects that can be "unegged", then, are ones that no egg >>project depends on, and which do not themselves depend on any eggs. The >>number of projects that are not depended on by other projects will be >>smaller and smaller over time, as will the number that do not depend on >>other eggs. > >Define "depends on". If this is "imports", I don't see a problem with >unegging the package. As you said, a false proposition implies any conclusion. It is you who is assuming "depends on" means "imports". Plugins are the simplest example of a "depends on" that goes beyond importing. >>In essence, trying to work around the absence of egg metadata is a >>bottomless pit, because over time there will be an ever-increasing amount >>of functionality in the field that is based on the use of metadata. > >That is really sad. Yes, we should all go back to C like real programmers. :) No, wait, then we would have to deal with all those messy .h files. But who needs interfaces and metadata like argument types? We should just put the memory addresses of the functions directly in our code, because then there will be fewer processing steps and we won't have all those .h files messing up the place. Plus, that whole concept of a "linker" seems awfully fragile to me. Who knows what address it might put my code at? Besides, I don't need a linker if I only use the code that I write, and those people who use other people's code are obviously just too lazy to write their own or even copy and paste it. Can you imagine? :) >>>I would add the complaint: >>>- it increases sys.path for no good reason. >> >>It is only true that it increases the length in the case of the two .egg >>forms, not the .egg-info form. > >Ok, then I think this is what Debian should use. Great! At least we are making some progress here. For non-setuptools packages (like ElementTree), it will suffice to place an empty 'projectname-version.egg-info' file or directory in site-packages alongside the installed package. I will modify setuptools 0.6a9 to parse the version from the file or directory name, and to accept a file instead of a directory. (Currently, it requires a PKG-INFO file inside an .egg-info directory and parses the Version: header from PKG-INFO.) If Debian adds this metadata marker for its non-setuptools Python packages, then the Python packages will be "eggs" in the sense that other eggs will be able to discover them via the pkg_resources API, and thus TurboGears users will be able to use the Debian-supplied versions of ElementTree and the like. Note, however, that the 'projectname-version' string has some precise escaping rules; the distutils are quite inconsistent about their processing of names and escaping, so I had to devise more specific rules for setuptools, because setuptools has to actually *use* the project names and versions, and parse them out of filenames: 1. The project name in a file or directory name is the setup(name=...) argument, with all runs of one or more non-alphanumeric characters replaced with '_'. (Note that this means there is never more than one '_' in a row in the filename.) So a project like "FooBar Tools" or "FooBar-Tools" would become "FooBar_Tools" in the filename. 2. The rules for the version are the same as for the name, *except* that the '.' character is allowed to remain unescaped, and spaces are converted to '.' before compacting non-alphanumeric runs. So, version '1.2 rc5' becomes '1.2.rc5', while '1.2-pl5' becomes '1.2_pl5'. >>The "no good reason" part is an interesting opinion, although in my view >>it is rather narrow-minded. Being able to support multi-version >>importing is a very good reason indeed, as is avoiding the need for a >>platform-specific package management tool in order to manage Python projects. > >I don't see why multi-version support necessarily requires to >increase sys.path. In the case of eggs, version dependencies are >expressed explicitly in the code (through require() calls), Actually, they're expressed in the egg metadata, and the wrappers on a project's scripts execute the require() calls, so that the code doesn't have to contain explicit require() calls except for more-dynamic situations, such as plugins and "optional extra features" that require additional projects to be present. > so >that essentially replace the standard Python import search algorithm. >Because of that, you could have a default version inside site-packages, >and additional versions elsewhere, only found when require() is >called. That's correct, and setuptools actually supports that scenario, but it doesn't currently provide tools for creating that arrangement on disk, since the "default version" you propose would be hard to manage without an external packaging tool, like Debian. (The proposed addition for 0.6a9 would be to make it possible to install such a thing, for use with external packaging tools.) Note that setuptools is in release 0.6a8 at the moment - it is obviously not a polished product, but it provides enough functionality to be quite useful to many Python developers. To this point, directly working on integration with external packaging tools has not been a focus, although I always have given top priority to responding to questions and requests from people working on integration with those tools (e.g. the volunteers who worked on easy_deb and the Gentoo stuff). I can't reasonably learn the technical details of every packaging system, so it is best to let volunteers familiar with individual packaging systems tell me what they need in order to effectively wrap the system. Up until now, my interactions with such volunteers have been most pleasant and positive. To my knowledge, it's not usual for packaging system developers to spew FUD at a project and look for ways to exclude or break the work of developers who've chosen to use it. I'm therefore more than a little surprised by some of the attitude I've received. I hope, though, that we can get past that soon, if only because it means I'll have more time to work on implementing and documenting whatever the resolution is. ;) From pje at telecommunity.com Thu Nov 24 00:12:11 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 23 Nov 2005 18:12:11 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <4384DA1E.4050702@v.loewis.de> References: <5.1.1.6.0.20051123144125.01f9a9f8@mail.telecommunity.com> <5.1.1.6.0.20051123105311.01f95960@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <5.1.1.6.0.20051122192931.01f84530@mail.telecommunity.com> <5.1.1.6.0.20051123105311.01f95960@mail.telecommunity.com> <5.1.1.6.0.20051123144125.01f9a9f8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051123180630.01f905b8@mail.telecommunity.com> At 10:07 PM 11/23/2005 +0100, Martin v. L?wis wrote: >Phillip J. Eby wrote: >>I try to use very long names for options that can have damaging effects >>if used indiscriminately. A project that's installed the "old-fashioned >>way" (which is what this does, apart from adding .egg-info) is hard to >>uninstall and may overwrite other projects' files. So, it is only safe >>to use if the files are being managed by some external package manager, >>and it further only works for a single installed version at a time. So >>the name is intended to advertise these facts, and to discourage people >>who are just reading the option list from trying it out to see what it >>does. :) > >And that is indeed a fine principle. The option would only occur >inside debian/rules (which is the Makefile to build the deb file), >so users would never need to type it manually (except for the DD >creating debian/rules). > >My request would be that the option is stable in its meaning, >e.g. if there is a need to extend/change its semantics, a different >option is introduced (unless it is certain that existing Debian >packages would continue to build correctly under the new meaning). Since this option exists only to provide a backward-compatible layout for external package managers, I can't see how it would change in terms of layout. I can imagine that across different versions, I might provide a different set of warnings, however. For example, it seems to me I might want to have this option check for existing files prior to installation, and abort if any are found. This seems like a good thing to do since external packaging uses would likely be run against a clean target or put into a temporary directory anyway. I'm also debating whether this option should also require the --record option, since there might otherwise be no way for an external packaging tool to know which files and directories belong to the installed package. From p.f.moore at gmail.com Thu Nov 24 00:17:59 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 23 Nov 2005 23:17:59 +0000 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051123125622.039b8008@mail.telecommunity.com> References: <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <E1Eeisx-0008Py-00@server.0x1.org> <5.1.1.6.0.20051122201306.03a9a090@mail.telecommunity.com> <5.1.1.6.0.20051123125622.039b8008@mail.telecommunity.com> Message-ID: <79990c6b0511231517w353a18e2lab0a9e1a7f7c4ffc@mail.gmail.com> On 11/23/05, Phillip J. Eby <pje at telecommunity.com> wrote: > At 04:53 PM 11/23/2005 +0000, Paul Moore wrote: > >If there was a way of building a Windows installer that installed > >packages in "egg" form, so I didn't have to use setup.py at all when > >installing, just double-click on the installer, that would suit me. > >This feels like what the Debian people want with their .deb format as > >well. > > Right - this would hopefully be covered at some point using an .msi or .exe > wrapper for eggs on Windows. So far, though, you're the only person I've > heard from about wanting this on Windows, versus the number of people who > want it for e.g. Debian. So I'm still considering Debian wrapper support > as being more urgent than .msi wrapper support. That's fair. It's quite possible that I'm unusual in my liking for Windows installers in this context - arguing that Windows' Add/Remove Programs applet is a good package management system is an uncommon stance to take, certainly :-) > My point to David was simply that egg packaging in the .egg form is more > akin to Stow than to CPAN, so most of the flaws of CPAN are not applicable > to them. Sorry, I don't know what Stow is, so that doesn't clarify things to me (but that's OK, I got your point from the previous paragraph, so if the clarification helps David, that's enough). > >The wording I'm using here is possibly not accurate - there seem to be > >a confusing mix of concepts and ideas going round, with no-one having > >a good understanding of all of them. My apologies - if there's a good > >glossary somewhere, which explains what to call things like > > > > * The executable I click on in Windows to install something > > * The .deb file that a Debian user downloads and installs > > I would call these "system packages", to distinguish them from Python > packages. You (and others) would like to ensure that any project you > install is wrapped in a system package. Gotcha. And you understand my position perfectly. > > * The set of files that end up in Python's site-packages > > A "project distribution", or just a "project" (which is a less accurate > term but more convenient). OK. > > - for a "normal" install (bdist_wininst, non-egg, whatever) > > This has never seemed to me to need a special term unto itself, so I don't > have one. I think I was more interested in it to allow me to define the difference (egg metadata) below, so that's fine. > > - for an egg > > An egg. :) :-) > > - that comprises the difference between the above two ( :-) ) > > Egg metadata. An egg is just an installed project distribution that > includes egg metadata, in one of the three possible layouts for putting > that metadata alongside the rest of the project contents. OK, I see it now. I need to reread your previous posts about the 3 layouts, as understanding those would probably give me the remaining pieces of the puzzle that I need. Regarding me being the only person interested in Windows installers which wrap eggs (which I don't dispute), I'd be curious to know what proportion of TurboGears (or any other egg-based project, I guess) users are on Windows. And of them, how many have it in "live" use (as how I'd be willing to install on a development box would differ from what I'd do - or possibly be allowed to do - on a production box). One final point - I think that naming of setup.py commands may be confusing things here. Before eggs, the main bdist_ commands (bdist_wininst, bdist_rpm, bdist_deb, bdist_msi, ...) created *system packages* by your definition above. And yet, bdist_egg doesn't - it creates an egg, which is a subtype of a project distribution. This leads (IMO) to confusion, in that we are now seeing interest in system packages which wrap eggs. Arguably, there should ultimately be distutils commands for this. But what to call them? bdist_egg_wininst, bdist_egg_deb, etc? Something else? This probably just proves that naming isn't my strong suit :-) But do you see my point that bdist_egg is the odd one out among the bdist_ commands (as it doesn't create a system package)? Paul From smurf at smurf.noris.de Thu Nov 24 00:25:28 2005 From: smurf at smurf.noris.de (Matthias Urlichs) Date: Thu, 24 Nov 2005 00:25:28 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051123180630.01f905b8@mail.telecommunity.com> References: <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <5.1.1.6.0.20051122192931.01f84530@mail.telecommunity.com> <5.1.1.6.0.20051123105311.01f95960@mail.telecommunity.com> <5.1.1.6.0.20051123144125.01f9a9f8@mail.telecommunity.com> <5.1.1.6.0.20051123180630.01f905b8@mail.telecommunity.com> Message-ID: <20051123232528.GL20960@kiste.smurf.noris.de> Hi, Phillip J. Eby: > I'm also debating whether this option should also require the --record > option, since there might otherwise be no way for an external packaging > tool to know which files and directories belong to the installed package. > Sure there is -- you install with --root (in Debian's case it's usually $pwd/debian/tmp), which was empty before you started, and declare that *all* files in there belong to the package you're creating. Or you use an external LD_PRELOADed file installation logger. Thus you could simply declare that somebody who does that either deserves to shoot themselves in the foot, or has a valid reason to do it (maybe I'm testing the latest version and don't want to fill my system with .pth files pointing to my development directories -- I, for one, *will* forget to delete them afterwards ... on the other hand, the mess I just created will nicely clean up after itself when I install the next official version on top, assuming I managed not to install random additional files). -- Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf at smurf.noris.de Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de - - "Remember, the German people are the chosen of God. On me the German Emperor, the spirit of God has descended. I am His sword, His weapon, and His vice-regent." [Kaiser Wilhelm II, 4 August 1914] From david at mantara.com Thu Nov 24 00:39:09 2005 From: david at mantara.com (David Arnold) Date: Thu, 24 Nov 2005 10:39:09 +1100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: Your message of "Wed, 23 Nov 2005 23:17:59 -0000." <79990c6b0511231517w353a18e2lab0a9e1a7f7c4ffc@mail.gmail.com> Message-ID: <E1Ef4Cj-0000yk-00@d.0x1.org> -->"Paul" == Paul Moore <p.f.moore at gmail.com> writes: >> My point to David was simply that egg packaging in the .egg form is >> more akin to Stow than to CPAN, so most of the flaws of CPAN are >> not applicable to them. Paul> Sorry, I don't know what Stow is, so that doesn't clarify things Paul> to me (but that's OK, I got your point from the previous Paul> paragraph, so if the clarification helps David, that's enough). In fact, I use stow for anything not supported by Debian. Stow allows you to maintain a repository of built "installations", and to "activate" overlapping things one at a time. Normally, this means different versions of the one "product". It achieves this by managing symlink farms. It's a fairly Unix-y thing :-) >> I would call these "system packages", to distinguish them from >> Python packages. You (and others) would like to ensure that any >> project you install is wrapped in a system package. Paul> Gotcha. And you understand my position perfectly. That's my wish also. I realise that other people (possibly more so on non-Debian systems?) don't have such an attachment to the way the system manages installed products. Don't get me wrong: I'm not disputing that .eggs are useful, nor that they provide capabilities that a Debian-packaged result of an installation using 'python setup.py install' might not (at least now, easy-deb aside). And, fwiw, as a developer of Python modules, a way of distributing them that allows others to safely and easily install different versions of my modules with different applications on the same machine is attractive. But I was hoping that I could help clarify the point of view of a Debian user, by pointing out that there's at least some part of the Debian user community that won't like installing .egg applications unless they're sanely converted to .debs d From pje at telecommunity.com Thu Nov 24 00:42:32 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 23 Nov 2005 18:42:32 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <79990c6b0511231517w353a18e2lab0a9e1a7f7c4ffc@mail.gmail.co m> References: <5.1.1.6.0.20051123125622.039b8008@mail.telecommunity.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <E1Eeisx-0008Py-00@server.0x1.org> <5.1.1.6.0.20051122201306.03a9a090@mail.telecommunity.com> <5.1.1.6.0.20051123125622.039b8008@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051123182553.01f905b8@mail.telecommunity.com> At 11:17 PM 11/23/2005 +0000, Paul Moore wrote: >OK, I see it now. I need to reread your previous posts about the 3 >layouts, as understanding those would probably give me the remaining >pieces of the puzzle that I need. To summarize, the layouts are .egg file (1) or directory (2), which both have a projectname-version-pyversion-platform.egg filename, and contain the project contents alongside an EGG-INFO subdirectory where the metadata lives. The third layout, originally designed for development rather than deployment, is just the project contents in an arbitrary directory, alongside a projectname.egg-info directory with the metadata in it. In setuptools 0.6a9, I will modify pkg_resources so that the .egg-info directory can also be a file (whose contents are ignored, for now) and so that its filename can also include the version, so that the PKG-INFO file can be optional in that case. This will make it easier for people wrapping non-setuptools packages as system-packaged eggs, because they will just need to "touch site-packages/projectname-version.egg-info" in order to let setuptools-using projects know that the project has been installed. In a sense, this approach sounds like a kind of "PEP 262 lite", in that it produces a basic database of installed packages, just as the .egg layout can be thought of as "PEP 262 plus", in that it offers extensible metadata as well as basic presence-and-version information. >Regarding me being the only person interested in Windows installers >which wrap eggs (which I don't dispute), Oh, I doubt you're really the only person *interested*, you're just the only person who's *asked* for it. :) > I'd be curious to know what >proportion of TurboGears (or any other egg-based project, I guess) >users are on Windows. And of them, how many have it in "live" use (as >how I'd be willing to install on a development box would differ from >what I'd do - or possibly be allowed to do - on a production box). Beats me; in my personal case, the intersection of "Windows" and "production box" is the empty set. :) >One final point - I think that naming of setup.py commands may be >confusing things here. Before eggs, the main bdist_ commands >(bdist_wininst, bdist_rpm, bdist_deb, bdist_msi, ...) created *system >packages* by your definition above. And yet, bdist_egg doesn't - it >creates an egg, which is a subtype of a project distribution. This >leads (IMO) to confusion, in that we are now seeing interest in system >packages which wrap eggs. That's a reasonable thought, except for bdist_dumb, which I think was already an exception to your otherwise-reasonable theory. > Arguably, there should ultimately be >distutils commands for this. But what to call them? bdist_egg_wininst, >bdist_egg_deb, etc? Something else? Actually, I think the better long-term approach is more likely to be tools like easy_deb that wrap easy_install. "Better" here meaning that it can save the system packager work, because it can handle finding and fetching and building in an automated way even for non-setuptools packages it has never seen before. While there are occasionally some issues with projects that have unusual customizations to the distutils, those customizations would potentially give a system packager similar troubles anyway. Conversely, if you tried to build system-packaged eggs for non-setuptools packages without easy_install, you'd have to patch their setup scripts in order to get at those hypothetical bdist_egg_foo commands. Last, but not least, system packagers vary widely in their essential build process, so it's probably easier and less fragile to let them wrap easy_install and then work from one of the three egg layouts, than to try to embed the packaging process entirely within the distutils. Of course, I haven't played around with anything but bdist_wininst and bdist_rpm, but it seems to me that at least bdist_rpm suffers from a lot of complexity by having to squeeze the process into the scope of a single distutils command, versus what it would be like if you just took an egg and turned it into an RPM. >This probably just proves that naming isn't my strong suit :-) But do >you see my point that bdist_egg is the odd one out among the bdist_ >commands (as it doesn't create a system package)? Yeah, except for the part where bdist_dumb isn't really a system packager. I always interpreted bdist as simply meaning a "binary" or "built" distribution; i.e., one that does not require a build step, just "installation". From pje at telecommunity.com Thu Nov 24 00:49:54 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 23 Nov 2005 18:49:54 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <E1Ef4Cj-0000yk-00@d.0x1.org> References: <Your message of "Wed, 23 Nov 2005 23:17:59 -0000." <79990c6b0511231517w353a18e2lab0a9e1a7f7c4ffc@mail.gmail.com> Message-ID: <5.1.1.6.0.20051123184351.030da5a0@mail.telecommunity.com> At 10:39 AM 11/24/2005 +1100, David Arnold wrote: >But I was hoping that I could help clarify the point of view of a Debian >user, by pointing out that there's at least some part of the Debian user >community that won't like installing .egg applications unless they're >sanely converted to .debs Certainly. There have been Debian users asking for this on the TurboGears mailing list from day 1, so I'm familiar with the viewpoint, and believe it's entirely rational. (The FAQ response for this is to install the eggs in /usr/local, or else use easy_deb to wrap the eggs in .debs, but I guess this started because somebody felt that there should be official .debs for TurboGears and its dependencies as well.) I just got the impression that you were saying that *any* use of eggs would be tantamount to CPAN-style chaos, which isn't true even in a non-packaged environment, since the architecture is basically that of Stow. (That is, install every version of every thing in its own directory, and then either find what you need at runtime or update pointers to the active versions.) From pje at telecommunity.com Thu Nov 24 00:59:28 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 23 Nov 2005 18:59:28 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <20051123232528.GL20960@kiste.smurf.noris.de> References: <5.1.1.6.0.20051123180630.01f905b8@mail.telecommunity.com> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <5.1.1.6.0.20051122192931.01f84530@mail.telecommunity.com> <5.1.1.6.0.20051123105311.01f95960@mail.telecommunity.com> <5.1.1.6.0.20051123144125.01f9a9f8@mail.telecommunity.com> <5.1.1.6.0.20051123180630.01f905b8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051123185038.032209c0@mail.telecommunity.com> At 12:25 AM 11/24/2005 +0100, Matthias Urlichs wrote: >Phillip J. Eby: > > I'm also debating whether this option should also require the --record > > option, since there might otherwise be no way for an external packaging > > tool to know which files and directories belong to the installed package. > > >Sure there is -- you install with --root (in Debian's case it's usually >$pwd/debian/tmp), which was empty before you started, and declare that >*all* files in there belong to the package you're creating. Or you use >an external LD_PRELOADed file installation logger. > >Thus you could simply declare that somebody who does that either >deserves to shoot themselves in the foot, or has a valid reason to do it Yeah, although historically I've preferred in such cases to require a further option like --yes-i-want-to-shoot-my-foot in order to enable foot-directed shooting features. It tends to keep people with holes in their feet from complaining afterwards. :) >(maybe I'm testing the latest version and don't want to fill my system >with .pth files pointing to my development directories -- I, for one, >*will* forget to delete them afterwards Well, they'll be overwritten when you change what version you're using, and ignored if you've deleted the source tree, and there'll only be one such file in the first place unless you're using the easy_deb approach of a directory full of them. > ... on the other hand, the mess >I just created will nicely clean up after itself when I install the next >official version on top, assuming I managed not to install random >additional files). I'm not sure if you mean if you install the new proposed way, or the existing setuptools way. In the existing setuptools way, your mess is strictly in your source tree or inside an .egg file or directory, so there's nothing to clean up except maybe a meaningless line in your easy-install.pth, which will indeed be cleaned up if you install an official version. In the new way, overwriting stuff in site-packages seems like a good way to implement the holes-in-feet protocol. :) Maybe what I should do is require *either* --root (which should in that case point to an empty directory) or --record in conjunction with --single-version-externally-managed. I don't really want to condone using this option in place of "setup.py develop", especially since the latter at least has an --uninstall option. :) From hawk78_it at yahoo.it Thu Nov 24 04:24:50 2005 From: hawk78_it at yahoo.it (Vincenzo Di Massa) Date: Thu, 24 Nov 2005 04:24:50 +0100 Subject: [Distutils] [Bulk] Re: .egg in Debian summary? In-Reply-To: <5.1.1.6.0.20051123123734.039adbb8@mail.telecommunity.com> References: <dm11id$pdi$1@sea.gmane.org> <5.1.1.6.0.20051123123734.039adbb8@mail.telecommunity.com> Message-ID: <200511240424.50892.hawk78_it@yahoo.it> Alle 18:56, mercoled? 23 novembre 2005, Phillip J. Eby ha scritto: > At 04:39 PM 11/23/2005 +0100, Vincenzo Di Massa wrote: > >I think setuptools could really help Debian developer as well as Fedora, > >Mandriva, Gentoo... by: > >1) providing functionality to install files outside of egg dir/zip (for > >example one could install code to site-packages and data-files to > > /usr/share/ or somewhere else) > > There will be some steps towards this in the 0.7 series, although initially > that's going to be by providing user-executed post-installation tools, > which would also be executable by packager-defined postinstallation > steps. Since easy_install is not a package manager, however, I would > prefer users be able to trust that running easy_install is only going to > make a Python package and some scripts available, and not do anything to > files elsewhere on their system. This position may evolve over time, since > there are a lot of other features that (in my mind at least) need to come > first before I can think deeply about the issue. (Related issues: header > file installation, documentation, etc.) That really sounds GREAT! What about providing an .egg-data file to be installed with the egg in site-packages or uncompressed and installed by an external packaging mechanism? Metadata in .egg packages could allow egg using projects to find the corresponding .egg-data package anywhere even if it unzipped and its contents spread all over the filesistem. I think this will be useful and nice. :-) > >2) providing a "fakeroot" option to fool setup.py (for example I want > >setuptools to install the files in > > debian/tmp/usr/lib/pythonX.Y/site-packages and scripts in > > debian/tmp/usr/bin and everithing must work when it copied to > > /usr/{bin,lib}. > > There is in fact such an option now; as far as I know the --root option to > "setup.py install" should do it, and I am pretty sure it's used by > bdist_rpm. You are right. And I did know about it :-) . I just asked for the feature in the wrong place: the --root would be usefull in easy_install too, not just in setup.py. This would allow you to prepare eggs distributions out of packages that are not using setuptools and put the stuff into a --root dir (which is what you are proposing Debian developers to do with the upcoming new setuptools version able to install "legacy-eggs"). > >2) providing a paseable output with setup process data like: dependencies > >(meet and unmett), files installed, author, license, and so on > > Perhaps you could provide a sample of what you have in mind? A patch would > be even better, although I would be using it mainly to understand what > you'd like to have and not necessarily as a basis for implementing it. I think that what I have in mind is to find out what is best to provide. There are many packaging systems out there and each one has its metadata format. My proposal is to find out what setuptools can do to generate a nice metadata file that describes the egg (version, author, wesite, gpg signature, .... anything that is usefull and doable), its dependencies (name, version, ???) and ???. That metadata file could be created when the egg file is created (using an option like --record). Parsing that file a python egg can be semi-automatically put into any pakage format (deb, rpm, tgz, ???) simply building a little app that converts setuptools metatada format to target platform metadata. > >All this will let people just "convert" the output of setuptools into an > > rpm specfile, debian dir, gentoo ebuild and so on. > >All that is needed is just a common metadata format that can be parsed by > >tools like easy-deb, easy-rmp easy-ebuild easy-(???). > > Note that easy_install and install have --record options that write the > names of all installed files or directories to a file for you. The > remaining metadata you're asking about can already be queried via the > pkg_resources API, since it's part of the resulting > eggs. 'someEgg.get_metadata("PKG-INFO")' will get you the PKG-INFO file > with all the standard distutils metadata, for example. The 'requires()' > method gives information about dependencies. It would be nice to arrange a set of metadata to be put toghether in just one file built when the egg is biult. Maybe you could add requires() output in PKG-INFO (making it comply with version 1.2 which is in PEP). So the output file cou then just be the 1.2 PKG-INFO. > >This way setuptools provides the information to distribution maintainers > > in a well documented, api independent way. > > Well, if you want API independence, you can directly look at the .egg-info > or EGG-INFO directory contents, as the data is in plain-text files like > requires.txt and PKG-INFO. PKG-INFO is a PEP-defined format, and the > requires.txt format has been stable for maybe 7 months now. I don't > anticipate any changes to it. > > > Distribution maintainers can use this > >info to reorganize, repackage, invent new package formats, but without > >involving setuptools and python developers. > > Fair enough, although you should keep in mind that there are only three > valid installation layouts for egg-specific projects: .egg zipfile, .egg > directory, and "old school" installation with an .egg-info directory > installed alongside. Anything else, and it would involve changing the egg > runtime to work with it. I know, and I think eggs are GOOD even because they are(or will be with newer versions of setuptools) easy to put into other packages types (like debs). But you know what packagers(meaning peoplewho does packaging) are like: they have policies and tend to reinvent the weel (and hot water too!!!) every time just becouse they have policies. It is sad to listen to good developers spreading FUDware thoughts about nice FLOSS projects like yours. I think that the best thing would be to give them a plaintext file to parse and the egg as is. They will find a very mind and time consuming way to those. But that is not your business anymore: people at DistroX don't like eggs? They can rewrite the egg in assembly if they like to... that will not hurt your project I think. If eggs are GOOD (as we think) other distros may have more success among python developers and users. Vincenzo > >About the pythonpath and pth files handlind: > >if a project A really wants to use project B eggs it can easily require > > them without the need for a pth file. > >If project C wants to use project B, but project C is not setuptools aware > >one > >could patch project C (just adding require() to it) instead of paching > >project B and project A removing egg support from them. > > Note that if C is an application rather than a library, one can add the > requirements to setup.py and use easy_install to install it, which will > create script wrappers that perform the necessary require() > operations. So, you don't have to patch C very deeply in this case. In > addition, the scripts will require() C itself, so you don't need to install > C with a .pth unless it is a library that is needed by something else that > is not packaged using this system. > > Basically, the whole thing is designed so that as long as a project is > setuptools-based, it doesn't need explicit require() calls, as these are > handled by setuptools. Only if you want a project to be *always* on > sys.path do you need a .pth file. I know, and if I were a Debian developer I would change the (holy) Debian Python Policy to let a module be instaleed without pth if that is ok! :-) > >Onother possible solution would be to have: > >python-A > >python-pypi-A > >python-B > >python-pypi-B > >python-C > > > >Where > >python-B) depends on python-pypi-A and just provides the pth file (handled > >like easy-deb does) > >python-pypi-B) contains the egg > > > >So project C will depend on python-B, which will install the pth. But if > >project B is required just by project A which is setuptools aware the > >dependency will be on python-pypi-B: the pth file will not be installed. > > So, you're saying that python-foo would install 'foo' such that it is > always on sys.path, but python-pypi-foo simply installs 'foo' in such a way > that is discoverable by the egg runtime. That's an interesting idea, I'll > have to think a while about the consequences of that before I comment on it > further. Vincenzo ___________________________________ Yahoo! Mail: gratis 1GB per i messaggi e allegati da 10MB http://mail.yahoo.it From joss at debian.org Thu Nov 24 13:44:02 2005 From: joss at debian.org (Josselin Mouette) Date: Thu, 24 Nov 2005 13:44:02 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> References: <438378C1.4050401@colorstudy.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> Message-ID: <1132836242.407.25.camel@silicium.ccc.cea.fr> Le mardi 22 novembre 2005 ? 17:05 -0500, Phillip J. Eby a ?crit : > And over the last few months, I believe we've also succeeded in stomping > most of the issues that people had with getting solid non-root > installations on their Linux distributions. So the reasons for developers > to prefer their dependencies to be managed as eggs will only improve over > time, as the egg system allows Python developers to control and introspect > their dependencies, rather than keeping that information hidden behind > diverse platform-specific packaging tools. This is useful for non-root installations, but it is only a hassle when you are making .deb's. Here, dependencies are already handled, and you rely on the system to provide correct versions for packages you depend upon. > It's only "competitive" if you feel that there must be only one way to do > it. (And if you do feel that way, then it also should be obvious that eggs > are the superior solution, since they don't take away any capabilities of > the old, only provide new ones.) They only introduce more complexity, instead of bringing real features. > Obviously, every individual distribution would like to have Python packages > conform to their individual system. However, on the whole, it is clearly > better for the Python developer to have practical dependency management > that doesn't tie their efforts to a single platform, packaging system, or > distribution. And that's why there are things like dh_python to adapt python distutils installations to be compliant to the Debian way of things. However, eggs make it so complicated that the adaptation layer would be unmaintainable. This is a good reason to think the whole egg system should be avoided. -- .''`. Josselin Mouette /\./\ : :' : josselin.mouette at ens-lyon.org `. `' joss at debian.org `- Debian GNU/Linux -- The power of freedom From joss at debian.org Thu Nov 24 14:14:13 2005 From: joss at debian.org (Josselin Mouette) Date: Thu, 24 Nov 2005 14:14:13 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051123161333.01f9b7e8@mail.telecommunity.com> References: <5.1.1.6.0.20051123105311.01f95960@mail.telecommunity.com> <5.1.1.6.0.20051122192931.01f84530@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <5.1.1.6.0.20051122192931.01f84530@mail.telecommunity.com> <5.1.1.6.0.20051123105311.01f95960@mail.telecommunity.com> <5.1.1.6.0.20051123161333.01f9b7e8@mail.telecommunity.com> Message-ID: <1132838053.12478.8.camel@silicium.ccc.cea.fr> Le mercredi 23 novembre 2005 ? 18:06 -0500, Phillip J. Eby a ?crit : > The degree of negativity from the Debian side at the outset of this > conversation (virtually all of it from you) has not been conducive to > making this happen. As a simple matter of practicality, I can't afford to > leave your comments unanswered, not because I feel any need to convince you > personally of anything, but because I don't want to leave anyone else with > the impression that your portrayal of these so-called "drawbacks" is a fair > one. Otherwise, I would have just ignored your comments and focused on > working with the people who seem more interested in finding solutions than > finding ways to declare a non-existence of the problem. As it is, I feel > forced to spend time replying to your comments point-by-point, that I could > otherwise spend on actually helping to resolve the issues. The reason why people from Debian are complaining is that all solutions you have proposed so far are not acceptable. Full stop. Currently, I can't see any way to package eggs in a clean and elegant way. > If I were to adopt your tone, I would be calling Debian a fragile and > broken system that is unable to deal well with simple matters like editing > a file upon installation, I suggest you go get yourself a cluebat. Editing a file upon installation is a completely stupid idea, and that's why we try to avoid it whenever possible. The reason should be obvious: while it's easy to add or remove files from the system, you can never guarantee a file will be successfully edited. Furthermore, editing a file for modifying the sys.path is an even more stupid idea, as python already has everything you need to deal with new modules: simply ship them in site-packages/ subdirectories. > or having multiple versions of a package > installed at the same time. Is this a joke? Installing several versions of a package is fragile, it's a security mistake, and takes place on the hard disk for no real use. I'm currently fighting so that we stop supporting so many python versions, and I hope we won't have to support several versions for each package, each of them for several python versions! This would become a complete nightmare. The whole purpose of eggs seems to be able to distribute whole projects, with their dependencies, in a simple format. Sorry, but it goes in the opposite direction of all Linux distributions. The whole point of inter-package dependencies is to avoid this, and believe me, there are lots of reasons to do that; otherwise, all binaries would be statically linked, or packed with their libraries. This is how things work in the Windows world, but dependencies were invented exactly to avoid it. > Sure, the limitation might exist, but is it > fair to call Debian fragile or broken because of it? Not a bit! I've > therefore been very careful to describe any such tradeoffs that Debian > makes in neutral terms rather than categorically pejorative ones. I would > prefer if you would extend me the same courtesy of not describing every > design tradeoff I make as being a "non-standard", "drawback", "for no good > reason". Refusing to edit files upon installation, for example, is not a tradeoff. It's a deliberate decision, based on the fact this is a bad idea. > If Debian adds this metadata marker for its non-setuptools Python packages, > then the Python packages will be "eggs" in the sense that other eggs will > be able to discover them via the pkg_resources API, and thus TurboGears > users will be able to use the Debian-supplied versions of ElementTree and > the like. Why would we need to modify hundreds of packages to contain empty files, just because another package cannot import its dependencies correctly? > >I don't see why multi-version support necessarily requires to > >increase sys.path. In the case of eggs, version dependencies are > >expressed explicitly in the code (through require() calls), > > Actually, they're expressed in the egg metadata, and the wrappers on a > project's scripts execute the require() calls, so that the code doesn't > have to contain explicit require() calls except for more-dynamic > situations, such as plugins and "optional extra features" that require > additional projects to be present. I'm used to remove such code checking for the version of dependencies in python packages, as it is broken most of the time. Introducing another broken way to check the dependencies can be worked around the same way, but it's sadly more burden on the package maintainers. Still, I can assure you it's less burden than adapting the system to handle eggs, as I understand them. > Note that setuptools is in release 0.6a8 at the moment - it is obviously > not a polished product, but it provides enough functionality to be quite > useful to many Python developers. To this point, directly working on > integration with external packaging tools has not been a focus, although I > always have given top priority to responding to questions and requests from > people working on integration with those tools (e.g. the volunteers who > worked on easy_deb and the Gentoo stuff). I can't reasonably learn the > technical details of every packaging system, so it is best to let > volunteers familiar with individual packaging systems tell me what they > need in order to effectively wrap the system. It seems to me that you haven't thought of integrating with anything except Windows, from the very beginning. Eggs are similar to .jar or .xpi files, that also come from the Windows world, and believe me, it's a real pain to maintain a package with such files. They aren't suited at all for Unix systems and you can expect many Debian maintainers to find ways to distribute python packages the way they should be distributed. > Up until now, my interactions with such volunteers have been most pleasant > and positive. To my knowledge, it's not usual for packaging system > developers to spew FUD at a project and look for ways to exclude or break > the work of developers who've chosen to use it. I'm therefore more than a > little surprised by some of the attitude I've received. I hope, though, > that we can get past that soon, if only because it means I'll have more > time to work on implementing and documenting whatever the resolution is. ;) If there is any implementation, it will be a crude hack. You'd better think of a way for python to completely bypass all egg metadata and still function properly. Then, we can think of ways to fill dependencies at the .deb level automatically, but it's really less important. Regards, -- .''`. Josselin Mouette /\./\ : :' : josselin.mouette at ens-lyon.org `. `' joss at debian.org `- Debian GNU/Linux -- The power of freedom From joss at debian.org Thu Nov 24 14:22:00 2005 From: joss at debian.org (Josselin Mouette) Date: Thu, 24 Nov 2005 14:22:00 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> References: <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> Message-ID: <1132838520.12478.15.camel@silicium.ccc.cea.fr> Le mardi 22 novembre 2005 ? 18:47 -0500, Phillip J. Eby a ?crit : > I don't understand you here. Are you saying that it's not possible for > dpkg to do a post-install or uninstall operation like adding or removing a > line from a file? It's possible, but it's fragile. > Of course, this creates additional work for package maintainers that > wouldn't be present with setuptools' normal .egg file/directory > distributions, and my assumption was that the maintainers would prefer to > be able to ignore such issues and get the benefit of dependencies defined > by the upstream developers. Debian is used to ignore dependencies defined by upstream developers, because in most cases they turn out to be unsuitable. That's why there's no tool to convert libtool versioning schemes to shlibs files, why there's no tool to convert configure.in requirements to build-dependencies, and why there's currently no tool to generate dependencies for python packages at all. The maintainer of each package is on his own to define dependencies, because he knows better. > Eggs keep each project in its own little > bubble, where it can't overwrite anything else and can be uninstalled > without removing any overlapping parts. I can't believe you are pushing a solution with such a "feature". Duplication is something we should avoid at all costs. -- .''`. Josselin Mouette /\./\ : :' : josselin.mouette at ens-lyon.org `. `' joss at debian.org `- Debian GNU/Linux -- The power of freedom From joss at debian.org Thu Nov 24 14:24:36 2005 From: joss at debian.org (Josselin Mouette) Date: Thu, 24 Nov 2005 14:24:36 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <200511221541.08761.tanner@real-time.com> References: <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122152901.030e2ef0@mail.telecommunity.com> <43838D58.7000801@egenix.com> <200511221541.08761.tanner@real-time.com> Message-ID: <1132838676.12478.18.camel@silicium.ccc.cea.fr> Le mardi 22 novembre 2005 ? 15:41 -0600, Bob Tanner a ?crit : > When I read the above, my knee-jerk reaction is: Where is the data to backup > this statement? > > Follow up questions are: > > How much slower? We talking milliseconds, seconds, minutes? Yes, there are > variables, here, but narrow them to a set number and compare zip vs > directory? > > Previous post talked about memory consumption, again bytes, kilobytes, > megabytes? Performance is not the key point, although a package that affects performance of other packages is obviously broken in some way. The problem is the added complexity for no real gain. The only thing you gain is multiple versions of a package installed at the same time. Something that should be avoided. -- .''`. Josselin Mouette /\./\ : :' : josselin.mouette at ens-lyon.org `. `' joss at debian.org `- Debian GNU/Linux -- The power of freedom From joss at debian.org Thu Nov 24 14:26:40 2005 From: joss at debian.org (Josselin Mouette) Date: Thu, 24 Nov 2005 14:26:40 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <43835988.3060106@colorstudy.com> References: <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> Message-ID: <1132838800.12478.21.camel@silicium.ccc.cea.fr> Le mardi 22 novembre 2005 ? 11:46 -0600, Ian Bicking a ?crit : > Eggs give room for package metadata that doesn't exist otherwise. > Putting dependencies aside, this is functionality that simply doesn't > exist with the standard distutils installation. You are advertising this metadata a lot, but what does it exactly contain? If it's for dependencies, we can really live better without them, or with a tool to convert them into .deb dependencies. -- .''`. Josselin Mouette /\./\ : :' : josselin.mouette at ens-lyon.org `. `' joss at debian.org `- Debian GNU/Linux -- The power of freedom From cmlenz at gmx.de Thu Nov 24 15:19:38 2005 From: cmlenz at gmx.de (Christopher Lenz) Date: Thu, 24 Nov 2005 15:19:38 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1132838800.12478.21.camel@silicium.ccc.cea.fr> References: <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> Message-ID: <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> Am 24.11.2005 um 14:26 schrieb Josselin Mouette: > Le mardi 22 novembre 2005 ? 11:46 -0600, Ian Bicking a ?crit : >> Eggs give room for package metadata that doesn't exist otherwise. >> Putting dependencies aside, this is functionality that simply doesn't >> exist with the standard distutils installation. > > You are advertising this metadata a lot, but what does it exactly > contain? If it's for dependencies, we can really live better without > them, or with a tool to convert them into .deb dependencies. You know, it is really unfair to make a statement like "The problem is the added complexity for no real gain" if you're gonna ask the above question in a follow-up message. You clearly don't understand the purpose of Python eggs, yet you feel qualified to judge whether or not it provides any benefit. Sigh. (And no, I'm not going to repeat the numerous attempts by Phillip to politely and comprehensively explain it all.) In the end, what matters is that Python eggs are being quickly adopted by Python application developers, because they allow us to easily build extensible systems. When Debian users want to install and use those systems, you're gonna gave to tell them why eggs aren't supported. The requirements of packaging systems aren't being ignored here. We're (well, Phillip is) listening, and trying to provide constructive input. It'd be nice if the Debian folks would be similarly forthcoming. Cheers, Chris -- Christopher Lenz cmlenz at gmx.de http://www.cmlenz.net/ From abo at minkirri.apana.org.au Thu Nov 24 15:24:24 2005 From: abo at minkirri.apana.org.au (Donovan Baarda) Date: Thu, 24 Nov 2005 14:24:24 +0000 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1132838053.12478.8.camel@silicium.ccc.cea.fr> References: <5.1.1.6.0.20051123105311.01f95960@mail.telecommunity.com> <5.1.1.6.0.20051122192931.01f84530@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <5.1.1.6.0.20051122153410.03c840e0@mail.telecommunity.com> <5.1.1.6.0.20051122181255.03a51d98@mail.telecommunity.com> <5.1.1.6.0.20051122192931.01f84530@mail.telecommunity.com> <5.1.1.6.0.20051123105311.01f95960@mail.telecommunity.com> <5.1.1.6.0.20051123161333.01f9b7e8@mail.telecommunity.com> <1132838053.12478.8.camel@silicium.ccc.cea.fr> Message-ID: <1132842264.25145.39.camel@warna.corp.google.com> On Thu, 2005-11-24 at 14:14 +0100, Josselin Mouette wrote: [...] > > or having multiple versions of a package > > installed at the same time. > > Is this a joke? Installing several versions of a package is fragile, > it's a security mistake, and takes place on the hard disk for no real > use. This is true for a production environment, but not for a development environment where you want to develop/test/support multiple versions of Python. Debian is used for both. It should support both. The broad diversity of Python applications, all with their own independent Python version dependencies and migration schedules, also makes this impractical for many production environments. Hence we should support multiple versions of Python. > I'm currently fighting so that we stop supporting so many python > versions, and I hope we won't have to support several versions for each [...] And there is still a lot of Debian Python people resisting :-) > package, each of them for several python versions! This would become a > complete nightmare. [...] I think we are pretty close to managing this already, although it could use some tweaks... mind you, I'm all talk and no action on this front, so I should probably be ignored. Disclaimer: This should not be interpreted as an argument for or against egg's. This thread is the first time I've even heard of them. -- Donovan Baarda <abo at minkirri.apana.org.au> http://minkirri.apana.org.au/~abo/ From p.f.moore at gmail.com Thu Nov 24 15:37:02 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 24 Nov 2005 14:37:02 +0000 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051123182553.01f905b8@mail.telecommunity.com> References: <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <E1Eeisx-0008Py-00@server.0x1.org> <5.1.1.6.0.20051122201306.03a9a090@mail.telecommunity.com> <5.1.1.6.0.20051123125622.039b8008@mail.telecommunity.com> <5.1.1.6.0.20051123182553.01f905b8@mail.telecommunity.com> Message-ID: <79990c6b0511240637l73554774u774d81bf1bda73d8@mail.gmail.com> On 11/23/05, Phillip J. Eby <pje at telecommunity.com> wrote: > Actually, I think the better long-term approach is more likely to be tools > like easy_deb that wrap easy_install. "Better" here meaning that it can > save the system packager work, because it can handle finding and fetching > and building in an automated way even for non-setuptools packages it has > never seen before. While there are occasionally some issues with projects > that have unusual customizations to the distutils, those customizations > would potentially give a system packager similar troubles anyway. I have now made 3 attempts to write a message responding to this, and got muddled each time. So I'm just going to try giving my key points, unadorned. I can provide supporting discussion, and will do so if you like, but I want to separate the threads first: 1. Windows is a special case here, as traditionally, Windows system packages are provided by the project itself, whereas Linux distributions have separate packaging groups who handle that outside the project. 2. Eggs don't compete with system packagers. And yet projects such as TurboGears and Kid are providing *only* eggs, and no bdist_wininst packages. This is because it is *not possible* at present for projects to provide system packages which wrap eggs. In other words, the rapid take-up of eggs is killing bdist_wininst as a standard distribution format, before a replacement is ready. 3. Easy_install hides this problem, to an extent. Because it *is* a competitor to system packagers. You say that easy_deb wraps easy_install in a Debian package - I don't see how that would work, and regardless, I can't envision an equivalent for Windows. Also, the same issue exists - it doesn't exist *yet*. This is all transition. Assuming that ultimately eggs are the standard, and system packager integration gets solved, there's no problem. I can wait. (If I couldn't wait, I'd be obliged to get off my backside and produce some code :-)) But I do think that projects should be aware of the trade-offs, and I *don't* think they've been clear up to now - that's why I'm writing this summary. One example, then I'll stop. I was recently considering having a look at Kid. I hadn't got round to it, but I now discover that Kid is only distributed as an egg. So I'll forget Kid for now, until the egg/system installer issues settle. Trying out Kid isn't important enough to me to mix two packaging systems. Fully-aware-that-I'm-a-minority-view-ly y'rs Paul. From smurf at smurf.noris.de Thu Nov 24 15:49:34 2005 From: smurf at smurf.noris.de (Matthias Urlichs) Date: Thu, 24 Nov 2005 15:49:34 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> References: <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> Message-ID: <20051124144933.GB20418@kiste.smurf.noris.de> Hi, Christopher Lenz: > (And no, I'm not going to repeat the numerous attempts by Phillip to > politely and comprehensively explain it all.) > Sorry -- I don't buy that. I've read all these messages too, and I also don't know what's in the metadata besides dependency information. Debian, rpm, and other Linux packaging systems don't need to install Python-specific dependency information in their packages. It's probably nice to have so that non-system Python packages can complain properly, instead of throwing an incomprehensible exception later (and leaving one with a partially-installed system or whatever), but that's a different problem. > When Debian users want to install and use those systems, you're gonna > gave to tell them why eggs aren't supported. > It seems that the next version of the .egg builder _will_ support a mode that works for Debian packaging, which is *great*, so ... -- Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf at smurf.noris.de Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de - - Order is the sanity of the mind, the health of the body, the peace of the city, the security of the state. As the beams to a house, as the bones to the microcosm of man, so is order to all things. -- Southey -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/distutils-sig/attachments/20051124/0c9e6b4e/attachment.pgp From pje at telecommunity.com Thu Nov 24 16:14:04 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 24 Nov 2005 10:14:04 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1132836242.407.25.camel@silicium.ccc.cea.fr> References: <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <438378C1.4050401@colorstudy.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> At 01:44 PM 11/24/2005 +0100, Josselin Mouette wrote: >They only introduce more complexity, instead of bringing real features. Please read the hundreds of kilobytes of messages I've already posted on this thread, and then when you're done, read these much shorter bits of documentation for some examples of the features they provide: http://peak.telecommunity.com/DevCenter/setuptools#dynamic-discovery-of-services-and-plugins http://peak.telecommunity.com/DevCenter/setuptools#defining-additional-metadata http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages http://peak.telecommunity.com/DevCenter/PkgResources#entry-points http://peak.telecommunity.com/DevCenter/PkgResources#metadata-api These are of course just the documentation for features that developers use, in order to create features of their own. I know that Ian's SQLObject project adds SQL-related metadata to projects using these APIs, and a variety of projects use entry points. The Chandler PIM will probably use project metadata for i18n stuff. > > Obviously, every individual distribution would like to have Python > packages > > conform to their individual system. However, on the whole, it is clearly > > better for the Python developer to have practical dependency management > > that doesn't tie their efforts to a single platform, packaging system, or > > distribution. > >And that's why there are things like dh_python to adapt python distutils >installations to be compliant to the Debian way of things. However, eggs >make it so complicated that the adaptation layer would be >unmaintainable. As I said, you should probably read the hundreds of K already written in this thread before jumping to conclusions like that. From joss at debian.org Thu Nov 24 16:17:23 2005 From: joss at debian.org (Josselin Mouette) Date: Thu, 24 Nov 2005 16:17:23 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> References: <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> Message-ID: <1132845443.12478.35.camel@silicium.ccc.cea.fr> Le jeudi 24 novembre 2005 ? 15:19 +0100, Christopher Lenz a ?crit : > > You are advertising this metadata a lot, but what does it exactly > > contain? If it's for dependencies, we can really live better without > > them, or with a tool to convert them into .deb dependencies. > > You know, it is really unfair to make a statement like "The problem > is the added complexity for no real gain" if you're gonna ask the > above question in a follow-up message. You clearly don't understand > the purpose of Python eggs, yet you feel qualified to judge whether > or not it provides any benefit. Sigh. > > (And no, I'm not going to repeat the numerous attempts by Phillip to > politely and comprehensively explain it all.) Maybe you should ask yourself the same question the other way around. How can it be possible, after reading all messages from Phillip to the list - and I haven't omitted a single one - I have no idea of what benefits can eggs bring? Let me repeat the question: what useful information is there in the metadata? What *problem* are eggs exactly trying to solve? > In the end, what matters is that Python eggs are being quickly > adopted by Python application developers, because they allow us to > easily build extensible systems. Indeed, it is nice to developers, but it entirely forgets system integrators. > When Debian users want to install > and use those systems, you're gonna gave to tell them why eggs aren't > supported. There's no reason to forbid people to install eggs in /usr/local or in their home directories. After all, they are suitable for that. What I'm standing against is shipping eggs in Debian packages. Regards, -- .''`. Josselin Mouette /\./\ : :' : josselin.mouette at ens-lyon.org `. `' joss at debian.org `- Debian GNU/Linux -- The power of freedom From joss at debian.org Thu Nov 24 16:20:47 2005 From: joss at debian.org (Josselin Mouette) Date: Thu, 24 Nov 2005 16:20:47 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> References: <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <438378C1.4050401@colorstudy.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> Message-ID: <1132845647.12478.39.camel@silicium.ccc.cea.fr> Le jeudi 24 novembre 2005 ? 10:14 -0500, Phillip J. Eby a ?crit : > At 01:44 PM 11/24/2005 +0100, Josselin Mouette wrote: > >They only introduce more complexity, instead of bringing real features. > > Please read the hundreds of kilobytes of messages I've already posted on > this thread I've read them. All of them. You're only babbling about how eggs are great, instead of explaining what problem they are trying to solve. -- .''`. Josselin Mouette /\./\ : :' : josselin.mouette at ens-lyon.org `. `' joss at debian.org `- Debian GNU/Linux -- The power of freedom From joss at debian.org Thu Nov 24 16:50:28 2005 From: joss at debian.org (Josselin Mouette) Date: Thu, 24 Nov 2005 16:50:28 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> References: <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <438378C1.4050401@colorstudy.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> Message-ID: <1132847428.12478.51.camel@silicium.ccc.cea.fr> Alright, now to the lots of (unclear) documentation you pointed me to. Le jeudi 24 novembre 2005 ? 10:14 -0500, Phillip J. Eby a ?crit : > http://peak.telecommunity.com/DevCenter/setuptools#dynamic-discovery-of-services-and-plugins > > http://peak.telecommunity.com/DevCenter/setuptools#defining-additional-metadata I fail to see why this feature needs eggs. A generic plugin system is definitely a good idea, but integrating it with the way python modules are provided is a bad one. Other developers didn't wait for eggs to develop elegant plugin systems that don't require any kind of extra metadata. > http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages This is definitely not a feature. There's no way retrieving a module from several locations should be possible, at least in the distribution-provided packages. > http://peak.telecommunity.com/DevCenter/PkgResources#entry-points I don't even understand what this is trying to achieve. > http://peak.telecommunity.com/DevCenter/PkgResources#metadata-api Generic metadata? Why isn't it put in files lying in the /usr/share tree? I don't understand why it has to come with the python files themselves. -- .''`. Josselin Mouette /\./\ : :' : josselin.mouette at ens-lyon.org `. `' joss at debian.org `- Debian GNU/Linux -- The power of freedom From schwehr at gmail.com Thu Nov 24 17:10:59 2005 From: schwehr at gmail.com (Kurt Schwehr) Date: Thu, 24 Nov 2005 08:10:59 -0800 Subject: [Distutils] formencode as .egg in Debian ?? Message-ID: <37de72610511240810p2cca4cf4v9560a748fe8cb918@mail.gmail.com> As a person who works as a packager with a debian based distribution (fink on Mac OSX) I am glad to see this discussion. I have no personal opinion about python eggs, but would really like to find a way to get some nice packages into the system and have been totally unable to figure out how to deal with eggs. I see the point of allowing developers to have multiple versions of packages when testing, but we definitely want to avoid that in the actual fink tree at all costs. It would be great to have a build system that can switch between what we need as stable distribution packagers and what people want/need in their local trees. My hope is that eggs in the long run can help the packagers track dependencies, but still let the packagers handle those dependencies. -kurt http://schwehr.org/blog -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20051124/46063ebf/attachment.html From pje at telecommunity.com Thu Nov 24 17:43:29 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 24 Nov 2005 11:43:29 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <20051124144933.GB20418@kiste.smurf.noris.de> References: <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> Message-ID: <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> At 03:49 PM 11/24/2005 +0100, Matthias Urlichs wrote: >Hi, > >Christopher Lenz: > > (And no, I'm not going to repeat the numerous attempts by Phillip to > > politely and comprehensively explain it all.) > > >Sorry -- I don't buy that. I've read all these messages too, and I also >don't know what's in the metadata besides dependency information. Whatever the project developer wants to put there, or wants people creating projects that use his project to put there. It's essentially an inter-project communication and configuration system: http://peak.telecommunity.com/DevCenter/setuptools#dynamic-discovery-of-services-and-plugins http://peak.telecommunity.com/DevCenter/setuptools#defining-additional-metadata http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages http://peak.telecommunity.com/DevCenter/PkgResources#entry-points http://peak.telecommunity.com/DevCenter/PkgResources#metadata-api These features are already being used by Trac, TurboGears, SQLObject, Paste, and others. Other projects are contemplating moving to the use of egg entry points to improve their extensibility, or increasing the use that they currently make of these features. >Debian, rpm, and other Linux packaging systems don't need to install >Python-specific dependency information in their packages. > >It's probably nice to have so that non-system Python packages can complain >properly, instead of throwing an incomprehensible exception later (and >leaving one with a partially-installed system or whatever), but that's a >different problem. That's an interesting perspective, but it's viewing the world through vendor-colored glasses. Unless the project developer is wearing similar glasses (i.e., has decided to commit to Debian as their sole platform), though, it's not a practical one. From the point of view of people like the author of TurboGears, it's the egg dependency system that allows them not to have to worry about which packaging system the user has, or doesn't have. I mention this not to be disagreeable, just to point out that the world in which egg dependencies are of no benefit, needless complexity, etc., is not the same world in which the project developers using eggs live. Case in point: this thread began because somebody wanted to package TurboGears and its dependencies for Debian. But that project wouldn't have been viable without the egg system already existing, and there was certainly no way for TurboGears to have started its life as anything but a "non-system Python package". One reason TurboGears is popular is because it's well-supported, and it's well-supported in part because it can "complain properly" (as you describe it above) no matter what platform it's running on. Another reason TurboGears is popular because of its high degree of reuse of existing code. Indeed, TurboGears has an unprecedented degree of reuse of code for a Python project. How many Debian Python packages have even *half* as many Python dependencies as TurboGears does? Meanwhile, TurboGears is viral with respect to eggs. When you create new web applications with it, those projects are also eggs, with dependencies on TurboGears. And of course people are creating TurboGears add-ons, like the identity package, and those have dependencies to TurboGears and other eggs. Even if it were *only* TurboGears that was doing these things (and it's not), you're going to have an "egg-splosion" on your hands here pretty soon. ("Pretty soon" == ~1 year, assuming the current growth rate of eggs is linear rather than exponential.) Again, I don't think you're "wrong" to think that egg dependencies are redundant - from within your particular point of view. But you need to understand that for *Python* developers, being able to practically depend on other packages in a cross-platform way is a new and powerful feature which is *not* provided by Debian or any other packaging system, unless it's wrapping eggs. So, from your perspective, you already have this feature, but for projects using eggs, you really *don't* have the feature, because the data is not economically accessible to them. And that's only *one* feature of eggs, not even getting into each new egg-specific feature that each project can create, building on the entry point and metadata features. Of course, those features have a different growth curve for adoption than the dependency features do, since plugin features are more of a hub-and-spokes model with limited network effects. And an individual plugin project is less likely to need to end up as a Debian package anyway, so the plugin impact will be more slow-moving unless somebody puts out a "killer app" that everybody wants plugins for. Anyway, I appreciate your frank and open-minded participation in this discussion, and I hope that together we can get the remaining issues resolved. From pje at telecommunity.com Thu Nov 24 17:46:58 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 24 Nov 2005 11:46:58 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1132845647.12478.39.camel@silicium.ccc.cea.fr> References: <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <438378C1.4050401@colorstudy.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> At 04:20 PM 11/24/2005 +0100, Josselin Mouette wrote: >Le jeudi 24 novembre 2005 ? 10:14 -0500, Phillip J. Eby a ?crit : > > At 01:44 PM 11/24/2005 +0100, Josselin Mouette wrote: > > >They only introduce more complexity, instead of bringing real features. > > > > Please read the hundreds of kilobytes of messages I've already posted on > > this thread > >I've read them. All of them. You're only babbling about how eggs are >great, instead of explaining what problem they are trying to solve. No, what's happening is that you're not paying attention, because you believe that Debian already solves those problems, even though it doesn't solve them for Python developers who want their projects to be usable on every platform. You're also ignoring my repeated explanations about how plugins work, and why Debian doesn't help with that problem either. And finally, and most importantly, you're ignoring the fact that this discussion began because a Debian developer wanted to package a successful egg-using project and its dependencies. Nobody came to you asking for your blessing to package eggs or your approval of how useful eggs are; somebody came and asked me for help in getting this to work with Debian. Help which I've been giving. For some reason, however, you seem to think that you are actually in a position to pass judgment on the usefulness of eggs. If their usefulness were in any meaningful question, we wouldn't be having this conversation in the first place! I've been answering Martin's posts to kill FUD and to make progress where progress could be made. But your posts aren't spreading any FUD I haven't already answered, and you're clearly not interested in making any progress, so I don't see any point in wasting either of our time any further on this thread. Nonetheless, I remain committed to helping any system integrator - Debian included - in helping their users by allowing them to satisfy egg dependencies using their existing packages, as well as in packaging egg-based projects in ways that work, not only for the project, but for the system integrator. In other words, setuptools 0.6a9 will have the features that at least Martin and Matthias have suggested will be an acceptable solution for packaging projects that explicitly contain egg metadata (such as FormEncode), and will also allow older Python packages to add dependency metadata with a single empty file. While I'm not a party to any Debian policy decisions regarding whether these features will be used for which packages and when, I will nonetheless be happy to continue offering technical input on these matters upon request. Luckily, the overall discussion has been productive despite the rancor. I've learned quite a lot that will be useful. From joss at debian.org Thu Nov 24 18:03:44 2005 From: joss at debian.org (Josselin Mouette) Date: Thu, 24 Nov 2005 18:03:44 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> References: <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <438378C1.4050401@colorstudy.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> Message-ID: <1132851824.12478.71.camel@silicium.ccc.cea.fr> Le jeudi 24 novembre 2005 ? 11:46 -0500, Phillip J. Eby a ?crit : > No, what's happening is that you're not paying attention, because you > believe that Debian already solves those problems, even though it doesn't > solve them for Python developers who want their projects to be usable on > every platform. While solving a "problem" (which problem?) for other systems, you've just created a new problem for all Linux distributions. > You're also ignoring my repeated explanations about how > plugins work, and why Debian doesn't help with that problem either. You haven't given any useful explanation so far. > And finally, and most importantly, you're ignoring the fact that this > discussion began because a Debian developer wanted to package a successful > egg-using project and its dependencies. Nobody came to you asking for your > blessing to package eggs or your approval of how useful eggs are; somebody > came and asked me for help in getting this to work with Debian. Help which > I've been giving. My preoccupation is to have a clean integration of all python packages in Debian. Something you've not been helpful about, proposing to change the clean, correct way python modules are packaged so that they can deal with your "improvements". > For some reason, however, you seem to think that you are actually in a > position to pass judgment on the usefulness of eggs. If their usefulness > were in any meaningful question, we wouldn't be having this conversation in > the first place! You have to bear with it. In the open source world, anyone can come and discuss the usefulness of your project. This won't happen if your project isn't causing trouble to other projects, but when you're talking about breaking the python packaging practises in Debian, I think any Debian developer has the moral right to discuss your suggestions. I have seen the nightmare that mozilla extensions packaging can become, and I urge maintainers of python-related packages not to rush into egg packaging, as it is obviously going to reach a similar level of insanity. > I've been answering Martin's posts to kill FUD and to make progress where > progress could be made. But your posts aren't spreading any FUD I haven't > already answered, and you're clearly not interested in making any progress, > so I don't see any point in wasting either of our time any further on this > thread. Nonetheless, I remain committed to helping any system integrator - > Debian included - in helping their users by allowing them to satisfy egg > dependencies using their existing packages, as well as in packaging > egg-based projects in ways that work, not only for the project, but for the > system integrator. You should demonstrate that will, instead of repeatedly proposing suggestions that are simply not implementable. > In other words, setuptools 0.6a9 will have the features that at least > Martin and Matthias have suggested will be an acceptable solution for > packaging projects that explicitly contain egg metadata (such as > FormEncode), and will also allow older Python packages to add dependency > metadata with a single empty file. Why the hell would a package having nothing to do with yours would have to add a file - especially an empty file - to cope with it? Again, intra-package dependencies is *not* something useful for us, distributors. They have repeatedly proved to be unreliable, in many contexts. Regards, -- .''`. Josselin Mouette /\./\ : :' : josselin.mouette at ens-lyon.org `. `' joss at debian.org `- Debian GNU/Linux -- The power of freedom From joss at debian.org Thu Nov 24 18:13:01 2005 From: joss at debian.org (Josselin Mouette) Date: Thu, 24 Nov 2005 18:13:01 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> References: <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> Message-ID: <1132852381.12478.80.camel@silicium.ccc.cea.fr> Le jeudi 24 novembre 2005 ? 11:43 -0500, Phillip J. Eby a ?crit : > That's an interesting perspective, but it's viewing the world through > vendor-colored glasses. Unless the project developer is wearing similar > glasses (i.e., has decided to commit to Debian as their sole platform), > though, it's not a practical one. From the point of view of people like > the author of TurboGears, it's the egg dependency system that allows them > not to have to worry about which packaging system the user has, or doesn't > have. > > I mention this not to be disagreeable, just to point out that the world in > which egg dependencies are of no benefit, needless complexity, etc., is not > the same world in which the project developers using eggs live. As I understand your explanation, developers who are using eggs are doing it so simplify *their* work, without a thought for users. Tools that simplify development processes are good and should be encouraged, but not if it means extra burden for users. One of the primary priorities of the Debian project is its users. Obviously this isn't yours. > Case in point: this thread began because somebody wanted to package > TurboGears and its dependencies for Debian. But that project wouldn't have > been viable without the egg system already existing, and there was > certainly no way for TurboGears to have started its life as anything but a > "non-system Python package". One reason TurboGears is popular is because > it's well-supported, and it's well-supported in part because it can > "complain properly" (as you describe it above) no matter what platform it's > running on. Again, the ability to distribute it as a single package is good, but it has been done by breaking existing practise for python packages distribution. I have nothing against eggs, I have something against being forced to use eggs to distribute a python project. > Again, I don't think you're "wrong" to think that egg dependencies are > redundant - from within your particular point of view. But you need to > understand that for *Python* developers, being able to practically depend > on other packages in a cross-platform way is a new and powerful feature > which is *not* provided by Debian or any other packaging system, unless > it's wrapping eggs. So, from your perspective, you already have this > feature, but for projects using eggs, you really *don't* have the feature, > because the data is not economically accessible to them. Yes, having that information is good. But there should be a way to ignore it. Simply. Nothing more, nothing less. If egg-enabled packages can also work without all this extra stuff, there's no problem for the distributor. Regards, -- .''`. Josselin Mouette /\./\ : :' : josselin.mouette at ens-lyon.org `. `' joss at debian.org `- Debian GNU/Linux -- The power of freedom From pje at telecommunity.com Thu Nov 24 18:49:59 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 24 Nov 2005 12:49:59 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1132851824.12478.71.camel@silicium.ccc.cea.fr> References: <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <438378C1.4050401@colorstudy.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051124122133.02e624a0@mail.telecommunity.com> At 06:03 PM 11/24/2005 +0100, Josselin Mouette wrote: >Le jeudi 24 novembre 2005 ? 11:46 -0500, Phillip J. Eby a ?crit : > > And finally, and most importantly, you're ignoring the fact that this > > discussion began because a Debian developer wanted to package a successful > > egg-using project and its dependencies. Nobody came to you asking for > your > > blessing to package eggs or your approval of how useful eggs are; somebody > > came and asked me for help in getting this to work with Debian. Help > which > > I've been giving. > > > For some reason, however, you seem to think that you are actually in a > > position to pass judgment on the usefulness of eggs. If their usefulness > > were in any meaningful question, we wouldn't be having this > conversation in > > the first place! > >You have to bear with it. In the open source world, anyone can come and >discuss the usefulness of your project. This won't happen if your >project isn't causing trouble to other projects, but when you're talking >about breaking the python packaging practises in Debian, I think any >Debian developer has the moral right to discuss your suggestions. My point is that I am not the one who just dropped by and asked somebody to change their practices. I was off doing other things and was pulled into this discussion by someone asking for assistance. Every proposal I've made in this thread has been directed at providing that assistance. Your dislike of or disagreement with any particular proposal doesn't excuse your attitude. I don't really care if you accept the proposals or not; you guys need to do whatever you think is best for Debian. I've only tried to educate you about your options regarding eggs, framed within the assumption that you *want* to distribute egg-based projects. The question of eggs' goodness or lack thereof is entirely irrelevant within that frame. Either you want to distribute those projects or you don't. If you don't, then this discussion is pointless. If you do, then arguing that people shouldn't use eggs is equally pointless. The only discussion that has any point is *how* we can get the needs of both Debian and the project developers met. >Why the hell would a package having nothing to do with yours would have >to add a file - especially an empty file - to cope with it? Geez, you make this sound like I just wandered in off the street and started telling people to redesign their packaging system. Somebody asked for help in Debianizing TurboGears. If you want to Debianize TurboGears, you'll have to add such a file to at least ElementTree's Debian package. The proposal I made is my attempt at offering a solution that doesn't require you to completely repackage ElementTree, while still allowing it to be used by TurboGears. Bluntly put, it's Debian that's asking me to change my practices (and those of egg-based project authors) to suit their needs, not the other way around. You seem to be under some delusion that we need you or want you to support those projects, when it's actually the existence of eggs that allows us to not care whether *you* exist. We wouldn't be having this discussion if somebody from your group hadn't asked for *our* help, so that Debian can play in *our* sandbox. All this discussion about the relative elegance of different packaging mechanisms is irrelevant. From pje at telecommunity.com Thu Nov 24 19:05:14 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 24 Nov 2005 13:05:14 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1132852381.12478.80.camel@silicium.ccc.cea.fr> References: <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051124125005.02e5b620@mail.telecommunity.com> At 06:13 PM 11/24/2005 +0100, Josselin Mouette wrote: >Le jeudi 24 novembre 2005 ? 11:43 -0500, Phillip J. Eby a ?crit : > > That's an interesting perspective, but it's viewing the world through > > vendor-colored glasses. Unless the project developer is wearing similar > > glasses (i.e., has decided to commit to Debian as their sole platform), > > though, it's not a practical one. From the point of view of people like > > the author of TurboGears, it's the egg dependency system that allows them > > not to have to worry about which packaging system the user has, or doesn't > > have. > > > > I mention this not to be disagreeable, just to point out that the world in > > which egg dependencies are of no benefit, needless complexity, etc., is > not > > the same world in which the project developers using eggs live. > >As I understand your explanation, developers who are using eggs are >doing it so simplify *their* work, without a thought for users. Tools >that simplify development processes are good and should be encouraged, >but not if it means extra burden for users. One of the primary >priorities of the Debian project is its users. Obviously this isn't >yours. You seem to be confusing "users" with "Debian packagers" and "Debian users", which are a subset of "users" where these projects are concerned. TurboGears targets Mac and Windows as well as Linux, and I've seen people on the TurboGears mailing list using at least three major packaging *tools* (e.g. .rpm, .deb, etc.), to say nothing of the count of Linux *distributions*. Obviously, Kevin *was* thinking of the users - eggs are the only thing that would let his project reach so many of them. > > Case in point: this thread began because somebody wanted to package > > TurboGears and its dependencies for Debian. But that project wouldn't > have > > been viable without the egg system already existing, and there was > > certainly no way for TurboGears to have started its life as anything but a > > "non-system Python package". One reason TurboGears is popular is because > > it's well-supported, and it's well-supported in part because it can > > "complain properly" (as you describe it above) no matter what platform > it's > > running on. > >Again, the ability to distribute it as a single package is good, What "single package" are you talking about? http://turbogears.org/download/ lists eggs and source packages for *10* different Python projects that it depends on, written by five different authors. Each is individually packaged, with eggs for Mac and Windows, and source packages that can build eggs for everything else. > > Again, I don't think you're "wrong" to think that egg dependencies are > > redundant - from within your particular point of view. But you need to > > understand that for *Python* developers, being able to practically depend > > on other packages in a cross-platform way is a new and powerful feature > > which is *not* provided by Debian or any other packaging system, unless > > it's wrapping eggs. So, from your perspective, you already have this > > feature, but for projects using eggs, you really *don't* have the feature, > > because the data is not economically accessible to them. > >Yes, having that information is good. But there should be a way to >ignore it. Simply. Nothing more, nothing less. If egg-enabled packages >can also work without all this extra stuff, there's no problem for the >distributor. What you're saying is, you want TurboGears to be able to blindly trust that its dependencies are installed. This doesn't help users, though, because it keeps the package author from being able to provide *end-user support* without learning the ins and outs of every distro and packaging system, so he can tell the user what to run to find out if the dependencies are *really* met. The funny thing here is that just the .egg names *alone* have been wildly useful; when a user posts an error message to the TurboGears mailing list, you can tell right away exactly what the project versions are for every piece of code in the stack trace. It dramatically cuts down on the number of, "Do you have version X of Y?" questions. Being able to support users is good for users. Being able to provide lots of functionality using off-the-shelf libraries that have their own documentation is good for users, too. This is *all* about the users. From pje at telecommunity.com Thu Nov 24 19:23:08 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 24 Nov 2005 13:23:08 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <37de72610511240810p2cca4cf4v9560a748fe8cb918@mail.gmail.co m> Message-ID: <5.1.1.6.0.20051124131638.042cc950@mail.telecommunity.com> At 08:10 AM 11/24/2005 -0800, Kurt Schwehr wrote: >As a person who works as a packager with a debian based distribution (fink >on Mac OSX) I am glad to see this discussion. I have no personal opinion >about python eggs, but would really like to find a way to get some nice >packages into the system and have been totally unable to figure out how to >deal with eggs. At the moment, easy_deb appears to be the most immediately available option for creating debs that support all egg features, from both egg-based and non-egg-based Python packages. If I understand Vincenzo correctly, easy_deb includes tools to allow switching the active package versions, although I don't know how or if it relates to dpkg-divert and the "alternatives" system, as I'm not a Debian user myself. >I see the point of allowing developers to have multiple versions of >packages when testing, but we definitely want to avoid that in the actual >fink tree at all costs. It would be great to have a build system that can >switch between what we need as stable distribution packagers and what >people want/need in their local trees. My hope is that eggs in the long >run can help the packagers track dependencies, but still let the packagers >handle those dependencies. On Mac OS X, we recommend that users install to ~/Library/Python2.x/site-packages, as this is private to the user and already supports .pth files. That way, they can switch between versions at will. See also: http://peak.telecommunity.com/DevCenter/EasyInstall#non-root-installation I would be happy to answer any questions or hear any concerns you may have about how any of this works or how you'd like it to. Thanks. From joss at debian.org Thu Nov 24 19:27:40 2005 From: joss at debian.org (Josselin Mouette) Date: Thu, 24 Nov 2005 19:27:40 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051124122133.02e624a0@mail.telecommunity.com> References: <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <438378C1.4050401@colorstudy.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> <5.1.1.6.0.20051124122133.02e624a0@mail.telecommunity.com> Message-ID: <1132856860.5916.17.camel@arrakis.localnet> Le jeudi 24 novembre 2005 ? 12:49 -0500, Phillip J. Eby a ?crit : > I don't really care if you accept the proposals or not; you guys need to do > whatever you think is best for Debian. I've only tried to educate you > about your options regarding eggs, framed within the assumption that you > *want* to distribute egg-based projects. The question of eggs' goodness or > lack thereof is entirely irrelevant within that frame. Either you want to > distribute those projects or you don't. If you don't, then this discussion > is pointless. If you do, then arguing that people shouldn't use eggs is > equally pointless. The only discussion that has any point is *how* we can > get the needs of both Debian and the project developers met. Indeed. I'm sure no sane Debian developer would want to distribute egg-based projects if there was another way to distribute them, but they are here and people will want them. You are making things worse by advocating the egg way of doing things, and the only thing I can do if you don't stop is to urge developers to think twice before shipping their projects this way. Maybe it will make their development a bit more difficult, but it will improve the situation for their users. > Bluntly put, it's Debian that's asking me to change my practices (and those > of egg-based project authors) to suit their needs, not the other way > around. You seem to be under some delusion that we need you or want you to > support those projects, when it's actually the existence of eggs that > allows us to not care whether *you* exist. We wouldn't be having this > discussion if somebody from your group hadn't asked for *our* help, so that > Debian can play in *our* sandbox. All this discussion about the relative > elegance of different packaging mechanisms is irrelevant. It seems that you can only think about things in a competitive way. "Use our stuff or die.", or "Change your way of doing things or you'll die.". Looks like I mistakenly hoped it was possible to change things on technical grounds, but if you're rebutting any technical discussion as "irrelevant", there's probably not much to do. I know what will happen now: people will package your crappy eggs, because users want the software that's inside. Packages will be crappy and users will complain, and you'll tell them their distribution sucks. And what do we get? Nothing interesting. If you don't want to do anything for users, it's pointless to try to discuss with you. My tone is aggressive because it's easy to feel irritated by so much foolishness, but on the grounds, this is still a request: please, change your distribution system so that it's possible to distribute your software in a sane way. Don't become like Mozilla. You can't imagine how much Debian time is wasted because Mozilla developers don't care about the Unix way of doing things. We are receiving weekly complaints of users about the quality of mozilla packages, and there's nothing we can do for them. Don't make python-related Debian maintainership look the same. Regards, -- .''`. Josselin Mouette /\./\ : :' : josselin.mouette at ens-lyon.org `. `' joss at debian.org `- Debian GNU/Linux -- The power of freedom -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message =?ISO-8859-1?Q?num=E9riquement?= =?ISO-8859-1?Q?_sign=E9e?= Url : http://mail.python.org/pipermail/distutils-sig/attachments/20051124/d33fe6d6/attachment.pgp From joss at debian.org Thu Nov 24 19:38:51 2005 From: joss at debian.org (Josselin Mouette) Date: Thu, 24 Nov 2005 19:38:51 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051124125005.02e5b620@mail.telecommunity.com> References: <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <5.1.1.6.0.20051124125005.02e5b620@mail.telecommunity.com> Message-ID: <1132857531.5916.28.camel@arrakis.localnet> Le jeudi 24 novembre 2005 ? 13:05 -0500, Phillip J. Eby a ?crit : > You seem to be confusing "users" with "Debian packagers" and "Debian > users", which are a subset of "users" where these projects are > concerned. TurboGears targets Mac and Windows as well as Linux, and I've > seen people on the TurboGears mailing list using at least three major > packaging *tools* (e.g. .rpm, .deb, etc.), to say nothing of the count of > Linux *distributions*. Obviously, Kevin *was* thinking of the users - eggs > are the only thing that would let his project reach so many of them. Of course, I'm talking about Unix users. You seem to have already taken care of Windows users. > >Again, the ability to distribute it as a single package is good, > > What "single package" are you talking > about? http://turbogears.org/download/ lists eggs and source packages for > *10* different Python projects that it depends on, written by five > different authors. Each is individually packaged, with eggs for Mac and > Windows, and source packages that can build eggs for everything else. You're reinventing the wheel. Really. This isn't a matter for the Windows world, as users are accustomed to various, broken tools to install stuff, and this one will probably be less broken than others. But MacOS and Linux distributions already have a unified packaging system, and I don't think it's useful to invent another package type, especially if it's only suitable for Python. > >Yes, having that information is good. But there should be a way to > >ignore it. Simply. Nothing more, nothing less. If egg-enabled packages > >can also work without all this extra stuff, there's no problem for the > >distributor. > > What you're saying is, you want TurboGears to be able to blindly trust that > its dependencies are installed. YES. Please. > This doesn't help users, though, because > it keeps the package author from being able to provide *end-user support* > without learning the ins and outs of every distro and packaging system, so > he can tell the user what to run to find out if the dependencies are > *really* met. No. Distributions all have their own bug-reporting tool, so that distribution-related issues can be filtered out before such issues are brought to you. > The funny thing here is that just the .egg names *alone* have been wildly > useful; when a user posts an error message to the TurboGears mailing list, > you can tell right away exactly what the project versions are for every > piece of code in the stack trace. It dramatically cuts down on the number > of, "Do you have version X of Y?" questions. Same answer: at least the Debian bug reporting tool does it. We are not trying to make your work harder, only to make our work easier. > Being able to support users is good for users. Being able to provide lots > of functionality using off-the-shelf libraries that have their own > documentation is good for users, too. This is *all* about the users. And making security bugs unfixable is good for users? We probably don't have the same users, indeed. -- .''`. Josselin Mouette /\./\ : :' : josselin.mouette at ens-lyon.org `. `' joss at debian.org `- Debian GNU/Linux -- The power of freedom -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message =?ISO-8859-1?Q?num=E9riquement?= =?ISO-8859-1?Q?_sign=E9e?= Url : http://mail.python.org/pipermail/distutils-sig/attachments/20051124/c23df812/attachment.pgp From pje at telecommunity.com Thu Nov 24 20:13:09 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 24 Nov 2005 14:13:09 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1132856860.5916.17.camel@arrakis.localnet> References: <5.1.1.6.0.20051124122133.02e624a0@mail.telecommunity.com> <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <438378C1.4050401@colorstudy.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> <5.1.1.6.0.20051124122133.02e624a0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051124134908.01fa4328@mail.telecommunity.com> At 07:27 PM 11/24/2005 +0100, Josselin Mouette wrote: >Looks like I mistakenly hoped it was possible to change things on >technical grounds, but if you're rebutting any technical discussion as >"irrelevant", there's probably not much to do. On the contrary, quite useful technical discussion about how to make this work has taken place, including proposed changes *which I have accepted* and will implement. You just haven't been participating in any of that discussion. >My tone is aggressive because it's easy to feel irritated by so much >foolishness, but on the grounds, this is still a request: please, change >your distribution system so that it's possible to distribute your >software in a sane way. Please help me do that by explaining "sane way" on grounds that don't resort to religion. Please explain why including a package's *own metadata* that the author *wished to include* is not sane. Please explain why providing a consistent way (i.e. a cross-distribution, cross-platform way) to identify what Python libraries are installed on a system is not "sane". Note, by the way, that those two things are the only essentials here, as best I can tell, and I've already stated my willingness to change *how* those two things get accomplished. For clarity, I will repeat yet again, in yet another way: 1. Egg-based projects need to install their published metadata, in a well-known location relative to the installation location of their code, so that it can be found by searching sys.path, so that it and other projects can locate the metadata for currently-importable projects, *without* needing to first import the project's code. 2. Egg-based projects need to be able to identify whether another Python project package is installed and what version it is, without requiring modification to that other project's code or needing to import it. (And this is independent of whether the depended-on project was packaged as an egg by its author.) As far as I'm aware, those are the irreducible technical minimum requirements for making an egg-based project work. *How* these requirements are met is quite flexible, as there are already three working layouts that achieve this. As I said before, I'm quite willing to implement a fourth. But nobody has been proposing anything that meets these requirements, because they're too busy trying to prove the requirements don't exist or are somehow not real. >Don't become like Mozilla. You can't imagine how >much Debian time is wasted because Mozilla developers don't care about >the Unix way of doing things. We are receiving weekly complaints of >users about the quality of mozilla packages, and there's nothing we can >do for them. Don't make python-related Debian maintainership look the >same. Since I'm neither a Debian nor Mozilla developer, I have no way to know what these problems you speak of are, nor any way to tell whether the alleged flaws of the Mozilla packaging relate in any meaningful way to the alleged flaws of eggs. It's you who's in the position of giving advocacy without providing any real information about the issues. It's plain that you don't want crappy packaging. But I've only been getting the barest hint of what "crappy packaging" consists of, except for the loud-and-clear message that it's defined as Anything But Debian. Since I'm providing for users beyond Debian, that isn't useful information and doesn't help me investigate fixing any alleged crappiness. It would be especially helpful to define what *specifically* are the problems, rather than continuing all this vague handwaving about elegance and sanity and the unspecified crappiness of all things non-Debian. So far, every argument except startup performance and the length of sys.path has seemed to boil down to, "we don't like it so you should stop," and "we don't trust package developers to specify their own dependencies, so anything that allows it is evil". Those aren't useful arguments, because there is nothing I can *do* about them. From cmlenz at gmx.de Thu Nov 24 20:37:38 2005 From: cmlenz at gmx.de (Christopher Lenz) Date: Thu, 24 Nov 2005 20:37:38 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1132856860.5916.17.camel@arrakis.localnet> References: <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <438378C1.4050401@colorstudy.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> <5.1.1.6.0.20051124122133.02e624a0@mail.telecommunity.com> <1132856860.5916.17.camel@arrakis.localnet> Message-ID: <C6E8E17D-627C-4457-B455-2205C3382ECF@gmx.de> Am 24.11.2005 um 19:27 schrieb Josselin Mouette: > Le jeudi 24 novembre 2005 ? 12:49 -0500, Phillip J. Eby a ?crit : >> I don't really care if you accept the proposals or not; you guys >> need to do >> whatever you think is best for Debian. I've only tried to educate >> you >> about your options regarding eggs, framed within the assumption >> that you >> *want* to distribute egg-based projects. The question of eggs' >> goodness or >> lack thereof is entirely irrelevant within that frame. Either you >> want to >> distribute those projects or you don't. If you don't, then this >> discussion >> is pointless. If you do, then arguing that people shouldn't use >> eggs is >> equally pointless. The only discussion that has any point is >> *how* we can >> get the needs of both Debian and the project developers met. > > Indeed. I'm sure no sane Debian developer would want to distribute > egg-based projects if there was another way to distribute them, but > they > are here and people will want them. You are making things worse by > advocating the egg way of doing things, and the only thing I can do if > you don't stop is to urge developers to think twice before shipping > their projects this way. Maybe it will make their development a bit > more > difficult, but it will improve the situation for their users. Eggs are not really a way to *ship* Python projects. It is a way to deploy Python projects so that they can provide discoverable meta- data at runtime. This meta-data can be used in many ways. An example: for the latest release of the Trac project management web-app we implemented a simple plugin system that is based on Python eggs. It uses the egg meta-data to discover other Python projects that provide plugins for itself. Thus, this meta-data is essential for using plugins in Trac, and if a package is installed without this meta-data, well, you could've just as well not installed it at all. Sure, every Python project that wanted to be extensible could just reinvent the wheel here and implement its own system for discovering and activating plugins. But Python eggs give us a unified solution to this problem, and a very nice one, too. The problems this approach may raise for package maintainers need to be discussed and addressed. But to simply say that eggs are a "crappy" technology is just ignorant. Cheers, Chris -- Christopher Lenz cmlenz at gmx.de http://www.cmlenz.net/ From robert.kern at gmail.com Thu Nov 24 20:36:29 2005 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 24 Nov 2005 11:36:29 -0800 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051124134908.01fa4328@mail.telecommunity.com> References: <5.1.1.6.0.20051124122133.02e624a0@mail.telecommunity.com> <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <438378C1.4050401@colorstudy.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> <5.1.1.6.0.20051124122133.02e624a0@mail.telecommunity.com> <1132856860.5916.17.camel@arrakis.localnet> <5.1.1.6.0.20051124134908.01fa4328@mail.telecommunity.com> Message-ID: <dm54nu$s3g$1@sea.gmane.org> Phillip J. Eby wrote: > Note, by the way, that those two things are the only essentials here, as > best I can tell, and I've already stated my willingness to change *how* > those two things get accomplished. For clarity, I will repeat yet again, > in yet another way: > > 1. Egg-based projects need to install their published metadata, in a > well-known location relative to the installation location of their code, so > that it can be found by searching sys.path, so that it and other projects > can locate the metadata for currently-importable projects, *without* > needing to first import the project's code. > > 2. Egg-based projects need to be able to identify whether another Python > project package is installed and what version it is, without requiring > modification to that other project's code or needing to import it. (And > this is independent of whether the depended-on project was packaged as an > egg by its author.) > > As far as I'm aware, those are the irreducible technical minimum > requirements for making an egg-based project work. *How* these > requirements are met is quite flexible, as there are already three working > layouts that achieve this. As I said before, I'm quite willing to > implement a fourth. But nobody has been proposing anything that meets > these requirements, because they're too busy trying to prove the > requirements don't exist or are somehow not real. [Note: I am a happy Debian user, though not a DD. I am also one of the developers of a Debian-packaged Python package, and we're considering using pkg_resources to implement certain new features. I swear, this is like watching two parents fight. Anyways...] I think one of the sticking points with the Debian developers has been that the .egg-info metadata is being put into /usr/lib/... when according to Debian policy and general UNIX lore, such should be placed somewhere in /usr/share/.... Would it be possible to treat /usr/share/pythonX.Y-egginfo/ as a proxy for /usr/lib/pythonX.Y/site-packages/ when searching for .egg-info directories? -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From joss at debian.org Thu Nov 24 21:10:50 2005 From: joss at debian.org (Josselin Mouette) Date: Thu, 24 Nov 2005 21:10:50 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051124134908.01fa4328@mail.telecommunity.com> References: <5.1.1.6.0.20051124122133.02e624a0@mail.telecommunity.com> <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <438378C1.4050401@colorstudy.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> <5.1.1.6.0.20051124122133.02e624a0@mail.telecommunity.com> <5.1.1.6.0.20051124134908.01fa4328@mail.telecommunity.com> Message-ID: <1132863050.5916.79.camel@arrakis.localnet> Le jeudi 24 novembre 2005 ? 14:13 -0500, Phillip J. Eby a ?crit : > On the contrary, quite useful technical discussion about how to make this > work has taken place, including proposed changes *which I have accepted* > and will implement. You just haven't been participating in any of that > discussion. Indeed, I arrived in the discussion a bit late, and didn't want to intervene before having read it all. However, the implementation that you proposed isn't clear from those messages. Putting things out of the zip archives and having a separate file for metadata is obviously a good thing. However, I'm still concerned about things not functioning without this metadata, and the requirement for adding metadata in packages that don't need it. As I still don't get the point of including this metadata in the package, I'd have a hard time proposing other implementations. > >My tone is aggressive because it's easy to feel irritated by so much > >foolishness, but on the grounds, this is still a request: please, change > >your distribution system so that it's possible to distribute your > >software in a sane way. > > Please help me do that by explaining "sane way" on grounds that don't > resort to religion. Please explain why including a package's *own > metadata* that the author *wished to include* is not sane. Please explain > why providing a consistent way (i.e. a cross-distribution, cross-platform > way) to identify what Python libraries are installed on a system is not "sane". A sane way, first of all, means a consistent way. Having two sorts of Debian python packages is a no-go. Therefore, if we want to switch to a new way of distributing packages, there has to be some serious grounds for it. Currently, the picture I have of eggs doesn't show any advantages over the plain old distutils we are currently using. A sane way means, when there are meaningful files that are accessed by the interpreter in an easily understandable way, to ship them as is in a package, not to hide them in an archive. A sane way means not to alter functionality of other software. For example, impacting the overall python startup time by making it unzip all eggs is not sane, even if the impact is currently small. Modifying the default sys.path enters in the same category. A sane way means compliance with standards, especially the filesystem hierarchy standard. When some people are trying to separate .py and .pyc/.pyo files to respect it, you're asking to put them in a single archive. > 1. Egg-based projects need to install their published metadata, in a > well-known location relative to the installation location of their code, so > that it can be found by searching sys.path, so that it and other projects > can locate the metadata for currently-importable projects, *without* > needing to first import the project's code. > > 2. Egg-based projects need to be able to identify whether another Python > project package is installed and what version it is, without requiring > modification to that other project's code or needing to import it. (And > this is independent of whether the depended-on project was packaged as an > egg by its author.) > > As far as I'm aware, those are the irreducible technical minimum > requirements for making an egg-based project work. *How* these > requirements are met is quite flexible, as there are already three working > layouts that achieve this. As I said before, I'm quite willing to > implement a fourth. But nobody has been proposing anything that meets > these requirements, because they're too busy trying to prove the > requirements don't exist or are somehow not real. Alright, let me try to propose something. How about defining a place in the python distribution, say, /usr/share/python2.x/eggs (but of course configurable when installing python), where this information should go? An installed package would look like a directory structure of .py files in /usr/lib/python2.x/site-packages, and a metadata file in /usr/share/python2.x/eggs. In fact, this approach is quite similar to the .egg-info approach, but it would be better to put the files at the right place from the very beginning, and, more importantly, to be able to deal with other packages not having such .egg-info files. > Since I'm neither a Debian nor Mozilla developer, I have no way to know > what these problems you speak of are, nor any way to tell whether the > alleged flaws of the Mozilla packaging relate in any meaningful way to the > alleged flaws of eggs. It's you who's in the position of giving advocacy > without providing any real information about the issues. This is true. In short, my two main concerns about mozilla-related packaging are total absence of library versioning, and the shipping of extensions in .xpi files, which are tarballs containing information and meta-information about the package. You can surely understand that my concerns about eggs approach those of .xpi files. > It's plain that you don't want crappy packaging. But I've only been > getting the barest hint of what "crappy packaging" consists of, except for > the loud-and-clear message that it's defined as Anything But Debian. Sorry, but you're jumping to conclusions. Crappy packaging will be the same for all package-based distributions. Regards, -- .''`. Josselin Mouette /\./\ : :' : josselin.mouette at ens-lyon.org `. `' joss at debian.org `- Debian GNU/Linux -- The power of freedom -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message =?ISO-8859-1?Q?num=E9riquement?= =?ISO-8859-1?Q?_sign=E9e?= Url : http://mail.python.org/pipermail/distutils-sig/attachments/20051124/7b1fb991/attachment.pgp From pje at telecommunity.com Thu Nov 24 21:17:23 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 24 Nov 2005 15:17:23 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <dm54nu$s3g$1@sea.gmane.org> References: <5.1.1.6.0.20051124134908.01fa4328@mail.telecommunity.com> <5.1.1.6.0.20051124122133.02e624a0@mail.telecommunity.com> <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <438378C1.4050401@colorstudy.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> <5.1.1.6.0.20051124122133.02e624a0@mail.telecommunity.com> <1132856860.5916.17.camel@arrakis.localnet> <5.1.1.6.0.20051124134908.01fa4328@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051124144838.01f7b560@mail.telecommunity.com> At 11:36 AM 11/24/2005 -0800, Robert Kern wrote: >Phillip J. Eby wrote: > > > Note, by the way, that those two things are the only essentials here, as > > best I can tell, and I've already stated my willingness to change *how* > > those two things get accomplished. For clarity, I will repeat yet again, > > in yet another way: > > > > 1. Egg-based projects need to install their published metadata, in a > > well-known location relative to the installation location of their > code, so > > that it can be found by searching sys.path, so that it and other projects > > can locate the metadata for currently-importable projects, *without* > > needing to first import the project's code. > > > > 2. Egg-based projects need to be able to identify whether another Python > > project package is installed and what version it is, without requiring > > modification to that other project's code or needing to import it. (And > > this is independent of whether the depended-on project was packaged as an > > egg by its author.) > > > > As far as I'm aware, those are the irreducible technical minimum > > requirements for making an egg-based project work. *How* these > > requirements are met is quite flexible, as there are already three working > > layouts that achieve this. As I said before, I'm quite willing to > > implement a fourth. But nobody has been proposing anything that meets > > these requirements, because they're too busy trying to prove the > > requirements don't exist or are somehow not real. > >[Note: I am a happy Debian user, though not a DD. I am also one of the >developers of a Debian-packaged Python package, and we're considering >using pkg_resources to implement certain new features. I swear, this is >like watching two parents fight. Anyways...] > >I think one of the sticking points with the Debian developers has been >that the .egg-info metadata is being put into /usr/lib/... when >according to Debian policy and general UNIX lore, such should be placed >somewhere in /usr/share/.... Would it be possible to treat >/usr/share/pythonX.Y-egginfo/ as a proxy for >/usr/lib/pythonX.Y/site-packages/ when searching for .egg-info directories? The sticking point is the relative relationship to a sys.path directory (line 2 of requirement 1, above). Since a program such as Trac can legitimately manipulate sys.path to e.g. add plugin directories, we need to be able to do some reasonable transformation to *every* sys.path entry, preferably without introducing some kind of distro-specific configuration to match special directories and redirect them. We really want to treat every sys.path entry as a valid place to find a project's metadata, since each is a valid place for the actual packages to live. There is of course also a performance issue; currently we need only scan each sys.path directory at most once in order to locate all packages and their metadata. (.egg directories don't need to be scanned unless looking for a dependency that's not already on sys.path.) The scan of non-.egg directories identifies any .egg-info directories there. So, for a cost of zero or one listdir() for each sys.path entry, we can find the locations of all the discoverable metadata. However, if the issue here is where the data files actually live, couldn't Debian just put the egg info wherever it wants, and symlink to it from site-packages? That would already work with the current runtime, although it seems to me like it's just making more work for the Debian packagers, for no benefit that I comprehend. (Especially since in the case of non-setuptools packages, it might be just an empty file anyway.) Note that Python packages have always been able to include data files inside their package directories, and in fact the *single most popular* distutils modification made by published Python packages is to *not* install data under /usr/share and instead put those files in the packages' directories, where the package can find it without needing to know the platform-specific or distro-local place for data. This is why setuptools pioneered an easier way to add package data almost 2 years ago, and that way was officially backported into the distutils in Python 2.4. See: http://python.org/doc/2.4.1/dist/node11.html Anyway, symlinks would work, but it seems to me rather inconsistent to push project-level data to /usr/share while leaving module-level data in /usr/lib. And pushing either one of the two to /usr/share seems like wasted effort on a non-problem to me. In neither case is the data going to be changed; it can be just as read-only as the code is. (But just because I don't think it's useful, I'm not going to act like it's not a real problem for the Debian team, or insist that putting files in /usr/share is undesirable. If it's desirable to them, it's desirable to them. The only relevant question is how can we both get our desires met.) From robert.kern at gmail.com Thu Nov 24 21:39:21 2005 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 24 Nov 2005 12:39:21 -0800 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051124144838.01f7b560@mail.telecommunity.com> References: <5.1.1.6.0.20051124134908.01fa4328@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <438378C1.4050401@colorstudy.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> <5.1.1.6.0.20051124122133.02e624a0@mail.telecommunity.com> <1132856860.5916.17.camel@arrakis.localnet> <5.1.1.6.0.20051124134908.01fa4328@mail.telecommunity.com> <dm54nu$s3g$1@sea.gmane.org> <5.1.1.6.0.20051124144838.01f7b560@mail.telecommunity.com> Message-ID: <dm58dq$6j8$1@sea.gmane.org> Phillip J. Eby wrote: > At 11:36 AM 11/24/2005 -0800, Robert Kern wrote: >>I think one of the sticking points with the Debian developers has been >>that the .egg-info metadata is being put into /usr/lib/... when >>according to Debian policy and general UNIX lore, such should be placed >>somewhere in /usr/share/.... Would it be possible to treat >>/usr/share/pythonX.Y-egginfo/ as a proxy for >>/usr/lib/pythonX.Y/site-packages/ when searching for .egg-info directories? > > The sticking point is the relative relationship to a sys.path directory > (line 2 of requirement 1, above). Since a program such as Trac can > legitimately manipulate sys.path to e.g. add plugin directories, we need to > be able to do some reasonable transformation to *every* sys.path entry, > preferably without introducing some kind of distro-specific configuration > to match special directories and redirect them. We really want to treat > every sys.path entry as a valid place to find a project's metadata, since > each is a valid place for the actual packages to live. I'm not suggesting that /usr/share/.../ should be the only place to find .egg-info directories. Simply that pkg_resources would scan sys.path+['/usr/share/.../'] and treat the ones found in /usr/share/.../ as if they were in /usr/lib/pythonX.Y/site-packages/. Everywhere else that a user could put Python packages (e.g. /usr/local/lib/pythonX.Y/site-packages/) is fine to put whatever the user wants. Eggs, .egg-info/; it doesn't matter. It doesn't need an alternate location. Debian packages need to be more disciplined, but all Debianized Python packages are installed to /usr/lib/pythonX.Y/site-packages/ so we only need the one alias, not many. It's possible that only the DD in charge of packaging setuptools will have to concern himself with coding this. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From david at mantara.com Thu Nov 24 21:54:31 2005 From: david at mantara.com (David Arnold) Date: Fri, 25 Nov 2005 07:54:31 +1100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: Your message of "Thu, 24 Nov 2005 14:13:09 CDT." <5.1.1.6.0.20051124134908.01fa4328@mail.telecommunity.com> Message-ID: <E1EfO6x-0002J0-00@d.0x1.org> -->"Phillip" == Phillip J Eby <pje at telecommunity.com> writes: Phillip> I've only been getting the barest hint of what "crappy Phillip> packaging" consists of, except for the loud-and-clear message Phillip> that it's defined as Anything But Debian. Since I'm Phillip> providing for users beyond Debian, that isn't useful Phillip> information and doesn't help me investigate fixing any Phillip> alleged crappiness. >From here, the problem looks like: Eggs provide a way of managing dependencies that is generic to Python, across all platforms. From a Python developer or user's viewpoint, this is great. Debs (and RPMs, and ebuilds, and PKGs, and so on) provide a way of managing dependencies that are generic to a Unix distribution. From a Unix developer or user's viewpoint, this is great. For each of these groups, the alternative approach is problematic. >From a Python-centric viewpoint, Debian's (and RedHat, Gentoo, Solaris, etc) packaging mechanism, however great, covers only one of the many possible platforms that an application might need to support. >From a Debian-centric viewpoint, dependency management is a solved problem, with a decade of experience behind it, and proposing that Python applications should have their dependencies managed separately is blatantly foolhardy. I kinda have a foot in both camps: I use Debian as my desktop, but I produce Python apps and "libraries" for both Unix and Windows users. So far as I know, Windows users don't have a standard way to manage dependencies. And to be fair, Windows users are far more numerous than those of any Unix/Linux distribution. So I can see the appeal of eggs, both for a popular platform, and as a Python developer, as a cross-platform do-it-once-only, mechanism. But, as I've said (and I believe Phillip has acknowledged), from a Unix point of view, this is problematic. Tools to convert Python-centric, egg-managed dependency information into Debian/Red Hat/Gentoo/Solaris/etc dependency information would appear to help bridge this gap. I'm not convinced that's the case (I'm not sure how reasonable it is to automatically convert 'require("FooBar>=1.2")' into the equivalent Debian package names and versions, but my expectation is that it's no more difficult than it currently is to package a "product" for Debian. My point here is: both sides of this discussion have a valid point. If eggs are going to see the claimed rapid adoption (and I have no reason to doubt the claims that this is happening), then systems that have an existing dependency management system will need to cope. Assistance from the Egg People to make this happen can only be useful. Finally, this is ignoring other possible uses of egg metadata, which might end up being far more problematic? d From pje at telecommunity.com Thu Nov 24 22:03:10 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 24 Nov 2005 16:03:10 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1132863050.5916.79.camel@arrakis.localnet> References: <5.1.1.6.0.20051124134908.01fa4328@mail.telecommunity.com> <5.1.1.6.0.20051124122133.02e624a0@mail.telecommunity.com> <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <438378C1.4050401@colorstudy.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> <5.1.1.6.0.20051124122133.02e624a0@mail.telecommunity.com> <5.1.1.6.0.20051124134908.01fa4328@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051124151906.05214310@mail.telecommunity.com> At 09:10 PM 11/24/2005 +0100, Josselin Mouette wrote: >A sane way, first of all, means a consistent way. Having two sorts of >Debian python packages is a no-go. Therefore, if we want to switch to a >new way of distributing packages, there has to be some serious grounds >for it. Currently, the picture I have of eggs doesn't show any >advantages over the plain old distutils we are currently using. What advantage does the internet have? It lets you be connected to the internet. Eggs are a network, packaging them lets your packages be part of that network. That's the only *possible* benefit to eggs from a system packager's point of view, which is why it has never been my policy to go to system packagers and ask them to include eggs. Instead, I leave it to their users to ask, because it's really none of my business whether a distributor packages eggs. That's between you and your users. I only get involved if someone asks for help in packaging the eggs, and I help anybody who asks. Which is why I find this discussion so frustrating; somehow or other, people keep spinning this back to some kind of advocacy thing. If you want to provide eggs, great. You don't want to provide them, great. It really makes no difference to me; it's an issue between you and your users. I'm only here to help if you *do* want to provide eggs and have issues with *how* to do it in a way that works for you. [snip items already discussed that have technical solutions already arrived at] >A sane way means compliance with standards, especially the filesystem >hierarchy standard. When some people are trying to separate .py >and .pyc/.pyo files to respect it, you're asking to put them in a single >archive. PEP 304 has been languishing for years, apparently due to lack of interest. In any case, its motivation was never FHS conformance, as far as I can tell. More immediately relevant, I'm 1) not asking anybody to do anything, and 2) an archive is only one of three already-valid ways to lay out the installation of an egg. > > 1. Egg-based projects need to install their published metadata, in a > > well-known location relative to the installation location of their > code, so > > that it can be found by searching sys.path, so that it and other projects > > can locate the metadata for currently-importable projects, *without* > > needing to first import the project's code. > > > > 2. Egg-based projects need to be able to identify whether another Python > > project package is installed and what version it is, without requiring > > modification to that other project's code or needing to import it. (And > > this is independent of whether the depended-on project was packaged as an > > egg by its author.) > > > > As far as I'm aware, those are the irreducible technical minimum > > requirements for making an egg-based project work. *How* these > > requirements are met is quite flexible, as there are already three working > > layouts that achieve this. As I said before, I'm quite willing to > > implement a fourth. But nobody has been proposing anything that meets > > these requirements, because they're too busy trying to prove the > > requirements don't exist or are somehow not real. > >Alright, let me try to propose something. How about defining a place in >the python distribution, say, /usr/share/python2.x/eggs (but of course >configurable when installing python), where this information should go? >An installed package would look like a directory structure of .py files >in /usr/lib/python2.x/site-packages, and a metadata file >in /usr/share/python2.x/eggs. See line 2 of item 1 above; we need a "well-known location located relative to the installation location of their code". Python programs have limited access to knowledge about platform directory structure, except as expressed in sys.path. Also, they need to be able to support development environments, application plugin directories, and other sys.path manipulation. So, ideally the approach needs to be able to be applied uniformly to each sys.path entry. One of the reasons for this is that it allows the runtime to reason about the contents of sys.path. If the metadata is in a particular directory, it knows that by adding or removing that directory from sys.path, what packages will be added or removed from importability, and it also knows which packages may be shadowing other packages. In the case of namespace packages, it knows what subdirectories of sys.path directories need to be added to the package's in-memory __path__ in order to merge separately-distributed components into a common virtual package. These goals seem unachievable with an absolute location that's divorced from any direct connection to a sys.path entry, although there are a couple of workarounds possible. See below. >In fact, this approach is quite similar to the .egg-info approach, but >it would be better to put the files at the right place from the very >beginning, and, more importantly, to be able to deal with other packages >not having such .egg-info files. It doesn't matter where the files actually live; it meets the technical requirements just fine if you symlink the metadata into site-packages, and would work with the existing runtime. (Or does the FHS disallow symlinking out of site-packages, too? I'm actually surprisd the FHS would be dictating anything about Python package internals anyway, but oh well.) Alternately, you could just put your proposed /usr/share directory on sys.path (just after site-packages) which would solve the issue of trying to have some new way of configuring the location, which setuptools would then need to be able to read. Sure, that would increase the length of sys.path, but only by one, no matter how many eggs are installed by Debian. > > Since I'm neither a Debian nor Mozilla developer, I have no way to know > > what these problems you speak of are, nor any way to tell whether the > > alleged flaws of the Mozilla packaging relate in any meaningful way to the > > alleged flaws of eggs. It's you who's in the position of giving advocacy > > without providing any real information about the issues. > >This is true. In short, my two main concerns about mozilla-related >packaging are total absence of library versioning, and the shipping of >extensions in .xpi files, which are tarballs containing information and >meta-information about the package. You can surely understand that my >concerns about eggs approach those of .xpi files. Yes - but I fear that you may be making an inaccurate comparison based on a superficial similarity. Distribution of binary eggs of anything but pure-Python libraries (i.e., ones with no native code) is understood to be impractical for anything but well-identified slow-moving platforms like Windows. If you observe e.g. TurboGears' distribution practices, you'll see that it relies on Python source packages for "impure" projects on anything but OS X and Windows. As a practical matter, eggs containing native code for Linux get built from source on the target machine, unless some system packager provides them. It sounds like your Mozilla complaints are directed at Mozilla-provided pre-packaged binaries that include native code, which is not something that most Python developers *want* to get involved in. There will undoubtedly be projects (especially commercial ones) that provide plugin binaries for various platforms, but even they will prefer to rely on system-packaged libraries for anything that's not bleeding edge. And the plugins themselves most likely will live in those apps' plugin directories, not system directories. (I'm of course making several assumptions here about what your concerns are and what kind of issues you've encountered, as you still haven't provided anything other than a description of similarity in form, and some vague hints (e.g. "library versioning") about what might actually be at issue.) From pje at telecommunity.com Thu Nov 24 22:29:33 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 24 Nov 2005 16:29:33 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <E1EfO6x-0002J0-00@d.0x1.org> References: <Your message of "Thu, 24 Nov 2005 14:13:09 CDT." <5.1.1.6.0.20051124134908.01fa4328@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051124160420.0521ca60@mail.telecommunity.com> At 07:54 AM 11/25/2005 +1100, David Arnold wrote: > From a Python-centric viewpoint, Debian's (and RedHat, Gentoo, Solaris, >etc) packaging mechanism, however great, covers only one of the many >possible platforms that an application might need to support. > > From a Debian-centric viewpoint, dependency management is a solved >problem, with a decade of experience behind it, and proposing that >Python applications should have their dependencies managed separately is >blatantly foolhardy. I prefer a slightly different spin: Python developers would *love* to have Debian manage their packages, they would simply like to be able to verify at runtime that the management has in fact been done. It's not that we don't trust you, it's just that we're paranoid. :) We'd like for Debian to advertise to our packages, precisely what versions of which of our dependencies are installed. We'd also like for Debian to include the metadata we provided with our packages, when it installs them. And we'd like all this to cleanly work with any locally-installed non-Debian eggs that might be in the mix, since we need to do development, beta testing, etc. >I'm not convinced that's the case (I'm not sure how reasonable it is to >automatically convert 'require("FooBar>=1.2")' into the equivalent >Debian package names and versions, but my expectation is that it's no >more difficult than it currently is to package a "product" for Debian. The 'FooBar' name generally refers to a well-known name registered with PyPI, and both the name and version come from metadata that is already supplied by setup.py for all distributions packaged with the distutils today. So you actually have this information *already*, for any distutils-based distribution you package with Debian. For the Python developer's purposes (i.e., verifying whether something is there), you do not *need* to match this up with Debian package naming, however. You only need to do that when you are packaging something that uses such dependencies, and as you say, it's probably not any harder than the dependency mapping you do now. (Except for the part where egg-based projects tend to have more dependencies simply because they *can*, encouraging reuse instead of reinvention.) Anyway, for advertising the presence of dependencies, we only need a standardized way for distributors to say, "FooBar 1.2 is installed here". If a developer's project needs FooBar 1.2, he's not really worried about what Debian package provides it, and the egg runtime isn't in the business of asking system packagers to install things. It just "properly complains" (as one person put it) about the absence of things that should be there. >Finally, this is ignoring other possible uses of egg metadata, which >might end up being far more problematic? I don't see why they would be any more problematic than any other kind of data that comes with Python packages. Again, we just need to be able to read this information at runtime to discover services, plugins, etc. without requiring user configuration besides just installing things. Here's a motivating example of what this kind of data can be used for. Suppose I said, "okay, I don't see how we can satisfy the Debian requirements cleanly, so suppose I make setuptools allow plugins to redefine how the metadata gets found?" I would then define an entry point group, let's say "setuptools.dependency_resolvers", and add some code to query these entry points when scanning for egg metadata. Debian could then create a separate Python package, and include in its metadata an entry point for this entry point group, which would tell setuptools where to import the plugin code from. Debian would then make sure this package and its metadata are discoverable to setuptools, and setuptools would then automatically do the right thing on any Debian installation, even if the user was running with a new, locally-installed version of setuptools. Of course, this scenario wouldn't actually work due to the bootstrapping issue. I just mention it so you can see an example of why packages want to be able to discover metadata besides the mere presence of a dependency. In this example, the hypothetical Debian-written plugin need not be known about by the provider of the plugin point, i.e. setuptools. Anyway, the point is that we simply need a way to link that kind of data to sys.path entries, so we can be sure that the things we need are there, and not shadowed by things that are before them on sys.path. From pje at telecommunity.com Thu Nov 24 22:32:08 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 24 Nov 2005 16:32:08 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <dm58dq$6j8$1@sea.gmane.org> References: <5.1.1.6.0.20051124144838.01f7b560@mail.telecommunity.com> <5.1.1.6.0.20051124134908.01fa4328@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <438378C1.4050401@colorstudy.com> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <4383642F.8030804@egenix.com> <438378C1.4050401@colorstudy.com> <5.1.1.6.0.20051122161120.03191258@mail.telecommunity.com> <5.1.1.6.0.20051124095605.03c341a0@mail.telecommunity.com> <5.1.1.6.0.20051124104922.02e3fb78@mail.telecommunity.com> <5.1.1.6.0.20051124122133.02e624a0@mail.telecommunity.com> <1132856860.5916.17.camel@arrakis.localnet> <5.1.1.6.0.20051124134908.01fa4328@mail.telecommunity.com> <dm54nu$s3g$1@sea.gmane.org> <5.1.1.6.0.20051124144838.01f7b560@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051124154836.03293bc8@mail.telecommunity.com> At 12:39 PM 11/24/2005 -0800, Robert Kern wrote: >I'm not suggesting that /usr/share/.../ should be the only place to find > .egg-info directories. Simply that pkg_resources would scan >sys.path+['/usr/share/.../'] and treat the ones found in /usr/share/.../ >as if they were in /usr/lib/pythonX.Y/site-packages/. Everywhere else >that a user could put Python packages (e.g. >/usr/local/lib/pythonX.Y/site-packages/) is fine to put whatever the >user wants. Eggs, .egg-info/; it doesn't matter. It doesn't need an >alternate location. Debian packages need to be more disciplined, but all >Debianized Python packages are installed to >/usr/lib/pythonX.Y/site-packages/ so we only need the one alias, not many. > >It's possible that only the DD in charge of packaging setuptools will >have to concern himself with coding this. Or they can just put the /usr/share/whatever directory on sys.path just after site-packages, with no need to modify setuptools then at all. This solution would work today even with a locally-installed (including user-home-directory-installed) version of setuptools. From david at mantara.com Thu Nov 24 23:30:14 2005 From: david at mantara.com (David Arnold) Date: Fri, 25 Nov 2005 09:30:14 +1100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: Your message of "Thu, 24 Nov 2005 16:29:33 CDT." <5.1.1.6.0.20051124160420.0521ca60@mail.telecommunity.com> Message-ID: <E1EfPba-0004S6-00@server.0x1.org> -->"Phillip" == Phillip J Eby <pje at telecommunity.com> writes: Phillip> Python developers would *love* to have Debian manage their Phillip> packages, they would simply like to be able to verify at Phillip> runtime that the management has in fact been done. It's not Phillip> that we don't trust you, it's just that we're paranoid. :) >From a Debian perspective, that's strictly your own affair. Debian packages "just work", and if they don't it's a bug. There's no need for anyone other than the packager to worry about this. Of course, the application is free to do its own verification of this, but it would need to use an internal mechanism: since there's no *need* for it, Debian doesn't expose this information from its internal dependency database. Phillip> We'd like for Debian to advertise to our packages, precisely Phillip> what versions of which of our dependencies are installed. In general, I don't see why packages should care. Either it works, in which case there's no problem, or it doesn't, in which case it's a packaging bug, and it will be fixed. Of course, when supporting packages fail to provide a stable API, this means you end up needing to install multiple versions, or have the application adapt internally. Supporting the installation and simultaneous use of multiple versions seems to be a goal of eggs? FWIW, that's kinda heretical on Unix (which might explain some of the antipathy). Phillip> We'd also like for Debian to include the metadata we provided Phillip> with our packages, when it installs them. It think that's a reasonable expectation. So long as Python packages are location-agnostic, and allow Debian to put such things where it thinks is reasonable? Phillip> And we'd like all this to cleanly work with any Phillip> locally-installed non-Debian eggs that might be in the mix, Phillip> since we need to do development, beta testing, etc. And non-egg packages as well, right? But again, I don't see why that would be a problem. Perhaps it's worth pointing out: Debian users typically don't recognise dependency management as a problem, because it's not a problem for them. If you can't apt-get something, the choice is to wait a month for it to arrive in unstable, or to enter a world of pain where you're dealing with other (often conflicting) dependency management systems or no dependency management at all (and btw, this is the commonality I think eggs have with CPAN and PEAR). Debian developers take their role of banishing this problem seriously, which I suppose is why this is a contentious topic. d From jjl at pobox.com Fri Nov 25 00:02:01 2005 From: jjl at pobox.com (John J Lee) Date: Thu, 24 Nov 2005 23:02:01 +0000 (UTC) Subject: [Distutils] setuptools doc patch Message-ID: <Pine.LNX.4.58.0511242255030.6279@alice> The comma removed by the patch below implies that users working from a Subversion checkout should not use EasyInstall to periodically fetch the latest version. That's not what you intended to say, I assume? http://peak.telecommunity.com/DevCenter/setuptools You should also inform your users of the need to run this command, if they -are working from a Subversion checkout, rather than using EasyInstall to +are working from a Subversion checkout rather than using EasyInstall to periodically fetch the latest version. Hmm, probably much better than the patch above: -You should also inform your users of the need to run this command, if they -are working from a Subversion checkout, rather than using EasyInstall to -periodically fetch the latest version. +If your users are working from a Subversion checkout rather than using +EasyInstall, you should also inform them of the need to run this command +to periodically fetch the latest version. John From pje at telecommunity.com Fri Nov 25 01:16:44 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 24 Nov 2005 19:16:44 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <E1EfPba-0004S6-00@server.0x1.org> References: <Your message of "Thu, 24 Nov 2005 16:29:33 CDT." <5.1.1.6.0.20051124160420.0521ca60@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> At 09:30 AM 11/25/2005 +1100, David Arnold wrote: >-->"Phillip" == Phillip J Eby <pje at telecommunity.com> writes: > > Phillip> Python developers would *love* to have Debian manage their > Phillip> packages, they would simply like to be able to verify at > Phillip> runtime that the management has in fact been done. It's not > Phillip> that we don't trust you, it's just that we're paranoid. :) > > From a Debian perspective, that's strictly your own affair. Debian >packages "just work", and if they don't it's a bug. There's no need for >anyone other than the packager to worry about this. And if we were Debian developers rather than Python developers, this would be a sensible approach to take. As a practical matter, we have to worry, because our users aren't strictly Debian users, and asking them all to move to Debian isn't practical for most of us. ;) Let me ask you this: does Debian ban people from putting $id$ strings in C code to allow identifying the version of the source that was used to build a library or executable? Because that's what *exactly* what we're talking about here in respect to dependencies, just for Python code instead of C. > Phillip> We'd like for Debian to advertise to our packages, precisely > Phillip> what versions of which of our dependencies are installed. > >In general, I don't see why packages should care. Either it works, in >which case there's no problem, or it doesn't, in which case it's a >packaging bug, and it will be fixed. But we can't tell if it's a packaging bug, or a bug against what version of our dependencies, unless we know what version the user is running. Given that sys.path can include *both* Debian system packages and local user-installed packages, we need a more comprehensive way of dealing with the issue, which is what eggs provide. >Supporting the installation and simultaneous use of multiple versions >seems to be a goal of eggs? Absolutely. > FWIW, that's kinda heretical on Unix (which >might explain some of the antipathy). The Python approach is "Practicality Beats Purity", which I thought was also a Unix value. > Phillip> We'd also like for Debian to include the metadata we provided > Phillip> with our packages, when it installs them. > >It think that's a reasonable expectation. So long as Python packages >are location-agnostic, and allow Debian to put such things where it >thinks is reasonable? As long as it can be found by the package, without needing any special configuration. Python packages are looked up relative to sys.path, so it's reasonable to expect to find the metadata on sys.path as well, preferably at a fixed location relative to the code. > Phillip> And we'd like all this to cleanly work with any > Phillip> locally-installed non-Debian eggs that might be in the mix, > Phillip> since we need to do development, beta testing, etc. > >And non-egg packages as well, right? There isn't any such thing, from an egg developer's perspective. Any distutils package can be made into an egg, because all of the metadata needed is supplied by the standard distutils setup script. So, if you have the source, you can make it an egg. With respect to the whole "Dependencies are a solved problem" thing, you're beating a dead horse. I freely admit you can manage *system* dependencies better. But that's only *part* of an application developer's problem. That's no critique of your efforts, only a recognition of their scope. For example, it doesn't help somebody developing a package while using Subversion-head versions of three other packages. It doesn't help Windows users. There's a whole bunch of things it just doesn't help with, that eggs do. That doesn't mean you should change Debian to eggs, because eggs aren't trying to solve the problems that you guys solve either. But if you'd like for your system packages of Python project *not* to have to be duplicated as separately distributed eggs, then we need to have some dependency info, so that your packages can play in those areas where you *aren't* providing a complete solution, but *can* provide a part of it. Thus, maybe the issue for some is that supplying the dependency information would be an admission that there really *are* problems Debian doesn't solve, as opposed to "merely" helping with. From david at mantara.com Fri Nov 25 02:54:12 2005 From: david at mantara.com (David Arnold) Date: Fri, 25 Nov 2005 12:54:12 +1100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: Your message of "Thu, 24 Nov 2005 19:16:44 CDT." <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> Message-ID: <E1EfSmy-0004aw-00@server.0x1.org> -->"Phillip" == Phillip J Eby <pje at telecommunity.com> writes: Phillip> Python developers would *love* to have Debian manage their Phillip> packages, they would simply like to be able to verify at Phillip> runtime that the management has in fact been done. It's not Phillip> that we don't trust you, it's just that we're paranoid. :) >> From a Debian perspective, that's strictly your own affair. >> Debian packages "just work", and if they don't it's a bug. There's >> no need for anyone other than the packager to worry about this. Phillip> And if we were Debian developers rather than Python Phillip> developers, this would be a sensible approach to take. As a Phillip> practical matter, we have to worry, because our users aren't Phillip> strictly Debian users, and asking them all to move to Debian Phillip> isn't practical for most of us. ;) Yep, I understand that totally. Phillip> Let me ask you this: does Debian ban people from putting $id$ Phillip> strings in C code to allow identifying the version of the Phillip> source that was used to build a library or executable? No. Phillip> Because that's what *exactly* what we're talking about here Phillip> in respect to dependencies, just for Python code instead of Phillip> C. So, if a system package, shipped by the upstream developer as an egg, is "unpacked" into a directory structure, and its metadata is maintained in a .egg-info file somewhere in sys.path, non-system eggs will have all they need to operate correctly? Phillip> We'd like for Debian to advertise to our packages, precisely Phillip> what versions of which of our dependencies are installed. >> In general, I don't see why packages should care. Either it works, >> in which case there's no problem, or it doesn't, in which case it's >> a packaging bug, and it will be fixed. Phillip> But we can't tell if it's a packaging bug, or a bug against Phillip> what version of our dependencies, unless we know what version Phillip> the user is running. So that's another goal of eggs? To provide information to a package maintainer to assist in determining if it's the user's PYTHONPATH or .pth files that are causing a bug? I can see this would be helpful. But Debian won't tell you that (as you said you'd like, above). Phillip> Given that sys.path can include *both* Debian system packages Phillip> and local user-installed packages, we need a more Phillip> comprehensive way of dealing with the issue, which is what Phillip> eggs provide. I can see that this is *nice*; I'd debate "need". But I'm happy to accept that for egg-based stuff, this is a nice feature. >> Supporting the installation and simultaneous use of multiple >> versions seems to be a goal of eggs? Phillip> Absolutely. >> FWIW, that's kinda heretical on Unix (which might explain some of >> the antipathy). Phillip> The Python approach is "Practicality Beats Purity", which I Phillip> thought was also a Unix value. I'm not going to try to assert "Unix values" here. My observation is that historically, Unix has installed things into one of a couple of directory hierarchies (/usr, /usr/local, /opt). Within those hierarchies, there has been scope for only one version of any given thing. This is ok because developers take care to maintain API compatibility at the source level, at least, and ABI compatibility if they're well behaved. If you need to break compatibility, it's normally done by renaming the new verion (ie. gtk vs gtk2). Without going into the merits of this, it's my observation that this has been the in-practice approach (and you can draw whatever philosophical "Unix values" you like from that). Phillip> And we'd like all this to cleanly work with any Phillip> locally-installed non-Debian eggs that might be in the mix, Phillip> since we need to do development, beta testing, etc. >> And non-egg packages as well, right? Phillip> There isn't any such thing, from an egg developer's Phillip> perspective. Really? So if I use one egg, everything has to be an egg? Phillip> Any distutils package can be made into an egg, because all of Phillip> the metadata needed is supplied by the standard distutils Phillip> setup script. So, if you have the source, you can make it an Phillip> egg. What if I don't have the source (or setup.py) ? Phillip> That doesn't mean you should change Debian to eggs, because Phillip> eggs aren't trying to solve the problems that you guys solve Phillip> either. It seems to me that it's likely Debian will need to cope with eggs. Phillip> But if you'd like for your system packages of Python project Phillip> *not* to have to be duplicated as separately distributed Phillip> eggs, I personally would appreciate that not being the case Phillip> then we need to have some dependency info, so that your Phillip> packages can play in those areas where you *aren't* providing Phillip> a complete solution, but *can* provide a part of it. That seems reasonable. Accepting that there will be parallel (I hesitate to say "competing") systems, and that keeping them in sync is both hard and necessary seems to be the open issue. Phillip> Thus, maybe the issue for some is that supplying the Phillip> dependency information would be an admission that there Phillip> really *are* problems Debian doesn't solve, as opposed to Phillip> "merely" helping with. I don't perceive that to be the issue (but perhaps I'm wrong). d From pje at telecommunity.com Fri Nov 25 07:33:05 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 25 Nov 2005 01:33:05 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <E1EfSmy-0004aw-00@server.0x1.org> References: <Your message of "Thu, 24 Nov 2005 19:16:44 CDT." <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> At 12:54 PM 11/25/2005 +1100, David Arnold wrote: >So, if a system package, shipped by the upstream developer as an egg, is >"unpacked" into a directory structure, and its metadata is maintained >in a .egg-info file somewhere in sys.path, non-system eggs will have all >they need to operate correctly? Yes, with a few clarifications. The internal structure of an egg, let's say foobar-1.2-py23.egg, would look something like: foobar/ __init__.py baz.py # plus .pyc files, etc. EGG-INFO/ PKG-INFO # distutils metadata like description/version requires.txt # optional and required dependencies # plus other metadata files, either setuptools-defined or # project specific If you unpack this as-is, but rename EGG-INFO to foobar.egg-info (today) or foobar-1.2.egg-info (when I release 0.6a9 of setuptools), and the whole tree above is in a directory on sys.path, this egg is good to go. I would like to clarify the phrase "shipped as an egg", though. To me, that would mean that the developer is distributing a binary .egg file, and I'm assuming that Debian is primarily interested in *source* packages, being a Free Software distribution. (A binary .egg doesn't have to contain source code at all; you can specifically build it with the source stripped if you desire.) The plan for setuptools 0.6a9 is to provide an option to "setup.py install" that will basically install the layout described above, with the correctly named .egg-info directory automatically created. (Normally, the whole tree above is instead nested in an .egg file or directory.) I think I should also clarify that whether the upstream developer sets out to package their project as an egg or not, it's possible to create an .egg-info directory and PKG-INFO file to identify that distribution, using setuptools' "easy_install" program and the source distribution. So if the developer of 'foobar' did not choose to create an egg or use setuptools, this doesn't stop a developer who wants to *use* foobar from simply running easy_install to create an .egg file for it. So, this is what I mean when I say there's no such thing as a non-egg package for an egg developer. Someone who depends on a package can simply say they depend on it, and when they build their package, they'll get eggs for their dependencies as a side effect. >So that's another goal of eggs? To provide information to a package >maintainer to assist in determining if it's the user's PYTHONPATH or >.pth files that are causing a bug? More specifically, what versions of what packages they're *actually* using, as opposed to what they think they installed or have on their system. PYTHONPATH and .pth files can of course be a factor in that, but also just people thinking they installed something, or not knowing that a bug is fixed in a particular version. Part of it too is finding out whether they're reporting a regression or whether they're just still using a version that has a bug that's been fixed. In the case of the TurboGears mailing list, it's often been the case that TurboGears users flush out a bug in a dependency, which then gets fixed, but then a new TurboGears user maybe reports the same problem, and then it's obvious from their error message whether or not they upgraded. I realize this is stuff you guys probably do all day for system packages, but eggs make the support job easier upstream too. >I can see that this is *nice*; I'd debate "need". But I'm happy to >accept that for egg-based stuff, this is a nice feature. Well, need is relative. A project like TurboGears "needs" this, because otherwise it would be uneconomical to provide the current level of support on as many platforms. So, one project's "nice to have" may be another project's lifeblood, depending on available resources. They've also made it easier for the authors of TurboGears' dependencies to assist in support as well. For me, I'm glad that these features have helped to make something like TurboGears possible and practical. >I'm not going to try to assert "Unix values" here. My observation is >that historically, Unix has installed things into one of a couple of >directory hierarchies (/usr, /usr/local, /opt). Within those >hierarchies, there has been scope for only one version of any given >thing. Um, sure. Not sure what this has to do with the present discussion. As a practical matter, only *one* version of an egg can be *active* (i.e. importable) on sys.path within a given process anyway. It's also clearly not going to be the case on a Debian system that somebody would have multiple versions of something living in /usr/lib, although they might do it for /usr/local or in a user-private directory. So, I think maybe I lost the train of thought on this point here. I was under the impression that the consensus of the Debian-Python folks so far was that of any egg format, the "single version externally managed" one using .egg-info directories was preferred, since it is basically the same as your current layout. (It's also convenient for me to implement, because it's basically the same as the format already used by the "setup.py develop" command for temporarily adding a project's source checkout to sys.path.) > Phillip> And we'd like all this to cleanly work with any > Phillip> locally-installed non-Debian eggs that might be in the mix, > Phillip> since we need to do development, beta testing, etc. > > >> And non-egg packages as well, right? > > Phillip> There isn't any such thing, from an egg developer's > Phillip> perspective. > >Really? So if I use one egg, everything has to be an egg? I'm not sure I follow you. If I'm an egg developer, and I want to use other Python packages in my project, I add their project names and versions to my setup.py, and then I get them installed for free. If an .egg-info on sys.path indicates that the project I want is already on my system, then the tools don't go hunting on PyPI and the runtime doesn't gripe about missing dependencies. Note again that the dependencies *don't* need to be distributed as eggs. They can be distributed as source, eggs, .exe installers (Windows only), or Subversion URLs, as long as either PyPI has a usable link, or if I supply one in my project configuration. These dependencies' authors don't even need to have heard of the concept of eggs, they just need a reasonably-standard Python distutils package with a setup.py. Thus, if I'm developing an egg, yes, all my dependencies have to be eggs, but this doesn't imply that I'm pushing eggification upstream, it just means that I can install their package as an egg locally, which essentially amounts to adding the PKG-INFO file in either an EGG-INFO or .egg-info directory. (The distutils normally generate this PKG-INFO file as part of creating a source distribution, so it's not even an egg-specific file format.) So, projects using setuptools get to take advantage of most any project using distutils, and the upstream projects are modified only by adding the egg-info, in order to allow the tools and runtime to know when a dependency has already been satisfied. While I don't advocate changing all Debian Python packages to add this metadata, I do suggest it's a practical way to deal with certain dependency issues. For example, TurboGears depends on ElementTree, which is not packaged as an egg by its author. (I think that Kid, which is also an egg-packaged TurboGears dependency, may depend on ElementTree as well.) Anyway, the quickest way to get all this stuff working without a lot of hacks to the dependency metadata would be to install an .egg-info marker with the ElementTree package, so that the egg tools and runtime on any user's machine will simply know what version of ElementTree is present, and be happy. I know - you can think of other ways to deal with this. However, most of the ways that have been suggested to date fail in the use case where a user has been using the Debian package, and Kevin moves to requiring a new version of ElementTree or some other dependency, perhaps a new SVN revision that hasn't been released -- foobar-1.3.dev-r4262, let's say. (Setuptools users can have their builds tagged with a repository revision number.) This release of foobar isn't going to be in Debian unless you're tracking subversion revisions of experimental projects daily - and maybe you are, I don't know. The point is that when the Debian package no longer satisfies the dependency, the egg tools move smoothly to downloading and installing wherever the user has configured their development environment to install it, say their ~/pydev directory. So now we've segued smoothly into "multiple versions" being installed, but the "system version" is still intact. A month later, a stable package is released and I upgrade my Debian install. This is a later version than the development version I have in ~/pydev, so the egg tools switch back to that as the preferred version unless I have a .pth file specifically requesting activation of the ~/pydev version as the active version for the other work I'm doing. (And even then it'll still prefer the Debian version if I don't have a ~/pydev version that satisfies something's dependency.) These transitions can only be so seamless if the Debian-installed version of foobar includes the egg-info marker so that the tools know what version is sitting in /usr/lib, as opposed to the version(s) I have hanging in my ~/pydev. > Phillip> Any distutils package can be made into an egg, because all of > Phillip> the metadata needed is supplied by the standard distutils > Phillip> setup script. So, if you have the source, you can make it an > Phillip> egg. > >What if I don't have the source (or setup.py) ? What do you have instead? There really aren't many formats for shipping binary Python packages. The only ones provided by the distutils are bdist_dumb, bdist_wininst, and bdist_rpm. It seems to me that all of these formats except bdist_dumb include enough metadata to be able to get the project name and version, which is all you need to create enough metadata to make a usable egg. The "easy_install" tool actually supports turning bdist_wininst packages into eggs directly. I'm not sure if you could do it with a bdist_dumb. A bdist_rpm probably has most of what you need just in the filename alone, at least if you're doing it manually. (Distutils-built distributions' filenames are too ambiguously formatted for automated parsing, alas, even though a human reader can usually tell what they mean.) Anyway, all you need to make a non-egg package into an egg is its project name and version number. If you have those two things, you can make a PKG-INFO file, and that's all you need for today's egg runtime. For 0.6a9, you won't even need to put the data in a file, just the filename. >Accepting that there will be parallel (I hesitate to say "competing") >systems, and that keeping them in sync is both hard and necessary seems >to be the open issue. I think this may actually be an illusion, perhaps brought about by preconceptions based on experiences with other packaging systems. All we need is that: 1. For Debian packages of setuptools-using packages (i.e., projects like FormEncode that explicitly set themselves up to be eggs), all the included metadata is installed in an .egg-info directory alongside the package. This is nothing more than including all the package's required contents, so there's no "parallel" anything going on here. 2. For Debian packages of non-setuptools packages, that are a dependency of a setuptools-using package, add an empty .egg-info file named for the dependency's project name and version number, as specified in its setup.py name/version options. This is just a simple addition to the packaging, and again doesn't seem to create any "parallel" anything. You do not need to go back and repackage every single Debian-Python package unless you feel that that's a more efficient way to handle it. You can simply add the .egg-info on an as-needed basis, when you package setuptools-using projects. Now, there is the separate issue of whether you want to create a separate pyegg or python-pypi namespace for these packages, so that you can keep a closer match between package names and PyPI project names. That's for you guys to decide, as that's a matter of policy and process. But I don't see anything forcing you to make such a split, so again I don't get the "parallel" part. From pje at telecommunity.com Fri Nov 25 07:54:49 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 25 Nov 2005 01:54:49 -0500 Subject: [Distutils] setuptools doc patch In-Reply-To: <Pine.LNX.4.58.0511242255030.6279@alice> Message-ID: <5.1.1.6.0.20051125014803.03c7cb88@mail.telecommunity.com> At 11:02 PM 11/24/2005 +0000, John J Lee wrote: >The comma removed by the patch below implies that users working from a >Subversion checkout should not use EasyInstall to periodically fetch the >latest version. That's not what you intended to say, I assume? > >http://peak.telecommunity.com/DevCenter/setuptools > > You should also inform your users of the need to run this command, if they >-are working from a Subversion checkout, rather than using EasyInstall to >+are working from a Subversion checkout rather than using EasyInstall to > periodically fetch the latest version. > >Hmm, probably much better than the patch above: > >-You should also inform your users of the need to run this command, if they >-are working from a Subversion checkout, rather than using EasyInstall to >-periodically fetch the latest version. >+If your users are working from a Subversion checkout rather than using >+EasyInstall, you should also inform them of the need to run this command >+to periodically fetch the latest version. Actually, what I meant to say was, if your users aren't using EasyInstall to fetch the latest version, but instead are working directly from a subversion checkout, they need to run "setup.py develop" after updates in order to update their egg-info, dependencies, etc. So, I'm changing this sentence to: "Be sure to also remind any of your users who check out your project from Subversion that they need to run ``setup.py develop`` after every update in order to keep their checkout completely in sync." Thanks for pointing out the ambiguity! From david at mantara.com Fri Nov 25 11:04:10 2005 From: david at mantara.com (David Arnold) Date: Fri, 25 Nov 2005 21:04:10 +1100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: Your message of "Fri, 25 Nov 2005 01:33:05 CDT." <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> Message-ID: <E1EfaR8-0008DS-00@d.0x1.org> -->"Phillip" == Phillip J Eby <pje at telecommunity.com> writes: (btw Phillip, thanks muchly for your patience with this thread) Phillip> If you unpack this as-is, but rename EGG-INFO to Phillip> foobar.egg-info (today) or foobar-1.2.egg-info (when I Phillip> release 0.6a9 of setuptools), and the whole tree above is in Phillip> a directory on sys.path, this egg is good to go. Ok. Phillip> I would like to clarify the phrase "shipped as an egg", Phillip> though. To me, that would mean that the developer is Phillip> distributing a binary .egg file, and I'm assuming that Debian Phillip> is primarily interested in *source* packages, being a Free Phillip> Software distribution. Debian ships both source packages and binary packages. It's my impression (although I could be wrong) that most users use only the binary packages. Phillip> (A binary .egg doesn't have to contain source code at all; Phillip> you can specifically build it with the source stripped if you Phillip> desire.) Normal Debian Python (binary) packages include .py files, which are compiled to both .pyc and .pyo as part of their installation, but C extensions are shipped as pre-built .so libraries. Phillip> While I don't advocate changing all Debian Python packages to Phillip> add this metadata, I do suggest it's a practical way to deal Phillip> with certain dependency issues. Assume a developer grabs some setuptools-using Python app, and tries to run it on a Debian system. It will look in sys.path for its dependencies. If it doesn't find the egg info, as I understand it, the app will then go out and grab those dependencies via PyPI until it's able to run. But, if compatible versions of those dependencies are already installed as Debian packages *without* egg metadata, will these be ignored? Even if it was possible for Debian to extend this downloading mechanism so that it looked for dependencies via apt-get before trying to install from the raw source or egg or whatever, it would usually be the case that the user running the newly downloaded Python application would not have permission to install system packages. Ultimately, I suppose this isn't a major problem -- if a user (or developer) runs such an application, they might end up pulling down and installing a bunch of dependencies in their home directory (that's where they'd go by default, right?) But once the app is packaged, both it and its dependencies will be available for all users, so it'll just be early adopters who will encounter this. d From abo at minkirri.apana.org.au Fri Nov 25 11:29:56 2005 From: abo at minkirri.apana.org.au (Donovan Baarda) Date: Fri, 25 Nov 2005 10:29:56 +0000 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> References: <Your message of "Thu, 24 Nov 2005 19:16:44 CDT." <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> Message-ID: <1132914596.28475.48.camel@warna.corp.google.com> On Fri, 2005-11-25 at 01:33 -0500, Phillip J. Eby wrote: [... long informative explanation of egg...] So it sounds like egg duplicates much of the functionality of the Debian Package manager, but also extends it slightly in python-specific ways. I can under stand the Debian packagers asking themselves; do we just wrap an egg in a Debian package and accept the duplicated functionality, or do we unwrap the egg, add the missing functionality some other way, and create a clean Debian package without the duplicated functionality? The biggest risk of just wrapping the duplicated functionality is; are there incompatibilities or policy violations in the egg way of doing things? In particular, will an egg wrapped inside a Debian package magically install other bits of software not from Debian packages? Will it install them in the correct places? > While I don't advocate changing all Debian Python packages to add this > metadata, I do suggest it's a practical way to deal with certain dependency > issues. For example, TurboGears depends on ElementTree, which is not > packaged as an egg by its author. (I think that Kid, which is also an > egg-packaged TurboGears dependency, may depend on ElementTree as > well.) Anyway, the quickest way to get all this stuff working without a > lot of hacks to the dependency metadata would be to install an .egg-info > marker with the ElementTree package, so that the egg tools and runtime on > any user's machine will simply know what version of ElementTree is present, > and be happy. On thing about this worried me; If TurboGears depends on an "egg'ed" ElementTree, what happens if a system has ElementTree installed as a non-egg package? Does installing TurboGears as an egg inside a Debian package require also installing another ElementTree as an egg inside a Debian package? Or worse, will it automatically download and install an egg'ed ElementTree not from a Debian package? Will you end up with two ElementTree's installed, one egg'ed and one not? > I know - you can think of other ways to deal with this. However, most of > the ways that have been suggested to date fail in the use case where a user > has been using the Debian package, and Kevin moves to requiring a new > version of ElementTree or some other dependency, perhaps a new SVN revision > that hasn't been released -- foobar-1.3.dev-r4262, let's say. (Setuptools > users can have their builds tagged with a repository revision > number.) This release of foobar isn't going to be in Debian unless you're > tracking subversion revisions of experimental projects daily - and maybe > you are, I don't know. The point is that when the Debian package no longer > satisfies the dependency, the egg tools move smoothly to downloading and > installing wherever the user has configured their development environment > to install it, say their ~/pydev directory. So now we've segued smoothly > into "multiple versions" being installed, but the "system version" is still > intact. Debian has versioned dependencies. If a package depends on a particular version of another package that doesn't exist yet, you can't install it; full stop. If you want to package something that depends on a SVN revision of some dependency, then you need to package that SVN version. This is part of Debian's quality control. Debian Packages are only installable if their dependencies are met, and incompatible combinations of packages cannot be installed at all. This enforces packagers to produce a distribution of packaged versions of everything that will work together. It sounds like egg's provide python specific package management focused on development. All that "automatically pull in all the eggs of versions **I** want" allows developers to specify bleeding edge dependencies and not have to worry about dependencies of other things on the system. Debian packages are more focused on stabilising for distribution releases. It encourages/forces you to agree on particular versions of everything for the whole system in a very controlled manner. I think they a both important. If I was using eggs, I'm not sure I'd want them to be packaged as anything other than an egg until I was ready to release, and then I'd want it **not** packaged as an egg, as a "proof of it's release readiness". It's perhaps unfortunate that egg's bundle package management with generic meta-data management. I know that package management uses meta-data, but if it was implemented as one wrapped around the other, you could "peel the package management layer off", leaving the metadata management there, and you could produce clean deb's using only Debian package management, but with the egg meta-data management still in place. If you ever want to install stuff that is not in a Debian package, it should be installed outside of the package-managed directories. On Debian, this usually means anywhere in /home or /usr/local. I'd be installing eggs in /usr/local until they were ready to be packaged as a proper Debian package. -- Donovan Baarda <abo at minkirri.apana.org.au> http://minkirri.apana.org.au/~abo/ From hawk78_it at yahoo.it Fri Nov 25 11:54:33 2005 From: hawk78_it at yahoo.it (Vincenzo Di Massa) Date: Fri, 25 Nov 2005 11:54:33 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <E1EfaR8-0008DS-00@d.0x1.org> References: <E1EfaR8-0008DS-00@d.0x1.org> Message-ID: <200511251154.33408.hawk78_it@yahoo.it> Alle 11:04, venerd? 25 novembre 2005, David Arnold ha scritto: > But, if compatible versions of those dependencies are already installed > as Debian packages *without* egg metadata, will these be ignored? Yes, they will. > Even if it was possible for Debian to extend this downloading mechanism > so that it looked for dependencies via apt-get before trying to install > from the raw source or egg or whatever, it would usually be the case > that the user running the newly downloaded Python application would not > have permission to install system packages. I think easy_install coud be patched by debian packagers (or made configurable by Phillip) to add a further check on dependency checking: it could check if a dependency is already provided by system packages. easy_install (and pkg_resources.require()) could have a config file telling what code execute to test for dependencies, configurable in a way that when I require foo-1.2.3 it can: 1) optionally check if apt (or urpmi or anything else) can provide that depenedency and, if apt has them, promt the user to either: 1.1) run apt-get as root 1.2) skip this step ( 1) ) and go to directly to 2) 2) check if it can find this dep like setuptools does today (installing them as eggs somewhere in the user's home) This simple modification allows distributors to re-implement the run-time dependency checking conforming to theyr packaging policies. Obviuosly it then is distributors business to maintain a database of mappings between distro specific package names and PyPI module names as expressed in requires sections of setup.py files. Philliph, do you like this idea? It is just adding a hook to dependency checking where distributors can call ther depency checking tools. That way the Name+Vesion problem is resolved without putting empty files anywhere. Vincenzo ___________________________________ Yahoo! Messenger: chiamate gratuite in tutto il mondo http://it.messenger.yahoo.com From hawk78_it at yahoo.it Fri Nov 25 12:10:54 2005 From: hawk78_it at yahoo.it (Vincenzo Di Massa) Date: Fri, 25 Nov 2005 12:10:54 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <200511251154.33408.hawk78_it@yahoo.it> References: <E1EfaR8-0008DS-00@d.0x1.org> <200511251154.33408.hawk78_it@yahoo.it> Message-ID: <200511251210.54991.hawk78_it@yahoo.it> I forgot to add that doing what I propose will make *all* debian python packages become valid eggs *without* touching them at all: it would be setuptools code tha becomes able to interpret a deb package meaning, considering it as if it were a real egg package. So the Debian policy, the filesistem layout and *all python packages* would remain untouched! Vincenzo ___________________________________ Yahoo! Mail: gratis 1GB per i messaggi e allegati da 10MB http://mail.yahoo.it From p.f.moore at gmail.com Fri Nov 25 12:34:16 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 25 Nov 2005 11:34:16 +0000 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1132914596.28475.48.camel@warna.corp.google.com> References: <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> <1132914596.28475.48.camel@warna.corp.google.com> Message-ID: <79990c6b0511250334v7602dbb4oc625d28ac2c3d039@mail.gmail.com> On 11/25/05, Donovan Baarda <abo at minkirri.apana.org.au> wrote: > On thing about this worried me; > > If TurboGears depends on an "egg'ed" ElementTree, what happens if a > system has ElementTree installed as a non-egg package? Does installing > TurboGears as an egg inside a Debian package require also installing > another ElementTree as an egg inside a Debian package? Or worse, will it > automatically download and install an egg'ed ElementTree not from a > Debian package? Will you end up with two ElementTree's installed, one > egg'ed and one not? Yes. That's the issue I'm not keen on, and I believe your interpretation is right - TurboGears depends on ElementTree, which is fine. But *because* TurboGears is an egg [1] it *needs* an egg-ified version of ElementTree. If I have ElementTree installed already, TG will *ignore* that, and grab an extra version from the internet (at install time, presumably, assuning I'm using EasyInstall - but what if I'm not?) just to get it in egg format. This is the viral aspect of eggs - if I use an egg, I have to switch to using eggs for all its dependencies, even if I'm currently a happy user of the non-egg version of them. To me, this sucks. Sorry, Philip, I know *why* you can't introspect a non-egg ElementTree well enough to avoid this, but it still sucks. I'm also aware that in the very near future, I'll be able to just add a single empty .egg-info file to satisfy the dependency tracking, but it's still maintenance *I* have to do to satisfy TG that a package I have had for ages exists... The suckiness is certainly decreasing, but it's still there. Paul. [1] It's not exactly because TG is an egg per se, as easy_install and dependency checking is involved as well, which not all eggs have to use. I'm slowly coming to understand that eggs (and setuptools, and easy_install, which are related but subtly different items) provide a host of largely unrelated functionality, and discussions are getting confused because these are not being separated out sufficiently. I'm trying to understand all the bits, and when I do, I'm hoping to post a summary. But as an example, eggs provide: - a way to store package metadata in a runtime-introspectable way - a standard way to store and locate package data files - a package location mechanism (plugins) - a dependency management infrastructure - a way of downloading dependencies automatically - a "just drop it in" distribution format (this clashes with the dependency management stuff, and seems to be less prominent these days - what would happen if I downloaded the TurboGears egg and just put it on sys.path - no easy_install or whatever?) plus many others, I think. These are *not* the same. Some (package data, plugins) have been invented before, probably many times, and eggs just provide a standardised drop-in approach. Some are relatively non-contentious useful tools. And some are much more contentious. I've more to say on this, but I'll wait until I've done my research. From no228 at cam.ac.uk Fri Nov 25 12:35:48 2005 From: no228 at cam.ac.uk (Noel O'Boyle) Date: Fri, 25 Nov 2005 11:35:48 +0000 Subject: [Distutils] Including package_data in a source distribution Message-ID: <1132918548.5285.27.camel@sandwi.ch.cam.ac.uk> Dear all, I have been trying to use either distutils or setuptools to include some package data in a source distribution ("python setup.py sdist") using the package_data option. It *is* included if I use "python setup.py bdist_egg". Can someone explain to me whether this is a feature or a bug? The documentation led me to expect that this would work, but it does not. I am using the example on http://www.python.org/dev/doc/devel/dist/node12.html. Regards, Noel O'Boyle. From mal at egenix.com Fri Nov 25 12:56:13 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 25 Nov 2005 12:56:13 +0100 Subject: [Distutils] Including package_data in a source distribution In-Reply-To: <1132918548.5285.27.camel@sandwi.ch.cam.ac.uk> References: <1132918548.5285.27.camel@sandwi.ch.cam.ac.uk> Message-ID: <4386FBDD.902@egenix.com> Noel O'Boyle wrote: > Dear all, > > I have been trying to use either distutils or setuptools to include some > package data in a source distribution ("python setup.py sdist") using > the package_data option. It *is* included if I use "python setup.py > bdist_egg". > > Can someone explain to me whether this is a feature or a bug? The > documentation led me to expect that this would work, but it does not. I > am using the example on > http://www.python.org/dev/doc/devel/dist/node12.html. This is probably a bug, since the package_data files are not taken into account by the source distribution command in distutils (sdist.py). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 25 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From hawk78_it at yahoo.it Fri Nov 25 13:04:17 2005 From: hawk78_it at yahoo.it (Vincenzo Di Massa) Date: Fri, 25 Nov 2005 13:04:17 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <79990c6b0511250334v7602dbb4oc625d28ac2c3d039@mail.gmail.com> References: <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <1132914596.28475.48.camel@warna.corp.google.com> <79990c6b0511250334v7602dbb4oc625d28ac2c3d039@mail.gmail.com> Message-ID: <200511251304.17984.hawk78_it@yahoo.it> Alle 12:34, venerd? 25 novembre 2005, Paul Moore ha scritto: > On 11/25/05, Donovan Baarda <abo at minkirri.apana.org.au> wrote: > > On thing about this worried me; > > > > If TurboGears depends on an "egg'ed" ElementTree, what happens if a > > system has ElementTree installed as a non-egg package? Does installing > > TurboGears as an egg inside a Debian package require also installing > > another ElementTree as an egg inside a Debian package? Or worse, will it > > automatically download and install an egg'ed ElementTree not from a > > Debian package? Will you end up with two ElementTree's installed, one > > egg'ed and one not? > > Yes. That's the issue I'm not keen on, and I believe your > interpretation is right - TurboGears depends on ElementTree, which is > fine. But *because* TurboGears is an egg [1] it *needs* an egg-ified > version of ElementTree. If I have ElementTree installed already, TG > will *ignore* that, and grab an extra version from the internet (at > install time, presumably, assuning I'm using EasyInstall - but what if > I'm not?) just to get it in egg format. > > This is the viral aspect of eggs - if I use an egg, I have to switch > to using eggs for all its dependencies, even if I'm currently a happy > user of the non-egg version of them. > > To me, this sucks. Sorry, Philip, I know *why* you can't introspect a > non-egg ElementTree well enough to avoid this, but it still sucks. Please read http://mail.python.org/pipermail/distutils-sig/2005-November/005520.html http://mail.python.org/pipermail/distutils-sig/2005-November/005521.html Can my proposal resolve the "suckiness". > I'm > also aware that in the very near future, I'll be able to just add a > single empty .egg-info file to satisfy the dependency tracking, but > it's still maintenance *I* have to do to satisfy TG that a package I > have had for ages exists... The suckiness is certainly decreasing, but > it's still there. > > Paul. > > [1] It's not exactly because TG is an egg per se, as easy_install and > dependency checking is involved as well, which not all eggs have to > use. I'm slowly coming to understand that eggs (and setuptools, and > easy_install, which are related but subtly different items) provide a > host of largely unrelated functionality, and discussions are getting > confused because these are not being separated out sufficiently. I'm > trying to understand all the bits, and when I do, I'm hoping to post a > summary. But as an example, eggs provide: > > - a way to store package metadata in a runtime-introspectable way > - a standard way to store and locate package data files > - a package location mechanism (plugins) > - a dependency management infrastructure > - a way of downloading dependencies automatically > - a "just drop it in" distribution format (this clashes with the More hooks could be provided (apart from the one thtat just checks for dependencies) to allow distributors redefine where things are installed and how to find them. This way the logical egg parts (code/metadata/plugins) will become a "guide" distributors will use to put things in the "Right Place (tm)". Accepting my proposal will make the actual setuptools one of the implementatios of the "egg concept" and the mother of all other implementations (in the some way distutils is the mother of setuptools). A distributor could provide a custom setuptools implementation that is just the original setuptools + some code (plugins to setuptools itself?, hooks anyway) that both checks for dependencies/metadata/plugins/... and acts in response to checks in a custom way. This way the egg concept will become othogonal(meaning unrelated) to the egg format. Anyway the custom setuptools will be able to fallback to the original setutools way of handling eggs (the only difference is that it will install things into user home or /usr/local). > dependency management stuff, and seems to be less prominent these days > - what would happen if I downloaded the TurboGears egg and just put it > on sys.path - no easy_install or whatever?) > > plus many others, I think. These are *not* the same. Some (package > data, plugins) have been invented before, probably many times, and > eggs just provide a standardised drop-in approach. Some are relatively > non-contentious useful tools. And some are much more contentious. I've > more to say on this, but I'll wait until I've done my research. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Vincenzo ___________________________________ Yahoo! Mail: gratis 1GB per i messaggi e allegati da 10MB http://mail.yahoo.it From hawk78_it at yahoo.it Fri Nov 25 13:04:17 2005 From: hawk78_it at yahoo.it (Vincenzo Di Massa) Date: Fri, 25 Nov 2005 13:04:17 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <79990c6b0511250334v7602dbb4oc625d28ac2c3d039@mail.gmail.com> References: <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <1132914596.28475.48.camel@warna.corp.google.com> <79990c6b0511250334v7602dbb4oc625d28ac2c3d039@mail.gmail.com> Message-ID: <200511251304.17984.hawk78_it@yahoo.it> Alle 12:34, venerd? 25 novembre 2005, Paul Moore ha scritto: > On 11/25/05, Donovan Baarda <abo at minkirri.apana.org.au> wrote: > > On thing about this worried me; > > > > If TurboGears depends on an "egg'ed" ElementTree, what happens if a > > system has ElementTree installed as a non-egg package? Does installing > > TurboGears as an egg inside a Debian package require also installing > > another ElementTree as an egg inside a Debian package? Or worse, will it > > automatically download and install an egg'ed ElementTree not from a > > Debian package? Will you end up with two ElementTree's installed, one > > egg'ed and one not? > > Yes. That's the issue I'm not keen on, and I believe your > interpretation is right - TurboGears depends on ElementTree, which is > fine. But *because* TurboGears is an egg [1] it *needs* an egg-ified > version of ElementTree. If I have ElementTree installed already, TG > will *ignore* that, and grab an extra version from the internet (at > install time, presumably, assuning I'm using EasyInstall - but what if > I'm not?) just to get it in egg format. > > This is the viral aspect of eggs - if I use an egg, I have to switch > to using eggs for all its dependencies, even if I'm currently a happy > user of the non-egg version of them. > > To me, this sucks. Sorry, Philip, I know *why* you can't introspect a > non-egg ElementTree well enough to avoid this, but it still sucks. Please read http://mail.python.org/pipermail/distutils-sig/2005-November/005520.html http://mail.python.org/pipermail/distutils-sig/2005-November/005521.html Can my proposal resolve the "suckiness". > I'm > also aware that in the very near future, I'll be able to just add a > single empty .egg-info file to satisfy the dependency tracking, but > it's still maintenance *I* have to do to satisfy TG that a package I > have had for ages exists... The suckiness is certainly decreasing, but > it's still there. > > Paul. > > [1] It's not exactly because TG is an egg per se, as easy_install and > dependency checking is involved as well, which not all eggs have to > use. I'm slowly coming to understand that eggs (and setuptools, and > easy_install, which are related but subtly different items) provide a > host of largely unrelated functionality, and discussions are getting > confused because these are not being separated out sufficiently. I'm > trying to understand all the bits, and when I do, I'm hoping to post a > summary. But as an example, eggs provide: > > - a way to store package metadata in a runtime-introspectable way > - a standard way to store and locate package data files > - a package location mechanism (plugins) > - a dependency management infrastructure > - a way of downloading dependencies automatically > - a "just drop it in" distribution format (this clashes with the More hooks could be provided (apart from the one thtat just checks for dependencies) to allow distributors redefine where things are installed and how to find them. This way the logical egg parts (code/metadata/plugins) will become a "guide" distributors will use to put things in the "Right Place (tm)". Accepting my proposal will make the actual setuptools one of the implementatios of the "egg concept" and the mother of all other implementations (in the some way distutils is the mother of setuptools). A distributor could provide a custom setuptools implementation that is just the original setuptools + some code (plugins to setuptools itself?, hooks anyway) that both checks for dependencies/metadata/plugins/... and acts in response to checks in a custom way. This way the egg concept will become othogonal(meaning unrelated) to the egg format. Anyway the custom setuptools will be able to fallback to the original setutools way of handling eggs (the only difference is that it will install things into user home or /usr/local). > dependency management stuff, and seems to be less prominent these days > - what would happen if I downloaded the TurboGears egg and just put it > on sys.path - no easy_install or whatever?) > > plus many others, I think. These are *not* the same. Some (package > data, plugins) have been invented before, probably many times, and > eggs just provide a standardised drop-in approach. Some are relatively > non-contentious useful tools. And some are much more contentious. I've > more to say on this, but I'll wait until I've done my research. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Vincenzo ___________________________________ Yahoo! Mail: gratis 1GB per i messaggi e allegati da 10MB http://mail.yahoo.it From alex at bofh.net.pl Fri Nov 25 13:07:03 2005 From: alex at bofh.net.pl (Janusz A. Urbanowicz) Date: Fri, 25 Nov 2005 13:07:03 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1132914596.28475.48.camel@warna.corp.google.com> References: <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> <1132914596.28475.48.camel@warna.corp.google.com> Message-ID: <20051125120703.GO29592@syjon.fantastyka.net> On Fri, Nov 25, 2005 at 10:29:56AM +0000, Donovan Baarda wrote: > On Fri, 2005-11-25 at 01:33 -0500, Phillip J. Eby wrote: > [... long informative explanation of egg...] > In particular, will an egg wrapped inside a Debian package magically > install other bits of software not from Debian packages? Will it install > them in the correct places? This is a dangerous practice from ore than one point of view: 1) it may pollute the system with non-DFSG-compliant stuff 2) as a both python developer and debian user and developer I DO NOT want software to download and run stuff without my knowledge and explicit consent > It sounds like egg's provide python specific package management focused > on development. All that "automatically pull in all the eggs of versions > **I** want" allows developers to specify bleeding edge dependencies and > not have to worry about dependencies of other things on the system. And in the long run this leads to many disasters. > Debian packages are more focused on stabilising for distribution > releases. It encourages/forces you to agree on particular versions of > everything for the whole system in a very controlled manner. > > I think they a both important. If I was using eggs, I'm not sure I'd > want them to be packaged as anything other than an egg until I was ready > to release, and then I'd want it **not** packaged as an egg, as a "proof > of it's release readiness". > > It's perhaps unfortunate that egg's bundle package management with > generic meta-data management. I know that package management uses > meta-data, but if it was implemented as one wrapped around the other, > you could "peel the package management layer off", leaving the metadata > management there, and you could produce clean deb's using only Debian > package management, but with the egg meta-data management still in > place. > > If you ever want to install stuff that is not in a Debian package, it > should be installed outside of the package-managed directories. On > Debian, this usually means anywhere in /home or /usr/local. I'd be > installing eggs in /usr/local until they were ready to be packaged as a > proper Debian package. but according to FHS /usr/local is a no-no for a package management to touch -- mors ab alto 0x46399138 From joss at debian.org Fri Nov 25 13:52:26 2005 From: joss at debian.org (Josselin Mouette) Date: Fri, 25 Nov 2005 13:52:26 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <200511251154.33408.hawk78_it@yahoo.it> References: <E1EfaR8-0008DS-00@d.0x1.org> <200511251154.33408.hawk78_it@yahoo.it> Message-ID: <1132923147.27619.3.camel@silicium.ccc.cea.fr> Le vendredi 25 novembre 2005 ? 11:54 +0100, Vincenzo Di Massa a ?crit : > Alle 11:04, venerd? 25 novembre 2005, David Arnold ha scritto: > > But, if compatible versions of those dependencies are already installed > > as Debian packages *without* egg metadata, will these be ignored? > > Yes, they will. Why? > > Even if it was possible for Debian to extend this downloading mechanism > > so that it looked for dependencies via apt-get before trying to install > > from the raw source or egg or whatever, it would usually be the case > > that the user running the newly downloaded Python application would not > > have permission to install system packages. > > I think easy_install coud be patched by debian packagers (or made configurable > by Phillip) to add a further check on dependency checking: it could check if a > dependency is already provided by system packages. It could do it by simply trying to import the module. > easy_install (and pkg_resources.require()) could have a config file telling > what code execute to test for dependencies, configurable in a way that when I > require foo-1.2.3 it can: > 1) optionally check if apt (or urpmi or anything else) can provide that > depenedency and, if apt has them, promt the user to either: > 1.1) run apt-get as root > 1.2) skip this step ( 1) ) and go to directly to 2) No. If the dependency is missing, this is a bug in the Debian package. No more, no less. Furthermore, it is a bad idea for a python script to mess up with the packaging system. Not only querying it is very slow, but it implies to make assumptions about the versioning scheme, that can become wrong at any time. > 2) check if it can find this dep like setuptools does today (installing them > as eggs somewhere in the user's home) I consider this as a security risk. I hope this can be disabled by default. Regards, -- .''`. Josselin Mouette /\./\ : :' : josselin.mouette at ens-lyon.org `. `' joss at debian.org `- Debian GNU/Linux -- The power of freedom From jjl at pobox.com Fri Nov 25 13:58:39 2005 From: jjl at pobox.com (John J Lee) Date: Fri, 25 Nov 2005 12:58:39 +0000 (GMT Standard Time) Subject: [Distutils] setuptools doc patch In-Reply-To: <5.1.1.6.0.20051125014803.03c7cb88@mail.telecommunity.com> References: <5.1.1.6.0.20051125014803.03c7cb88@mail.telecommunity.com> Message-ID: <Pine.WNT.4.64.0511251244190.2040@shaolin> On Fri, 25 Nov 2005, Phillip J. Eby wrote: > At 11:02 PM 11/24/2005 +0000, John J Lee wrote: [...] >> -You should also inform your users of the need to run this command, if they >> -are working from a Subversion checkout, rather than using EasyInstall to >> -periodically fetch the latest version. >> +If your users are working from a Subversion checkout rather than using >> +EasyInstall, you should also inform them of the need to run this command >> +to periodically fetch the latest version. > > Actually, what I meant to say was, if your users aren't using EasyInstall to > fetch the latest version, but instead are working directly from a subversion > checkout, they need to run "setup.py develop" after updates in order to > update their egg-info, dependencies, etc. Yes, that's what I'd understood by it... > So, I'm changing this sentence to: > > "Be sure to also remind any of your users who check out your project > from Subversion that they need to run ``setup.py develop`` after every update > in order to keep their checkout completely in sync." ...hmm, this still reads ambiguously to me (your fault for writing such flexible software ;-) An easy_install myproject==dev may be an SVN checkout too, but I assume there's no need to run setup.py develop (yes, one doesn't even have a setup.py in that case, but it's still confusing to me). How about: "Be sure to also remind any of your users who checked out your project from Subversion and ran ``setup.py develop`` that they need to run ``setup.py develop`` after every update in order to keep their checkout completely in sync." John From abo at minkirri.apana.org.au Fri Nov 25 14:06:06 2005 From: abo at minkirri.apana.org.au (Donovan Baarda) Date: Fri, 25 Nov 2005 13:06:06 +0000 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <20051125120703.GO29592@syjon.fantastyka.net> References: <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> <1132914596.28475.48.camel@warna.corp.google.com> <20051125120703.GO29592@syjon.fantastyka.net> Message-ID: <1132923966.9042.3.camel@warna.corp.google.com> On Fri, 2005-11-25 at 13:07 +0100, Janusz A. Urbanowicz wrote: > On Fri, Nov 25, 2005 at 10:29:56AM +0000, Donovan Baarda wrote: > > On Fri, 2005-11-25 at 01:33 -0500, Phillip J. Eby wrote: [...] > > If you ever want to install stuff that is not in a Debian package, it > > should be installed outside of the package-managed directories. On > > Debian, this usually means anywhere in /home or /usr/local. I'd be > > installing eggs in /usr/local until they were ready to be packaged as a > > proper Debian package. > > but according to FHS /usr/local is a no-no for a package management to touch Sorry... that's what I meant; don't deb an egg... install it as an egg, outside of the Debian package management system. As an egg, it is under development and not ready for release as a deb. When the software is ready for release, strip the egg package management layer and create a proper deb out of it. -- Donovan Baarda <abo at minkirri.apana.org.au> http://minkirri.apana.org.au/~abo/ From pje at telecommunity.com Fri Nov 25 14:19:11 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 25 Nov 2005 08:19:11 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <E1EfaR8-0008DS-00@d.0x1.org> References: <Your message of "Fri, 25 Nov 2005 01:33:05 CDT." <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051125075214.01feee98@mail.telecommunity.com> At 09:04 PM 11/25/2005 +1100, David Arnold wrote: >-->"Phillip" == Phillip J Eby <pje at telecommunity.com> writes: > Phillip> I would like to clarify the phrase "shipped as an egg", > Phillip> though. To me, that would mean that the developer is > Phillip> distributing a binary .egg file, and I'm assuming that Debian > Phillip> is primarily interested in *source* packages, being a Free > Phillip> Software distribution. > >Debian ships both source packages and binary packages. It's my >impression (although I could be wrong) that most users use only the >binary packages. I was saying I assume Debian packages (source or binary) are built from upstream source packages, but you were asking about upstream packages "shipped as an egg". .egg files do not include the full source of a disutils package, except perhaps the .py files. C source would not be included in a distributed .egg. Anyway, I assume that Debian Python packages are currently built from upstream source, not an upstream binary, and .egg files would be considered a binary. >Assume a developer grabs some setuptools-using Python app, and tries to >run it on a Debian system. It will look in sys.path for its >dependencies. If it doesn't find the egg info, as I understand it, the >app will then go out and grab those dependencies via PyPI until it's >able to run. No, the default behavior will be to throw a DependencyNotFound exception describing the problem. A particular application can in principle catch it and do something about it, or pass in an installer hook that will be tried before throwing the exception, but that's up to the application. The setuptools runtime by default is just going to give an exception. Indeed, if it's a required dependency (as opposed to a dependency needed for an optional feature of the application), then you're not going to be able to start the application at all; you'll just get the DependencyNotFound exception currently. >But, if compatible versions of those dependencies are already installed >as Debian packages *without* egg metadata, will these be ignored? That's correct; the wrapper scripts presently have no way to tell such packages exist, and no reliable uniform way to check their version. One possible feature that has been discussed is allowing package authors to specify fallback hooks in their application to try to verify the presence of a package by importing it and testing for the existence of certain symbols. A couple of developers have suggested that for some dependencies, they would be willing to specify such fallbacks in order to take advantage of non-egg system packages. At this point, the means by which this could happen is incompletely specified, and it would be tricky to implement because it means importing code from partially processed dependencies in order to process *their* dependencies, which means the dependency resolution process can't be safely unwound and retried. The other thought I'd had about this (aside from adding .egg-info files to every Python package), is to provide an entry point group that could be registered by a module provided by a system packager that would use the packager's native API to test for the presence of a dependency, and supply a mock Distribution object with the metadata. Of course, this hypothetical packager API would have to be able to map PyPI names and versions to native package names and versions and obtain the metadata, so I'm not sure that in the general case such a thing would be any more useful. (Aside from the fact that such a feature would allow you to put the metadata in /usr/share without adding that directory to sys.path.) >Even if it was possible for Debian to extend this downloading mechanism >so that it looked for dependencies via apt-get before trying to install >from the raw source or egg or whatever, Again, this is not the default behavior. It's the "easy_install" tool that does such downloading and installing, or "setup.py install". Downloading and running things from the internet by default didn't seem to me like a safe thing to do. :) There are various "setup.py" commands that will do it (such as "develop" and "test"), but by running a setup.py you're trusting the author to install arbitrary stuff on your machine anyway. The installed application, unless it provides some explicit user functionality to say, download plugins, will not have this download-and-install behavior. I'm assuming that a Debian user will not be using "easy_install" except to get and run bleeding edge stuff, and that in that case they will be directing it to install in /usr/local or a personal directory. >it would usually be the case >that the user running the newly downloaded Python application would not >have permission to install system packages. > >Ultimately, I suppose this isn't a major problem -- if a user (or >developer) runs such an application, they might end up pulling down and >installing a bunch of dependencies in their home directory (that's where >they'd go by default, right?) No. The default is determined by the standard distutils configuration file chain. If an install location isn't specified in the user's ~/.pydistutils.cfg, the systemwide distutils.cfg will be checked, and finally the default will be /usr/lib/.../site-packages. Debian, by the way, apparently ships with this default unchanged; it should presumably point to /usr/local/.../site-packages instead, so that locally-installed packages would go to the right place. Or you could set a default location such that easy_install will install to a home-relative location by default, and include that directory in your .pth processing (as is done by Mac OS Python with the ~/Library/.../site-packages directory). > But once the app is packaged, both it and >its dependencies will be available for all users, so it'll just be early >adopters who will encounter this. Correct, and only if they are using easy_install or a setup.py to install the package in the first place. And a Debian user using Debian packages should have no reason to touch easy_install, unless they lack permission to install packages. From p.f.moore at gmail.com Fri Nov 25 14:18:27 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 25 Nov 2005 13:18:27 +0000 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <200511251304.17984.hawk78_it@yahoo.it> References: <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <1132914596.28475.48.camel@warna.corp.google.com> <79990c6b0511250334v7602dbb4oc625d28ac2c3d039@mail.gmail.com> <200511251304.17984.hawk78_it@yahoo.it> Message-ID: <79990c6b0511250518r3dc332f9g6415d173a53571d@mail.gmail.com> On 11/25/05, Vincenzo Di Massa <hawk78_it at yahoo.it> wrote: > > To me, this sucks. Sorry, Philip, I know *why* you can't introspect a > > non-egg ElementTree well enough to avoid this, but it still sucks. > > Please read > http://mail.python.org/pipermail/distutils-sig/2005-November/005520.html > http://mail.python.org/pipermail/distutils-sig/2005-November/005521.html > > Can my proposal resolve the "suckiness". Not for me, unfortunately (I use Windows, not Debian), but it seems like a sensible and flexible suggestion for platforms where the system packager maintains that sort of information. Paul. From abo at minkirri.apana.org.au Fri Nov 25 14:23:32 2005 From: abo at minkirri.apana.org.au (Donovan Baarda) Date: Fri, 25 Nov 2005 13:23:32 +0000 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <200511251304.17984.hawk78_it@yahoo.it> References: <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <1132914596.28475.48.camel@warna.corp.google.com> <79990c6b0511250334v7602dbb4oc625d28ac2c3d039@mail.gmail.com> <200511251304.17984.hawk78_it@yahoo.it> Message-ID: <1132925012.9042.13.camel@warna.corp.google.com> On Fri, 2005-11-25 at 13:04 +0100, Vincenzo Di Massa wrote: > Alle 12:34, venerd? 25 novembre 2005, Paul Moore ha scritto: > > On 11/25/05, Donovan Baarda <abo at minkirri.apana.org.au> wrote: [...] > Please read > http://mail.python.org/pipermail/distutils-sig/2005-November/005520.html > http://mail.python.org/pipermail/distutils-sig/2005-November/005521.html > > Can my proposal resolve the "suckiness". [...] Some of it, yes. In any case, I think it is worthwhile "fixing" eggs so that they consider an installed deb as an egg... this way non-deb eggs installed in /home/ or /usr/local can leverage of already installed debs managed by the Debian Package manager. The phrase "run-time dependency checking" in the first post disturbs me in a not-really-related-to-this-discussion way. It feels wrong; surely dependency checking should be done at install time? Perhaps my perspective is distorted by Debian's excellent install-time dependency checking... The only reasons I can see to do run-time dependency checking is to continue to operate in a degraded mode when an optional dependency is not available. I hope like hell egg's don't do run-time dependency installation... that would be really bad. Installing is a privileged operation. Running is not. -- Donovan Baarda <abo at minkirri.apana.org.au> http://minkirri.apana.org.au/~abo/ From p.f.moore at gmail.com Fri Nov 25 14:39:12 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 25 Nov 2005 13:39:12 +0000 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1132923966.9042.3.camel@warna.corp.google.com> References: <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> <1132914596.28475.48.camel@warna.corp.google.com> <20051125120703.GO29592@syjon.fantastyka.net> <1132923966.9042.3.camel@warna.corp.google.com> Message-ID: <79990c6b0511250539y65dce49ew3a5939debf313bdb@mail.gmail.com> On 11/25/05, Donovan Baarda <abo at minkirri.apana.org.au> wrote: > Sorry... that's what I meant; don't deb an egg... install it as an egg, > outside of the Debian package management system. As an egg, it is under > development and not ready for release as a deb. When the software is > ready for release, strip the egg package management layer and create a > proper deb out of it. Again, this is something I picked up from previous discussions of this nature, but you can't "strip the egg package management layer" without also stripping out the other features of eggs, such as plugin support, metadata introspection, etc, which are useful whether or not you care about package management. So, as a package developer, if I want to use egg metadata or plugin support, I have to buy into the whole egg experience, including the installation stuff. I'd argue that the different features should be separated, so that people can choose the ones they want to use (and as a consequence, the "egg package management layer" could be stripped out independently). But that doesn't seem to be the direction the development is going. Apologies if this is incorrect. I'm still researching... Paul. From hawk78_it at yahoo.it Fri Nov 25 14:49:18 2005 From: hawk78_it at yahoo.it (Vincenzo Di Massa) Date: Fri, 25 Nov 2005 14:49:18 +0100 Subject: [Distutils] [Bulk] Re: formencode as .egg in Debian ?? In-Reply-To: <1132923147.27619.3.camel@silicium.ccc.cea.fr> References: <E1EfaR8-0008DS-00@d.0x1.org> <200511251154.33408.hawk78_it@yahoo.it> <1132923147.27619.3.camel@silicium.ccc.cea.fr> Message-ID: <200511251449.18296.hawk78_it@yahoo.it> Alle 13:52, venerd? 25 novembre 2005, hai scritto: > Le vendredi 25 novembre 2005 ? 11:54 +0100, Vincenzo Di Massa a ?crit : > > Alle 11:04, venerd? 25 novembre 2005, David Arnold ha scritto: > > > But, if compatible versions of those dependencies are already installed > > > as Debian packages *without* egg metadata, will these be ignored? > > > > Yes, they will. > > Why? > > > > Even if it was possible for Debian to extend this downloading mechanism > > > so that it looked for dependencies via apt-get before trying to install > > > from the raw source or egg or whatever, it would usually be the case > > > that the user running the newly downloaded Python application would not > > > have permission to install system packages. > > > > I think easy_install coud be patched by debian packagers (or made > > configurable by Phillip) to add a further check on dependency checking: > > it could check if a dependency is already provided by system packages. > > It could do it by simply trying to import the module. > > > easy_install (and pkg_resources.require()) could have a config file > > telling what code execute to test for dependencies, configurable in a way > > that when I require foo-1.2.3 it can: > > 1) optionally check if apt (or urpmi or anything else) can provide that > > depenedency and, if apt has them, promt the user to either: > > 1.1) run apt-get as root > > 1.2) skip this step ( 1) ) and go to directly to 2) > > No. If the dependency is missing, this is a bug in the Debian package. > No more, no less. > > Furthermore, it is a bad idea for a python script to mess up with the > packaging system. Not only querying it is very slow, but it implies to > make assumptions about the versioning scheme, that can become wrong at > any time. > > > 2) check if it can find this dep like setuptools does today (installing > > them as eggs somewhere in the user's home) > > I consider this as a security risk. I hope this can be disabled by > default. > > Regards, ___________________________________ Yahoo! Mail: gratis 1GB per i messaggi e allegati da 10MB http://mail.yahoo.it From pje at telecommunity.com Fri Nov 25 14:57:18 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 25 Nov 2005 08:57:18 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1132914596.28475.48.camel@warna.corp.google.com> References: <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> <Your message of "Thu, 24 Nov 2005 19:16:44 CDT." <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051125081920.03d61d90@mail.telecommunity.com> At 10:29 AM 11/25/2005 +0000, Donovan Baarda wrote: >On Fri, 2005-11-25 at 01:33 -0500, Phillip J. Eby wrote: >[... long informative explanation of egg...] > >So it sounds like egg duplicates much of the functionality of the Debian >Package manager, but also extends it slightly in python-specific ways. > >I can under stand the Debian packagers asking themselves; do we just >wrap an egg in a Debian package and accept the duplicated functionality, It's not duplicated functionality in normal operation. In the case where a user decides to use easy_install to install something instead of using a Debian package (e.g. because it's not available as one yet), it simply allows them to *also* use Debian packages to satisfy dependencies. If the user wants to pull an up-to-the-minute TurboGears, which uses a mix of other bleeding-edge stuff and stable stuff, it'd be nice for them to be able to use their system packages to satisfy those more stable dependencies. In the case where the user is *not* using easy_install, then all dependencies will be met by system packages, and the only "duplication" is that the paranoid setuptools integrity-check wrappers will want to be assured that yes, everything is still there. >The biggest risk of just wrapping the duplicated functionality is; are >there incompatibilities or policy violations in the egg way of doing >things? > >In particular, will an egg wrapped inside a Debian package magically >install other bits of software not from Debian packages? Will it install >them in the correct places? Not unless the application was specifically written to do this. For example, I believe Trac has a web interface that allows you to upload plugin eggs to the Trac application, which will go in that specific Trac installation's plugin directory. However, there is *no* automatic download-and-install for eggs *except* by running setup.py commands or by using easy_install. Any download-and-install features provided by a particular application or library are that author's responsibility. And I deliberately didn't make it *too* easy to implement that functionality, because I believe people should think carefully about the ramifications of download-and-install without the user explicitly running setup.py or easy_install. In particular, most applications needing such functionality should be downloading into an application-specific plugin directory (which they then add to their sys.path at runtime), and definitely not to system locations. >If TurboGears depends on an "egg'ed" ElementTree, what happens if a >system has ElementTree installed as a non-egg package? Does installing >TurboGears as an egg inside a Debian package require also installing >another ElementTree as an egg inside a Debian package? Or worse, will it >automatically download and install an egg'ed ElementTree not from a >Debian package? Will you end up with two ElementTree's installed, one >egg'ed and one not? If you are running easy_install or setup.py to install TurboGears, and an egged ElementTree is not available, one is going to get installed for you -- because in that scenario you are asking for installation and are presumably doing it to /usr/local or to ~/somewhere. If you are merely *using* TurboGears and an egged version of a dependency is not available, you will get a DistributionNotFound exception at startup or during runtime, depending on how the author defined the dependency and whether it was an optional one. (Optional dependencies allow the package author to define optional features for which additional dependencies are required. Usually this is in the form of self-registered entry points so that the dependencies are checked just before trying to import and invoke the code that implements the feature. The application can then trap any DistributionNotFound or VersionConflict errors and continue without the feature.) For more info about optional features, see: http://peak.telecommunity.com/DevCenter/setuptools#declaring-extras-optional-features-with-their-own-dependencies >Debian has versioned dependencies. If a package depends on a particular >version of another package that doesn't exist yet, you can't install it; >full stop. If you want to package something that depends on a SVN >revision of some dependency, then you need to package that SVN version. Right; the point here is that if somebody is going *outside* of Debian packages and explicitly using setup.py or easy_install to install it, then it would be nice for them to *still* be able to use system packages to meet some of the dependencies, rather than having to re-download, re-build, and re-install new copies of the stuff they already have. This isn't about changing Debian to support bleeding edge things, it's about allowing people doing bleeding edge things to not need to duplicate what Debian already provides. >I think they a both important. If I was using eggs, I'm not sure I'd >want them to be packaged as anything other than an egg until I was ready >to release, and then I'd want it **not** packaged as an egg, as a "proof >of it's release readiness". Well, that would only work if you weren't developing something using entry points to tie an extensible application or framework together. And that didn't use egg metadata for anything else. But if all you used it for was dependencies, then that might make some sense... except that it would also mean that nobody else could *use* your stable version to do *their* development, which is one of the goals here. >It's perhaps unfortunate that egg's bundle package management with >generic meta-data management. It's not really bundled - integrity checking and inter-package linking do not a package manager make. EasyInstall is the (rather primitive) package manager, and it only comes into play if you are manually installing things. For the normal apt-get scenario, EasyInstall isn't involved, with the possible exception of applications like Trac installing plugins into their private plugins directories. > I know that package management uses >meta-data, but if it was implemented as one wrapped around the other, >you could "peel the package management layer off", leaving the metadata >management there, and you could produce clean deb's using only Debian >package management, but with the egg meta-data management still in >place. There's nothing to peel off, unless you believe that it's wrong to have runtime integrity checking. That's all the egg runtime does, unless an application developer codes up something special for their application by wrapping easy_install. When an application says to the runtime, "I need foobar", the runtime just checks if foobar is there. This is useful for optional feature dependencies, for example, and safe integrity checking is definitely needed for apps with plugins (which is what eggs were originally invented for: extensible Python applications like Zope, Trac, Chandler, etc.). Vincenzo's idea about having easy_install be able to wrap apt-get is interesting, in that it would allow apps that want plugin installation to satisfy dependencies via system packages that *aren't* already installed, but in practice I think such applications are going to be restricted to local or specific-user installation by the very nature of their intended uses, so being able to have easy_install get automatically hooked to apt-get doesn't seem necessary to me. In the Debian world, easy_install should probably only be used for user-private and/or system-local non-Debian stuff, including apps' private plugins. (And if you did want easy_install to be able to use apt-get to satisfy dependencies, it probably means you should just be using something like easy_deb instead, to build debs of the stuff you want.) From pje at telecommunity.com Fri Nov 25 15:05:46 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 25 Nov 2005 09:05:46 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <200511251154.33408.hawk78_it@yahoo.it> References: <E1EfaR8-0008DS-00@d.0x1.org> <E1EfaR8-0008DS-00@d.0x1.org> Message-ID: <5.1.1.6.0.20051125085757.033eb008@mail.telecommunity.com> At 11:54 AM 11/25/2005 +0100, Vincenzo Di Massa wrote: >Alle 11:04, venerd? 25 novembre 2005, David Arnold ha scritto: > > But, if compatible versions of those dependencies are already installed > > as Debian packages *without* egg metadata, will these be ignored? > >Yes, they will. > > > Even if it was possible for Debian to extend this downloading mechanism > > so that it looked for dependencies via apt-get before trying to install > > from the raw source or egg or whatever, it would usually be the case > > that the user running the newly downloaded Python application would not > > have permission to install system packages. > >I think easy_install coud be patched by debian packagers (or made >configurable >by Phillip) to add a further check on dependency checking: it could check >if a >dependency is already provided by system packages. This could be done with entry points that are then satisfied by a Debian-speciifc plugin. But, I don't think this should be used to apt-get things, just to check for availability. If you want to fetch stuff as Debian system packages you should use easy_deb in the case where a Debian package isn't already available. However, the problem in creating such a dependency checking plugin is that it *still* needs to map PyPI name+version -> Debian name+version, which is a harder problem, I think, than simply adding the .egg-info files to the Debian dependencies. >Philliph, do you like this idea? It is just adding a hook to dependency >checking where distributors can call ther depency checking tools. That way >the Name+Vesion problem is resolved without putting empty files anywhere. Yeah, except that as you pointed out, you still have to have the mapping exist somewhere as metadata for that hook to read. Which in the simplest case would consist of some sort of flag files installed with the packages... like an .egg-info file. So, it seems like it would be just as easy to do that, and not have to: 1. maintain the hook in setuptools, which in this case is somewhat tricky to bootstrap because the dependency resolver needs to be working before the hook is activated, but the hook would be part of the dependency resolver. So, it's a bit more fragile than the common sort of hook provided by setuptools. 2. have the Debian folks maintaining both a module to hook into this, and a database of package mappings. It seems to me like the simpler solution is just to have the packages install the .egg-info, and nobody has to maintain a mapping database because the setup.py's the .debs are built from already have the info needed, and there's no special code having to be jointly maintained to make the detection work. From hawk78_it at yahoo.it Fri Nov 25 15:09:13 2005 From: hawk78_it at yahoo.it (Vincenzo Di Massa) Date: Fri, 25 Nov 2005 15:09:13 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1132925012.9042.13.camel@warna.corp.google.com> References: <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <200511251304.17984.hawk78_it@yahoo.it> <1132925012.9042.13.camel@warna.corp.google.com> Message-ID: <200511251509.13150.hawk78_it@yahoo.it> Alle 14:23, venerd? 25 novembre 2005, Donovan Baarda ha scritto: > The phrase "run-time dependency checking" in the first post disturbs me > in a not-really-related-to-this-discussion way. It feels wrong; surely > dependency checking should be done at install time? We are talking about unpackaged modules laying in your home. Installation time (from the Debian Point Of View) never occurs to this software. That is why for unpackaged software (there are python suppoerted platforms where software is always unpackaged) you need run-time dependency checking. > Perhaps my perspective is distorted by Debian's excellent install-time > dependency checking... > > The only reasons I can see to do run-time dependency checking is to > continue to operate in a degraded mode when an optional dependency is > not available. Or when you use software that depends (like TG as of today) on not packaged modules. I know one should only use software in debs, but if one wants to use unpackaged softwre, I think Debian could give him the tools to do this without interfering with the Debian installed filesystem. Right? > I hope like hell egg's don't do run-time dependency installation... that > would be really bad. Installing is a privileged operation. Running is > not. The problem are not eggs: they don't download anything. The problem is in the "egg runtime" and in particular in easy_install. Eggs just tell (have metadata describing) what deps they need to work, it is up to an external tool to install them. Vincenzo ___________________________________ Yahoo! Mail: gratis 1GB per i messaggi e allegati da 10MB http://mail.yahoo.it From pje at telecommunity.com Fri Nov 25 15:15:07 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 25 Nov 2005 09:15:07 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <79990c6b0511250334v7602dbb4oc625d28ac2c3d039@mail.gmail.co m> References: <1132914596.28475.48.camel@warna.corp.google.com> <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> <1132914596.28475.48.camel@warna.corp.google.com> Message-ID: <5.1.1.6.0.20051125090603.02034040@mail.telecommunity.com> At 11:34 AM 11/25/2005 +0000, Paul Moore wrote: >On 11/25/05, Donovan Baarda <abo at minkirri.apana.org.au> wrote: > > On thing about this worried me; > > > > If TurboGears depends on an "egg'ed" ElementTree, what happens if a > > system has ElementTree installed as a non-egg package? Does installing > > TurboGears as an egg inside a Debian package require also installing > > another ElementTree as an egg inside a Debian package? Or worse, will it > > automatically download and install an egg'ed ElementTree not from a > > Debian package? Will you end up with two ElementTree's installed, one > > egg'ed and one not? > >Yes. That's the issue I'm not keen on, and I believe your >interpretation is right - TurboGears depends on ElementTree, which is >fine. But *because* TurboGears is an egg [1] it *needs* an egg-ified >version of ElementTree. If I have ElementTree installed already, TG >will *ignore* that, and grab an extra version from the internet (at >install time, presumably, assuning I'm using EasyInstall - but what if >I'm not?) If you're not then you'll just get an error at runtime telling you the integrity checking failed; i.e. a DistributionNotFound error. >just to get it in egg format. More precisely, just to get something it knows for sure is an ElementTree whose version is in the desired range. >[1] It's not exactly because TG is an egg per se, as easy_install and >dependency checking is involved as well, which not all eggs have to >use. I'm slowly coming to understand that eggs (and setuptools, and >easy_install, which are related but subtly different items) provide a >host of largely unrelated functionality, Yep; this is why they have separate web pages. ;-) (And also why some people call it the "eggs/easy_install/setuptools trifecta".) >and discussions are getting >confused because these are not being separated out sufficiently. +1 :) >- what would happen if I downloaded the TurboGears egg and just put it >on sys.path - no easy_install or whatever?) You could import stuff from it, but it might or might not work. Where it explicitly require()'s things or tries to load a plugin, you'll get DependencyNotFound errors. The drop-in-and-go aspect of eggs is mainly used for application plugins in systems like Trac, although in truth they actually use the discovery machinery to scan their plugins directory and identify which plugins are loadable - that is, which plugins can have all their dependencies met. This is one reason that runtime integrity checks are important; an app can detect that it doesn't have what's needed for a given plugin, and skip loading it. This is also why dependency resolution needs to be able to roll back if it fails, even for deeply-nested dependencies. From pje at telecommunity.com Fri Nov 25 15:20:03 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 25 Nov 2005 09:20:03 -0500 Subject: [Distutils] Including package_data in a source distribution In-Reply-To: <1132918548.5285.27.camel@sandwi.ch.cam.ac.uk> Message-ID: <5.1.1.6.0.20051125091516.033a5350@mail.telecommunity.com> At 11:35 AM 11/25/2005 +0000, Noel O'Boyle wrote: >I have been trying to use either distutils or setuptools to include some >package data in a source distribution ("python setup.py sdist") using >the package_data option. It *is* included if I use "python setup.py >bdist_egg". > >Can someone explain to me whether this is a feature or a bug? The >documentation led me to expect that this would work, but it does not. I >am using the example on >http://www.python.org/dev/doc/devel/dist/node12.html. Actually, if you look at the documentation for the "sdist" command, you'll see that it does not support automatically including package_data in source distributions; you have to add it to the MANIFEST.in. If you are using the latest Subversion revision of setuptools, you can specify 'include_package_data=True' instead of listing individual package data files, and any data files under CVS or SVN source control will automatically be added to the package data for you. If you are not using CVS or SVN, you can instead add the files to your MANIFEST.in, and then setuptools will pick them up from there. If you're using an older version of setuptools, you can use the package_data form, and if the data files are under CVS or SVN control, they'll be included in your source distribution. If you're using distutils, you'll have to duplicate the info in MANIFEST.in. From pje at telecommunity.com Fri Nov 25 15:23:04 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 25 Nov 2005 09:23:04 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <20051125120703.GO29592@syjon.fantastyka.net> References: <1132914596.28475.48.camel@warna.corp.google.com> <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> <1132914596.28475.48.camel@warna.corp.google.com> Message-ID: <5.1.1.6.0.20051125092036.02018450@mail.telecommunity.com> At 01:07 PM 11/25/2005 +0100, Janusz A. Urbanowicz wrote: >On Fri, Nov 25, 2005 at 10:29:56AM +0000, Donovan Baarda wrote: > > On Fri, 2005-11-25 at 01:33 -0500, Phillip J. Eby wrote: > > [... long informative explanation of egg...] > > In particular, will an egg wrapped inside a Debian package magically > > install other bits of software not from Debian packages? Will it install > > them in the correct places? > >This is a dangerous practice from ore than one point of view: > >1) it may pollute the system with non-DFSG-compliant stuff > >2) as a both python developer and debian user and developer I DO NOT want >software to download and run stuff without my knowledge and explicit consent It does neither; you have to explicitly be using easy_install or setup.py to get any download-and-run behavior. Now, it's possible for an individual coder to write an application or library that invokes easy_install itself, but anybody can write bad code and that's what you have a QA process for, no? From hawk78_it at yahoo.it Fri Nov 25 15:23:15 2005 From: hawk78_it at yahoo.it (Vincenzo Di Massa) Date: Fri, 25 Nov 2005 15:23:15 +0100 Subject: [Distutils] [Bulk] Re: formencode as .egg in Debian ?? In-Reply-To: <200511251449.18296.hawk78_it@yahoo.it> References: <E1EfaR8-0008DS-00@d.0x1.org> <1132923147.27619.3.camel@silicium.ccc.cea.fr> <200511251449.18296.hawk78_it@yahoo.it> Message-ID: <200511251523.15545.hawk78_it@yahoo.it> (Sorry for the prevoius empty post) Alle 13:52, venerd? 25 novembre 2005, hai scritto: > Le vendredi 25 novembre 2005 ? 11:54 +0100, Vincenzo Di Massa a ?crit : > > Alle 11:04, venerd? 25 novembre 2005, David Arnold ha scritto: > > > But, if compatible versions of those dependencies are already installed > > > as Debian packages *without* egg metadata, will these be ignored? > > > > Yes, they will. > > Why? The reason is explained by Phillip in the previous e-mails. > > > Even if it was possible for Debian to extend this downloading mechanism > > > so that it looked for dependencies via apt-get before trying to install > > > from the raw source or egg or whatever, it would usually be the case > > > that the user running the newly downloaded Python application would not > > > have permission to install system packages. > > > > I think easy_install coud be patched by debian packagers (or made > > configurable by Phillip) to add a further check on dependency checking: > > it could check if a dependency is already provided by system packages. > > It could do it by simply trying to import the module. But it will not know about the version of the module that is imported! I know we who are using debian packaged modules don't care to have this checks at runtime. But there are circumstances where it is not possible/wanted to install a debian packaged python module (maybe becouse: you have not root access, you want to install a personal/test/unpackaged version in your home... you are not a debian user, (thus you are using something that is inferior to Debian ;-) ). Python developers need a way to check deps at runtime becose python MUST work on platforms inferior to debian. So they put and will put code in theyr projects that does this: check for versin X of modlule foo. Distributors (where vensions are already taken care of) can safely remove that checks (monky patching every upcoming python module released). But if they don't remove the checks (and simply manage them in a custom and wise way) they will allow thir users to install local copies of new/unpackages software that plays well with packaged modules (and the don't need to pathc evey(maybe) single new python app!). This seems a *real benefit*, no? > > easy_install (and pkg_resources.require()) could have a config file > > telling what code execute to test for dependencies, configurable in a way > > that when I require foo-1.2.3 it can: > > 1) optionally check if apt (or urpmi or anything else) can provide that > > depenedency and, if apt has them, promt the user to either: > > 1.1) run apt-get as root > > 1.2) skip this step ( 1) ) and go to directly to 2) > > No. If the dependency is missing, this is a bug in the Debian package. > No more, no less. You are not understanding... I'm talking about unpackaged modules. So there is no buggy package: there is no package at all, just a freshly downloaded (from somewere) python app or lib that you want to run from (for example) your home. Is it desiderable that an user can test a new app by just downloading the python source and installing it in his home using easy_install without the need to also install in his home the dependencies debian already satisfies correctly . No? > Furthermore, it is a bad idea for a python script to mess up with the > packaging system. In fact I'm not proposing this! The python script will just tell the user to run apt-get as root if needed. > Not only querying it is very slow, but it implies to > make assumptions about the versioning scheme, that can become wrong at > any time. The query can be cached, or the post-install script could populate a fastly searcheble (module,version) database of installed modules. It is not so difficolt, is it? > > 2) check if it can find this dep like setuptools does today (installing > > them as eggs somewhere in the user's home) > > I consider this as a security risk. I hope this can be disabled by > default. That ha nothing to do with the system: this fallback will just execute for software that is not already provided by Debian. So you are not going to accidentally install something becouse of this. Moreover the user is expected to insert the path where the fallback mecanism can install deps as egg. That could be similar to a make install in C world: if you don't specify a destination things go to /usr/local. > Regards, Vincenzo ___________________________________ Yahoo! Mail: gratis 1GB per i messaggi e allegati da 10MB http://mail.yahoo.it From pje at telecommunity.com Fri Nov 25 15:28:03 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 25 Nov 2005 09:28:03 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1132923147.27619.3.camel@silicium.ccc.cea.fr> References: <200511251154.33408.hawk78_it@yahoo.it> <E1EfaR8-0008DS-00@d.0x1.org> <200511251154.33408.hawk78_it@yahoo.it> Message-ID: <5.1.1.6.0.20051125092318.02034c48@mail.telecommunity.com> At 01:52 PM 11/25/2005 +0100, Josselin Mouette wrote: >It could do it by simply trying to import the module. Python's import mechanism is fragile in the presence of failed imports that aren't explicitly trapped, and in Python 2.3 it's even more fragile because only the *first* attempt to import a module that has a missing dependency will fail; subsequent imports of that module will return a broken module. Diagnosing such problems is non-trivial. > > 2) check if it can find this dep like setuptools does today (installing > them > > as eggs somewhere in the user's home) > >I consider this as a security risk. I hope this can be disabled by >default. It *is*. You have to explicitly run easy_install or setup.py to get any installation of any software, or use an explicit application feature like Trac's "upload a plugin" facility. And I don't agree with Vincenzo's proposal to make easy_install run apt-get in any case. If you want to fetch debs to satisfy a non-Debian Python project, you should use easy_deb. Using easy_install only makes sense if you are using bleeding-edge stuff installed to your home directory or a special project directory, so you don't want to fetch any debs in that case because you're not updating the *system* packages. From hawk78_it at yahoo.it Fri Nov 25 15:30:24 2005 From: hawk78_it at yahoo.it (Vincenzo Di Massa) Date: Fri, 25 Nov 2005 15:30:24 +0100 Subject: [Distutils] [Bulk] Re: formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051125085757.033eb008@mail.telecommunity.com> References: <E1EfaR8-0008DS-00@d.0x1.org> <5.1.1.6.0.20051125085757.033eb008@mail.telecommunity.com> Message-ID: <200511251530.24612.hawk78_it@yahoo.it> Alle 15:05, venerd? 25 novembre 2005, Phillip J. Eby ha scritto: > At 11:54 AM 11/25/2005 +0100, Vincenzo Di Massa wrote: > >Alle 11:04, venerd? 25 novembre 2005, David Arnold ha scritto: > > > But, if compatible versions of those dependencies are already installed > > > as Debian packages *without* egg metadata, will these be ignored? > > > >Yes, they will. > > > > > Even if it was possible for Debian to extend this downloading mechanism > > > so that it looked for dependencies via apt-get before trying to install > > > from the raw source or egg or whatever, it would usually be the case > > > that the user running the newly downloaded Python application would not > > > have permission to install system packages. > > > >I think easy_install coud be patched by debian packagers (or made > >configurable > >by Phillip) to add a further check on dependency checking: it could check > >if a > >dependency is already provided by system packages. > > This could be done with entry points that are then satisfied by a > Debian-speciifc plugin. But, I don't think this should be used to apt-get > things, just to check for availability. Right! What I propose is to check and report the result to the user. Like -->you must first install module foo typing "apt-get install python-foo" > If you want to fetch stuff as > Debian system packages you should use easy_deb in the case where a Debian > package isn't already available. > > However, the problem in creating such a dependency checking plugin is that > it *still* needs to map PyPI name+version -> Debian name+version, which is > a harder problem, I think, than simply adding the .egg-info files to the > Debian dependencies. Debian folks are happy to do hard work if that helps to develop a clean and well done Distribution. Right? > >Philliph, do you like this idea? It is just adding a hook to dependency > >checking where distributors can call ther depency checking tools. That way > >the Name+Vesion problem is resolved without putting empty files anywhere. > > Yeah, except that as you pointed out, you still have to have the mapping > exist somewhere as metadata for that hook to read. Which in the simplest > case would consist of some sort of flag files installed with the > packages... like an .egg-info file. So, it seems like it would be just as > easy to do that, and not have to: > > 1. maintain the hook in setuptools, which in this case is somewhat tricky > to bootstrap because the dependency resolver needs to be working before the > hook is activated, but the hook would be part of the dependency > resolver. So, it's a bit more fragile than the common sort of hook > provided by setuptools. > > 2. have the Debian folks maintaining both a module to hook into this, and a > database of package mappings. > > It seems to me like the simpler solution is just to have the packages > install the .egg-info, and nobody has to maintain a mapping database > because the setup.py's the .debs are built from already have the info > needed, and there's no special code having to be jointly maintained to make > the detection work. The simpler solution has pitfalls consisting of: 1) you need rebuilding every single python, rpm, tgz... package 2) as you told it has problems with namespace packages that coud be resolved providing more hooks ( http://mail.python.org/pipermail/distutils-sig/2005-November/005542.html ) 2) you force a layout of module installation when it is not strictly needed 3) you must provide functionality to distributors when they need (they are not encouraged to "subclass" your project and do what they want) (what when fedora or mandriva disagree with you or with Debian? will you add more and more platform specific --do-this-if-you-are-on-Debian-Sarge-with-backports and --do-this-if-you-are-on-Fedora-3.0?) :-) The best property of my proposal is IMHO the othogonality thing! Regards (and thanks) Vincenzo ___________________________________ Yahoo! Mail: gratis 1GB per i messaggi e allegati da 10MB http://mail.yahoo.it From no228 at cam.ac.uk Fri Nov 25 15:34:23 2005 From: no228 at cam.ac.uk (Noel O'Boyle) Date: Fri, 25 Nov 2005 14:34:23 +0000 Subject: [Distutils] Including package_data in a source distribution In-Reply-To: <5.1.1.6.0.20051125091516.033a5350@mail.telecommunity.com> References: <5.1.1.6.0.20051125091516.033a5350@mail.telecommunity.com> Message-ID: <1132929263.5285.52.camel@sandwi.ch.cam.ac.uk> On Fri, 2005-11-25 at 09:20 -0500, Phillip J. Eby wrote: > At 11:35 AM 11/25/2005 +0000, Noel O'Boyle wrote: > >I have been trying to use either distutils or setuptools to include some > >package data in a source distribution ("python setup.py sdist") using > >the package_data option. It *is* included if I use "python setup.py > >bdist_egg". > > > >Can someone explain to me whether this is a feature or a bug? The > >documentation led me to expect that this would work, but it does not. I > >am using the example on > >http://www.python.org/dev/doc/devel/dist/node12.html. > > Actually, if you look at the documentation for the "sdist" command, you'll > see that it does not support automatically including package_data in source > distributions; you have to add it to the MANIFEST.in. Thank you for the pointer. However this means that earlier pages are misleading. It would be useful if the documentation on package_data and data_files made it clear that it doesn't work for sdist, and gave a link to the sdist page with its completely different method. Maybe I'm an idiot, but I spent quite some time trying every possible combination of package_data and data_files, while sdist ignored them silently. > If you are using the latest Subversion revision of setuptools, you can > specify 'include_package_data=True' instead of listing individual package > data files, and any data files under CVS or SVN source control will > automatically be added to the package data for you. If you are not using > CVS or SVN, you can instead add the files to your MANIFEST.in, and then > setuptools will pick them up from there. > > If you're using an older version of setuptools, you can use the > package_data form, and if the data files are under CVS or SVN control, > they'll be included in your source distribution. > > If you're using distutils, you'll have to duplicate the info in MANIFEST.in. Thanks again, Noel From pje at telecommunity.com Fri Nov 25 15:36:43 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 25 Nov 2005 09:36:43 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <79990c6b0511250539y65dce49ew3a5939debf313bdb@mail.gmail.co m> References: <1132923966.9042.3.camel@warna.corp.google.com> <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> <1132914596.28475.48.camel@warna.corp.google.com> <20051125120703.GO29592@syjon.fantastyka.net> <1132923966.9042.3.camel@warna.corp.google.com> Message-ID: <5.1.1.6.0.20051125092931.034323f0@mail.telecommunity.com> At 01:39 PM 11/25/2005 +0000, Paul Moore wrote: >On 11/25/05, Donovan Baarda <abo at minkirri.apana.org.au> wrote: > > Sorry... that's what I meant; don't deb an egg... install it as an egg, > > outside of the Debian package management system. As an egg, it is under > > development and not ready for release as a deb. When the software is > > ready for release, strip the egg package management layer and create a > > proper deb out of it. > >Again, this is something I picked up from previous discussions of this >nature, but you can't "strip the egg package management layer" without >also stripping out the other features of eggs, such as plugin support, >metadata introspection, etc, which are useful whether or not you care >about package management. > >So, as a package developer, if I want to use egg metadata or plugin >support, I have to buy into the whole egg experience, including the >installation stuff. Actually, I thought you wanted eggs to be wrapped in system packages, so that you don't have to use the "installation stuff". That is, after all, what this extended thread is about: providing system packages of egg projects, and allowing egg dependencies to be met by system packages. >I'd argue that the different features should be separated, so that >people can choose the ones they want to use (and as a consequence, the >"egg package management layer" could be stripped out independently). >But that doesn't seem to be the direction the development is going. > >Apologies if this is incorrect. I'm still researching... The issue is integrity checking. Python imports as an integrity check are not safe in the general case, *especially* in Python 2.3, where a failed nested import results in an importable but broken module. Integrity checks are also essential for plugin support, since many plugins will probably *not* be system-managed, so the app has to be able to tell if it can load the plugin or not. A plugin, however, may have dependencies that can be satisfied by system packages, like if the plugin uses ReportLab or something to add PDF support to an application's export menu. That's why you can't really just "peel off" the egg shell here; the whole stack was designed to support quality plugin systems for Python apps, similar to the Eclipse or OSGi systems in the Java world. From smurf at smurf.noris.de Fri Nov 25 15:54:37 2005 From: smurf at smurf.noris.de (Matthias Urlichs) Date: Fri, 25 Nov 2005 15:54:37 +0100 Subject: [Distutils] [Bulk] Re: formencode as .egg in Debian ?? In-Reply-To: <200511251530.24612.hawk78_it@yahoo.it> References: <E1EfaR8-0008DS-00@d.0x1.org> <5.1.1.6.0.20051125085757.033eb008@mail.telecommunity.com> <200511251530.24612.hawk78_it@yahoo.it> Message-ID: <20051125145437.GD20418@kiste.smurf.noris.de> Hi, Vincenzo Di Massa: > 2) you force a layout of module installation when it is not strictly needed Umm... the layout problem has been resolved by now. -- Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf at smurf.noris.de Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de - - Hello. Jim Rockford's machine, this is Larry Doheny's machine. Will you please have your master call my master at his convenience? Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. -- "The Rockford Files" From alex at bofh.net.pl Fri Nov 25 16:22:53 2005 From: alex at bofh.net.pl (Janusz A. Urbanowicz) Date: Fri, 25 Nov 2005 16:22:53 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051125092036.02018450@mail.telecommunity.com> References: <1132914596.28475.48.camel@warna.corp.google.com> <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> <1132914596.28475.48.camel@warna.corp.google.com> <5.1.1.6.0.20051125092036.02018450@mail.telecommunity.com> Message-ID: <20051125152252.GP29592@syjon.fantastyka.net> On Fri, Nov 25, 2005 at 09:23:04AM -0500, Phillip J. Eby wrote: > At 01:07 PM 11/25/2005 +0100, Janusz A. Urbanowicz wrote: > >On Fri, Nov 25, 2005 at 10:29:56AM +0000, Donovan Baarda wrote: > >> On Fri, 2005-11-25 at 01:33 -0500, Phillip J. Eby wrote: > >> [... long informative explanation of egg...] > >> In particular, will an egg wrapped inside a Debian package magically > >> install other bits of software not from Debian packages? Will it install > >> them in the correct places? > > > >This is a dangerous practice from ore than one point of view: > > > >1) it may pollute the system with non-DFSG-compliant stuff > > > >2) as a both python developer and debian user and developer I DO NOT want > >software to download and run stuff without my knowledge and explicit > >consent > > It does neither; you have to explicitly be using easy_install or setup.py > to get any download-and-run behavior. > > Now, it's possible for an individual coder to write an application or > library that invokes easy_install itself, but anybody can write bad code > and that's what you have a QA process for, no? Yes and no; malicious code of this kind gets eventually weed out _after a while_, but this is wrong by design, not to be corrected by QA process. -- mors ab alto 0x46399138 From pje at telecommunity.com Fri Nov 25 16:28:53 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 25 Nov 2005 10:28:53 -0500 Subject: [Distutils] [Bulk] Re: formencode as .egg in Debian ?? In-Reply-To: <200511251530.24612.hawk78_it@yahoo.it> References: <5.1.1.6.0.20051125085757.033eb008@mail.telecommunity.com> <E1EfaR8-0008DS-00@d.0x1.org> <5.1.1.6.0.20051125085757.033eb008@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051125100225.034468c0@mail.telecommunity.com> At 03:30 PM 11/25/2005 +0100, Vincenzo Di Massa wrote: >Alle 15:05, venerd? 25 novembre 2005, Phillip J. Eby ha scritto: > > It seems to me like the simpler solution is just to have the packages > > install the .egg-info, and nobody has to maintain a mapping database > > because the setup.py's the .debs are built from already have the info > > needed, and there's no special code having to be jointly maintained to make > > the detection work. > >The simpler solution has pitfalls consisting of: >1) you need rebuilding every single python, rpm, tgz... package Only those, such as ElementTree, that are depended on by a setuptools-based package, such as TurboGears. While this will be increasingly widespread over time, that doesn't mean that Debian needs to rebuild all those packages *now*, unless that's what is desired. >2) as you told it has problems with namespace packages that coud be resolved >providing more hooks >( http://mail.python.org/pipermail/distutils-sig/2005-November/005542.html ) Your approach doesn't fix this; it's a fundamental problem with the standard distutils package layout. The only simple fix for namespace packages is to use .egg directories for projects that use them. If that's not acceptable, namespace packages will need to use the "dummy Debian package for the __init__.py" approach. Currently, only PEAK, Paste, and maybe some of the LivingLogic packages actually use namespace packages in the field, so far as I know. Zope 3 makes heavy use of namespace packages, but is still in the early stages of moving to eggs, so I'm pretty sure they don't have any eggs with namespace packages yet. I believe they are only using eggs for Zope 2 plugins so far, not actually distributing Zope 3 or Zope itself that way. So, I don't believe namespace packages are an immediate issue for Debian, but they might be soon, depending on whether TurboGears starts using Paste packages. I believe TurboGears 0.9 or 1.0 might have some plans to leverage PasteDeploy for creating applications that will be launchable by WSGI servers. >3) you must provide functionality to distributors when they need (they are >not >encouraged to "subclass" your project and do what they want) (what when >fedora or mandriva disagree with you or with Debian? will you add more and >more platform specific --do-this-if-you-are-on-Debian-Sarge-with-backports >and --do-this-if-you-are-on-Fedora-3.0?) :-) Actually, I'm increasingly tempted to PEP the idea of a name-version.pkg-info file being installed alongside the code as a standard Python distutils feature. This discussion has made it clearer (even to me!) how useful such a thing would be, and if it became a standard part of the distutils, it would simply become part of Debian and every other system packager unless explicitly removed. Such a thing would compete with PEP 262, but PEP 262 also seems like more competition with system package managers than this idea is. Arguably, normal Python code doesn't need all the extra baggage in the PEP 262 database; the PKG-INFO part alone is sufficient when there's an existing system package manager. And if you don't have a system package manager, using eggs solves most of the other problems PEP 262 was intended to address. Perhaps it's worth a show of hands on Distutils-SIG, as far as whether there are any people (Marc-Andre? Martin? Paul?) who are outright opposed to making .pkg-info files be a standard feature of distutils installation, as a simpler substitute for the sketchy state of PEP 262. If there's nobody actively opposed, it might well be worth PEPping. At that point, it would be an easier sell to package system maintainers, in that it would be expected for Python 2.5, and courteously requested as an advance feature with current Pythons to support leading-edge software development using Python. (Note that such a PEP would *not* be an endorsement of any .egg format, setuptools, or the concept of eggs in general, which is why I would propose .pkg-info rather than .egg-info there, in order to avoid any premature implication that eggs are a standard Python feature. Changing setuptools to support the alternate naming is a trivial matter with negligible performance or other impacts.) From pje at telecommunity.com Fri Nov 25 16:37:51 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 25 Nov 2005 10:37:51 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <20051125152252.GP29592@syjon.fantastyka.net> References: <5.1.1.6.0.20051125092036.02018450@mail.telecommunity.com> <1132914596.28475.48.camel@warna.corp.google.com> <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> <1132914596.28475.48.camel@warna.corp.google.com> <5.1.1.6.0.20051125092036.02018450@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051125102920.03410da8@mail.telecommunity.com> At 04:22 PM 11/25/2005 +0100, Janusz A. Urbanowicz wrote: >On Fri, Nov 25, 2005 at 09:23:04AM -0500, Phillip J. Eby wrote: > > Now, it's possible for an individual coder to write an application or > > library that invokes easy_install itself, but anybody can write bad code > > and that's what you have a QA process for, no? > >Yes and no; malicious code of this kind gets eventually weed out _after a >while_, but this is wrong by design, not to be corrected by QA process. I'm just pointing out that *I* can't control what some arbitrary author chooses to write. But the libraries that *I* wrote are not going to download and install something automatically unless the user ran easy_install or a setup.py file. Also, there is a trivial way to "opt out" of any such automatic downloading that some arbitrary author may write; the configuration files for easy_install can include allow_hosts=localhost, which will prevent easy_install from downloading any URL that's spelled with a host name other than localhost. You can also use wildcard patterns to set hostname masks for acceptable download locations. Of course, an application author can override the options set by the config files, or write their own downloading tools. I'm just pointing out that easy_install does allow a user to place restrictions on where packages can be obtained from (and thereby whether they can be obtained at all), even when the user has explicitly chosen to run easy_install or a setup.py. From alex at bofh.net.pl Fri Nov 25 17:04:40 2005 From: alex at bofh.net.pl (Janusz A. Urbanowicz) Date: Fri, 25 Nov 2005 17:04:40 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051125102920.03410da8@mail.telecommunity.com> References: <5.1.1.6.0.20051125092036.02018450@mail.telecommunity.com> <1132914596.28475.48.camel@warna.corp.google.com> <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> <1132914596.28475.48.camel@warna.corp.google.com> <5.1.1.6.0.20051125092036.02018450@mail.telecommunity.com> <5.1.1.6.0.20051125102920.03410da8@mail.telecommunity.com> Message-ID: <20051125160440.GQ29592@syjon.fantastyka.net> On Fri, Nov 25, 2005 at 10:37:51AM -0500, Phillip J. Eby wrote: > At 04:22 PM 11/25/2005 +0100, Janusz A. Urbanowicz wrote: > >On Fri, Nov 25, 2005 at 09:23:04AM -0500, Phillip J. Eby wrote: > >> Now, it's possible for an individual coder to write an application or > >> library that invokes easy_install itself, but anybody can write bad code > >> and that's what you have a QA process for, no? > > > >Yes and no; malicious code of this kind gets eventually weed out _after a > >while_, but this is wrong by design, not to be corrected by QA process. > > I'm just pointing out that *I* can't control what some arbitrary author > chooses to write. But the libraries that *I* wrote are not going to > download and install something automatically unless the user ran > easy_install or a setup.py file. > > Also, there is a trivial way to "opt out" of any such automatic downloading > that some arbitrary author may write; the configuration files for > easy_install can include allow_hosts=localhost, which will prevent > easy_install from downloading any URL that's spelled with a host name other > than localhost. You can also use wildcard patterns to set hostname masks > for acceptable download locations. > > Of course, an application author can override the options set by the config > files, or write their own downloading tools. I'm just pointing out that > easy_install does allow a user to place restrictions on where packages can > be obtained from (and thereby whether they can be obtained at all), even > when the user has explicitly chosen to run easy_install or a setup.py. The more interesting question is that it is possible to accept only SSL sites with proper certificate credentials. From the proper security point of view, this should be the other way, disabled by default and enabled on per event and per host basis. Opt out is evil in any circumstances, it should be always opt in, and not too easy to do so. -- mors ab alto 0x46399138 From pje at telecommunity.com Fri Nov 25 17:08:05 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 25 Nov 2005 11:08:05 -0500 Subject: [Distutils] [Bulk] Re: formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051125100225.034468c0@mail.telecommunity.com> References: <200511251530.24612.hawk78_it@yahoo.it> <5.1.1.6.0.20051125085757.033eb008@mail.telecommunity.com> <E1EfaR8-0008DS-00@d.0x1.org> <5.1.1.6.0.20051125085757.033eb008@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051125110130.01fd2da0@mail.telecommunity.com> At 10:28 AM 11/25/2005 -0500, Phillip J. Eby wrote: >Actually, I'm increasingly tempted to PEP the idea of a >name-version.pkg-info file being installed alongside the code as a standard >Python distutils feature. This discussion has made it clearer (even to >me!) how useful such a thing would be, and if it became a standard part of >the distutils, it would simply become part of Debian and every other system >packager unless explicitly removed. Also, it just occurred to me that this would ultimately fix one of Paul Moore's complaints about eggs as well. If bdist_wininst included this .pkg-info, then even Windows "managed" packages would be supported in this scheme, and setuptools could include a bdist_wininst that made similar .exe's to wrap eggs. (Of course, the full solution for Paul's issue wouldn't arrive until Python 2.5 in that case, unless the people doing the .exe packaging are using some special tool to add the .pkg-info to non-setuptools .exe's.) From p.f.moore at gmail.com Fri Nov 25 17:19:13 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 25 Nov 2005 16:19:13 +0000 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051125090603.02034040@mail.telecommunity.com> References: <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> <1132914596.28475.48.camel@warna.corp.google.com> <5.1.1.6.0.20051125090603.02034040@mail.telecommunity.com> Message-ID: <79990c6b0511250819s7480a899pb030dc732a0269ed@mail.gmail.com> On 11/25/05, Phillip J. Eby <pje at telecommunity.com> wrote: > At 11:34 AM 11/25/2005 +0000, Paul Moore wrote: > >- what would happen if I downloaded the TurboGears egg and just put it > >on sys.path - no easy_install or whatever?) > > You could import stuff from it, but it might or might not work. Where it > explicitly require()'s things or tries to load a plugin, you'll get > DependencyNotFound errors. OK, I made an assumption which I wasn't clear about here. Assume I've done what I currently do, and installed all of its dependencies. Will it work now? If not, why not? *As a user* I want to opt out of the easy-install experience (you state elsewhere "EasyInstall is the (rather primitive) package manager..." - and I'm happy sticking with the equally primitive Windows package management approach). Actually, I think I know the answer (although I've thought that before as well :-)) - it will work, as long as I didn't screw up with my manual dependency management. A screw-up equates to me getting a DependencyNotFound error. If that's correct, then I'm more or less fine with it. But... To install a non-egg Python package is just a matter of: 1. Running a bdist_wininst exe (or apt-get on Debian, etc), or 2. Dumping the package directory in site-packages (Debian folks, please don't scream :-)) For an egg, like TurboGears, my options (assuming I don't want to use easy_install) are: 1. Wait for someone to wrap the TG egg in a system package, or 2. Put the egg somewhere (likely site-packages), then create or modify a .pth file by hand to add the egg to sys.path. In both cases, (1) is the managed option, and (2) is the unmanaged one. And (2) is *harder* for eggs. (1) is just as easy for me either way, but I assume that's just because the extra difficulty is passed onto the developer of the system packaging process. I find this jarring, because eggs are presented as a "zero installation" format. And I'd be curious to know what sort of reaction I'd get from the TG community if I posted a message saying I'd dumped various eggs into site-packages, manually fiddled with .pth files, and TG wasn't working. :-) > The drop-in-and-go aspect of eggs is mainly used for application plugins in > systems like Trac, although in truth they actually use the discovery > machinery to scan their plugins directory and identify which plugins are > loadable - that is, which plugins can have all their dependencies met. Yes. Plugins are a clear benefit. I don't think *anyone* is arguing that this is an issue. > This is one reason that runtime integrity checks are important; an app can > detect that it doesn't have what's needed for a given plugin, and skip > loading it. This is also why dependency resolution needs to be able to > roll back if it fails, even for deeply-nested dependencies. OK. And I see that a plugin can need a system library (a Trac "progress chart" plugin needing PIL installed, for example). And non-egg installs can't provide that discovery mechanism directly. OK. You've got that covered (as best you can) with the single egg-info file option. (From another posting) >>So, as a package developer, if I want to use egg metadata or plugin >>support, I have to buy into the whole egg experience, including the >>installation stuff. >Actually, I thought you wanted eggs to be wrapped in system packages, so >that you don't have to use the "installation stuff". That is, after all, >what this extended thread is about: providing system packages of egg >projects, and allowing egg dependencies to be met by system packages. Different contexts. If I'm *developing* a package (not something I do often) I can see the benefits of metadata, package resources, etc. I don't care one way or the other about the installation side - but it comes as part of the whole bundle, for better or worse. As a *user* of other people's packages, I want to use system managed packages. As a *Windows* user, I'm a pain, because I expect the developer to supply that system managed installer, rather than the suppliers of my distribution. (and round full circle) As a *developer* again, I don't have a way of satisfying my Windows user's requirement for a wrapped version of my egg. It's somewhat like the old days, when python setup.py install existed, but bdist_wininst didn't. Windows users had to take pot luck as to whether a particular package would be available to them (because they couldn't use the "standard" setup.py install when C extensions were involved). But while I'm only developing packages for my own use, it's pointless building fancy infrastructure. And clearly people like the TurboGears developers, who are aiming at a significant user base, don't have many users asking for Windows installers at the moment, so things will stay as they are for the moment. [An only vaguely related question - is the feature of generating wrappers for scripts based on entry points an easy_install feature, or an egg feature? If so, what happens to it with a system package wrapping the egg? I ask because I see that Kid has swiched from bdist_wininst to egg as its distribution format, as far as I can tell, just for the script-wrapper feature - so can I use the manual option (2) above to install Kid???] Must stop reading and replying to posts, and do the research... Paul. From pje at telecommunity.com Fri Nov 25 17:27:38 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 25 Nov 2005 11:27:38 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <20051125160440.GQ29592@syjon.fantastyka.net> References: <5.1.1.6.0.20051125102920.03410da8@mail.telecommunity.com> <5.1.1.6.0.20051125092036.02018450@mail.telecommunity.com> <1132914596.28475.48.camel@warna.corp.google.com> <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> <1132914596.28475.48.camel@warna.corp.google.com> <5.1.1.6.0.20051125092036.02018450@mail.telecommunity.com> <5.1.1.6.0.20051125102920.03410da8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051125112437.01fe4730@mail.telecommunity.com> At 05:04 PM 11/25/2005 +0100, Janusz A. Urbanowicz wrote: >Opt out is evil in any circumstances, it should be always opt in, and not >too easy to do so. Um, running setup.py or easy_install *is* opting in. I was just pointing out that there are ways to restrict what you're opting in *to*, and that you can also curb the enthusiasm of a too-helpful author who's embedded a call to easy_install in their program. From p.f.moore at gmail.com Fri Nov 25 17:33:45 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 25 Nov 2005 16:33:45 +0000 Subject: [Distutils] [Bulk] Re: formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051125110130.01fd2da0@mail.telecommunity.com> References: <200511251530.24612.hawk78_it@yahoo.it> <E1EfaR8-0008DS-00@d.0x1.org> <5.1.1.6.0.20051125085757.033eb008@mail.telecommunity.com> <5.1.1.6.0.20051125100225.034468c0@mail.telecommunity.com> <5.1.1.6.0.20051125110130.01fd2da0@mail.telecommunity.com> Message-ID: <79990c6b0511250833y6d5d28c5v79b4f3c9d9d75f81@mail.gmail.com> On 11/25/05, Phillip J. Eby <pje at telecommunity.com> wrote: > At 10:28 AM 11/25/2005 -0500, Phillip J. Eby wrote: > >Actually, I'm increasingly tempted to PEP the idea of a > >name-version.pkg-info file being installed alongside the code as a standard > >Python distutils feature. This discussion has made it clearer (even to > >me!) how useful such a thing would be, and if it became a standard part of > >the distutils, it would simply become part of Debian and every other system > >packager unless explicitly removed. > > Also, it just occurred to me that this would ultimately fix one of Paul > Moore's complaints about eggs as well. If bdist_wininst included this > .pkg-info, then even Windows "managed" packages would be supported in this > scheme, That would be a major bonus for me, certainly. > and setuptools could include a bdist_wininst that made similar > .exe's to wrap eggs. That would be the other half of the equation. I had avoided suggesting precisely that, because it's daft of me to expect you to do this work just to stop me moaning (and installer-writing is enough of a black art to me that I can't assume it's an easy change to the existing bdist_wininst code). > (Of course, the full solution for Paul's issue wouldn't arrive until Python > 2.5 in that case, unless the people doing the .exe packaging are using some > special tool to add the .pkg-info to non-setuptools .exe's.) No problem for me, 2.5 would be a sensible timescale. Paul. From pje at telecommunity.com Fri Nov 25 17:46:58 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 25 Nov 2005 11:46:58 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <79990c6b0511250819s7480a899pb030dc732a0269ed@mail.gmail.co m> References: <5.1.1.6.0.20051125090603.02034040@mail.telecommunity.com> <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> <1132914596.28475.48.camel@warna.corp.google.com> <5.1.1.6.0.20051125090603.02034040@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051125112951.03293bc8@mail.telecommunity.com> [debian folks trimmed from the cc: list since this is mainly Windows-related] At 04:19 PM 11/25/2005 +0000, Paul Moore wrote: >On 11/25/05, Phillip J. Eby <pje at telecommunity.com> wrote: > > At 11:34 AM 11/25/2005 +0000, Paul Moore wrote: > > >- what would happen if I downloaded the TurboGears egg and just put it > > >on sys.path - no easy_install or whatever?) > > > > You could import stuff from it, but it might or might not work. Where it > > explicitly require()'s things or tries to load a plugin, you'll get > > DependencyNotFound errors. > >OK, I made an assumption which I wasn't clear about here. Assume I've >done what I currently do, and installed all of its dependencies. > >Will it work now? If not, why not? See what I said above; It answers both questions. >Actually, I think I know the answer (although I've thought that before >as well :-)) - it will work, as long as I didn't screw up with my >manual dependency management. A screw-up equates to me getting a >DependencyNotFound error. If that's correct, then I'm more or less >fine with it. If the system-packaged dependencies don't have some way for the egg runtime to identify them, you'll also get DependencyNotFound errors for those. >But... > >To install a non-egg Python package is just a matter of: > 1. Running a bdist_wininst exe (or apt-get on Debian, etc), or > 2. Dumping the package directory in site-packages (Debian folks, >please don't scream :-)) > >For an egg, like TurboGears, my options (assuming I don't want to use >easy_install) are: > 1. Wait for someone to wrap the TG egg in a system package, or > 2. Put the egg somewhere (likely site-packages), then create or >modify a .pth file by hand to add the egg to sys.path. > >In both cases, (1) is the managed option, and (2) is the unmanaged >one. And (2) is *harder* for eggs. ... >I find this jarring, because eggs are presented as a "zero installation" >format. Eggs were really only intended to be "zero installation" for application plugins. The package management stuff came later; the purpose of dependencies was originally there to allow pre-import plugin integrity checking to better preserve application stability. However, at some point I realized that the way eggs were designed meant that you could build some simple package management on top, and easy_install was born. Anyway, I would say that "Wait for someone to wrap the TG egg in a system package" is the correct approach for people who want system packages, and don't want to use easy_install. >And I'd be curious to know what sort of reaction I'd get from the TG >community if I posted a message saying I'd dumped various eggs into >site-packages, manually fiddled with .pth files, and TG wasn't >working. :-) Well, as far as I know, TurboGears does a require("TurboGears") when you import it, which will promptly fail with DependencyNotFound or VersionConflict errors if you haven't done the job right, describing the first missing or screwed-up thing. > > The drop-in-and-go aspect of eggs is mainly used for application plugins in > > systems like Trac, although in truth they actually use the discovery > > machinery to scan their plugins directory and identify which plugins are > > loadable - that is, which plugins can have all their dependencies met. > >Yes. Plugins are a clear benefit. I don't think *anyone* is arguing >that this is an issue. > > > This is one reason that runtime integrity checks are important; an app can > > detect that it doesn't have what's needed for a given plugin, and skip > > loading it. This is also why dependency resolution needs to be able to > > roll back if it fails, even for deeply-nested dependencies. > >OK. And I see that a plugin can need a system library (a Trac >"progress chart" plugin needing PIL installed, for example). And >non-egg installs can't provide that discovery mechanism directly. OK. >You've got that covered (as best you can) with the single egg-info >file option. That's the idea. >As a *user* of other people's packages, I want to use system managed >packages. As a *Windows* user, I'm a pain, because I expect the >developer to supply that system managed installer, rather than the >suppliers of my distribution. Yeah, expecting Microsoft to ship .msi's of Free software is a bit much. :) >(and round full circle) As a *developer* again, I don't have a way of >satisfying my Windows user's requirement for a wrapped version of my >egg. This could be solved by making setuptools' bdist_wininst build an egg in an .exe. The other half of the issue would be resolved by making a .pkg-info inclusion be a standard distutils feature. With these two things done, any egg can be had as an .exe, and any .exe can be had as an egg. In the long run, .msi might be a better format if it supports dependencies, but at least with .exe-wrapped eggs you have the integrity checks to catch when essential stuff got removed. >[An only vaguely related question - is the feature of generating >wrappers for scripts based on entry points an easy_install feature, or >an egg feature? If so, what happens to it with a system package >wrapping the egg? I ask because I see that Kid has swiched from >bdist_wininst to egg as its distribution format, as far as I can tell, >just for the script-wrapper feature - so can I use the manual option >(2) above to install Kid???] Script wrappers are implemented by easy_install, but the entry points themselves are an egg API feature. You can invoke the entry points whether or not there's a script wrapper, you just can't do it from the command line without typing some code. :) From pje at telecommunity.com Fri Nov 25 17:53:17 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 25 Nov 2005 11:53:17 -0500 Subject: [Distutils] [Bulk] Re: formencode as .egg in Debian ?? In-Reply-To: <79990c6b0511250833y6d5d28c5v79b4f3c9d9d75f81@mail.gmail.co m> References: <5.1.1.6.0.20051125110130.01fd2da0@mail.telecommunity.com> <200511251530.24612.hawk78_it@yahoo.it> <E1EfaR8-0008DS-00@d.0x1.org> <5.1.1.6.0.20051125085757.033eb008@mail.telecommunity.com> <5.1.1.6.0.20051125100225.034468c0@mail.telecommunity.com> <5.1.1.6.0.20051125110130.01fd2da0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051125114939.0328f718@mail.telecommunity.com> [Debian trimmed since this is Windows-specific] At 04:33 PM 11/25/2005 +0000, Paul Moore wrote: >On 11/25/05, Phillip J. Eby <pje at telecommunity.com> wrote: > > At 10:28 AM 11/25/2005 -0500, Phillip J. Eby wrote: > > >Actually, I'm increasingly tempted to PEP the idea of a > > >name-version.pkg-info file being installed alongside the code as a > standard > > >Python distutils feature. This discussion has made it clearer (even to > > >me!) how useful such a thing would be, and if it became a standard part of > > >the distutils, it would simply become part of Debian and every other > system > > >packager unless explicitly removed. > > > > Also, it just occurred to me that this would ultimately fix one of Paul > > Moore's complaints about eggs as well. If bdist_wininst included this > > .pkg-info, then even Windows "managed" packages would be supported in this > > scheme, > >That would be a major bonus for me, certainly. > > > and setuptools could include a bdist_wininst that made similar > > .exe's to wrap eggs. > >That would be the other half of the equation. I had avoided suggesting >precisely that, because it's daft of me to expect you to do this work >just to stop me moaning (and installer-writing is enough of a black >art to me that I can't assume it's an easy change to the existing >bdist_wininst code). Actually, the only thing weighing against it in my priorities is the other half of the equation. Until regular win32.exe's include pkg-info, it doesn't make sense to only half-solve this problem; it's better to focus on improving the egg-centric tools (like easy_install), since these can provide a complete solution, and you're the only person who has said they would rather download and install individual .exe packages than use easy_install to take care of the whole business. From sharkypt at gmail.com Sat Nov 26 14:15:01 2005 From: sharkypt at gmail.com (Sharky On PTNet) Date: Sat, 26 Nov 2005 13:15:01 +0000 Subject: [Distutils] problems installing elementTree Message-ID: <40b686290511260515q64024de3l@mail.gmail.com> Hi, I'm getting problems instaling Turbogears when easy_install try to install cElementTree. I have ubuntu breezy (5.10) with python 2.4.2. Any one can helpme with this? Thanks, Helio Pereira Sharky @ PTNet [ERROR] Using /usr/lib/python2.4/site-packages/TurboGears-0.8a4-py2.4.egg Processing dependencies for turbogears Searching for cElementTree>=1.0.2 Reading http://www.python.org/pypi/cElementTree/ Reading http://www.effbot.org/zone/celementtree.htm Reading http://effbot.org/downloads#celementtree Best match: cElementTree 1.0.2-20050302 Downloading http://effbot.org/downloads/index.cgi/cElementTree-1.0.2-20050302.zip?index error: Unexpected HTML page found at http://effbot.org/downloads/index.cgi/cElementTree-1.0.2-20050302.zip?index From pje at telecommunity.com Sat Nov 26 20:06:52 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 26 Nov 2005 14:06:52 -0500 Subject: [Distutils] problems installing elementTree In-Reply-To: <40b686290511260515q64024de3l@mail.gmail.com> Message-ID: <5.1.1.6.0.20051126135155.01fc1730@mail.telecommunity.com> At 01:15 PM 11/26/2005 +0000, Sharky On PTNet wrote: >Hi, > > I'm getting problems instaling Turbogears when easy_install try to >install cElementTree. > > I have ubuntu breezy (5.10) with python 2.4.2. Any one can helpme with > this? I've just checked in a fix. The problem was that the effbot.org pages contained links that look like download links (because they include cElementTree-1.0.2-20050302.zip as the last path part), but which are actually links to a CGI that lets you view the *contents* of the zipfile, as an HTML page. The way that EasyInstall was sorting candidate links was causing it to prefer longer links over shorter ones, assuming that the two links appeared to be pointing to the same package, version number, and format. I've now changed it to prefer shorter links over longer ones, assuming all else is equal. And I've verified that this makes it ignore the effbot.org CGI links and prefer the real download links. You can upgrade your EasyInstall using: easy_install setuptools==dev this will upgrade you to the latest revision from Subversion. (Note: Windows users, please use "ez_setup.py setuptools==dev", as easy_install.exe isn't currently able to updgrade itself without permission problems.) From ianb at colorstudy.com Sun Nov 27 20:30:11 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 27 Nov 2005 13:30:11 -0600 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1132857531.5916.28.camel@arrakis.localnet> References: <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <5.1.1.6.0.20051124125005.02e5b620@mail.telecommunity.com> <1132857531.5916.28.camel@arrakis.localnet> Message-ID: <438A0943.80105@colorstudy.com> Josselin Mouette wrote: >>>Again, the ability to distribute it as a single package is good, >> >>What "single package" are you talking >>about? http://turbogears.org/download/ lists eggs and source packages for >>*10* different Python projects that it depends on, written by five >>different authors. Each is individually packaged, with eggs for Mac and >>Windows, and source packages that can build eggs for everything else. > > > You're reinventing the wheel. Really. This isn't a matter for the > Windows world, as users are accustomed to various, broken tools to > install stuff, and this one will probably be less broken than others. > But MacOS and Linux distributions already have a unified packaging > system, and I don't think it's useful to invent another package type, > especially if it's only suitable for Python. I haven't followed this entire thread. But I'll chime in here: I'm an Egg consumer, and I want Egg features. Some Python software I've built *requires* eggs right now (Paste), some will require them in the near future (SQLObject -- at least with the full featureset). This isn't just dependency information, but the additional metadata Eggs allow for (entry points specifically). I do not know of any similar system for Python plugins. I'm at least passingly familiar with a lot of Python code, and I haven't seen a plugin system that doesn't suck in comparison. If you patch my projects and remove the dependency checks, I won't care that much (though it might be a nuisance); but if the Egg metadata goes away, the projects will be broken. So, perhaps in contrast to Phillip, I'll go further and say that Debian must support Eggs if it is going to package the software I'm writing (SQLObject, FormEncode, Paste), because if it doesn't then... well, then it will just be broken -- if not right away then soon -- and that's not a very good package then, is it? I think the same goes for TurboGears and maybe Trac. No one is forcing Debian to package any of these. I'd like it if Debian did contain some of these packages. But if that means that someone isn't willing to upgrade to SQLObject 0.8 because they have SQLObject 0.7 installed and Debian hasn't packaged 0.8, then I think the Debian packages have become pointless and even counterproductive. And that happens frequently, at least for developer libraries (as opposed to fully encapsulated applications). As to the question of what new things Eggs provide: Eggs -- putting aside the dependency checking -- solve a problem that is currently handled (from what I can tell) with a variety of ad hoc policies in Debian. An example might be, say, /etc/emacs/site-start.d, where a package that provides something puts a file in a well-known directory, and a package that consumes something looks in that directory. However, extra policy is not required for new consumers and producers using Eggs, as the tools all take that into account. So Eggs reduce the work a maintainer has to go through to adapt these kinds of systems to Debian. So, using TurboGears as an example, TurboGears will be able to see extensions provided by other packages, without any special effort. Back to dependency checking... Eggs also provide a cross-platform way to handle installation and dependencies. I use Debian, but a majority of the people using my software don't, and even I don't use it exclusively. I can't maintain Debian-specific data. However, I don't see why there's this resistence to map the data I maintain upstream directly to the Debian package data. You should see the duplication as a positive feature. But it should also go both ways -- it can be painful to do development on Debian because you reach a comfortable state, until the available Debian packages don't fit your needs for some reason, and then you have to abandon those packages and create a whole separate set of installed software. But if Debian packages provide the Egg metadata, then it will be much more reliable and comfortable to install updated versions of just a few libraries without using a Debian package. I don't develop my software as Debian packages. I don't think anyone does development like that. I use a variety of software, in a variety of versions, with some of the stable bits (e.g., MySQLdb) installed as Debian packages. I'd rather not step on the toes of installed packages, but I also want a development environment that tracks the projects I care about. Maybe you are just saying that Debian packages are unsuitable for software development. Maybe that is true; is that what you've decided? If so, then perhaps many of these projects simply shouldn't be packaged at all. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From joss at debian.org Sun Nov 27 21:08:27 2005 From: joss at debian.org (Josselin Mouette) Date: Sun, 27 Nov 2005 21:08:27 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <438A0943.80105@colorstudy.com> References: <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <5.1.1.6.0.20051124125005.02e5b620@mail.telecommunity.com> <1132857531.5916.28.camel@arrakis.localnet> <438A0943.80105@colorstudy.com> Message-ID: <1133122108.8232.14.camel@arrakis.localnet> Le dimanche 27 novembre 2005 ? 13:30 -0600, Ian Bicking a ?crit : > > You're reinventing the wheel. Really. This isn't a matter for the > > Windows world, as users are accustomed to various, broken tools to > > install stuff, and this one will probably be less broken than others. > > But MacOS and Linux distributions already have a unified packaging > > system, and I don't think it's useful to invent another package type, > > especially if it's only suitable for Python. > > I haven't followed this entire thread. But I'll chime in here: > > I'm an Egg consumer, and I want Egg features. Some Python software I've > built *requires* eggs right now (Paste), some will require them in the > near future (SQLObject -- at least with the full featureset). This > isn't just dependency information, but the additional metadata Eggs > allow for (entry points specifically). I do not know of any similar > system for Python plugins. I'm at least passingly familiar with a lot > of Python code, and I haven't seen a plugin system that doesn't suck in > comparison. As I understand it, dependencies and generic metadata are two slightly different features that come together with eggs. While the latter is merely an annoyance for us, the former is a nice feature, even if I dislike the implementation. > If you patch my projects and remove the dependency checks, > I won't care that much (though it might be a nuisance); but if the Egg > metadata goes away, the projects will be broken. Removing the dependency checks entirely and replacing them with Debian dependencies is definitely an option, and I happen to prefer it slightly to the necessity of adding empty .egg-info files to all python module packages. > So, perhaps in > contrast to Phillip, I'll go further and say that Debian must support > Eggs if it is going to package the software I'm writing (SQLObject, > FormEncode, Paste), because if it doesn't then... well, then it will > just be broken -- if not right away then soon -- and that's not a very > good package then, is it? I think the same goes for TurboGears and > maybe Trac. > > No one is forcing Debian to package any of these. Of course you are forcing Debian to package these. As long as your projects have enough users, someone will want to build a Debian package. The whole point of this thread is to make this package not suck. > I'd like it if Debian > did contain some of these packages. But if that means that someone > isn't willing to upgrade to SQLObject 0.8 because they have SQLObject > 0.7 installed and Debian hasn't packaged 0.8, then I think the Debian > packages have become pointless and even counterproductive. And that > happens frequently, at least for developer libraries (as opposed to > fully encapsulated applications). I'm happy you see stable releases as "counterproductive". However many of our users happen to like them, as they don't like to upgrade their software every other day. Again, eggs are a useful tool only for developers, not for regular users. > As to the question of what new things Eggs provide: > > Eggs -- putting aside the dependency checking -- solve a problem that is > currently handled (from what I can tell) with a variety of ad hoc > policies in Debian. An example might be, say, /etc/emacs/site-start.d, > where a package that provides something puts a file in a well-known > directory, and a package that consumes something looks in that > directory. However, extra policy is not required for new consumers and > producers using Eggs, as the tools all take that into account. So Eggs > reduce the work a maintainer has to go through to adapt these kinds of > systems to Debian. So, using TurboGears as an example, TurboGears will > be able to see extensions provided by other packages, without any > special effort. This is slightly different from dependencies. It seems weird to use the same framework to bring both features. > Back to dependency checking... > > Eggs also provide a cross-platform way to handle installation and > dependencies. I use Debian, but a majority of the people using my > software don't, and even I don't use it exclusively. I can't maintain > Debian-specific data. However, I don't see why there's this resistence > to map the data I maintain upstream directly to the Debian package data. It is not possible to do it automatically - not in a reliable way. > You should see the duplication as a positive feature. But it should > also go both ways -- it can be painful to do development on Debian > because you reach a comfortable state, until the available Debian > packages don't fit your needs for some reason, and then you have to > abandon those packages and create a whole separate set of installed > software. But if Debian packages provide the Egg metadata, then it will > be much more reliable and comfortable to install updated versions of > just a few libraries without using a Debian package. I fail to see how it changes anything. If it's to be able to install two versions of the same package at once, I have already explained why it's a bad idea to even support it. > I don't develop my software as Debian packages. I don't think anyone > does development like that. I use a variety of software, in a variety > of versions, with some of the stable bits (e.g., MySQLdb) installed as > Debian packages. I'd rather not step on the toes of installed packages, > but I also want a development environment that tracks the projects I > care about. Maybe you are just saying that Debian packages are > unsuitable for software development. Maybe that is true; is that what > you've decided? If so, then perhaps many of these projects simply > shouldn't be packaged at all. It should be obvious Debian packages aren't useful for the development of the packages themselves. They are a way to install software in an easy and generic way for our users. If your software isn't useful to anyone else than the people developing it, it should indeed not be packaged for Debian. Regards, -- .''`. Josselin Mouette /\./\ : :' : josselin.mouette at ens-lyon.org `. `' joss at debian.org `- Debian GNU/Linux -- The power of freedom -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message =?ISO-8859-1?Q?num=E9riquement?= =?ISO-8859-1?Q?_sign=E9e?= Url : http://mail.python.org/pipermail/distutils-sig/attachments/20051127/43fffed2/attachment.pgp From ianb at colorstudy.com Sun Nov 27 23:26:09 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 27 Nov 2005 16:26:09 -0600 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1133122108.8232.14.camel@arrakis.localnet> References: <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <5.1.1.6.0.20051124125005.02e5b620@mail.telecommunity.com> <1132857531.5916.28.camel@arrakis.localnet> <438A0943.80105@colorstudy.com> <1133122108.8232.14.camel@arrakis.localnet> Message-ID: <438A3281.9000602@colorstudy.com> Josselin Mouette wrote: >>If you patch my projects and remove the dependency checks, >>I won't care that much (though it might be a nuisance); but if the Egg >>metadata goes away, the projects will be broken. > > > Removing the dependency checks entirely and replacing them with Debian > dependencies is definitely an option, and I happen to prefer it slightly > to the necessity of adding empty .egg-info files to all python module > packages. This will make it harder for people who use additional non-packaged Python libraries when they depend on packaged libraries. >>So, perhaps in >>contrast to Phillip, I'll go further and say that Debian must support >>Eggs if it is going to package the software I'm writing (SQLObject, >>FormEncode, Paste), because if it doesn't then... well, then it will >>just be broken -- if not right away then soon -- and that's not a very >>good package then, is it? I think the same goes for TurboGears and >>maybe Trac. >> >>No one is forcing Debian to package any of these. > > > Of course you are forcing Debian to package these. As long as your > projects have enough users, someone will want to build a Debian package. > The whole point of this thread is to make this package not suck. easy_install works, right now, for these packages. There are some outstanding issues, and those issues can probably be resolved in easy_install, without any intervention by Debian. Maybe the simplest resolution is adding an option like: easy_install --already-provided=1.2.6 ElementTree that will create a .egg-info directory. And we can encourage people to fix Debian by putting a distutils.cfg file in place that installs distutil software in /usr/local by default, as opposed to the current situation where I believe distutils installs go alongside Debian packages. >>I'd like it if Debian >>did contain some of these packages. But if that means that someone >>isn't willing to upgrade to SQLObject 0.8 because they have SQLObject >>0.7 installed and Debian hasn't packaged 0.8, then I think the Debian >>packages have become pointless and even counterproductive. And that >>happens frequently, at least for developer libraries (as opposed to >>fully encapsulated applications). > > > I'm happy you see stable releases as "counterproductive". However many > of our users happen to like them, as they don't like to upgrade their > software every other day. Again, eggs are a useful tool only for > developers, not for regular users. These are developer tools, regular users *are* developers. If someone reports a bug, and I fix it, and they can't get access to that because they can't install a new version of the package, then that is counterproductive. It discourages feedback. >>You should see the duplication as a positive feature. But it should >>also go both ways -- it can be painful to do development on Debian >>because you reach a comfortable state, until the available Debian >>packages don't fit your needs for some reason, and then you have to >>abandon those packages and create a whole separate set of installed >>software. But if Debian packages provide the Egg metadata, then it will >>be much more reliable and comfortable to install updated versions of >>just a few libraries without using a Debian package. > > > I fail to see how it changes anything. If it's to be able to install two > versions of the same package at once, I have already explained why it's > a bad idea to even support it. Then you are cutting off me, the upstream maintainer, from the users, because everything has to go through the Debian packager, and a separate release cycle. *Or*, we develop various ad hoc techniques to allow people to install other versions of packages *in spite* of the Debian packages. I just don't think it is an option to say that only one version of a package can be installed. In /usr/lib/pythonX.Y/site-packages, sure, and there can potentially be other restrictions. But you are limiting your users too much if there can never be multiple versions installed. >>I don't develop my software as Debian packages. I don't think anyone >>does development like that. I use a variety of software, in a variety >>of versions, with some of the stable bits (e.g., MySQLdb) installed as >>Debian packages. I'd rather not step on the toes of installed packages, >>but I also want a development environment that tracks the projects I >>care about. Maybe you are just saying that Debian packages are >>unsuitable for software development. Maybe that is true; is that what >>you've decided? If so, then perhaps many of these projects simply >>shouldn't be packaged at all. > > > It should be obvious Debian packages aren't useful for the development > of the packages themselves. They are a way to install software in an > easy and generic way for our users. If your software isn't useful to > anyone else than the people developing it, it should indeed not be > packaged for Debian. This is where the whole thing becomes weird, IMHO. I'm a Debian user (among several distributions). I'm also a developer. As such, the Debian system doesn't work well for me. The system doesn't need to be primarily targetted at my use case, but I think my use case at least needs to be taken into account. Much of the software we are currently talking about is primarily targeted at people like myself (the exception would be Trac, though even that has a developer focus). This may change at some point -- there might be useful applications that Debian users want easy access to, and those applications in turn require one of these libraries. But right now it is Python programmers who are asking for these packages. It's entirely valid to say that these libraries aren't ready to become Debian packages; I'm not really arguing that they should. *But*, Debian policy can make things easier or more difficult for Debian users who are using Eggs, regardless of whether there are packages involved. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From pje at telecommunity.com Mon Nov 28 04:49:00 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 27 Nov 2005 22:49:00 -0500 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1133122108.8232.14.camel@arrakis.localnet> References: <438A0943.80105@colorstudy.com> <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <5.1.1.6.0.20051124125005.02e5b620@mail.telecommunity.com> <1132857531.5916.28.camel@arrakis.localnet> <438A0943.80105@colorstudy.com> Message-ID: <5.1.1.6.0.20051127224128.021ea008@mail.telecommunity.com> At 09:08 PM 11/27/2005 +0100, Josselin Mouette wrote: >Removing the dependency checks entirely and replacing them with Debian >dependencies is definitely an option, and I happen to prefer it slightly >to the necessity of adding empty .egg-info files to all python module >packages. Just a reminder: without the .egg-info metadata (or something similar), we aren't fulfilling one of the primary use cases, which is Debian users wanting their bleeding-edge installs to not duplicate stable packages. This is probably the number one gripe of Debian users on the TurboGears mailing list: "why doesn't easy_install see my Debian ElementTree package?" (or some other dependency). They don't seem to mind easy_install adding stuff to /usr/local that there's not a Debian system package for, but they'd like it to be able to "see" the Debian packages they've installed and not rebuild/reinstall them. (One of their other big complaints would be fixed by having Debian set distutils.cfg such that the default install location for distutils stuff is /usr/local/.../site-packages, rather than /usr. But I gather one such user opened a bug for this issue separately, though.) From sanxiyn at gmail.com Mon Nov 28 09:22:50 2005 From: sanxiyn at gmail.com (Sanghyeon Seo) Date: Mon, 28 Nov 2005 17:22:50 +0900 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051127224128.021ea008@mail.telecommunity.com> References: <5bnlf-5nA-35@gated-at.bofh.it> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <5.1.1.6.0.20051124125005.02e5b620@mail.telecommunity.com> <1132857531.5916.28.camel@arrakis.localnet> <438A0943.80105@colorstudy.com> <1133122108.8232.14.camel@arrakis.localnet> <5.1.1.6.0.20051127224128.021ea008@mail.telecommunity.com> Message-ID: <5b0248170511280022w46c2b2cbi@mail.gmail.com> 2005/11/28, Phillip J. Eby <pje at telecommunity.com>: > > (One of their other big complaints would be fixed by having Debian set > distutils.cfg such that the default install location for distutils stuff is > /usr/local/.../site-packages, rather than /usr. But I gather one such user > opened a bug for this issue separately, though.) http://bugs.debian.org/338572 I filed it. Seo Sanghyeon From wl at flexis.de Mon Nov 28 09:25:22 2005 From: wl at flexis.de (Wolfgang Langner) Date: Mon, 28 Nov 2005 09:25:22 +0100 Subject: [Distutils] specify zip method (ZIP_STORED, ZIP_DEFLATED) Message-ID: <dmeeti$rmu$1@sea.gmane.org> Hello, is there a way to specify the zip method like ZIP_STORED or ZIP_DEFLATED as in python zipfile module ? I think it is possible to speed up something if ZIP_STORED is used to "compress" egg packages. (not tested only thoughts) And on a platform with no zlib it schould be possible to create .egg files. I have searched the docu but nothing found on that topic. bye by Wolfgang From joss at debian.org Mon Nov 28 09:45:20 2005 From: joss at debian.org (Josselin Mouette) Date: Mon, 28 Nov 2005 09:45:20 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <438A3281.9000602@colorstudy.com> References: <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <5.1.1.6.0.20051124125005.02e5b620@mail.telecommunity.com> <1132857531.5916.28.camel@arrakis.localnet> <438A0943.80105@colorstudy.com> <1133122108.8232.14.camel@arrakis.localnet> <438A3281.9000602@colorstudy.com> Message-ID: <1133167520.4203.19.camel@silicium.ccc.cea.fr> Le dimanche 27 novembre 2005 ? 16:26 -0600, Ian Bicking a ?crit : > >>No one is forcing Debian to package any of these. > > > > Of course you are forcing Debian to package these. As long as your > > projects have enough users, someone will want to build a Debian package. > > The whole point of this thread is to make this package not suck. > > easy_install works, right now, for these packages. There are some > outstanding issues, and those issues can probably be resolved in > easy_install, without any intervention by Debian. Maybe the simplest > resolution is adding an option like: > > easy_install --already-provided=1.2.6 ElementTree We're not going to tell our users to use easy_install for some kind of packages and some other thing for another package, and so on. They expect to be able to install anything, just by typing "apt-get install foo". > that will create a .egg-info directory. And we can encourage people to > fix Debian by putting a distutils.cfg file in place that installs > distutil software in /usr/local by default, as opposed to the current > situation where I believe distutils installs go alongside Debian packages. This is an entirely different matter, which indeed deserves a bug report if it isn't done already. > > I'm happy you see stable releases as "counterproductive". However many > > of our users happen to like them, as they don't like to upgrade their > > software every other day. Again, eggs are a useful tool only for > > developers, not for regular users. > > These are developer tools, regular users *are* developers. > > If someone reports a bug, and I fix it, and they can't get access to > that because they can't install a new version of the package, then that > is counterproductive. It discourages feedback. Stable releases are not meant for developers like you, but there are developers who want a stable development platform, even if it has some bugs. No bug fixing also means no regressions. The fact you don't understand those people's needs - and forget them when designing your tools - doesn't make all stable releases "counterproductive". Developers who want a bleeding edge development platform should be using Debian testing or unstable, where the new versions become available as soon as possible. > > I fail to see how it changes anything. If it's to be able to install two > > versions of the same package at once, I have already explained why it's > > a bad idea to even support it. > > Then you are cutting off me, the upstream maintainer, from the users, > because everything has to go through the Debian packager, and a separate > release cycle. Yes. Bugs in the Debian package should go to the Debian packager. > *Or*, we develop various ad hoc techniques to allow > people to install other versions of packages *in spite* of the Debian > packages. If your packages can't be installed in /usr/local, where they take precedence over the version in /usr, your packaging system is flawed. > I just don't think it is an option to say that only one version of a > package can be installed. In /usr/lib/pythonX.Y/site-packages, sure, > and there can potentially be other restrictions. But you are limiting > your users too much if there can never be multiple versions installed. Stable users will want to keep the same version, and allowing them to install several versions provides a way to easily break their systems. It also cuts them off the security fixes. Testing/unstable users have the new packages as soon as possible, and the new package should replace the old one entirely. > > It should be obvious Debian packages aren't useful for the development > > of the packages themselves. They are a way to install software in an > > easy and generic way for our users. If your software isn't useful to > > anyone else than the people developing it, it should indeed not be > > packaged for Debian. > > This is where the whole thing becomes weird, IMHO. I'm a Debian user > (among several distributions). I'm also a developer. As such, the > Debian system doesn't work well for me. The system doesn't need to be > primarily targetted at my use case, but I think my use case at least > needs to be taken into account. Much of the software we are currently > talking about is primarily targeted at people like myself (the exception > would be Trac, though even that has a developer focus). This may change > at some point -- there might be useful applications that Debian users > want easy access to, and those applications in turn require one of these > libraries. I think you are missing my point. When I'm developing software, I can use Debian for that. However I won't rely on the Debian packages of my own software, this simply makes no sense. Now, if you are developing software based on some python modules, if these modules are packaged in Debian, it's perfectly possible, as long as you're not going to hack the modules themselves. And when it's packaged in Debian, you don't need egg-style dependencies, as Debian dependencies already take care of it. Regards, -- .''`. Josselin Mouette /\./\ : :' : josselin.mouette at ens-lyon.org `. `' joss at debian.org `- Debian GNU/Linux -- The power of freedom From joss at debian.org Mon Nov 28 09:52:27 2005 From: joss at debian.org (Josselin Mouette) Date: Mon, 28 Nov 2005 09:52:27 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051127224128.021ea008@mail.telecommunity.com> References: <438A0943.80105@colorstudy.com> <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <5.1.1.6.0.20051124125005.02e5b620@mail.telecommunity.com> <1132857531.5916.28.camel@arrakis.localnet> <438A0943.80105@colorstudy.com> <5.1.1.6.0.20051127224128.021ea008@mail.telecommunity.com> Message-ID: <1133167947.4203.25.camel@silicium.ccc.cea.fr> Le dimanche 27 novembre 2005 ? 22:49 -0500, Phillip J. Eby a ?crit : > Just a reminder: without the .egg-info metadata (or something similar), we > aren't fulfilling one of the primary use cases, which is Debian users > wanting their bleeding-edge installs to not duplicate stable packages. > > This is probably the number one gripe of Debian users on the TurboGears > mailing list: "why doesn't easy_install see my Debian ElementTree package?" > (or some other dependency). They don't seem to mind easy_install adding > stuff to /usr/local that there's not a Debian system package for, but > they'd like it to be able to "see" the Debian packages they've installed > and not rebuild/reinstall them. I see it as a flaw in the way eggs work, not in our Debian packages. You shouldn't forget Debian stable isn't going to move for still about a year. Even if we add one of your suggested hacks to unstable packages, sarge users still won't be able to install such packages, and that's the same for stable RHEL users, for example. > (One of their other big complaints would be fixed by having Debian set > distutils.cfg such that the default install location for distutils stuff is > /usr/local/.../site-packages, rather than /usr. But I gather one such user > opened a bug for this issue separately, though.) This is indeed a valid complaint. -- .''`. Josselin Mouette /\./\ : :' : josselin.mouette at ens-lyon.org `. `' joss at debian.org `- Debian GNU/Linux -- The power of freedom From joss at debian.org Mon Nov 28 09:52:27 2005 From: joss at debian.org (Josselin Mouette) Date: Mon, 28 Nov 2005 09:52:27 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051127224128.021ea008@mail.telecommunity.com> References: <438A0943.80105@colorstudy.com> <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <5.1.1.6.0.20051124125005.02e5b620@mail.telecommunity.com> <1132857531.5916.28.camel@arrakis.localnet> <438A0943.80105@colorstudy.com> <5.1.1.6.0.20051127224128.021ea008@mail.telecommunity.com> Message-ID: <1133167947.4203.25.camel@silicium.ccc.cea.fr> Le dimanche 27 novembre 2005 ? 22:49 -0500, Phillip J. Eby a ?crit : > Just a reminder: without the .egg-info metadata (or something similar), we > aren't fulfilling one of the primary use cases, which is Debian users > wanting their bleeding-edge installs to not duplicate stable packages. > > This is probably the number one gripe of Debian users on the TurboGears > mailing list: "why doesn't easy_install see my Debian ElementTree package?" > (or some other dependency). They don't seem to mind easy_install adding > stuff to /usr/local that there's not a Debian system package for, but > they'd like it to be able to "see" the Debian packages they've installed > and not rebuild/reinstall them. I see it as a flaw in the way eggs work, not in our Debian packages. You shouldn't forget Debian stable isn't going to move for still about a year. Even if we add one of your suggested hacks to unstable packages, sarge users still won't be able to install such packages, and that's the same for stable RHEL users, for example. > (One of their other big complaints would be fixed by having Debian set > distutils.cfg such that the default install location for distutils stuff is > /usr/local/.../site-packages, rather than /usr. But I gather one such user > opened a bug for this issue separately, though.) This is indeed a valid complaint. -- .''`. Josselin Mouette /\./\ : :' : josselin.mouette at ens-lyon.org `. `' joss at debian.org `- Debian GNU/Linux -- The power of freedom From p.f.moore at gmail.com Mon Nov 28 11:09:20 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 28 Nov 2005 10:09:20 +0000 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <438A3281.9000602@colorstudy.com> References: <5bnlf-5nA-35@gated-at.bofh.it> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <5.1.1.6.0.20051124125005.02e5b620@mail.telecommunity.com> <1132857531.5916.28.camel@arrakis.localnet> <438A0943.80105@colorstudy.com> <1133122108.8232.14.camel@arrakis.localnet> <438A3281.9000602@colorstudy.com> Message-ID: <79990c6b0511280209m2b68ac2o91b552ab4a1ed9bb@mail.gmail.com> On 11/27/05, Ian Bicking <ianb at colorstudy.com> wrote: > easy_install --already-provided=1.2.6 ElementTree > > that will create a .egg-info directory. [Windows POV, so I removed the Debian crossposts] Hmm, but what if I then upgrade (uninstall & install a new version) ElementTree? The egg dependency information is wrong, I guess. Sounds like a potential problem. OTOH, do I really care? Is version 1.2.7 really going to break all my software??? Paul. From sharkypt at gmail.com Mon Nov 28 13:49:49 2005 From: sharkypt at gmail.com (Sharky On PTNet) Date: Mon, 28 Nov 2005 12:49:49 +0000 Subject: [Distutils] Problems instaling Kid template Message-ID: <40b686290511280449i29ecb9e8l@mail.gmail.com> Hi again :P I'm having trobles trying to install Kid template with easy_install. Don't know if is a problem related with kid package or something else. Here is the output of the error: # easy_install kid Searching for kid Reading http://www.python.org/pypi/kid/ Reading http://kid.lesscode.org/ Best match: kid 0.7.1 Downloading http://lesscode.org/dist/kid/0.7.1/kid-0.7.1.tar.gz Processing kid-0.7.1.tar.gz Running kid-0.7.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-gJouzW/kid-0.7.1/egg-dist-tmp-noS3cf error: SandboxViolation: mkdir('/home/sharky/.python-eggs', 511) {} The package setup script has attempted to modify files on your system that are not within the EasyInstall build area, and has been aborted. This package cannot be safely installed by EasyInstall, and may not support alternate installation locations even if you run its setup script by hand. Please inform the package's author and the EasyInstall maintainers to find out if a fix or workaround is available. From abo at minkirri.apana.org.au Mon Nov 28 15:25:11 2005 From: abo at minkirri.apana.org.au (Donovan Baarda) Date: Mon, 28 Nov 2005 14:25:11 +0000 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1133167520.4203.19.camel@silicium.ccc.cea.fr> References: <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <5.1.1.6.0.20051124125005.02e5b620@mail.telecommunity.com> <1132857531.5916.28.camel@arrakis.localnet> <438A0943.80105@colorstudy.com> <1133122108.8232.14.camel@arrakis.localnet> <438A3281.9000602@colorstudy.com> <1133167520.4203.19.camel@silicium.ccc.cea.fr> Message-ID: <1133187911.14336.11.camel@warna.dub.corp.google.com> On Mon, 2005-11-28 at 09:45 +0100, Josselin Mouette wrote: > Le dimanche 27 novembre 2005 ? 16:26 -0600, Ian Bicking a ?crit : [...] > > easy_install works, right now, for these packages. There are some > > outstanding issues, and those issues can probably be resolved in > > easy_install, without any intervention by Debian. Maybe the simplest > > resolution is adding an option like: > > > > easy_install --already-provided=1.2.6 ElementTree > > We're not going to tell our users to use easy_install for some kind of > packages and some other thing for another package, and so on. They > expect to be able to install anything, just by typing "apt-get install > foo". Actually, provided easy_install puts things in /usr/local, I'm happy with this. I regularly don't use apt-get install when I want something bleeding edge which is not (and I don't expect to be) packaged. The point is; this is not a deb, provided it behaves as a non-deb should, it is OK. However, when I want production ready software, I look for debs. If it ain't deb'ed, it's almost certainly not ready for production. If you want your software taken seriously, you want it well deb'ed :-) -- Donovan Baarda <abo at minkirri.apana.org.au> http://minkirri.apana.org.au/~abo/ From abo at minkirri.apana.org.au Mon Nov 28 16:11:05 2005 From: abo at minkirri.apana.org.au (Donovan Baarda) Date: Mon, 28 Nov 2005 15:11:05 +0000 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051125081920.03d61d90@mail.telecommunity.com> References: <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> <Your message of "Thu, 24 Nov 2005 19:16:44 CDT." <5.1.1.6.0.20051124184858.01f9c2c8@mail.telecommunity.com> <5.1.1.6.0.20051125003059.02c5de70@mail.telecommunity.com> <5.1.1.6.0.20051125081920.03d61d90@mail.telecommunity.com> Message-ID: <1133190665.14336.22.camel@warna.dub.corp.google.com> On Fri, 2005-11-25 at 08:57 -0500, Phillip J. Eby wrote: > At 10:29 AM 11/25/2005 +0000, Donovan Baarda wrote: > >On Fri, 2005-11-25 at 01:33 -0500, Phillip J. Eby wrote: [...] > In the case where the user is *not* using easy_install, then all > dependencies will be met by system packages, and the only "duplication" is > that the paranoid setuptools integrity-check wrappers will want to be > assured that yes, everything is still there. [...] > If you are running easy_install or setup.py to install TurboGears, and an > egged ElementTree is not available, one is going to get installed for you > -- because in that scenario you are asking for installation and are > presumably doing it to /usr/local or to ~/somewhere. If you are merely > *using* TurboGears and an egged version of a dependency is not available, [...] > Right; the point here is that if somebody is going *outside* of Debian > packages and explicitly using setup.py or easy_install to install it, then > it would be nice for them to *still* be able to use system packages to meet > some of the dependencies, rather than having to re-download, re-build, and > re-install new copies of the stuff they already have. This isn't about > changing Debian to support bleeding edge things, it's about allowing people > doing bleeding edge things to not need to duplicate what Debian already > provides. I'm not sure at this point whether you are describing the way eggs currently work, or the way you have figured out they should work to behave nice with Debian... it sounds OK to me :-) > >I think they a both important. If I was using eggs, I'm not sure I'd > >want them to be packaged as anything other than an egg until I was ready > >to release, and then I'd want it **not** packaged as an egg, as a "proof > >of it's release readiness". > > Well, that would only work if you weren't developing something using entry > points to tie an extensible application or framework together. And that > didn't use egg metadata for anything else. But if all you used it for was I was referring here to stripping the egg package management "layer". > >It's perhaps unfortunate that egg's bundle package management with > >generic meta-data management. > > It's not really bundled - integrity checking and inter-package linking do > not a package manager make. EasyInstall is the (rather primitive) package > manager, and it only comes into play if you are manually installing > things. For the normal apt-get scenario, EasyInstall isn't involved, with > the possible exception of applications like Trac installing plugins into > their private plugins directories. It sounds like EasyInstall **is** the package management layer I was suggesting should be stripped when making a deb. > > I know that package management uses > >meta-data, but if it was implemented as one wrapped around the other, > >you could "peel the package management layer off", leaving the metadata > >management there, and you could produce clean deb's using only Debian > >package management, but with the egg meta-data management still in > >place. > > There's nothing to peel off, unless you believe that it's wrong to have > runtime integrity checking. That's all the egg runtime does, unless an [...] I think runtime integrity checking is an unnecessary overhead when you have decent install-time integrity checking. However, I can see how this is useful for detecting and using optional dependencies... > Vincenzo's idea about having easy_install be able to wrap apt-get is > interesting, in that it would allow apps that want plugin installation to > satisfy dependencies via system packages that *aren't* already installed, In some ways I'd feel mildly disturbed by this. I'd tend to think of apt-get and easy_install as seperate tools; one for managing Debian packages, and the other for installing non-debian packaged egg's in /usr/local. > but in practice I think such applications are going to be restricted to > local or specific-user installation by the very nature of their intended > uses, so being able to have easy_install get automatically hooked to > apt-get doesn't seem necessary to me. In the Debian world, easy_install agreed. > should probably only be used for user-private and/or system-local > non-Debian stuff, including apps' private plugins. (And if you did want > easy_install to be able to use apt-get to satisfy dependencies, it probably > means you should just be using something like easy_deb instead, to build > debs of the stuff you want.) dunno how useful that is... In some ways I'd prefer a /usr/local non-deb install than a custom-built deb. -- Donovan Baarda <abo at minkirri.apana.org.au> http://minkirri.apana.org.au/~abo/ From alex at bofh.net.pl Mon Nov 28 17:39:09 2005 From: alex at bofh.net.pl (Janusz A. Urbanowicz) Date: Mon, 28 Nov 2005 17:39:09 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <5.1.1.6.0.20051127224128.021ea008@mail.telecommunity.com> References: <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <5.1.1.6.0.20051124125005.02e5b620@mail.telecommunity.com> <1132857531.5916.28.camel@arrakis.localnet> <438A0943.80105@colorstudy.com> <5.1.1.6.0.20051127224128.021ea008@mail.telecommunity.com> Message-ID: <20051128163908.GU29592@syjon.fantastyka.net> On Sun, Nov 27, 2005 at 10:49:00PM -0500, Phillip J. Eby wrote: > At 09:08 PM 11/27/2005 +0100, Josselin Mouette wrote: > >Removing the dependency checks entirely and replacing them with Debian > >dependencies is definitely an option, and I happen to prefer it slightly > >to the necessity of adding empty .egg-info files to all python module > >packages. > > Just a reminder: without the .egg-info metadata (or something similar), we > aren't fulfilling one of the primary use cases, which is Debian users > wanting their bleeding-edge installs to not duplicate stable packages. If a dedina user wants bleeding edge, she should be running sid or etch where this is packaged. Or ubuntu. > This is probably the number one gripe of Debian users on the TurboGears > mailing list: "why doesn't easy_install see my Debian ElementTree package?" > (or some other dependency). They don't seem to mind easy_install adding > stuff to /usr/local that there's not a Debian system package for, but > they'd like it to be able to "see" the Debian packages they've installed > and not rebuild/reinstall them. This should be reported as a bug toElementTree package. a -- mors ab alto 0x46399138 From pje at telecommunity.com Mon Nov 28 17:53:17 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 28 Nov 2005 11:53:17 -0500 Subject: [Distutils] Problems instaling Kid template In-Reply-To: <40b686290511280449i29ecb9e8l@mail.gmail.com> Message-ID: <5.1.1.6.0.20051128115030.022c6668@mail.telecommunity.com> At 12:49 PM 11/28/2005 +0000, Sharky On PTNet wrote: >Hi again :P > > I'm having trobles trying to install Kid template with >easy_install. Don't know if is a problem related with kid package or >something else. See this thread: http://mail.python.org/pipermail/distutils-sig/2005-November/005401.html I'm still working on fixing this; basically I need to exempt pkg_resources operations from the sandbox checking used by easy_install. In the meantime, you can use these steps to install manually: easy_install -eb . kid cd kid python setup.py bdist_egg easy_install dist/kid*.egg cd ..; rm -rf kid From ianb at colorstudy.com Mon Nov 28 18:27:37 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 28 Nov 2005 11:27:37 -0600 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1133167520.4203.19.camel@silicium.ccc.cea.fr> References: <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <5.1.1.6.0.20051124125005.02e5b620@mail.telecommunity.com> <1132857531.5916.28.camel@arrakis.localnet> <438A0943.80105@colorstudy.com> <1133122108.8232.14.camel@arrakis.localnet> <438A3281.9000602@colorstudy.com> <1133167520.4203.19.camel@silicium.ccc.cea.fr> Message-ID: <438B3E09.8070801@colorstudy.com> Josselin Mouette wrote: > Le dimanche 27 novembre 2005 ? 16:26 -0600, Ian Bicking a ?crit : > >>>>No one is forcing Debian to package any of these. >>> >>>Of course you are forcing Debian to package these. As long as your >>>projects have enough users, someone will want to build a Debian package. >>>The whole point of this thread is to make this package not suck. >> >>easy_install works, right now, for these packages. There are some >>outstanding issues, and those issues can probably be resolved in >>easy_install, without any intervention by Debian. Maybe the simplest >>resolution is adding an option like: >> >> easy_install --already-provided=1.2.6 ElementTree > > > We're not going to tell our users to use easy_install for some kind of > packages and some other thing for another package, and so on. They > expect to be able to install anything, just by typing "apt-get install > foo". That's okay; I'll just tell my users to use it when Debian packages are not available (which is the case for the majority of Python libraries, and probably always will be unless libraries are automatically packaged). easy_install provides a much better and safer experience than downloading tarballs and using the normal "setup.py install" routine; it is an improvement on the status quo. The other option would be to allow people to easily create debs, in the same way they can easily create RPMs right now; but I get the strong impression you don't think that's a good idea. >>>I'm happy you see stable releases as "counterproductive". However many >>>of our users happen to like them, as they don't like to upgrade their >>>software every other day. Again, eggs are a useful tool only for >>>developers, not for regular users. >> >>These are developer tools, regular users *are* developers. >> >>If someone reports a bug, and I fix it, and they can't get access to >>that because they can't install a new version of the package, then that >>is counterproductive. It discourages feedback. > > > Stable releases are not meant for developers like you, but there are > developers who want a stable development platform, even if it has some > bugs. No bug fixing also means no regressions. The fact you don't > understand those people's needs - and forget them when designing your > tools - doesn't make all stable releases "counterproductive". I'm speaking in terms of the open source ecosystem -- gaining contributors is far more valuable than gaining users. So from my perspective -- which isn't out of line from Debian's, I think -- cutting off feedback is damaging. > Developers who want a bleeding edge development platform should be using > Debian testing or unstable, where the new versions become available as > soon as possible. I don't think those really provide the same experience. If you are really serious about the bleeding edge, you should be using a checkout from the upstream repository. Which, incidentally, setuptools and easy_install handle nicely. But again, I say that from the perspective of someone who wants to gain contributors, not just users. >>*Or*, we develop various ad hoc techniques to allow >>people to install other versions of packages *in spite* of the Debian >>packages. > > > If your packages can't be installed in /usr/local, where they take > precedence over the version in /usr, your packaging system is flawed. They can be, with no problems; easy_install uses the standard distutils.cfg configuration, which can indicate prefix=/usr/local. Even without it, easy_install/setuptools is much less likely to clobber Debian installed packages, compared to normal distutils. >>I just don't think it is an option to say that only one version of a >>package can be installed. In /usr/lib/pythonX.Y/site-packages, sure, >>and there can potentially be other restrictions. But you are limiting >>your users too much if there can never be multiple versions installed. > > > Stable users will want to keep the same version, and allowing them to > install several versions provides a way to easily break their systems. > It also cuts them off the security fixes. New versions don't overwrite old versions. I believe that if you have a library in /usr/lib, and install a new version to /usr/local/lib, that the new version won't ever be accessible until you specifically pkg_resources.require() it (or implicitly require it due to dependencies). *However*, to make that work nicely an easy_install'ed package should see metadata about Debian installed packages; otherwise duplicate versions of libraries have to be installed, because it's not reasonable to discover at runtime what Debian packages are installed (if only because the Debian package database is much too slow). Though if you really wanted, setuptools/easy_install could be patched to read the debian package database on installation, maybe even writing a .egg-info directory then (putting it in /usr/local/lib, but it can refer back to a library that was installed in /usr/lib). That mostly depends on the enthusiasm of the setuptools packager. Anyway, in spite of all the disagreement, I think we actually have reached a sort of conclusion... For the packages in question (ultimately leading to packaging TurboGears) there's only one current Debian package of consequence: ElementTree. I think it's fine to remove that package in particular from the egg dependencies. So the timescale for changes to Debian aren't that much of an issue because these are new packages. Most of the libraries are developed with setuptools upstream... I think all of them, in fact (setuptools itself, TurboGears, Kid, SQLObject, FormEncode, probably nose as well). There are some questions about zip files -- simplest answer: don't install them as zips. Some of them can't be installed that way anyway. There are some questions about the performance implications of installing them in subdirectories of site-packages, and extending sys.path for each installed package. You can revert to normal installation with an .egg-info directory, and Phillip will (or already has?) added support for putting the version number in those directory names, so that will all work fine. I believe if you require a version of a package from /usr/local/lib, it will do the Right Thing and change sys.path appropriately. As I think about it, not putting Debian-packaged libraries entirely in subdirectories (the typical egg installation) seems best -- keeping libraries in subdirectories makes package management easier (by easy_install, nest, or whatever), but in this case we definitely *don't* want those tools to try to manage those packages. Maybe we could even add a flag to indicate to setuptools/easy_install when a library is managed by some external tool? I think that would add some important safety. I would suggest something like a Package.egg-info/managed_by.txt file, whose existance indicates the package is externally managed, and whose content explains what manages it (so setuptools could provide a nice error message that directs the user to apt-get on Debian, or some other native tool on other platforms). A distutils.cfg file (bug 338572) will allow people to use easy_install on Debian systems, and still follow Debian policy. But I assume that such a change will take some time to propogate even if it is applied immediately to sid. But even without it, easy_install won't overwrite any Debian package files without that configuration file. I think easy_install also gives some scary-looking warnings in that case, but that might be possible to handle in documentation, especially if we all agree on what steps users should be following. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From joss at debian.org Mon Nov 28 22:41:14 2005 From: joss at debian.org (Josselin Mouette) Date: Mon, 28 Nov 2005 22:41:14 +0100 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <438B3E09.8070801@colorstudy.com> References: <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <5.1.1.6.0.20051124125005.02e5b620@mail.telecommunity.com> <1132857531.5916.28.camel@arrakis.localnet> <438A0943.80105@colorstudy.com> <1133122108.8232.14.camel@arrakis.localnet> <438A3281.9000602@colorstudy.com> <1133167520.4203.19.camel@silicium.ccc.cea.fr> <438B3E09.8070801@colorstudy.com> Message-ID: <1133214074.15714.35.camel@arrakis.localnet> Le lundi 28 novembre 2005 ? 11:27 -0600, Ian Bicking a ?crit : > That's okay; I'll just tell my users to use it when Debian packages are > not available (which is the case for the majority of Python libraries, > and probably always will be unless libraries are automatically > packaged). easy_install provides a much better and safer experience > than downloading tarballs and using the normal "setup.py install" > routine; it is an improvement on the status quo. The other option would > be to allow people to easily create debs, in the same way they can > easily create RPMs right now; but I get the strong impression you don't > think that's a good idea. Creating debs easily is a good idea if done correctly, but only for packages that will eventually enter the archive. It's a very bad idea to encourage people to build their own versions of Debian packages. It would lead to a horrible cluttering of package sources, and to the disappearance of any coherent versioning scheme. However, if a semi-automated way to produce Debian source packages is available, like it is for CPAN modules, it can only be a good thing, as it will be used to make loads of packages that can enter the archive. And there's no reason why all these python modules couldn't enter the archive. > > Stable releases are not meant for developers like you, but there are > > developers who want a stable development platform, even if it has some > > bugs. No bug fixing also means no regressions. The fact you don't > > understand those people's needs - and forget them when designing your > > tools - doesn't make all stable releases "counterproductive". > > I'm speaking in terms of the open source ecosystem -- gaining > contributors is far more valuable than gaining users. So from my > perspective -- which isn't out of line from Debian's, I think -- cutting > off feedback is damaging. More users also mean more contributors. Stable releases mean more users, especially corporate users who aren't likely to contribute from the beginning, but who may become contributors later. And I strongly disagree with your statement we are cutting off feedback. You seem to forget your software isn't the only one out there; we are providing a centralized way to get feedback, redistributing it when necessary. This makes things much easier for users when they have only one way to give feedback. > > Developers who want a bleeding edge development platform should be using > > Debian testing or unstable, where the new versions become available as > > soon as possible. > > I don't think those really provide the same experience. If you are > really serious about the bleeding edge, you should be using a checkout > from the upstream repository. Which, incidentally, setuptools and > easy_install handle nicely. But again, I say that from the perspective > of someone who wants to gain contributors, not just users. What if you want bleeding edge, yet tested software? You're going to use the latest release of the software, not the CVS snapshot. When you're developing a project based on TurboGears, I'm sure you don't want to use the CVS tree. You're going to use the features as soon as they are available in the release. And releases are available in Debian unstable. > New versions don't overwrite old versions. I believe that if you have a > library in /usr/lib, and install a new version to /usr/local/lib, that > the new version won't ever be accessible until you specifically > pkg_resources.require() it (or implicitly require it due to > dependencies). *However*, to make that work nicely an easy_install'ed > package should see metadata about Debian installed packages; otherwise > duplicate versions of libraries have to be installed, because it's not > reasonable to discover at runtime what Debian packages are installed (if > only because the Debian package database is much too slow). Though if > you really wanted, setuptools/easy_install could be patched to read the > debian package database on installation, maybe even writing a .egg-info > directory then (putting it in /usr/local/lib, but it can refer back to a > library that was installed in /usr/lib). That mostly depends on the > enthusiasm of the setuptools packager. You're mixing things again, and again, and again. Debian dependencies should NOT be mixed with lower-level dependencies. A clean way of doing things would be a shlibs-like mechanism. However, it would require dependencies to be written only in metadata files. As I understand it, they can also be asked directly in python code, in which case there's no reliable way to tell a package needs another package. > Anyway, in spite of all the disagreement, I think we actually have > reached a sort of conclusion... > > For the packages in question (ultimately leading to packaging > TurboGears) there's only one current Debian package of consequence: > ElementTree. I think it's fine to remove that package in particular > from the egg dependencies. So the timescale for changes to Debian > aren't that much of an issue because these are new packages. Most of > the libraries are developed with setuptools upstream... I think all of > them, in fact (setuptools itself, TurboGears, Kid, SQLObject, > FormEncode, probably nose as well). That can be done in the unstable package. However, you're still going to get complaints from sarge users because your software won't see the stable ElementTree package. The same goes for RHEL4 users. Regards, -- .''`. Josselin Mouette /\./\ : :' : josselin.mouette at ens-lyon.org `. `' joss at debian.org `- Debian GNU/Linux -- The power of freedom -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message =?ISO-8859-1?Q?num=E9riquement?= =?ISO-8859-1?Q?_sign=E9e?= Url : http://mail.python.org/pipermail/distutils-sig/attachments/20051128/2992aad2/attachment.pgp From chewie at wookimus.net Tue Nov 29 08:02:12 2005 From: chewie at wookimus.net (Chad Walstrom) Date: Tue, 29 Nov 2005 01:02:12 -0600 Subject: [Distutils] formencode as .egg in Debian ?? In-Reply-To: <1133214074.15714.35.camel@arrakis.localnet> References: <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <4382B786.2080705@v.loewis.de> <5bnlf-5nA-35@gated-at.bofh.it> <5bsO3-54O-21@gated-at.bofh.it> <4382B786.2080705@v.loewis.de> <5.1.1.6.0.20051122102248.01fc1bd0@mail.telecommunity.com> <43835658.30606@egenix.com> <43835988.3060106@colorstudy.com> <1132838800.12478.21.camel@silicium.ccc.cea.fr> <64447770-0719-4D4A-8570-9B4B632394C4@gmx.de> <5.1.1.6.0.20051124101703.021648f0@mail.telecommunity.com> <5.1.1.6.0.20051124125005.02e5b620@mail.telecommunity.com> <1132857531.5916.28.camel@arrakis.localnet> <438A0943.80105@colorstudy.com> <1133122108.8232.14.camel@arrakis.localnet> <438A3281.9000602@colorstudy.com> <1133167520.4203.19.camel@silicium.ccc.cea.fr> <438B3E09.8070801@colorstudy.com> <1133214074.15714.35.camel@arrakis.localnet> Message-ID: <20051129070212.617FE3153@skuld.wookimus.net> Josselin Mouette <joss at Debian.org> wrote: > Creating debs easily is a good idea if done correctly, but only for > packages that will eventually enter the archive. It's a very bad > idea to encourage people to build their own versions of Debian > packages. It would lead to a horrible cluttering of package sources, > and to the disappearance of any coherent versioning scheme. As a fellow Debian Developer, I say, "Hogwash!" If Debian isn't providing *.deb's, then there's nothing wrong with upstream providing their own, regardless of vaporish eventualities. In fact, it's presumptuous to assume that Debian has *any* control over upstream's practices. I applaud Phillip for his assistance in trying to resolve the issue at hand. I for one encourage upstream to provide packages in any and all formats that they're willing to. All the better if the package is policy compliant, has sane dependences, and compiled against latest stable and unstable archives, but these are by no means a requirement. Given that Debian developers are volunteers and may not be as responsive to user's bug reports as upstream may be, the centralized nature of Debian you tout as being so grand isn't always so. I admit freely that had I more time, I might be able to respond more quickly to bug reports and feature requests. (Not to mention the politics and overly-strict, glacial policies.) So, I again encourage software developers to provide packages to any system they might want to support, and if they need help with packaging software for official inclusion to Debian, there are plenty of developers and new maintainer candidates willing to help. > However, if a semi-automated way to produce Debian source packages > is available, like it is for CPAN modules, it can only be a good > thing, as it will be used to make loads of packages that can enter > the archive. And there's no reason why all these python modules > couldn't enter the archive. Agreed. > ...You seem to forget your software isn't the only one out there; Presumptuous, shortsighted troll, and totally uncalled for. In fact, I believe Phillip, et. al. have been arguing the exact opposite. Folks, we don't all act this way. > we are providing a centralized way to get feedback, redistributing > it when necessary. This makes things much easier for users when they > have only one way to give feedback. ...which can have response problems proportional to the time investment of the developer maintaining the package. > ...You're going to use the features as soon as they are available in > the release. And releases are available in Debian unstable. s/are/may be/ Of course, that is if we can come to some sort of technical agreement here. > You're mixing things again, and again, and again. Debian > dependencies should NOT be mixed with lower-level dependencies. It depends (no pun intended -- O.K. maybe a little), and you state so in your next sentence. > A clean way of doing things would be a shlibs-like mechanism. For those unfamiliar with such terminology, shlibs is a debhelper script that heuristicly resolves dependencies found at package build time and then updates the appropriate field in the binary package(s) built for the source. In fact, I believe this has been suggested a number of times in this thread already. > However, it would require dependencies to be written only in > metadata files. As I understand it, they can also be asked directly > in python code, in which case there's no reliable way to tell a > package needs another package. Yet there's no reason to exclude the possibility of creating a Debian-specific hook in for easy_setup/setup.py to resolve these introspectively as well. Regardless, it's up to the packager to add appropriate package dependencies that cannot be resolved by automated scripts. This is not a new issue or one isolated to eggs. > That can be done in the unstable package. However, you're still > going to get complaints from sarge users because your software won't > see the stable ElementTree package. The same goes for RHEL4 users. Not an issue. Sarge (stable) users can get backports of all their favorite bleeding-edge software like they always could through upstream developers who package their own *.debs or through interested Debian developers who upload to sites such as http://backports.org. -- Chad Walstrom <chewie at wookimus.net> http://www.wookimus.net/ assert(expired(knowledge)); /* core dump */ From sharkypt at gmail.com Tue Nov 29 13:51:56 2005 From: sharkypt at gmail.com (Sharky On PTNet) Date: Tue, 29 Nov 2005 12:51:56 +0000 Subject: [Distutils] [DistUtils] - Problem with versions... Message-ID: <40b686290511290451k6dc396c6p@mail.gmail.com> Hi, Trying to upgrade cherrypy pkg with ease_install, i notice that easy_install don't detect the newer version. You can see that 2.1.0 is the final and not 2.1.0-rc2 in the sourceforge page: http://sourceforge.net/project/showfiles.php?group_id=56099 Thanks for all Sharkypt @ Helio Pereira # easy_install -U cherrypy Searching for cherrypy Reading http://www.python.org/pypi/cherrypy/ Couldn't find index page for 'cherrypy' (maybe misspelled?) Scanning index of all packages (this may take a while) Reading http://www.python.org/pypi/ Reading http://www.python.org/pypi/CherryPy/2.1.0 Reading http://www.cherrypy.org Reading http://sourceforge.net/project/showfiles.php?group_id=56099 Best match: CherryPy 2.1.0-rc2 Processing CherryPy-2.1.0_rc2-py2.4.egg CherryPy 2.1.0-rc2 is already the active version in easy-install.pth Using /usr/lib/python2.4/site-packages/CherryPy-2.1.0_rc2-py2.4.egg Processing dependencies for cherrypy From pje at telecommunity.com Tue Nov 29 19:10:42 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 29 Nov 2005 13:10:42 -0500 Subject: [Distutils] [DistUtils] - Problem with versions... In-Reply-To: <40b686290511290451k6dc396c6p@mail.gmail.com> Message-ID: <5.1.1.6.0.20051129130726.02232010@mail.telecommunity.com> At 12:51 PM 11/29/2005 +0000, Sharky On PTNet wrote: >Hi, > > Trying to upgrade cherrypy pkg with ease_install, i notice that >easy_install don't detect the newer version. > > You can see that 2.1.0 is the final and not 2.1.0-rc2 in the >sourceforge page: >http://sourceforge.net/project/showfiles.php?group_id=56099 The problem is that "2.1.0-rc2" > "2.1.0" > "2.1.0rc2", as "-" is most commonly used in versions to denote a post-release patch or port, whereas a plain "rc" is normally a pre-release candidate. Use "easy_install CherryPy==2.1.0" to get the exact release. (TurboGears works around this by depending on "CherryPy>=2.1,!=2.1-rc1,!=2.1-rc2".) From sharkypt at gmail.com Tue Nov 29 22:13:33 2005 From: sharkypt at gmail.com (Sharky On PTNet) Date: Tue, 29 Nov 2005 21:13:33 +0000 Subject: [Distutils] [DistUtils] - Problem with versions... In-Reply-To: <5.1.1.6.0.20051129130726.02232010@mail.telecommunity.com> References: <40b686290511290451k6dc396c6p@mail.gmail.com> <5.1.1.6.0.20051129130726.02232010@mail.telecommunity.com> Message-ID: <40b686290511291313y2df0a7cek@mail.gmail.com> Thanks for everything Phillip... 2005/11/29, Phillip J. Eby <pje at telecommunity.com>: > At 12:51 PM 11/29/2005 +0000, Sharky On PTNet wrote: > >Hi, > > > > Trying to upgrade cherrypy pkg with ease_install, i notice that > >easy_install don't detect the newer version. > > > > You can see that 2.1.0 is the final and not 2.1.0-rc2 in the > >sourceforge page: > >http://sourceforge.net/project/showfiles.php?group_id=56099 > > > The problem is that "2.1.0-rc2" > "2.1.0" > "2.1.0rc2", as "-" is most > commonly used in versions to denote a post-release patch or port, whereas a > plain "rc" is normally a pre-release candidate. Use "easy_install > CherryPy==2.1.0" to get the exact release. (TurboGears works around this > by depending on "CherryPy>=2.1,!=2.1-rc1,!=2.1-rc2".) > > From skip at pobox.com Wed Nov 30 16:50:29 2005 From: skip at pobox.com (skip@pobox.com) Date: Wed, 30 Nov 2005 09:50:29 -0600 Subject: [Distutils] Trouble installing Myghty with --prefix Message-ID: <17293.51781.473397.360361@montanaro.dyndns.org> I tried installing Myghty 0.99a using this command: python setup.py install --prefix=/usr/local/mojam (e.g., non-standard install location). It downloaded setuptools and installed a bunch of other stuff (Paste, etc), all available as eggs. Unfortunately, it didn't create packages and didn't make the pkg_resources module available. Any hints about how to get this working would be appreciated. (I have no idea what other details you'll need, never having used setuptools - directly or indirectly. Let me know if there's more I can do to explain the situation.) Thx, -- Skip Montanaro Katrina Benefit Concerts: http://www.musi-cal.com/katrina skip at pobox.com From ianb at colorstudy.com Wed Nov 30 17:32:21 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 30 Nov 2005 10:32:21 -0600 Subject: [Distutils] Trouble installing Myghty with --prefix In-Reply-To: <17293.51781.473397.360361@montanaro.dyndns.org> References: <17293.51781.473397.360361@montanaro.dyndns.org> Message-ID: <438DD415.4050403@colorstudy.com> skip at pobox.com wrote: > I tried installing Myghty 0.99a using this command: > > python setup.py install --prefix=/usr/local/mojam > > (e.g., non-standard install location). It downloaded setuptools and > installed a bunch of other stuff (Paste, etc), all available as eggs. > Unfortunately, it didn't create packages and didn't make the pkg_resources > module available. Any hints about how to get this working would be > appreciated. (I have no idea what other details you'll need, never having > used setuptools - directly or indirectly. Let me know if there's more I can > do to explain the situation.) I suspect it's because setuptools didn't think that /usr/local/mojam/lib/... was on the normal site path, and so .pth files aren't read from the location. And when .pth files aren't read, the eggs won't be put on the path automatically (though I don't know quite how setuptools can work at all when that happens). There's non-root installation instructions on the setuptools site (and maybe included in easy_install now?) that will create a full Python environment in a non-standard location, which would probably resolve this issue. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Wed Nov 30 18:03:39 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 30 Nov 2005 12:03:39 -0500 Subject: [Distutils] Trouble installing Myghty with --prefix In-Reply-To: <438DD415.4050403@colorstudy.com> References: <17293.51781.473397.360361@montanaro.dyndns.org> <17293.51781.473397.360361@montanaro.dyndns.org> Message-ID: <5.1.1.6.0.20051130115201.01fcb5c8@mail.telecommunity.com> At 10:32 AM 11/30/2005 -0600, Ian Bicking wrote: >skip at pobox.com wrote: > > I tried installing Myghty 0.99a using this command: > > > > python setup.py install --prefix=/usr/local/mojam > > > > (e.g., non-standard install location). It downloaded setuptools and > > installed a bunch of other stuff (Paste, etc), all available as eggs. > > Unfortunately, it didn't create packages and didn't make the pkg_resources > > module available. Any hints about how to get this working would be > > appreciated. (I have no idea what other details you'll need, never having > > used setuptools - directly or indirectly. Let me know if there's more > I can > > do to explain the situation.) > >I suspect it's because setuptools didn't think that >/usr/local/mojam/lib/... was on the normal site path, and so .pth files >aren't read from the location. And when .pth files aren't read, the >eggs won't be put on the path automatically (though I don't know quite >how setuptools can work at all when that happens). That's correct. Skip, if you want to use a custom install location using PYTHONPATH, you need to read this section: http://peak.telecommunity.com/DevCenter/EasyInstall#traditional-pythonpath-based-installation Your configuration, however, would be something like: [install] prefix = /usr/local/mojam [easy_install] site_dirs = /usr/local/mojam/lib/python2.4/site-packages instead of the configurations shown there. >There's non-root installation instructions on the setuptools site (and >maybe included in easy_install now?) that will create a full Python >environment in a non-standard location, which would probably resolve >this issue. Right; the other option is to use this approach: http://peak.telecommunity.com/DevCenter/EasyInstall#creating-a-virtual-python There's a 'virtual-python' script linked there that will set it up for you; you would run: virtual-python.py --prefix=/usr/local/mojam to set it up. This should not delete any existing files there, just add symlinks and create a /usr/local/mojam/bin/python executable that will think its sys.prefix is /usr/local/mojam. (It will overwrite /usr/local/mojam/bin/python, though, if it already exists.) In contrast to the other approach above, the "virtual" Python installation will not need any special configuration, because as far as it knows, it was built to live in /usr/local/mojam and all its defaults and installation locations will work fine. At that point you should make sure that /usr/local/mojam/bin is on PATH, and use that Python for everything from then on. From skip at pobox.com Wed Nov 30 18:58:52 2005 From: skip at pobox.com (skip@pobox.com) Date: Wed, 30 Nov 2005 11:58:52 -0600 Subject: [Distutils] Trouble installing Myghty with --prefix In-Reply-To: <438DD415.4050403@colorstudy.com> References: <17293.51781.473397.360361@montanaro.dyndns.org> <438DD415.4050403@colorstudy.com> Message-ID: <17293.59484.8092.849782@montanaro.dyndns.org> Ian> There's non-root installation instructions on the setuptools site Ian> (and maybe included in easy_install now?) that will create a full Ian> Python environment in a non-standard location, which would probably Ian> resolve this issue. Thanks, but that's not what I'm after. On my CentOS 4 server, Python 2.3.x is installed with /usr as the prefix (that is, I use the CentOS-provided version of Python). I install all third-party packages (including my own code) with a prefix of /usr/local/mojam so I can a) avoid polluting the base package and b) install things without becoming root. Accordingly, I dump a mojam.pth file in the system site-packages directory which contains this one line: /usr/local/mojam/lib/python2.3/site-packages That is my only pollution of the default Python installation. I'd like to keep it that way. Can setuptools (be made to) do the job? Thx, Skip From skip at pobox.com Wed Nov 30 19:21:20 2005 From: skip at pobox.com (skip@pobox.com) Date: Wed, 30 Nov 2005 12:21:20 -0600 Subject: [Distutils] Trouble installing Myghty with --prefix In-Reply-To: <5.1.1.6.0.20051130115201.01fcb5c8@mail.telecommunity.com> References: <17293.51781.473397.360361@montanaro.dyndns.org> <5.1.1.6.0.20051130115201.01fcb5c8@mail.telecommunity.com> Message-ID: <17293.60832.239064.263462@montanaro.dyndns.org> Phillip> Skip, if you want to use a custom install location using Phillip> PYTHONPATH, you need to read this section: Phillip> http://peak.telecommunity.com/DevCenter/EasyInstall#traditional-pythonpath-based-installation Folks, Please note a couple things: 1. I use pth files, not PYTHONPATH. They are much easier to control than insuring the every user of the system (even the daemon-ish ones) have PYTHONPATH set properly. I have one pth file in the system install. That's the full extent of the pollution of that directory tree. I want to keep it that way. ISTM that setuptools should be able to easily deduce what the default sys.path is, since Python does the heavy lifting of parsing/executing pth files for you: python -c 'import sys ; print ":".join(sys.path)' Voila! Now setuptools should know that /usr/local/mojam/lib/ python2.3/site-packages *is* in sys.path and that I'm not trying to install packages for some sort of offbeat Python installation. 2. I didn't use setuptools directly. In fact, when I executed python setup.py install --prefix=/usr/local/mojam in the Myghty 0.99a source directory I had no idea setuptools would be used at all. Mike Bayer (Myghty's author) tells me that was a requirement because Myghty depends on something called Paste. The whole notion of using setuptools is highly indirect for me. Phillip> Your configuration, however, would be something like: Phillip> [install] Phillip> prefix = /usr/local/mojam Phillip> [easy_install] Phillip> site_dirs = /usr/local/mojam/lib/python2.4/site-packages Phillip> instead of the configurations shown there. I'll take a look. At first blush I'm a bit put off by the notion it seems to promote that somehow using .pth files is wrong. I think you should be able to deal with them straight out of the box, no tricks required, especially considering they are blessed as part of the default site.py, and not likely to go away soon. Phillip> Right; the other option is to use this approach: Phillip> http://peak.telecommunity.com/DevCenter/EasyInstall#creating-a-virtual-python Phillip> There's a 'virtual-python' script linked there that will set it Phillip> up for you; you would run: Phillip> virtual-python.py --prefix=/usr/local/mojam Phillip> to set it up. This should not delete any existing files there, Phillip> just add symlinks and create a /usr/local/mojam/bin/python Phillip> executable that will think its sys.prefix is /usr/local/mojam. Phillip> (It will overwrite /usr/local/mojam/bin/python, though, if it Phillip> already exists.) That's definitely not what I want. I want to "/usr/bin/env python" to resolve to /usr/bin/python in all cases (like when a CGI script executes as nobody or when procmail runs spambayes) but use a single pth file to tell it where my local stuff lives. Such tools are almost certainly never going to have /usr/local/mojam/bin in their PATH. Skip From pje at telecommunity.com Wed Nov 30 19:43:01 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 30 Nov 2005 13:43:01 -0500 Subject: [Distutils] Trouble installing Myghty with --prefix Message-ID: <5.1.1.6.0.20051130134300.01fa4458@mail.telecommunity.com> At 11:58 AM 11/30/2005 -0600, skip at pobox.com wrote: > Ian> There's non-root installation instructions on the setuptools site > Ian> (and maybe included in easy_install now?) that will create a full > Ian> Python environment in a non-standard location, which would probably > Ian> resolve this issue. > >Thanks, but that's not what I'm after. On my CentOS 4 server, Python 2.3.x >is installed with /usr as the prefix (that is, I use the CentOS-provided >version of Python). I install all third-party packages (including my own >code) with a prefix of /usr/local/mojam so I can a) avoid polluting the base >package and b) install things without becoming root. Accordingly, I dump a >mojam.pth file in the system site-packages directory which contains this one >line: > > /usr/local/mojam/lib/python2.3/site-packages > >That is my only pollution of the default Python installation. I'd like to >keep it that way. Can setuptools (be made to) do the job? Yes, but you need two things: 1. Change your .pth file to read: import site; site.addsitedir('/usr/local/mojam/lib/python2.3/site-packages') 2. Create a distutils.cfg in /usr/lib/python2.3/distutils/ with the contents I mentioned before. Basically, without #1, .pth files in /usr/local/mojam/lib/python2.3/site-package won't be processed by Python, and without #2, setuptools won't know that Python is processing the .pth files. From pje at telecommunity.com Wed Nov 30 19:52:29 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 30 Nov 2005 13:52:29 -0500 Subject: [Distutils] Trouble installing Myghty with --prefix In-Reply-To: <17293.60832.239064.263462@montanaro.dyndns.org> References: <5.1.1.6.0.20051130115201.01fcb5c8@mail.telecommunity.com> <17293.51781.473397.360361@montanaro.dyndns.org> <5.1.1.6.0.20051130115201.01fcb5c8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051130134308.020634c0@mail.telecommunity.com> At 12:21 PM 11/30/2005 -0600, skip at pobox.com wrote: > Phillip> Skip, if you want to use a custom install location using > Phillip> PYTHONPATH, you need to read this section: > > Phillip> > http://peak.telecommunity.com/DevCenter/EasyInstall#traditional-pythonpath-based-installation > >Folks, > >Please note a couple things: > > 1. I use pth files, not PYTHONPATH. They are much easier to control > than insuring the every user of the system (even the daemon-ish ones) > have PYTHONPATH set properly. I have one pth file in the system > install. That's the full extent of the pollution of that directory > tree. I want to keep it that way. ISTM that setuptools should be > able to easily deduce what the default sys.path is, since Python does > the heavy lifting of parsing/executing pth files for you: > > python -c 'import sys ; print ":".join(sys.path)' > > Voila! Now setuptools should know that /usr/local/mojam/lib/ > python2.3/site-packages *is* in sys.path and that I'm not trying to > install packages for some sort of offbeat Python installation. This isn't the issue; setuptools uses .pth files itself, but Python won't process .pth files in your custom directory unless you take additional steps. The two approaches I linked to are two different ways of getting Python to process the .pth files that setuptools uses (and easy_install installs). There's also a third way, which I just posted, and which is also described under the --site-dirs option of the EasyInstall docs. >I'll take a look. At first blush I'm a bit put off by the notion it seems >to promote that somehow using .pth files is wrong. On the contrary, setuptools deals with them just fine, and uses them just fine. The problem is that your *Python* won't process the .pth files installed by easy_install unless you take additional steps. This is literally a Python problem, in that Bob Ippolito has previously filed bugs and patches requesting that .pth processing be transitive. That is, that if you list a directory in a .pth, any .pth files in *that* directory should be processed. Guido has vetoed this at least once already, though. The workaround is to use an 'import' hack in your .pth file and call site.addsitedir(). It's a pain to set up, but it works. > I think you should be >able to deal with them straight out of the box, no tricks required, >especially considering they are blessed as part of the default site.py, and >not likely to go away soon. Unfortunately, Python provides no way to know which directories will have .pth files processed. EasyInstall therefore uses the default Python algorithm for determining "site" directories, which will fail if Python has been patched (e.g. under Debian to add /usr/local stuff) or if you use the 'import' hack to fix the non-transitive .pth problem.