From pje at telecommunity.com Wed Mar 1 16:08:54 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 01 Mar 2006 09:08:54 -0600 Subject: [Distutils] problem with EasyInstall 0.6a10 + pkg manager In-Reply-To: <4405BF35.2040506@gmx.de> References: <4405BF35.2040506@gmx.de> Message-ID: <7.0.1.0.0.20060301090555.020b2678@telecommunity.com> At 09:35 AM 3/1/2006, Ronny Pfannschmidt wrote: >With the latest version of EasyInstall i had a major problem with the >"exhaustive testing of the install directory, including a spawn test for >.pth file support" >since my pkg manager uses `setup.py install --root=Dest` to install the >pkg in a image directory, before installing it on the system. > >actually i work around this by creating the needed dirs and adding them >to PYTHONPATH > >is there a nicer way too work with such pkg managers ? Yes. Use "setup.py install --single-version-externally-managed --root=Dest", which will do the installation in a way that's package manager friendly. I may change this at some point so that a --root (other than /) automatically triggers the other option, but at the moment I'm not sure whether that would interfere with anything else. From pje at telecommunity.com Wed Mar 1 16:59:05 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 01 Mar 2006 09:59:05 -0600 Subject: [Distutils] [PEAK] Re: problem with EasyInstall 0.6a10 + pkg manager In-Reply-To: <4405D196.3020602@gmx.de> References: <4405BF35.2040506@gmx.de> <7.0.1.0.0.20060301090555.020b2678@telecommunity.com> <4405D196.3020602@gmx.de> Message-ID: <7.0.1.0.0.20060301095623.02019c00@telecommunity.com> At 10:53 AM 3/1/2006, Ronny Pfannschmidt wrote: >Phillip J. Eby wrote: > >>At 09:35 AM 3/1/2006, Ronny Pfannschmidt wrote: >> >>>With the latest version of EasyInstall i had a major problem with the >>>"exhaustive testing of the install directory, including a spawn test for >>>.pth file support" >>>since my pkg manager uses `setup.py install --root=Dest` to install the >>>pkg in a image directory, before installing it on the system. >>> >>>actually i work around this by creating the needed dirs and adding them >>>to PYTHONPATH >>> >>>is there a nicer way too work with such pkg managers ? >> >> >>Yes. Use "setup.py install --single-version-externally-managed >>--root=Dest", which will do the installation in a way that's >>package manager friendly. >> >>I may change this at some point so that a --root (other than /) >>automatically triggers the other option, but at the moment I'm not >>sure whether that would interfere with anything else. >i tried the option, and found several problems >while the python modules where installed fine much of the other data >is thrown right into /usr (wich isnt very nice) > >as result the program (pida) was not working anymore ( while it >worked with the dirty workaround), since it was written for a egg >image, wich implies, that much of the data is in the root of the egg >image and usually the path can be resolved by the module path Ah. Don't do that. If your packages have data, put them inside a package directory, and use the pkg_resources module APIs to access them. Then your code will work no matter how it's installed. I would go so far in fact as to say that you should *never* use the disutils' data files installation for constant data files that your program needs access to at runtime. The "data files" option should be used only for example configuration files, documentation, etc. If your packages need data, install it as package data. From pje at telecommunity.com Thu Mar 2 05:50:43 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 01 Mar 2006 23:50:43 -0500 Subject: [Distutils] setuptools and ordering... In-Reply-To: <87irr3mmvw.fsf@localhost.localdomain> Message-ID: <5.1.1.6.0.20060301234101.02d6d878@mail.telecommunity.com> At 06:19 PM 2/25/2006 +0200, Iwan Vosloo wrote: >Or, I'd have to write code that builds the dependency tree (of >Distributions) and then flatten it into a list that would make >sense. And THEN search for entry points in that order. I don't >suppose something like this exists? Not really. I'd probably do something like: def flatten(dists, ws=pkg_resources.working_set, memo=None): if memo is None: memo = {} for dist in dists: if dist in memo: continue memo[dist] = True predecessors = ws.resolve(dist.as_requirement()) for d in flatten(predecessors, ws, memo): yield d yield dist Calling "flatten(working_set)" would then yield every distribution (at most once) in an ordering guaranteed to have dependency targets before the distributions that depend on them. Once you've got the distributions in that order, you can then do as you wish with their entry points. From pje at telecommunity.com Thu Mar 2 05:54:32 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 01 Mar 2006 23:54:32 -0500 Subject: [Distutils] Testing with stubbed entry points? In-Reply-To: <87accio6ki.fsf@localhost.localdomain> References: <5.1.1.6.0.20060222110556.01dfce90@mail.telecommunity.com> <5.1.1.6.0.20060222110556.01dfce90@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060301235236.02edee08@mail.telecommunity.com> At 09:52 AM 2/23/2006 +0200, Iwan Vosloo wrote: >"Phillip J. Eby" writes: > > You then have to add that Distribution to a WorkingSet instance, and > > you need to make your code use that WorkingSet instead of the > > default working_set. > >Why can't I add the Distribution to the default working_set? Well, you could, but then your test may end up inadvertently depending on some local sys.path quirk. For a "pure" unit test, I'd want to use a WorkingSet with known contents, that's all. >And... do I need to .activate() the distribution? Not if you're not actually importing anything from the dummy egg, no. From strawman at astraw.com Fri Mar 3 01:50:41 2006 From: strawman at astraw.com (Andrew Straw) Date: Thu, 02 Mar 2006 16:50:41 -0800 Subject: [Distutils] excluding files from sdist using setuptools? Message-ID: <440792E1.10407@astraw.com> How do I do the equivalent of an "exclude" in MANIFEST.in using setuptools? I can't find this documented anywhere. From pje at telecommunity.com Fri Mar 3 02:45:15 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 02 Mar 2006 20:45:15 -0500 Subject: [Distutils] excluding files from sdist using setuptools? In-Reply-To: <440792E1.10407@astraw.com> Message-ID: <5.1.1.6.0.20060302204346.01df8260@mail.telecommunity.com> At 04:50 PM 3/2/2006 -0800, Andrew Straw wrote: >How do I do the equivalent of an "exclude" in MANIFEST.in using >setuptools? I can't find this documented anywhere. Setuptools processes MANIFEST.in after its default processing, so if you need MANIFEST.in features, you can still use them. It's just that the defaults include everything under source control. From ben at groovie.org Fri Mar 3 22:56:54 2006 From: ben at groovie.org (Ben Bangert) Date: Fri, 3 Mar 2006 13:56:54 -0800 Subject: [Distutils] Paste broken? (likely setuptools installation problem) In-Reply-To: <4408B4BD.8030604@rtl.fmailbox.com> References: <44082316.5030200@rtl.fmailbox.com> <44082812.6070007@pythonweb.org> <44083F12.9030206@rtl.fmailbox.com> <4408B4BD.8030604@rtl.fmailbox.com> Message-ID: <882B9AF0-F6DE-47BD-8FC9-A1A636FD25D6@groovie.org> On Mar 3, 2006, at 1:27 PM, Robert Leftwich wrote: > In an attempt to solve this problem and get Pylons working again > I've spent the > last 2hrs trying various approaches - ending up with deleting > everything even > remotely connected to my app and Pylons from the site-packages, > cleaning up > easy-install.pth to remove all references, deleting all eggs - > basically > everything I could find, even to the point of blowing away ez- > setup.py. I'm trying to reproduce this problem, clearing Paste and running easy_install appears to fetch it all properly. I've done this on several different machines (FreeBSD 4.2, FreeBSD 5.2, OSX 10.4), I can't seem to reproduce it anywhere. I'm cc'ing dist-utils on this as it appears to be an issue with setuptools. > Running python ez_setup.py Pylons==dev results in: > > ... (some successful installs elided) > > Installed /usr/lib/python2.4/site-packages/PasteScript-0.5dev_r4816- > py2.4.egg > Searching for PasteDeploy==dev,>=0.5dev-r4624 > Reading http://www.python.org/pypi/PasteDeploy/ > Reading http://pythonpaste.org/deploy/ > Best match: PasteDeploy dev > Downloading http://svn.pythonpaste.org/Paste/Deploy/ > trunk#egg=PasteDeploy-dev > Doing subversion checkout from http://svn.pythonpaste.org/Paste/ > Deploy/trunk to > /tmp/easy_install-9WT2is/trunk > Processing trunk > Running setup.py -q bdist_egg --dist-dir > /tmp/easy_install-9WT2is/trunk/egg-dist-tmp-s4UzvR > warning: manifest_maker: standard file not found: should have one > of README, > README.txt > warning: no files found matching 'docs/*.html' > warning: no previously-included files found matching 'docs/rebuild' > ---------------------------------------------------------------------- > --- > CONFLICT WARNING: > > The following modules or packages have the same names as modules or > packages being installed, and will be *before* the installed > packages in > Python's search path. You MUST remove all of the relevant files and > directories before you will be able to use the package(s) you are > installing: > > /usr/lib/python2.4/site-packages/PasteScript-0.5dev_r4816- > py2.4.egg/paste > > Note: you can attempt this installation again with EasyInstall, and > use > either the --delete-conflicting (-D) option or the > --ignore-conflicts-at-my-risk option, to either delete the above files > and directories, or to ignore the conflicts, respectively. Note > that if > you ignore the conflicts, the installed package(s) may not work. > Running with -D succeeds (sort of), except that doing a "paster create > --template=pylons test" fails with: > > > Traceback (most recent call last): > File "/usr/bin/paster", line 5, in ? > pkg_resources.run_script('PasteScript==0.5dev-r4816', 'paster') > File > "/usr/lib/python2.4/site-packages/setuptools-0.6a11dev_r42684- > py2.4.egg/pkg_resources.py", > line 407, in run_script > self.require(requires)[0].run_script(script_name, ns) > File > "/usr/lib/python2.4/site-packages/setuptools-0.6a11dev_r42684- > py2.4.egg/pkg_resources.py", > line 1043, in run_script > execfile(script_filename, namespace, namespace) > File > "/usr/lib/python2.4/site-packages/PasteScript-0.5dev_r4816- > py2.4.egg/EGG-INFO/scripts/paster", > line 17, in ? > from paste.script import command > ImportError: No module named script > > I've tried all the combinations I can think of (using setuptools > a10 and dev as > shown above), easy_install and ez_setup.py, even installing the > Paste* dev > releases directly using easy_install, but all w/o success. > > Can anyone offer any alternative approaches to getting Pylons to > work again? Ok, I'm not sure how this is happening but my guess is that for some bizarre reason PasteScript was seen as a conflict to PasteDeploy. When you ran with -D, it of course happily blew away PasteScript during the PasteDeploy installation. So of course, when paster tries to use something from PasteScript, there is no longer any PasteScript registered as being around anymore thus it can't be imported. Perhaps try installing just PasteScript: easy_install -U PasteScript==dev Then run paster again? Alternatively, if something is borked in your setuptools settings somewhere, which cause PasteDeploy to think PasteScript was in conflict... hopefully one of the distutils guys can think of either a reason that would happen, or how you could prevent it from occuring. Hope that helps, Ben From pje at telecommunity.com Sat Mar 4 00:37:55 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 03 Mar 2006 18:37:55 -0500 Subject: [Distutils] Paste broken? (likely setuptools installation problem) In-Reply-To: <882B9AF0-F6DE-47BD-8FC9-A1A636FD25D6@groovie.org> References: <4408B4BD.8030604@rtl.fmailbox.com> <44082316.5030200@rtl.fmailbox.com> <44082812.6070007@pythonweb.org> <44083F12.9030206@rtl.fmailbox.com> <4408B4BD.8030604@rtl.fmailbox.com> Message-ID: <5.1.1.6.0.20060303183513.01df8420@mail.telecommunity.com> At 01:56 PM 3/3/2006 -0800, Ben Bangert wrote: >Ok, I'm not sure how this is happening but my guess is that for some >bizarre reason PasteScript was seen as a conflict to PasteDeploy. >When you ran with -D, it of course happily blew away PasteScript >during the PasteDeploy installation. So of course, when paster tries >to use something from PasteScript, there is no longer any PasteScript >registered as being around anymore thus it can't be imported. Ouch. That sounds like a nasty problem. I'll see if I can figure out how it's happening. The conflict detection code currently sucks rather badly, I'm afraid, so maybe this will provide a good excuse to do some refactoring. :) From titus at caltech.edu Sun Mar 5 06:17:49 2006 From: titus at caltech.edu (Titus Brown) Date: Sat, 4 Mar 2006 21:17:49 -0800 Subject: [Distutils] easy_install problem with Twisted-2.1.0 In-Reply-To: <5.1.1.6.0.20060209192853.03686c60@mail.telecommunity.com> References: <20060209230709.GA8662@caltech.edu> <5.1.1.6.0.20060209192853.03686c60@mail.telecommunity.com> Message-ID: <20060305051749.GB2200@caltech.edu> On Thu, Feb 09, 2006 at 07:31:44PM -0500, Phillip J. Eby wrote: -> At 03:07 PM 2/9/2006 -0800, titus at caltech.edu wrote: -> >Hi all, -> > -> >here's a quick bug report for ya. Happened with both Python 2.3 and -> >2.4, using latest easy_install. -> -> -> Please see -> http://mail.python.org/pipermail/distutils-sig/2006-February/005960.html -> -> Better yet, would you update the experiences page: -> -> http://peak.telecommunity.com/DevCenter/PackageNotes -> -> to link there? This is unfortunately not fixable in 0.6, since setuptools -> doesn't yet offer an optional-extension feature, at least not one I'd want -> to try talking the Twisted folks into using. :) Added, with appropriate linkages. cheers, --titus p.s. sorry for the delay -- I plead pycon ;) From chris at kateandchris.net Mon Mar 6 16:48:41 2006 From: chris at kateandchris.net (Chris Lambacher) Date: Mon, 6 Mar 2006 10:48:41 -0500 Subject: [Distutils] Easy Install Documentation Error Message-ID: <20060306154841.GA27796@kateandchris.net> Hi, I was having a bit of trouble installing python module packages on Gentoo after following the "Administrator Installation" section of the documentation. I kept getting packages installed to /var/tmp/homedir/python2.4/ or something like that. I couldn't figure out what was going on until I came accross a gentoo bug that someone filed and then closed citing installation of setuptools as the culprit. Thats when I realised that I added the following to my distutils.cfg: [install] install_lib = ~/lib/python2.3 # This next line is optional but often quite useful; it directs EasyInstall # and the distutils to install scripts in the user's "bin" directory. For # Mac OS X framework Python builds, you should use /usr/local/bin instead, # because neither ~/bin nor the default script installation location are on # the system PATH. # install_scripts = ~/bin Which would cause all distutils based installations to do that. Further up in the documenation it talks about using the [easy_install] section of distutils.cfg to set some preferences. I changed the above to: [easy_install] install_dir = ~/lib/python2.3 scripts_dir = ~/bin I would recommend changing the documenation to avoid confusion since most people probably want to leave their old distutils operation alone and add these settings for easy_install. -Chris From ianb at colorstudy.com Tue Mar 7 19:01:55 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 07 Mar 2006 12:01:55 -0600 Subject: [Distutils] Working environment Message-ID: <440DCA93.2030207@colorstudy.com> So, coming back to the idea of a working environment, an isolated and more-or-less self-contained environment for holding installed packages. Sorry if this is a little scattered. I'm just summarizing my thoughts and the open issues I see, in no particular order (because I'm not sure what order to approach these things in). I'm assuming such an environment will be encapsulated in a single directory, looking something like: env/ bin/ lib/python2.4/ src/ conf/ The conf/ directory doesn't really relate to much of this specifically, but in many situations it would be useful. Depending on the situation, other subdirectories may exist. Each of the scripts in bin/ should know what their working environment is. This is slightly tricky, depending on what that means. If it is a totally isolated environment -- no site-packages on sys.path -- then I feel like the script wrappers have to be shell scripts, to invoke Python with -S (which is hard to do portably on the #! line). I don't know the details of doing the same thing on Windows, but I assume it is possible. The actual directory location should be portable -- all paths should be relative, and you should be able to move the directory around. lib/python2.4/ is for packages. I'm almost inclined to say that --single-version-externally-managed makes sense on some level, with a record kept in some standard place (lib/python2.4/install-record.txt?) -- but I'm basically indifferent. I at least don't see a need for multiple simultaneous versions in this setup, and multiple versions do lead to confusion. Version metadata is still nice, of course. src/ is for checkouts, where each package is installed with setup.py develop. These are naturally single-version, which is part of why I like the idea of only using single-version setups. I'm a little unsure of how src/ should be layed out. In practice I want all "my" packages to be installed in src/ as checkouts, either from tags or the trunk (or a branch or whatever). So I'm not sure if I should name the subdirectories after the package, or maybe even the package plus a tag name. One of the things SwitchTower (now "Cappucino", I think) does in Rails land is it makes a dated checkout, then activates that checkout (it does it with a symlink, we'd do it with setup.py develop). It then rolls back by switching to an existing checkout. Of course svn switch + svn up does this in place, and with less checkout trash laying around, even if rollbacks aren't as fast as a result. So, I'm thinking just src/PackageName/ There's an installation issue there -- it would be nice if I could say "these are the packages I want to install as editable" and easy_install would pick those up (maybe detecting based on what package index the package was found in) and install them in src/ as editable. sys.path would contain /usr/lib/python2.4, optionally /usr/lib/python2.4/site-packages, and env/lib/python2.4/, and all the similar directories. Unfortunately figuring out what "similar" directories there are is hard. sys.path on my machine now has 63 entries normally and 12 with python -S. I guess I'd really like to start with 12 and build up, instead of 63 and try to strip them down. Installation as a whole is an open issue. Putting in env/setup.cfg with the setting specific to that working environment works to a degree -- easy_install will pick it up if invoked from there. But that doesn't work with setup.py develop, or setup.py install, or some other scenarios. The system distutils.cfg doesn't really work, because the only expansion it knows how to do is of user directories, so there's little way to pass interesting information in (like a "this is my setup.cfg" environmental variable or something). Maybe with PYTHONPATH to indicate the working environment, and a distutils monkeypatch put into lib/python2.4/distutils/__init__.py? I played around with putting the path setup in sitecustomize, but that runs after site.py, and doesn't run at all if python -S is used, so it seems like it brings in too much before it can remove stuff. Another option is a completely new python interpreter bound to the environment. Basically the virtual-python.py option (http://peak.telecommunity.com/DevCenter/EasyInstall#creating-a-virtual-python). In this model using env/bin/python indicate the proper environment, and you'd have local installs of *everything* including easy_install. This fixes so many problems without crazy hacks that it strongly appeals to me, especially if we can make it somewhat lighter. I get this /usr/lib/python2.4.zip on my path, that doesn't usually exist (does it ever get created by default); if we could create that somehow on demand and use such a big-bundle-zip, that seems lighter and faster and nicer. If we just put .pyc files in it, and those .pyc files refer back to the actual module source (in /usr/lib/python2.4/), then tracebacks should also still work, right? No actual symlinks either, so it should work on Windows. I'm not entirely sure where I'm going with this, though. Sorry for the length. I've been stewing on this without a lot of progress since PyCon so I thought I'd just throw out my current thoughts. Maybe what I really want to do is hack on virtual-python.py some more and see where that gets me. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ben at groovie.org Tue Mar 7 20:36:49 2006 From: ben at groovie.org (Ben Bangert) Date: Tue, 7 Mar 2006 11:36:49 -0800 Subject: [Distutils] sandbox trying to run things before they're installed...again? Message-ID: <524EACC5-49B0-4CF0-AD46-F21205536A0C@groovie.org> I think I had this problem earlier, and it went away, but now its back for some reason. Several users have reported it to me as well. Apparently the sandbox testing runs, and tries to import a package that wasn't installed yet. Is there anyway to make it actually install dependencies *before* doing its sandbox stuff? Thanks, Ben The traceback looks like this: [ben-bangerts-power-mac-g4:~] ben% sudo easy_install -U Pylons==dev Searching for Pylons==dev Reading http://www.python.org/pypi/Pylons/ Reading http://pylons.groovie.org/ Best match: Pylons dev Downloading http://pylons.groovie.org/svn/Pylons/trunk#egg=Pylons-dev Doing subversion checkout from http://pylons.groovie.org/svn/Pylons/ trunk to /tmp/easy_install-8oXRPG/trunk Processing trunk Running setup.py -q bdist_egg --dist-dir /tmp/easy_install-8oXRPG/ trunk/egg-dist-tmp-vH9ebx Traceback (most recent call last): File "/usr/local/bin/easy_install", line 6, in ? sys.exit( File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a10-py2.4.egg/setuptools/ command/easy_install.py", line 1506, in main with_ei_usage(lambda: File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a10-py2.4.egg/setuptools/ command/easy_install.py", line 1495, in with_ei_usage return f() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a10-py2.4.egg/setuptools/ command/easy_install.py", line 1509, in distclass=DistributionWithoutHelpCommands, **kw File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/core.py", line 149, in setup dist.run_commands() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a10-py2.4.egg/setuptools/ command/easy_install.py", line 211, in run self.easy_install(spec, not self.no_deps) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a10-py2.4.egg/setuptools/ command/easy_install.py", line 446, in easy_install return self.install_item(spec, dist.location, tmpdir, deps) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a10-py2.4.egg/setuptools/ command/easy_install.py", line 461, in install_item dists = self.install_eggs(spec, download, tmpdir) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a10-py2.4.egg/setuptools/ command/easy_install.py", line 655, in install_eggs return self.build_and_install(setup_script, setup_base) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a10-py2.4.egg/setuptools/ command/easy_install.py", line 930, in build_and_install self.run_setup(setup_script, setup_base, args) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a10-py2.4.egg/setuptools/ command/easy_install.py", line 919, in run_setup run_setup(setup_script, args) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a10-py2.4.egg/setuptools/ sandbox.py", line 26, in run_setup DirectorySandbox(setup_dir).run( File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a10-py2.4.egg/setuptools/ sandbox.py", line 63, in run return func() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6a10-py2.4.egg/setuptools/ sandbox.py", line 29, in {'__file__':setup_script, '__name__':'__main__'} File "/Users/ben/Programming/Python/pudge/setup.py", line 5, in ? import os File "/tmp/easy_install-8oXRPG/trunk/pylons/__init__.py", line 6, in ? File "/tmp/easy_install-8oXRPG/trunk/pylons/decorators.py", line 4, in ? ImportError: No module named simplejson From pje at telecommunity.com Tue Mar 7 20:55:05 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 07 Mar 2006 14:55:05 -0500 Subject: [Distutils] sandbox trying to run things before they're installed...again? In-Reply-To: <524EACC5-49B0-4CF0-AD46-F21205536A0C@groovie.org> Message-ID: <5.1.1.6.0.20060307145322.01fd5760@mail.telecommunity.com> At 11:36 AM 3/7/2006 -0800, Ben Bangert wrote: >I think I had this problem earlier, and it went away, but now its >back for some reason. Several users have reported it to me as well. >Apparently the sandbox testing runs, and tries to import a package >that wasn't installed yet. Is there anyway to make it actually >install dependencies *before* doing its sandbox stuff? No; the setup script has to be run in order to find out what the dependencies are. Note that as a general rule, distutils setup scripts shouldn't import the code they're trying to install. It tends to lead to problems on users' machines that don't show up on the developer's machine. From ben at groovie.org Tue Mar 7 21:28:57 2006 From: ben at groovie.org (Ben Bangert) Date: Tue, 7 Mar 2006 12:28:57 -0800 Subject: [Distutils] sandbox trying to run things before they're installed...again? In-Reply-To: <5.1.1.6.0.20060307145322.01fd5760@mail.telecommunity.com> References: <5.1.1.6.0.20060307145322.01fd5760@mail.telecommunity.com> Message-ID: <5777D9FC-FB03-4DEF-AD88-89A62D5B6BC5@groovie.org> On Mar 7, 2006, at 11:55 AM, Phillip J. Eby wrote: > No; the setup script has to be run in order to find out what the > dependencies are. > > Note that as a general rule, distutils setup scripts shouldn't > import the code they're trying to install. It tends to lead to > problems on users' machines that don't show up on the developer's > machine. So.... the setup script runs the code to see what the dependencies are, the code dies a horrid death because of course the dependency has yet to be installed... thus everything dies and nothing is installed. Isn't that a bit of a problem? I mean, the code can't be run, because the dependency has yet to be installed.... which means the user is left manually installing dependencies because setuptools refuses to install the package before trying to run code from it. Shouldn't setuptools see that the install is failing because the dependency needs to be installed, and install the dependency? Thanks, Ben From pje at telecommunity.com Tue Mar 7 21:47:12 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 07 Mar 2006 15:47:12 -0500 Subject: [Distutils] sandbox trying to run things before they're installed...again? In-Reply-To: <5777D9FC-FB03-4DEF-AD88-89A62D5B6BC5@groovie.org> References: <5.1.1.6.0.20060307145322.01fd5760@mail.telecommunity.com> <5.1.1.6.0.20060307145322.01fd5760@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060307154359.01fd7af8@mail.telecommunity.com> At 12:28 PM 3/7/2006 -0800, Ben Bangert wrote: >Shouldn't setuptools see that the install is failing because the >dependency needs to be installed, and install the dependency? If it knew what the dependencies were, there'd be no need to run the setup script *first*. (Note that most setup scripts that import themselves or their dependencies do it *before* they call setup(), so if the script fails, it's not like setuptools has any idea what arguments they were *going to* call setup() with.) From ben at groovie.org Tue Mar 7 22:30:54 2006 From: ben at groovie.org (Ben Bangert) Date: Tue, 7 Mar 2006 13:30:54 -0800 Subject: [Distutils] sandbox trying to run things before they're installed...again? In-Reply-To: <5.1.1.6.0.20060307154359.01fd7af8@mail.telecommunity.com> References: <5.1.1.6.0.20060307145322.01fd5760@mail.telecommunity.com> <5.1.1.6.0.20060307145322.01fd5760@mail.telecommunity.com> <5.1.1.6.0.20060307154359.01fd7af8@mail.telecommunity.com> Message-ID: <2D2CAFE2-837D-4DF3-92A2-29EF775D5078@groovie.org> On Mar 7, 2006, at 12:47 PM, Phillip J. Eby wrote: > At 12:28 PM 3/7/2006 -0800, Ben Bangert wrote: >> Shouldn't setuptools see that the install is failing because the >> dependency needs to be installed, and install the dependency? > > If it knew what the dependencies were, there'd be no need to run > the setup script *first*. > > (Note that most setup scripts that import themselves or their > dependencies do it *before* they call setup(), so if the script > fails, it's not like setuptools has any idea what arguments they > were *going to* call setup() with. Indeed, I just noticed that setup.py was importing version from the package, and the location it was importing, then imported other stuff that had the dependencies. Thanks for the pointer on what was going wrong. - Ben From dangoor at gmail.com Tue Mar 7 22:37:59 2006 From: dangoor at gmail.com (Kevin Dangoor) Date: Tue, 7 Mar 2006 16:37:59 -0500 Subject: [Distutils] sandbox trying to run things before they're installed...again? In-Reply-To: <2D2CAFE2-837D-4DF3-92A2-29EF775D5078@groovie.org> References: <5.1.1.6.0.20060307145322.01fd5760@mail.telecommunity.com> <5.1.1.6.0.20060307154359.01fd7af8@mail.telecommunity.com> <2D2CAFE2-837D-4DF3-92A2-29EF775D5078@groovie.org> Message-ID: <3f085ecd0603071337w6b7195b6l56a95d330d540f39@mail.gmail.com> On 3/7/06, Ben Bangert wrote: > Indeed, I just noticed that setup.py was importing version from the > package, and the location it was importing, then imported other stuff > that had the dependencies. Thanks for the pointer on what was going > wrong. This appears to be the general solution to that particular problem. In setup.py: execfile(os.path.join("turbogears", "release.py")) or something along those lines. Kevin From pje at telecommunity.com Tue Mar 7 23:00:30 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 07 Mar 2006 17:00:30 -0500 Subject: [Distutils] sandbox trying to run things before they're installed...again? In-Reply-To: <3f085ecd0603071337w6b7195b6l56a95d330d540f39@mail.gmail.co m> References: <2D2CAFE2-837D-4DF3-92A2-29EF775D5078@groovie.org> <5.1.1.6.0.20060307145322.01fd5760@mail.telecommunity.com> <5.1.1.6.0.20060307154359.01fd7af8@mail.telecommunity.com> <2D2CAFE2-837D-4DF3-92A2-29EF775D5078@groovie.org> Message-ID: <5.1.1.6.0.20060307165930.036f3b18@mail.telecommunity.com> At 04:37 PM 3/7/2006 -0500, Kevin Dangoor wrote: >On 3/7/06, Ben Bangert wrote: > > Indeed, I just noticed that setup.py was importing version from the > > package, and the location it was importing, then imported other stuff > > that had the dependencies. Thanks for the pointer on what was going > > wrong. > >This appears to be the general solution to that particular problem. In >setup.py: > >execfile(os.path.join("turbogears", "release.py")) > >or something along those lines. That only works if 'release.py' doesn't import any dependencies. :) That is, the issue doesn't have anything to do with imports per se, only imports that cause dependencies to come into play. From iv at lantic.net Wed Mar 8 08:15:12 2006 From: iv at lantic.net (Iwan Vosloo) Date: 08 Mar 2006 09:15:12 +0200 Subject: [Distutils] Working environment In-Reply-To: <440DCA93.2030207@colorstudy.com> References: <440DCA93.2030207@colorstudy.com> Message-ID: <87oe0hz9tr.fsf@localhost.localdomain> Ian Bicking writes: > Another option is a completely new python interpreter bound to the > environment. Basically the virtual-python.py option > (http://peak.telecommunity.com/DevCenter/EasyInstall#creating-a-virtual-python). > In this model using env/bin/python indicate the proper environment, > and you'd have local installs of *everything* including easy_install. > This fixes so many problems without crazy hacks that it strongly appeals > to me, especially if we can make it somewhat lighter. I get this > /usr/lib/python2.4.zip on my path, that doesn't usually exist (does it > ever get created by default); if we could create that somehow on demand > and use such a big-bundle-zip, that seems lighter and faster and nicer. > If we just put .pyc files in it, and those .pyc files refer back to > the actual module source (in /usr/lib/python2.4/), then tracebacks > should also still work, right? No actual symlinks either, so it should > work on Windows. I'm not entirely sure where I'm going with this, though. Just a thought... This sounds like a clean approach and powerful. What if, for example, you work on a machine with python 2.4 installed, but you want an environment with python 2.5? With what you're proposing here you could go totally heavyweight and have the whole python distribution installed in your working invironment only. Or distro's could have a python-dev type package which installs in a location other than their official python installation, so that you can still do your more lightweight thing. Reminds me of a database 'installation' with seperate 'instances' of it on the same machine, each with its own data. I'm actually thinking that this is useful especially in the case where you're working on a distribution which is tightly bound to an older version of python - but you want to develop on a newer version. -i From anthony at interlink.com.au Wed Mar 8 14:35:32 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu, 9 Mar 2006 00:35:32 +1100 Subject: [Distutils] Working environment In-Reply-To: <87oe0hz9tr.fsf@localhost.localdomain> References: <440DCA93.2030207@colorstudy.com> <87oe0hz9tr.fsf@localhost.localdomain> Message-ID: <200603090035.33612.anthony@interlink.com.au> On Wednesday 08 March 2006 18:15, Iwan Vosloo wrote: > Just a thought... This sounds like a clean approach and powerful. > What if, for example, you work on a machine with python 2.4 > installed, but you want an environment with python 2.5? With what > you're proposing here you could go totally heavyweight and have the > whole python distribution installed in your working invironment > only. Ick. I'm sure I'm not the only one who remembers telling perl's CPAN module to auto-build all dependencies, only to come back and find it building a new version of Perl to install in a local directory. I really don't like this idea at all. I should point out that I don't mind the working environment idea, just not the idea of a version of Python being built in there. You can already install multiple versions of Python with "make altinstall" (or your distro's packaging system). Anthony -- Anthony Baxter It's never too late to have a happy childhood. From pje at telecommunity.com Thu Mar 9 00:31:01 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 08 Mar 2006 18:31:01 -0500 Subject: [Distutils] Smart conflict management/resolution for setuptools Message-ID: <5.1.1.6.0.20060308162152.01fce9a8@mail.telecommunity.com> Okay, so I've been looking over the various remaining issues that people have encountered with setuptools installation, and there's a kind of theme emerging: conflict management and resolution. First, let me give some of the problems. First off, the conflict detection system at installation time isn't very bright. It'll happily report conflicts with system-installed packages, even ones that are installed using --single-version-externally-managed. It doesn't pay attention to what order stuff is installed in on sys.path, either; it just says, "hey, this *might* conflict" and kicks up a fuss. The "develop" command currently doesn't check for conflicts at all. And finally, the --delete-conflicting flag will happily attempt to delete system-installed packages, even if they're .egg-info ones. Ouch. And those are just the problems with install-time conflict detection. Runtime conflict resolution's not that great, either. Once an egg gets in the working set (e.g., due to being the default or system-installed version of something), it never comes out, meaning that you can't really override the default version. There's a special trick that's used to bypass this issue for scripts, that for all practical purposes just throws out sys.path and starts over if there's a conflict, but it's a little drastic and in any case can't be generalized. Finally, Guido has proposed allowing setuptools to get into the stdlib for Python 2.5, but because it's a moving target, we need a way (ala PyXML) to allow users to safely install newer versions of setuptools in such a way as to replace the stdlib-supplied version. After a fair amount of head-scratching, I think I have some ideas that can fix all of these problems, but I want to air them out here before starting implementation, to see if anybody can find any holes in the plan or improvements to it. The runtime part is pretty easy and less likely to have any controversial impacts, so I'll save that part for last, and deal with the installation conflict issues first. Installation Conflicts ====================== The original reason for having installation conflict detection was that eggs added using .pth files normally ended up at the tail end of sys.path, after any site-packages or other locally-installed packages. In effect, eggs were always the lowest man on the totem pole, so you *had* to get rid of any other installed version in order for it to work. But in setuptools 0.6a6, I added logic to pkg_resources that automatically placed activated eggs *before* their containing directory in sys.path. Thus, an egg in site-packages would be placed (at runtime) *before* the site-packages directory in sys.path, thereby overriding any "legacy" or "system-installed" packages there. This is also true of any other directory on sys.path: if it contains an egg, and you activate the egg, the egg will be placed just before that directory in sys.path. So in theory, conflict detection shouldn't have been needed since 0.6a6. In practice, however, people need to be able to use eggs *without* importing pkg_resources and munging sys.path, so this is only a partial fix. However, it does point the way to a full fix. If we could get .pth files to do essentially the same trick, we'd be all set. Better still, if we could do it in such a way that the added entries were before the stdlib on sys.path, then we would at the same time accomplish the other need: upgrading a stdlib-provided version of setuptools! At PyCon, I had a few hallway conversations that touched on this issue, and as I mentioned then, the simplest possible hack is something like adding this line to a .pth file (e.g. setuptools.pth): import sys; sys.path.insert(0,"/path/to/setuptools.egg") But this isn't very helpful in the case where you might have more than one sys.path directory with an easy-install.pth file in it. Why? Because the later the directory is on sys.path, the *earlier* on sys.path the egg will be inserted. Thus, the eggs end up in reverse order, with eggs installed in site-packages would take precedence over even those on PYTHONPATH! A Half-baked Solution --------------------- So, I've been racking my brain for a solution to this. Mostly, I was thinking in terms of tracking what egg version is selected, but today I realized there is a simpler solution. We just need to ensure that the insertion point *moves forward* each time we insert an egg, e.g.: import sys; sys.__egginsert=p=getattr(sys,'__egginsert',-1)+1; \ sys.path.insert(p,"/path/to/the.egg") I've used a \ continuation for clarity here, but of course the real .pth file couldn't do this; the whole thing has to be on one line. I've considered a few other ways to do this, like inserting a marker value directly into sys.path: import sys; '@@@' not in sys.path and sys.path.insert(0,'@@@'); \ sys.path.insert(sys.path.index('@@@'),"/path/to/the.egg") But this has the problem of needing to choose a suitable marker value that won't be a valid import string, and doesn't introduce security problems, etc. So I came up with the __egginsert approach as a way to fix that. The main (potential) problems I see remaining with __egginsert are: 1. Change in semantics of current install (can't be helped, but needs investigation) 2. If somebody else is crazy enough to do something similar, hilarity may ensue as hacks in different .pth files use insert points that don't go where they think. (Not that likely; few people have even heard of .pth files, and in any case the insertions will retain a consistent order *within* each set, putting an upper limit on the possible chaos.) 3. EasyInstall will have to be able to *read* these lines as well as write them, and parsing one correctly looks a bit daunting to do. 4. The 'site' module has to 'exec' lines beginning with 'import', so there's a compile-and-exec delay imposed by each line, thus increasing the incremental startup cost for each installed egg. A More Complete Solution ------------------------ So here's a new idea, made up fresh while writing this: import sys; sys.__oldpath=sys.path[:]; sys.path[:]=[] ... normal .pth lines go here ... import sys; new=sys.path[:]; sys.path[:]=sys.__oldpath; \ p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; \ sys.__egginsert = p+len(new) (Again, the \ continuations won't be in the real file.) Anyway, what this does is temporarily nuke sys.path, then let the normal .pth processing logic run until we get to the end of the file, where we restore it and slap in the list of directories that were added by the 'site' module. Or, perhaps more simply: import sys; sys.__plen = len(sys.path) ... normal .pth lines go here ... import sys; new=sys.path[sys.__plen:]; \ del sys.path[sys.__plen:]; \ p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; \ sys.__egginsert = p+len(new) This doesn't wipe out sys.path during the interim processing, although I suppose it is really just as complex as the previous version. Somehow, it just feels "safer" not to ditch sys.path during the processing. Anyway, either of these will keep the number of 'exec's capped at a maximum of 2 lines, *and* they'll make easy_install's parsing job easy, because it can just ignore the "import" lines when reading, and just regenerate them when writing. This change would also thus affect only easy_install'ed eggs; it would have no effect on normal .pth processing. But I would have to slightly alter the existing 'site.py' hack that setuptools installs in PYTHONPATH directories, so that it resets the sys.__egginsert position to zero before processing the .pth files on PYTHONPATH, and then restores its original value afterward. (That way, eggs installed on PYTHONPATH will come before those installed in site-packages, but any subsequent addsitedir() operations will add to the right spot, more or less.) That does suggest that there's some fragility involved in this process, if you use addsitedir() at runtime. This is probably the biggest issue with this approach. I did a bit of googling, however, and was only able to find 3 or 4 projects that do any kind of addsitedir stuff, including the usual suspects such as Twisted and Zope. :) I didn't find anything that looked like it would be negatively affected, although I admit I didn't study the surrounding code too deeply. Mostly, they were doing things that were basically the same as what we're trying to do here: i.e., support .pth files in PYTHONPATH and in some cases, put them first on sys.path. The only place where confusing results are likely, is the case where somebody calls addsitedir() *after* the pkg_resources module is imported, thus causing the working_set to be out of sync with sys.path. Of course, this is *already* a possible problem if you munge sys.path after importing pkg_resources, which suggests that some additional conflict detection there would also be helpful. (Side rant: this is one of those times where the singleton nature of certain Python features is really really annoying. If only one could instantiate interpreters in standard Python!) Anyway, sys.path manipulation is already fraught with numerous dangers, so I won't dwell on that issue for now. Perhaps in 0.7 or 0.8 I'll revamp the WorkingSet class so it can deal with dynamic manipulation of an externally-supplied list. Proposed Installation Strategy ------------------------------ To summarize my proposal for handling installation "conflicts": * EasyInstall will write special .pth files with a header and trailer "import" hack. It will ignore "import" lines when reading them. * I'll change the existing site.py hack to accomodate the out-of-order insertion, and make EasyInstall update any existing hacked site.py files. * EasyInstall will stop checking for (and deleting) installation conflicts, because there will *no longer be such a thing*. The options that control conflict detection will remain, but will have no function except to issue deprecation warnings. And to summarize the effects of these changes: * Eggs installed by EasyInstall will have sys.path precedence over everything else, including the standard library, script directory, etc. They will, however, have a precedence order amongst themselves that reflects the sequence in which the .pth files were loaded. * Invoking site.addsitedir() after pkg_resources is imported may produce slightly weirder results than it already does. :) * EasyInstall will be able to upgrade stdlib packages, not just via PYTHONPATH installs, but also via site-packages installs. And the issues that remain open are: * Should --single-version-externally-managed installations check for conflicts? Of course, users of the option are theoretically more experienced users/packagers, and are also accustomed to the distutils' default (and thoroughly unsafe) behavior, so perhaps they should be left to their own devices. * Should pkg_resources resort to more extreme measures to track the runtime value of sys.path? We'll perhaps look more at this a bit more closely below, in the section on runtime conflict management. Does anybody see any issues I haven't covered? Any ideas for improvements? Runtime Conflicts ================= The solution to runtime conflicts was a little easier to figure out, although now that I've figured out the installation conflicts side, I think the runtime side is actually going to be a lot tougher to implement correctly, as there are a lot more moving parts. But the idea is relatively simple to explain: when trying to resolve dependencies, handle VersionConflict errors by: 1. Removing the conflicting package from the working set, if and only if it's safe to do so. 2. Retry the operation, and repeat. If a package isn't safe to remove, reraise the conflict error after restoring any previously-removed packages, so that the working set is rolled back to its original state. At this level, it's all pretty simple. The tricky parts are in the details: 1. This isn't really compatible with the existing API, for anything but the require() operation. There's no sane way to change the resolve() API to work with this, because resolve() isn't supposed to have side effects, and it only returns a list of distributions to *add*, not ones to *remove*. 2. Defining "safe to remove" is complex. As a first approximation, it's safe to remove a distribution from the working set if: a) it's the only distribution with that file/directory name (Think of .egg-info dirs all sitting in site-packages; you can't remove one of those babies without removing *all* system-installed eggs in site-packages, since they all share a single sys.path entry!) b) it hasn't had anything imported from it yet c) no namespace packages it participates in have been imported Those latter two conditions are interesting, because they basically mean you have to go through every sys.modules entry looking for names starting with anything in the egg's top_level list, and then check those modules' __file__ and __path__ values to see if anything starts with the egg's location. This is likely to be slow, but that's not really a big deal; this is a fallback for resolving a runtime version conflict, after all. But it's somewhat tricky to test well, due to the singletons (e.g. sys.modules) that are involved. 3. Working sets don't actually allow you remove anything currently, and I'm not sure how to actually implement that feature! So, after reviewing this, I think I'm going to avoid touching this during the remainder of the 0.6 development cycle, because it seems too likely to introduce instability. Also, requirement 2c above isn't really practical before 0.7, because of the current eager loading of namespace packages. It would basically rule out conflict resolution for eggs that currently participate in namespace packages. Last, but not least, if I put it off till 0.7, I can go whole hog on refactoring the WorkingSet class, to also support better sys.path tracking at runtime. And if some corners of the API have to get tweaked a bit to make that work in 0.7, I can probably live with it. Any other thoughts or suggestions? From pje at telecommunity.com Sat Mar 11 01:43:41 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 10 Mar 2006 19:43:41 -0500 Subject: [Distutils] Better conflict handling & system packager support In-Reply-To: <5.1.1.6.0.20060308162152.01fce9a8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060310131913.04053250@mail.telecommunity.com> The SVN head of setuptools now implements the strategy described below, allowing it to safely interoperate with system package managers, and even to allow safely installing replacements for packages found in the stdlib. The only potential downside is that Python startup imports may now be a bit slower, since the eggs are placed ahead of the stdlib on sys.path. However, since the people who are really concerned about super-fast startup time have already chosen not to use easy_install, I figure this won't hurt them. :) But it *will* help everybody who's trying to have system packages interoperate with EasyInstall. For example, if you have a .deb, .rpm, or win32.exe of a package installed --single-version-externally-managed, or if you just have some legacy versions of things floating around, this new feature will do you a lot of good. (By the way, the new SVN version also automatically sets --single-version-externally-managed when the "--root" option is given to the "install" command. This should improve compatibility with most third-party "bdist_*" commands and other custom build/install processes that people might have, and fixes a problem in 0.6a10 where install refuses to run with "--root" if you *haven't* set --single-version-externally-managed.) Please upgrade (using "ez_setup.py setuptools==dev")and let me know how the new features work for you. Thanks. At 06:31 PM 3/8/2006 -0500, Phillip J. Eby wrote: >Proposed Installation Strategy >------------------------------ > >To summarize my proposal for handling installation "conflicts": > >* EasyInstall will write special .pth files with a header and trailer >"import" hack. It will ignore "import" lines when reading them. > >* I'll change the existing site.py hack to accomodate the out-of-order >insertion, and make EasyInstall update any existing hacked site.py files. > >* EasyInstall will stop checking for (and deleting) installation conflicts, >because there will *no longer be such a thing*. The options that control >conflict detection will remain, but will have no function except to issue >deprecation warnings. > >And to summarize the effects of these changes: > >* Eggs installed by EasyInstall will have sys.path precedence over >everything else, including the standard library, script directory, >etc. They will, however, have a precedence order amongst themselves that >reflects the sequence in which the .pth files were loaded. > >* Invoking site.addsitedir() after pkg_resources is imported may produce >slightly weirder results than it already does. :) > >* EasyInstall will be able to upgrade stdlib packages, not just via >PYTHONPATH installs, but also via site-packages installs. From pje at telecommunity.com Sat Mar 11 01:55:35 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 10 Mar 2006 19:55:35 -0500 Subject: [Distutils] Double-click install for eggs on Windows Message-ID: <5.1.1.6.0.20060310194357.01fb79a0@mail.telecommunity.com> Just FYI, if you want to make double-clicking .egg files automatically install them with EasyInstall, you can do it by putting the following entries in the Windows registry (Python 2.3 example shown): Windows Registry Editor Version 5.00 [HKEY_CLASSES_ROOT\.egg] @="Python.Egg" [HKEY_CLASSES_ROOT\Python.Egg] "EditFlags"=dword:00000000 "BrowserFlags"=dword:00000008 @="Python Egg" "AlwaysShowExt"="" [HKEY_CLASSES_ROOT\Python.Egg\DefaultIcon] @="C:\\Python23\\python.exe,0" [HKEY_CLASSES_ROOT\Python.Egg\shell] @="Install Egg for Python 2.3" [HKEY_CLASSES_ROOT\Python.Egg\shell\Install Egg for Python 2.3] [HKEY_CLASSES_ROOT\Python.Egg\shell\Install Egg for Python 2.3\command] @="c:\\windows\\command\\cmd.exe /D /K C:\\Python23\\Scripts\\easy_install \"%L\"" I wasn't able to get Firefox to directly install straight from a download, but saving to disk and then double-clicking worked. It might be interesting to add a script to setuptools that would make these entries for you, automatically adjusting paths where necessary. :) Note that the '/K' part of the cmd.exe line ensures that a command shell window is left open for confirmation when easy_install finishes. That way, you can see any errors, warnings, etc. Thanks to Tiago Cogumbreiro, by the way, for inadvertently giving me the idea that something like this might be possible without writing any new code. :) From jim at zope.com Sat Mar 11 16:47:07 2006 From: jim at zope.com (Jim Fulton) Date: Sat, 11 Mar 2006 10:47:07 -0500 Subject: [Distutils] Working environment In-Reply-To: <440DCA93.2030207@colorstudy.com> References: <440DCA93.2030207@colorstudy.com> Message-ID: <4412F0FB.9090907@zope.com> Ian Bicking wrote: > So, coming back to the idea of a working environment, an isolated and > more-or-less self-contained environment for holding installed packages. > Sorry if this is a little scattered. I'm just summarizing my thoughts > and the open issues I see, in no particular order (because I'm not sure > what order to approach these things in). > > I'm assuming such an environment will be encapsulated in a single > directory, looking something like: > > env/ > bin/ > lib/python2.4/ > src/ > conf/ > > The conf/ directory doesn't really relate to much of this specifically, > but in many situations it would be useful. Depending on the situation, > other subdirectories may exist. OK, I'll pretend that it doesn't exist. ;) Actually, I'll have a bit to say about configuration below. > Each of the scripts in bin/ should know what their working environment > is. This is slightly tricky, depending on what that means. If it is a > totally isolated environment -- no site-packages on sys.path -- then I > feel like the script wrappers have to be shell scripts, to invoke Python > with -S (which is hard to do portably on the #! line). I'll note that this isn't important to me at all. I'm not opposed to allowing it, but I don't need it. I never use the python that comes with my system. I always build Python from source and I can therefore keep that install clean, complete, and relatively predictable. (I am often annoyed by Python's *implicit* decision of which extension modules to install. Hopefully, someday, the presense of a packaging system will enable saner approaches to optional extensions.) In fact, I usually have multiple versions of Python available. I then will have many working environments that use these custtom Pythons. (The custom Python's are not part of any working environment.) Similarly, in production environments, we install custom builds of Python that our applications use. ... > lib/python2.4/ is for packages. Minor note: this needs to be flexible. I'd be more inclined to go with something shallower and simpler, like just "lib", > I'm almost inclined to say that > --single-version-externally-managed makes sense on some level, with a > record kept in some standard place (lib/python2.4/install-record.txt?) > -- but I'm basically indifferent. I at least don't see a need for > multiple simultaneous versions in this setup, and multiple versions do > lead to confusion. Version metadata is still nice, of course. Increasingly, projects I work on involve multiple Python applications with each application requiring different sets of packages and, sometimes, different package versions. Basically, I want a single version for each application, but multiple versions in the environment. What appeals to me is a compromise between the single-version and multi-version install. With a single-version install, you end up with an easy-install.pth that names the packages (and package versions) being used. This has the advantage of being deterministic. You always get the packages listed there. This affects all applications running in the environment. In a multi-version install, multiple versions are stored in the environment. Different applications can use different versions. When an application is run, a root package is specified in a script (the package containing the entry point) and pkg_resources tries to find a suitable combination of packages that satisfy the closure of the root package's dependencies. This process is somewhat non-deterministic. Installation of a new package could change the set of packages used beyond that individual package, causing implicit and unintended "upgrades" that you refered to in your talk at PyCon. What I want (or think I want :) is per-application (per-script) .pth files. When I install a script, I want the packages needed by the script to be determined at that time and written to a script-specific .pth file. So, for example, if I install a script "foo" (foo.exe, whatever), I also get foo.pth and have the generated foo script prepend the contents of that file to sys.path on startup. I don't mind if that file is required to be in the same directory as the script. I also wouldn't mind if the contents if the path entries were simply embedded in the script. If I want to upgrade the packages being used, I simply update the scripts. There are veriopus ways that this could be made easy. Even better, the upgrade process could optionally make backups of the .pth files or scripts, allowing easy rollbacks of upgrades. I think that this approach combines the benefits of single- and multi-version install. Different applications in an enevironment can require different package versions. The choice of packages for an application is determined at discrete explicit install times, rather than at run time. ... > Installation as a whole is an open issue. Putting in env/setup.cfg with > the setting specific to that working environment works to a degree -- > easy_install will pick it up if invoked from there. But that doesn't > work with setup.py develop, or setup.py install, or some other > scenarios. I don't follow this. It seems to work for us, at least for setup.py develop. The main lamosity is depending on the current working directory. Personally, I'd like (at least optionally) the generated easy_install script to include information about the environment that was present when it was installed. That is, I'd be happy to see setup.cfg consuled when easy_install is installed and then ignored thereafter. If you wanted to change the environment definition, then simply update setup.cfg amd reinstall easy_install. Or, better yet, when installing easy_install, have the generated script remember the full path of the setup.cfg file used and reread that file when run, regardless of where easy_install is run from. This brings me to the topic of configuration. Today, I write wrapper scripts "by hand", I may have some application like Zope, or ZEO or our test runner that is implemented by a an entry point in a module. Then there's a wrapper script that imports the module and calls the entry point. The wrapper script is written (manually or with some custom installation script) to include the path to be used and configuration data, which may be the location of a configuration file. I really like the fact that easy_install will generate wrapper scripts for me, but I really need more control over how these scripts are generated to include *both* path and configuration information. ... > Another option is a completely new python interpreter bound to the > environment. Basically the virtual-python.py option > (http://peak.telecommunity.com/DevCenter/EasyInstall#creating-a-virtual-python). > In this model using env/bin/python indicate the proper environment, > and you'd have local installs of *everything* including easy_install. > This fixes so many problems without crazy hacks that it strongly appeals > to me, especially if we can make it somewhat lighter. I get this > /usr/lib/python2.4.zip on my path, that doesn't usually exist (does it > ever get created by default); if we could create that somehow on demand > and use such a big-bundle-zip, that seems lighter and faster and nicer. > If we just put .pyc files in it, and those .pyc files refer back to > the actual module source (in /usr/lib/python2.4/), then tracebacks > should also still work, right? No actual symlinks either, so it should > work on Windows. I'm not entirely sure where I'm going with this, though. I think that something much simpler can be made to work. I think a little more control over how scripts get generated would go a long way. (There is still the question of the interactive interpreter...) Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From bkc at murkworks.com Sat Mar 11 19:40:35 2006 From: bkc at murkworks.com (Brad Clements) Date: Sat, 11 Mar 2006 13:40:35 -0500 Subject: [Distutils] Jython support? Message-ID: <441319A3.1030202@murkworks.com> I would like to use setuptools, eggs with entry points, and so forth with Jython. Is this possible? If not, what's needed? From ianb at colorstudy.com Sat Mar 11 22:22:28 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Sat, 11 Mar 2006 15:22:28 -0600 Subject: [Distutils] working-env.py Message-ID: <44133F94.8080505@colorstudy.com> I wrote a script last night that implements some of the ideas of a working environment, similar in goals to virtual-python.py, but it doesn't actually work like that. http://svn.colorstudy.com/home/ianb/working-env.py When you run "python working-env.py env" you get: env/ bin/ activate (something you source in your shell to activate this environment) conf/ src/ lib/python2.4/ distutils/__init__.py distutils/distutils.cfg setuptools/__init__.py setuptools.pth site.py You activate the environment by adding env/lib/python2.4 to your PYTHONPATH, and maybe adding env/bin/ to your PATH. The custom site.py accomplishes most of the work, I guess. It's just like the normal site.py, except it doesn't add site-packages (I should probably make an option so you can set it up either way). distutils/__init__.py is Ronald Oussoren's suggestion for monkeypatching distutils. It both manages to get the working environment's distutils.cfg to be used (comes for free, since distutils.dist looks alongside distutils.__file__), and it substitutes __WORKING__ for the actual working directory. The setuptools monkeypatch changes how scripts are generated. But I don't like how it works. It changes scripts to look more like: #!/usr/bin/python -S import sys, os join, dirname = os.path.join, os.path.dirname lib_dir = join(dirname(dirname(__file__)), 'lib', 'python%s.%s' % tuple(sys.version_info[:2])) sys.path.insert(0, lib_dir) import site ... normal stuff ... Monkeypatching distutils is fine, because it's not going anywhere, but monkeypatching setuptools is not going to be so reliable. But I guess we just need to figure out exactly what we want to do, then move those changes to setuptools. Also, distutils should probably be monkeypatched directly in some fashion, so that distutils install also installs scripts with this header. A goal I have -- that is pretty much accomplished so far -- is that the working environment be movable, and all paths be relative. But setuptools creates some absolute directory names in .pth files, I believe. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From ianb at colorstudy.com Sat Mar 11 22:39:18 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Sat, 11 Mar 2006 15:39:18 -0600 Subject: [Distutils] Working environment In-Reply-To: <4412F0FB.9090907@zope.com> References: <440DCA93.2030207@colorstudy.com> <4412F0FB.9090907@zope.com> Message-ID: <44134386.3080205@colorstudy.com> Jim Fulton wrote: >> Each of the scripts in bin/ should know what their working environment >> is. This is slightly tricky, depending on what that means. If it is >> a totally isolated environment -- no site-packages on sys.path -- then >> I feel like the script wrappers have to be shell scripts, to invoke >> Python with -S (which is hard to do portably on the #! line). > > I'll note that this isn't important to me at all. I'm not opposed > to allowing it, but I don't need it. I can go both ways. I think this should be configurable when you set up the working environment, hopefully someplace where it is easy to change. >> lib/python2.4/ is for packages. > > Minor note: this needs to be flexible. I'd be more inclined to go > with something shallower and simpler, like just "lib", Why? Top-level packages aren't portable, since .pyc files aren't portable. Eggs are portable, since they contain the Python version. > > I'm almost inclined to say that >> --single-version-externally-managed makes sense on some level, with a >> record kept in some standard place (lib/python2.4/install-record.txt?) >> -- but I'm basically indifferent. I at least don't see a need for >> multiple simultaneous versions in this setup, and multiple versions do >> lead to confusion. Version metadata is still nice, of course. > > Increasingly, projects I work on involve multiple Python applications > with each application requiring different sets of packages and, sometimes, > different package versions. Basically, I want a single version for each > application, but multiple versions in the environment. What appeals to me > is a compromise between the single-version and multi-version install. It's probably not important to get rid of an existing feature of setuptools. The problems I have are better served with better tool support. Mostly I get a huge number of old and expired eggs sitting around, and there needs to be a collection process, probably a process that is run on every installation. I guess the collection would start from all the packages listed in .pth files, and maybe a collection of scripts, and then anything those packages require. And anything left over is garbage. In part I'm personally moving to using setup.py develop installs for more software, and that's more naturally single-version (though I suppose it doesn't have to be). >> Installation as a whole is an open issue. Putting in env/setup.cfg >> with the setting specific to that working environment works to a >> degree -- easy_install will pick it up if invoked from there. But >> that doesn't work with setup.py develop, or setup.py install, or some >> other scenarios. > > I don't follow this. It seems to work for us, at least for > setup.py develop. The main lamosity is depending on the current working > directory. I don't want users to have to give particular options to manage the installation. I want them to activate a specific environment, and for everything to just work. working-env.py mostly works like this. > This brings me to the topic of configuration. Today, I write wrapper > scripts "by hand", I may have some application like Zope, or ZEO > or our test runner that is implemented by a an entry point in a module. > Then there's a wrapper script that imports the module and calls the > entry point. > The wrapper script is written (manually or with some custom installation > script) to include the path to be used and configuration data, > which may be the location of a configuration file. I really like > the fact that easy_install will generate wrapper scripts for me, but > I really need more control over how these scripts are generated to > include *both* path and configuration information. I'm not sure what to think of this. I don't think of it as a script. It's like a specific invocation of the script. A shell script. Maybe we can improve on shell scripts, but I think it's a different idea than the script alone. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From pje at telecommunity.com Sat Mar 11 23:37:02 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 11 Mar 2006 17:37:02 -0500 Subject: [Distutils] Jython support? In-Reply-To: <441319A3.1030202@murkworks.com> Message-ID: <5.1.1.6.0.20060311173604.01fcad50@mail.telecommunity.com> At 01:40 PM 3/11/2006 -0500, Brad Clements wrote: >I would like to use setuptools, eggs with entry points, and so forth >with Jython. > >Is this possible? Not as far as I know. >If not, what's needed? A release of Jython that's equivalent to Python 2.3 or better, including PEP 302 support and the distutils. From pje at telecommunity.com Sat Mar 11 23:40:10 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 11 Mar 2006 17:40:10 -0500 Subject: [Distutils] working-env.py In-Reply-To: <44133F94.8080505@colorstudy.com> Message-ID: <5.1.1.6.0.20060311173739.01fb5ab8@mail.telecommunity.com> At 03:22 PM 3/11/2006 -0600, Ian Bicking wrote: > #!/usr/bin/python -S > import sys, os > join, dirname = os.path.join, os.path.dirname > lib_dir = join(dirname(dirname(__file__)), 'lib', 'python%s.%s' % >tuple(sys.version_info[:2])) > sys.path.insert(0, lib_dir) > import site > ... normal stuff ... FYI, I plan to write a proposal next week for script path freezing, along the lines of what I talked with you and Jim Fulton at PyCon about. The basic idea is that there'll be a '.pylibs' file alongside the script (e.g. 'foo.pylibs' alongside a 'foo' script) that lists what should be at the front of sys.path. There are still some details to be worked out, but the basic idea is that this would happen in such a way as to ensure that precisely the libraries needed by the script would be first on sys.path, so that it would "inherit" default versions from the environment for plugins or dynamic dependencies only. From ianb at colorstudy.com Sat Mar 11 23:49:56 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Sat, 11 Mar 2006 16:49:56 -0600 Subject: [Distutils] working-env.py In-Reply-To: <5.1.1.6.0.20060311173739.01fb5ab8@mail.telecommunity.com> References: <5.1.1.6.0.20060311173739.01fb5ab8@mail.telecommunity.com> Message-ID: <44135414.9020704@colorstudy.com> Phillip J. Eby wrote: > At 03:22 PM 3/11/2006 -0600, Ian Bicking wrote: >> #!/usr/bin/python -S >> import sys, os >> join, dirname = os.path.join, os.path.dirname >> lib_dir = join(dirname(dirname(__file__)), 'lib', 'python%s.%s' % >> tuple(sys.version_info[:2])) >> sys.path.insert(0, lib_dir) >> import site >> ... normal stuff ... > > FYI, I plan to write a proposal next week for script path freezing, > along the lines of what I talked with you and Jim Fulton at PyCon > about. The basic idea is that there'll be a '.pylibs' file alongside > the script (e.g. 'foo.pylibs' alongside a 'foo' script) that lists what > should be at the front of sys.path. Will this happen before the import of site.py? I would very much like it to. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From jim at zope.com Sat Mar 11 23:53:15 2006 From: jim at zope.com (Jim Fulton) Date: Sat, 11 Mar 2006 17:53:15 -0500 Subject: [Distutils] Working environment In-Reply-To: <44134386.3080205@colorstudy.com> References: <440DCA93.2030207@colorstudy.com> <4412F0FB.9090907@zope.com> <44134386.3080205@colorstudy.com> Message-ID: <441354DB.8050405@zope.com> Ian Bicking wrote: > Jim Fulton wrote: > ... >>> lib/python2.4/ is for packages. >> >> >> Minor note: this needs to be flexible. I'd be more inclined to go >> with something shallower and simpler, like just "lib", > > > Why? Top-level packages aren't portable, since .pyc files aren't > portable. Eggs are portable, since they contain the Python version. I have no idea what you are saying or how it relates to whether or not packages go in lib/python2.4 or lib. ... >> This brings me to the topic of configuration. Today, I write wrapper >> scripts "by hand", I may have some application like Zope, or ZEO >> or our test runner that is implemented by a an entry point in a module. >> Then there's a wrapper script that imports the module and calls the >> entry point. >> The wrapper script is written (manually or with some custom installation >> script) to include the path to be used and configuration data, >> which may be the location of a configuration file. I really like >> the fact that easy_install will generate wrapper scripts for me, but >> I really need more control over how these scripts are generated to >> include *both* path and configuration information. > > > I'm not sure what to think of this. I don't think of it as a script. > It's like a specific invocation of the script. A shell script. Maybe > we can improve on shell scripts, but I think it's a different idea than > the script alone. What "it" are you talking about? Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From ianb at colorstudy.com Sun Mar 12 00:10:45 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Sat, 11 Mar 2006 17:10:45 -0600 Subject: [Distutils] Working environment In-Reply-To: <441354DB.8050405@zope.com> References: <440DCA93.2030207@colorstudy.com> <4412F0FB.9090907@zope.com> <44134386.3080205@colorstudy.com> <441354DB.8050405@zope.com> Message-ID: <441358F5.1000208@colorstudy.com> Jim Fulton wrote: > Ian Bicking wrote: >> Jim Fulton wrote: >> > ... >>>> lib/python2.4/ is for packages. >>> >>> >>> Minor note: this needs to be flexible. I'd be more inclined to go >>> with something shallower and simpler, like just "lib", >> >> >> Why? Top-level packages aren't portable, since .pyc files aren't >> portable. Eggs are portable, since they contain the Python version. > > I have no idea what you are saying or how it relates to whether or not > packages go in lib/python2.4 or lib. lib/foo/__init__.pyc is a file that is specific to a version of Python. lib/python2.4/foo/__init__.pyc removes any possibility of conflict. Though I suppose it is arguable that a working environment should only support one major version of Python. >>> This brings me to the topic of configuration. Today, I write wrapper >>> scripts "by hand", I may have some application like Zope, or ZEO >>> or our test runner that is implemented by a an entry point in a module. >>> Then there's a wrapper script that imports the module and calls the >>> entry point. >>> The wrapper script is written (manually or with some custom installation >>> script) to include the path to be used and configuration data, >>> which may be the location of a configuration file. I really like >>> the fact that easy_install will generate wrapper scripts for me, but >>> I really need more control over how these scripts are generated to >>> include *both* path and configuration information. >> >> >> I'm not sure what to think of this. I don't think of it as a script. >> It's like a specific invocation of the script. A shell script. Maybe >> we can improve on shell scripts, but I think it's a different idea >> than the script alone. > > What "it" are you talking about? This script+config invocation. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From jim at zope.com Sun Mar 12 00:17:10 2006 From: jim at zope.com (Jim Fulton) Date: Sat, 11 Mar 2006 18:17:10 -0500 Subject: [Distutils] Working environment In-Reply-To: <441358F5.1000208@colorstudy.com> References: <440DCA93.2030207@colorstudy.com> <4412F0FB.9090907@zope.com> <44134386.3080205@colorstudy.com> <441354DB.8050405@zope.com> <441358F5.1000208@colorstudy.com> Message-ID: <44135A76.8030605@zope.com> Ian Bicking wrote: > Jim Fulton wrote: > >> Ian Bicking wrote: >> >>> Jim Fulton wrote: >>> >> ... >> >>>>> lib/python2.4/ is for packages. >>>> >>>> >>>> >>>> Minor note: this needs to be flexible. I'd be more inclined to go >>>> with something shallower and simpler, like just "lib", >>> >>> >>> >>> Why? Top-level packages aren't portable, since .pyc files aren't >>> portable. Eggs are portable, since they contain the Python version. >> >> >> I have no idea what you are saying or how it relates to whether or not >> packages go in lib/python2.4 or lib. > > > lib/foo/__init__.pyc is a file that is specific to a version of Python. > lib/python2.4/foo/__init__.pyc removes any possibility of conflict. > Though I suppose it is arguable that a working environment should only > support one major version of Python. Yup, at least most of the time. >>>> This brings me to the topic of configuration. Today, I write wrapper >>>> scripts "by hand", I may have some application like Zope, or ZEO >>>> or our test runner that is implemented by a an entry point in a module. >>>> Then there's a wrapper script that imports the module and calls the >>>> entry point. >>>> The wrapper script is written (manually or with some custom >>>> installation >>>> script) to include the path to be used and configuration data, >>>> which may be the location of a configuration file. I really like >>>> the fact that easy_install will generate wrapper scripts for me, but >>>> I really need more control over how these scripts are generated to >>>> include *both* path and configuration information. >>> >>> >>> >>> I'm not sure what to think of this. I don't think of it as a script. >>> It's like a specific invocation of the script. A shell script. Maybe >>> we can improve on shell scripts, but I think it's a different idea >>> than the script alone. >> >> >> What "it" are you talking about? > > > This script+config invocation. OK. Do you realize that for us, these are already combined? That is, if you install Zope and make an instance, the scripts that are installed have paths and configuration baked into them. For us, these are just installed scripts. There's nothing particularly exciting going on here. I want the same thing with the scripts generated by easy_install (or something like it that we build, if necessary). Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From ianb at colorstudy.com Sun Mar 12 01:27:43 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Sat, 11 Mar 2006 18:27:43 -0600 Subject: [Distutils] Working environment In-Reply-To: <44135A76.8030605@zope.com> References: <440DCA93.2030207@colorstudy.com> <4412F0FB.9090907@zope.com> <44134386.3080205@colorstudy.com> <441354DB.8050405@zope.com> <441358F5.1000208@colorstudy.com> <44135A76.8030605@zope.com> Message-ID: <44136AFF.3060702@colorstudy.com> Jim Fulton wrote: >>>> I'm not sure what to think of this. I don't think of it as a >>>> script. It's like a specific invocation of the script. A shell >>>> script. Maybe >>>> we can improve on shell scripts, but I think it's a different idea >>>> than the script alone. >>> >>> >>> What "it" are you talking about? >> >> >> This script+config invocation. > > OK. Do you realize that for us, these are already combined? > That is, if you install Zope and make an instance, the scripts that > are installed have paths and configuration baked into them. > For us, these are just installed scripts. There's nothing particularly > exciting going on here. I want the same thing with the scripts generated > by easy_install (or something like it that we build, if necessary). Well, I can certainly see that it would be useful to give the script access to the location of the working environment, and from there it can load default configuration files. That the script can detect intelligent default locations for files -- entirely reasonable -- isn't the same as coding the configuration file in the script. That might be okay too, but config+script isn't the same as a script, it's something else. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From jim at zope.com Sun Mar 12 15:56:58 2006 From: jim at zope.com (Jim Fulton) Date: Sun, 12 Mar 2006 09:56:58 -0500 Subject: [Distutils] Working environment In-Reply-To: <44136AFF.3060702@colorstudy.com> References: <440DCA93.2030207@colorstudy.com> <4412F0FB.9090907@zope.com> <44134386.3080205@colorstudy.com> <441354DB.8050405@zope.com> <441358F5.1000208@colorstudy.com> <44135A76.8030605@zope.com> <44136AFF.3060702@colorstudy.com> Message-ID: <441436BA.5060903@zope.com> Ian Bicking wrote: > Jim Fulton wrote: > >>>>> I'm not sure what to think of this. I don't think of it as a >>>>> script. It's like a specific invocation of the script. A shell >>>>> script. Maybe >>>>> we can improve on shell scripts, but I think it's a different idea >>>>> than the script alone. >>>> >>>> >>>> >>>> What "it" are you talking about? >>> >>> >>> >>> This script+config invocation. >> >> >> OK. Do you realize that for us, these are already combined? >> That is, if you install Zope and make an instance, the scripts that >> are installed have paths and configuration baked into them. >> For us, these are just installed scripts. There's nothing particularly >> exciting going on here. I want the same thing with the scripts generated >> by easy_install (or something like it that we build, if necessary). > > > Well, I can certainly see that it would be useful to give the script > access to the location of the working environment, and from there it can > load default configuration files. That could be a start. > That the script can detect > intelligent default locations for files -- entirely reasonable -- But probably too implicit, especially as a general approach. > isn't > the same as coding the configuration file in the script. That might be > okay too, but config+script isn't the same as a script, it's something > else. OK, we disagree. People encode this sort of information in scripts now, Saying it is something else doesn't make it so. If no one else is interested in this, we'll try to figure something out and share what we've learned. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Sun Mar 12 19:22:32 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 12 Mar 2006 13:22:32 -0500 Subject: [Distutils] Working environment In-Reply-To: <441436BA.5060903@zope.com> References: <44136AFF.3060702@colorstudy.com> <440DCA93.2030207@colorstudy.com> <4412F0FB.9090907@zope.com> <44134386.3080205@colorstudy.com> <441354DB.8050405@zope.com> <441358F5.1000208@colorstudy.com> <44135A76.8030605@zope.com> <44136AFF.3060702@colorstudy.com> Message-ID: <5.1.1.6.0.20060312131509.01fd17f8@mail.telecommunity.com> At 09:56 AM 3/12/2006 -0500, Jim Fulton wrote: >OK, we disagree. People encode this sort of information in scripts >now, Saying it is something else doesn't make it so. If no one else >is interested in this, we'll try to figure something out and share what >we've learned. FWIW, I'm probably going to generalize script writing to be entry-point based. Right now, the various entry point groups for scripts are hard-coded, but I want to make it extensible. That way, you'll be able to effectively have custom install-time hooks, within certain restrictions that I haven't yet figured out. (Apart from obvious restrictions like not being interactive, not writing stuff to arbitrary locations, etc.) The complex thing about doing this is that when an egg's scripts are written, its dependencies may not yet be installed. (Especially in the system package manager case.) So, I may have to refactor the whole installation process to queue scripts for installation only at the end of installing all dependencies. Not-so-coincidentally, a similar refactoring may be needed to implement path freezing for scripts. :) (Because until the dependencies are installed, you don't know what paths to freeze.) From ianb at colorstudy.com Sun Mar 12 20:04:56 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 12 Mar 2006 13:04:56 -0600 Subject: [Distutils] Working environment In-Reply-To: <441436BA.5060903@zope.com> References: <440DCA93.2030207@colorstudy.com> <4412F0FB.9090907@zope.com> <44134386.3080205@colorstudy.com> <441354DB.8050405@zope.com> <441358F5.1000208@colorstudy.com> <44135A76.8030605@zope.com> <44136AFF.3060702@colorstudy.com> <441436BA.5060903@zope.com> Message-ID: <441470D8.3050404@colorstudy.com> Jim Fulton wrote: > > isn't >> the same as coding the configuration file in the script. That might >> be okay too, but config+script isn't the same as a script, it's >> something else. > > OK, we disagree. People encode this sort of information in scripts > now, Saying it is something else doesn't make it so. If no one else > is interested in this, we'll try to figure something out and share what > we've learned. Maybe I should clarify what I mean. The script that setuptools generates, as enumerated in the setup.py file, isn't something that would naturally be bound to any configuration file. There's not a one-to-one mapping between those scripts, an installation, and a single configuration file. That *a* script might be bound to a configuration file, sure. Such a binding might also include any number of environment setups; the typical items in a shell script. Unless you are thinking about a different kind of configuration than I am. I suppose configuration like an enumeration of activated plugins would seem to fit more closely with a working environment. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From vivainio at gmail.com Sun Mar 12 22:11:54 2006 From: vivainio at gmail.com (Ville Vainio) Date: Sun, 12 Mar 2006 23:11:54 +0200 Subject: [Distutils] easy_install ./mymodule.py Message-ID: <46cb515a0603121311n3ccd368ag39992e21fdc2ea81@mail.gmail.com> I can't remember whether this has been discussed previously, but I'd like to see the ability to quickly install simple python modules globally, with the usage of easy_install ./mymodule.py or even easy_install http://somesite.com/mymodule.py Additionally, it would be groovy to be able to specify the entry points in .py files, a'la # setuptools_entrypoint = myscript:main def main(): """ will be called by 'myscript' script.""" -- Ville Vainio - vivainio.googlepages.com vainio.blogspot.com - g[mail | talk]='vivainio' From pje at telecommunity.com Sun Mar 12 23:00:26 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 12 Mar 2006 17:00:26 -0500 Subject: [Distutils] easy_install ./mymodule.py In-Reply-To: <46cb515a0603121311n3ccd368ag39992e21fdc2ea81@mail.gmail.co m> Message-ID: <5.1.1.6.0.20060312165520.03fbf698@mail.telecommunity.com> At 11:11 PM 3/12/2006 +0200, Ville Vainio wrote: >I can't remember whether this has been discussed previously, but I'd >like to see the ability to quickly install simple python modules >globally, with the usage of > >easy_install ./mymodule.py > >or even > >easy_install http://somesite.com/mymodule.py If you do this: easy_install "http://somesite.com/mymodule.py#egg=MyProject-0.1" EasyInstall will treat the .py file as being an egg named MyProject, with a version number of 0.1. This should also work with file: URLs, and can also be used as links in a download page or --find-links option. EasyInstall automatically generates a setup.py to go with the .py file, and runs it to create the egg which it then installs. >Additionally, it would be groovy to be able to specify the entry >points in .py files, a'la > ># setuptools_entrypoint = myscript:main > >def main(): > """ will be called by 'myscript' script.""" That won't work, but it's possible I could have it look for an assignment like this: __setuptools_entry_points__ = """ [console_scripts] foo = mymodule:main """ On the other hand, this seems like a path to diminishing returns. You might as well start using a setup.py if you need entry points. From vivainio at gmail.com Mon Mar 13 08:55:40 2006 From: vivainio at gmail.com (Ville Vainio) Date: Mon, 13 Mar 2006 09:55:40 +0200 Subject: [Distutils] easy_install ./mymodule.py In-Reply-To: <5.1.1.6.0.20060312165520.03fbf698@mail.telecommunity.com> References: <5.1.1.6.0.20060312165520.03fbf698@mail.telecommunity.com> Message-ID: <46cb515a0603122355v1243385el8569a57f3d566528@mail.gmail.com> On 3/13/06, Phillip J. Eby wrote: > If you do this: > > easy_install "http://somesite.com/mymodule.py#egg=MyProject-0.1" > > EasyInstall will treat the .py file as being an egg named MyProject, with a > version number of 0.1. Ok, thanks for the pointer. However, would it perhaps make sense to have the option of automatically generating the egg name from the module name (and using the name 'Mymodule", for example). > That won't work, but it's possible I could have it look for an assignment > like this: > > __setuptools_entry_points__ = """ > [console_scripts] > foo = mymodule:main > """ > > On the other hand, this seems like a path to diminishing returns. You > might as well start using a setup.py if you need entry points. I'm thinking of the extreme "scaling down" scenario here, shipping only a single .py file (a "script"). When you have a setup.py script you need to bundle it with the module. When in the past you had a directory full of .py scripts that you added to PATH, now you could just go there and do easy_install *.py. -- Ville Vainio - vivainio.googlepages.com vainio.blogspot.com - g[mail | talk]='vivainio' From fuzzyman at voidspace.org.uk Mon Mar 13 10:21:44 2006 From: fuzzyman at voidspace.org.uk (Fuzzyman) Date: Mon, 13 Mar 2006 09:21:44 +0000 Subject: [Distutils] easy_install ./mymodule.py In-Reply-To: <46cb515a0603122355v1243385el8569a57f3d566528@mail.gmail.com> References: <5.1.1.6.0.20060312165520.03fbf698@mail.telecommunity.com> <46cb515a0603122355v1243385el8569a57f3d566528@mail.gmail.com> Message-ID: <441539A8.6040606@voidspace.org.uk> Ville Vainio wrote: > On 3/13/06, Phillip J. Eby wrote: > > >> If you do this: >> >> easy_install "http://somesite.com/mymodule.py#egg=MyProject-0.1" >> >> EasyInstall will treat the .py file as being an egg named MyProject, with a >> version number of 0.1. >> > > Ok, thanks for the pointer. However, would it perhaps make sense to > have the option of automatically generating the egg name from the > module name (and using the name 'Mymodule", for example). > > It looks like setuptools determines the name of the file from the URL. I use a Python CGI to serve my files for download. This means that anyone attempting : easy_install "http://www.voidspace.org.uk/cgi-bin/voidspace/downman.py?file=configobj.py" Gets an egg containing 'downman.py'. The CGI does send the correct http header containing the filename : Content-Type: application/octet-stream Content-Disposition: attachment; filename= "%s" Content-Length: %s Shouldn't setuptools honour the "Content-Disposition" header ? All the best, Michael Foord http://www.voidspace.org.uk/python/index.shtml -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20060313/709b4d69/attachment.htm From pje at telecommunity.com Mon Mar 13 16:17:57 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 13 Mar 2006 10:17:57 -0500 Subject: [Distutils] easy_install ./mymodule.py In-Reply-To: <441539A8.6040606@voidspace.org.uk> References: <46cb515a0603122355v1243385el8569a57f3d566528@mail.gmail.com> <5.1.1.6.0.20060312165520.03fbf698@mail.telecommunity.com> <46cb515a0603122355v1243385el8569a57f3d566528@mail.gmail.com> Message-ID: <5.1.1.6.0.20060313100603.022495f8@mail.telecommunity.com> At 09:21 AM 3/13/2006 +0000, Fuzzyman wrote: >Ville Vainio wrote: >>On 3/13/06, Phillip J. Eby >> wrote: >>>If you do this: >>> >>> easy_install "http://somesite.com/mymodule.py#egg=MyProject-0.1" >>> >>>EasyInstall will treat the .py file as being an egg named MyProject, with a >>>version number of 0.1. >> >>Ok, thanks for the pointer. However, would it perhaps make sense to >>have the option of automatically generating the egg name from the >>module name (and using the name 'Mymodule", for example). >> >It looks like setuptools determines the name of the file from the URL. > >I use a Python CGI to serve my files for download. This means that anyone >attempting >: > > easy_install > "http://www.voidspace.org.uk/cgi-bin/voidspace/downman.py?file=configobj.py" > >Gets an egg containing 'downman.py'. The CGI does send the correct http >header containing the filename : > > Content-Type: application/octet-stream > Content-Disposition: attachment; filename= "%s" > Content-Length: %s > >Shouldn't setuptools honour the "Content-Disposition" header ? EasyInstall has to determine the project name and version from the URL, because otherwise it has no way to know *which* URL to download (in the case where it's scanning pages for links. If you need it to deal with URLs like the one you gave above, you'll need to use the "#egg=Projectname-Version" suffix on the URL. Now, if you mean that when you *do* use that approach, you end up with the wrong filename for the .py, I suppose you have a point. OTOH, why not just change your downman.py to use PATH_INFO? e.g.: http://www.voidspace.org.uk/cgi-bin/voidspace/downman.py/configobj.py I don't process Content-Disposition because currently the choice of filename has to be fixed before the download begins. Changing it would only be useful for the "single .py file" case, because in all other situations the downloaded filename doesn't matter; only the info that was gotten from the URL. But it seems odd to support weird download options for single .py files; why not just put the file directly on the server and be done with it? From pje at telecommunity.com Mon Mar 13 16:19:11 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 13 Mar 2006 10:19:11 -0500 Subject: [Distutils] easy_install ./mymodule.py In-Reply-To: <46cb515a0603122355v1243385el8569a57f3d566528@mail.gmail.co m> References: <5.1.1.6.0.20060312165520.03fbf698@mail.telecommunity.com> <5.1.1.6.0.20060312165520.03fbf698@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060313101817.0445ee70@mail.telecommunity.com> At 09:55 AM 3/13/2006 +0200, Ville Vainio wrote: >I'm thinking of the extreme "scaling down" scenario here, shipping >only a single .py file (a "script"). When you have a setup.py script >you need to bundle it with the module. > >When in the past you had a directory full of .py scripts that you >added to PATH, now you could just go there and do easy_install *.py. I'm not following why that would be any better. Why not just add a setup.py to that directory? From vivainio at gmail.com Mon Mar 13 17:34:19 2006 From: vivainio at gmail.com (Ville Vainio) Date: Mon, 13 Mar 2006 18:34:19 +0200 Subject: [Distutils] easy_install ./mymodule.py In-Reply-To: <5.1.1.6.0.20060313101817.0445ee70@mail.telecommunity.com> References: <5.1.1.6.0.20060312165520.03fbf698@mail.telecommunity.com> <5.1.1.6.0.20060313101817.0445ee70@mail.telecommunity.com> Message-ID: <46cb515a0603130834q12417f0oe6c6299496dfc2ea@mail.gmail.com> On 3/13/06, Phillip J. Eby wrote: > I'm not following why that would be any better. Why not just add a > setup.py to that directory? It's a slight drag to create, esp. if you need to keep it up-to-date when you make changes to directory layout, add/delete scripts etc. and the scripts are not necessarily related to each other (each script is standalone). Basically what I'm saying is that I'm lazy. Perhaps if there was find_modules and find_entrypoints (like there is find_packages now) I wouldn't care, but now creating and maintaining a setup.py is an extra complication. I guess creating my own setuppy-wizard would solve this issue in a cleaner fashion. -- Ville Vainio - vivainio.googlepages.com vainio.blogspot.com - g[mail | talk]='vivainio' From ianb at colorstudy.com Mon Mar 13 19:03:57 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 13 Mar 2006 12:03:57 -0600 Subject: [Distutils] Further extending of setup.py Message-ID: <4415B40D.4010601@colorstudy.com> I'd like to get rid of the "paster" script, moving some of what it does into setup.py or elsewhere. To a degree that is possible, but I think requires some additions to setuptools: * An entry point group that is not globally looked up, like distutils.commands. This would be a set of commands the package provides only for itself. I expect it to look like: [distutils.commands.local] sql_record = sqlobject.manage.distextension:sql_record Exactly like what we have currently, just not looked up globally. * Additionally, or probably even better, something that enumerates locations for commands. E.g.: extra_commands=['SQLObject'] And then it would look in the SQLObject egg for distutils.commands.local, and apply everything found there to this package. Right now, for instance, buildutils adds a "pytest" command to every project, even though it only applies to some projects (for some commands this is ok, like "info", so two different entry points makes sense). A project could list itself to provide its own custom commands; I think that won't be too circular. Typically commands you provide for yourself or someone else are different -- e.g., the SQLObject commands don't apply to SQLObject itself. * Everything that can be done on a deployed egg will probably go in app/egg-specific command-line scripts, and maybe I'll make a little framework to make these easier to write, but that's entirely an implementation detail. But I'm also thinking that extra_commands could be used as a hint to that framework, and would kind of facilitate coordination of in-development commands with after-deployment commands. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From tseaver at palladion.com Tue Mar 14 16:23:31 2006 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 14 Mar 2006 10:23:31 -0500 Subject: [Distutils] setup.py test doesn't honor --find-links Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Can we extend the test command (deriving from easy_install, I guess, instead of Command), to take the same options it does? I tried hacking at it a bit, but without much success so far: I've gotten it to accept the '--find-links' option, but not actually to *use* it: $ cd ~/projects/eggz/src/zope.tales $ ../../bin/python setup.py test \ --find-links="http://download.zope.org/distribution" running test running egg_info writing requirements to src/zope.tales.egg-info/requires.txt writing src/zope.tales.egg-info/PKG-INFO writing top-level names to src/zope.tales.egg-info/top_level.txt writing namespace_packages to src/zope.tales.egg-info/namespace_packages.txt warning: manifest_maker: standard file not found: should have one of README, README.txt writing manifest file 'src/zope.tales.egg-info/SOURCES.txt' running build_ext Checking .pth file support in . /home/tseaver/projects/PyCon2006/eggification/sandbox/src/zope.tales/../../bin/python - -E -c pass Searching for zope.testing Reading http://www.python.org/pypi/zope.testing/ Couldn't find index page for 'zope.testing' (maybe misspelled?) Scanning index of all packages (this may take a while) Reading http://www.python.org/pypi/ No local packages or download links found for zope.testing error: Could not find suitable distribution for Requirement.parse('zope.testing') I'm attaching the patch against the current SVN head, Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFEFt/z+gerLs4ltQ4RAvdFAKCRSHP44tnDHIKdnHP7dEtZxvyqBgCdFWBE dULlC535P+qeO8m5MZLogig= =lM5x -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: setuptools_test_find_links.patch Type: text/x-patch Size: 1988 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060314/534e45c1/attachment.bin From FETCHMAIL-DAEMON at ns3565.ovh.net Wed Mar 15 15:03:03 2006 From: FETCHMAIL-DAEMON at ns3565.ovh.net (FETCHMAIL-DAEMON at ns3565.ovh.net) Date: Wed, 15 Mar 2006 15:03:03 +0100 (CET) Subject: [Distutils] (no subject) Message-ID: <20060315140303.E60C41E4009@bag.python.org> This message was too large (SMTP error 552). -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/rfc822-headers Size: 527 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060315/52a6789d/attachment.bin From Juha.Tuomala at iki.fi Wed Mar 15 14:55:36 2006 From: Juha.Tuomala at iki.fi (Juha Tuomala) Date: Wed, 15 Mar 2006 15:55:36 +0200 Subject: [Distutils] Finding dependencies and Limiting download Message-ID: <200603151555.43460.Juha.Tuomala@iki.fi> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi I'm new with setuptools and probably asking wrong questions. :-) I'm making RPM package from python-paste for CentOS Linux installation and came across some problems. I do the installation with: setup.py install --root $RPM_BUILD_ROOT and it seems to work pretty well, but it doesn't find my already installed cheetah package below site-pacakges and thus starts searching it from the net. Does the dependencies have to be installed as an egg so that setuptools could find them? (I doubt that) Second thing is that how do I limit the download using 'install' ? With 'easy_install' I could use --allow-hosts=None or -H, but is it so that it's not available for 'install'? Any help would be appreciated. BR, Tuju - -- Ajatteleva ihminen tarvitsee unta. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.1 (GNU/Linux) iD8DBQFEGBzfQnPP29fw0xMRAinEAJ9acsH9V/upVnx+Z2vTfVwaxHC//gCfdGsp mn39L3REbs9DKXFOIYzILbI= =nn4X -----END PGP SIGNATURE----- From Juha.Tuomala at iki.fi Wed Mar 15 14:55:36 2006 From: Juha.Tuomala at iki.fi (Juha Tuomala) Date: Wed, 15 Mar 2006 15:55:36 +0200 Subject: [Distutils] Finding dependencies and Limiting download Message-ID: <200603151555.43460.Juha.Tuomala@iki.fi> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi I'm new with setuptools and probably asking wrong questions. :-) I'm making RPM package from python-paste for CentOS Linux installation and came across some problems. I do the installation with: setup.py install --root $RPM_BUILD_ROOT and it seems to work pretty well, but it doesn't find my already installed cheetah package below site-pacakges and thus starts searching it from the net. Does the dependencies have to be installed as an egg so that setuptools could find them? (I doubt that) Second thing is that how do I limit the download using 'install' ? With 'easy_install' I could use --allow-hosts=None or -H, but is it so that it's not available for 'install'? Any help would be appreciated. BR, Tuju - -- Ajatteleva ihminen tarvitsee unta. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.1 (GNU/Linux) iD8DBQFEGBzfQnPP29fw0xMRAinEAJ9acsH9V/upVnx+Z2vTfVwaxHC//gCfdGsp mn39L3REbs9DKXFOIYzILbI= =nn4X -----END PGP SIGNATURE----- From Juha.Tuomala at iki.fi Wed Mar 15 14:55:36 2006 From: Juha.Tuomala at iki.fi (Juha Tuomala) Date: Wed, 15 Mar 2006 15:55:36 +0200 Subject: [Distutils] Finding dependencies and Limiting download Message-ID: <200603151555.43460.Juha.Tuomala@iki.fi> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi I'm new with setuptools and probably asking wrong questions. :-) I'm making RPM package from python-paste for CentOS Linux installation and came across some problems. I do the installation with: setup.py install --root $RPM_BUILD_ROOT and it seems to work pretty well, but it doesn't find my already installed cheetah package below site-pacakges and thus starts searching it from the net. Does the dependencies have to be installed as an egg so that setuptools could find them? (I doubt that) Second thing is that how do I limit the download using 'install' ? With 'easy_install' I could use --allow-hosts=None or -H, but is it so that it's not available for 'install'? Any help would be appreciated. BR, Tuju - -- Ajatteleva ihminen tarvitsee unta. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.1 (GNU/Linux) iD8DBQFEGBzfQnPP29fw0xMRAinEAJ9acsH9V/upVnx+Z2vTfVwaxHC//gCfdGsp mn39L3REbs9DKXFOIYzILbI= =nn4X -----END PGP SIGNATURE----- From pje at telecommunity.com Wed Mar 15 17:30:20 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 15 Mar 2006 11:30:20 -0500 Subject: [Distutils] Finding dependencies and Limiting download In-Reply-To: <200603151555.43460.Juha.Tuomala@iki.fi> Message-ID: <5.1.1.6.0.20060315112402.01faca78@mail.telecommunity.com> At 03:55 PM 3/15/2006 +0200, Juha Tuomala wrote: >-----BEGIN PGP SIGNED MESSAGE----- >Hash: SHA1 > > > >Hi > >I'm new with setuptools and probably asking wrong questions. :-) > >I'm making RPM package from python-paste for CentOS >Linux installation and came across some problems. > >I do the installation with: > > setup.py install --root $RPM_BUILD_ROOT Use "setup.py bdist_rpm" to build an RPM, and all of the details (except RPM-level dependencies) will be handled automatically. >and it seems to work pretty well, but it doesn't find >my already installed cheetah package below site-pacakges >and thus starts searching it from the net. That's because you're using 'install' rather than 'bdist_rpm'. If for some reason you can't use bdist_rpm, you should use the --single-version-externally-managed option to the 'install' command, which tells easy_install that you're building a system package, so it won't try to do dependencies and it won't build an egg file or directory. Instead, it will install packages in the "traditional" way, with the addition of a .egg-info/ directory that contains the package's metadata so it will still work correctly at runtime. > Does the >dependencies have to be installed as an egg so >that setuptools could find them? Yes, if the package uses the pkg_resources module at runtime to access the dependencies. At minimum, the dependencies would then need .egg-info files or directories to indicate what is installed. If you use setuptools to build new RPMs for those dependencies, you'll get that info added to the RPMs automatically. >Second thing is that how do I limit the download using >'install' ? With 'easy_install' I could use --allow-hosts=None >or -H, but is it so that it's not available for 'install'? If you want to get the full set of easy_install features, just go to the setup directory and run "easy_install .", it will install the package that's in the current directory, and thus you can use all the normal easy_install options. Note also that if you use 'bdist_rpm', or 'install --single-version-externally-managed', no dependencies will be sought or installed at build time. (But this doesn't stop you from needing .egg-info metadata for the dependencies if the installed packages are queried at runtime.) From ianb at colorstudy.com Wed Mar 15 17:31:42 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 15 Mar 2006 10:31:42 -0600 Subject: [Distutils] Finding dependencies and Limiting download In-Reply-To: <5.1.1.6.0.20060315112402.01faca78@mail.telecommunity.com> References: <5.1.1.6.0.20060315112402.01faca78@mail.telecommunity.com> Message-ID: <4418416E.9000706@colorstudy.com> Phillip J. Eby wrote: > That's because you're using 'install' rather than 'bdist_rpm'. If for some > reason you can't use bdist_rpm, you should use the > --single-version-externally-managed option to the 'install' command, which > tells easy_install that you're building a system package, so it won't try > to do dependencies and it won't build an egg file or directory. Instead, > it will install packages in the "traditional" way, with the addition of a > .egg-info/ directory that contains the package's metadata so it will still > work correctly at runtime. One thing that came up is that --single-version-... doesn't seem to work with Paste's namespace packages. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Wed Mar 15 17:36:39 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 15 Mar 2006 11:36:39 -0500 Subject: [Distutils] setup.py test doesn't honor --find-links In-Reply-To: Message-ID: <5.1.1.6.0.20060315113145.043b29d8@mail.telecommunity.com> At 10:23 AM 3/14/2006 -0500, Tres Seaver wrote: >-----BEGIN PGP SIGNED MESSAGE----- >Hash: SHA1 > >Can we extend the test command (deriving from easy_install, I guess, >instead of Command), to take the same options it does? I don't think that approach is a good idea, as it's more complex and isn't orthogonal to other setuptools features that install eggs for build purposes (e.g. setup_requires). It would probably be better to implement the "dependency_links" feature described here: http://mail.python.org/pipermail/distutils-sig/2005-October/005209.html and make it be used by the setuptools.dist.Distribution class's "fetch_build_egg()" method, which is used for both setup_requires and tests_require. I've left that feature open for a while in the hopes that it would provide some enterprising soul with an easy project to learn about extending setuptools, as it's mostly pretty shallow to implement. The only part such an enterprising party might need help with is the bit that touches easy_install; the rest should be a matter of just copying existing entry points defined in setuptools' setup.py. From pje at telecommunity.com Wed Mar 15 17:38:45 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 15 Mar 2006 11:38:45 -0500 Subject: [Distutils] Finding dependencies and Limiting download In-Reply-To: <4418416E.9000706@colorstudy.com> References: <5.1.1.6.0.20060315112402.01faca78@mail.telecommunity.com> <5.1.1.6.0.20060315112402.01faca78@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060315113838.043b9430@mail.telecommunity.com> At 10:31 AM 3/15/2006 -0600, Ian Bicking wrote: >Phillip J. Eby wrote: >>That's because you're using 'install' rather than 'bdist_rpm'. If for >>some reason you can't use bdist_rpm, you should use the >>--single-version-externally-managed option to the 'install' command, >>which tells easy_install that you're building a system package, so it >>won't try to do dependencies and it won't build an egg file or >>directory. Instead, it will install packages in the "traditional" way, >>with the addition of a .egg-info/ directory that contains the package's >>metadata so it will still work correctly at runtime. > >One thing that came up is that --single-version-... doesn't seem to work >with Paste's namespace packages. How so? From ianb at colorstudy.com Wed Mar 15 17:41:03 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 15 Mar 2006 10:41:03 -0600 Subject: [Distutils] Finding dependencies and Limiting download In-Reply-To: <5.1.1.6.0.20060315113838.043b9430@mail.telecommunity.com> References: <5.1.1.6.0.20060315112402.01faca78@mail.telecommunity.com> <5.1.1.6.0.20060315112402.01faca78@mail.telecommunity.com> <5.1.1.6.0.20060315113838.043b9430@mail.telecommunity.com> Message-ID: <4418439F.9020504@colorstudy.com> Phillip J. Eby wrote: > At 10:31 AM 3/15/2006 -0600, Ian Bicking wrote: > >> Phillip J. Eby wrote: >> >>> That's because you're using 'install' rather than 'bdist_rpm'. If >>> for some reason you can't use bdist_rpm, you should use the >>> --single-version-externally-managed option to the 'install' command, >>> which tells easy_install that you're building a system package, so it >>> won't try to do dependencies and it won't build an egg file or >>> directory. Instead, it will install packages in the "traditional" >>> way, with the addition of a .egg-info/ directory that contains the >>> package's metadata so it will still work correctly at runtime. >> >> >> One thing that came up is that --single-version-... doesn't seem to >> work with Paste's namespace packages. > > > How so? Well, I should say, it *works*, but as RPMs it doesn't work, because they all write to the same __init__.py. They all write the same content to __init__.py, but RPM doesn't like that. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From Juha.Tuomala at iki.fi Wed Mar 15 17:50:03 2006 From: Juha.Tuomala at iki.fi (Juha Tuomala) Date: Wed, 15 Mar 2006 18:50:03 +0200 Subject: [Distutils] Finding dependencies and Limiting download In-Reply-To: <4418439F.9020504@colorstudy.com> References: <5.1.1.6.0.20060315112402.01faca78@mail.telecommunity.com> <5.1.1.6.0.20060315113838.043b9430@mail.telecommunity.com> <4418439F.9020504@colorstudy.com> Message-ID: <200603151850.21813.Juha.Tuomala@iki.fi> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday 15 March 2006 18:41, Ian Bicking wrote: > Well, I should say, it *works*, but as RPMs it doesn't work, because > they all write to the same __init__.py. They all write the same content > to __init__.py, but RPM doesn't like that. That's true. During the RPM build process, the stuff is installed into empty 'chroot' where RPM looks for files and adds them into its own bookkeeping/metadata. All packages thus will have their own version those overlapping files (due empty chroot, nothing gets overwritten during packaging) and conflicts appear only until someone tries to install second package containing this overlapping file. BR, Tuju - -- Ajatteleva ihminen tarvitsee unta. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.1 (GNU/Linux) iD8DBQFEGEXNQnPP29fw0xMRAh6wAJ9TSPGfdnwTuXCFxkyf+0TgPZVLrACcCH0q CxnA3vtecK83IAERcWoT8Sk= =cyg+ -----END PGP SIGNATURE----- From pje at telecommunity.com Wed Mar 15 20:33:47 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 15 Mar 2006 14:33:47 -0500 Subject: [Distutils] Finding dependencies and Limiting download In-Reply-To: <4418439F.9020504@colorstudy.com> References: <5.1.1.6.0.20060315113838.043b9430@mail.telecommunity.com> <5.1.1.6.0.20060315112402.01faca78@mail.telecommunity.com> <5.1.1.6.0.20060315112402.01faca78@mail.telecommunity.com> <5.1.1.6.0.20060315113838.043b9430@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060315142727.02053150@mail.telecommunity.com> At 10:41 AM 3/15/2006 -0600, Ian Bicking wrote: >Phillip J. Eby wrote: >>At 10:31 AM 3/15/2006 -0600, Ian Bicking wrote: >> >>>Phillip J. Eby wrote: >>> >>>>That's because you're using 'install' rather than 'bdist_rpm'. If for >>>>some reason you can't use bdist_rpm, you should use the >>>>--single-version-externally-managed option to the 'install' command, >>>>which tells easy_install that you're building a system package, so it >>>>won't try to do dependencies and it won't build an egg file or >>>>directory. Instead, it will install packages in the "traditional" way, >>>>with the addition of a .egg-info/ directory that contains the package's >>>>metadata so it will still work correctly at runtime. >>> >>> >>>One thing that came up is that --single-version-... doesn't seem to work >>>with Paste's namespace packages. >> >>How so? > >Well, I should say, it *works*, but as RPMs it doesn't work, because they >all write to the same __init__.py. They all write the same content to >__init__.py, but RPM doesn't like that. Well, you can work around this, but it's a pain because it makes it impossible for you to use setup.py develop. What you do is delete the __init__.py from all the packages, except for a special extra project that includes only the __init__.py. Then you make all the packages depend on that one. (And each package with a missing __init__ needs to have the package name explicitly listed in setup(), since find_packages() won't see it's there otherwise.) At that point, setuptools will add automatically-created __init__.py files into any eggs it makes, but it will not include them in --single-version-externally-managed installs. The downside, however, is that it makes "setup.py develop" useless for the original source packages. IOW, right now there isn't a good solution for using namespace packages with system packagers that use --single-version-externally-managed, and which don't accept any files in common between different system packages. Ideas for solutions are welcome. :) From Juha.Tuomala at iki.fi Thu Mar 16 13:56:52 2006 From: Juha.Tuomala at iki.fi (Juha Tuomala) Date: Thu, 16 Mar 2006 14:56:52 +0200 Subject: [Distutils] Missing packages Message-ID: <200603161456.52971.Juha.Tuomala@iki.fi> % paster Traceback (most recent call last): File "/usr/bin/paster", line 5, in ? from pkg_resources import load_entry_point File "/usr/lib/python2.3/site-packages/pkg_resources.py", line 2356, in ? working_set.require(__requires__) File "/usr/lib/python2.3/site-packages/pkg_resources.py", line 585, in require needed = self.resolve(parse_requirements(requirements)) File "/usr/lib/python2.3/site-packages/pkg_resources.py", line 483, in resolve raise DistributionNotFound(req) # XXX put more info here pkg_resources.DistributionNotFound: Cheetah % It looks like paster cannot find cheetach, even it's installed. I've compiled and installed cheetach RPM package. What is the thing what setuptools looks and doesn't find? That egg info? % rpm -qil python-cheetah Name : python-cheetah Relocations: (not relocatable) Version : 1.0 Vendor: (none) Release : 1 Build Date: Thu 16 Mar 2006 01:52:16 PM EET Install Date: Thu 16 Mar 2006 01:52:33 PM EET Build Host: XXXX.XXXXX.fi Group : Development/Libraries Source RPM: python-cheetah-1.0-1.src.rpm Size : 1993082 License: MIT Signature : (none) Packager : Juha Tuomala URL : http://sourceforge.net/projects/cheetahtemplate Summary : Template engine and code-generator Description : Cheetah is a Python-powered template engine and code-generator. It is similar to the Jakarta project's Velocity. /usr/bin/cheetah /usr/bin/cheetah-compile /usr/lib64/python2.3/site-packages/Cheetah /usr/lib64/python2.3/site-packages/Cheetah/CacheRegion.py /usr/lib64/python2.3/site-packages/Cheetah/CacheRegion.pyc /usr/lib64/python2.3/site-packages/Cheetah/CheetahWrapper.py /usr/lib64/python2.3/site-packages/Cheetah/CheetahWrapper.pyc /usr/lib64/python2.3/site-packages/Cheetah/Compiler.py /usr/lib64/python2.3/site-packages/Cheetah/Compiler.pyc /usr/lib64/python2.3/site-packages/Cheetah/DummyTransaction.py /usr/lib64/python2.3/site-packages/Cheetah/DummyTransaction.pyc /usr/lib64/python2.3/site-packages/Cheetah/ErrorCatchers.py /usr/lib64/python2.3/site-packages/Cheetah/ErrorCatchers.pyc /usr/lib64/python2.3/site-packages/Cheetah/FileUtils.py /usr/lib64/python2.3/site-packages/Cheetah/FileUtils.pyc /usr/lib64/python2.3/site-packages/Cheetah/Filters.py /usr/lib64/python2.3/site-packages/Cheetah/Filters.pyc /usr/lib64/python2.3/site-packages/Cheetah/ImportHooks.py /usr/lib64/python2.3/site-packages/Cheetah/ImportHooks.pyc /usr/lib64/python2.3/site-packages/Cheetah/ImportManager.py /usr/lib64/python2.3/site-packages/Cheetah/ImportManager.pyc /usr/lib64/python2.3/site-packages/Cheetah/NameMapper.py /usr/lib64/python2.3/site-packages/Cheetah/NameMapper.pyc /usr/lib64/python2.3/site-packages/Cheetah/Parser.py /usr/lib64/python2.3/site-packages/Cheetah/Parser.pyc /usr/lib64/python2.3/site-packages/Cheetah/Servlet.py /usr/lib64/python2.3/site-packages/Cheetah/Servlet.pyc /usr/lib64/python2.3/site-packages/Cheetah/SettingsManager.py /usr/lib64/python2.3/site-packages/Cheetah/SettingsManager.pyc /usr/lib64/python2.3/site-packages/Cheetah/SourceReader.py /usr/lib64/python2.3/site-packages/Cheetah/SourceReader.pyc /usr/lib64/python2.3/site-packages/Cheetah/Template.py /usr/lib64/python2.3/site-packages/Cheetah/Template.pyc /usr/lib64/python2.3/site-packages/Cheetah/TemplateCmdLineIface.py /usr/lib64/python2.3/site-packages/Cheetah/TemplateCmdLineIface.pyc /usr/lib64/python2.3/site-packages/Cheetah/Templates /usr/lib64/python2.3/site-packages/Cheetah/Templates/SkeletonPage.py /usr/lib64/python2.3/site-packages/Cheetah/Templates/SkeletonPage.pyc /usr/lib64/python2.3/site-packages/Cheetah/Templates/SkeletonPage.tmpl /usr/lib64/python2.3/site-packages/Cheetah/Templates/_SkeletonPage.py /usr/lib64/python2.3/site-packages/Cheetah/Templates/_SkeletonPage.pyc /usr/lib64/python2.3/site-packages/Cheetah/Templates/__init__.py /usr/lib64/python2.3/site-packages/Cheetah/Templates/__init__.pyc /usr/lib64/python2.3/site-packages/Cheetah/Tests /usr/lib64/python2.3/site-packages/Cheetah/Tests/CheetahWrapper.py /usr/lib64/python2.3/site-packages/Cheetah/Tests/CheetahWrapper.pyc /usr/lib64/python2.3/site-packages/Cheetah/Tests/FileRefresh.py /usr/lib64/python2.3/site-packages/Cheetah/Tests/FileRefresh.pyc /usr/lib64/python2.3/site-packages/Cheetah/Tests/NameMapper.py /usr/lib64/python2.3/site-packages/Cheetah/Tests/NameMapper.pyc /usr/lib64/python2.3/site-packages/Cheetah/Tests/SyntaxAndOutput.py /usr/lib64/python2.3/site-packages/Cheetah/Tests/SyntaxAndOutput.pyc /usr/lib64/python2.3/site-packages/Cheetah/Tests/Template.py /usr/lib64/python2.3/site-packages/Cheetah/Tests/Template.pyc /usr/lib64/python2.3/site-packages/Cheetah/Tests/Test.py /usr/lib64/python2.3/site-packages/Cheetah/Tests/Test.pyc /usr/lib64/python2.3/site-packages/Cheetah/Tests/__init__.py /usr/lib64/python2.3/site-packages/Cheetah/Tests/__init__.pyc /usr/lib64/python2.3/site-packages/Cheetah/Tests/unittest_local_copy.py /usr/lib64/python2.3/site-packages/Cheetah/Tests/unittest_local_copy.pyc /usr/lib64/python2.3/site-packages/Cheetah/Tools /usr/lib64/python2.3/site-packages/Cheetah/Tools/CGITemplate.py /usr/lib64/python2.3/site-packages/Cheetah/Tools/CGITemplate.pyc /usr/lib64/python2.3/site-packages/Cheetah/Tools/MondoReport.py /usr/lib64/python2.3/site-packages/Cheetah/Tools/MondoReport.pyc /usr/lib64/python2.3/site-packages/Cheetah/Tools/MondoReportDoc.txt /usr/lib64/python2.3/site-packages/Cheetah/Tools/RecursiveNull.py /usr/lib64/python2.3/site-packages/Cheetah/Tools/RecursiveNull.pyc /usr/lib64/python2.3/site-packages/Cheetah/Tools/SiteHierarchy.py /usr/lib64/python2.3/site-packages/Cheetah/Tools/SiteHierarchy.pyc /usr/lib64/python2.3/site-packages/Cheetah/Tools/__init__.py /usr/lib64/python2.3/site-packages/Cheetah/Tools/__init__.pyc /usr/lib64/python2.3/site-packages/Cheetah/Utils /usr/lib64/python2.3/site-packages/Cheetah/Utils/Indenter.py /usr/lib64/python2.3/site-packages/Cheetah/Utils/Indenter.pyc /usr/lib64/python2.3/site-packages/Cheetah/Utils/Misc.py /usr/lib64/python2.3/site-packages/Cheetah/Utils/Misc.pyc /usr/lib64/python2.3/site-packages/Cheetah/Utils/VerifyType.py /usr/lib64/python2.3/site-packages/Cheetah/Utils/VerifyType.pyc /usr/lib64/python2.3/site-packages/Cheetah/Utils/WebInputMixin.py /usr/lib64/python2.3/site-packages/Cheetah/Utils/WebInputMixin.pyc /usr/lib64/python2.3/site-packages/Cheetah/Utils/__init__.py /usr/lib64/python2.3/site-packages/Cheetah/Utils/__init__.pyc /usr/lib64/python2.3/site-packages/Cheetah/Utils/optik /usr/lib64/python2.3/site-packages/Cheetah/Utils/optik/__init__.py /usr/lib64/python2.3/site-packages/Cheetah/Utils/optik/__init__.pyc /usr/lib64/python2.3/site-packages/Cheetah/Utils/optik/errors.py /usr/lib64/python2.3/site-packages/Cheetah/Utils/optik/errors.pyc /usr/lib64/python2.3/site-packages/Cheetah/Utils/optik/option.py /usr/lib64/python2.3/site-packages/Cheetah/Utils/optik/option.pyc /usr/lib64/python2.3/site-packages/Cheetah/Utils/optik/option_parser.py /usr/lib64/python2.3/site-packages/Cheetah/Utils/optik/option_parser.pyc /usr/lib64/python2.3/site-packages/Cheetah/Version.py /usr/lib64/python2.3/site-packages/Cheetah/Version.pyc /usr/lib64/python2.3/site-packages/Cheetah/__init__.py /usr/lib64/python2.3/site-packages/Cheetah/__init__.pyc /usr/lib64/python2.3/site-packages/Cheetah/_namemapper.so /usr/share/doc/python-cheetah-1.0 /usr/share/doc/python-cheetah-1.0/CHANGES /usr/share/doc/python-cheetah-1.0/LICENSE /usr/share/doc/python-cheetah-1.0/README /usr/share/doc/python-cheetah-1.0/TODO % -- Ajatteleva ihminen tarvitsee unta. From tseaver at palladion.com Thu Mar 16 16:52:36 2006 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 16 Mar 2006 10:52:36 -0500 Subject: [Distutils] setup.py test doesn't honor --find-links In-Reply-To: <5.1.1.6.0.20060315113145.043b29d8@mail.telecommunity.com> References: <5.1.1.6.0.20060315113145.043b29d8@mail.telecommunity.com> Message-ID: <441989C4.4000401@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Phillip J. Eby wrote: > At 10:23 AM 3/14/2006 -0500, Tres Seaver wrote: > >>-----BEGIN PGP SIGNED MESSAGE----- >>Hash: SHA1 >> >>Can we extend the test command (deriving from easy_install, I guess, >>instead of Command), to take the same options it does? > > > I don't think that approach is a good idea, as it's more complex and isn't > orthogonal to other setuptools features that install eggs for build > purposes (e.g. setup_requires). It would probably be better to implement > the "dependency_links" feature described here: > > http://mail.python.org/pipermail/distutils-sig/2005-October/005209.html > > and make it be used by the setuptools.dist.Distribution class's > "fetch_build_egg()" method, which is used for both setup_requires and > tests_require. > > I've left that feature open for a while in the hopes that it would provide > some enterprising soul with an easy project to learn about extending > setuptools, as it's mostly pretty shallow to implement. The only part such > an enterprising party might need help with is the bit that touches > easy_install; the rest should be a matter of just copying existing entry > points defined in setuptools' setup.py. I'm a bit lost as to the rationale for the separate metadata filee in that post: why wouldn't we just use the 'dependency_links' attribute of the distribution? At any rate, I'm attaching a partially-working patch. I could get the 'dependency_links.txt' file to be created, but couldn't figure out how to use the pkg_resources facilities for reading it while inside 'fetch_build_eggs'. I alos ran into the problem that, since the 'test' command doesn't support the usual 'easy_install' arguments (my original reason for posting), I couldn't control where the downloaded packages would be installed: they went directly in my local sandbox. In case it wasn't clear from my original post, I have the following use case: - Check out the "packaging bits" for a Zope egg I want to build, e.g.: $ svn co svn+ssh://svn.zope.org/repos/main/zope.interface/trunk zi $ cd zi - Run the tests, first downloading and installing any dependencies (including'test_requires'). Preferably, those dependencies should *not* be installed into 'site-packages'; I want to verify that the package correctly identifies all of its own dependencies, without relying on any other environment except a "bare" virtual python: $ /path/to/python setup.py test ... No local packages or download links found for zope.testing error: Could not find suitable distribution for \ Requirement.parse('zope.testing') My original plan was to add the 'easy_install' command line args to the 'test' command, which would have allowed: $ /path/to/python setup.py test \ --find-links="http://download.zope.org/distribution" Under the mechanism you redireted me to, I need to add the equivalent URL into zi/setup.py as a 'dependency_link' argument to 'setup', and then arrange for that value to be used as the 'find_links' value for the 'test' command. Is that what you intended? Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFEGYnE+gerLs4ltQ4RAvoNAJ0bXH1PaiSiwCrkUNX9vKcUKmZd6wCdHusa nbxSITAWe1kk4L9e300djFA= =YA3h -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: setuptools-dependency_links.patch Type: text/x-patch Size: 3867 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060316/595eba95/attachment.bin From pje at telecommunity.com Thu Mar 16 20:49:36 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 16 Mar 2006 14:49:36 -0500 Subject: [Distutils] setup.py test doesn't honor --find-links In-Reply-To: <441989C4.4000401@palladion.com> References: <5.1.1.6.0.20060315113145.043b29d8@mail.telecommunity.com> <5.1.1.6.0.20060315113145.043b29d8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060316143730.02f98cb8@mail.telecommunity.com> At 10:52 AM 3/16/2006 -0500, Tres Seaver wrote: >I'm a bit lost as to the rationale for the separate metadata filee in >that post: why wouldn't we just use the 'dependency_links' attribute of >the distribution? Because the original post is discussing the scenario where you are installing an egg that was already built by someone else. You're correct that the fetch_build_eggs() method can and should just use the attribute directly. > At any rate, I'm attaching a partially-working patch. I'll take it for a spin. It looks pretty good, except that easy_install doesn't use it. I can go ahead and add that bit. The doc is a bit, um, terse, but I can flesh it out a bit I guess. :) >I could get the 'dependency_links.txt' file to be created, but couldn't >figure out how to use the pkg_resources facilities for reading it while >inside 'fetch_build_eggs'. You don't need to; it's the easy_install command that needs to do that, in process_distribution() I believe. > I alos ran into the problem that, since the >'test' command doesn't support the usual 'easy_install' arguments (my >original reason for posting), I couldn't control where the downloaded >packages would be installed: they went directly in my local sandbox. That's where build dependencies are supposed to go, actually; building and testing a package shouldn't cause build-time dependencies to be permanently installed on the build system, without an explicit action. You can always "easy_install *.egg" if you want to make them permanent. >My original plan was to add the 'easy_install' command line args to >the 'test' command, which would have allowed: > > $ /path/to/python setup.py test \ > --find-links="http://download.zope.org/distribution" > >Under the mechanism you redireted me to, I need to add the equivalent >URL into zi/setup.py as a 'dependency_link' argument to 'setup', and >then arrange for that value to be used as the 'find_links' value for >the 'test' command. Is that what you intended? Yes; it should get passed to the find_links argument given to the easy_install command instance created by Distribution.fetch_build_eggs(), which is what your patch does. From pje at telecommunity.com Thu Mar 16 20:52:23 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 16 Mar 2006 14:52:23 -0500 Subject: [Distutils] Missing packages In-Reply-To: <200603161456.52971.Juha.Tuomala@iki.fi> Message-ID: <5.1.1.6.0.20060316144940.02f9a0a8@mail.telecommunity.com> At 02:56 PM 3/16/2006 +0200, Juha Tuomala wrote: >It looks like paster cannot find cheetach, even it's installed. >I've compiled and installed cheetach RPM package. What is the thing >what setuptools looks and doesn't find? That egg info? Yes. The simplest way to get it is probably to build the Cheetah RPM using setuptools, by getting the source and then running this in the directory with its setup.py: python -c "import setuptools; execfile('setup.py')" bdist_rpm ... where '...' is any arguments you would normally pass to "setup.py bdist_rpm". You can probably fetch the Cheetah source using easy_install, e.g.: easy_install -eb. Cheetah which will create a 'cheetah' directory in the current directory, containing the source distribution. You can then 'cd cheetah' and run the setuptools command shown above. From pje at telecommunity.com Thu Mar 16 21:31:49 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 16 Mar 2006 15:31:49 -0500 Subject: [Distutils] Finding dependencies and Limiting download In-Reply-To: <44198E98.1060706@colorstudy.com> References: <200603161224.18999.Juha.Tuomala@iki.fi> <5.1.1.6.0.20060315113838.043b9430@mail.telecommunity.com> <5.1.1.6.0.20060315142727.02053150@mail.telecommunity.com> <200603161224.18999.Juha.Tuomala@iki.fi> Message-ID: <5.1.1.6.0.20060316145300.0363f5e0@mail.telecommunity.com> At 10:13 AM 3/16/2006 -0600, Ian Bicking wrote: >It occurs to me that maybe it would be easiest to create an RPM that >includes Paste, PasteDeploy, and PasteScript all together? That would >simplify all of this, it seems. That would certainly fix the conflict problem in the short term. In the long run, I'd like a better solution, though. Separate components for namespace packages are too good a thing to give up on without a fight. I'm tempted to see if there isn't a way to create an import hook in pkg_resources that would allow namespace packages to work without having an __init__.py. That would eliminate the file conflicts, albeit at the expense of requiring pkg_resources to be imported before any namespace packages. That seems problematic, though, since it would only be needed for --svem pacakges. The fundamental problem is that you end up having to have dummy packages to hold the __init__.py, or else you have to have something that gets imported first. It seems a bit fragile. One hack would be to add a .pth file for each --svem install of a namespace package, something like "PasteDeploy-nspkg.pth", that contains some code to activate the namespace package, like: import sys,new; \ m = sys.modules.setdefault('paste',new.module('paste')); \ p = os.path.join(sys._getframe(1).f_locals['sitedir'], 'paste'); \ mp = m.__path__ = getattr(m,'__path__',[]); \ (p not in mp) and mp.append(p) Yeah, that would do the trick, if --svem installs left out the __init__.py for declared namespace packages. Of course, this slows down Python startup time when you have a lot of --svem namespace packages installed, and doesn't work with "python -S", but what else can we do? If a given system packager wants to handle namespace packages differently, we could probably accomodate that via modifiers to the --svem mode. Really, the hardest part of implementing this is going to be the bits of setuptools that need to munge the build_py and install_lib commands to leave out __init__ files for namespace packages. I'll probably also need to document how to deal with certain legacy situations where there is an official package that "owns" the namespace and needs to install an __init__.py that actually contains code. From ianb at colorstudy.com Thu Mar 16 21:37:54 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 16 Mar 2006 14:37:54 -0600 Subject: [Distutils] Finding dependencies and Limiting download In-Reply-To: <5.1.1.6.0.20060316145300.0363f5e0@mail.telecommunity.com> References: <200603161224.18999.Juha.Tuomala@iki.fi> <5.1.1.6.0.20060315113838.043b9430@mail.telecommunity.com> <5.1.1.6.0.20060315142727.02053150@mail.telecommunity.com> <200603161224.18999.Juha.Tuomala@iki.fi> <5.1.1.6.0.20060316145300.0363f5e0@mail.telecommunity.com> Message-ID: <4419CCA2.1060502@colorstudy.com> Phillip J. Eby wrote: > At 10:13 AM 3/16/2006 -0600, Ian Bicking wrote: > >> It occurs to me that maybe it would be easiest to create an RPM that >> includes Paste, PasteDeploy, and PasteScript all together? That would >> simplify all of this, it seems. > > > That would certainly fix the conflict problem in the short term. > > In the long run, I'd like a better solution, though. Separate > components for namespace packages are too good a thing to give up on > without a fight. I'm tempted to see if there isn't a way to create an > import hook in pkg_resources that would allow namespace packages to work > without having an __init__.py. That would eliminate the file conflicts, > albeit at the expense of requiring pkg_resources to be imported before > any namespace packages. Another solution is at the package level -- RPMs can overwrite each other's files, if they are spec'd out to do that somehow. This is not very clean -- I don't know which if any package is allowed to ultimately delete that file. With some post-install hooks you could just create the file, and delete when there's nothing left in the package. This would have to be handled for every packaging system. But there's only a couple anyone uses. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Thu Mar 16 21:46:47 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 16 Mar 2006 15:46:47 -0500 Subject: [Distutils] Finding dependencies and Limiting download In-Reply-To: <200603161224.18999.Juha.Tuomala@iki.fi> References: <5.1.1.6.0.20060315142727.02053150@mail.telecommunity.com> <5.1.1.6.0.20060315113838.043b9430@mail.telecommunity.com> <5.1.1.6.0.20060315142727.02053150@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060316153638.02faac68@mail.telecommunity.com> At 12:24 PM 3/16/2006 +0200, Juha Tuomala wrote: >On Wednesday 15 March 2006 21:33, Phillip J. Eby wrote: > > Well, you can work around this, but it's a pain because it makes it > > impossible for you to use setup.py develop. What you do is delete the > > __init__.py from all the packages, except for a special extra project that > > includes only the __init__.py. Then you make all the packages depend on > > that one. (And each package with a missing __init__ needs to have the > > package name explicitly listed in setup(), since find_packages() won't see > > it's there otherwise.) > >Where those __init__.py files should be deleted? From SVN snapshot >taken from Script? Or from installed target chroot during RPM packaging? Either the source tree you're building from, or the installed chroot. > > At that point, setuptools will add automatically-created __init__.py files > > into any eggs it makes, but it will not include them in > > --single-version-externally-managed installs. > >Currently I'm doing > > %{__python} setup.py install --root $RPM_BUILD_ROOT >- --single-version-externally-managed > >And it installs below: > > /usr/lib/python2.3/site-packages/paste/script > >which is logcical but also below: > > /usr/lib/python2.3/site-packages/PasteScript-0.5.1dev-py2.3.egg-info/ > >so that must be the egg-info you talked about. Yes. Again, please note that "setup.py bdist_rpm" can do a lot of this for you without you needing to do it manually. > > The downside, however, is that it makes "setup.py develop" useless for the > > original source packages. > >I don't even know what that 'develop' does, recall reading the docs that it >helps with syncing the svn snapshot or something. Yeah, I wouldn't worry about it, given what you're trying to do. I was just pointing out why Ian can't just remove the __init__.py files from the original source code; it would interfere with him working on the package normally. >So, all what I would like to do is do > > make install > >and say > > paster serve > >and if that would work, it would be great. Somehow I'm stuck with this >setuptools/egg stuff what looks that it's making things complicated. > >Doing it manually is not an option as everything is installed with RPM >what allows strict management. Actually, it's the way in which you're building the RPMs that's making things complicated here. If you just did this: export PYTHONPATH=/some/project/dir easy_install -d /some/project/dir Paste PasteDeploy PasteScript You'd get everything neatly installed in /some/project/dir with nothing installed anywhere else, and no need for an RPM. (It would also install a copy of Cheetah with the missing egg-info.) But if you really *must* have an RPM, you can just make an RPM of the resulting directory, as long as you don't mind it always being installed to /some/project/dir. (Because the created files will contain some absolute paths.) Anyway, you could do something like "/usr/local/paste" as the target directory, and just put in whatever you need to either symlink the executables from there to a bin directory. If you don't want to have to set PYTHONPATH at runtime, you can use the "administrator install" setup instructions for easy_install to make /usr/local/paste an automatic "site" directory when Python starts. All this having been said, I think there *is* a reasonable way to work around the __init__.py issue for packaging tools like RPM, but what I've just given you should work now, so that you don't have to wait for any new features to come out in setuptools. From pje at telecommunity.com Thu Mar 16 22:27:51 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 16 Mar 2006 16:27:51 -0500 Subject: [Distutils] Finding dependencies and Limiting download In-Reply-To: <4419CCA2.1060502@colorstudy.com> References: <5.1.1.6.0.20060316145300.0363f5e0@mail.telecommunity.com> <200603161224.18999.Juha.Tuomala@iki.fi> <5.1.1.6.0.20060315113838.043b9430@mail.telecommunity.com> <5.1.1.6.0.20060315142727.02053150@mail.telecommunity.com> <200603161224.18999.Juha.Tuomala@iki.fi> <5.1.1.6.0.20060316145300.0363f5e0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060316162201.03225018@mail.telecommunity.com> At 02:37 PM 3/16/2006 -0600, Ian Bicking wrote: >Another solution is at the package level -- RPMs can overwrite each >other's files, if they are spec'd out to do that somehow. This is not >very clean -- I don't know which if any package is allowed to ultimately >delete that file. With some post-install hooks you could just create the >file, and delete when there's nothing left in the package. This would >have to be handled for every packaging system. But there's only a couple >anyone uses. Er, define "couple" and "anyone". :) I've heard from folks using Gentoo, RPM, .deb, and win32.exe just to start with. And let's not forget the people doing darwinports and other BSD systems. And that's all only talking about packaging systems that have existing users wrapping setuptools and setuptools-based packages, not general Python-using systems! So it's a bit more than a couple. And I suspect that most, if not all, would just as soon that "setup.py install --root" should "just work" without having to change their build processes, and my proposed solution would ensure that. Indeed, I believe that this fix for namespace packages is probably the last remaining impediment to setuptools' ability to achieve Total World Domination, except for the introduction of .egg-info files in Python 2.5. As far as I can tell, these are the only things left before system packages and user packages can all "just get along". From mdaniel at scdi.com Thu Mar 16 22:05:06 2006 From: mdaniel at scdi.com (Matthew L Daniel) Date: Thu, 16 Mar 2006 16:05:06 -0500 Subject: [Distutils] Missing packages In-Reply-To: <200603161456.52971.Juha.Tuomala@iki.fi> References: <200603161456.52971.Juha.Tuomala@iki.fi> Message-ID: <20060316210506.GA2926@cafebabe.intra> > /usr/lib64/python2.3/site-packages/Cheetah ------^^^^^^ Be cautious, I have experienced Python packages which hard-code 'lib' in their search path. I cannot speak to your situation, or certainly to distutils (yet), but wanted to mention that in case it rang bells with someone. -- /v\atthew From Juha.Tuomala at iki.fi Fri Mar 17 10:39:24 2006 From: Juha.Tuomala at iki.fi (Juha Tuomala) Date: Fri, 17 Mar 2006 11:39:24 +0200 Subject: [Distutils] Finding dependencies and Limiting download In-Reply-To: <4419CCA2.1060502@colorstudy.com> References: <200603161224.18999.Juha.Tuomala@iki.fi> <5.1.1.6.0.20060316145300.0363f5e0@mail.telecommunity.com> <4419CCA2.1060502@colorstudy.com> Message-ID: <200603171139.32605.Juha.Tuomala@iki.fi> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday 16 March 2006 22:37, Ian Bicking wrote: > Another solution is at the package level -- RPMs can overwrite each > other's files, if they are spec'd out to do that somehow. Yes, you could do that in the %post section where post installation script can be put. It's executed as root so it can do rm -rf / if needed. > This is not > very clean -- I don't know which if any package is allowed to ultimately > delete that file. All packages are installed as root. So any package could do that. There where the trust comes into picture. I trust fedoraproject.org as my OS supplier and they digitally sign every package, so it's unlikely that any package would behave badly in my system. > With some post-install hooks you could just create > the file, and delete when there's nothing left in the package. This > would have to be handled for every packaging system. But there's only a > couple anyone uses. First, doing so would never be accepted in most Linux distrubtions and it would never pass their QA examination. It would break the package checksums. If I would like to check that 'what is wrong in that software' and would run $ rpm -V postfix S.5....T. c /etc/postfix/main.cf And I can see that that file's size, md5sum and time differes, but it doesn't matter as it is a condifuration file. If that would have listed other files, I would have removed and reinstalled it to fix it. It would also break file listing done with rpm -q -l . Screwing the RPM database from a kitchen door will screw the package metadata and ruin the whole purpose of using RPM. So, deleting files from the package bookeeping would flag the file missing and create an error. I think that there is something wrong in the installation if it overwrites each other's files during the process. BR, Tuju - -- Ajatteleva ihminen tarvitsee unta. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.1 (GNU/Linux) iD8DBQFEGoPUQnPP29fw0xMRAohpAJwPdBcPkc1cohf+W0oXEcKKqhRHSACcDO3w KwJgoVrYabii6SctDZuF2XU= =XrO4 -----END PGP SIGNATURE----- From Juha.Tuomala at iki.fi Fri Mar 17 15:07:40 2006 From: Juha.Tuomala at iki.fi (Juha Tuomala) Date: Fri, 17 Mar 2006 16:07:40 +0200 Subject: [Distutils] Missing packages In-Reply-To: <5.1.1.6.0.20060316144940.02f9a0a8@mail.telecommunity.com> References: <5.1.1.6.0.20060316144940.02f9a0a8@mail.telecommunity.com> Message-ID: <200603171607.48066.Juha.Tuomala@iki.fi> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday 16 March 2006 21:52, Phillip J. Eby wrote: > Yes. The simplest way to get it is probably to build the Cheetah RPM using > setuptools, by getting the source and then running this in the directory > with its setup.py: > > python -c "import setuptools; execfile('setup.py')" bdist_rpm ... These switches are pobably used: --spec-only only regenerate spec file --source-only only generate source RPM --binary-only only generate binary RPM I tried --spec-only and the spec file what it generates might work, but would never pass to Fedora/Red Hat/Suste distro as such. Linux distributions simply are that different from each other that you can't mix specs, you need to write its own to every distro itself. In Linux world dependencies are handled with package management with RPM and yum/apt like tools. For me it looks that setuptools handles same area which is overlapping. I can see it's importance in systems like Solaris where its own package tool is so ancient what doesn't meet todays needs but in case of RPM or Linux, it would be just better to be able to do simple install and let the system native tools to handle the rest. BR, Tuju - -- Ajatteleva ihminen tarvitsee unta. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.1 (GNU/Linux) iD8DBQFEGsKzQnPP29fw0xMRAnLWAJ93LY98D1AVNdL2uyh4aK6pf8Ot2gCfcpy5 FTpVxbwHi/plVlNJRhZW3Q0= =eWVv -----END PGP SIGNATURE----- From jjl at pobox.com Fri Mar 17 15:40:55 2006 From: jjl at pobox.com (John J Lee) Date: Fri, 17 Mar 2006 14:40:55 +0000 (GMT Standard Time) Subject: [Distutils] Missing packages In-Reply-To: <200603171607.48066.Juha.Tuomala@iki.fi> References: <5.1.1.6.0.20060316144940.02f9a0a8@mail.telecommunity.com> <200603171607.48066.Juha.Tuomala@iki.fi> Message-ID: On Fri, 17 Mar 2006, Juha Tuomala wrote: [...] > In Linux world dependencies are handled with package management > with RPM and yum/apt like tools. For me it looks that setuptools > handles same area which is overlapping. I can see it's importance > in systems like Solaris where its own package tool is so ancient > what doesn't meet todays needs but in case of RPM or Linux, it > would be just better to be able to do simple install and let the > system native tools to handle the rest. This argument has already been gone through at great length on this list. In brief, Phillip has a number of goals that are not met by any system such as RPM. I'm sure it would be appreciated on all sides if you read the archived posts on this, wait for two days to cool off (not saying you're tetchy, but other people certainly were ;-), and then decide if you want to post more on this point. John From Juha.Tuomala at iki.fi Fri Mar 17 17:13:22 2006 From: Juha.Tuomala at iki.fi (Juha Tuomala) Date: Fri, 17 Mar 2006 18:13:22 +0200 Subject: [Distutils] Missing packages In-Reply-To: References: <5.1.1.6.0.20060316144940.02f9a0a8@mail.telecommunity.com> <200603171607.48066.Juha.Tuomala@iki.fi> Message-ID: <200603171813.30390.Juha.Tuomala@iki.fi> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday 17 March 2006 16:40, John J Lee wrote: > This argument has already been gone through at great length on this list. > In brief, Phillip has a number of goals that are not met by any system > such as RPM. Maybe. I look forward when I can install my binary kernel as setuptools egg. Or did you mean that the goal was not to unify sw management so that computer illeterates could install software without reading zillion different README's and learning different systems? > (not saying you're tetchy, but other people certainly were ;-), > and then decide if you want to post more on this point. You got me wrong in that point. I just try not to express it publicly as it typically doesn't help anything. :-) > I'm sure it would be appreciated on all sides if you read > the archived posts on this, wait for two days to cool off I did for this year. If I got the right picture, I try to stay away anything what uses setuptools. I think that it's just trying to solve problems inside python sandbox what is just 1% of the whole sw management issue and at the same time it's making it impossible to be solved properly where it should be done. BR, Tuju - -- Ajatteleva ihminen tarvitsee unta. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.1 (GNU/Linux) iD8DBQFEGuAqQnPP29fw0xMRApqvAJ93MtURVESNcLJPqMfZ6902X2Q68QCfYaN9 Ht1+luRG/w3Vp+9FMgeleA8= =1ZxO -----END PGP SIGNATURE----- From ianb at colorstudy.com Fri Mar 17 17:21:58 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 17 Mar 2006 10:21:58 -0600 Subject: [Distutils] Finding dependencies and Limiting download In-Reply-To: <200603171139.32605.Juha.Tuomala@iki.fi> References: <200603161224.18999.Juha.Tuomala@iki.fi> <5.1.1.6.0.20060316145300.0363f5e0@mail.telecommunity.com> <4419CCA2.1060502@colorstudy.com> <200603171139.32605.Juha.Tuomala@iki.fi> Message-ID: <441AE226.6050205@colorstudy.com> Juha Tuomala wrote: > On Thursday 16 March 2006 22:37, Ian Bicking wrote: > >>Another solution is at the package level -- RPMs can overwrite each >>other's files, if they are spec'd out to do that somehow. > > > Yes, you could do that in the %post section where post installation > script can be put. It's executed as root so it can do rm -rf / > if needed. > > >>This is not >>very clean -- I don't know which if any package is allowed to ultimately >>delete that file. > > > All packages are installed as root. So any package could do that. > There where the trust comes into picture. I trust fedoraproject.org > as my OS supplier and they digitally sign every package, so it's unlikely > that any package would behave badly in my system. Well, the question is which should. E.g., if you install PasteScript, then Paste, then remove PasteScript, PasteScript should leave paste/__init__.py since Paste also needs it. In practice, I tend to notice files like this are just left around by packagers. I don't think that would be appropriate at all in this situation, though (it would mean there would still be an importable "paste" even though there'd be nothing in it). > I think that there is something wrong in the installation if > it overwrites each other's files during the process. This is why Eggs have their namespace package support, but it relies on the packages being installed in separate hierarchies even though they share the same package space. If Phillip can fix that, then great, we should all be happy ;) Yet another option would be to make Paste a requirement for PasteDeploy and PasteScript, and rely on Paste installing paste/__init__.py Namespace packages *are* rather annoying, I'll admit. If I knew when I first wrote them how I wanted paste.* factored into three packages, I probably wouldn't have used a namespace package. The only real reason I see for it is legacy package names, or when you really really like hierarchy (and I don't). The issue with installing as an egg is that either you have to have postinstall scripts handle easy-install.pth (or rpm-install.pth or something), or you have to handle the fact that no egg will be installed by default. But otherwise you can just install an egg instead of using --single-version... -- and I don't know if there's really a reason not to do that at this time. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Fri Mar 17 17:29:52 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 17 Mar 2006 10:29:52 -0600 Subject: [Distutils] Another packaging strategy... Message-ID: <441AE400.4090807@colorstudy.com> There's another strategy for making packages that I thought I'd bring up -- using PYTHONPATH and creating a large bundled package. That means, if you have an application, distribute all the dependencies in the application's package. Stuff them in /usr/share/PKG/lib or something. This doesn't work for installation of libraries, but then you never *need* to install a library as a package. It's the application you are ultimately interested in deploying and distributing. And I think the non-system-package installation techniques are generally much more appropriate for the actual development process. I don't think the bundled package is necessarily *easy* to build right now. But the result seems easy to manage. I don't really know much about the build process for packages, so what is ultimately involved to do this I'm not sure. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Fri Mar 17 17:34:03 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 17 Mar 2006 11:34:03 -0500 Subject: [Distutils] Finding dependencies and Limiting download In-Reply-To: <441AE226.6050205@colorstudy.com> References: <200603171139.32605.Juha.Tuomala@iki.fi> <200603161224.18999.Juha.Tuomala@iki.fi> <5.1.1.6.0.20060316145300.0363f5e0@mail.telecommunity.com> <4419CCA2.1060502@colorstudy.com> <200603171139.32605.Juha.Tuomala@iki.fi> Message-ID: <5.1.1.6.0.20060317113107.01fc1c50@mail.telecommunity.com> At 10:21 AM 3/17/2006 -0600, Ian Bicking wrote: >If Phillip can fix that, then great, we >should all be happy ;) FYI, I've got a patch in my source checkout now and it appears to work great for the simple cases. I still have a few more complex scenarios to test, like bdist_wininst roundtripping with easy_install, and some documentation/release notes to update. I hope to have the fix in Subversion in a few hours. From grflanagan at yahoo.co.uk Fri Mar 17 17:46:56 2006 From: grflanagan at yahoo.co.uk (Ger Flanagan) Date: Fri, 17 Mar 2006 16:46:56 +0000 (GMT) Subject: [Distutils] paster.py failure on WinXP Message-ID: <20060317164656.27793.qmail@web26413.mail.ukl.yahoo.com> Hello all I posted the following message to the 'pylons-discuss' newsgroup, but maybe someone here can help, as I think it's related to setuptools? (Myghty installed the package 'setuptools-0.6a5-py2.4.egg') Thanks Gerard ----- Message ------ Hello all If this is off-topic, I apologise in advance. I'm interested in Pylons but I thought I'd check out Myghty first - I installed the latest version (Myghty-1.0.1 on Windows XP) and I can run the example shoppingcart application with no problems. However when I try to create a simple test app of my own using the 'paster.py' script, I get this error: >C:\DEVELOPMENT\myghty\test>paster create --template=myghty_simple simple C:\Program: can't open file 'Files\Python\Python24\python.exe': [Errno 2] No such file or directory As you can see, the location of my 'python.exe' is C:\Program Files\Python\Python24 but it looks like the space in the path may be causing a problem. (This has never been a problem before). I see that Pylons has a 'paster' command - presumably this is the same script that comes with the Myghty installation - has anyone come across this? Any advice would be welcome. Thanks. Gerard ------ End Message ------- ___________________________________________________________ Win a BlackBerry device from O2 with Yahoo!. Enter now. http://www.yahoo.co.uk/blackberry From pje at telecommunity.com Fri Mar 17 18:00:39 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 17 Mar 2006 12:00:39 -0500 Subject: [Distutils] Missing packages In-Reply-To: <200603171607.48066.Juha.Tuomala@iki.fi> References: <5.1.1.6.0.20060316144940.02f9a0a8@mail.telecommunity.com> <5.1.1.6.0.20060316144940.02f9a0a8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060317100403.0374d078@mail.telecommunity.com> At 04:07 PM 3/17/2006 +0200, Juha Tuomala wrote: >On Thursday 16 March 2006 21:52, Phillip J. Eby wrote: > > Yes. The simplest way to get it is probably to build the Cheetah RPM > using > > setuptools, by getting the source and then running this in the directory > > with its setup.py: > > > > python -c "import setuptools; execfile('setup.py')" bdist_rpm ... > >These switches are pobably used: > > --spec-only only regenerate spec file > --source-only only generate source RPM > --binary-only only generate binary RPM > >I tried --spec-only and the spec file what it generates >might work, but would never pass to Fedora/Red Hat/Suste distro >as such. Okay, then make the spec file do this for the install (instead of "python setup.py install --root"): python -c "import setuptools; execfile('setup.py')" install --root=... etc. >In Linux world dependencies are handled with package management >with RPM and yum/apt like tools. For me it looks that setuptools >handles same area which is overlapping. I can see it's importance >in systems like Solaris where its own package tool is so ancient >what doesn't meet todays needs but in case of RPM or Linux, it >would be just better to be able to do simple install and let the >system native tools to handle the rest. And that's exactly what you'll be able to do when Python 2.5 comes out. As it happens, you encountered only two actual limitations of the existing interface between setuptools and system packaging tools: 1. Namespace packages (which are used by the Paste* projects) and system packages don't work together properly yet 2. Not every Python project includes egg-info yet (e.g. Cheetah) Item #1 is now fixed in Subversion. Item #2 will be completely fixed when Python 2.5 comes out, as all distutils-based packages will install a .egg-info file, not just setuptools-based ones. Until Python 2.5 comes out, however, setuptools-based packages with dependencies will require that their dependency packages be repackaged using setuptools so that they install the .egg-info. This should only require a 1-line change to the dependency's spec file, as shown above. Thank you for having the patience to stick with it this long, and for providing me with the information I needed in order to fix problem #1. I hope you will find that the latest version of setuptools in Subversion makes it possible for you to proceed now. From pje at telecommunity.com Fri Mar 17 18:12:15 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 17 Mar 2006 12:12:15 -0500 Subject: [Distutils] paster.py failure on WinXP In-Reply-To: <20060317164656.27793.qmail@web26413.mail.ukl.yahoo.com> Message-ID: <5.1.1.6.0.20060317121122.042438c0@mail.telecommunity.com> At 04:46 PM 3/17/2006 +0000, Ger Flanagan wrote: >Hello all > >I posted the following message to the 'pylons-discuss' >newsgroup, but maybe someone here can help, as I think >it's related to setuptools? (Myghty installed the >package 'setuptools-0.6a5-py2.4.egg') Upgrade to the current version of setuptools (0.6a10); the "space in Python interpreter filename" problem was fixed quite a while back. From pje at telecommunity.com Fri Mar 17 18:23:44 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 17 Mar 2006 12:23:44 -0500 Subject: [Distutils] Using namespace packages? READ THIS Message-ID: <5.1.1.6.0.20060317120044.0423e5f0@mail.telecommunity.com> Conversely, if you're not using namespace packages with setuptools, you don't need to read this. Remain calm, everything is fine. :) If you do use them, please read on. Most of what's in here will probably not affect most people anyway, even if they do use namespace packages. But just take a few moments to check and make sure these issues don't apply to you. I just checked in a fix for the issue with namespace packages in system package installation scenarios. This fix unfortunately puts some significant limitations on what you can have in a namespace package __init__.py. Basically, *all* you should have in *every* namespace package __init__.py is: __import__('pkg_resources').declare_namespace(__name__) because the __init__.py files are now *not installed* when using --root or --single-version-externally-managed. So anything else you put in these files, is just not going to work in that case. Now, if you're doing what the ll-libs were doing, where you have one project containing the __init__.py, and the others not containing an __init__.py, and the main package doesn't declare a namespace but the others do, then that scenario will still work. If you don't know what this means, don't worry about it. :) Here's the really goofy part, that you need to deal with if you are currently distributing any win32.exe installers for a project that uses namespace packages: If you build a win32.exe with the latest setuptools in SVN, and your users use easy_install to install it, they *must* upgrade to the latest setuptools as well, or they will get a bunch of stuff about easy_install not knowing how to process various files, and the install will fail. (This is because setuptools now includes additional .pth files to munge namespace package stuff, and the old .exe-to-.egg conversion doesn't know how to handle it.) So anyway, if you care about this scenario (you distribute bdist_wininst win32.exe files but *not* win32.egg files), you should make sure you and your users are on the same setuptools version. But if you provide win32.egg files or do *not* provide win32.exe files, everything's cool. From trentm at ActiveState.com Sat Mar 18 02:31:31 2006 From: trentm at ActiveState.com (Trent Mick) Date: Fri, 17 Mar 2006 17:31:31 -0800 Subject: [Distutils] Using namespace packages? READ THIS In-Reply-To: <5.1.1.6.0.20060317120044.0423e5f0@mail.telecommunity.com> References: <5.1.1.6.0.20060317120044.0423e5f0@mail.telecommunity.com> Message-ID: <20060318013131.GK17401@activestate.com> [Phillip J. Eby wrote] > I just checked in a fix for the issue with namespace packages in system > package installation scenarios. This fix unfortunately puts some > significant limitations on what you can have in a namespace package > __init__.py. > > Basically, *all* you should have in *every* namespace package __init__.py is: > > __import__('pkg_resources').declare_namespace(__name__) Pardon my ignorance here. I'd just like to make sure this doesn't affect me if I move my stuff to setuptools. Is a "namespace package __init__.py" file different than, say, the "__init__.py" used in my personal (and mythical) "logging" alternative package: flogging/ __init__.py <---- this guy handlers.py ... ? Trent -- Trent Mick TrentM at ActiveState.com From bob at redivi.com Sat Mar 18 03:00:54 2006 From: bob at redivi.com (Bob Ippolito) Date: Fri, 17 Mar 2006 18:00:54 -0800 Subject: [Distutils] Using namespace packages? READ THIS In-Reply-To: <20060318013131.GK17401@activestate.com> References: <5.1.1.6.0.20060317120044.0423e5f0@mail.telecommunity.com> <20060318013131.GK17401@activestate.com> Message-ID: <58B5C1CF-C067-434D-A91D-8BF1959ACF39@redivi.com> On Mar 17, 2006, at 5:31 PM, Trent Mick wrote: > [Phillip J. Eby wrote] >> I just checked in a fix for the issue with namespace packages in >> system >> package installation scenarios. This fix unfortunately puts some >> significant limitations on what you can have in a namespace package >> __init__.py. >> >> Basically, *all* you should have in *every* namespace package >> __init__.py is: >> >> __import__('pkg_resources').declare_namespace(__name__) > > Pardon my ignorance here. I'd just like to make sure this doesn't > affect > me if I move my stuff to setuptools. Is a "namespace package > __init__.py" file different than, say, the "__init__.py" used in my > personal (and mythical) "logging" alternative package: > > flogging/ > __init__.py <---- this guy > handlers.py > ... Namespace packages support scenarios like zope or twisted where you have a bunch of different packages (zope.interfaces, etc.) that share a common top-level namespace but are distributed separately. zope would be the namespace package. Normal packages, like your theoretical flogging package, are not normally going to be namespace packages. -bob From pje at telecommunity.com Sat Mar 18 03:46:36 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 17 Mar 2006 21:46:36 -0500 Subject: [Distutils] Using namespace packages? READ THIS In-Reply-To: <20060318013131.GK17401@activestate.com> References: <5.1.1.6.0.20060317120044.0423e5f0@mail.telecommunity.com> <5.1.1.6.0.20060317120044.0423e5f0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060317214332.01fd5ef0@mail.telecommunity.com> At 05:31 PM 3/17/2006 -0800, Trent Mick wrote: >Pardon my ignorance here. I'd just like to make sure this doesn't affect >me if I move my stuff to setuptools. Is a "namespace package >__init__.py" file different than, say, the "__init__.py" used in my >personal (and mythical) "logging" alternative package: > > flogging/ > __init__.py <---- this guy > handlers.py > ... Yes, it's different. If you don't know what a namespace package is, you're not using one and don't need to worry. :) Sorry for the false alarm. (A namespace package is one whose contents are distributed as separately-installed components, and it's something that's somewhat tricky to do with the distutils by themselves.) From pje at telecommunity.com Sat Mar 18 04:07:29 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 17 Mar 2006 22:07:29 -0500 Subject: [Distutils] easy_install: bugs with pathing and multi-version In-Reply-To: <3b9601c64946$9da1e8d0$c921cdd1@norada.com> Message-ID: <5.1.1.6.0.20060317214921.04671e98@home.unrestrained.com> At 10:11 PM 3/16/2006 +0000, Maris Fogels wrote: >All of our applications are installed in a flat structure; an >application directory is one big file-soup. The applications are so >large that we do not have the budget to correct them, so I must >emulate this flat structure in the installation target directory. The >emulations requires the use of a setup.cfg and the distutils '$base' >varible: > ># example setup.cfg >[install] >install-base=/data/myproject >install-lib=$base >install-scripts=$base >install-headers=$base >install-data=$base Note that easy_install only has the equivalent of an install_lib and install_scripts; it doesn't install headers or any data other than package data embedded in eggs. >1. I unpacked the project sources in ~/src/myproject-1.0. When I tried to >use easy_install from the ~/src directory it ignores the install-base >directory >setting specified in setup.cfg: By design, easy_install only loads its own settings from ./setup.cfg. Any setup scripts it runs will read their own setup.cfg files, but since easy_install only runs "setup.py bdist_egg" on its targets, installation-related settings have no effect. This is intentional, since it's not good practice for a distributed setup.cfg to customize the install location. If easy_install honored those settings in the general case, it could result in quite a bit of chaos. >The command works if I cd into the project directory itself, or if I >specify the install directory with '-d' Yes, those are the two ways you can do that. >2. After successfully installing I attempted to uninstall the project >as per the instructions on the EasyInstall wiki page, by changing the >installed files to use a multi-install. It produces the following error: > >$ cd ~/src/myproject-1.0 >$ easy_install -m . I suggest using "easy_install -m myproject", as this does not need to run a setup script. Instead, it will simply detect that 'myproject' is already installed and remove it from sys.path. >zipimport.ZipImportError: bad local file header in >/data/myproject/myproject-1.0-py2.3.egg > >If I re-run the 'easy_install -m .' command it works properly. >Everything in the installed .egg file appears to work correctly, so I >could not figure out why a ZipImportError was being raised by a >working .egg. Me either. The only time I've seen anything like that happen is with Python 2.3 on a 64-bit platform. If you can figure out how to reproduce it, please let me know. >3. This may be a problem with distutils or setuptools, depending on >how the install command is dispatched. I thought it would be nice to put the >install record in the same directory I installed the other files in, >so I appended the following to setup.cfg: > >record=$base/uninstall.dat > >Alas, $base is not expanded for the 'record' configuration option. I think this feature can be added by simply inlcuding 'record' in the list of variables that are expanded in line 125 of setuptools/command/easy_install.py. If you have a chance to try that, please let me know if it works for you, and then I'll add it to the official version. From jjl at pobox.com Mon Mar 20 21:36:48 2006 From: jjl at pobox.com (John J Lee) Date: Mon, 20 Mar 2006 20:36:48 +0000 (UTC) Subject: [Distutils] easy_install2.3, easy_install2.4 etc.? Message-ID: Seems it would be nice to have the easy_install bootstrap installer create easy_installX.Y links for each version X.Y of Python. As it is, you have to fiddle with shell backticks, or type the full path to easy_install if on Windows. John From dangoor at gmail.com Mon Mar 20 22:02:45 2006 From: dangoor at gmail.com (Kevin Dangoor) Date: Mon, 20 Mar 2006 16:02:45 -0500 Subject: [Distutils] entry point for new installations? Message-ID: <3f085ecd0603201302l7a7720fdy3d7a50f5ec9797f8@mail.gmail.com> Someone just prodded me about automatic application reloading, which is always a hairy topic. The autoreload code that many people run in development mode forks to run the actual application in a child process, and everything in sys.modules is watched for a change. If one of the files change, the child exits and the parent process fires off a new one. It's ugly, but it's reliable and is generally fine for development. For TurboGears apps, I'm of the opinion that the right way to deploy an app to production is to turn it into an egg and install it. I had the (not entirely fleshed out) thought that it might be interesting if there was an entry_point that got called when new eggs were installed. This would make it possible to notify a running server that it needs to be restarted. Thoughts? Kevin -- Kevin Dangoor Author of the Zesty News RSS newsreader email: kid at blazingthings.com company: http://www.BlazingThings.com blog: http://www.BlueSkyOnMars.com From pje at telecommunity.com Mon Mar 20 23:11:59 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 20 Mar 2006 17:11:59 -0500 Subject: [Distutils] entry point for new installations? In-Reply-To: <3f085ecd0603201302l7a7720fdy3d7a50f5ec9797f8@mail.gmail.co m> Message-ID: <5.1.1.6.0.20060320161724.0369f3c8@mail.telecommunity.com> At 04:02 PM 3/20/2006 -0500, Kevin Dangoor wrote: >Someone just prodded me about automatic application reloading, which >is always a hairy topic. The autoreload code that many people run in >development mode forks to run the actual application in a child >process, and everything in sys.modules is watched for a change. If one >of the files change, the child exits and the parent process fires off >a new one. It's ugly, but it's reliable and is generally fine for >development. > >For TurboGears apps, I'm of the opinion that the right way to deploy >an app to production is to turn it into an egg and install it. I had >the (not entirely fleshed out) thought that it might be interesting if >there was an entry_point that got called when new eggs were installed. >This would make it possible to notify a running server that it needs >to be restarted. > >Thoughts? An entry point won't really help you much here, because it would have to know how to perform this notification. Presumably, it would either have to talk to a socket or modify a file somewhere. However, if you're going to have to monitor for file changes anyway, then you might as well just monitor sys.path directories for new .egg files, or monitor changes to easy-install.pth files. The only other thing I can think of that would work is if you had server processes write their process IDs some place, and then the hypothetical entry point could send them SIGHUPs or something. But monitoring file/directory changes is a bit more cross-platform, doesn't require adding any entry point code, is probably part of existing reload code, and doesn't require a pidfile arena, with all the associated permissions and cleanup headaches thereof. From dangoor at gmail.com Mon Mar 20 23:26:14 2006 From: dangoor at gmail.com (Kevin Dangoor) Date: Mon, 20 Mar 2006 17:26:14 -0500 Subject: [Distutils] entry point for new installations? In-Reply-To: <5.1.1.6.0.20060320161724.0369f3c8@mail.telecommunity.com> References: <5.1.1.6.0.20060320161724.0369f3c8@mail.telecommunity.com> Message-ID: <3f085ecd0603201426h6faa4d5fq34ffc9c292cf7656@mail.gmail.com> On 3/20/06, Phillip J. Eby wrote: > An entry point won't really help you much here, because it would have to > know how to perform this notification. Presumably, it would either have to > talk to a socket or modify a file somewhere. The egg that has that entry point would have to know how to find its configuration file to know what to notify, yes. It's not entirely pleasant, but it's still doable. > However, if you're going to have to monitor for file changes anyway, then > you might as well just monitor sys.path directories for new .egg files, or > monitor changes to easy-install.pth files. My thought was that no file monitoring would be required. easy_install would keep track of the eggs it installed and then pass that list along to everything that defines that entry point. > The only other thing I can think of that would work is if you had server > processes write their process IDs some place, and then the hypothetical > entry point could send them SIGHUPs or something. But monitoring > file/directory changes is a bit more cross-platform, doesn't require adding > any entry point code, is probably part of existing reload code, and doesn't > require a pidfile arena, with all the associated permissions and cleanup > headaches thereof. The entry point is probably a premature optimization. File modification time lookups are pretty cheap. And, if it's only watching easy-install.pth, that would be dirt cheap. Kevin From pje at telecommunity.com Mon Mar 20 23:31:18 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 20 Mar 2006 17:31:18 -0500 Subject: [Distutils] easy_install2.3, easy_install2.4 etc.? In-Reply-To: Message-ID: <5.1.1.6.0.20060320173027.01f950a8@mail.telecommunity.com> At 08:36 PM 3/20/2006 +0000, John J Lee wrote: >Seems it would be nice to have the easy_install bootstrap installer create >easy_installX.Y links for each version X.Y of Python. As it is, you have >to fiddle with shell backticks, or type the full path to easy_install if >on Windows. Done. Update your setuptools installation and enjoy. :) (I went with a script name of easy_install-X.Y, btw.) From pje at telecommunity.com Mon Mar 20 23:38:38 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 20 Mar 2006 17:38:38 -0500 Subject: [Distutils] entry point for new installations? In-Reply-To: <3f085ecd0603201426h6faa4d5fq34ffc9c292cf7656@mail.gmail.co m> References: <5.1.1.6.0.20060320161724.0369f3c8@mail.telecommunity.com> <5.1.1.6.0.20060320161724.0369f3c8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060320173309.038772a0@mail.telecommunity.com> At 05:26 PM 3/20/2006 -0500, Kevin Dangoor wrote: >On 3/20/06, Phillip J. Eby wrote: > > An entry point won't really help you much here, because it would have to > > know how to perform this notification. Presumably, it would either have to > > talk to a socket or modify a file somewhere. > >The egg that has that entry point would have to know how to find its >configuration file to know what to notify, yes. configuration file*s* - you might have multiple copies of something. And Python packages don't really (i.e. shouldn't) own configuration files of their own. > > However, if you're going to have to monitor for file changes anyway, then > > you might as well just monitor sys.path directories for new .egg files, or > > monitor changes to easy-install.pth files. > >My thought was that no file monitoring would be required. easy_install >would keep track of the eggs it installed and then pass that list >along to everything that defines that entry point. Yeah, except that your approach effectively requires building an egg for every *instance* of every app on the box, and also requires you to have a privileged process running an individual user's code. I know that you probably have a rather more restricted deployment scenario in mind, I'm just pointing out that the approach you were proposing falls apart as soon as multiple instances or users with different privileges are involved. (Heck, even my SIGHUP approach has some issues with that, since a high-privilege installer could then be tricked into signalling some other process.) > > The only other thing I can think of that would work is if you had server > > processes write their process IDs some place, and then the hypothetical > > entry point could send them SIGHUPs or something. But monitoring > > file/directory changes is a bit more cross-platform, doesn't require adding > > any entry point code, is probably part of existing reload code, and doesn't > > require a pidfile arena, with all the associated permissions and cleanup > > headaches thereof. > >The entry point is probably a premature optimization. File >modification time lookups are pretty cheap. And, if it's only watching >easy-install.pth, that would be dirt cheap. Yeah, and this latter exchange has convinced me that it's the only simple way to avoid privilege escalation attacks. That is, it doesn't require low/medium-privilege server processes to be able to write stuff (like entry point code, pids, or configuration files) that gets read or executed (or otherwise relied upon) by the high-privilege installation process. From dangoor at gmail.com Mon Mar 20 23:55:47 2006 From: dangoor at gmail.com (Kevin Dangoor) Date: Mon, 20 Mar 2006 17:55:47 -0500 Subject: [Distutils] entry point for new installations? In-Reply-To: <5.1.1.6.0.20060320173309.038772a0@mail.telecommunity.com> References: <5.1.1.6.0.20060320161724.0369f3c8@mail.telecommunity.com> <5.1.1.6.0.20060320173309.038772a0@mail.telecommunity.com> Message-ID: <3f085ecd0603201455i4d00936dmdb8bb65b3485b13a@mail.gmail.com> On 3/20/06, Phillip J. Eby wrote: > Yeah, and this latter exchange has convinced me that it's the only simple > way to avoid privilege escalation attacks. That is, it doesn't require > low/medium-privilege server processes to be able to write stuff (like entry > point code, pids, or configuration files) that gets read or executed (or > otherwise relied upon) by the high-privilege installation process. Excellent point. Clearly, I hadn't thought of privilege escalation... Kevin From mfogels at gmail.com Fri Mar 24 15:23:53 2006 From: mfogels at gmail.com (Mars) Date: Fri, 24 Mar 2006 09:23:53 -0500 Subject: [Distutils] easy_install: bugs with pathing and multi-version Message-ID: <3676e0c0603240623i31bb0023qdd1509fb31d255d0@mail.gmail.com> > From: pje at telecommunity.com (Phillip J. Eby) > Date: 3/17/2006 10:07 PM (Eastern Standard Time) > To: mfogels at interactdirect.com (Maris Fogels) > Cc: distutils-sig at python.org > Subject: Re: easy_install: bugs with pathing and multi-version > > At 10:11 PM 3/16/2006 +0000, Maris Fogels wrote: > >zipimport.ZipImportError: bad local file header in > >/data/myproject/myproject-1.0-py2.3.egg > > > >If I re-run the 'easy_install -m .' command it works properly. > >Everything in the installed .egg file appears to work correctly, so I > >could not figure out why a ZipImportError was being raised by a > >working .egg. > > Me either. The only time I've seen anything like that happen is with > Python 2.3 on a 64-bit platform. If you can figure out how to reproduce > it, please let me know. > Thankfully the problem is trivial for me to reproduce, and I was able to run the script through pdb. I believe I have isolated the cause of the problem, but I have not had the time to devise a solution. The problem starts with a normal installation, in a clean directory, using the command 'easy_install .'. There is a custom setup.cfg file there that specifies the install-directory. During installation a temporary egg file is created, along with the projectname.egg-info directory. The temporary egg is copied into the installation directory and renamed. I noted that the newly created SOURCES.txt manifest on my system does not contain an entry for itself. But the [incorrect] manifest file is properly wrapped into the egg. To uninstall, we run the command 'easy_install -m .'. This creates a new egg file, along with a new manifest. This new manifest does contain a reference to itself. This changes the egg file size and the contained zip-file offsets. At this point we have two different egg files and two different egg file definitions, with different zip-file offsets. The new egg file definiton is used to open the egg in the install directory. However, the egg that the install script finds there is a copy of the orginal temporary egg file, and the old file offsets do not match the new definition. Thus a ZipImportError is raised. I was under the impression that a fix to the class setuptools.command.egg_info.manifest_maker was in order, but the code already ensures that the manifest contains the needed self-reference. A run through the debugger should either verify this or pick up the bug. Maris From pje at telecommunity.com Fri Mar 24 18:33:24 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 24 Mar 2006 12:33:24 -0500 Subject: [Distutils] easy_install: bugs with pathing and multi-version In-Reply-To: <3676e0c0603240623i31bb0023qdd1509fb31d255d0@mail.gmail.com > Message-ID: <5.1.1.6.0.20060324122446.03725008@mail.telecommunity.com> At 09:23 AM 3/24/2006 -0500, Mars wrote: >The new egg file definiton is used to open the egg in the install directory. >However, the egg that the install script finds there is a copy of the >orginal temporary egg file, and the old file offsets do not match the new >definition. Thus a ZipImportError is raised. It sounds like easy_install needs to clear the zipimport directory cache, or at least remove any entry corresponding to the file it's installing, in the case where it's overwriting an existing file. As for the SOURCES.txt problem, it seems to be a quirk of the way the distutils FileList class works, such that if a file doesn't exist before SOURCES.txt is built, it doesn't end up listed in SOURCES.txt. So, I'll have to add a workaround for that as well. Thanks for taking the time to follow up on this. From pje at telecommunity.com Fri Mar 24 18:51:15 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 24 Mar 2006 12:51:15 -0500 Subject: [Distutils] easy_install: bugs with pathing and multi-version In-Reply-To: <5.1.1.6.0.20060324122446.03725008@mail.telecommunity.com> References: <3676e0c0603240623i31bb0023qdd1509fb31d255d0@mail.gmail.com > Message-ID: <5.1.1.6.0.20060324125037.020376b8@mail.telecommunity.com> At 12:33 PM 3/24/2006 -0500, Phillip J. Eby wrote: >At 09:23 AM 3/24/2006 -0500, Mars wrote: > >The new egg file definiton is used to open the egg in the install directory. > >However, the egg that the install script finds there is a copy of the > >orginal temporary egg file, and the old file offsets do not match the new > >definition. Thus a ZipImportError is raised. > >It sounds like easy_install needs to clear the zipimport directory cache, >or at least remove any entry corresponding to the file it's installing, in >the case where it's overwriting an existing file. > >As for the SOURCES.txt problem, it seems to be a quirk of the way the >distutils FileList class works, such that if a file doesn't exist before >SOURCES.txt is built, it doesn't end up listed in SOURCES.txt. So, I'll >have to add a workaround for that as well. > >Thanks for taking the time to follow up on this. By the way, both of these problems should now be fixed in setuptools SVN. Let me know if you have any further problems. Thanks. From pje at telecommunity.com Sun Mar 26 05:04:50 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 25 Mar 2006 22:04:50 -0500 Subject: [Distutils] Breakage of fetch in setuptools-0.6a10-py2.4.egg In-Reply-To: <20060326013241.76195.qmail@web54509.mail.yahoo.com> References: <5.1.1.6.0.20060317130710.0423e5f0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060325215905.02057708@mail.telecommunity.com> At 05:32 PM 3/25/2006 -0800, Grig Gheorghiu wrote: >Hi, Philip > >2 questions: > >1. Is there any chance you can release a new setuptools version that >includes the fix for the fetch breakage? I expect to release that version within a week. Nothing stops you, however, from using intermediate snapshots. Run this command to create an egg in the current directory that's a snapshot of the current SVN version: easy_install -zmaxd. setuptools==dev You can then distribute this egg with your package if needed, by passing the correct version info to the ez_setup() function in ez_setup.py (so it uses your copy of the snapshot. >2. Another thing that seems to be broken, and that I was depending on >in cheesecake, is installing a package to a non-default location via >the --home option. > >I'm not sure when this started to break in terms of setuptools, but it >used to work at least a couple of months ago. Now I get something >similar to this when I try to run setuptools-based setup.py scripts >with --home: > >================ >[ggheo at concord twill-0.8.4]$ python setup.py install >--home=/tmp/install_twill/ >running install >Checking .pth file support in /tmp/install_twill//lib/python/ >error: can't create or remove files in install directory > >The following error occurred while trying to add or remove files in the >installation directory: > > [Errno 2] No such file or directory: >'/tmp/install_twill//lib/python/test-easy-install-12889.pth' > >The installation directory you specified (via --install-dir, --prefix, >or >the distutils default setting) was: > > /tmp/install_twill//lib/python/ > >This directory does not currently exist. Please create it and try >again, or >choose a different installation directory (using the -d or >--install-dir >option). >================ > >Is there a better/less-prone-to-breakage way to test whether a package >can be installed to an alternate location, other than using --home? I >want this to work with distutils-based versions of setup.py too. If you use --root instead of --home, it will work nicely with both, assuming you use the current SVN version of setuptools. When setuptools' "install" command sees the --root option, it doesn't try to verify that the installation location is valid. This should be ideal for your purposes, although you may have to fiddle with some of the other options a little. --root also tells setuptools to use its "legacy installation" format, which will more closely mimic what the distutils would do in the same circumstances. From mfogels at gmail.com Tue Mar 28 18:24:14 2006 From: mfogels at gmail.com (Mars) Date: Tue, 28 Mar 2006 11:24:14 -0500 Subject: [Distutils] easy_install: bugs with pathing and multi-version In-Reply-To: <5.1.1.6.0.20060324125037.020376b8@mail.telecommunity.com> References: <5.1.1.6.0.20060324122446.03725008@mail.telecommunity.com> <5.1.1.6.0.20060324125037.020376b8@mail.telecommunity.com> Message-ID: <3676e0c0603280824y3b9a829fx4387852057cf2f47@mail.gmail.com> On 3/24/06, Phillip J. Eby wrote: > At 12:33 PM 3/24/2006 -0500, Phillip J. Eby wrote: > >At 09:23 AM 3/24/2006 -0500, Mars wrote: > > >The new egg file definiton is used to open the egg in the install directory. > > >However, the egg that the install script finds there is a copy of the > > >orginal temporary egg file, and the old file offsets do not match the new > > >definition. Thus a ZipImportError is raised. > > > >It sounds like easy_install needs to clear the zipimport directory cache, > >or at least remove any entry corresponding to the file it's installing, in > >the case where it's overwriting an existing file. > > > >As for the SOURCES.txt problem, it seems to be a quirk of the way the > >distutils FileList class works, such that if a file doesn't exist before > >SOURCES.txt is built, it doesn't end up listed in SOURCES.txt. So, I'll > >have to add a workaround for that as well. > > > >Thanks for taking the time to follow up on this. > > By the way, both of these problems should now be fixed in setuptools > SVN. Let me know if you have any further problems. Thanks. > > The error is fixed as of setuptools version 0.6a11dev_r43297. Thanks! Maris From schwehr at gmail.com Wed Mar 29 17:40:57 2006 From: schwehr at gmail.com (Kurt Schwehr) Date: Wed, 29 Mar 2006 10:40:57 -0500 Subject: [Distutils] setuptools in fink Message-ID: <37de72610603290740h654bf744md4cc4ac45be3fe9b@mail.gmail.com> Hi All, I am working on getting setup tools into fink. If anyone has time, please take a look at the setup file and let me know what you think. http://schwehr.org/software/fink/setuptools-py.info Once setuptools is installed, I am having trouble with SQLAlchemy which I am using as my test case to see if setuptools is installed correctly. Clearly I am doing something wrong. Can someone please set me straight? I am working with the 10.4-transitional tree within fink. Thanks! -kurt http://schwehr.org/software/fink/sqlalchemy-py.info This makes me think setup tools is ok since it is in the default site-packages: & /sw/bin/python2.4 Python 2.4.2 (#1, Feb 24 2006, 17:05:21) [GCC 4.0.1 (Apple Computer, Inc. build 5247)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from ez_setup import use_setuptools But then the SQLAlchemy build tries to pull down setuptools, which I don't want it to do: CC=gcc-3.3 /sw/bin/python2.4 setup.py build --------------------------------------------------------------------------- This script requires setuptools version 0.6a5 to run (even to display help). I will attempt to download it for you (from http://cheeseshop.python.org/packages/2.4/s/setuptools/), but you may need to enable firewall access for this script first. I will start the download in 15 seconds. --------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20060329/08365118/attachment.htm From pje at telecommunity.com Wed Mar 29 19:44:29 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 29 Mar 2006 12:44:29 -0500 Subject: [Distutils] setuptools in fink In-Reply-To: <37de72610603290740h654bf744md4cc4ac45be3fe9b@mail.gmail.co m> Message-ID: <5.1.1.6.0.20060329124158.0390efc8@mail.telecommunity.com> At 10:40 AM 3/29/2006 -0500, Kurt Schwehr wrote: >This makes me think setup tools is ok since it is in the default >site-packages: > >& /sw/bin/python2.4 >Python 2.4.2 (#1, Feb 24 2006, 17:05:21) >[GCC 4.0.1 (Apple Computer, Inc. build 5247)] on darwin >Type "help", "copyright", "credits" or "license" for more information. > >>> from ez_setup import use_setuptools You didn't *call* use_setuptools, or import setuptools itself, so this doesn't actually mean that setuptools is okay. The error message below is displayed when use_setuptools() is *called*, not when it's imported. My guess is that setuptools isn't actually on the path, either because it didn't install correctly, or because the configuration for building SQLAlchemy doesn't specify that its build requires setuptools. >But then the SQLAlchemy build tries to pull down setuptools, which I don't >want it to do: > >CC=gcc-3.3 /sw/bin/python2.4 setup.py build > >--------------------------------------------------------------------------- >This script requires setuptools version 0.6a5 to run (even to display >help). I will attempt to download it for you (from >http://cheeseshop.python.org/packages/2.4/s/setuptools/ >), but >you may need to enable firewall access for this script first. >I will start the download in 15 seconds. >--------------------------------------------------------------------------- From pje at telecommunity.com Thu Mar 30 00:27:49 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 29 Mar 2006 17:27:49 -0500 Subject: [Distutils] Revision control plugin support for setuptools Message-ID: <5.1.1.6.0.20060329172515.044c1cc8@mail.telecommunity.com> I've just checked in support for revision control plugins in setuptools. This means that folks using any of the various revision control systems out there other than CVS and SVN can now write a plugin for setuptools to support automatically finding data and source files when building packages and their source distributions. Please see setuptools.txt in the source for documentation, and let me know if you have any questions or problems. From pje at telecommunity.com Thu Mar 30 01:38:38 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 29 Mar 2006 18:38:38 -0500 Subject: [Distutils] Custom test loader support in setuptools Message-ID: <5.1.1.6.0.20060329183418.0379c278@mail.telecommunity.com> I've just added a ``test_loader`` keyword argument to ``setup()`` in setuptools' Subversion edition. This lets you specify a string of the form "modulename:classname" to specify an alternative to unittest:TestLoader for finding tests. Note that an instance of the specified class will receive a name that comes from the ``test_suite`` argument, but it is free to interpret that string in any way it wishes. For example, a "nose" loader class could be created that parses command line arguments from the test_suite string. The default test_loader is "setuptools.command.test:ScanningLoader", which scans submodules/subpackages for tests as well as invoking any module-level "additional_tests()" functions it finds. You can use a test_loader of "unittest:TestLoader" to force the old (non-scanning) behavior. And of course the field is wide open for people to create new test loaders, or wrap other testing frameworks as test loaders. Please see setuptools.txt in the source, under the "New and Changed ``setup()`` Keywords" section, for more details on the ``test_loader`` argument. From schwehr at gmail.com Thu Mar 30 01:42:41 2006 From: schwehr at gmail.com (Kurt Schwehr) Date: Wed, 29 Mar 2006 18:42:41 -0500 Subject: [Distutils] setuptools in fink In-Reply-To: <5.1.1.6.0.20060329124158.0390efc8@mail.telecommunity.com> References: <5.1.1.6.0.20060329124158.0390efc8@mail.telecommunity.com> Message-ID: <37de72610603291542y26e10a17t886e0f8528ba8ced@mail.gmail.com> Phillip and others, Does this look like it is installed correctly? Or is it that I am calling setup.py when I should be calling ez_setup? And what should I pass to ez_setup? It does look like setuptools are installed incorrectly since easy_install -h does not run. So what is the right way? Thanks! -kurt easy_install -h Traceback (most recent call last): File "/sw/bin/easy_install", line 5, in ? from pkg_resources import load_entry_point ImportError: No module named pkg_resources cat /sw/lib/python2.4/site-packages/setuptools.pth /sw/src/fink.build/root-setuptools-py24-0.6a10-1 /sw/lib/python2.4/site-packages/setuptools-0.6a10-py2.4.egg dpkg -L setuptools-py24 | head -n 15 /. /sw /sw/bin /sw/bin/easy_install /sw/bin/easy_install.py /sw/lib /sw/lib/python2.4 /sw/lib/python2.4/site-packages /sw/lib/python2.4/site-packages/easy-install.pth /sw/lib/python2.4/site-packages/setuptools-0.6a10-py2.4.egg /sw/lib/python2.4/site-packages/setuptools-0.6a10-py2.4.egg/easy_install.py /sw/lib/python2.4/site-packages/setuptools-0.6a10-py2.4.egg/easy_install.pyc /sw/lib/python2.4/site-packages/setuptools-0.6a10-py2.4.egg/EGG-INFO /sw/lib/python2.4/site-packages/setuptools-0.6a10-py2.4.egg /EGG-INFO/entry_points.txt /sw/lib/python2.4/site-packages/setuptools-0.6a10-py2.4.egg /EGG-INFO/not-zip-safe On 3/29/06, Phillip J. Eby wrote: > > At 10:40 AM 3/29/2006 -0500, Kurt Schwehr wrote: > >This makes me think setup tools is ok since it is in the default > >site-packages: > > > >& /sw/bin/python2.4 > >Python 2.4.2 (#1, Feb 24 2006, 17:05:21) > >[GCC 4.0.1 (Apple Computer, Inc. build 5247)] on darwin > >Type "help", "copyright", "credits" or "license" for more information. > > >>> from ez_setup import use_setuptools > > You didn't *call* use_setuptools, or import setuptools itself, so this > doesn't actually mean that setuptools is okay. The error message below is > displayed when use_setuptools() is *called*, not when it's imported. > > My guess is that setuptools isn't actually on the path, either because it > didn't install correctly, or because the configuration for building > SQLAlchemy doesn't specify that its build requires setuptools. > > > >But then the SQLAlchemy build tries to pull down setuptools, which I > don't > >want it to do: > > > >CC=gcc-3.3 /sw/bin/python2.4 setup.py build > > > > >--------------------------------------------------------------------------- > >This script requires setuptools version 0.6a5 to run (even to display > >help). I will attempt to download it for you (from > > > http://cheeseshop.python.org/packages/2.4/s/setuptools/ > >), but > >you may need to enable firewall access for this script first. > >I will start the download in 15 seconds. > > >--------------------------------------------------------------------------- > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20060329/9f4843fd/attachment.html From pje at telecommunity.com Thu Mar 30 02:08:15 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 29 Mar 2006 19:08:15 -0500 Subject: [Distutils] setuptools in fink In-Reply-To: <37de72610603291542y26e10a17t886e0f8528ba8ced@mail.gmail.co m> References: <5.1.1.6.0.20060329124158.0390efc8@mail.telecommunity.com> <5.1.1.6.0.20060329124158.0390efc8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060329190018.01fa9830@mail.telecommunity.com> At 06:42 PM 3/29/2006 -0500, Kurt Schwehr wrote: >Phillip and others, > >Does this look like it is installed correctly? Or is it that I am calling >setup.py when I should be calling ez_setup? And what should I pass to >ez_setup? It does look like setuptools are installed incorrectly since >easy_install -h does not run. So what is the right way? If you are packaging setuptools for use with a system packager, you should use: python setup.py install --root=/some/pseudoroot If you are using setuptools 0.6a10 or earlier, you *also* need --single-version-externally-managed for this to work. 0.6a11 (which I just released a few minutes ago) automatically sets --single-version-externally-managed if you specify a --root, so you might want to just go ahead and upgrade, especially since 0.6a11 has a lot of other changes to improve compatibility with system packagers. For example, while 0.6a10 will complain about system-installed versions of a package as conflicting with a package that is being installed, 0.6a11 installs things such that there are no conflicts. 0.6a11 also supports making system packages for projects containing namespace packages, without causing inter-package conflicts for the packaging system. From pje at telecommunity.com Thu Mar 30 02:16:07 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 29 Mar 2006 19:16:07 -0500 Subject: [Distutils] setuptools 0.6a11 released! Message-ID: <5.1.1.6.0.20060329190856.03ff26c0@mail.telecommunity.com> This is a major new feature and bugfix release of setuptools, including: * Overhauled conflict management: no more -D/--ignore-conflicts-at-my-risk junk, it Just Works(TM). * Namespace package support that's compatible with system packagers * Lots of new setup() features including dependency_links (thanks to Tres Seaver), test_loader, and enhanced test_suite finder * Upload supports --identity (thanks to Stefan Behnel) * Improved error messages for unwritable cache directory, and added a special exception for wrapping such errors * Revision control plugin support * And a ton of minor bugfixes and creature comforts/"fit-and-finish" issues, ranging from version-specific "easy_install-2.x" executables to relative paths in .pth files. Please see the docs for details. Barring any new bugs, this should be the last 0.6 alpha release before we go beta. The 0.6 beta branch will then become a candidate for inclusion in Python 2.5 alpha 2, sometime in the next few weeks. So please report any issues as soon as practical. Thank you to everyone who reported problems, asked questions, and supplied patches. From schwehr at gmail.com Thu Mar 30 05:33:02 2006 From: schwehr at gmail.com (Kurt Schwehr) Date: Wed, 29 Mar 2006 22:33:02 -0500 Subject: [Distutils] setuptools in fink In-Reply-To: <5.1.1.6.0.20060329190018.01fa9830@mail.telecommunity.com> References: <5.1.1.6.0.20060329124158.0390efc8@mail.telecommunity.com> <5.1.1.6.0.20060329190018.01fa9830@mail.telecommunity.com> Message-ID: <37de72610603291933k703fddceqce194c4e35e1bd55@mail.gmail.com> I think I have it figured out. Fink requires installing a package in a temporary root install that is then turned into a deb. This makes the pth files have bad paths. If I manage the paths in the pth files after the install, then things seem to work. It looks like i will have to manage the easy_install.pth anyway since if it ends up in a deb, then other packages can't work with it. I'm going to create PostInstall, PostRm commands to take care of that. /sw/bin/easy_install -h is now running so I think I am on the right track. -kurt On 3/29/06, Phillip J. Eby wrote: > > At 06:42 PM 3/29/2006 -0500, Kurt Schwehr wrote: > >Phillip and others, > > > >Does this look like it is installed correctly? Or is it that I am > calling > >setup.py when I should be calling ez_setup? And what should I pass to > >ez_setup? It does look like setuptools are installed incorrectly since > >easy_install -h does not run. So what is the right way? > > If you are packaging setuptools for use with a system packager, you should > use: > > python setup.py install --root=/some/pseudoroot > > If you are using setuptools 0.6a10 or earlier, you *also* need > --single-version-externally-managed for this to work. 0.6a11 (which I > just > released a few minutes ago) automatically sets > --single-version-externally-managed if you specify a --root, so you might > want to just go ahead and upgrade, especially since 0.6a11 has a lot of > other changes to improve compatibility with system packagers. > > For example, while 0.6a10 will complain about system-installed versions of > a package as conflicting with a package that is being installed, 0.6a11 > installs things such that there are no conflicts. 0.6a11 also supports > making system packages for projects containing namespace packages, without > causing inter-package conflicts for the packaging system. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20060329/560c287f/attachment-0001.htm From pje at telecommunity.com Thu Mar 30 08:39:00 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 30 Mar 2006 01:39:00 -0500 Subject: [Distutils] setuptools in fink In-Reply-To: <37de72610603291933k703fddceqce194c4e35e1bd55@mail.gmail.co m> References: <5.1.1.6.0.20060329190018.01fa9830@mail.telecommunity.com> <5.1.1.6.0.20060329124158.0390efc8@mail.telecommunity.com> <5.1.1.6.0.20060329190018.01fa9830@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060330013620.01e24a30@mail.telecommunity.com> At 10:33 PM 3/29/2006 -0500, Kurt Schwehr wrote: >I think I have it figured out. Fink requires installing a package in a >temporary root install that is then turned into a deb. This makes the pth >files have bad paths. And if you use --single-version-externally-managed as well as --root, then it will not have bad paths. Also, 0.6a11 uses relative paths, and in any case defaults to using --single-version-externally-managed when you specify --root. >If I manage the paths in the pth files after the install, then things seem >to work. It looks like i will have to manage the easy_install.pth anyway >since if it ends up in a deb, then other packages can't work with it. I'm >going to create PostInstall, PostRm commands to take care of that. Don't go to all that trouble, it's not necessary. Just use 0.6a11 with "install --root" and you won't have to worry about any of this. If you run into any problems with it, let me know, as 0.6a11 is supposed to "Just Work" for system packagers using --root. If it doesn't, it's my fault, so tell me and I'll fix it. :) From schwehr at gmail.com Thu Mar 30 14:47:51 2006 From: schwehr at gmail.com (Kurt Schwehr) Date: Thu, 30 Mar 2006 07:47:51 -0500 Subject: [Distutils] setuptools in fink In-Reply-To: <5.1.1.6.0.20060330013620.01e24a30@mail.telecommunity.com> References: <5.1.1.6.0.20060329124158.0390efc8@mail.telecommunity.com> <5.1.1.6.0.20060329190018.01fa9830@mail.telecommunity.com> <5.1.1.6.0.20060330013620.01e24a30@mail.telecommunity.com> Message-ID: <37de72610603300447s27d630d6v106474e25388187e@mail.gmail.com> Nice! Thanks much for the help. The packages are all set. -kurt On 3/30/06, Phillip J. Eby wrote: > > > And if you use --single-version-externally-managed as well as --root, then > it will not have bad paths. Also, 0.6a11 uses relative paths, and in any > case defaults to using --single-version-externally-managed when you > specify > --root. Don't go to all that trouble, it's not necessary. Just use 0.6a11 with > "install --root" and you won't have to worry about any of this. If you > run > into any problems with it, let me know, as 0.6a11 is supposed to "Just > Work" for system packagers using --root. If it doesn't, it's my fault, so > tell me and I'll fix it. :) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20060330/9364c443/attachment.htm From natalie.l.doe at gsk.com Thu Mar 30 19:45:29 2006 From: natalie.l.doe at gsk.com (natalie.l.doe at gsk.com) Date: Thu, 30 Mar 2006 12:45:29 -0500 Subject: [Distutils] PyGraphViz Message-ID: Does anyone have any experience using PyGraphViz or GraphViz on windows with python? We are having some trouble with compiling and wonder if anyone has had previous experience with this. Beyond just the compilation does anyone have any experience using PyGraphVix with PyQt? We are interested in the level of interactivity that can be achieved with the combination of the packages. Please e-mail back directly. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20060330/5afbe47a/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 12313 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060330/5afbe47a/attachment.jpe From pje at telecommunity.com Fri Mar 31 05:03:20 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 30 Mar 2006 22:03:20 -0500 Subject: [Distutils] Breakage of fetch in setuptools-0.6a10-py2.4.egg In-Reply-To: <20060331015534.34450.qmail@web54501.mail.yahoo.com> References: <5.1.1.6.0.20060325215905.02057708@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060330215814.01dedd30@mail.telecommunity.com> At 05:55 PM 3/30/2006 -0800, Grig Gheorghiu wrote: >Can I do it with a setuptools-aware setup.py installation file? > >I ran easy_install with the options you mentioned and it created >setuptools-0.6a11dev_r43297-py2.4.egg. I then tried to require this >version in setup.py like this: > > install_requires = ["setuptools==0.6a11dev_r43297", > ], A better way to do this is to do: from ez_setup import use_setuptools use_setuptools() And this will ensure that the user updates to 0.6a11 before running. See these links for more info: http://peak.telecommunity.com/DevCenter/setuptools#using-setuptools-without-bundling-it http://peak.telecommunity.com/DevCenter/setuptools#managing-multiple-projects