From python at jwp.name Tue Apr 1 02:24:49 2008 From: python at jwp.name (James William Pye) Date: Mon, 31 Mar 2008 17:24:49 -0700 Subject: [Distutils] setup.py develop un-egg-spected result In-Reply-To: <20080325172228.B1F843A4074@sparrow.telecommunity.com> References: <20080325103931.o6t2dvs7pxhpckw8@webmail.utoronto.ca> <20080325151819.199753A40FF@sparrow.telecommunity.com> <20080325120721.l7065o2nznwoo84s@webmail.utoronto.ca> <20080325164235.4C9D13A4074@sparrow.telecommunity.com> <20080325125203.iaa6yzdanon48swk@webmail.utoronto.ca> <20080325172228.B1F843A4074@sparrow.telecommunity.com> Message-ID: <20080401002449.GA30016@lit.jwp.name> On Tue, Mar 25, 2008 at 01:22:28PM -0400, Phillip J. Eby wrote: > If we were replacing the distutils, I'd simply have a "source root" > option, rather than a dictionary with a magic key. FWIW, +1 on the "source root" option. package_dir has always seemed rather confusing to me. I used it for a source root at one time, but I just use packages now with a physical layout. It's easier to understand what's going on(not just for my sake, but others as well). From mhammond at skippinet.com.au Tue Apr 1 11:45:57 2008 From: mhammond at skippinet.com.au (Mark Hammond) Date: Tue, 1 Apr 2008 20:45:57 +1100 Subject: [Distutils] [Python-Dev] FW: [issue2513] 64bit cross compilation on windows In-Reply-To: <002401c892ff$a8254630$f86fd290$@com.au> References: <002401c892ff$a8254630$f86fd290$@com.au> Message-ID: <00c801c893dd$32d03320$98709960$@com.au> I wrote: > FYI, I've uploaded a patch that provides for cross-compilation on > Windows between 32 and 64 bit platforms - all comments invited! While I have some people's attention I'd like to re-raise another issue I foresee for x64 builds. I've mentioned this over the last couple of months, but haven't got much of a response, so this seems like a reasonable opportunity to try one more time :) Currently, the "official" (by way of being de-facto) directory structure for a build tree is 'PCBuild/.' for x86 builds and 'PCBuild/amd64' for x64 platforms. I believe this might cause problems for people trying to port their applications to 64bit platforms. My proposal is that we change this subtly - that both 'PCBuild/x86', and 'PCBuild/amd64' exist, while 'PCBuild/.' is always the 'native' platform - ie, that 'PCBuild/.' be a copy of one of the platform subdirectories. The rationale is fairly simple: I'm quite certain that this new directory layout will break existing "native only" build processes (ie, those that aren't aware of cross-compilation). I'm quite certain that Mozilla will break, for example, and no cross-compilation process exists that I am aware of. Existing build tools already know about the PCBuild directory, and are focused almost exclusively towards native compilation - all such tools on non-x86 platforms are currently broken. My proposal would help avoid breakage for existing build processes or tools that try to natively target x64, and give a reasonable story for build processes that explicitly support cross-compilation. I believe the costs are fairly small - disk space is cheap and the extra complexity in the VS project files should be reasonable. What do people think? I think it's time to take the approach of "silent ascent", so unless I hear objections I'll upload a new patch which implements this and also takes it into account for Distutils builds. Cheers, Mark From anton at headnet.dk Tue Apr 1 17:07:41 2008 From: anton at headnet.dk (Anton Stonor) Date: Tue, 01 Apr 2008 17:07:41 +0200 Subject: [Distutils] Can't generate console script with buildout - even when using zc.recipe.egg Message-ID: Hi there, I was hoping a buildout guru could give a hand on this one: The "RelStorage" package comes as an egg and installing it in a buildout environment works fine. However I can't find a way to let buildout generate the zodbconvert console script that comes with RelStorage. easy_install RelStorage adds the script fine. I know you should use zc.recipe.egg in this case, but no result. The RelStorage setup.py comes with this: scripts=['scripts/zodbconvert.py'] but no "console_scripts" entry points (maybe that is needed?) I found a kind-of workaround: entry-points = zodbconvert=zodbconvert:main extra-paths = /home/stonor/projects/mybuildout/eggs/RelStorage-1.0.1-py2.4.egg/EGG-INFO/scripts But it is annoying to hard code the path to RelStorage-1.0.1-py2.4.egg/EGG-INFO/scripts. Is there a way to dynamically look up the path to an egg with bulidout or a another way to solve my issue? /Anton Stonor From jim at zope.com Tue Apr 1 18:45:34 2008 From: jim at zope.com (Jim Fulton) Date: Tue, 1 Apr 2008 12:45:34 -0400 Subject: [Distutils] Can't generate console script with buildout - even when using zc.recipe.egg In-Reply-To: References: Message-ID: <195B1D75-B453-4B1E-9485-354FFBDB27BB@zope.com> On Apr 1, 2008, at 11:07 AM, Anton Stonor wrote: > Hi there, > > I was hoping a buildout guru could give a hand on this one: > > The "RelStorage" package comes as an egg and installing it in a > buildout > environment works fine. > > However I can't find a way to let buildout generate the zodbconvert > console script that comes with RelStorage. > > easy_install RelStorage adds the script fine. > > I know you should use zc.recipe.egg in this case, but no result. > > The RelStorage setup.py comes with this: > > scripts=['scripts/zodbconvert.py'] > > but no "console_scripts" entry points (maybe that is needed?) Yes, buildout needs that. buildout only installs scripts declared as entry points. As you discovered, you can specify the entry points yourself. Buildout also needs the entry point to be importable from the egg. Unfortunately, old-style distutils script handling doesn't put the scripts in a place where they are importable. > > > I found a kind-of workaround: > > entry-points = zodbconvert=zodbconvert:main > extra-paths = > /home/stonor/projects/mybuildout/eggs/RelStorage-1.0.1-py2.4.egg/EGG- > INFO/scripts > > But it is annoying to hard code the path to > RelStorage-1.0.1-py2.4.egg/EGG-INFO/scripts. > > Is there a way to dynamically look up the path to an egg with bulidout > or a another way to solve my issue? No, because the scripts aren't included in the RelStorage Python package and aren't in the egg path. You should get Shane to move this script into a module that can be imported from the egg. This is what I did with the ZODB scripts a while back. Perhaps buildout should support the traditional script-installation mechanism, but that mechanism assumes that the modules needed by the script are installed into site packages and that's not a model that buildout provides. Virtualenv supports this use case because it emulates a Python install. Jim -- Jim Fulton Zope Corporation From anton at headnet.dk Tue Apr 1 21:28:38 2008 From: anton at headnet.dk (Anton Stonor) Date: Tue, 01 Apr 2008 21:28:38 +0200 Subject: [Distutils] Can't generate console script with buildout - even when using zc.recipe.egg In-Reply-To: <195B1D75-B453-4B1E-9485-354FFBDB27BB@zope.com> References: <195B1D75-B453-4B1E-9485-354FFBDB27BB@zope.com> Message-ID: Jim Fulton wrote: >> The RelStorage setup.py comes with this: >> >> scripts=['scripts/zodbconvert.py'] >> >> but no "console_scripts" entry points (maybe that is needed?) > > Yes, buildout needs that. buildout only installs scripts declared as > entry points. As you discovered, you can specify the entry points > yourself. Buildout also needs the entry point to be importable from > the egg. Unfortunately, old-style distutils script handling doesn't > put the scripts in a place where they are importable. Thanks for a quick and precise answer. I'll poke Shane. Anton From martin at v.loewis.de Wed Apr 2 01:12:35 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 02 Apr 2008 01:12:35 +0200 Subject: [Distutils] [Python-Dev] FW: [issue2513] 64bit cross compilation on windows In-Reply-To: <00c801c893dd$32d03320$98709960$@com.au> References: <002401c892ff$a8254630$f86fd290$@com.au> <00c801c893dd$32d03320$98709960$@com.au> Message-ID: <47F2C163.9000402@v.loewis.de> > Currently, the "official" (by way of being de-facto) directory structure for > a build tree is 'PCBuild/.' for x86 builds and 'PCBuild/amd64' for x64 > platforms. I believe this might cause problems for people trying to port > their applications to 64bit platforms. My proposal is that we change this > subtly - that both 'PCBuild/x86', and 'PCBuild/amd64' exist, while > 'PCBuild/.' is always the 'native' platform - ie, that 'PCBuild/.' be a copy > of one of the platform subdirectories. Can you explain what you mean by "native platform"? I usually perform cross-builds for amd64, so would then this directory remain empty? > The rationale is fairly simple: I'm quite certain that this new directory > layout will break existing "native only" build processes (ie, those that > aren't aware of cross-compilation). I'm quite certain that Mozilla will > break, for example, and no cross-compilation process exists that I am aware > of. Existing build tools already know about the PCBuild directory, and are > focused almost exclusively towards native compilation - all such tools on > non-x86 platforms are currently broken. The reverse may also be true: some tools may expect PCbuild to contain always x86 binaries, even on AMD64 - as they don't support non-x86 at all. Those tools might break if PCbuild suddenly starts containing AMD64 binaries. If this is all about *just* Mozilla, then I'd be much more in favour than if it was for some theoretical package. So if Mozilla builds with that kind of setup, and works correctly - go for it. If you have to fix Mozilla anyway, consider fixing it so it looks in the amd64 directory. Regards, Martin From mhammond at skippinet.com.au Wed Apr 2 03:26:22 2008 From: mhammond at skippinet.com.au (Mark Hammond) Date: Wed, 2 Apr 2008 12:26:22 +1100 Subject: [Distutils] [Python-Dev] FW: [issue2513] 64bit cross compilation on windows In-Reply-To: <47F2C163.9000402@v.loewis.de> References: <002401c892ff$a8254630$f86fd290$@com.au> <00c801c893dd$32d03320$98709960$@com.au> <47F2C163.9000402@v.loewis.de> Message-ID: <016001c89460$944f10c0$bced3240$@com.au> Martin quoting me: > > Currently, the "official" (by way of being de-facto) directory > > structure for a build tree is 'PCBuild/.' for x86 builds and > > 'PCBuild/amd64' for x64 > > platforms. I believe this might cause problems for people trying to > > port their applications to 64bit platforms. My proposal is that we > > change this subtly - that both 'PCBuild/x86', and 'PCBuild/amd64' > > exist, while 'PCBuild/.' is always the 'native' platform - ie, > > that 'PCBuild/.' be a copy of one of the platform subdirectories. > > Can you explain what you mean by "native platform"? I usually perform > cross-builds for amd64, so would then this directory remain empty? By "native platform", I mean it contains binaries that are native for the current platform. IOW, on x86 platforms, PCBuild would contain x86 binaries, and binaries compiled for AMD64 would live in the AMD64 directory. If in a given directory tree, you *only* did a cross-compile, then yes, PCBuild would remain empty - but I don't think that is a problem; tools that know how to cross-compile (such as the distutils after my patch is applied) will not look in the PCBuild directory, and tools that do not cross-compile will be unable to use the x64 binaries from the x86 platform anyway. In other words, I can't think of anything that would break by having an empty PCBuild directory in a tree that only contains binaries that can't be executed on the current platform. > > The rationale is fairly simple: I'm quite certain that this new > > directory layout will break existing "native only" build processes > > (ie, those that aren't aware of cross-compilation). I'm quite > > certain that Mozilla will break, for example, and no > > cross-compilation process exists that I am aware > > of. Existing build tools already know about the PCBuild directory, > > and are focused almost exclusively towards native compilation - all > > such tools on non-x86 platforms are currently broken. > > The reverse may also be true: some tools may expect PCbuild to contain > always x86 binaries, even on AMD64 - as they don't support non-x86 at > all. Those tools might break if PCbuild suddenly starts containing > AMD64 binaries. I agree. However, it is my assertion that there are very few build tools which expect the layout you describe, simply as the scheme has only been around for a few months, and only in Python 2.6 builds. Further, I assert that there are a greater number of build tools which do not support cross-compilation, but will build natively on x64 and expect 'PCBuild' to have libraries they can link with to create an x64 binary. I accept my assertions may be incorrect - in which case keeping the existing layout is indeed the best options. But I assert that my assertions are correct :) > If this is all about *just* Mozilla, then I'd be much more in favour > than if it was for some theoretical package. So if Mozilla builds with > that kind of setup, and works correctly - go for it. If you have to > fix Mozilla anyway, consider fixing it so it looks in the amd64 > directory. Nope - it's not about *just* Mozilla at all, and I don't expect that I personally will be involved in any 64bit issues for that platform. That just happens to be one build process that I know about, but I except it does some "typical" things other tools might do - like executing the python executable to sniff sys.prefix and looking for (eg) "Libs" and "PCBuild" to locate libraries to link with seems fairly likely to me. Even looking for the executable itself directly in PCBuild seems likely. On an x64 platform, such tools may well find the 32bit version, and it may even seem to work - but it is unlikely to be doing what they expect. Or they may find no binary at all. It seems that you previously agreed with this. From http://mail.python.org/pipermail/distutils-sig/2007-May/007668.html; you are replying to (I think) Kristj?n: | > I am baffled about why the build environment's layout matters, | > but once an .msi install can place the binaries in any | > old place it wants. The build structure doesn't have to | > reflect the final installed structure at all. | No. But still, people like to be able to "run" Python out of | a source check-out. This has been supported for a long time, | and more and more stuff was added to support it. For examples | within Python itself, see the support in distutils, getpathp.c, | PCbuild/rt.bat, and Tools/buildbot/*.bat. Reportedly (by | Mark), building Mozilla (the web browser) also "knows" | about PCbuild. Your comments exactly reflect my concern, and I believe them relevant for people working with native binaries on x64. But as implied above, I actually have zero personal interest in this at the moment - unlike the distutils cross-compile patch, which does scratch an itch of mine. It just seems it would make life easier for people down the track - but I'm happy to let it go if your position has changed or I have mis-understood it. Thanks, Mark From martin at v.loewis.de Wed Apr 2 04:15:22 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 02 Apr 2008 04:15:22 +0200 Subject: [Distutils] [Python-Dev] FW: [issue2513] 64bit cross compilation on windows In-Reply-To: <016001c89460$944f10c0$bced3240$@com.au> References: <002401c892ff$a8254630$f86fd290$@com.au> <00c801c893dd$32d03320$98709960$@com.au> <47F2C163.9000402@v.loewis.de> <016001c89460$944f10c0$bced3240$@com.au> Message-ID: <47F2EC3A.9090300@v.loewis.de> >> The reverse may also be true: some tools may expect PCbuild to contain >> always x86 binaries, even on AMD64 - as they don't support non-x86 at >> all. Those tools might break if PCbuild suddenly starts containing >> AMD64 binaries. > > I agree. However, it is my assertion that there are very few build tools > which expect the layout you describe, simply as the scheme has only been > around for a few months, and only in Python 2.6 builds. That PCbuild exists and contains x86 binaries? This scheme has been in place ever since the PCbuild directory got created, several years ago! > Nope - it's not about *just* Mozilla at all, and I don't expect that I > personally will be involved in any 64bit issues for that platform. That > just happens to be one build process that I know about, but I except it does > some "typical" things other tools might do - like executing the python > executable to sniff sys.prefix and looking for (eg) "Libs" and "PCBuild" to > locate libraries to link with seems fairly likely to me. Even looking for > the executable itself directly in PCBuild seems likely. On an x64 platform, > such tools may well find the 32bit version, and it may even seem to work - > but it is unlikely to be doing what they expect. Unless the *want* the x86 binaries, of course, which always had been in this place. > Your comments exactly reflect my concern, and I believe them relevant for > people working with native binaries on x64. No, I never cared about what binaries are "native". Instead, I wanted to see those binaries in PCbuild which had been built last. If you just built the x86 binaries, PCbuild should contain x86 binaries, even if you were running Win64. This is different from what you propose, but only slightly so (but perhaps importantly, depending on the scenario). > But as implied above, I > actually have zero personal interest in this at the moment - unlike the > distutils cross-compile patch, which does scratch an itch of mine. It just > seems it would make life easier for people down the track - but I'm happy to > let it go if your position has changed or I have mis-understood it. OK, if we don't have an actual use case, I suggest to leave things as is. People running into this problem should propose solutions themselves. Regards, Martin From dpeterson at enthought.com Wed Apr 2 06:30:17 2008 From: dpeterson at enthought.com (Dave Peterson) Date: Tue, 01 Apr 2008 23:30:17 -0500 Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and beyond In-Reply-To: <014c01c88e2e$e4aad5a0$ae0080e0$@com.au> References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com> <47E40726.1090209@egenix.com> <20080321212127.791B63A4074@sparrow.telecommunity.com> <47E4330C.5070601@egenix.com> <20080321224110.657C63A40A5@sparrow.telecommunity.com> <47E46176.4090205@v.loewis.de> <20080322020447.3B1033A4074@sparrow.telecommunity.com> <47E4EE9D.2090605@v.loewis.de> <20080322151918.D9EBC3A40B1@sparrow.telecommunity.com> <47E525F5.40705@v.loewis.de> <20080322154449.BDA423A40A5@sparrow.telecommunity.com> <47E52F70.1010806@v.loewis.de> <47E8276B.8030800@palladion.com> <014c01c88e2e$e4aad5a0$ae0080e0$@com.au> Message-ID: <47F30BD9.9080605@enthought.com> Mark Hammond wrote: >> (Note: I'm aware that people believe it to be necessary to munge the >> Windows registry when installing Python packages; I just don't agree >> with the practice, and don't think we should distort Python's process >> to coddle it). >> > > Whoever thinks it necessary is misguided. Installing a package doesn't > require munging the registry and none of the popular installation techniques > do. Installing Python itself does, and some packages have special > requirements (eg, pywin32 registering COM objects), but in general, Python > packages on Windows can ignore the registry. > I guess I'll step up and volunteer myself as 'misguided' then. :-) I would very much like to be able to package up my application or library such that it created desktop/start menu/quicklaunch icons for various things like running the app, running examples, opening docs for browsing, etc. I'd also like to be able to associate certain file types with executables/scripts/shortcuts provided by the installation of my project. After all, Windows users in general are not as technically adapt when it comes to command line tools so setting up these nice GUI ways of doing things adds usability significantly. You could argue (like Phillip has) that these operations should be done via an explicitly user-invoked script, and I can buy that for the standard version of the tool. You could also argue that installing applications (which is generally where these kinds of desires come into play) is not the job of the tools we're discussing. But it seems to me that the existing capabilities of setuptools are 80% (or more) of the effort in creating a tool that would allow installation and efficient maintenance of large Python-based applications, such as what my employer delivers to customers. All that being said, I'm fine with the idea that the standard library version of the tool does not enable this. Just so long as nothing is done to actively prevent extensions of that tool to do this sort of thing for those who need or want it. -- Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080401/cc6aa0b3/attachment-0001.htm From dpeterson at enthought.com Wed Apr 2 06:40:20 2008 From: dpeterson at enthought.com (Dave Peterson) Date: Tue, 01 Apr 2008 23:40:20 -0500 Subject: [Distutils] setuptools bug report: _get_unpatched() is way too clever (and also returns the wrong module) In-Reply-To: <3219B2C8-AD42-4C74-8ECE-714EB66B63F8@zooko.com> References: <3219B2C8-AD42-4C74-8ECE-714EB66B63F8@zooko.com> Message-ID: <47F30E34.3080204@enthought.com> zooko wrote: > I would be willing to host a trac instance to serve as a wiki and > issue tracker for setuptools, and a buildbot. (But I don't want to > have anything to do with subversion -- I much prefer darcs.) > I've previously volunteered Enthought to do about the same thing (except to include svn with the Trac instance) but there wasn't much response beyond Jeff Rush saying he was already setting one up, and the response by Phillip to use the python dev bug-tracker and svn instead. (No one previously talked about buildbot but that makes great sense.) I haven't seen any follow-up on those discussions though. Did I miss it? ;-) I'd really like to get this resolved, along with the issues of who can commit to the codebase, so that more people can start working on some of this stuff being discussed. Phillip, I think everyone is waiting for you to provide direction so can you make some decisions about how setuptools evolves to a larger community effort? -- Dave From pje at telecommunity.com Wed Apr 2 08:58:22 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 02 Apr 2008 02:58:22 -0400 Subject: [Distutils] setuptools bug report: _get_unpatched() is way too clever (and also returns the wrong module) In-Reply-To: <47F30E34.3080204@enthought.com> References: <3219B2C8-AD42-4C74-8ECE-714EB66B63F8@zooko.com> <47F30E34.3080204@enthought.com> Message-ID: <20080402065817.5A90F3A40AE@sparrow.telecommunity.com> At 11:40 PM 4/1/2008 -0500, Dave Peterson wrote: >I've previously volunteered Enthought to do about the same thing (except >to include svn with the Trac instance) but there wasn't much response >beyond Jeff Rush saying he was already setting one up, and the response >by Phillip to use the python dev bug-tracker and svn instead. ... >Phillip, I think everyone is waiting for you to provide direction so can >you make some decisions about how setuptools evolves to a larger >community effort? As I believe you just pointed out, I already did that. :) From tnelson at onresolve.com Wed Apr 2 09:37:14 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Wed, 2 Apr 2008 00:37:14 -0700 Subject: [Distutils] [Python-Dev] FW: [issue2513] 64bit cross compilation on windows In-Reply-To: <016001c89460$944f10c0$bced3240$@com.au> References: <002401c892ff$a8254630$f86fd290$@com.au> <00c801c893dd$32d03320$98709960$@com.au> <47F2C163.9000402@v.loewis.de> <016001c89460$944f10c0$bced3240$@com.au> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E381BA@EXMBX04.exchhosting.com> > Further, I > assert that there are a greater number of build tools which do not support > cross-compilation, but will build natively on x64 and expect 'PCBuild' > to have libraries they can link with to create an x64 binary. I'm with Martin on this one as well I think. If I understand correctly, you're proposing: PCbuild - may have contents of x86 or x64 depending on the build machine's architecture PCbuild/amd64 - always x64 PCbuild/x86 - always x86 And what we've currently got is: PCbuild/ - always x86 PCbuild/amd64 - always x64 I think this is fine; we don't really have a notion of compiling for a native platform, nor is the build machine's architecture factored into the equation. I believe this is a Good Thing. If you want a 32-bit build, use the 'Win32' target platform in VS; if you want a 64-bit build, use 'x64'. Trent. From gael.varoquaux at normalesup.org Wed Apr 2 11:31:35 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 2 Apr 2008 11:31:35 +0200 Subject: [Distutils] Making setuptools play nicer with virtualenv Message-ID: <20080402093135.GF11334@phare.normalesup.org> Hi, I have just out-clevered myself using setuptools and virtualenv: * install foo using "python setup.py develop" (foo being ipython). * download some module bar you want to work on in an isolated environment * create this isolated environment using "virtualenv bar" * in the isolated environment "python setup.py develop" the bar module. * still in the isolated environment, try to import bar in a script installed by foo (aka ipython) --> fails because foo uses the system python, and virtualenv wants you to use its own python One very easy solution to make this work is to have the setuptools generated scripts use, under unices, "#!/usr/bin/env python" rather than "#!/usr/bin/python". This seems to me like a good solution, in general, to follow the user's expectations. Is this a change that would be possible? Cheers, Ga?l From gael.varoquaux at normalesup.org Wed Apr 2 11:43:09 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 2 Apr 2008 11:43:09 +0200 Subject: [Distutils] Making setuptools play nicer with virtualenv Message-ID: <20080402094309.GH11334@phare.normalesup.org> Hi, I have just out-clevered myself using setuptools and virtualenv: * install foo using "python setup.py develop" (foo being ipython). * download some module bar you want to work on in an isolated environment * create this isolated environment using "virtualenv bar" * in the isolated environment "python setup.py develop" the bar module. * still in the isolated environment, try to import bar in a script installed by foo (aka ipython) --> fails because foo uses the system python, and virtualenv wants you to use its own python One very easy solution to make this work is to have the setuptools generated scripts use, under unices, "#!/usr/bin/env python" rather than "#!/usr/bin/python". This seems to me like a good solution, in general, to follow the user's expectations. Is this a change that would be possible? Cheers, Ga?l From lxander.m at gmail.com Wed Apr 2 13:23:41 2008 From: lxander.m at gmail.com (Alexander Michael) Date: Wed, 2 Apr 2008 07:23:41 -0400 Subject: [Distutils] Making setuptools play nicer with virtualenv In-Reply-To: <20080402093135.GF11334@phare.normalesup.org> References: <20080402093135.GF11334@phare.normalesup.org> Message-ID: <525f23e80804020423wae6cf2cnf7b0695993bff653@mail.gmail.com> On Wed, Apr 2, 2008 at 5:31 AM, Gael Varoquaux wrote: > Hi, > > I have just out-clevered myself using setuptools and virtualenv: > > * install foo using "python setup.py develop" (foo being ipython). > > * download some module bar you want to work on in an isolated environment > > * create this isolated environment using "virtualenv bar" > > * in the isolated environment "python setup.py develop" the bar module. > > * still in the isolated environment, try to import bar in a script > installed by foo (aka ipython) > > --> fails because foo uses the system python, and virtualenv wants > you to use its own python Currently, virtualenv only inherits libraries (i.e. site-packages), not scripts (i.e. entry points in bin or Scripts). I believe the idea is that you want to create a semi-isolated application environment of a sort that only provides the targeted application. This should be documented more explicitly. > One very easy solution to make this work is to have the setuptools > generated scripts use, under unices, "#!/usr/bin/env python" rather than > "#!/usr/bin/python". This seems to me like a good solution, in general, > to follow the user's expectations. > > Is this a change that would be possible? This won't work. One of the cool things about virtualenv is that you don't actually need to "activate" the environment to use applications/scripts it provides. The correct solution is to create a custom bootstrap that either installs ipython, or copies the entry point wrapper from the parent environment to the virtualenv, replacing the shebang line with the path to the virtualenv provided interpreter. Scripts are versioned entry points assigned to a particular environment. As far as modifying setuptools to install scripts with the suggested shebang line, I used to patch my local setuptools to do just this but have learned the hard way the wisdom of being explicit here after users installed their own python and then complained that the site scripts broke. It also makes things less fragile when you want to start transitioning to newer python releases along-side the current (or system) python. HTH, Alex From ianb at colorstudy.com Wed Apr 2 18:06:04 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 02 Apr 2008 11:06:04 -0500 Subject: [Distutils] Making setuptools play nicer with virtualenv In-Reply-To: <20080402093135.GF11334@phare.normalesup.org> References: <20080402093135.GF11334@phare.normalesup.org> Message-ID: <47F3AEEC.7080404@colorstudy.com> Gael Varoquaux wrote: > I have just out-clevered myself using setuptools and virtualenv: > > * install foo using "python setup.py develop" (foo being ipython). > > * download some module bar you want to work on in an isolated environment > > * create this isolated environment using "virtualenv bar" > > * in the isolated environment "python setup.py develop" the bar module. > > * still in the isolated environment, try to import bar in a script > installed by foo (aka ipython) > > --> fails because foo uses the system python, and virtualenv wants > you to use its own python > > One very easy solution to make this work is to have the setuptools > generated scripts use, under unices, "#!/usr/bin/env python" rather than > "#!/usr/bin/python". This seems to me like a good solution, in general, > to follow the user's expectations. > > Is this a change that would be possible? Sometimes you want to inherit the environment you've activated, but in my experience usually this isn't what you'll want. I find it easier to just reinstall any tools (like ipython, nose, etc) that I want to use in the virtualenv. In an ideal situation they could share eggs with the system packages, but this only kind of works. (Sometimes, for reasons I don't always understand, easy_install will find and install globally-installed packages, creating an executable bound to the virtualenv.) Ian From dpeterson at enthought.com Wed Apr 2 21:11:28 2008 From: dpeterson at enthought.com (Dave Peterson) Date: Wed, 02 Apr 2008 14:11:28 -0500 Subject: [Distutils] Setting up a setuptools community In-Reply-To: <20080402065817.5A90F3A40AE@sparrow.telecommunity.com> References: <3219B2C8-AD42-4C74-8ECE-714EB66B63F8@zooko.com> <47F30E34.3080204@enthought.com> <20080402065817.5A90F3A40AE@sparrow.telecommunity.com> Message-ID: <47F3DA60.3040108@enthought.com> Phillip J. Eby wrote: > At 11:40 PM 4/1/2008 -0500, Dave Peterson wrote: >> I've previously volunteered Enthought to do about the same thing (except >> to include svn with the Trac instance) but there wasn't much response >> beyond Jeff Rush saying he was already setting one up, and the response >> by Phillip to use the python dev bug-tracker and svn instead. > ... > >> Phillip, I think everyone is waiting for you to provide direction so can >> you make some decisions about how setuptools evolves to a larger >> community effort? > > As I believe you just pointed out, I already did that. :) Egggcellent. Doing a search for 'setuptools' in the issue tracker only returns two open issues currently. I'm guessing I'm not catching all the issues as a result. It would probably help if we had a clear component to use (i.e. Setuptools). Anyone have any idea who to contact to get that done? I've so far had no luck reading the info on www.python.org/dev looking for this sort of thing. Phillip, I don't believe you responded to either my, nor Jeff Rush's, volunteering for being patch reviewers. At least I haven't seen any response -- perhaps that's you trying to tell me something? :-) What's the process you want to follow on this? Ideally things should be opened up where more people are 'blessed' to commit directly, but I'm not sure how you want to do the 'bless'ing part. Finally, what is the build and test process you've been following? Is there a buildbot that's building and publishing results somewhere or does that also need to be set up? -- Dave From plone at hannosch.info Wed Apr 2 21:44:27 2008 From: plone at hannosch.info (Hanno Schlichting) Date: Wed, 02 Apr 2008 21:44:27 +0200 Subject: [Distutils] [zc.buildout] upgrade with changed explicit recipe version Message-ID: Hi. I ran into a problem with zc.buildout while upgrading a customers buildout and wondered if the problem I encountered is a general one. The buildout in question had one recipe pinned down to an exact version as in: [buildout] parts = plone [plone] recipe = plone.recipe.plone==3.0.2 To upgrade this environment I changed the explicit version to 3.0.6: [buildout] parts = plone [plone] recipe = plone.recipe.plone==3.0.6 When rerunning the buildout script I got an error during uninstall of the plone part. It complained that the plone.recipe.plone recipe version 3.0.2 was not available. From my understanding of this, buildout parses the version requirements first and at the point where it wants to uninstall the plone part, only the newer 3.0.6 version of the recipe is available. The workaround was to change .installed.cfg and remove the version restriction, so any version of the recipe would be accepted to uninstall the part, but I wonder if there is a general problem here or if we are misusing the version restrictions? I have the feeling that buildout should either add the old recipe to the environment to satisfy the uninstall requirement or accept newer versions of a recipe to uninstall older versions. Thoughts? Hanno From jim at zope.com Wed Apr 2 21:54:58 2008 From: jim at zope.com (Jim Fulton) Date: Wed, 2 Apr 2008 15:54:58 -0400 Subject: [Distutils] [zc.buildout] upgrade with changed explicit recipe version In-Reply-To: References: Message-ID: On Apr 2, 2008, at 3:44 PM, Hanno Schlichting wrote: > Hi. > > I ran into a problem with zc.buildout while upgrading a customers > buildout and wondered if the problem I encountered is a general one. > > The buildout in question had one recipe pinned down to an exact > version > as in: > > [buildout] > parts = > plone > > [plone] > recipe = plone.recipe.plone==3.0.2 > > To upgrade this environment I changed the explicit version to 3.0.6: > > [buildout] > parts = > plone > > [plone] > recipe = plone.recipe.plone==3.0.6 > > When rerunning the buildout script I got an error during uninstall of > the plone part. It complained that the plone.recipe.plone recipe > version > 3.0.2 was not available. > > From my understanding of this, buildout parses the version > requirements > first and at the point where it wants to uninstall the plone part, > only > the newer 3.0.6 version of the recipe is available. > > The workaround was to change .installed.cfg and remove the version > restriction, so any version of the recipe would be accepted to > uninstall > the part, but I wonder if there is a general problem here or if we are > misusing the version restrictions? I think there is a general problem and no, you aren't miss-using the the version restriction. > I have the feeling that buildout should either add the old recipe to > the > environment to satisfy the uninstall requirement Unfortunately, there can be only one version in the environment at a time. > or accept newer > versions of a recipe to uninstall older versions. That is probably better. Another option might be to uninstall in a separate process. (Just thinking out loud.) > Thoughts? I need to think about this. :) I haven't seen this situation before, although I should have expected it. I'll think about this and respond in a day or 2. If I forget, bug me, or create a launchpad issue. Jim -- Jim Fulton Zope Corporation From mhammond at skippinet.com.au Thu Apr 3 00:35:59 2008 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 3 Apr 2008 09:35:59 +1100 Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and beyond In-Reply-To: <47F30BD9.9080605@enthought.com> References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com> <47E40726.1090209@egenix.com> <20080321212127.791B63A4074@sparrow.telecommunity.com> <47E4330C.5070601@egenix.com> <20080321224110.657C63A40A5@sparrow.telecommunity.com> <47E46176.4090205@v.loewis.de> <20080322020447.3B1033A4074@sparrow.telecommunity.com> <47E4EE9D.2090605@v.loewis.de> <20080322151918.D9EBC3A40B1@sparrow.telecommunity.com> <47E525F5.40705@v.loewis.de> <20080322154449.BDA423A40A5@sparrow.telecommunity.com> <47E52F70.1010806@v.loewis.de> <47E8276B.8030800@palladion.com> <014c01c88e2e$e4aad5a0$ae0080e0$@com.au> <47F30BD9.9080605@enthought.com> Message-ID: <021001c89511$ef806190$ce8124b0$@com.au> Even installing shortcuts doesn't need to munge the registry! But regardless, you seem to be arguing that setuptools should morph into a full-blown, general purpose installer for python build apps, capable of doing all kinds of platform specific things which any app may desire - and while I don't agree with that, it wasn't what was being discussed at all. Writing and installing a Python extension does not need to munge the registry. Installing a general purpose application, written in Python or anything else, will require, in the general case, anything you could possible imagine. Mark From: Dave Peterson [mailto:dpeterson at enthought.com] Sent: Wednesday, 2 April 2008 3:30 PM Cc: Mark Hammond; distutils-sig Subject: Re: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and beyond Mark Hammond wrote: (Note: I'm aware that people believe it to be necessary to munge the Windows registry when installing Python packages; I just don't agree with the practice, and don't think we should distort Python's process to coddle it). Whoever thinks it necessary is misguided. Installing a package doesn't require munging the registry and none of the popular installation techniques do. Installing Python itself does, and some packages have special requirements (eg, pywin32 registering COM objects), but in general, Python packages on Windows can ignore the registry. I guess I'll step up and volunteer myself as 'misguided' then. :-) I would very much like to be able to package up my application or library such that it created desktop/start menu/quicklaunch icons for various things like running the app, running examples, opening docs for browsing, etc. I'd also like to be able to associate certain file types with executables/scripts/shortcuts provided by the installation of my project. After all, Windows users in general are not as technically adapt when it comes to command line tools so setting up these nice GUI ways of doing things adds usability significantly. You could argue (like Phillip has) that these operations should be done via an explicitly user-invoked script, and I can buy that for the standard version of the tool. You could also argue that installing applications (which is generally where these kinds of desires come into play) is not the job of the tools we're discussing. But it seems to me that the existing capabilities of setuptools are 80% (or more) of the effort in creating a tool that would allow installation and efficient maintenance of large Python-based applications, such as what my employer delivers to customers. All that being said, I'm fine with the idea that the standard library version of the tool does not enable this. Just so long as nothing is done to actively prevent extensions of that tool to do this sort of thing for those who need or want it. -- Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080403/42551350/attachment-0001.htm From mhammond at skippinet.com.au Thu Apr 3 00:50:10 2008 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 3 Apr 2008 09:50:10 +1100 Subject: [Distutils] [Python-Dev] FW: [issue2513] 64bit cross compilation on windows In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E381BA@EXMBX04.exchhosting.com> References: <002401c892ff$a8254630$f86fd290$@com.au> <00c801c893dd$32d03320$98709960$@com.au> <47F2C163.9000402@v.loewis.de> <016001c89460$944f10c0$bced3240$@com.au> <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E381BA@EXMBX04.exchhosting.com> Message-ID: <022b01c89513$ef278140$cd7683c0$@com.au> > I think this is fine; we don't really have a notion of compiling for a > native platform, nor is the build machine's architecture factored into > the equation. Actually, I think it is slightly. IIUC, the AMD64 build currently assumes it can execute x86 executables in various places. To fix this, the build process for each of the platforms would be slightly different. My proposal would allow (eg) 'PCBuild\python.exe' or 'PCBuild\make_versioninfo.exe' to specified in build processes, with the knowledge it will work, regardless of what platform it is being run on. For example, with my idea, you could write code such as: exe = os.path.join(os.getenv("PYTHON_HOME"), "PCBuild", "python.exe")) popen(exe, ...) and expect it to work on all platforms. If I understand correctly, I now need to write something like: if "AMD64" in sys.platform: exe = os.path.join(os.getenv("PYTHON_HOME"), "PCBuild", "amd64", "python.exe")) else if "32 bit" in sys.platform: exe = os.path.join(os.getenv("PYTHON_HOME"), "PCBuild", "python.exe")) else: # hrm - I obviously don?t know about new platforms yet... popen(exe, ...) In other words, I think there is real advantage to simple scripts knowing that './python' or './PCBuild/python.exe' will always be executable (and always give the "native" version if the platform does emulation) But I accept I'm waving appendages in the wind, and without a concrete use-case I will take Martin's advice and let it go. Cheers, Mark From dpeterson at enthought.com Thu Apr 3 01:32:13 2008 From: dpeterson at enthought.com (Dave Peterson) Date: Wed, 02 Apr 2008 18:32:13 -0500 Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and beyond In-Reply-To: <021001c89511$ef806190$ce8124b0$@com.au> References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com> <47E40726.1090209@egenix.com> <20080321212127.791B63A4074@sparrow.telecommunity.com> <47E4330C.5070601@egenix.com> <20080321224110.657C63A40A5@sparrow.telecommunity.com> <47E46176.4090205@v.loewis.de> <20080322020447.3B1033A4074@sparrow.telecommunity.com> <47E4EE9D.2090605@v.loewis.de> <20080322151918.D9EBC3A40B1@sparrow.telecommunity.com> <47E525F5.40705@v.loewis.de> <20080322154449.BDA423A40A5@sparrow.telecommunity.com> <47E52F70.1010806@v.loewis.de> <47E8276B.8030800@palladion.com> <014c01c88e2e$e4aad5a0$ae0080e0$@com.au> <47F30BD9.9080605@enthought.com> <021001c89511$ef806190$ce8124b0$@com.au> Message-ID: <47F4177D.8030705@enthought.com> Hi Mark, Shortcuts don't, but file associations to those shortcuts do (at least to the best of my understanding,) adding paths does, etc. I agree that a library (extensions or no) doesn't need to do these things. But anything targeted at an end-user on Windows has a reasonable chance of wanting them. Here I thought I was being pretty clear that I was only requesting that things not be done to explicitly make it hard (or impossible) for someone else to extend setuptools (or whatever replaces / refactors it) to do these things. -- Dave Mark Hammond wrote: > > Even installing shortcuts doesn't need to munge the registry! But > regardless, you seem to be arguing that setuptools should morph into a > full-blown, general purpose installer for python build apps, capable > of doing all kinds of platform specific things which any app may > desire -- and while I don't agree with that, it wasn't what was being > discussed at all. Writing and installing a Python extension does not > need to munge the registry. Installing a general purpose application, > written in Python or anything else, will require, in the general case, > anything you could possible imagine. > > > > Mark > > > > *From:* Dave Peterson [mailto:dpeterson at enthought.com] > *Sent:* Wednesday, 2 April 2008 3:30 PM > *Cc:* Mark Hammond; distutils-sig > *Subject:* Re: [Distutils] [Python-Dev] How we can get rid of eggs for > 2.6 and beyond > > > > Mark Hammond wrote: > > (Note: I'm aware that people believe it to be necessary to munge the > > Windows registry when installing Python packages; I just don't agree > > with the practice, and don't think we should distort Python's process > > to coddle it). > > > > > Whoever thinks it necessary is misguided. Installing a package doesn't > require munging the registry and none of the popular installation techniques > do. Installing Python itself does, and some packages have special > requirements (eg, pywin32 registering COM objects), but in general, Python > packages on Windows can ignore the registry. > > > > I guess I'll step up and volunteer myself as 'misguided' then. :-) > > I would very much like to be able to package up my application or > library such that it created desktop/start menu/quicklaunch icons for > various things like running the app, running examples, opening docs > for browsing, etc. I'd also like to be able to associate certain file > types with executables/scripts/shortcuts provided by the installation > of my project. After all, Windows users in general are not as > technically adapt when it comes to command line tools so setting up > these nice GUI ways of doing things adds usability significantly. > > You could argue (like Phillip has) that these operations should be > done via an explicitly user-invoked script, and I can buy that for the > standard version of the tool. You could also argue that installing > applications (which is generally where these kinds of desires come > into play) is not the job of the tools we're discussing. But it > seems to me that the existing capabilities of setuptools are 80% (or > more) of the effort in creating a tool that would allow installation > and efficient maintenance of large Python-based applications, such as > what my employer delivers to customers. > > All that being said, I'm fine with the idea that the standard library > version of the tool does not enable this. Just so long as nothing is > done to actively prevent extensions of that tool to do this sort of > thing for those who need or want it. > > > -- Dave > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080402/1c978521/attachment.htm From gael.varoquaux at normalesup.org Thu Apr 3 01:37:55 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 3 Apr 2008 01:37:55 +0200 Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and beyond In-Reply-To: <47F4177D.8030705@enthought.com> References: <47E4EE9D.2090605@v.loewis.de> <20080322151918.D9EBC3A40B1@sparrow.telecommunity.com> <47E525F5.40705@v.loewis.de> <20080322154449.BDA423A40A5@sparrow.telecommunity.com> <47E52F70.1010806@v.loewis.de> <47E8276B.8030800@palladion.com> <014c01c88e2e$e4aad5a0$ae0080e0$@com.au> <47F30BD9.9080605@enthought.com> <021001c89511$ef806190$ce8124b0$@com.au> <47F4177D.8030705@enthought.com> Message-ID: <20080402233755.GO4585@phare.normalesup.org> On Wed, Apr 02, 2008 at 06:32:13PM -0500, Dave Peterson wrote: > Shortcuts don't, but file associations to those shortcuts do (at least to > the best of my understanding,) adding paths does, etc. I agree that a > library (extensions or no) doesn't need to do these things. But anything > targeted at an end-user on Windows has a reasonable chance of wanting > them. > Here I thought I was being pretty clear that I was only requesting that > things not be done to explicitly make it hard (or impossible) for someone > else to extend setuptools (or whatever replaces / refactors it) to do > these things. I am +1 on this request. I am not a Windows users, but I would still like to see my applications appear in the menus when I ship them. And now free-desktop has a nice specification to do this, so it is technically possible. My 2 cents, Ga?l From martin at v.loewis.de Thu Apr 3 08:41:44 2008 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 03 Apr 2008 08:41:44 +0200 Subject: [Distutils] [Python-Dev] FW: [issue2513] 64bit cross compilation on windows In-Reply-To: <022b01c89513$ef278140$cd7683c0$@com.au> References: <002401c892ff$a8254630$f86fd290$@com.au> <00c801c893dd$32d03320$98709960$@com.au> <47F2C163.9000402@v.loewis.de> <016001c89460$944f10c0$bced3240$@com.au> <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E381BA@EXMBX04.exchhosting.com> <022b01c89513$ef278140$cd7683c0$@com.au> Message-ID: <47F47C28.9090203@v.loewis.de> > Actually, I think it is slightly. IIUC, the AMD64 build currently assumes > it can execute x86 executables in various places. To fix this, the build > process for each of the platforms would be slightly different. Why does that need fixing? The AMD64 build *can* execute x86 binaries, whether it is performed on Win32 or Win64. A problem only arises if the AMD64 build assumes that it can execute AMD64 binaries, and it is a cross-compilation. As a consequence, we must never execute the python.exe that we just built during the build process. > My proposal > would allow (eg) 'PCBuild\python.exe' or 'PCBuild\make_versioninfo.exe' to > specified in build processes, with the knowledge it will work, regardless of > what platform it is being run on. No to python.exe, yes to make_versioninfo.exe. make_versioninfo.exe is *always* an x86 binary, so it is always safe to execute on Windows. If you tried to execute PCBuild\python.exe, it would break my builds: I don't *have* x86 binaries in PCBuild when I cross-compile for AMD64. Instead, I've changed the build process to run HOST_PYTHON if that is set, and only fall back to PCBuild\python.exe if it isn't. Regards, Martin From jeff at taupro.com Thu Apr 3 08:41:21 2008 From: jeff at taupro.com (Jeff Rush) Date: Thu, 03 Apr 2008 01:41:21 -0500 Subject: [Distutils] Setting up a setuptools community In-Reply-To: <47F3DA60.3040108@enthought.com> References: <3219B2C8-AD42-4C74-8ECE-714EB66B63F8@zooko.com> <47F30E34.3080204@enthought.com> <20080402065817.5A90F3A40AE@sparrow.telecommunity.com> <47F3DA60.3040108@enthought.com> Message-ID: <47F47C11.4010003@taupro.com> Dave Peterson wrote: > Phillip J. Eby wrote: >> At 11:40 PM 4/1/2008 -0500, Dave Peterson wrote: >>> I've previously volunteered Enthought to do about the same thing (except >>> to include svn with the Trac instance) but there wasn't much response >>> beyond Jeff Rush saying he was already setting one up, and the response >>> by Phillip to use the python dev bug-tracker and svn instead. >> ... >> >>> Phillip, I think everyone is waiting for you to provide direction so can >>> you make some decisions about how setuptools evolves to a larger >>> community effort? >> As I believe you just pointed out, I already did that. :) > > Egggcellent. > > Doing a search for 'setuptools' in the issue tracker only returns two > open issues currently. I'm guessing I'm not catching all the issues as > a result. It would probably help if we had a clear component to use > (i.e. Setuptools). Anyone have any idea who to contact to get that > done? I've so far had no luck reading the info on www.python.org/dev > looking for this sort of thing. Hi Dave, At PyCon I asked the core team about using the Python tracker for setuptools but Brett Cannon said that no software outside of the Python core is currently using it and it wouldn't be appropriate for setuptools to do so. They were willing to give us our own instance of a tracker, similar to how they have for Jython. Martin v. L?wis has been in the process since PyCon of doing so, but is busy and apparently the Roundup software used is tricky to configure for this and,f for the email address, requires ISP intervention. The plan when this is done is to have a tracker at http://bugs.python.org/setuptools/ with an email address for submission/commenting of setuptools at bugs.python.org. Sorry for the delay - I didn't realize this would be such a big deal to get started. We could also start using Launchpad instantly, but I figured reaching concensus to do so, with Phillip and others more familiar with the Python tracker, would be difficult. -Jeff From lists at zopyx.com Thu Apr 3 16:56:56 2008 From: lists at zopyx.com (Andreas Jung) Date: Thu, 03 Apr 2008 16:56:56 +0200 Subject: [Distutils] [zc.buildout] buildout unable to parse 'simple' view Message-ID: <2430BB03575C506E8261F96A@[192.168.0.24]> We're running a local egg repository based on haufe.eggserver. Buildout won't find an egg called 'haufe.recipe.svnsetup' that is stored on the server. The server is configured properly within the find-links section. And I can see a successful HTTP request to the page in the logs. The underlying template generates the following HTML: .... haufe.recipe.subversion-0.1.0dev_r942-py2.4.egg
haufe.recipe.subversion-0.1.1dev_r997-py2.4.egg
haufe.recipe.svnbuild-0.1.1dev_r1057-py2.4.egg
... I wonder why buildout won't parse this HTML properly and find the related distribution?! (Error: Couldn't distribution for 'haufe.recipe.svnsetup'). Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20080403/45a76287/attachment.pgp From deron.meranda at gmail.com Thu Apr 3 20:52:03 2008 From: deron.meranda at gmail.com (Deron Meranda) Date: Thu, 3 Apr 2008 14:52:03 -0400 Subject: [Distutils] Easy_install fails, package somehow depends on itself and won't resolve Message-ID: <5c06fa770804031152p39be618dg90840f46333f4ab7@mail.gmail.com> I'm having an issue with easy_install from setuptools 0.6c8 with egg files containing C extensions. I have created an egg from a package which includes C code. When I try to install it on another system though, the egg itself will install, but then easy_install aborts with an error while trying to resolve it's dependencies (which it shouldn't have any dependencies). The created egg file is named pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg easy_installing it though yields an error: ----- $ easy_install -v -H "" pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg Processing pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg Copying pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg to /....../lib/python2.5/site-packages Removing pysqlite 2.3.2 from easy-install.pth file Adding pysqlite 2.4.1 to easy-install.pth file Saving /....../lib/python2.5/site-packages/easy-install.pth Installed /....../lib/python2.5/site-packages/pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg Processing dependencies for pysqlite==2.4.1 Searching for pysqlite==2.4.1 Link to http://pypi.python.org/simple/pysqlite/ ***BLOCKED*** by --allow-hosts Couldn't retrieve index page for 'pysqlite' Scanning index of all packages (this may take a while) Link to http://pypi.python.org/simple/ ***BLOCKED*** by --allow-hosts No local packages or download links found for pysqlite==2.4.1 error: Could not find suitable distribution for Requirement.parse('pysqlite==2.4.1') ----- However I can still use the package just fine, and it seems the same require condition works: >>> import pkg_resources >>> pkg_resources.require('pysqlite==2.4.1') [pysqlite 2.4.1 (/....../lib/python2.5/site-packages/pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg)] >>> import pysqlite2 >>> pysqlite2.__path__ ['/....../lib/python2.5/site-packages/pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg/pysqlite2'] Everything else works just fine too. The C library gets extracted into ~/.python-eggs and using the package works flawlessly. The only thing I can think is that the system I'm installing on has a slightly different OS version. But it should be (and is) binary compatible. System use to build egg: $ uname -s -r -m HP-UX B.11.00 9000/800 System trying to install on: $ uname -s -r -m HP-UX B.11.11 9000/800 I should also note that the python executable (and its libraries) are identical on each system. Python was actually built on the first system, and copied to the second. Yes I know sqlite is built into python 2.5 so pysqlite isn't needed, but this should work anyway, right? Can anybody help me understand what is going on, and how can I get the egg file to install without an error? Thanks -- Deron Meranda From jim at zope.com Thu Apr 3 21:24:23 2008 From: jim at zope.com (Jim Fulton) Date: Thu, 3 Apr 2008 15:24:23 -0400 Subject: [Distutils] [zc.buildout] buildout unable to parse 'simple' view In-Reply-To: <2430BB03575C506E8261F96A@[192.168.0.24]> References: <2430BB03575C506E8261F96A@[192.168.0.24]> Message-ID: <70572D41-441C-4CE5-B764-2363323BEAB4@zope.com> On Apr 3, 2008, at 10:56 AM, Andreas Jung wrote: > > We're running a local egg repository based on haufe.eggserver. > Buildout won't find an egg called 'haufe.recipe.svnsetup' that is > stored on the server. The server is configured properly within the > find-links section. I don't know what you mean by a "simple" view. > I wonder why buildout won't parse this HTML properly and find the > related > distribution?! (Error: Couldn't distribution for > 'haufe.recipe.svnsetup'). I don't know what it's actually looking at. Can you give a URL? Note that an index can have a nested structure that won't work when used in a find-links option. So a valid index may not work with find links. Jim -- Jim Fulton Zope Corporation From plone at hannosch.info Fri Apr 4 09:47:07 2008 From: plone at hannosch.info (Hanno Schlichting) Date: Fri, 04 Apr 2008 09:47:07 +0200 Subject: [Distutils] [zc.buildout] upgrade with changed explicit recipe version In-Reply-To: References: Message-ID: Hi. Jim Fulton wrote: > On Apr 2, 2008, at 3:44 PM, Hanno Schlichting wrote: > I think there is a general problem and no, you aren't miss-using the > the version restriction. > >> I have the feeling that buildout should either add the old recipe to >> the >> environment to satisfy the uninstall requirement > > Unfortunately, there can be only one version in the environment at a > time. > >> or accept newer >> versions of a recipe to uninstall older versions. > > That is probably better. > > Another option might be to uninstall in a separate process. (Just > thinking out loud.) > > I haven't seen this situation before, although I should have expected > it. > > I'll think about this and respond in a day or 2. If I forget, bug me, > or create a launchpad issue. There has already been a report in November last year by Wichert: https://bugs.launchpad.net/zc.buildout/+bug/163776 I added the information from this conversation to the report. Hanno From pje at telecommunity.com Fri Apr 4 17:40:47 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 04 Apr 2008 11:40:47 -0400 Subject: [Distutils] Easy_install fails, package somehow depends on itself and won't resolve In-Reply-To: <5c06fa770804031152p39be618dg90840f46333f4ab7@mail.gmail.co m> References: <5c06fa770804031152p39be618dg90840f46333f4ab7@mail.gmail.com> Message-ID: <20080404154054.275523A40AF@sparrow.telecommunity.com> At 02:52 PM 4/3/2008 -0400, Deron Meranda wrote: >The only thing I can think is that the system I'm installing on has a >slightly different >OS version. But it should be (and is) binary compatible. Setuptools currently uses exact matching on OS strings, so that's almost certainly the problem. If you want to contribute code to do platform matching, check out the pkg_resources docs under "Platform Utilities" and then look at the code for the existing Mac OS version compatibility checking. Be sure to include tests in your patch, and the tests (as well as the version comparison code itself) must be able to run on all platforms. Feel free to ask on the list if you run into any problems/questions. From dpeterson at enthought.com Fri Apr 4 23:57:40 2008 From: dpeterson at enthought.com (Dave Peterson) Date: Fri, 04 Apr 2008 16:57:40 -0500 Subject: [Distutils] Setting up a setuptools community In-Reply-To: <47F47C11.4010003@taupro.com> References: <3219B2C8-AD42-4C74-8ECE-714EB66B63F8@zooko.com> <47F30E34.3080204@enthought.com> <20080402065817.5A90F3A40AE@sparrow.telecommunity.com> <47F3DA60.3040108@enthought.com> <47F47C11.4010003@taupro.com> Message-ID: <47F6A454.1070307@enthought.com> Jeff Rush wrote: > Dave Peterson wrote: > >> Phillip J. Eby wrote: >> >>> At 11:40 PM 4/1/2008 -0500, Dave Peterson wrote: >>> >>>> I've previously volunteered Enthought to do about the same thing (except >>>> to include svn with the Trac instance) but there wasn't much response >>>> beyond Jeff Rush saying he was already setting one up, and the response >>>> by Phillip to use the python dev bug-tracker and svn instead. >>>> >>> ... >>> >>> >>>> Phillip, I think everyone is waiting for you to provide direction so can >>>> you make some decisions about how setuptools evolves to a larger >>>> community effort? >>>> >>> As I believe you just pointed out, I already did that. :) >>> >> Egggcellent. >> >> Doing a search for 'setuptools' in the issue tracker only returns two >> open issues currently. I'm guessing I'm not catching all the issues as >> a result. It would probably help if we had a clear component to use >> (i.e. Setuptools). Anyone have any idea who to contact to get that >> done? I've so far had no luck reading the info on www.python.org/dev >> looking for this sort of thing. >> > > Hi Dave, > > At PyCon I asked the core team about using the Python tracker for setuptools > but Brett Cannon said that no software outside of the Python core is currently > using it and it wouldn't be appropriate for setuptools to do so. They were > willing to give us our own instance of a tracker, similar to how they have for > Jython. > > Martin v. L?wis has been in the process since PyCon of doing so, but is busy > and apparently the Roundup software used is tricky to configure for this and,f > for the email address, requires ISP intervention. The plan when this is done > is to have a tracker at http://bugs.python.org/setuptools/ with an email > address for submission/commenting of setuptools at bugs.python.org. > > Sorry for the delay - I didn't realize this would be such a big deal to get > started. We could also start using Launchpad instantly, but I figured > reaching concensus to do so, with Phillip and others more familiar with the > Python tracker, would be difficult. > While I'm glad to hear there has been progress, I'm concerned we risk losing momentum by waiting. Can we at least start by using their wiki to capture some thoughts on priorities and roadmap? Or does that need to be separate too? Or does Phillip already have that somewhere I haven't seen? -- Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080404/16abe2c7/attachment.htm From zooko at zooko.com Sat Apr 5 04:31:08 2008 From: zooko at zooko.com (zooko) Date: Fri, 4 Apr 2008 20:31:08 -0600 Subject: [Distutils] Setting up a setuptools community In-Reply-To: <47F6A454.1070307@enthought.com> References: <3219B2C8-AD42-4C74-8ECE-714EB66B63F8@zooko.com> <47F30E34.3080204@enthought.com> <20080402065817.5A90F3A40AE@sparrow.telecommunity.com> <47F3DA60.3040108@enthought.com> <47F47C11.4010003@taupro.com> <47F6A454.1070307@enthought.com> Message-ID: <8AF9B681-8527-402A-8D00-5B7D80B71851@zooko.com> >> Martin v. L?wis has been in the process since PyCon of doing so, >> but is busy and apparently the Roundup software used is tricky to >> configure for this and,f for the email address, requires ISP >> intervention. The plan when this is done is to have a tracker at >> http://bugs.python.org/setuptools/ with an email address for >> submission/commenting of setuptools at bugs.python.org. Sorry for the >> delay - I didn't realize this would be such a big deal to get >> started. We could also start using Launchpad instantly, but I >> figured reaching concensus to do so, with Phillip and others more >> familiar with the Python tracker, would be difficult. > > While I'm glad to hear there has been progress, I'm concerned we > risk losing momentum by waiting. > > Can we at least start by using their wiki to capture some thoughts > on priorities and roadmap? Or does that need to be separate too? > Or does Phillip already have that somewhere I haven't seen? Just to give us all a kick in the pants, I set up a trac instance and imported setuptools revision control history into it: http://allmydata.org/trac/setuptools It took me only a few minutes -- I'm getting familiar with the process of setting up trac instances for open source projects. Hm, the timeline for the last 90 days: http://allmydata.org/trac/setuptools/timeline? from=2008-04-04&daysback=90&ticket=on&changeset=on&milestone=on&update=U pdate The issue tracker is currently empty: http://allmydata.org/trac/setuptools/report/1 You can register and start scribbling on the wiki if you like -- anything you write there can be copied over to a new site once it is operational. The "buildbot" link yields a 404 Not Found -- I haven't set up a buildbot yet. Whether or not we use trac, and whether or not we use hosting provided by Allmydata, Inc., on http://allmydata.org , we should definitely set up a buildbot. Regards, Zooko From ziade.tarek at gmail.com Sat Apr 5 09:17:57 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 5 Apr 2008 09:17:57 +0200 Subject: [Distutils] --allow-host and local files Message-ID: <94bdd2610804050017g1a1904dbq490b319b2d1e1ddd@mail.gmail.com> Hello, while playing with the allow-host options in setuptools, I have noticed that it is restricted to URLs because url_ok() uses urlparse over the regular expressions that are provided This means that it is not possible to allow local folders to be visited since a "file://*" expression for example, will lead to an empty string: >>> import urlparse >>> urlparse.urlparse('file:///tmp/my/local/file.tgz')[1] '' This will make some links blocked and impossible to add as authorized resource: ... Link to file:///tmp/tmpE-LbUpbuildouttests/setuptools/ ***BLOCKED*** by --allow-hosts ... I would like to propose something: I think this would be easy to change by calling URL_SCHEME() over the url in url_ok(), before urlparse is called. If it is not an url we could then consider that the url is "safe" and return immediatly. if you think it is a good idea, i can provide a patch with a test, Regards, Tarek -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080405/b1e0df53/attachment.htm From klaus_zimmermann at gmx.de Sat Apr 5 17:15:09 2008 From: klaus_zimmermann at gmx.de (Klaus Zimmermann) Date: Sat, 05 Apr 2008 17:15:09 +0200 Subject: [Distutils] [PATCH][setuptools] Add test_runner support to setup Message-ID: <1207408509.6012.9.camel@localhost> Hi all, for our nightly build system I needed the capability to use a different test runner, namely one that outputs junit compatible xml files. In order to support that through the setup.py test command I hacked on setuptools. Is there a simpler way to do this? If not you might want to consider attached patch for addition (Should be against latest svn.) Cheers, Klaus -------------- next part -------------- A non-text attachment was scrubbed... Name: test_runner.patch Type: text/x-patch Size: 3061 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20080405/09da56bb/attachment.bin From floris.bruynooghe at gmail.com Sat Apr 5 23:07:49 2008 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Sat, 5 Apr 2008 22:07:49 +0100 Subject: [Distutils] A Modest Proposal for "A Database of Installed Packages" In-Reply-To: <525f23e80803280802r507322car646b6110799934b@mail.gmail.com> References: <525f23e80803280802r507322car646b6110799934b@mail.gmail.com> Message-ID: <20080405210749.GA7978@laurie.devork> Hello On Fri, Mar 28, 2008 at 11:02:19AM -0400, Alexander Michael wrote: > I'll continue my fool hearty effort [1] to build a concrete proposal > for "a database of installed packages" by offering up a sketch of a > possible straw-man "solution". I realize that this is likely > oversimplified to a fault, but I hope it will help us move forward. > Apologies if the equivalent of this has been proposed and rejected > before. My proposal is basically to make PKG-INFO functional and > usable by: > > * Fixing the technical issues with requirements (i.e. dependencies) > and naming as specified by PEP 314/345. > > * Modifying distutils to install PKG-INFO alongside each module file > or package directory as a side-car file of the same name but with a > special extension (.pyi or whatever). These files would be the place > to include the optional list of installed files as well as the > optional md5sums, if desired by the installer. Files in the package > will be listed using relative paths, while far flung files (bin, > shared, etc) will get full paths so that there is full allowance for > relocating simple (nothing in bin or shared) modules and packages. > Although optional, "python setup.py install" will include the > installed file list by default. > > That's it. This proposal has been here about a week now, with no comments on it. I take that as positive as no one has had major objections. :-) Personally I think it is a good proposal, it does basically what an installation database would have to do while being minimally invasive. The important question is however: Is this enough for setuptools to work withouth doing all it's path magic? Would this be a workable solution for setuptools? Now my own thoughts about the technicalities (sorry this got long)... Distutils does create a ${pkgname}-${version}.egg-info file right now with the PKG-INFO data in. From earlier discussions it seems the .egg-info extension is not very loved, so a change to .pyi could be done (also, it has little to do with eggs). Secondly I'm not sure how useful it is for the version number to be encoded in the filename. It seems the .egg-info file does get installed in the site-packages root currently. This will likely give conflicts when we're starting to use namespace packages. We can't put the .pyi *in* the package since then we lose support for simple modules, so we have to place it *next* to the package. So if "bar" is a namespace package inside "foo" then we would have: site-packages/foo/bar.pyi site-packages/foo/bar/__init__.py This means any package tool will need to recursively scan the site-packages directory to find the files, but that doesn't seem like to much a penalty? The alternative is to have a separate directory for the intalldb files: site-packages/foo/bar/__init__.py site-packages/install.db/foo/bar.pyi This could significantly reduce the scanning time since there are far fewer files too walk. I chose a name with a "." for install.db so we're not stealing a possible module or package name. Other then that the name of the directory can by anything we manage to agree on. :-) Using this approach might create confusion about relative paths mentioned in .pyi files though (is the root the current direcotry or do we pretend the .pyi was actually next to the package/module?). Distribution not providing a package/module or with a different distribution name then the package(s)/module(s) provided would end up in the top-level of the database (in both scenarios), effectively stealing package/module names but that seems to be the current behaviour of distutils already anyway. Namespace sub-distributions (bar in the example above) with a different distribution name as package/module name would steal names from it's namespace. Namespace packages are not fully handled yet, there is still the issue of who owns site-packages/foo/__init__.py. That would logically be defined by site-packages/foo.pyi, but we don't want the user to have to install yet another package for this. So for a namespace package the .pyi could look like this: Metadata-Version: 1.0 Name: foo ... Owner: setuptools Namespace: True Directory: foo/ File: foo/__init__.py It might be possible that a namespace package doesn't need an owner so that a different tool is allowed to clean it up, but I'm not sure about that. When "bar" gets uninstalled now it should know if it can clean up it's namespace "foo" too (if it is empty). So bar.pyi should have: Metadata-Version: 1.0 Name: bar ... Owner: setuptools Requires-Namespaces: foo Directory: foo/bar/ File: foo/bar/__init__.py Here "foo" could also have been a dotted name: "foo.sub.package". So the "foo.sub" package would have both the Namespace: and Requires-Namespace: fields in it's .pyi. AFAIK this should cover namespace packages. So the new headers to turn the PKG-INFO into a .pyi would be: Owner: The owner of this distribution. This would be any string representing the package tool, e.g.: distutils, setuptools, zc.buildout, rpm, dpkg, etc. Provides: Copied from PEP262. Don't like this in it's original form since it's ambiguous. So this lists the *distributions* provided by this package on top of it's native name in the Name: field. Optional (and very rare). Modules: List of packages/modules provided. If no packages or modules are installed this doesn't need to be present. You could argue that this should be called Packages: or so. Derived from PEP262. Namespace: The value of this doesn't matter, when present it indicates this .pyi file describes a namespace package. Requires: Copied from PEP262 (also ambiguous). Optional. Distributions that must be installed for this distribution to work. Requires-Modules: Optional list of packages/modules required. No need to list modules in the standard library. (Figuring out if this site-packages tree is of the correct python version is of no use for the installdb). Derived from PEP262. Requires-Namespaces: This package requires a namespace. The value is a list of dotted names of the namespace packages, as they would appear in an import statement. Directory: A directory from this package. Relative to this .pyi or absolute. For directories inside site-packages they *should* be relative, for outside site-packges they *should* be absolute. File: The value of this is first an optional MD5 hash (or SHA1?) of the file, followed by the path of the installed file (absolute or relative, same rules as for Directory: above). The only restriction this makes on a filename is that you can't have a file in the current directory that is also a valid hash and does not have a hash itself. You can work around this by prepending the filename with ./ however - but why would you want such a file? The only issue I can think of right now is with File:. It is not very extensible if a tool wants too keep track of extra info like file permissions. AFAIK RFC822 requires you to keep the order of the fields, if so we could make this: File: foo.py File-MD5: XXXXXXXXXXXX X-MyTool-File-perms: -rw-rw-r- File: bar.py ... Lastly --and I'm not sure how happy I'm about this, should have thought of this earlier-- the python packaging tools need to support giving away ownership at install time! Since Debian and Redhat etc just call setup.py that would mean each package they install would be owned by distutils/setuptools/... That's bad. I propose that setup.py needs to honour an environment variable: PYI_OWNER so that distros can set this to their custom name (dpkg, rpm, ...). Although I can imagine in Debian's case that it's better to change the dh_py* tools to go and modify the .pyi files. So if all distros are happy with having to modify installed files this might not be necessary. Another a nice/required feature for distros would be to ask the tool to only install the namespace package or omit the namespace packge. This could just be a command line switch to setup.py I think. Again this is not a hard requirement, I can imagine Debian's dh_py* tools to scan the .pyi files, detect namespace packages and (re)move them as required. But once more I don't know enough about other distro's. Phew, thanks for reading this far! I hope this is useful, if it is we should probably start writing the text for the new PEP262 on a wiki somewhere while we discus details. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From pje at telecommunity.com Sun Apr 6 01:50:19 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 05 Apr 2008 19:50:19 -0400 Subject: [Distutils] A Modest Proposal for "A Database of Installed Packages" In-Reply-To: <20080405210749.GA7978@laurie.devork> References: <525f23e80803280802r507322car646b6110799934b@mail.gmail.com> <20080405210749.GA7978@laurie.devork> Message-ID: <20080405235028.152133A4042@sparrow.telecommunity.com> At 10:07 PM 4/5/2008 +0100, Floris Bruynooghe wrote: >This proposal has been here about a week now, with no comments on it. >I take that as positive as no one has had major objections. :-) It's more that there are some holes and handwaving; I haven't really had the mental bandwidth to offer comments on the original proposal as yet. (One comment, though: I really don't like the idea of extending PKG-INFO to include installation data; it's only incidentally related and there are other contexts in which we use PKG-INFO where having that data included would make no sense. Plus, it's really not an ideal file format for including data about a potentially rather large number of files.) >Secondly I'm not sure how >useful it is for the version number to be encoded in the filename. It's very useful for setuptools, as it avoids the need to open and parse the file when searching for a suitable version of a desired package. >It seems the .egg-info file does get installed in the site-packages >root currently. This will likely give conflicts when we're starting >to use namespace packages. This doesn't make sense. Namespace packages and project names are not in the same namespace and have nothing to do with each other. For example, I have a project called DecoratorTools that installs a module in the peak.util namespace package. Its egg-info would be something like DecoratorTools-1.6.egg-info. So I think you are confused about something here. > We can't put the .pyi *in* the package >since then we lose support for simple modules, so we have to place it >*next* to the package. No, it just goes to the --install-lib directory, which in the default case is site-packages. (But could be a PYTHONPATH or other directory.) > So if "bar" is a namespace package inside >"foo" then we would have: > >site-packages/foo/bar.pyi >site-packages/foo/bar/__init__.py Ah, I see... you are definitely confusing package names and project names. >This means any package tool will need to recursively scan the >site-packages directory to find the files, but that doesn't seem like >to much a penalty? The alternative is to have a separate directory >for the intalldb files: > >site-packages/foo/bar/__init__.py >site-packages/install.db/foo/bar.pyi > >This could significantly reduce the scanning time since there are far >fewer files too walk. I chose a name with a "." for install.db so >we're not stealing a possible module or package name. Other then that >the name of the directory can by anything we manage to agree on. :-) >Using this approach might create confusion about relative paths >mentioned in .pyi files though (is the root the current direcotry or >do we pretend the .pyi was actually next to the package/module?). > >Distribution not providing a package/module or with a different >distribution name then the package(s)/module(s) provided would end up >in the top-level of the database (in both scenarios), effectively >stealing package/module names but that seems to be the current >behaviour of distutils already anyway. Namespace sub-distributions >(bar in the example above) with a different distribution name as >package/module name would steal names from it's namespace. All of this is moot, since project/distribution names are unrelated to package names. >Namespace packages are not fully handled yet, ... > >AFAIK this should cover namespace packages. Unfortunately, this doesn't fix the problem, since either *some* package has to own the __init__.py, or there has to be a way for Python to treat the directory as a package without one. And for system package managers (esp. on Linux), some *one* system package must own the file - it can't be owned by multiple system packages. My guess is that this is true, *even if* the file is automatically generated. Some system packaging folks will need to chime in here. >Lastly --and I'm not sure how happy I'm about this, should have >thought of this earlier-- the python packaging tools need to support >giving away ownership at install time! Since Debian and Redhat etc >just call setup.py that would mean each package they install would be >owned by distutils/setuptools/... That's bad. > >I propose that setup.py needs to honour an environment variable: >PYI_OWNER so that distros can set this to their custom name (dpkg, >rpm, ...). A command-line option to 'install' that's inherited by 'install_egg_info' would handle this; I don't think an environment variable is a good idea for this -- too implicit. Note that bdist_rpm, for example, would need to encode this as a command-line option in the .spec file, anyway. >Phew, thanks for reading this far! I hope this is useful, if it is we >should probably start writing the text for the new PEP262 on a wiki >somewhere while we discus details. The major issues at the moment are that 1) your spec is confused about packages vs. projects or distributions (and thus needs to be revamped with that in mind), and 2) PKG-INFO is a really lousy place to put this, from a formatting perspective. It's one thing to include the PKG-INFO in the install DB, and another thing entirely to include the install db into the PKG-INFO! I think PEP 262 had the right idea, even though I'm not overjoyed by its proposed format, either. From pje at telecommunity.com Sun Apr 6 01:53:08 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 05 Apr 2008 19:53:08 -0400 Subject: [Distutils] [PATCH][setuptools] Add test_runner support to setup In-Reply-To: <1207408509.6012.9.camel@localhost> References: <1207408509.6012.9.camel@localhost> Message-ID: <20080405235300.795DE3A4042@sparrow.telecommunity.com> At 05:15 PM 4/5/2008 +0200, Klaus Zimmermann wrote: >Hi all, > >for our nightly build system I needed the capability to use a different >test runner, namely one that outputs junit compatible xml files. >In order to support that through the setup.py test command I hacked on >setuptools. > >Is there a simpler way to do this? >If not you might want to consider attached patch for addition >(Should be against latest svn.) Add documentation to the patch and you've got yourself a deal. :) From ziade.tarek at gmail.com Sun Apr 6 02:25:32 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 6 Apr 2008 02:25:32 +0200 Subject: [Distutils] --allow-host and local files In-Reply-To: <94bdd2610804050017g1a1904dbq490b319b2d1e1ddd@mail.gmail.com> References: <94bdd2610804050017g1a1904dbq490b319b2d1e1ddd@mail.gmail.com> Message-ID: <94bdd2610804051725i45fd261fsa7d239ac87edaaf9@mail.gmail.com> Here's a patch, with the test case and the fix On Sat, Apr 5, 2008 at 9:17 AM, Tarek Ziad? wrote: > Hello, > > while playing with the allow-host options in setuptools, I have noticed > that it is restricted to URLs because url_ok() uses urlparse > over the regular expressions that are provided > > This means that it is not possible to allow local folders to be visited > since a "file://*" expression for example, > will lead to an empty string: > > >>> import urlparse > >>> urlparse.urlparse('file:///tmp/my/local/file.tgz')[1] > '' > > This will make some links blocked and impossible to add as authorized > resource: > ... > Link to file:///tmp/tmpE-LbUpbuildouttests/setuptools/ ***BLOCKED*** by > --allow-hosts > ... > > I would like to propose something: > > I think this would be easy to change by calling URL_SCHEME() over the url > in url_ok(), > before urlparse is called. If it is not an url we could then consider that > the url is "safe" > and return immediatly. > > if you think it is a good idea, i can provide a patch with a test, > > Regards, > > Tarek > > -- > Tarek Ziad? | Association AfPy | www.afpy.org > Blog FR | http://programmation-python.org > Blog EN | http://tarekziade.wordpress.com/ -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080406/1aadbc2e/attachment-0001.htm -------------- next part -------------- A non-text attachment was scrubbed... Name: package_index.patch Type: application/octet-stream Size: 1281 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20080406/1aadbc2e/attachment-0001.obj From floris.bruynooghe at gmail.com Sun Apr 6 03:18:42 2008 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Sun, 6 Apr 2008 02:18:42 +0100 Subject: [Distutils] A Modest Proposal for "A Database of Installed Packages" In-Reply-To: <20080405235028.152133A4042@sparrow.telecommunity.com> References: <525f23e80803280802r507322car646b6110799934b@mail.gmail.com> <20080405210749.GA7978@laurie.devork> <20080405235028.152133A4042@sparrow.telecommunity.com> Message-ID: <20080406011842.GA26228@laurie.devork> On Sat, Apr 05, 2008 at 07:50:19PM -0400, Phillip J. Eby wrote: > At 10:07 PM 4/5/2008 +0100, Floris Bruynooghe wrote: > (One comment, though: I really don't like the idea of extending PKG-INFO > to include installation data; it's only incidentally related and there > are other contexts in which we use PKG-INFO where having that data > included would make no sense. Plus, it's really not an ideal file format > for including data about a potentially rather large number of files.) That's fair. Blowing up the files with the PKG-INFO information in could have bad performance effects. rfc822 in the stdlib reads everything in memory AFAIK. >> Secondly I'm not sure how >> useful it is for the version number to be encoded in the filename. > > It's very useful for setuptools, as it avoids the need to open and parse > the file when searching for a suitable version of a desired package. Hmm, it's not that much work to read the contents of a .egg-info. Just seems odd to me to have this info in two places so close to each other. [...] > All of this is moot, since project/distribution names are unrelated to > package names. So this means there is a flat namespace for all project names and nested namespace for modules. When I was saying that project names "steal" names from modules that is because they end up in the same directory. I.e. project "foo" with foo_1.0.egg-info provides module "bar", while project "bar" with bar_1.0.egg-info provides module "bar2". Not ideal. What I was trying to get at was to prefix project names that provide a sub-module for a namespace with the namespace module name. Inside the hypothetical installdb that is. But maybe that makes the whole project namespace vs modules namespace just more confusing (thinking of it definatly a bad idea if the project of the sub-package also installs a script or so). The second part was introducing a "virtual project" for pure namespace packages, where the project name would have to be the same as the package name in order to find it. >> AFAIK this should cover namespace packages. > > Unfortunately, this doesn't fix the problem, since either *some* package > has to own the __init__.py, or there has to be a way for Python to treat > the directory as a package without one. And for system package managers > (esp. on Linux), some *one* system package must own the file - it can't > be owned by multiple system packages. With the format I suggested a package tool could detect on install if a required pure namespace package was already installed or still needed to be installed/created. Similar on removal it is possible to detect if the pure namespace package is still required (by checking if it's directory contains any other files then those provided by the namespace package) on removal of a sub-package. > My guess is that this is true, *even if* the file is automatically > generated. Some system packaging folks will need to chime in here. System packagers would create 2 packages out of a package requiring a namespace package. One the pure namespace, the other with the sub-package. Other sub-packages then just need to depend on the pure namespace one. >> Lastly --and I'm not sure how happy I'm about this, should have >> thought of this earlier-- the python packaging tools need to support >> giving away ownership at install time! Since Debian and Redhat etc >> just call setup.py that would mean each package they install would be >> owned by distutils/setuptools/... That's bad. >> >> I propose that setup.py needs to honour an environment variable: >> PYI_OWNER so that distros can set this to their custom name (dpkg, >> rpm, ...). > > A command-line option to 'install' that's inherited by > 'install_egg_info' would handle this; I don't think an environment > variable is a good idea for this -- too implicit. Note that bdist_rpm, > for example, would need to encode this as a command-line option in the > .spec file, anyway. I picked an environment variable here because then it would be possible to call setup.py identical whether or not it provides this new installdb. Providing a non-existing command line option tends to cause more problems. >> Phew, thanks for reading this far! I hope this is useful, if it is we >> should probably start writing the text for the new PEP262 on a wiki >> somewhere while we discus details. > > The major issues at the moment are that 1) your spec is confused about > packages vs. projects or distributions (and thus needs to be revamped > with that in mind), See clarification above. > and 2) PKG-INFO is a really lousy place to put this, > from a formatting perspective. It's one thing to include the PKG-INFO in > the install DB, and another thing entirely to include the install db into > the PKG-INFO! I think PEP 262 had the right idea, even though I'm not > overjoyed by its proposed format, either. Not wanting to blow up PKG-INFO is laudable, but OTOH separating out the data is dubious as is replicating data (PKG-INFO data in .egg-info AND the installdb). PKG-INFO was just simple as it's there and tools can use it already. Maybe we're making it too hard by wanting to cover *every* file installed by python projects? The main reason for this installdb, as I understand it, is so that a package tool can install a sub-project in a namespace package installed by someone else. And similarly that someone else doesn't wipe away the sub-package when it thinks it can remove the namespace package. Ah, this make me think of the people that complain on comp.lang.python that Python namespaces are too tightly bound to files and directories... It all makes sense now, we wouldn't even be having this discussion if a package could declare it's namespace in the code! ;-) Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From noah.gift at gmail.com Sun Apr 6 02:19:03 2008 From: noah.gift at gmail.com (Noah Gift) Date: Sat, 5 Apr 2008 20:19:03 -0400 Subject: [Distutils] ReST formatting errors when registering package can invalidate HTML conversion Message-ID: <04DC863B-771E-439D-ACAF-76373BE45D03@gmail.com> I thought I would post a generally help tip that Ian Bicking helped me with. I couldn't figure out why my ReST formatted long_description field would not get converted to HTML when I registered my egg with the cheeseshop: python setup.py register It turns out I had a minor error in my ReST formatting, I didn't underline a headline with enough length. So, the moral of the story is if you wonder why you cannot get HTML formatted ReST to display when you register your egg, it is possible you have a formatting error. It was tricky for me to discover, because I did not get any error message when I uploaded to the cheeseshop, and I just missed the minor formatting error when I was viewing the converted HTML document I used to preview what I created. Noah Gift / http://noahgift.com From mhammond at skippinet.com.au Sun Apr 6 04:34:54 2008 From: mhammond at skippinet.com.au (Mark Hammond) Date: Sun, 6 Apr 2008 12:34:54 +1000 Subject: [Distutils] Remove "current" Windows executables from Lib/distutils/command in svn? Message-ID: <018701c8978e$ce620660$6b261320$@com.au> As I was preparing to check-in my cross-compilation patch, which includes a new x64 executable in the Lib/distutils/command directory, it occurs to me that we don't actually need it in subversion - and nor do we need wininst-9.0.exe. I believe these executables are in svn for historical reasons. The build process for these executables are now integrated into the core Python build process (although they are disabled by default). There is a good use-case for keeping executables built with older MSVC versions, but I don't see one for executables built with the current version. I'd like to propose we delete Lib/Distutils/command/wininst-9.0.exe, and enable the building of that project by default in the standard build process (and I'll setup the x64 build of the executable similarly). They will then need to be re-added when a future version moves away from Visual Studio 2008, but that seems like a reasonable way to approach things. Are there any objections to this, or should we stick with the status-quo and I'll add the new x64 executable? Or do things that way *just* for x64? Thanks, Mark. From pje at telecommunity.com Sun Apr 6 04:49:24 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 05 Apr 2008 22:49:24 -0400 Subject: [Distutils] A Modest Proposal for "A Database of Installed Packages" In-Reply-To: <20080406011842.GA26228@laurie.devork> References: <525f23e80803280802r507322car646b6110799934b@mail.gmail.com> <20080405210749.GA7978@laurie.devork> <20080405235028.152133A4042@sparrow.telecommunity.com> <20080406011842.GA26228@laurie.devork> Message-ID: <20080406024917.8FEDF3A4042@sparrow.telecommunity.com> At 02:18 AM 4/6/2008 +0100, Floris Bruynooghe wrote: >On Sat, Apr 05, 2008 at 07:50:19PM -0400, Phillip J. Eby wrote: > > At 10:07 PM 4/5/2008 +0100, Floris Bruynooghe wrote: > > (One comment, though: I really don't like the idea of extending PKG-INFO > > to include installation data; it's only incidentally related and there > > are other contexts in which we use PKG-INFO where having that data > > included would make no sense. Plus, it's really not an ideal file format > > for including data about a potentially rather large number of files.) > >That's fair. Blowing up the files with the PKG-INFO information in >could have bad performance effects. rfc822 in the stdlib reads >everything in memory AFAIK. > > >> Secondly I'm not sure how > >> useful it is for the version number to be encoded in the filename. > > > > It's very useful for setuptools, as it avoids the need to open and parse > > the file when searching for a suitable version of a desired package. > >Hmm, it's not that much work to read the contents of a .egg-info. >Just seems odd to me to have this info in two places so close to each >other. It allows pkg_resources to grok the entire contents of a directory using only a single listdir operation -- not an unbounded number of open-and-read operations. Of course, if we're going to *also* have a properly-named .egg-info file, then using just the project name is sufficient for the install db. >[...] > > All of this is moot, since project/distribution names are unrelated to > > package names. > >So this means there is a flat namespace for all project names and >nested namespace for modules. When I was saying that project names >"steal" names from modules that is because they end up in the same >directory. I.e. project "foo" with foo_1.0.egg-info provides module >"bar", while project "bar" with bar_1.0.egg-info provides module >"bar2". Not ideal. I have no idea what you're saying here. There is absolutely no relationship between project names and the Python package/module namespace. None. Thus, any attempt to talk about them as though they are related is just noise to me. >The second part was introducing a "virtual project" for pure namespace >packages, where the project name would have to be the same as the >package name in order to find it. I think there would also need to be some prefix to the name, to prevent confusion in the event that there exists a normal project name that happens to use that package name. (Again: the two namespaces are unrelated, so a new/reserved namespace would be required for these virtual projects.) > >> AFAIK this should cover namespace packages. > > > > Unfortunately, this doesn't fix the problem, since either *some* package > > has to own the __init__.py, or there has to be a way for Python to treat > > the directory as a package without one. And for system package managers > > (esp. on Linux), some *one* system package must own the file - it can't > > be owned by multiple system packages. > >With the format I suggested a package tool could detect on install if >a required pure namespace package was already installed or still >needed to be installed/created. Similar on removal it is possible to >detect if the pure namespace package is still required (by checking if >it's directory contains any other files then those provided by the >namespace package) on removal of a sub-package. Again... some system packaging folks need to speak up on this, because my understanding is that some tools simply can't do something like this. They need to make explicit what a given package depends on, and install that, not dynamically decide what dependencies something has. (And then there is the possibility of a problem if a non-system packager installs the namespace, and then you install a system package for something that includes packages in that namespace.) > >> Lastly --and I'm not sure how happy I'm about this, should have > >> thought of this earlier-- the python packaging tools need to support > >> giving away ownership at install time! Since Debian and Redhat etc > >> just call setup.py that would mean each package they install would be > >> owned by distutils/setuptools/... That's bad. > >> > >> I propose that setup.py needs to honour an environment variable: > >> PYI_OWNER so that distros can set this to their custom name (dpkg, > >> rpm, ...). > > > > A command-line option to 'install' that's inherited by > > 'install_egg_info' would handle this; I don't think an environment > > variable is a good idea for this -- too implicit. Note that bdist_rpm, > > for example, would need to encode this as a command-line option in the > > .spec file, anyway. > >I picked an environment variable here because then it would be >possible to call setup.py identical whether or not it provides this >new installdb. Providing a non-existing command line option tends to >cause more problems. How so, if this is going into a new version of Python? >Not wanting to blow up PKG-INFO is laudable, but OTOH separating out >the data is dubious as is replicating data (PKG-INFO data in .egg-info >AND the installdb). PKG-INFO was just simple as it's there and tools >can use it already. > >Maybe we're making it too hard by wanting to cover *every* file >installed by python projects? The main reason for this installdb, as >I understand it, is so that a package tool can install a sub-project >in a namespace package installed by someone else. And similarly that >someone else doesn't wipe away the sub-package when it thinks it can >remove the namespace package. It's not just about namespace packages, it's about any package or module. We also want to know about installed scripts, data, etc., so that they can be cleaned up by a tool that does uninstalls. > Ah, this make me think of the people >that complain on comp.lang.python that Python namespaces are too >tightly bound to files and directories... It all makes sense now, we >wouldn't even be having this discussion if a package could declare >it's namespace in the code! ;-) Or if you could import from directories without needing there to be an __init__.py, and Python supported namespace packages by default. From martin at v.loewis.de Sun Apr 6 08:15:32 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 06 Apr 2008 08:15:32 +0200 Subject: [Distutils] [Python-Dev] Remove "current" Windows executables from Lib/distutils/command in svn? In-Reply-To: <018701c8978e$ce620660$6b261320$@com.au> References: <018701c8978e$ce620660$6b261320$@com.au> Message-ID: <47F86A84.6000204@v.loewis.de> > I'd like to propose we delete Lib/Distutils/command/wininst-9.0.exe, and > enable the building of that project by default in the standard build process > (and I'll setup the x64 build of the executable similarly). There are two issues here: a) how does the binary get into the release tarball? You might argue that it doesn't have to, as it is sufficient when it is included in the Windows installer, however, as currently implemented, bdist_wininst also runs on Unix, and people use it that way. b) IIRC, upx was used to compress these executables. I don't think the the current build process covers this, yet (but then, the current binaries might not be compressed with upx anymore, either - not sure whether that would be a bug). Of course, upx would probably not currently apply to Win64-amd64. As another alternative to providing a separate AMD64 binary, you could also try to make the existing binary work properly on Win64, and tell it with a config flag whether to look for Win32 or Win64 (or either) on the target system. Regards, Martin From ziade.tarek at gmail.com Sun Apr 6 11:02:55 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 6 Apr 2008 11:02:55 +0200 Subject: [Distutils] ReST formatting errors when registering package can invalidate HTML conversion In-Reply-To: <04DC863B-771E-439D-ACAF-76373BE45D03@gmail.com> References: <04DC863B-771E-439D-ACAF-76373BE45D03@gmail.com> Message-ID: <94bdd2610804060202i171abc28l1650f396a24c7ede@mail.gmail.com> I would be in favor of a warning message generated by PyPI when a long_description field is not reST compliant. I don't think it is a client-side job. In any case you can test your long_description field like this: $ python setup.py --long-description | rst2html.py --strict > /dev/null If something goes wrong in the conversion you'll get a warning here On Sun, Apr 6, 2008 at 2:19 AM, Noah Gift wrote: > I thought I would post a generally help tip that Ian Bicking helped me > with. I couldn't figure out why my ReST formatted long_description > field would not get converted to HTML when I registered my egg with > the cheeseshop: > > python setup.py register > > It turns out I had a minor error in my ReST formatting, I didn't > underline a headline with enough length. So, the moral of the story > is if you wonder why you cannot get HTML formatted ReST to display > when you register your egg, it is possible you have a formatting > error. It was tricky for me to discover, because I did not get any > error message when I uploaded to the cheeseshop, and I just missed the > minor formatting error when I was viewing the converted HTML document > I used to preview what I created. > > > Noah Gift / http://noahgift.com > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080406/fa9cb371/attachment-0001.htm From ziade.tarek at gmail.com Sun Apr 6 12:10:46 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 6 Apr 2008 12:10:46 +0200 Subject: [Distutils] [setuptools] how to report bugs and patchs ? Message-ID: <94bdd2610804060310v5f6e894dja0da73186a6b56f@mail.gmail.com> Hello, Sorry I must be naive, but I don't understand how setuptools community works at this time. In the README.txt of the package, it is written "Questions, comments, and bug reports should be directed to the distutils-sig mailing list..." But what happens when some bug reports or patches are just ignored in the mailing list ? (even after months) I would expect patch or bug reports to be added in a bug tracker, then rejected or accepted, even if it takes months for them to be treated. At least, registered somewhere. I have read that a bug tracker was going to be set for setuptools, but the current status is rather fuzzy, so I was just wondering if this is stilled planned... Regards, Tarek -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080406/ad5f67ff/attachment.htm From mhammond at skippinet.com.au Sun Apr 6 15:19:21 2008 From: mhammond at skippinet.com.au (Mark Hammond) Date: Sun, 6 Apr 2008 23:19:21 +1000 Subject: [Distutils] [Python-Dev] Remove "current" Windows executables from Lib/distutils/command in svn? In-Reply-To: <47F86A84.6000204@v.loewis.de> References: <018701c8978e$ce620660$6b261320$@com.au> <47F86A84.6000204@v.loewis.de> Message-ID: <01c701c897e8$d8ce3a60$8a6aaf20$@com.au> > As another alternative to providing a separate AMD64 binary, you could > also try to make the existing binary work properly on Win64, and tell > it with a config flag whether to look for Win32 or Win64 (or either) > on the target system. I'm afraid that isn't an option for me in the short term. Should I just check-in my cross-compilation patch as it stands then? Mark From klaus_zimmermann at gmx.de Sun Apr 6 13:40:58 2008 From: klaus_zimmermann at gmx.de (Klaus Zimmermann) Date: Sun, 06 Apr 2008 13:40:58 +0200 Subject: [Distutils] [PATCH][setuptools] Add test_runner support to setup In-Reply-To: <20080405235300.795DE3A4042@sparrow.telecommunity.com> References: <1207408509.6012.9.camel@localhost> <20080405235300.795DE3A4042@sparrow.telecommunity.com> Message-ID: <1207482058.5957.0.camel@localhost> Am Samstag, den 05.04.2008, 19:53 -0400 schrieb Phillip J. Eby: > At 05:15 PM 4/5/2008 +0200, Klaus Zimmermann wrote: > >Hi all, > > > >for our nightly build system I needed the capability to use a different > >test runner, namely one that outputs junit compatible xml files. > >In order to support that through the setup.py test command I hacked on > >setuptools. > > > >Is there a simpler way to do this? > >If not you might want to consider attached patch for addition > >(Should be against latest svn.) > > Add documentation to the patch and you've got yourself a deal. :) Updated patch attached. Have fun, Klaus -------------- next part -------------- A non-text attachment was scrubbed... Name: test_runner.patch Type: text/x-patch Size: 4821 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20080406/a990b369/attachment.bin From martin at v.loewis.de Sun Apr 6 15:54:48 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 06 Apr 2008 15:54:48 +0200 Subject: [Distutils] [Python-Dev] Remove "current" Windows executables from Lib/distutils/command in svn? In-Reply-To: <01c701c897e8$d8ce3a60$8a6aaf20$@com.au> References: <018701c8978e$ce620660$6b261320$@com.au> <47F86A84.6000204@v.loewis.de> <01c701c897e8$d8ce3a60$8a6aaf20$@com.au> Message-ID: <47F8D628.1030404@v.loewis.de> >> As another alternative to providing a separate AMD64 binary, you could >> also try to make the existing binary work properly on Win64, and tell >> it with a config flag whether to look for Win32 or Win64 (or either) >> on the target system. > > I'm afraid that isn't an option for me in the short term. Should I just > check-in my cross-compilation patch as it stands then? Sure, go ahead. Regards, Martin From noah.gift at gmail.com Sun Apr 6 17:29:59 2008 From: noah.gift at gmail.com (Noah Gift) Date: Sun, 6 Apr 2008 11:29:59 -0400 Subject: [Distutils] ReST formatting errors when registering package can invalidate HTML conversion In-Reply-To: <94bdd2610804060202i171abc28l1650f396a24c7ede@mail.gmail.com> References: <04DC863B-771E-439D-ACAF-76373BE45D03@gmail.com> <94bdd2610804060202i171abc28l1650f396a24c7ede@mail.gmail.com> Message-ID: +1 I eventually found it just the way you describe Tarek, and with Ian's suggestion. On Sun, Apr 6, 2008 at 5:02 AM, Tarek Ziad? wrote: > I would be in favor of a warning message generated by PyPI when a > long_description > field is not reST compliant. I don't think it is a client-side job. > > In any case you can test your long_description field like this: > > $ python setup.py --long-description | rst2html.py --strict > /dev/null > > If something goes wrong in the conversion you'll get a warning here > > On Sun, Apr 6, 2008 at 2:19 AM, Noah Gift wrote: > > > I thought I would post a generally help tip that Ian Bicking helped me > > with. I couldn't figure out why my ReST formatted long_description > > field would not get converted to HTML when I registered my egg with > > the cheeseshop: > > > > python setup.py register > > > > It turns out I had a minor error in my ReST formatting, I didn't > > underline a headline with enough length. So, the moral of the story > > is if you wonder why you cannot get HTML formatted ReST to display > > when you register your egg, it is possible you have a formatting > > error. It was tricky for me to discover, because I did not get any > > error message when I uploaded to the cheeseshop, and I just missed the > > minor formatting error when I was viewing the converted HTML document > > I used to preview what I created. > > > > > > Noah Gift / http://noahgift.com > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > http://mail.python.org/mailman/listinfo/distutils-sig > > > > > > -- > Tarek Ziad? | Association AfPy | www.afpy.org > Blog FR | http://programmation-python.org > Blog EN | http://tarekziade.wordpress.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080406/d7efc5c7/attachment.htm From zooko at zooko.com Sun Apr 6 18:03:47 2008 From: zooko at zooko.com (zooko) Date: Sun, 6 Apr 2008 10:03:47 -0600 Subject: [Distutils] [setuptools] how to report bugs and patchs ? In-Reply-To: <94bdd2610804060310v5f6e894dja0da73186a6b56f@mail.gmail.com> References: <94bdd2610804060310v5f6e894dja0da73186a6b56f@mail.gmail.com> Message-ID: On Apr 6, 2008, at 4:10 AM, Tarek Ziad? wrote: > I would expect patch or bug reports to be added in a bug tracker, > then rejected or accepted, even if it takes > months for them to be treated. At least, registered somewhere. > > I have read that a bug tracker was going to be set for setuptools, > but the current status is rather fuzzy, > so I was just wondering if this is stilled planned... You can go ahead and submit your patches and bug reports on the trac that I set up: http://allmydata.org/trac/setuptools I haven't heard from anyone else that they are willing to use this trac instance, but on the other hand it already works and you can record your contributions for posterity. I'll make sure that all tickets and patches that have accumulated in that trac get moved if someone sets up a new issue tracker that gets more, um, traction. Regards, Zooko From d at 0x1.org Mon Apr 7 03:51:45 2008 From: d at 0x1.org (David Arnold) Date: Sun, 6 Apr 2008 21:51:45 -0400 Subject: [Distutils] [Python-Dev] Remove "current" Windows executables from Lib/distutils/command in svn? In-Reply-To: <47F86A84.6000204@v.loewis.de> References: <018701c8978e$ce620660$6b261320$@com.au> <47F86A84.6000204@v.loewis.de> Message-ID: On 06/04/2008, at 2:15 AM, Martin v. L?wis wrote: > a) how does the binary get into the release tarball? You might argue > that it doesn't have to, as it is sufficient when it is included in > the Windows installer, however, as currently implemented, > bdist_wininst also runs on Unix, and people use it that way. It would be a shame to lose this ability. It's very handy. d From theller at ctypes.org Mon Apr 7 08:45:43 2008 From: theller at ctypes.org (Thomas Heller) Date: Mon, 07 Apr 2008 08:45:43 +0200 Subject: [Distutils] Remove "current" Windows executables from Lib/distutils/command in svn? In-Reply-To: <47F86A84.6000204@v.loewis.de> References: <018701c8978e$ce620660$6b261320$@com.au> <47F86A84.6000204@v.loewis.de> Message-ID: <47F9C317.9060701@ctypes.org> Martin v. L?wis schrieb: >> I'd like to propose we delete Lib/Distutils/command/wininst-9.0.exe, and >> enable the building of that project by default in the standard build process >> (and I'll setup the x64 build of the executable similarly). > > There are two issues here: > a) how does the binary get into the release tarball? You might argue > that it doesn't have to, as it is sufficient when it is included in > the Windows installer, however, as currently implemented, > bdist_wininst also runs on Unix, and people use it that way. > b) IIRC, upx was used to compress these executables. I don't think the > the current build process covers this, yet (but then, the current > binaries might not be compressed with upx anymore, either - not > sure whether that would be a bug). Of course, upx would probably > not currently apply to Win64-amd64. I did not compress them with upx anymore because of licensing issues with the upx-created executables, but I do not remember the details anymore. Thomas From ziade.tarek at gmail.com Mon Apr 7 10:32:14 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 7 Apr 2008 10:32:14 +0200 Subject: [Distutils] Unicode in distutils meta-data Message-ID: <94bdd2610804070132k5ab9e73dn94a676e6f19947de@mail.gmail.com> Hello, I would like to summarize here http://bugs.python.org/issue2562. Currently, distutils does not allows the usage of Unicode for some meta-data fields that should be able to use it. These fields are: - author - maintainer - description - long_description For instance, if you use u"Barnab?" in the author field, the DistributionMetada.write_pkg_file will fail when it tries to serialize the information into a file. The problem is that the current implementation makes the assumption that all fields are ascii string. This won't be a problem in Python 3k, of course, but currently concerns the 2.x series. One possible solution would be to move these fields to Unicode for the 2.6 series. In the meantime, many people are using str type for those fields, so as Martin mentioned, a backward compatibility would be better to support either plain string either Unicode. Other fields should be left imho in ascii, since an url for instance has to be ascii. But maybe it would be better, as Martin mentioned, to use Unicode for all fields. In any case, if we do use Unicode for some fields, we will need to provide the codec to be used to serialize the data in a file. My proposal here would be to add a 'encoding' field in the metadata that defaults to 'utf8', and that would let people explicitely indicates the encoding. I have written a quick patch here of a possible implementation, so you can see the problem : http://bugs.python.org/file9967/unicode.metadata.patch Regards, Tarek -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080407/9f233548/attachment.htm From me at lbruno.org Mon Apr 7 11:53:44 2008 From: me at lbruno.org (Luis Bruno) Date: Mon, 07 Apr 2008 10:53:44 +0100 Subject: [Distutils] [setuptools] Windows AMD64 installation Message-ID: <47F9EF28.4060909@lbruno.org> Hi all, I installed setuptools-0.6c8 in a Python 2.5 AMD64 machine by dropping its .egg in site-packages, a 'temp.pth' file alongside pointing to it, and doing "python -m easy_install setuptools". Was there an easier way? From lists at cheimes.de Mon Apr 7 11:57:27 2008 From: lists at cheimes.de (Christian Heimes) Date: Mon, 07 Apr 2008 11:57:27 +0200 Subject: [Distutils] Remove "current" Windows executables from Lib/distutils/command in svn? In-Reply-To: <47F86A84.6000204@v.loewis.de> References: <018701c8978e$ce620660$6b261320$@com.au> <47F86A84.6000204@v.loewis.de> Message-ID: <47F9F007.2060903@cheimes.de> Martin v. L?wis schrieb: > b) IIRC, upx was used to compress these executables. I don't think the > the current build process covers this, yet (but then, the current > binaries might not be compressed with upx anymore, either - not > sure whether that would be a bug). Of course, upx would probably > not currently apply to Win64-amd64. The 9.0 binary hasn't been compressed with upx (yet). Christian From lists at cheimes.de Mon Apr 7 11:57:27 2008 From: lists at cheimes.de (Christian Heimes) Date: Mon, 07 Apr 2008 11:57:27 +0200 Subject: [Distutils] Remove "current" Windows executables from Lib/distutils/command in svn? In-Reply-To: <47F86A84.6000204@v.loewis.de> References: <018701c8978e$ce620660$6b261320$@com.au> <47F86A84.6000204@v.loewis.de> Message-ID: <47F9F007.2060903@cheimes.de> Martin v. L?wis schrieb: > b) IIRC, upx was used to compress these executables. I don't think the > the current build process covers this, yet (but then, the current > binaries might not be compressed with upx anymore, either - not > sure whether that would be a bug). Of course, upx would probably > not currently apply to Win64-amd64. The 9.0 binary hasn't been compressed with upx (yet). Christian From stef.mientki at gmail.com Mon Apr 7 13:36:25 2008 From: stef.mientki at gmail.com (Stef Mientki) Date: Mon, 07 Apr 2008 13:36:25 +0200 Subject: [Distutils] Py2exe, on PyGame gives problems ... Message-ID: <47FA0739.4000106@gmail.com> hello, I'm using the orginal Enthought suite 2.4.3 and added Pygame to it. Now everything works fine, until .. ... I try to build a windows executable with Py2Exe. All kinds of numarray modules are not detected by Py2Exe. Adding the missing numarray (which one, Enthought contains more of them), later on manual to the dirstibution didn't help. Does anyone recognize this problem, and maybe even have a solution ? thanks, Stef Mientki From lxander.m at gmail.com Mon Apr 7 23:05:11 2008 From: lxander.m at gmail.com (Alexander Michael) Date: Mon, 7 Apr 2008 17:05:11 -0400 Subject: [Distutils] A Modest Proposal for "A Database of Installed Packages" In-Reply-To: <20080406024917.8FEDF3A4042@sparrow.telecommunity.com> References: <525f23e80803280802r507322car646b6110799934b@mail.gmail.com> <20080405210749.GA7978@laurie.devork> <20080405235028.152133A4042@sparrow.telecommunity.com> <20080406011842.GA26228@laurie.devork> <20080406024917.8FEDF3A4042@sparrow.telecommunity.com> Message-ID: <525f23e80804071405j18db193ekf960c2757f64beed@mail.gmail.com> Rather than post my comments in-line, I will summarize what I see as the key points raised by the discussion over the weekend. 1. The strawman proposal did not explicitly mention how python packages (and modules) would be assigned to a distribution and make clear the distinction between packages and distributions 2. The strawman proposal did not explicitly address how optional add-on tools (like setuptools) might manage namespace packages. 3. PKG-INFO possibly makes a poor the conduit for the proposed installation metadata both because its usage in my original proposal confuses packages with distributions and its file format is perhaps inefficient for the purpose. 4. Concerns were raised about the performance penalty for using the side-car style files without version numbers possibly not all of which were located at the top-most level of the directory listed in the python path. I will respond to each of these in turn below. 1. The strawman proposal did not explicitly mention how python packages (and modules) would be assigned to a distribution and make clear the distinction between packages and distributions The unstated thought was that the side-car file would contain a line like: Provided-By: SomeDistribution that would assign the python package to a distribution. The side-car files would be named like the package, and there would no standard centralized database of distributions. The reasons for proposing it like this are: a. I believe that having side-car files that sit alongside packages because they have the same base name makes the database more transparent to the uninitiated. Just browsing a directory of python packages will allow you to see what's going on. Moving like-names files around manually maintains the integrity and availability of the data. I think that having magic entries in an essentially "hidden" directory somewhere will cause all sorts of trouble that could be avoiding at the cost of a small bit of duplication. b. I assume, perhaps incorrectly, that most distributions contain only a single package. That said, I do agree that if you are primarily interested in a database of *distributions* (as opposed to *packages*) then something like is proposed in PEP 262 makes more sense (but it would have to be per directory and not site-wide due to the dynamic nature of the python path). This is a trade-off between putting the metadata up front in an obvious and easy to understand way so that it will hopefully have a better chance of being noticed and maintained, versus tucking it away hidden someplace so that even though it is broken, it doesn't bother anyone until they care enough to fix it. *It is this trade-off that I am exploring with this strawman "counter" proposal to PEP 262.* 2. The strawman proposal did not explicitly address how optional add-on tools (like setuptools) might manage namespace packages. I agree with Floris that the best way to avoid magic is to actually have the sub-packages in a namespace share the same parent directory on disk. Since the goal of my proposal is to create the necessary metadata infrastructure so that add-on tools can be used to manage a standard python installation (i.e. no runtime support), I don't see any other way to support this feature in the proposal. Of course, non-standard features like zipped eggs and such could still be deployed using whatever tools and trickery are necessary to achieve the desired ends. To support this, we could indeed add a flag inside the side-car file indicating that the package is a namespace package and that one would need to recurse into it to see what is installed. Python-based installers could create the namespace directory on the fly by default or optionally when needed and system packagers could require a namespace system-level package. 3. PKG-INFO possibly makes a poor the conduit for the proposed installation metadata both because its usage in my original proposal confuses packages with distributions and its file format is perhaps inefficient for the purpose. Using PKG-INFO was just an attempt to be incremental and make use of what is already there. With the practice of including more than cursory documentation in the Description, perhaps it is too much and should be pared down for this purpose, or thrown out altogether if it really isn't the right thing. I'll address performance in the next point. 4. Concerns were raised about the performance penalty for using the side-car style files without version numbers possibly not all of which were located at the top-most level of the directory listed in the python path. Any add-on tool that actually used the data would very likely need to build a cache of the data using a more efficient representation, particularly if the add-on tool had distribution oriented view of the installation. The goal is not to support runtime scanning and manipulation of the data for use by add-on tools that work with the python path in non-standard ways, but to put in place a mechanism to merely make the metadata available for those who opt-in to the usage of such tools as well as for non-tool users to manually inspect. Once a user opts-in to such an add-on tool, they might be expected to use for all of their installations if they want to avoid rebuilding the database cache etc., but could always resync with whats on disk by explicitly rebuilding the database. From mhammond at skippinet.com.au Tue Apr 8 04:57:10 2008 From: mhammond at skippinet.com.au (Mark Hammond) Date: Tue, 8 Apr 2008 12:57:10 +1000 Subject: [Distutils] FW: [issue2581] Vista UAC/elevation support for bdist_wininst Message-ID: <031e01c89924$3ecc06d0$bc641470$@com.au> Hi all, I've just added this patch which adds basic UAC support to bdist_wininst in a fairly unobtrusive manner. I've added Thomas to the nosy list, but I'd welcome any feedback or concerns from everyone. Thanks, Mark -----Original Message----- From: Mark Hammond [mailto:report at bugs.python.org] Sent: Tuesday, 8 April 2008 12:43 PM To: mhammond at users.sourceforge.net Subject: [issue2581] Vista UAC/elevation support for bdist_wininst New submission from Mark Hammond : The attached patch adds basic UAC support to bdist_wininst created installers. A new option '--user-access-control' has been added to bdist_wininst, which is written to the INI file read by the installer. The installer reads this value: if it is 'force', elevation is always requested, if it is 'auto', elevation is requested when Python is installed in HKLM. 'none' (the default) means nothing UAC related happens at all. The elevation happens by having the installer re-execute itself using ShellExecute. I've also ensured the code builds for all versions of VS we support. As a result, it was necessary to change the old bdist_wininst project files to point to the later zlib version Python itself uses. All these changes are in the patch. ---------- assignee: mhammond components: Distutils files: bdist_wininst_uac.patch keywords: patch, patch messages: 65139 nosy: mhammond, theller severity: normal status: open title: Vista UAC/elevation support for bdist_wininst type: behavior versions: Python 2.6 Added file: http://bugs.python.org/file9978/bdist_wininst_uac.patch __________________________________ Tracker __________________________________ From pje at telecommunity.com Tue Apr 8 05:18:42 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 07 Apr 2008 23:18:42 -0400 Subject: [Distutils] A Modest Proposal for "A Database of Installed Packages" In-Reply-To: <525f23e80804071405j18db193ekf960c2757f64beed@mail.gmail.co m> References: <525f23e80803280802r507322car646b6110799934b@mail.gmail.com> <20080405210749.GA7978@laurie.devork> <20080405235028.152133A4042@sparrow.telecommunity.com> <20080406011842.GA26228@laurie.devork> <20080406024917.8FEDF3A4042@sparrow.telecommunity.com> <525f23e80804071405j18db193ekf960c2757f64beed@mail.gmail.com> Message-ID: <20080408031848.132E93A409D@sparrow.telecommunity.com> At 05:05 PM 4/7/2008 -0400, Alexander Michael wrote: >a. I believe that having side-car files that sit alongside >packages because they have the same base name makes the database more >transparent to the uninitiated. I'm not aware that this was ever a stated design goal, nor why it should have any priority. OTOH, files named by distribution would be at least as, if not even *more* transparent than package names, so I don't see any particular benefit to this. >Just browsing a directory of python >packages will allow you to see what's going on. Moving like-names >files around manually maintains the integrity and availability of the >data. Moving anything manually, other than the *entire* directory, will be unlikely to retain any form of integrity, so it's best not to give the false impression that it would. >I think that having magic entries in an essentially "hidden" >directory somewhere will cause all sorts of trouble that could be >avoiding at the cost of a small bit of duplication. > b. I assume, perhaps incorrectly, that most distributions contain >only a single package. Very incorrectly, unless you mean a single top-level package. Odds are fairly good that if there's a package, there's probably at least a subpackage, too, like perhaps a tests subpackage. >That said, I do agree that if you are primarily interested in a >database of *distributions* (as opposed to *packages*) then something >like is proposed in PEP 262 makes more sense (but it would have to be >per directory and not site-wide due to the dynamic nature of the >python path). That's exactly what I want. The only reason I didn't just implement easy_install using a per-directory form of PEP 262 is that I wanted something done rather more immediately. That was years ago, so I can afford to be more patient now. :) >This is a trade-off between putting the metadata up >front in an obvious and easy to understand way so that it will >hopefully have a better chance of being noticed and maintained, versus >tucking it away hidden someplace so that even though it is broken, it >doesn't bother anyone until they care enough to fix it. *It is this >trade-off that I am exploring with this strawman "counter" proposal to >PEP 262.* Someone would have to be crazy to maintain this information by hand. So I'd actually consider it an advantage if the file format made this fact plain, by using something that's difficult for a human being to maintain, like say a pickle. ;-) OTOH, it's possible that some system packagers will not wish to use Python to generate the files, so using something a bit less complex would be a good idea. The format proposed by PEP 262 isn't really that bad of a trade-off in those terms. >2. The strawman proposal did not explicitly address how optional >add-on tools (like setuptools) might manage namespace packages. I think there's some mistunderstanding here about the proposal's goals. If the proposal doesn't work for setuptools, it doesn't work, period. The entire point is to allow setuptools to do its work without annoying the people who don't want to use it. >I agree with Floris that the best way to avoid magic is to actually >have the sub-packages in a namespace share the same parent directory >on disk. I agree with this also. The issue is that an __init__.py must exist for this to happen, but most system packaging tools (e.g. RPM) require that a given file be owned by at most one system package (i.e., distribution), whereas the contents of a namespace package are assembled from multiple distributions. That's the problem that needs solving, not runtime support for the namespace itself. >4. Concerns were raised about the performance penalty for using the >side-car style files without version numbers possibly not all of which >were located at the top-most level of the directory listed in the >python path. > >Any add-on tool that actually used the data would very likely need to >build a cache of the data using a more efficient representation, This is a misunderstanding of the point I raised. Floris merely asked why there were version numbers in .egg-info files, and I answered him. That doesn't actually have much, if anything, to do with the package database proposal. It's merely how installed distributions' versions can be recognized quickly at runtime, not anything to do with how potential installation conflicts are handled at installation time. easy_install uses eggs for installation simply because it need never worry about file ownership conflicts. There is a direct mapping from a distribution to its files: the contents of a zipfile or subdirectory. This also allows for (relatively) straightforward uninstallation. The goal of the proposal, then, is to have a way for easy_install to have another way to map from a distribution to its owned files (and vice versa), so that eggs are not necessary for normal, single-version installations. From lists at zopyx.com Tue Apr 8 05:57:23 2008 From: lists at zopyx.com (Andreas Jung) Date: Tue, 08 Apr 2008 05:57:23 +0200 Subject: [Distutils] Unicode in distutils meta-data In-Reply-To: <94bdd2610804070132k5ab9e73dn94a676e6f19947de@mail.gmail.com> References: <94bdd2610804070132k5ab9e73dn94a676e6f19947de@mail.gmail.com> Message-ID: --On 7. April 2008 10:32:14 +0200 Tarek Ziad? wrote: > > My proposal here would be to add a 'encoding' field in the metadata > that defaults to 'utf8', and that would let people explicitely > +1 Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20080408/fb71ce33/attachment.pgp From zooko at zooko.com Tue Apr 8 19:01:23 2008 From: zooko at zooko.com (zooko) Date: Tue, 8 Apr 2008 10:01:23 -0700 Subject: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <47EB07A6.6040800@plope.com> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> Message-ID: <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> On Mar 26, 2008, at 7:34 PM, Chris McDonough wrote: > zooko wrote: http://mail.python.org/pipermail/python-dev/2008-March/078243.html >> Here is a simple proposal: make the standard Python "import" >> mechanism notice eggs on the PYTHONPATH and insert them (into the >> *same* location) on the sys.path. >> This eliminates the #1 problem with eggs -- that they don't >> easily work when installing them into places other than your site- >> packages and that if you allow any of them to be installed on >> your system then they take precedence over your non-egg packages >> even you explicitly put those other packages earlier in your >> PYTHONPATH. (That latter behavior is very disagreeable to more >> than a few prorgammers.) > > Sorry if I'm out of the loop and there's some subtlety here that > I'm disregarding, but it doesn't appear that either of the issues > you mention is a actually problem with eggs. These are instead > problems with how eggs get installed by easy_install (which uses > a .pth file to extend sys.path). It's reasonable to put eggs on > the PYTHONPATH manually (e.g. sys.path.append('/path/to/some.egg')) > instead of using easy_install to install them. Yes, you are missing something. While many programmers, such as yourself and Lennart Regebro (who posted to this thread) find the current eggs system to be perfectly convenient and to Just Work, many others, such as Glyph Lefkowitz (who posted to a related thread) find them to be so annoying that they actively ensure that no eggs are ever allowed to touch their system. The reasons for this latter problem are two: 1. You can't conveniently install eggs into a non-system directory, such as ~/my-python-stuff. 2. If you allow even a single egg to be installed into your PYTHONPATH, it will change the semantics of your PYTHONPATH. Both of these problems are directly caused by the need for eggs to hack your site.py. If Python automatically added eggs found in the PYTHONPATH to the sys.path, both of these problems would go away. I am skeptical that the current proposals to define a new database for installed packages will fare any better than the current eggs scheme does in this respect. This issue is important to me, because the benefits of eggs grow superlinearly with the number of good programmers who use them. They are a tool for re-using source code -- a tool for cooperation between programmers. To gain the greatest benefits at this point we do not need to add new features to eggs, we need to make them more palatable to more good programmers. I am skeptical that prorgammers are going to be willing to use a new database format. They already have a database -- their filesystem -- and they already have the tools to control it -- mv, rm, and PYTHONPATH. Many of them already hate the existence the "easy_instlal.pth" database file, and I don't see why a new database file would be any different. My proposal makes the current benefits of eggs -- clean, easy code re- use among programmers -- more compatible with their current tools -- mv, rm, and PYTHONPATH. It is also forward-compatible with more sophisticated proposals to add features like uninstall and operating system integration. By the way, since I posted my proposal two weeks ago I have pointed a couple of Python hackers who currently refuse to use eggs at the URL: http://mail.python.org/pipermail/python-dev/2008-March/078243.html They both agreed that it made perfect sense. I told one of them about the alternate proposal to define a new database file to contain a list of installed packages, and he sighed and rolled his eyes and said "So they are planning to reinvent apt!". Regards, Zooko From pje at telecommunity.com Tue Apr 8 19:42:08 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 08 Apr 2008 13:42:08 -0400 Subject: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> Message-ID: <20080408174307.79E453A4108@sparrow.telecommunity.com> At 10:01 AM 4/8/2008 -0700, zooko wrote: >On Mar 26, 2008, at 7:34 PM, Chris McDonough wrote: > > zooko wrote: > >http://mail.python.org/pipermail/python-dev/2008-March/078243.html > > >> Here is a simple proposal: make the standard Python "import" > >> mechanism notice eggs on the PYTHONPATH and insert them (into the > >> *same* location) on the sys.path. > >> This eliminates the #1 problem with eggs -- that they don't > >> easily work when installing them into places other than your site- > >> packages and that if you allow any of them to be installed on > >> your system then they take precedence over your non-egg packages > >> even you explicitly put those other packages earlier in your > >> PYTHONPATH. (That latter behavior is very disagreeable to more > >> than a few prorgammers.) > > > > Sorry if I'm out of the loop and there's some subtlety here that > > I'm disregarding, but it doesn't appear that either of the issues > > you mention is a actually problem with eggs. These are instead > > problems with how eggs get installed by easy_install (which uses > > a .pth file to extend sys.path). It's reasonable to put eggs on > > the PYTHONPATH manually (e.g. sys.path.append('/path/to/some.egg')) > > instead of using easy_install to install them. > >Yes, you are missing something. While many programmers, such as >yourself and Lennart Regebro (who posted to this thread) find the >current eggs system to be perfectly convenient and to Just Work, many >others, such as Glyph Lefkowitz (who posted to a related thread) find >them to be so annoying that they actively ensure that no eggs are >ever allowed to touch their system. > >The reasons for this latter problem are two: > >1. You can't conveniently install eggs into a non-system directory, >such as ~/my-python-stuff. Wha? >2. If you allow even a single egg to be installed into your >PYTHONPATH, it will change the semantics of your PYTHONPATH. Only in the same way that manually putting an egg on the front of PYTHONPATH can be considered to "change the semantics" of your PYTHONPATH. >Both of these problems are directly caused by the need for eggs to >hack your site.py. If Python automatically added eggs found in the >PYTHONPATH to the sys.path, both of these problems would go away. And add new ones. >I am skeptical that the current proposals to define a new database >for installed packages will fare any better than the current eggs >scheme does in this respect. The purpose for the installation database is to allow easy_install to eschew the use of .egg files or directories for anything other than multi-version installs. Thus, no need to add those .egg files or directories to the head of the PYTHONPATH. Conflicts would be handled at install time rather than runtime, in other words. >I am skeptical that prorgammers are going to be willing to use a new >database format. They already have a database -- their filesystem -- >and they already have the tools to control it -- mv, rm, and >PYTHONPATH. Many of them already hate the existence the >"easy_instlal.pth" database file, and I don't see why a new database >file would be any different. PEP 262 does not propose a database file -- it proposes the inclusion of a metadata file for each installed distribution. >My proposal makes the current benefits of eggs -- clean, easy code re- >use among programmers -- more compatible with their current tools -- >mv, rm, and PYTHONPATH. It is also forward-compatible with more >sophisticated proposals to add features like uninstall and operating >system integration. Actually, your current proposal doesn't work, unless you at least have some way to indicate which *version* of an egg should be automatically added to sys.path -- and some way to change that. Otherwise, you might as well use the -m option to easy_install, and require() the eggs at runtime. (Which needs neither .pth files nor site.py hacking.) Meanwhile, my understanding is that the people who dislike eggs, dislike them because when they install a setuptools-based package, it's installed as an egg by default. The installation database proposal (and by the way, people really should read and understand PEP 262, including the open issues, before trying to compete with it), will allow setuptools-based packages to install the "old-fashioned" way by default. That is, not as eggs. Similarly, easy_install would be able to skip installing .eggs unless you wanted multi-version support. So, people who don't like eggs would never see them, since the only way you'd ever get them would be via easy_install -m, and they would never use it. >By the way, since I posted my proposal two weeks ago I have pointed a >couple of Python hackers who currently refuse to use eggs at the URL: > >http://mail.python.org/pipermail/python-dev/2008-March/078243.html > >They both agreed that it made perfect sense. I told one of them >about the alternate proposal to define a new database file to contain >a list of installed packages, and he sighed and rolled his eyes and >said "So they are planning to reinvent apt!". No, we're planning to make it possible for easy_install not to overwrite things that would break your system, and allow distutils and setuptools to uninstall what they installed. That's a considerably less ambitious goal, by far. :) From p.f.moore at gmail.com Tue Apr 8 19:43:55 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 8 Apr 2008 18:43:55 +0100 Subject: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> Message-ID: <79990c6b0804081043ld93ea2fua01b9eb96be9f5c@mail.gmail.com> On 08/04/2008, zooko wrote: > By the way, since I posted my proposal two weeks ago I have pointed a > couple of Python hackers who currently refuse to use eggs at the URL: > > http://mail.python.org/pipermail/python-dev/2008-March/078243.html > > They both agreed that it made perfect sense. I told one of them > about the alternate proposal to define a new database file to contain > a list of installed packages, and he sighed and rolled his eyes and > said "So they are planning to reinvent apt!". I do think that a simple solution like that has some merit. It has two significant downsides, however: 1. It requires that core Python "bless" the egg format to some extent - something Guido has said he is unwilling to do. 2. It ignores the issue of package management completely. Personally, I avoid anything that doesn't integrate with a unified package manager (whether that be the Windows add/remove feature, or an as-yet-to-be-built custom Python package manager is not important). Filesystem commands do not a package manager make... Paul. From chrism at plope.com Tue Apr 8 19:52:19 2008 From: chrism at plope.com (Chris McDonough) Date: Tue, 08 Apr 2008 13:52:19 -0400 Subject: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> Message-ID: <47FBB0D3.2090405@plope.com> zooko wrote: > > On Mar 26, 2008, at 7:34 PM, Chris McDonough wrote: >> zooko wrote: > > http://mail.python.org/pipermail/python-dev/2008-March/078243.html > >>> Here is a simple proposal: make the standard Python "import" >>> mechanism notice eggs on the PYTHONPATH and insert them (into the >>> *same* location) on the sys.path. >>> This eliminates the #1 problem with eggs -- that they don't easily >>> work when installing them into places other than your site-packages >>> and that if you allow any of them to be installed on your system >>> then they take precedence over your non-egg packages even you >>> explicitly put those other packages earlier in your PYTHONPATH. >>> (That latter behavior is very disagreeable to more than a few >>> prorgammers.) >> >> Sorry if I'm out of the loop and there's some subtlety here that I'm >> disregarding, but it doesn't appear that either of the issues you >> mention is a actually problem with eggs. These are instead problems >> with how eggs get installed by easy_install (which uses a .pth file to >> extend sys.path). It's reasonable to put eggs on the PYTHONPATH >> manually (e.g. sys.path.append('/path/to/some.egg')) instead of using >> easy_install to install them. > > Yes, you are missing something. While many programmers, such as > yourself and Lennart Regebro (who posted to this thread) find the > current eggs system to be perfectly convenient and to Just Work, many > others, such as Glyph Lefkowitz (who posted to a related thread) find > them to be so annoying that they actively ensure that no eggs are ever > allowed to touch their system. > > The reasons for this latter problem are two: > > 1. You can't conveniently install eggs into a non-system directory, > such as ~/my-python-stuff. That's just not true. $ export PYTHONPATH=/home/you/my-python-stuff/foo-1.3.egg $ python >>> import foo Eggs are directories (or zipfiles) that contain packages. They happen to contain other metadata directories too, but these can be ignored if you just want to *use* them (as opposed to wanting to introspect metadata about them). > 2. If you allow even a single egg to be installed into your PYTHONPATH, > it will change the semantics of your PYTHONPATH. I think you are mistaken. The use of the .pth file that changes sys.path is a feature of easy_install, not of eggs. You don't need to use any .pth file to put eggs on your PYTHONPATH. Note that zc.buildout is a framework that installs eggs, and it doesn't rely at all on .pth files to automatically hack sys.path. Instead, it munges console scripts to add each egg dir to sys.path. This is pretty nasty too, but it does prove the point. It is however true that you need to change sys.path to use an egg. Is that what you're objecting to fundamentally? You just don't want to have to change sys.path at all to use an egg? Maybe you're objecting to the notion that an egg can contain more than one package. There is functionally no difference between an egg and a directory full of packages. > Both of these problems are directly caused by the need for eggs to hack > your site.py. If Python automatically added eggs found in the > PYTHONPATH to the sys.path, both of these problems would go away. I'm not sure what you mean. Eggs don't hack site.py. Eggs are just a packaging format. easy_install doesn't hack site.py either. It just puts a .pth file (the parsing of which is a feature of "core" Python itself, not any setuptools magic) in site packages and manages it. It seems like you're advocating adding magic that you can't turn off (magical detection of eggs in an already site.py-approved packages directory) to defeat magic that you can turn off (by not using easy_install or .pth files). At some level the magic you want to see built in to Python would almost certainly wind up doing what you hate by hacking sys.path unless you wanted to restrict eggs to containing a single package only. And you wouldn't be able to turn it off except through some obscure environment variable setting. > By the way, since I posted my proposal two weeks ago I have pointed a > couple of Python hackers who currently refuse to use eggs at the URL: > > http://mail.python.org/pipermail/python-dev/2008-March/078243.html > > They both agreed that it made perfect sense. I told one of them about > the alternate proposal to define a new database file to contain a list > of installed packages, and he sighed and rolled his eyes and said "So > they are planning to reinvent apt!". I think changing the Python core is the worst possible answer to this problem. "Don't use easy_install" is currently the best, AFAICT. - C From python at venix.com Tue Apr 8 20:27:39 2008 From: python at venix.com (Lloyd Kvam) Date: Tue, 08 Apr 2008 14:27:39 -0400 Subject: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> Message-ID: <1207679260.3474.74.camel@localhost.localdomain> On Tue, 2008-04-08 at 10:01 -0700, zooko wrote: > They both agreed that it made perfect sense. I told one of them > about the alternate proposal to define a new database file to > contain > a list of installed packages, and he sighed and rolled his eyes and > said "So they are planning to reinvent apt!". When I wear my sysadmin hat, eggs become a nuisance. They are not listed in the system packages; if zipped they won't work when the apache user tries to import them; easy_install can produce unexpected upgrades. The system package manager (apt or yum) is much preferred. As a developer, eggs are great. If a python module is not already available from my system packagers, easy_install will find it, get it, and install it. I waste almost no time with system administration issues while developing. Fortunately, distutils includes tools like bdist_rpm so that python modules can be packaged for easy processing by the system package manager. So once I need to switch back to a sysadmin role, I can use the system tools to install and track packages. -- Lloyd Kvam Venix Corp DLSLUG/GNHLUG library http://www.librarything.com/catalog/dlslug http://www.librarything.com/profile/dlslug http://www.librarything.com/rsshtml/recent/dlslug From zooko at zooko.com Tue Apr 8 20:46:27 2008 From: zooko at zooko.com (zooko) Date: Tue, 8 Apr 2008 11:46:27 -0700 Subject: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <1207679260.3474.74.camel@localhost.localdomain> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> <1207679260.3474.74.camel@localhost.localdomain> Message-ID: <236B9285-BE59-45CA-B727-74B3B3B764BA@zooko.com> On Apr 8, 2008, at 11:27 AM, Lloyd Kvam wrote: > > When I wear my sysadmin hat, eggs become a nuisance. ... > As a developer, eggs are great. ... > Fortunately, distutils includes tools like bdist_rpm so that python > modules can be packaged for easy processing by the system package > manager. So once I need to switch back to a sysadmin role, I can use > the system tools to install and track packages. This is the same experience I have. I rely on setuptools and eggs extensively in developing our software, and I use setuptools and eggs as the primary method of giving our source code to other programmers. But no software is ever installed on our production servers unless that software is in .deb form in an apt-gettable repository, and this requirement is unlikely to change for the forseeable future. Regards, Zooko From bignose+hates-spam at benfinney.id.au Wed Apr 9 03:37:07 2008 From: bignose+hates-spam at benfinney.id.au (Ben Finney) Date: Wed, 09 Apr 2008 11:37:07 +1000 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> Message-ID: <87bq4jolks.fsf@benfinney.id.au> zooko writes: > I am skeptical that prorgammers are going to be willing to use a new > database format. They already have a database -- their filesystem -- > and they already have the tools to control it -- mv, rm, and > PYTHONPATH. Many of them already hate the existence the > "easy_instlal.pth" database file, and I don't see why a new database > file would be any different. Moreover, many of us already have a database of *all* packages on the system, not just Python-language ones: the package database of our operating system. Adding another, parallel, database which needs separate maintenance, and only applies to Python packages, is not a step forward in such a situation. > They both agreed that it made perfect sense. I told one of them > about the alternate proposal to define a new database file to > contain a list of installed packages, and he sighed and rolled his > eyes and said "So they are planning to reinvent apt!". That's pretty much my reaction, too. -- \ "Contentment is a pearl of great price, and whosoever procures | `\ it at the expense of ten thousand desires makes a wise and | _o__) happy purchase." -- J. Balguy | Ben Finney From sklein at cpcug.org Wed Apr 9 04:49:26 2008 From: sklein at cpcug.org (Stanley A. Klein) Date: Tue, 8 Apr 2008 22:49:26 -0400 (EDT) Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: References: Message-ID: <54951.71.163.202.204.1207709366.squirrel@www.cpcug.org> On Tue, April 8, 2008 9:37 pm, Ben Finney wrote: > Date: Wed, 09 Apr 2008 11:37:07 +1000 > From: Ben Finney > Subject: Re: [Distutils] how to easily consume just the parts of eggs > that are good for you > To: Distutils-Sig at Python.Org > > > zooko writes: > >> I am skeptical that prorgammers are going to be willing to use a new >> database format. They already have a database -- their filesystem -- >> and they already have the tools to control it -- mv, rm, and >> PYTHONPATH. Many of them already hate the existence the >> "easy_instlal.pth" database file, and I don't see why a new database >> file would be any different. > > Moreover, many of us already have a database of *all* packages on the > system, not just Python-language ones: the package database of our > operating system. Adding another, parallel, database which needs > separate maintenance, and only applies to Python packages, is not a > step forward in such a situation. > >> They both agreed that it made perfect sense. I told one of them >> about the alternate proposal to define a new database file to >> contain a list of installed packages, and he sighed and rolled his >> eyes and said "So they are planning to reinvent apt!". > > That's pretty much my reaction, too. I have the same reaction. I don't install eggs (unless they are installed through my operating system package manager) or use easy_install. My systems have rpms (for Fedora and CENTOS) and debs (for Ubuntu). There are also a Linux Standards Base and a Unix Filesystem Hierarchy Standard (cited by the LSB) that rpms and debs generally enforce, but eggs often do not. I have tried in the past to use easy_install, but have run into problems because there is no communication between easy_install and the rpm database, resulting in failure of easy_install to recognize that dependencies have already been installed using rpms. Also, there are tools provided with rpm and apt to perform functions such as querying installed packages for their contents. I use "rpm -qil" frequently to find what files a package has installed on my system and where they are installed. A database focused only on Python packages is highly inappropriate for Linux systems, violates the Linux standards, and creates problems because eggs are not coordinated with the operating system package manager. The way to achieve a database for Python would be to provide tools for conversion of eggs to rpms and debs, to have eggs support conformance to the LSB and FHS, and to use rpm or apt for package management. Stan Klein From pje at telecommunity.com Wed Apr 9 06:41:32 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Apr 2008 00:41:32 -0400 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you Message-ID: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> At 10:49 PM 4/8/2008 -0400, Stanley A. Klein wrote: >On Tue, April 8, 2008 9:37 pm, Ben Finney > wrote: > > Date: Wed, 09 Apr 2008 11:37:07 +1000 > > From: Ben Finney > > Subject: Re: [Distutils] how to easily consume just the parts of eggs > > that are good for you > > To: Distutils-Sig at Python.Org > > > > > > zooko writes: > >> eyes and said "So they are planning to reinvent apt!". > > > > That's pretty much my reaction, too. > >I have the same reaction. I'm curious. Have any of you actually read PEP 262 in any detail? I have seen precious little discussion so far that doesn't appear to be based on significant misunderstandings of either the purpose of reviving the PEP, or the mechanics of its proposed implementation. >I have tried in the past to use easy_install, but have run into problems >because there is no communication between easy_install and the rpm >database, resulting in failure of easy_install to recognize that >dependencies have already been installed using rpms. This problem doesn't exist with Python 2.5, unless you're using a platform that willfully strips out the installation information that Python 2.5 provides for these packages. >A database focused only on Python packages is highly inappropriate for >Linux systems, violates the Linux standards, and creates problems because >eggs are not coordinated with the operating system package manager. The revamp of PEP 262 is aimed at removing .egg files and directories from the process, by allowing system packagers to tell Python what files belong to them and should not be messed with. And conversely, allowing systems and installation targets *without* package managers to safely manage their Python installations. > The >way to achieve a database for Python would be to provide tools for >conversion of eggs to rpms and debs, Such tools already exist, although the conversion takes place from source distributions rather than egg distributions. >to have eggs support conformance to >the LSB and FHS, Applying LSB and FHS to the innards of Python packages makes as much sense as applying them to the contents of Java .jar files -- i.e., none. If it's unchanging data that's part of a program or library, then it's a program or library, just like static data declared in a C program or library. Whether the file extension is .py, .so, or even .png is irrelevant. From gael.varoquaux at normalesup.org Wed Apr 9 09:54:16 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 9 Apr 2008 09:54:16 +0200 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <87bq4jolks.fsf@benfinney.id.au> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> <87bq4jolks.fsf@benfinney.id.au> Message-ID: <20080409075416.GA1761@phare.normalesup.org> On Wed, Apr 09, 2008 at 11:37:07AM +1000, Ben Finney wrote: > zooko writes: > > I am skeptical that prorgammers are going to be willing to use a new > > database format. They already have a database -- their filesystem -- > > and they already have the tools to control it -- mv, rm, and > > PYTHONPATH. Many of them already hate the existence the > > "easy_instlal.pth" database file, and I don't see why a new database > > file would be any different. > Moreover, many of us already have a database of *all* packages on the > system, not just Python-language ones: the package database of our > operating system. Adding another, parallel, database which needs > separate maintenance, and only applies to Python packages, is not a > step forward in such a situation. 90 % (at least) of the world does not have such database. I, and probably you, have such a very nice database. I works well, and we can choose to forget the problems our users are facing. It does not solve them though. In addition, packaging is system-specific. I recently had to learn some Debian packaging, because I wanted my Ubuntu and Debian users to be able to use my projects seamlessly. What about RPMs for RHEL, Fedora, Mandriva? ... and coronary packages? and MSIs? ... When do I find time to do development if I have to learn all this packaging. It would be fantastic to have an abstraction on all these packaging systems, including, as you point out, their database. I do agree that reusing the system packaging's database is great, and would be the best option for system-wide install. However one of the very neat features of setuptools and eggs is that you don't need administrator access to install the packages, and that is great in a shared environment, like a computation cluster. The system's database is thus unfortunately not a complete solution to the problem. My 2 cents, Ga?l From gael.varoquaux at normalesup.org Wed Apr 9 10:00:16 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 9 Apr 2008 10:00:16 +0200 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> Message-ID: <20080409080016.GB1761@phare.normalesup.org> On Wed, Apr 09, 2008 at 12:41:32AM -0400, Phillip J. Eby wrote: > >The way to achieve a database for Python would be to provide tools for > >conversion of eggs to rpms and debs, > Such tools already exist, although the conversion takes place from > source distributions rather than egg distributions. What is the status of the deb backend? The only one I know is unofficial maintained by Andrew Straw, but my information my be lagging behind. By the way, if these tools work well, they are priceless! Cheers, Ga?l From pje at telecommunity.com Wed Apr 9 15:00:01 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Apr 2008 09:00:01 -0400 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409080016.GB1761@phare.normalesup.org> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <20080409080016.GB1761@phare.normalesup.org> Message-ID: <20080409125953.99EE63A40D7@sparrow.telecommunity.com> At 10:00 AM 4/9/2008 +0200, Gael Varoquaux wrote: >On Wed, Apr 09, 2008 at 12:41:32AM -0400, Phillip J. Eby wrote: > > >The way to achieve a database for Python would be to provide tools for > > >conversion of eggs to rpms and debs, > > > Such tools already exist, although the conversion takes place from > > source distributions rather than egg distributions. > >What is the status of the deb backend? The only one I know is unofficial >maintained by Andrew Straw, but my information my be lagging behind. I was under the impression that there were 2 .deb tools, neither one "official" in any sense, any more than 'bdist_rpm' is really "official" for RPM-based systems. >By the way, if these tools work well, they are priceless! I haven't had need to use any of them, so I don't really know. From sklein at cpcug.org Wed Apr 9 17:52:53 2008 From: sklein at cpcug.org (Stanley A. Klein) Date: Wed, 9 Apr 2008 11:52:53 -0400 (EDT) Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> Message-ID: <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> On Wed, April 9, 2008 12:41 am, Phillip J. Eby wrote: > At 10:49 PM 4/8/2008 -0400, Stanley A. Klein wrote: >>On Tue, April 8, 2008 9:37 pm, Ben Finney >> wrote: >> > Date: Wed, 09 Apr 2008 11:37:07 +1000 >> > From: Ben Finney >> > Subject: Re: [Distutils] how to easily consume just the parts of eggs >> > that are good for you >> > To: Distutils-Sig at Python.Org >> > >> > >> > zooko writes: >> >> eyes and said "So they are planning to reinvent apt!". >> > >> > That's pretty much my reaction, too. >> >>I have the same reaction. > > I'm curious. Have any of you actually read PEP 262 in any detail? I > have seen precious little discussion so far that doesn't appear to be > based on significant misunderstandings of either the purpose of > reviving the PEP, or the mechanics of its proposed implementation. I haven't read the PEP at all. I generally don't read PEP's. > > >>I have tried in the past to use easy_install, but have run into problems >>because there is no communication between easy_install and the rpm >>database, resulting in failure of easy_install to recognize that >>dependencies have already been installed using rpms. > > This problem doesn't exist with Python 2.5, unless you're using a > platform that willfully strips out the installation information that > Python 2.5 provides for these packages. > IIRC, I have had the problem with Python 2.5 on Fedora 7. Until recently, Fedora packagers did strip out the egg information included with Python packages they packaged. I left those files in when packaging myself using bdist_rpm. However, are you implying that the installation information for Python egg packages accesses and coordinates with the rpm database? I found myself having to go into the setup.py for the relevant package(s) and delete any statements regarding dependencies. Otherwise, IIRC, the packaging couldn't proceed because the Python packaging tool couldn't find the dependencies that had already been installed as rpms. After installation, Python managed to find the relevant files, but the packaging tool couldn't. > >>A database focused only on Python packages is highly inappropriate for >>Linux systems, violates the Linux standards, and creates problems because >>eggs are not coordinated with the operating system package manager. > > The revamp of PEP 262 is aimed at removing .egg files and directories > from the process, by allowing system packagers to tell Python what > files belong to them and should not be messed with. And conversely, > allowing systems and installation targets *without* package managers > to safely manage their Python installations. IMHO, the main system without a package manager is Windows. A reasonable way to deal with Windows would be to create a package manager for it that could be used by Python and anyone else who wanted to use it. The package manager could establish a file hierarchy similar to the Unix FHS and install files appropriately, except for what is needed to satisfy the Windows OS. That would probably go a long way to addressing the issues being discussed here. This is primarily a Windows problem, not a Python problem. > > >> The >>way to achieve a database for Python would be to provide tools for >>conversion of eggs to rpms and debs, > > Such tools already exist, although the conversion takes place from > source distributions rather than egg distributions. > You are talking here about bdist_rpm and not about a tool that would take a Python package distributed as an egg file and convert the egg to an rpm or a deb. Unfortunately, some Python packagers are beginning to limit their focus only to egg distribution. That creates a problem for users who have native operating system package management. > >>to have eggs support conformance to >>the LSB and FHS, > > Applying LSB and FHS to the innards of Python packages makes as much > sense as applying them to the contents of Java .jar files -- i.e., > none. If it's unchanging data that's part of a program or library, > then it's a program or library, just like static data declared in a C > program or library. Whether the file extension is .py, .so, or even > .png is irrelevant. The FHS defines places to put specific kinds of files, such as command scripts (/bin, /usr/bin, /sbin, or /usr/sbin), documentation (/usr/share/doc/package-name), and configuration files (/etc). There are several kinds of files identified and places defined to put them. Distribution by eggs has a tendency to scoop up all of those files and put them in /usr/lib/python/site-packages, regardless of where they belong. Having eggs support conformance to FHS would mean recognizing and tagging the relevant files. A tool for converting eggs to rpms or debs would essentially reformat the egg to rpm or deb and put files where they belong. Stan Klein From p.f.moore at gmail.com Wed Apr 9 19:23:50 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 9 Apr 2008 18:23:50 +0100 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> Message-ID: <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> On 09/04/2008, Stanley A. Klein wrote: > IMHO, the main system without a package manager is Windows. A reasonable > way to deal with Windows would be to create a package manager for it that > could be used by Python and anyone else who wanted to use it. The package > manager could establish a file hierarchy similar to the Unix FHS and > install files appropriately, except for what is needed to satisfy the > Windows OS. That would probably go a long way to addressing the issues > being discussed here. This is primarily a Windows problem, not a Python > problem. Windows does have a package manager - the add/remove programs application. It's extremely limited, and doesn't make any attempt at doing dependency resolution, certainly - but that's a separate issue. I don't know if you use Windows (as in, develop programs using Python on Windows). If you do, then I'd be interested in your views on bdist_wininst and bdist_msi installers, and how they fit into the setuptools/egg environment, particularly with regard to the package manager you are proposing. If you don't use Windows, then I don't see how you can usefully comment. Personally, as I've said before, I don't have a problem with a Python-only package manager, as long as it replaces or integrates bdist_wininst and bdist_msi. Having two package managers is far worse than having none - and claiming that add/remove programs "isn't a package manager" is just ignoring reality (if it isn't, then why do bdist_wininst and bdist_msi exist?). Are the Linux users happy with having a Python package manager that ignores RPM/apt? Why should Windows users be any happier? Sorry - I'm feeling a little grumpy. I've read one too many "Windows is so broken that people who use it obviously don't care about doing things right" postings this week :-( Paul. From zooko at zooko.com Wed Apr 9 19:30:55 2008 From: zooko at zooko.com (zooko) Date: Wed, 9 Apr 2008 10:30:55 -0700 Subject: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> Message-ID: <593FD433-F1A0-4445-AA9D-BD7B75DE8125@zooko.com> On Apr 8, 2008, at 9:41 PM, Phillip J. Eby wrote: > > I'm curious. Have any of you actually read PEP 262 in any detail? I read it, though not in fine detail. I didn't write that you are planning to reinvent apt. I wrote that when programmers hear about this PEP they exclaim "They are planning to reinvent apt!". That is a matter of perception and marketing -- the value that I want to gain from Python packages is the value of a critical mass of good programmers using compatible tools for code re-use. If a lot of programmers hate an idea, then it doesn't matter what the details are -- it isn't going to provide this value to me. I think part of our disagreement is that we are talking about two overlapping use cases: programmer and sysadmin (and "end user" which is much like sysadmin). I am not, at this time, interested in the sysadmin use case. As I've mentioned, my sysadmin needs are currently well satisfied by apt (and sometimes by GNU stow), and my fellow sysadmins with whom I work are absolutely not going to relax their "apt-only policy" for the forseeable future, so I cannot use such a tool unless it is named "apt" and written by Debian/Ubuntu. On the other hand I am very interested in the programmer use case, because setuptools/easy_install already works pretty well for that, and we are already very close to achieving a critical mass of good programmers. Recently several more packages that my project [1] relies on have become easy_installable -- Twisted, pywin32 (thanks to you, PJE), and foolscap -- and several more have had bugfixes which make them work better with easy_install/setuptools -- Nevow and zope.interface -- and there are some patches in the queue to make another one compatible with setuptools -- pyflakes [2, 3]. So setuptools/easy_install is already (slowly) winning. I want to accelerate that process by reducing the degree to which it is incompatible, inconvenient, or objectionable to other programmers. PEP 262 sounds like a non-starter to me because 1. It appears to be backwards-incompatible with setuptools/ easy_install/eggs, thus losing all the recently gained cooperation that I mentioned in the previous paragraph, and 2. It defines a new database file, where I would prefer either: a. Doing away with database files entirely and relying on the filesystem alone to hold that information, or b. Continuing to use the current ".pth" database file format, possibly improved by having native support for .pth files in the Python import machinery. 3. Because of #2, it triggers programmers to exclaim "They are planning to reinvent apt!", thus making it unlikely that the new proposal will recapture the cooperation that setuptools has already (slowly) gained. I'm sorry, PJE -- I know it must be frustrating to you to have your proposal criticized on social rather than technical grounds -- but social benefits are what I am seeking right now. Perhaps PEP 262 and my proposal are not actually alternatives, but are complementary. I do not object to Python maintaining a database of installed packages for itself (although I cannot *rely* upon such behavior, not least because I will be maintaining backwards compatibility with Python 2.4 for at least the next several years, and with Python 2.5 for at least the next several years after that). What I want is for the already implemented, tested, and deployed code- re-use features of setuptools/easy_install to be more widely accepted. This is best and most easily achieved by fixing the two most frequent objections to setuptools/easy_install: 1. That you can't conveniently install into an arbitrary directory, and 2. that it subverts the meaning of your PYTHONPATH. Regards, Zooko [1] http://allmydata.org/source/tahoe/trunk/docs/install.html [2] http://divmod.org/trac/ticket/2535 [3] http://divmod.org/trac/ticket/2048 From sklein at cpcug.org Wed Apr 9 20:26:31 2008 From: sklein at cpcug.org (Stanley A. Klein) Date: Wed, 9 Apr 2008 14:26:31 -0400 (EDT) Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> Message-ID: <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> All my development is done on Linux. I use Windows very minimally (such as for tax preparation) and unless forced to do so for specific circumstances (such as submittal to grants.gov) do not expose Windows to the Internet. In the future there may possibly arise a need for us to port some Linux-developed Python code to Windows, but we will have to cross that bridge when we get there. I think you raise an interesting issue: What is a package manager? I have minimal experience installing packages on Windows over the last 5-10 years, but in my experience a Windows package comes as an executable that, when run, installs itself. Unless a third-party program monitors the installation, uninstalling is a nasty chore, as is finding out what files were installed or where they went. The rpm and deb package managers (and their yum and other higher level dependency managers) do a lot of things: 1. They install packages and maintain databases of what packages were installed 2. They manage dependencies 3. They support clean uninstalling of packages 4. They can query packages, both installed (via their databases) and not yet installed (e.g., as rpm or deb files), to determine attributes, such as files they install, dependencies, and other information defined at packaging time. 5. They build packages and (in some cases) can rebuild packages. 6. They can verify packages for integrity and security purposes. 7. They can download package files and maintain archives of installed package files for use as local repositories. There may be other functions, but the above is a top-of-the-head list. I can say that I'm not terribly happy with Python packaging that is only minimally compatible with rpm. I haven't used Ubuntu all that much. I do like Ubuntu's packaging and package management, and I do know that there are programs, such as alias, that can translate from rpm to deb formats. I don't think I ever said that Windows is broken in the area of package management. My own experience is that the files of Windows programs tend to be put in a directory devoted to the program, rather than put in directories with other files having similar purposes. At one time, the default location in Windows for word processing files was even in a sub-directory of the word processing program. That changed to having a form of user home directory, but it didn't change much for the program files themselves. Unix/Linux puts the files in specific areas of the file system having functional commonality. One could almost say that the Windows default approach to structuring its filesystem avoids or minimizes the need for package management. I repeat that this issue mainly arises because Windows doesn't have the same kind of filesystem structure (and therefore the need for package management) that other systems have. I don't know what Windows add/remove programs function does, but all it might do is to run the executable to install packages and record the installation (as was previously done by third party programs) to facilitate clean removal. Unless you can perform more of the other functions I listed above, I doubt I would call add/remove a package manager. Stan Klein On Wed, April 9, 2008 1:23 pm, Paul Moore wrote: > On 09/04/2008, Stanley A. Klein wrote: >> IMHO, the main system without a package manager is Windows. A >> reasonable >> way to deal with Windows would be to create a package manager for it >> that >> could be used by Python and anyone else who wanted to use it. The >> package >> manager could establish a file hierarchy similar to the Unix FHS and >> install files appropriately, except for what is needed to satisfy the >> Windows OS. That would probably go a long way to addressing the issues >> being discussed here. This is primarily a Windows problem, not a >> Python >> problem. > > Windows does have a package manager - the add/remove programs > application. It's extremely limited, and doesn't make any attempt at > doing dependency resolution, certainly - but that's a separate issue. > > I don't know if you use Windows (as in, develop programs using Python > on Windows). If you do, then I'd be interested in your views on > bdist_wininst and bdist_msi installers, and how they fit into the > setuptools/egg environment, particularly with regard to the package > manager you are proposing. If you don't use Windows, then I don't see > how you can usefully comment. > > Personally, as I've said before, I don't have a problem with a > Python-only package manager, as long as it replaces or integrates > bdist_wininst and bdist_msi. Having two package managers is far worse > than having none - and claiming that add/remove programs "isn't a > package manager" is just ignoring reality (if it isn't, then why do > bdist_wininst and bdist_msi exist?). > > Are the Linux users happy with having a Python package manager that > ignores RPM/apt? Why should Windows users be any happier? > > Sorry - I'm feeling a little grumpy. I've read one too many "Windows > is so broken that people who use it obviously don't care about doing > things right" postings this week :-( > > Paul. > -- From pje at telecommunity.com Wed Apr 9 21:40:53 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Apr 2008 15:40:53 -0400 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> Message-ID: <20080409194044.1A4E03A406A@sparrow.telecommunity.com> At 11:52 AM 4/9/2008 -0400, Stanley A. Klein wrote: >However, are you implying that the installation information for Python egg >packages accesses and coordinates with the rpm database? Yes, when the information isn't stripped out. Try a more recent Fedora. >IMHO, the main system without a package manager is Windows. You're ignoring shared environments and development environments. (Not to mention Mac OS.) > A reasonable >way to deal with Windows would be to create a package manager for it that >could be used by Python and anyone else who wanted to use it. Let us know when you've finished it, along with the one for Mac OS. :) Of course this still won't do anything for shared environments and development environments. >You are talking here about bdist_rpm and not about a tool that would take >a Python package distributed as an egg file and convert the egg to an rpm >or a deb. Unfortunately, some Python packagers are beginning to limit >their focus only to egg distribution. That creates a problem for users >who have native operating system package management. That is indeed a problem -- but it's a social one, not a technical one. It's trivial for the publisher of an egg to change their command line from "setup.py bdist_egg upload" to "setup.py sdist bdist_egg upload", as soon as their users (politely) request that they do so. > > Applying LSB and FHS to the innards of Python packages makes as much > > sense as applying them to the contents of Java .jar files -- i.e., > > none. If it's unchanging data that's part of a program or library, > > then it's a program or library, just like static data declared in a C > > program or library. Whether the file extension is .py, .so, or even > > .png is irrelevant. > >The FHS defines places to put specific kinds of files, such as command >scripts (/bin, /usr/bin, /sbin, or /usr/sbin), documentation >(/usr/share/doc/package-name), and configuration files (/etc). There are >several kinds of files identified and places defined to put them. >Distribution by eggs has a tendency to scoop up all of those files and put >them in /usr/lib/python/site-packages, regardless of where they belong. Eggs don't include documentation or configuration files, and they install scripts in script directories, so I don't get what you're talking about here. For any other data that a package accesses at runtime, my earlier comments apply. >Having eggs support conformance to FHS would mean recognizing and tagging >the relevant files. A tool for converting eggs to rpms or debs would >essentially reformat the egg to rpm or deb and put files where they >belong. No, because such files as you describe don't exist. If you think they do, then either you have misunderstood the nature of the files in question, or the developer has incorrectly placed non-runtime files in their installation tree. From pje at telecommunity.com Wed Apr 9 21:57:13 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Apr 2008 15:57:13 -0400 Subject: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you Message-ID: <20080409195704.32E5F3A406A@sparrow.telecommunity.com> At 10:30 AM 4/9/2008 -0700, zooko wrote: >PEP 262 sounds like a non-starter to me because > >1. It appears to be backwards-incompatible with setuptools/ >easy_install/eggs, thus losing all the recently gained cooperation >that I mentioned in the previous paragraph, and No. It provides a forward-compatibility path for the distutils, so that easy_install doesn't have to install things in .egg format in the future. There's no compatibility breakage at all. >2. It defines a new database file, No, it defines *files*. One file per installed distribution, containing (among other things) an installation manifest. >where I would prefer either: > a. Doing away with database files entirely and relying on the >filesystem alone to hold that information, or ...which is what PEP 262 *does*. Unfortunately, PEP 262's title is bad for marketing, as you've effectively pointed out. It would be better titled something "package installation manifests" or "package contents files", or something of that sort. > b. Continuing to use the current ".pth" database file format, >possibly improved by having native support for .pth files in the >Python import machinery. These mechanisms are orthogonal to this issue. >3. Because of #2, it triggers programmers to exclaim "They are >planning to reinvent apt!", thus making it unlikely that the new >proposal will recapture the cooperation that setuptools has already >(slowly) gained. Yeah, we need a new name. Everybody is going off of "database of installed packages" and thinking "apt", because they aren't paying any closer attention. However, given that we are discussing this on Python-Dev and distutils-sig, I do think it's reasonable to expect (if perhaps not reasonable to receive) that people discussing the PEP have *read* the freaking PEP first, prior to trashing it or offering alternatives. And it's not like I'm personally offended or anything -- I didn't even write the PEP in question. But what's the point of having PEPs if people read nothing but their titles? We could just delete everything but PEP 0. :) >Perhaps PEP 262 and my proposal are not actually alternatives, but >are complementary. As I've already pointed out, your proposal does not address multiple installed versions of a package, and I see no sane way to modify it to do so. >What I want is for the already implemented, tested, and deployed >code- re-use features of setuptools/easy_install to be more widely >accepted. This is best and most easily achieved by fixing the two >most frequent objections to setuptools/easy_install: 1. That you >can't conveniently install into an arbitrary directory, and 2. that >it subverts the meaning of your PYTHONPATH. As I've already stated, the only way for these problems to be fixed is for easy_install to not install files in .egg form -- which also solves the general objection to using .eggs in the first place. And the only way to do that, is to have a way to keep track of what files are installed. Rather than have easy_install come up with its own way of doing that, I would prefer to share a standard with the distutils. Hence, the PEP discussion. For earlier versions of Python, it will still be possible to install and uninstall with setuptools using this approach. You just won't be able to uninstall pure distutils-based packages, unless you installed them using easy_install. Meanwhile, it has occurred to me that the easiest way of handling compatibility is not to require that other packaging tools mark their files for non-removability, but simply not allow easy_install to remove or overwrite anything that *isn't* claimed by a manifest. In that way, easy_install would be immediately usable in the new mode, without any updates to Python or to system packaging tools. From sklein at cpcug.org Wed Apr 9 22:00:18 2008 From: sklein at cpcug.org (Stanley A. Klein) Date: Wed, 9 Apr 2008 16:00:18 -0400 (EDT) Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409191948.GA11856@phare.normalesup.org> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> <20080409191948.GA11856@phare.normalesup.org> Message-ID: <12119.207.188.248.157.1207771218.squirrel@www.cpcug.org> On Wed, April 9, 2008 3:19 pm, Gael Varoquaux wrote: > On Wed, Apr 09, 2008 at 02:26:31PM -0400, Stanley A. Klein wrote: >> The rpm and deb package managers (and their yum and other higher level >> dependency managers) do a lot of things: > >> 1. They install packages and maintain databases of what packages were >> installed >> 2. They manage dependencies >> 3. They support clean uninstalling of packages >> 4. They can query packages, both installed (via their databases) and >> not >> yet installed (e.g., as rpm or deb files), to determine attributes, such >> as files they install, dependencies, and other information defined at >> packaging time. >> 5. They build packages and (in some cases) can rebuild packages. >> 6. They can verify packages for integrity and security purposes. >> 7. They can download package files and maintain archives of installed >> package files for use as local repositories. > > You are collapsing three different functionalities in one: > > * Dealing with repositories and downloading: yum/apt > > * Installing + uninstalling packages, and dealing with system > consistency (thus checking the dependencies are available): rpm/dpkg > > * Building > > For me it is important that the 3 are separated: > > * I may want to download the dependencies of a package to burn to a CD > for a computer that does not have internet access. > > * I may want to send a tarball to a build server that does the building, > but no install (so as not to corrupt my working system). > > Cheers, > > Ga?l > Gael - The functionalities are combined in programs but are not necessarily required to be used all at the same time. I'm not that familiar with apt, but yum also installs, including downloading both a package and its dependencies. Yum also has a query capability (yum list, yum info). I think synaptic does the same thing yum does, and adds a GUI and search capabilities similar to yum info as well. The build capabilities of rpm were moved to rpmbuild, but the building remains part of the rpm system. IIRC, bdist_rpm actually calls rpmbuild as part of its processing. Also, IIRC, rpmbuild can build from a tarball if it contains an rpm spec. It does not install in the same process. That is a separate step. You would not corrupt your working system by building an rpm from a tarball on it. BTW, I would not want to do dependencies with rpm if yum is available. Doing dependencies with rpm is very difficult and it is easy to wind up in "dependency hell". Yum will find the dependencies and install them as long as they are in repositories that are registered in the yum configuration. I looked at "man yum" and couldn't find an option to download dependencies to the local repository without installing. However, if you did install a package and its dependencies, and if you have selected the option of retaining the cache and not cleaning it after installation, the rpms (e.g., for updates) are in /var/cache/yum/updates/packages/. They can be copied from there to a CD for a system without internet connectivity. Also, both Fedora and Ubuntu have software for building installable live CD's, although I don't know how they get their package files. Stan Klein From floris.bruynooghe at gmail.com Wed Apr 9 22:04:52 2008 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Wed, 9 Apr 2008 21:04:52 +0100 Subject: [Distutils] A Modest Proposal for "A Database of Installed Packages" In-Reply-To: <20080406024917.8FEDF3A4042@sparrow.telecommunity.com> References: <525f23e80803280802r507322car646b6110799934b@mail.gmail.com> <20080405210749.GA7978@laurie.devork> <20080405235028.152133A4042@sparrow.telecommunity.com> <20080406011842.GA26228@laurie.devork> <20080406024917.8FEDF3A4042@sparrow.telecommunity.com> Message-ID: <20080409200452.GA15562@laurie.devork> On Sat, Apr 05, 2008 at 10:49:24PM -0400, Phillip J. Eby wrote: > At 02:18 AM 4/6/2008 +0100, Floris Bruynooghe wrote: >> On Sat, Apr 05, 2008 at 07:50:19PM -0400, Phillip J. Eby wrote: >> > At 10:07 PM 4/5/2008 +0100, Floris Bruynooghe wrote: >> > (One comment, though: I really don't like the idea of extending PKG-INFO >> > to include installation data; it's only incidentally related and there >> > are other contexts in which we use PKG-INFO where having that data >> > included would make no sense. Plus, it's really not an ideal file format >> > for including data about a potentially rather large number of files.) >> >> That's fair. Blowing up the files with the PKG-INFO information in >> could have bad performance effects. rfc822 in the stdlib reads >> everything in memory AFAIK. >> >> >> Secondly I'm not sure how >> >> useful it is for the version number to be encoded in the filename. >> > >> > It's very useful for setuptools, as it avoids the need to open and parse >> > the file when searching for a suitable version of a desired package. >> >> Hmm, it's not that much work to read the contents of a .egg-info. >> Just seems odd to me to have this info in two places so close to each >> other. > > It allows pkg_resources to grok the entire contents of a directory using > only a single listdir operation -- not an unbounded number of > open-and-read operations. I'm still not thrilled. To quote the "Rejected Suggestions" section of PEP 262: "First, performance is probably not an extremely pressing concern as the database is only used when installing or removing software, a relatively infrequent task." Yet, it's a done fact so there's no point in me complaining about it - I'll live with it. >> The second part was introducing a "virtual project" for pure namespace >> packages, where the project name would have to be the same as the >> package name in order to find it. > > I think there would also need to be some prefix to the name, to prevent > confusion in the event that there exists a normal project name that > happens to use that package name. (Again: the two namespaces are > unrelated, so a new/reserved namespace would be required for these > virtual projects.) Sounds sensible. >> >> AFAIK this should cover namespace packages. >> > >> > Unfortunately, this doesn't fix the problem, since either *some* package >> > has to own the __init__.py, or there has to be a way for Python to treat >> > the directory as a package without one. And for system package managers >> > (esp. on Linux), some *one* system package must own the file - it can't >> > be owned by multiple system packages. >> >> With the format I suggested a package tool could detect on install if >> a required pure namespace package was already installed or still >> needed to be installed/created. Similar on removal it is possible to >> detect if the pure namespace package is still required (by checking if >> it's directory contains any other files then those provided by the >> namespace package) on removal of a sub-package. > > Again... some system packaging folks need to speak up on this, because > my understanding is that some tools simply can't do something like this. > They need to make explicit what a given package depends on, and install > that, not dynamically decide what dependencies something has. (And then > there is the possibility of a problem if a non-system packager installs > the namespace, and then you install a system package for something that > includes packages in that namespace.) As for dpkg it will just overwirte an existing __init__.py in the namespace package if it doesn't own it. It won't even tell you it did so (I was surprised at this). However --and I know you don't like this-- this still is no problem. What we are concerned here is that a user or sysadmin owned directory on the sys.path can be managed sanely. dpkg and co will keep out of those, they have /usr/lib to play in, and sysadmins or users should stay out of /usr/lib in their turn. What is needed to cooperate with system packagers is: 1. Detect existing packages on other directories of sys.path and accept them to satisfy dependencies on the distribution being installed. 2. Find a solution for a namespace package spread out over two directories of sys.path. >> Maybe we're making it too hard by wanting to cover *every* file >> installed by python projects? The main reason for this installdb, as >> I understand it, is so that a package tool can install a sub-project >> in a namespace package installed by someone else. And similarly that >> someone else doesn't wipe away the sub-package when it thinks it can >> remove the namespace package. > > It's not just about namespace packages, it's about any package or > module. We also want to know about installed scripts, data, etc., so > that they can be cleaned up by a tool that does uninstalls. No, it's only about namespace packages. Everything else is easy, each tool can keep their own database of installed package in a suitable location if it wants to do that. If you didn't install a file you don't remove it. >> Ah, this make me think of the people >> that complain on comp.lang.python that Python namespaces are too >> tightly bound to files and directories... It all makes sense now, we >> wouldn't even be having this discussion if a package could declare >> it's namespace in the code! ;-) > > Or if you could import from directories without needing there to be an > __init__.py, and Python supported namespace packages by default. Also good point. I'm sure people can come up with negative site-effect of this but I can't come up with any myself now. So any takers? Is this a possible option to solve the problem? What is the reason for requiring __init__.py? The longer this discussion goes on the less I like the idea of a full PEP 262 style database (I do admit that at first it seemed like a reasonable idea to me). One issue I've always had with it is that it suddenly stores management data in library directories (it should live in /var). The .egg-info files do already do this, but then they only really provide the sort of information that can be found in .so files of shared libraries but for python files. To summarise what I think are the issues: * Python packaging tools (distutils, setuptools) need to be able to detect packages on all sys.path directories and use them to satisfy dependencies. AIUI this is already done in Python 2.5 with the .egg-info files. * Python packaging tools need to be able to share namespace packages in a user owned sys.path/site-packages directory. Installation and removal of the __init__.py needs coordination between the different tools. This is what PEP 262 could solve, but it's not necesarily the best or most loved solution. * Namespace packages need to be able to be spread over multiple sys.path directories so that the system can provide part of it, the sysadmin some more and the user yet another sub-package. -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From floris.bruynooghe at gmail.com Wed Apr 9 22:21:09 2008 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Wed, 9 Apr 2008 21:21:09 +0100 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> Message-ID: <20080409202109.GB15562@laurie.devork> On Wed, Apr 09, 2008 at 02:26:31PM -0400, Stanley A. Klein wrote: > I don't know what Windows add/remove > programs function does, but all it might do is to run the executable to > install packages and record the installation (as was previously done by > third party programs) to facilitate clean removal. Unless you can perform > more of the other functions I listed above, I doubt I would call > add/remove a package manager. Ugh, you have yet to discover the horrors/wonders of MSI (I wish I was as naive as you here!). A properly installed windows program will install using an MSI database, registering each file, registry setting etc. Often a setup.exe will still interface with the MSI database in the background (they should, there's a C API for it too). MSI will even do stuff like reference counting how many programs need a certain file (in case you have something installed in a shared directory), figure out what to do on conflict etc. They have many anc complicated rules, options and features. As far as I am concerned MSI (and thus Add/Remove Programs) can be treated as a package manager in windows. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From pje at telecommunity.com Wed Apr 9 22:27:23 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Apr 2008 16:27:23 -0400 Subject: [Distutils] A Modest Proposal for "A Database of Installed Packages" In-Reply-To: <20080409200452.GA15562@laurie.devork> References: <525f23e80803280802r507322car646b6110799934b@mail.gmail.com> <20080405210749.GA7978@laurie.devork> <20080405235028.152133A4042@sparrow.telecommunity.com> <20080406011842.GA26228@laurie.devork> <20080406024917.8FEDF3A4042@sparrow.telecommunity.com> <20080409200452.GA15562@laurie.devork> Message-ID: <20080409202713.D337E3A406A@sparrow.telecommunity.com> At 09:04 PM 4/9/2008 +0100, Floris Bruynooghe wrote: >On Sat, Apr 05, 2008 at 10:49:24PM -0400, Phillip J. Eby wrote: > > At 02:18 AM 4/6/2008 +0100, Floris Bruynooghe wrote: > >> On Sat, Apr 05, 2008 at 07:50:19PM -0400, Phillip J. Eby wrote: > >> > At 10:07 PM 4/5/2008 +0100, Floris Bruynooghe wrote: > >> > (One comment, though: I really don't like the idea of extending PKG-INFO > >> > to include installation data; it's only incidentally related and there > >> > are other contexts in which we use PKG-INFO where having that data > >> > included would make no sense. Plus, it's really not an ideal > file format > >> > for including data about a potentially rather large number of files.) > >> > >> That's fair. Blowing up the files with the PKG-INFO information in > >> could have bad performance effects. rfc822 in the stdlib reads > >> everything in memory AFAIK. > >> > >> >> Secondly I'm not sure how > >> >> useful it is for the version number to be encoded in the filename. > >> > > >> > It's very useful for setuptools, as it avoids the need to open and parse > >> > the file when searching for a suitable version of a desired package. > >> > >> Hmm, it's not that much work to read the contents of a .egg-info. > >> Just seems odd to me to have this info in two places so close to each > >> other. > > > > It allows pkg_resources to grok the entire contents of a directory using > > only a single listdir operation -- not an unbounded number of > > open-and-read operations. > >I'm still not thrilled. To quote the "Rejected Suggestions" section >of PEP 262: "First, performance is probably not an extremely pressing >concern as the database is only used when installing or removing >software, a relatively infrequent task." > >Yet, it's a done fact so there's no point in me complaining about it - >I'll live with it. You're conflating .egg-info and PEP 262 -- there is no connection between the two, except the similarity of using a single file per installed distribution to implement a database of sorts. >What is needed to cooperate with system packagers is: > >1. Detect existing packages on other directories of sys.path and > accept them to satisfy dependencies on the distribution being > installed. This part is already handled by .egg-info. >2. Find a solution for a namespace package spread out over two > directories of sys.path. That part's easy - pkg_resources will already do that. It's the handling of namespace packages when they're being installed by a system packager where things get dicey. However, at least the existing .pth-based solution used by setuptools will work for a setuptools-based package. And setuptools could use a setuptools-based solution elsewhere (i.e., handle the overlapping packages' use of __init__.py). The only place where a problem could come in is if you install other namespace-packaged things to the same directory as your system package manager... but I suppose we could just say, "don't do that." :) >No, it's only about namespace packages. Everything else is easy, each >tool can keep their own database of installed package in a suitable >location if it wants to do that. If you didn't install a file you >don't remove it. Well, if that were true, then we could handle namespace packages in the same way. :) However, I would like setuptools and distutils at least, to use the same format or a compatible format. >The longer this discussion goes on the less I like the idea of a full >PEP 262 style database (I do admit that at first it seemed like a >reasonable idea to me). One issue I've always had with it is that it >suddenly stores management data in library directories (it should live >in /var). Keep in mind that there are platforms and use cases where the FHS makes no sense to start with. FHS is for systems, not applications, for example. Does Firefox split its user profile directories across lib, var, and etc? After all, they contain code, data, and configuration. >To summarise what I think are the issues: > >* Python packaging tools (distutils, setuptools) need to be able to > detect packages on all sys.path directories and use them to satisfy > dependencies. AIUI this is already done in Python 2.5 with the > .egg-info files. Yep. >* Python packaging tools need to be able to share namespace packages > in a user owned sys.path/site-packages directory. Installation and > removal of the __init__.py needs coordination between the different > tools. This is what PEP 262 could solve, but it's not necesarily > the best or most loved solution. Right; this really does seem to be the main issue. Setuptools solves it for "site" directories (e.g. site-packages) using .pth files, but it is not an ideal solution. It also won't work for non-site directories, which means I'd have to keep the site.py hack for PYTHONPATH dirs. Not having a uniform way to address it is also an implementation issue, since setuptools will need to know which way it's solving the problem. I suppose easy_install could detect the presence of other nspkg.pth files, and choose to use that method in that event. But I'd much rather get rid of the nspkg.pth files, as they are second only to the easy_install site.py hack in their nastiness. >* Namespace packages need to be able to be spread over multiple > sys.path directories so that the system can provide part of it, the > sysadmin some more and the user yet another sub-package. This part is already solved by pkg_resources, or for that matter, by pkgutil. (pkgutil is only suitable if you're not using eggs, though.) From floris.bruynooghe at gmail.com Wed Apr 9 22:42:03 2008 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Wed, 9 Apr 2008 21:42:03 +0100 Subject: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409195704.32E5F3A406A@sparrow.telecommunity.com> References: <20080409195704.32E5F3A406A@sparrow.telecommunity.com> Message-ID: <20080409204203.GC15562@laurie.devork> On Wed, Apr 09, 2008 at 03:57:13PM -0400, Phillip J. Eby wrote: > At 10:30 AM 4/9/2008 -0700, zooko wrote: > >where I would prefer either: > > a. Doing away with database files entirely and relying on the > >filesystem alone to hold that information, or > > ...which is what PEP 262 *does*. > > Unfortunately, PEP 262's title is bad for marketing, as you've > effectively pointed out. It would be better titled something > "package installation manifests" or "package contents files", or > something of that sort. Heh, *package* appears in both of your suggestions. Surely this means tools that want to support installing script or data files need to keep their own manifests separately. This would greatly simplify any PEP 262 style implementation. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From sklein at cpcug.org Wed Apr 9 22:43:32 2008 From: sklein at cpcug.org (Stanley A. Klein) Date: Wed, 9 Apr 2008 16:43:32 -0400 (EDT) Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409194044.1A4E03A406A@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> Message-ID: <12121.207.188.248.157.1207773812.squirrel@www.cpcug.org> On Wed, April 9, 2008 3:40 pm, Phillip J. Eby wrote: > At 11:52 AM 4/9/2008 -0400, Stanley A. Klein wrote: >>However, are you implying that the installation information for Python >> egg >>packages accesses and coordinates with the rpm database? > > Yes, when the information isn't stripped out. Try a more recent Fedora. > > >>IMHO, the main system without a package manager is Windows. > > You're ignoring shared environments and development > environments. (Not to mention Mac OS.) > I don't understand what you mean by "shared environments and development environments". I also don't know much about Mac OS, except that its underlying Darwin system is a version of BSD (that I assume would follow the Unix FHS). > >> A reasonable >>way to deal with Windows would be to create a package manager for it that >>could be used by Python and anyone else who wanted to use it. > > Let us know when you've finished it, along with the one for Mac OS. :) I have enough trouble with what I'm already doing. :-) > > Of course this still won't do anything for shared environments and > development environments. > > >>You are talking here about bdist_rpm and not about a tool that would take >>a Python package distributed as an egg file and convert the egg to an rpm >>or a deb. Unfortunately, some Python packagers are beginning to limit >>their focus only to egg distribution. That creates a problem for users >>who have native operating system package management. > > That is indeed a problem -- but it's a social one, not a technical > one. It's trivial for the publisher of an egg to change their > command line from "setup.py bdist_egg upload" to "setup.py sdist > bdist_egg upload", as soon as their users (politely) request that they do > so. > I agree that we are dealing with a combination of technical and social issues here. However, I think it takes a lot more understanding for a publisher to get everything straight. > >> > Applying LSB and FHS to the innards of Python packages makes as much >> > sense as applying them to the contents of Java .jar files -- i.e., >> > none. If it's unchanging data that's part of a program or library, >> > then it's a program or library, just like static data declared in a C >> > program or library. Whether the file extension is .py, .so, or even >> > .png is irrelevant. >> >>The FHS defines places to put specific kinds of files, such as command >>scripts (/bin, /usr/bin, /sbin, or /usr/sbin), documentation >>(/usr/share/doc/package-name), and configuration files (/etc). There are >>several kinds of files identified and places defined to put them. >>Distribution by eggs has a tendency to scoop up all of those files and >> put >>them in /usr/lib/python/site-packages, regardless of where they belong. > > Eggs don't include documentation or configuration files, and they > install scripts in script directories, so I don't get what you're > talking about here. For any other data that a package accesses at > runtime, my earlier comments apply. > But rpms and debs do include these files, plus manual pages, localization files and a lot of other ancillary stuff. IIRC, you once mentioned that you have a CENTOS system. Do an "rpm -qa |sort|less" to get an alphabetized list of your installed packages, and then an "rpm -qil" on some of the packages, and you will see the range of different kinds of files in there. > >>Having eggs support conformance to FHS would mean recognizing and tagging >>the relevant files. A tool for converting eggs to rpms or debs would >>essentially reformat the egg to rpm or deb and put files where they >>belong. > > No, because such files as you describe don't exist. If you think > they do, then either you have misunderstood the nature of the files > in question, or the developer has incorrectly placed non-runtime > files in their installation tree. > Most of the Python tarballs I have downloaded have all kinds of files in their installation trees. This is a major pain in the you-know-what for someone trying to use bdist_rpm and get proper, FHS-compliant rpms. If eggs are supposed to be strictly runtime files, I think very few developers actually understand that. Better yet, how do you define what should be included in an installation? It sounds like the egg concept doesn't include several kinds of files that rpm and deb would include in an installation. I think that may be an important issue here. Stan Klein From sklein at cpcug.org Wed Apr 9 23:22:17 2008 From: sklein at cpcug.org (Stanley A. Klein) Date: Wed, 9 Apr 2008 17:22:17 -0400 (EDT) Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: References: Message-ID: <12206.207.188.248.157.1207776137.squirrel@www.cpcug.org> On Wed, April 9, 2008 4:27 pm, distutils-sig-request at python.org wrote: > Message: 5 > Date: Wed, 9 Apr 2008 21:21:09 +0100 > From: Floris Bruynooghe > Subject: Re: [Distutils] how to easily consume just the parts of eggs > that are good for you > To: distutils-sig at python.org > > On Wed, Apr 09, 2008 at 02:26:31PM -0400, Stanley A. Klein wrote: >> I don't know what Windows add/remove >> programs function does, but all it might do is to run the executable to >> install packages and record the installation (as was previously done by >> third party programs) to facilitate clean removal. Unless you can >> perform >> more of the other functions I listed above, I doubt I would call >> add/remove a package manager. > > Ugh, you have yet to discover the horrors/wonders of MSI (I wish I was > as naive as you here!). A properly installed windows program will > install using an MSI database, registering each file, registry setting > etc. Often a setup.exe will still interface with the MSI database in > the background (they should, there's a C API for it too). MSI will > even do stuff like reference counting how many programs need a certain > file (in case you have something installed in a shared directory), > figure out what to do on conflict etc. They have many anc complicated > rules, options and features. > > As far as I am concerned MSI (and thus Add/Remove Programs) can be > treated as a package manager in windows. > I have 3 desktops and a laptop. They are all at least dual boot. One of the systems on each machine is Windows. All the others are Linux, including Fedora 7, Fedora Core 5, Ubuntu 7, or CENTOS 5 in some combination on each machine. My greatest use of Windows is at tax time, because the good tax programs aren't released for Linux. I also have a mid-1990's version of QuickBooks that I still use. Aside from those applications, my use of Windows is sporadic at best, maybe a few times every few months. I do everything else in Linux. My exposure to Windows is minimal, so my exposure to MSI is even less. I wouldn't call it naivete. I just don't do Windows. :-) Stan Klein From pje at telecommunity.com Wed Apr 9 23:25:31 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Apr 2008 17:25:31 -0400 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <12121.207.188.248.157.1207773812.squirrel@www.cpcug.org> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> <12121.207.188.248.157.1207773812.squirrel@www.cpcug.org> Message-ID: <20080409212522.9CDE13A409E@sparrow.telecommunity.com> At 04:43 PM 4/9/2008 -0400, Stanley A. Klein wrote: >I don't understand what you mean by "shared environments and development > environments". I mean that in a shared or development environment, a system packager isn't useful, since it expects things to live in *one* place, and usually to have only one *version*, as well. >I agree that we are dealing with a combination of technical and social >issues here. However, I think it takes a lot more understanding for a >publisher to get everything straight. If they provide you with the source distribution, you can make any sort of package you want. > > Eggs don't include documentation or configuration files, and they > > install scripts in script directories, so I don't get what you're > > talking about here. For any other data that a package accesses at > > runtime, my earlier comments apply. > > > >But rpms and debs do include these files, plus manual pages, localization >files and a lot of other ancillary stuff. ...just one of the many reasons that eggs are not a replacement for rpms and debs. :) >Most of the Python tarballs I have downloaded have all kinds of files in >their installation trees. This is a major pain in the you-know-what for >someone trying to use bdist_rpm and get proper, FHS-compliant rpms. If >eggs are supposed to be strictly runtime files, I think very few >developers actually understand that. Better yet, how do you define what >should be included in an installation? It sounds like the egg concept >doesn't include several kinds of files that rpm and deb would include in >an installation. I think that may be an important issue here. It would be, if .eggs were a packaging format, rather than a binary distribution/runtime format. Remember "eggs are to Python as jars are to Java" -- a Java .jar doesn't contain documentation either, unless it's needed at runtime. Same for configuration files. They're not system packages, in other words. The assumption that they are is another marketing failure, due to conflation of "package == distribution of python code" and "package == thing you manage with a system packager". People see, "I put my package in an .egg" and think it's the latter definition, when it's barely even the former. :) From pje at telecommunity.com Wed Apr 9 23:39:54 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Apr 2008 17:39:54 -0400 Subject: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409204203.GC15562@laurie.devork> References: <20080409195704.32E5F3A406A@sparrow.telecommunity.com> <20080409204203.GC15562@laurie.devork> Message-ID: <20080409213945.A21493A409E@sparrow.telecommunity.com> At 09:42 PM 4/9/2008 +0100, Floris Bruynooghe wrote: >On Wed, Apr 09, 2008 at 03:57:13PM -0400, Phillip J. Eby wrote: > > At 10:30 AM 4/9/2008 -0700, zooko wrote: > > >where I would prefer either: > > > a. Doing away with database files entirely and relying on the > > >filesystem alone to hold that information, or > > > > ...which is what PEP 262 *does*. > > > > Unfortunately, PEP 262's title is bad for marketing, as you've > > effectively pointed out. It would be better titled something > > "package installation manifests" or "package contents files", or > > something of that sort. > >Heh, *package* appears in both of your suggestions. Surely this means >tools that want to support installing script or data files need to >keep their own manifests separately. This would greatly simplify any >PEP 262 style implementation. Sadly, "package" has the additional meaning of "thing you distribute" in a distutils "distribution". At this point, btw, I've pretty much come to the conclusion that the simplest way to solve the installdb problem is just going to be for setuptools to do its own thing, yet again. :( The only piece of the puzzle left, then, is the namespace package bit, but at least there the problem space has narrowed considerably. The only issue there will be how to make the new (non-egg) installation method interoperate with the existing nspkg.pth solution. And even there, the only problem to resolve is if you install something to a PYTHONPATH directory that shares a namespace with a system package. Currently, the only way I know of to fix that would be to ensure that the PYTHONPATH directory supports .pth files, and to install an appropriate .pth there as well. (Where "appropriate" means, an even hairier hack than today's nspkg.pth files.) I'd really like to avoid that, if possible. It would be best if we could solve the virtual-package thing in co-operation with the major system packagers (meaning the people, not just the tools). From zooko at zooko.com Thu Apr 10 00:11:01 2008 From: zooko at zooko.com (zooko) Date: Wed, 9 Apr 2008 15:11:01 -0700 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409125953.99EE63A40D7@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <20080409080016.GB1761@phare.normalesup.org> <20080409125953.99EE63A40D7@sparrow.telecommunity.com> Message-ID: <943A6CC1-6A89-4761-AB7A-332DCDCC7F81@zooko.com> On Apr 9, 2008, at 6:00 AM, Phillip J. Eby wrote: >> >> By the way, if these tools work well, they are priceless! > > I haven't had need to use any of them, so I don't really know. They are easydeb [1] and stddeb [2]. The former appears to be incomplete and unmaintained. The latter appears to be usable, but somewhat incomplete -- substantial manual labor is required to use it successfully, as documented by my programming partner Brian Warner in this ticket: [3]. Regards, Zooko [1] http://easy-deb.sourceforge.net/ [2] http://stdeb.python-hosting.com/ [3] http://allmydata.org/trac/tahoe/ticket/251 From zooko at zooko.com Thu Apr 10 00:20:30 2008 From: zooko at zooko.com (zooko) Date: Wed, 9 Apr 2008 15:20:30 -0700 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409194044.1A4E03A406A@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> Message-ID: <390014FF-5088-42C1-8213-17D0B25D2FFB@zooko.com> On Apr 9, 2008, at 12:40 PM, Phillip J. Eby wrote: >> >> You are talking here about bdist_rpm and not about a tool that >> would take >> a Python package distributed as an egg file and convert the egg to >> an rpm >> or a deb. Unfortunately, some Python packagers are beginning to >> limit >> their focus only to egg distribution. That creates a problem for >> users >> who have native operating system package management. > > That is indeed a problem -- but it's a social one, not a technical > one. It's trivial for the publisher of an egg to change their > command line from "setup.py bdist_egg upload" to "setup.py sdist > bdist_egg upload", as soon as their users (politely) request that > they do so. In general, it would be good if eggs came with .py files by default instead of .pyc files. I've opened a ticket on my setuptools trac about this proposal: http://allmydata.org/trac/setuptools/ticket/5 # binary eggs should come with .py files by default, rather than .pyc files Regards, Zooko From p.f.moore at gmail.com Thu Apr 10 00:46:19 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 9 Apr 2008 23:46:19 +0100 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> Message-ID: <79990c6b0804091546p5cf67f2ta800c675e7baa256@mail.gmail.com> On 09/04/2008, Stanley A. Klein wrote: > I think you raise an interesting issue: What is a package manager? My (very simplistic) answer is that it's whatever someone uses to manage packages. What level of functionality it has is irrelevant, as long as it suits an individual's way of working. > I don't think I ever said that Windows is broken in the area of package > management. I hope I didn't say you had - but it is an often-raised point, and it does grate on me (as by implication, it says that what I do isn't "real" package management). It's just another flavour of the old Windows vs Linux OS wars, which I don't want to participate in :-) > My own experience is that the files of Windows programs tend > to be put in a directory devoted to the program, rather than put in > directories with other files having similar purposes. At one time, the > default location in Windows for word processing files was even in a > sub-directory of the word processing program. That changed to having a > form of user home directory, but it didn't change much for the program > files themselves. Unix/Linux puts the files in specific areas of the file > system having functional commonality. One could almost say that the > Windows default approach to structuring its filesystem avoids or minimizes > the need for package management. Agreed. The minimal package manager on Windows is arguably reasonable precisely because the standard layout doesn't require much more. On the other hand, Microsoft has a bad habit of changing their "standards", and in particular Vista changes a lot of things. But that's very off-topic, so I'll avoid going into detail here. > I repeat that this issue mainly arises because Windows doesn't have the > same kind of filesystem structure (and therefore the need for package > management) that other systems have. I don't know what Windows add/remove > programs function does, but all it might do is to run the executable to > install packages and record the installation (as was previously done by > third party programs) to facilitate clean removal. Precisely. I could argue that the Linix filesystem structure is over complex, and causes the need for complex package managers, precisely because it's impossible to manually keep track of what file is owned by what package. But this way lies OS wars, so I won't make a major point of this, but just ask that people consider it as a possibility. (I believe that Mac OS X goes for an even simpler structure - applications store *everything* in the one directory, so that install/uninstall is simply a directory copy/remove. I won't comment on what that proves...) > Unless you can perform > more of the other functions I listed above, I doubt I would call > add/remove a package manager. OK. I would, though, as I don't really want much more. Later, you said: > I just don't do Windows. :-) And you've been pretty clear that you're looking for information rather than trying to explain how you think Windows should work. But many people aren't so balanced in their views, and that makes it hard to have a discussion - I'd suggest that people without Windows experience leave the Windows side to the Windows people, and accept that when they say "XXX won't work for us", that the statement is probably true. I find this whole discussion hugely confusing, because a lot of people are stating opinions about environments which it seems they don't use, or know much about. I don't know how to avoid this, but it does make it highly unlikely that any practical progress will get made. Paul. From bignose+hates-spam at benfinney.id.au Thu Apr 10 00:47:06 2008 From: bignose+hates-spam at benfinney.id.au (Ben Finney) Date: Thu, 10 Apr 2008 08:47:06 +1000 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> Message-ID: <87iqyqir2t.fsf@benfinney.id.au> "Stanley A. Klein" writes: > IMHO, the main system without a package manager is Windows. AFAICT the MacOS platform also lacks in this area. > A reasonable way to deal with Windows would be to create a package > manager for it that could be used by Python and anyone else who > wanted to use it. [...] This is primarily a Windows problem, not a > Python problem. I'd rephrase this as: If you *must* re-invent package management for those legacy systems without it, please *don't* make it specific to Python. -- \ ?The right to search for truth implies also a duty; one must | `\ not conceal any part of what one has recognized to be true.? | _o__) ?Albert Einstein | Ben Finney From p.f.moore at gmail.com Thu Apr 10 00:48:57 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 9 Apr 2008 23:48:57 +0100 Subject: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409212522.9CDE13A409E@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> <12121.207.188.248.157.1207773812.squirrel@www.cpcug.org> <20080409212522.9CDE13A409E@sparrow.telecommunity.com> Message-ID: <79990c6b0804091548p533e4dear5fb500341e4b87af@mail.gmail.com> On 09/04/2008, Phillip J. Eby wrote: > It would be, if .eggs were a packaging format, rather than a binary > distribution/runtime format. > > Remember "eggs are to Python as jars are to Java" -- a Java .jar > doesn't contain documentation either, unless it's needed at > runtime. Same for configuration files. And yet, Java doesn't have an equivalent of easy_install for jar files, to my knowledge. If Python had eggs but no easy_install, maybe this whole discussion wouldn't be taking place. (I know I personally like the idea of egg-as-jar-file, but *hate* the idea of egg-as-dependency-handling-tool-and-everything-else). Paul. From p.f.moore at gmail.com Thu Apr 10 00:52:08 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 9 Apr 2008 23:52:08 +0100 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <87iqyqir2t.fsf@benfinney.id.au> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <87iqyqir2t.fsf@benfinney.id.au> Message-ID: <79990c6b0804091552h4ee7f9f6t5512ed5f5aa70d2a@mail.gmail.com> On 09/04/2008, Ben Finney wrote: > "Stanley A. Klein" writes: > > A reasonable way to deal with Windows would be to create a package > > manager for it that could be used by Python and anyone else who > > wanted to use it. [...] This is primarily a Windows problem, not a > > Python problem. > > I'd rephrase this as: If you *must* re-invent package management for > those legacy systems without it, please *don't* make it specific to > Python. And I would say that Windows doesn't have a problem. Are any Windows users proposing building a package management system for Windows (Python-specific or otherwise)? It's a genuine question - is this something that Windows users are after, or is it just Linux users trying to show Windows users what they are missing? (BTW, it's unreasonable to call Windows "legacy" - whatever that word means, Windows ain't it (yet...)) Paul. From gael.varoquaux at normalesup.org Thu Apr 10 00:51:38 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 10 Apr 2008 00:51:38 +0200 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <79990c6b0804091546p5cf67f2ta800c675e7baa256@mail.gmail.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> <79990c6b0804091546p5cf67f2ta800c675e7baa256@mail.gmail.com> Message-ID: <20080409225138.GH28763@phare.normalesup.org> On Wed, Apr 09, 2008 at 11:46:19PM +0100, Paul Moore wrote: > I find this whole discussion hugely confusing, because a lot of people > are stating opinions about environments which it seems they don't use, > or know much about. I don't know how to avoid this, but it does make > it highly unlikely that any practical progress will get made. I find that something that doesn't help at all the discussion move forward is that everybody has different usecases in mind, on different platforms, and is not interested in other people's usecases. Hopefuly I am wrong, Cheers, Ga?l From gael.varoquaux at normalesup.org Thu Apr 10 00:59:24 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 10 Apr 2008 00:59:24 +0200 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <79990c6b0804091552h4ee7f9f6t5512ed5f5aa70d2a@mail.gmail.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <87iqyqir2t.fsf@benfinney.id.au> <79990c6b0804091552h4ee7f9f6t5512ed5f5aa70d2a@mail.gmail.com> Message-ID: <20080409225924.GI28763@phare.normalesup.org> On Wed, Apr 09, 2008 at 11:52:08PM +0100, Paul Moore wrote: > And I would say that Windows doesn't have a problem. Are any Windows > users proposing building a package management system for Windows > (Python-specific or otherwise)? It's a genuine question - is this > something that Windows users are after, or is it just Linux users > trying to show Windows users what they are missing? Well, users don't phrase this that way, because they don't know what package management (or rather automatic dependency tracking) is, but yes, they are some usecases. It is nowadays really tedious to deploy Python applications making uses of many packages on Python. The scientific community is a domain in which this problem is crucial, as we are trying to ship desktop applications to non-computer-savy people, with many dependencies outside the standard library. Enthought is working on shipping a Python distribution with some sort of package management for this purpose ( see http://code.enthought.com/enstaller/ ), and finding it is not an easy problem. Cheers, Gael From pje at telecommunity.com Thu Apr 10 01:12:55 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Apr 2008 19:12:55 -0400 Subject: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <390014FF-5088-42C1-8213-17D0B25D2FFB@zooko.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> <390014FF-5088-42C1-8213-17D0B25D2FFB@zooko.com> Message-ID: <20080409231247.7F3A23A406A@sparrow.telecommunity.com> At 03:20 PM 4/9/2008 -0700, zooko wrote: >I've opened a ticket on my setuptools trac about this proposal: > >http://allmydata.org/trac/setuptools/ticket/5 # binary eggs should >come with .py files by default, rather than .pyc files Filling your tracker with already-rejected proposals isn't likely to encourage me to look at it, especially when I've personally rejected them to you in IRC. That goes for your ticket #4 as well. From pje at telecommunity.com Thu Apr 10 01:16:20 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Apr 2008 19:16:20 -0400 Subject: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <79990c6b0804091548p533e4dear5fb500341e4b87af@mail.gmail.co m> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> <12121.207.188.248.157.1207773812.squirrel@www.cpcug.org> <20080409212522.9CDE13A409E@sparrow.telecommunity.com> <79990c6b0804091548p533e4dear5fb500341e4b87af@mail.gmail.com> Message-ID: <20080409231611.646C53A406A@sparrow.telecommunity.com> At 11:48 PM 4/9/2008 +0100, Paul Moore wrote: >On 09/04/2008, Phillip J. Eby wrote: > > It would be, if .eggs were a packaging format, rather than a binary > > distribution/runtime format. > > > > Remember "eggs are to Python as jars are to Java" -- a Java .jar > > doesn't contain documentation either, unless it's needed at > > runtime. Same for configuration files. > >And yet, Java doesn't have an equivalent of easy_install for jar >files, to my knowledge. Actually, OSGi and Eclipse plugins and "feature sites" come quite close, and setuptools rips off many of its features from them. OSGi is basically a standard for additional .jar metadata to encompass dependencies and other info. From dpeterson at enthought.com Thu Apr 10 01:17:49 2008 From: dpeterson at enthought.com (Dave Peterson) Date: Wed, 09 Apr 2008 18:17:49 -0500 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409194044.1A4E03A406A@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> Message-ID: <47FD4E9D.8060600@enthought.com> Phillip J. Eby wrote: > >>> Applying LSB and FHS to the innards of Python packages makes as much >>> sense as applying them to the contents of Java .jar files -- i.e., >>> none. If it's unchanging data that's part of a program or library, >>> then it's a program or library, just like static data declared in a C >>> program or library. Whether the file extension is .py, .so, or even >>> .png is irrelevant. >>> >> The FHS defines places to put specific kinds of files, such as command >> scripts (/bin, /usr/bin, /sbin, or /usr/sbin), documentation >> (/usr/share/doc/package-name), and configuration files (/etc). There are >> several kinds of files identified and places defined to put them. >> Distribution by eggs has a tendency to scoop up all of those files and put >> them in /usr/lib/python/site-packages, regardless of where they belong. >> > > Eggs don't include documentation or configuration files, and they > install scripts in script directories, so I don't get what you're > talking about here. For any other data that a package accesses at > runtime, my earlier comments apply. > We've talked a bit about this before, and IIRC, at that time you (Phillip) were willing to consider patches to setuptools that allowed for the inclusion of documentation in eggs such that it was placed into an LSB/FHS appropriate directory (or some standard dir for non-LSB systems) during the install process. I'm assuming that something similar for config files wouldn't be a problem either? Or is this whole idea out the window given the way the discussion has trended and the reiteration above that eggs are meant to be similar in principal to jars? Not that I have a patch yet, but we've been working on it in our "spare time" over here at Enthought. I'm now wondering if we're wasting our time. :-) I think the biggest use-case confusion in the current discussion is whether we're talking about applications or libraries? If we're talking about libraries, then clearly distribution of only executables is sufficient because anything else should be handled by the application distribution when that library is used in an app. Whereas if we're talking about applications, you probably *do* want to include documentation, configuration info, etc. in your distribution. I think I can sum up any further points by simply asking: "Should it be safe to assume I can distribute my application via eggs / easy_install just because it is written in Python?" -- Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080409/95acebba/attachment.htm From dpeterson at enthought.com Thu Apr 10 01:20:15 2008 From: dpeterson at enthought.com (Dave Peterson) Date: Wed, 09 Apr 2008 18:20:15 -0500 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <79990c6b0804091552h4ee7f9f6t5512ed5f5aa70d2a@mail.gmail.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <87iqyqir2t.fsf@benfinney.id.au> <79990c6b0804091552h4ee7f9f6t5512ed5f5aa70d2a@mail.gmail.com> Message-ID: <47FD4F2F.1000505@enthought.com> Paul Moore wrote: > On 09/04/2008, Ben Finney wrote: > >> "Stanley A. Klein" writes: >> > A reasonable way to deal with Windows would be to create a package >> > manager for it that could be used by Python and anyone else who >> >>> wanted to use it. [...] This is primarily a Windows problem, not a >>> >> > Python problem. >> >> I'd rephrase this as: If you *must* re-invent package management for >> those legacy systems without it, please *don't* make it specific to >> Python. >> > > And I would say that Windows doesn't have a problem. Are any Windows > users proposing building a package management system for Windows > (Python-specific or otherwise)? It's a genuine question - is this > something that Windows users are after, or is it just Linux users > trying to show Windows users what they are missing? > Depending on definition of package management system (which was recently debated here on the list,) I'd say the combination of setuptools, easy_install, and yolk make up a reasonable attempt at a system for Python projects (I prefer to use the term projects rather than avoid confusion between a distribution package and a python package.) Yes, they don't integrate with add/remove programs, but you can easily install a project by name and have it depend on other projects, list the installed projects, and there is a documented process to uninstall them as well. -- Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080409/8515a962/attachment-0001.htm From rrt at sc3d.org Thu Apr 10 01:09:45 2008 From: rrt at sc3d.org (Reuben Thomas) Date: Thu, 10 Apr 2008 00:09:45 +0100 (BST) Subject: [Distutils] Distutils setting permissions during build Message-ID: Hi, This is the only obvious up-to-date address or mailing list I could find, so feel free to direct me where I should go. I'm a Linux user, and normally I have a 0027 umask. This means that when I build a Python package using distutils, the files in the build directory are not world-readable (Python copies them there without preserving permissions, according to the comment in build_py.py: # XXX copy_file by default preserves mode, which appears to be the # wrong thing to do: if a file is read-only in the working # directory, we want it to be installed read/write so that the next # installation of the same module distribution can overwrite it # without problems. (This might be a Unix-specific issue.) Thus # we turn off 'preserve_mode' when copying to the build directory, # since the build directory is supposed to be exactly what the # installation will look like (ie. we preserve mode when # installing). This is all very well, but turning off "preserve_mode" is not enough to ensure this in the presence of a restrictive umask like mine; the permissions need rather to be explicitly set (as done for example by the GNU install tool, though that assumes that permissions are set at install time rather than build time). The obvious workaround is for me to remember to change my umask when I'm building python packages. That's OK, but is this problem recognised as such and might it be fixed in future? -- http://rrt.sc3d.org/ | God is the name of our ignorance (Grayling) From pje at telecommunity.com Thu Apr 10 01:24:46 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Apr 2008 19:24:46 -0400 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409225138.GH28763@phare.normalesup.org> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> <79990c6b0804091546p5cf67f2ta800c675e7baa256@mail.gmail.com> <20080409225138.GH28763@phare.normalesup.org> Message-ID: <20080409232437.7D74F3A406A@sparrow.telecommunity.com> At 12:51 AM 4/10/2008 +0200, Gael Varoquaux wrote: >On Wed, Apr 09, 2008 at 11:46:19PM +0100, Paul Moore wrote: > > I find this whole discussion hugely confusing, because a lot of people > > are stating opinions about environments which it seems they don't use, > > or know much about. I don't know how to avoid this, but it does make > > it highly unlikely that any practical progress will get made. > >I find that something that doesn't help at all the discussion move >forward is that everybody has different usecases in mind, on different >platforms, and is not interested in other people's usecases. > >Hopefuly I am wrong, You're not wrong at all. I have to deal with *all* the platforms and use cases, which makes it quite frustrating when people who haven't even read the requirements are making proposals to "solve" things in ways that break for everyone except their own niche platform+usecase combination. Guido, can I borrow the time machine and go back and NOT try to improve on the distutils? Or is there already too much collateral damage to the timeline? ;-) From zooko at zooko.com Thu Apr 10 02:27:22 2008 From: zooko at zooko.com (zooko) Date: Wed, 9 Apr 2008 17:27:22 -0700 Subject: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409231247.7F3A23A406A@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> <390014FF-5088-42C1-8213-17D0B25D2FFB@zooko.com> <20080409231247.7F3A23A406A@sparrow.telecommunity.com> Message-ID: <09411577-DD0F-4E0C-AE69-6CE4F1D97CD5@zooko.com> On Apr 9, 2008, at 4:12 PM, Phillip J. Eby wrote: >> http://allmydata.org/trac/setuptools/ticket/5 # binary eggs should >> come with .py files by default, rather than .pyc files > > Filling your tracker with already-rejected proposals isn't likely > to encourage me to look at it, especially when I've personally > rejected them to you in IRC. That goes for your ticket #4 as well. Part of my motivation in maintaining this tracker is to take issue discussions from IRC, and from mailing lists, and make them more permanent and structured. This part is useful even for rejected proposals, as an historical record that other people interested in those issues can consult. I will mark those two tickets as "rejected by PJE". Could you please repeat (so that I don't misrepresent you due to my faulty memory of our IRC discussion from more than a year ago) your reason for rejecting these two: http://allmydata.org/trac/setuptools/ticket/4 (when considering whether to zip, err on the side of safety rather than performance) http://allmydata.org/trac/setuptools/ticket/5 (binary eggs should come with .py files by default, rather than .pyc files) You are of course welcome to log in to that trac and update those tickets yourself, but if you prefer not to then I will paste your reasons into those tickets. Regards, Zooko From greg.ewing at canterbury.ac.nz Thu Apr 10 02:59:39 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 10 Apr 2008 12:59:39 +1200 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <79990c6b0804091546p5cf67f2ta800c675e7baa256@mail.gmail.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> <79990c6b0804091546p5cf67f2ta800c675e7baa256@mail.gmail.com> Message-ID: <47FD667B.9040602@canterbury.ac.nz> Paul Moore wrote: > I believe that Mac OS X goes for an even simpler structure - > applications store *everything* in the one directory, so that > install/uninstall is simply a directory copy/remove. Yep, and thereby cuts the whole gordian knot, throws the pieces on the fire and burns them. :-) Package managers have always seemed to me to be an excessively complex solution to a problem that needn't have existed in the first place. I keep hoping that someday Linux will support something like MacOSX application bundles and frameworks, but I haven't seen any sign of it yet. -- Greg From greg.ewing at canterbury.ac.nz Thu Apr 10 03:11:11 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 10 Apr 2008 13:11:11 +1200 Subject: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <79990c6b0804091552h4ee7f9f6t5512ed5f5aa70d2a@mail.gmail.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <87iqyqir2t.fsf@benfinney.id.au> <79990c6b0804091552h4ee7f9f6t5512ed5f5aa70d2a@mail.gmail.com> Message-ID: <47FD692F.2070308@canterbury.ac.nz> Paul Moore wrote: > And I would say that Windows doesn't have a problem. Are any Windows > users proposing building a package management system for Windows > (Python-specific or otherwise)? It's a genuine question - is this > something that Windows users are after, or is it just Linux users > trying to show Windows users what they are missing? I think the requirements for a package manager are different on different platforms. On Linux, you need to be able to cope with files scattered all over the system, and complex webs of dependencies between packages. On Windows, you need to be able to cope with scattered files and multiple applications sharing a file, but not usually with dependencies, because each application typically comes with all the files it needs (even if some of them might not get installed because they're already there for another application). On MacOSX, applications are usually completely self-contained, include all their dependencies and are not spread around, so there's really nothing for a package manager to do. What all this means for Python package management, I really don't know. Whatever is done, I'd like to see it kept as dirt-simple as possible. Ideal would be the MacOSX situation where the package is just a directory of files that you put somewhere obvious, and you can tell what it is just by looking at it, and get rid of it by dragging it to the trash -- so you don't need a package manager. -- Greg From lxander.m at gmail.com Thu Apr 10 03:54:05 2008 From: lxander.m at gmail.com (Alexander Michael) Date: Wed, 9 Apr 2008 21:54:05 -0400 Subject: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409194044.1A4E03A406A@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> Message-ID: <525f23e80804091854g19460b02pd426016b56177272@mail.gmail.com> On Wed, Apr 9, 2008 at 3:40 PM, Phillip J. Eby wrote: > That is indeed a problem -- but it's a social one, not a technical > one. It's trivial for the publisher of an egg to change their > command line from "setup.py bdist_egg upload" to "setup.py sdist > bdist_egg upload", as soon as their users (politely) request that they do so. I know you say it in , but perhaps you should be more explicit about the best practice being to distribute an sdist as well as one or more eggs and highlight this with some standout text like: """ When uploading your project to PyPI, always upload your sdist in addition to any eggs unless you have a good idiosyncratic reason not to. The easy_install tool can also download and install sdist's on all platforms (universally for pure-python packages and provided compilers are available for distributions containing python extensions), so uploading them gives your project the widest possible audience. """ Perhaps this will help some the social challenges. Regards, Alex From sklein at cpcug.org Thu Apr 10 03:58:57 2008 From: sklein at cpcug.org (Stanley A. Klein) Date: Wed, 9 Apr 2008 21:58:57 -0400 (EDT) Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <47FD4E9D.8060600@enthought.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> <47FD4E9D.8060600@enthought.com> Message-ID: <43869.71.178.83.244.1207792737.squirrel@www.cpcug.org> On Wed, 2008-04-09 at 18:17 -0500, Dave Peterson wrote: > I think I can sum up any further points by simply asking: "Should it > be safe to assume I can distribute my application via eggs / > easy_install just because it is written in Python?" I think that based on this discussion the bottom line answer to this question is "No". Stan Klein -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080409/ac2a1381/attachment.htm From lxander.m at gmail.com Thu Apr 10 04:05:01 2008 From: lxander.m at gmail.com (Alexander Michael) Date: Wed, 9 Apr 2008 22:05:01 -0400 Subject: [Distutils] A Modest Proposal for "A Database of Installed Packages" In-Reply-To: <20080408031848.132E93A409D@sparrow.telecommunity.com> References: <525f23e80803280802r507322car646b6110799934b@mail.gmail.com> <20080405210749.GA7978@laurie.devork> <20080405235028.152133A4042@sparrow.telecommunity.com> <20080406011842.GA26228@laurie.devork> <20080406024917.8FEDF3A4042@sparrow.telecommunity.com> <525f23e80804071405j18db193ekf960c2757f64beed@mail.gmail.com> <20080408031848.132E93A409D@sparrow.telecommunity.com> Message-ID: <525f23e80804091905m3a1b379bv58df56275ce2dfbf@mail.gmail.com> On Mon, Apr 7, 2008 at 11:18 PM, Phillip J. Eby wrote: > At 05:05 PM 4/7/2008 -0400, Alexander Michael wrote: > > > a. I believe that having side-car files that sit alongside > > packages because they have the same base name makes the database more > > transparent to the uninitiated. > > > > I'm not aware that this was ever a stated design goal, nor why it should > have any priority. OTOH, files named by distribution would be at least as, > if not even *more* transparent than package names, so I don't see any > particular benefit to this. Your right. I didn't state this in my rationale. I'm suggesting it *partly* because I thought that subverting the "database" aspect by inverting the relationship between distributions and packages would help solve the social problem. Now it could be that a technical solution to a social problem is suspect, but the other part of why I suggested it this way was an honest attempt at improving transparency by putting the metadata up front. I did indeed read PEP 262 and upon reading it decided it sounded too much like a "system packager" and not enough like a way to get the missing metadata out of PKG-INFO and into the installed packages. I appreciate your patience with me. I know I've earned little social capital in this community and am likely trying too hard to be helpful, but I'm earnestly trying to be helpful and I have read-up a fair bit in attempting to do so. That said, I think your probably right and inverting the distribution/package name relationship is too problematic, if only because of the redundancy it results in for distributions that contain more than one package (or worse, a slew of modules). This pretty much leaves us with the egg-info files I've been reading about. Since I use setuptools for everything but wxPython (whose WinXP installer doesn't seem to include them) and these aren't included in the standard library except for wsgiref, I don't really see these files, even though I see the code to produce them in distutils. But this is perhaps beside the point. > > Just browsing a directory of python > > packages will allow you to see what's going on. Moving like-names > > files around manually maintains the integrity and availability of the > > data. > > Moving anything manually, other than the *entire* directory, will be > unlikely to retain any form of integrity, so it's best not to give the false > impression that it would. I disagree with this. Certainly it decreases in likelihood when the side-car files are named by distribution and not package and if the distribution contains more than one package, but other than at, it seems pretty easy (e.g. hmm.. maybe I should move the mypkg.pkg-info file along with the mypkg directory. let's look inside. oh! I see how this works!) > > I think that having magic entries in an essentially "hidden" > > directory somewhere will cause all sorts of trouble that could be > > avoiding at the cost of a small bit of duplication. > > b. I assume, perhaps incorrectly, that most distributions contain > > only a single package. > > Very incorrectly, unless you mean a single top-level package. Odds are > fairly good that if there's a package, there's probably at least a > subpackage, too, like perhaps a tests subpackage. I do mean single top-level package (but one setup.py), thanks for clarifying. > > That said, I do agree that if you are primarily interested in a > > database of *distributions* (as opposed to *packages*) then something > > like is proposed in PEP 262 makes more sense (but it would have to be > > per directory and not site-wide due to the dynamic nature of the > > python path). > > That's exactly what I want. The only reason I didn't just implement > easy_install using a per-directory form of PEP 262 is that I wanted > something done rather more immediately. That was years ago, so I can afford > to be more patient now. :) Its ironic how impatience is rewarded! ;) > > This is a trade-off between putting the metadata up > > front in an obvious and easy to understand way so that it will > > hopefully have a better chance of being noticed and maintained, versus > > tucking it away hidden someplace so that even though it is broken, it > > doesn't bother anyone until they care enough to fix it. *It is this > > trade-off that I am exploring with this strawman "counter" proposal to > > PEP 262.* > > > > Someone would have to be crazy to maintain this information by hand. So > I'd actually consider it an advantage if the file format made this fact > plain, by using something that's difficult for a human being to maintain, > like say a pickle. ;-) OTOH, it's possible that some system packagers will > not wish to use Python to generate the files, so using something a bit less > complex would be a good idea. The format proposed by PEP 262 isn't really > that bad of a trade-off in those terms. What made you think people were rational? :) I do think that being able to maintain it by hand will aid in transparency and clarity which linux users just love, so maybe it will help win them over to the idea of python knowing a little bit about itself. ;) > > 2. The strawman proposal did not explicitly address how optional > > add-on tools (like setuptools) might manage namespace packages. > > > > I think there's some mistunderstanding here about the proposal's goals. If > the proposal doesn't work for setuptools, it doesn't work, period. > > The entire point is to allow setuptools to do its work without annoying the > people who don't want to use it. This is my misunderstanding. I was trying to take setuptools out of the equation (if only to avoid the social backlash of "trying to get setuptools into the standard library") by providing a proposal that met the objectives of my installation-tool agnostic rationale. . > > 4. Concerns were raised about the performance penalty for using the > > side-car style files without version numbers possibly not all of which > > were located at the top-most level of the directory listed in the > > python path. > > > > Any add-on tool that actually used the data would very likely need to > > build a cache of the data using a more efficient representation, > > > > This is a misunderstanding of the point I raised. Floris merely asked why > there were version numbers in .egg-info files, and I answered him. That > doesn't actually have much, if anything, to do with the package database > proposal. It's merely how installed distributions' versions can be > recognized quickly at runtime, not anything to do with how potential > installation conflicts are handled at installation time. Apologies for confusing point of fact with objective. > easy_install uses eggs for installation simply because it need never worry > about file ownership conflicts. There is a direct mapping from a > distribution to its files: the contents of a zipfile or subdirectory. This > also allows for (relatively) straightforward uninstallation. I actually like zipped eggs (much more than easy_install as package manager), but that is besides the point since the BDFL vetoed them. > The goal of the proposal, then, is to have a way for easy_install to have > another way to map from a distribution to its owned files (and vice versa), > so that eggs are not necessary for normal, single-version installations. This is where we misunderstood each other and where I've probably gone astray as I wasn't trying to propose anything at all for easy_install (it wasn't in my attempt at a rationale), but a generic common ground that tools like easy_install (but not exlcusively) could use without stepping on people's toes like easy_install does. I really wanted the proposal to standalone from easy_install so that easy_install haters wouldn't have to fear it as well as provide some utility for those who don't even use such tools. But this is probably just tilting at windmills. Thanks again for your patience. You must be overwhelmed by all of the opinions and misdirected attempts to help (including mine). I did wish we could have come-up with a setuptools agnostic support layer for installation managers suitable for inclusion in the standard library, but for some reason that doesn't seem desired by too many people. Apologies (and thanks for all your hard work). From stephen at xemacs.org Thu Apr 10 06:12:40 2008 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 10 Apr 2008 13:12:40 +0900 Subject: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <87iqyqir2t.fsf@benfinney.id.au> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <87iqyqir2t.fsf@benfinney.id.au> Message-ID: <87ej9eic07.fsf@uwakimon.sk.tsukuba.ac.jp> Ben Finney writes: > "Stanley A. Klein" writes: > > > IMHO, the main system without a package manager is Windows. > > AFAICT the MacOS platform also lacks in this area. Actually, they both have them. Windows has Cygwin (rpm-based), while for MacOS Fink (deb-based), MacPorts (FreeBSD ports-like), and NetBSD's pkgsrc are all viable options if you want packaging support for 3rd-party packages. From him at online.de Thu Apr 10 09:58:27 2008 From: him at online.de (=?ISO-8859-1?Q?Joachim_K=F6nig?=) Date: Thu, 10 Apr 2008 09:58:27 +0200 Subject: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409212522.9CDE13A409E@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> <12121.207.188.248.157.1207773812.squirrel@www.cpcug.org> <20080409212522.9CDE13A409E@sparrow.telecommunity.com> Message-ID: <47FDC8A3.7090300@online.de> Phillip J. Eby wrote: > It would be, if .eggs were a packaging format, rather than a binary > distribution/runtime format. > > Remember "eggs are to Python as jars are to Java" -- a Java .jar > doesn't contain documentation either, unless it's needed at > runtime. Same for configuration files. > But there's generally no need to easily have a look into a .class file with a tool the user is used to whereas for python, we're often interested in knowing the details. And having a zip file in my way to the source has left me frustrated often enough. If you want to be consequent and consistent leave out the .py files from eggs, a jar file normally doesn't contain .java sources files either. > They're not system packages, in other words. The assumption that > they are is another marketing failure, due to conflation of "package > == distribution of python code" and "package == thing you manage with > a system packager". People see, "I put my package in an .egg" and > think it's the latter definition, when it's barely even the former. :) > I agree that they are not system packages, but I would have prefered to install multiple versions of a package to separate "site-packages" directories, something that is really well understood by most unsofisticated python programmers. The selection of the version could then be made at runtime by a PYTHONPATH setting and not by fiddling with .pth files (something that could be autmated by a tool and persisted in batch files). From gael.varoquaux at normalesup.org Wed Apr 9 21:19:48 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 9 Apr 2008 21:19:48 +0200 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> Message-ID: <20080409191948.GA11856@phare.normalesup.org> On Wed, Apr 09, 2008 at 02:26:31PM -0400, Stanley A. Klein wrote: > The rpm and deb package managers (and their yum and other higher level > dependency managers) do a lot of things: > 1. They install packages and maintain databases of what packages were > installed > 2. They manage dependencies > 3. They support clean uninstalling of packages > 4. They can query packages, both installed (via their databases) and not > yet installed (e.g., as rpm or deb files), to determine attributes, such > as files they install, dependencies, and other information defined at > packaging time. > 5. They build packages and (in some cases) can rebuild packages. > 6. They can verify packages for integrity and security purposes. > 7. They can download package files and maintain archives of installed > package files for use as local repositories. You are collapsing three different functionalities in one: * Dealing with repositories and downloading: yum/apt * Installing + uninstalling packages, and dealing with system consistency (thus checking the dependencies are available): rpm/dpkg * Building For me it is important that the 3 are separated: * I may want to download the dependencies of a package to burn to a CD for a computer that does not have internet access. * I may want to send a tarball to a build server that does the building, but no install (so as not to corrupt my working system). Cheers, Ga?l From p.f.moore at gmail.com Thu Apr 10 10:10:39 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 10 Apr 2008 09:10:39 +0100 Subject: [Distutils] Proposal: egg -> bdist_wininst converter Message-ID: <79990c6b0804100110r7db5420ewff3cf4192fdd7de5@mail.gmail.com> For me, on Windows, the biggest negative aspect of eggs is when people provide binary installations (ones with compiled C extensions) in eggs only. I prefer bdist_wininst/bdist_msi installers, and with compiled code, I can't always build from source (complex library dependencies, etc). So, given that an egg already has everything that is needed at runtime, how easy would it be to create a tool which converted an egg into a bdist_wininst installer? With that, I could happily forget about eggs. Paul. PS I am assuming that "eggs are only a packaging format" - and that all of the setuptools goodies like namespace support, entry points etc, that some projects benefit from, would still work whether the package was an egg or a conventionally-installed bdist_wininst... From sklein at cpcug.org Thu Apr 10 16:11:38 2008 From: sklein at cpcug.org (Stanley A. Klein) Date: Thu, 10 Apr 2008 10:11:38 -0400 (EDT) Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: References: Message-ID: <42456.207.188.248.157.1207836698.squirrel@www.cpcug.org> On Wed, April 9, 2008 10:05 pm, Greg Ewing wrote: > Message: 4 > Date: Thu, 10 Apr 2008 12:59:39 +1200 > From: Greg Ewing > Subject: Re: [Distutils] how to easily consume just the parts of eggs > that are good for you > To: distutils-sig at python.org > > Paul Moore wrote: > >> I believe that Mac OS X goes for an even simpler structure - >> applications store *everything* in the one directory, so that >> install/uninstall is simply a directory copy/remove. > > Yep, and thereby cuts the whole gordian knot, throws the > pieces on the fire and burns them. :-) > > Package managers have always seemed to me to be an > excessively complex solution to a problem that needn't > have existed in the first place. > > I keep hoping that someday Linux will support something > like MacOSX application bundles and frameworks, but I > haven't seen any sign of it yet. > I think this discussion arises because the operating systems we are dealing with are based on very different concepts. Windows and Mac are fundamentally single user systems that have added capabilities for multiple users and are intended to be used with proprietary software. Those considerations lead to minimal dependencies among packages (each proprietary provider needs to control its own package, except for the OS), individual users serving as their own sysadmins, and similar factors. Any dependencies in the proprietary software are hidden from the user because the provider has compiled the dependencies into the binary code they supply. Unix/Linux are fundamentally multi-user systems with a distinct role for a sysadmin. Linux and the BSD's are intended to be used with Free/Open Source Software (FOSS), and Unix originated the Software Tools concept in which individual, relatively simple, separately-developed tools are combined to perform complex tasks. Both FOSS and the Software Tools concept encourage dependencies. The need for package managers arises out of the Unix FHS. If you look at the FHS, it is clearly designed to simplify the job of the sysadmin in a multi-user system that uses FOSS and Software Tools. For example, deciding what to backup and how often to do it in a Unix/Linux system is relatively easy for the sysadmin. All the installed software is in certain top-level directories. All the config files are in /etc. All the logs. caches, spools, web pages, process locks, and certain other data are in /var. All the user data is in /home or its sysadmin-determined equivalents. If the sysadmin needs space, deleting files in /tmp will not cause a problem. In summary, Python is being used on systems that have very different underlying OS use cases. To some extent, the natural use case for Python is closest to that of Linux/Unix. Running Python on Windows/Mac requires adapting for those platforms some of the kinds of tools that simplify operations on Linux/Unix systems. This discussion is essentially about how far that goes, how to accomplish it, and how to remain compatible with the existing tools on Linux/Unix. Stan Klein From pje at telecommunity.com Thu Apr 10 16:31:53 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 10 Apr 2008 10:31:53 -0400 Subject: [Distutils] Proposal: egg -> bdist_wininst converter In-Reply-To: <79990c6b0804100110r7db5420ewff3cf4192fdd7de5@mail.gmail.co m> References: <79990c6b0804100110r7db5420ewff3cf4192fdd7de5@mail.gmail.com> Message-ID: <20080410143147.062133A406A@sparrow.telecommunity.com> At 09:10 AM 4/10/2008 +0100, Paul Moore wrote: >For me, on Windows, the biggest negative aspect of eggs is when people >provide binary installations (ones with compiled C extensions) in eggs >only. I prefer bdist_wininst/bdist_msi installers, and with compiled >code, I can't always build from source (complex library dependencies, >etc). > >So, given that an egg already has everything that is needed at >runtime, how easy would it be to create a tool which converted an egg >into a bdist_wininst installer? With that, I could happily forget >about eggs. http://mail.python.org/pipermail/python-dev/2008-March/077959.html Basically what you need to do is rewrite the zipfile index such that everything is prefixed with PURELIB/ or PLATLIB/, except for the EGG-INFO directory which must be renamed to match the original egg's filename, plus '-info' -- i.e., foo-1.2-py2.3-win32.egg should have its EGG-INFO renamed to PURELIB/foo-1.2-py2.3-win32.egg-info or PLATLIB/foo-1.2-py2.3-win32.egg-info. Once the egg is reformatted in this way, it's a perfectly valid zipfile that the existing bdist_wininst code could be reused to turn into an .exe. From p.f.moore at gmail.com Thu Apr 10 16:47:01 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 10 Apr 2008 15:47:01 +0100 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <42456.207.188.248.157.1207836698.squirrel@www.cpcug.org> References: <42456.207.188.248.157.1207836698.squirrel@www.cpcug.org> Message-ID: <79990c6b0804100747s1005ca6drc867bbd2a10c50c2@mail.gmail.com> On 10/04/2008, Stanley A. Klein wrote: > In summary, Python is being used on systems that have very different > underlying OS use cases. To some extent, the natural use case for Python > is closest to that of Linux/Unix. Running Python on Windows/Mac requires > adapting for those platforms some of the kinds of tools that simplify > operations on Linux/Unix systems. This discussion is essentially about > how far that goes, how to accomplish it, and how to remain compatible with > the existing tools on Linux/Unix. Thanks, that's a good summary. I would dispute your comment that "the natural use case for Python is closest to that of Linux/Unix", however. I think Python is perfectly adaptable to both environments, and from *my* point of view, the issue is that Python is currently well adapted to a Windows environment. It seems that the Unix/Linux users find it less well adapted, and need changes as a result - but in doing so, their changes are disrupting the Windows situation. However, this is from the POV of a Windows developer, who has no sysadmin experience on Windows, and little experience with Unix. So it's certainly biased. But from where I sit, there's no Windows issue to solve, and while I'm happy for the Unix people to address the problems they have, I'd be unhappy if in doing so they *make* problems for Windows. A windows sysadmin may have a different perspective, though. Paul. From p.f.moore at gmail.com Thu Apr 10 16:52:33 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 10 Apr 2008 15:52:33 +0100 Subject: [Distutils] Proposal: egg -> bdist_wininst converter In-Reply-To: <20080410143147.062133A406A@sparrow.telecommunity.com> References: <79990c6b0804100110r7db5420ewff3cf4192fdd7de5@mail.gmail.com> <20080410143147.062133A406A@sparrow.telecommunity.com> Message-ID: <79990c6b0804100752n434c5419n79c637bcf7d545c8@mail.gmail.com> On 10/04/2008, Phillip J. Eby wrote: > At 09:10 AM 4/10/2008 +0100, Paul Moore wrote: > > So, given that an egg already has everything that is needed at > > runtime, how easy would it be to create a tool which converted an egg > > into a bdist_wininst installer? With that, I could happily forget > > about eggs. > > > > http://mail.python.org/pipermail/python-dev/2008-March/077959.html Ah, yes. I thought this had come up before. The problem is that all the talking in the world doesn't make things happen - and I don't have the time to write this sort of thing myself. Until someone steps up who is willing to write code, the discussion is going round in circles (as proved here!) and so think I'm going to restrain myself from posting any more on the topic... Paul. From sklein at cpcug.org Thu Apr 10 17:36:21 2008 From: sklein at cpcug.org (Stanley A. Klein) Date: Thu, 10 Apr 2008 11:36:21 -0400 (EDT) Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <79990c6b0804100747s1005ca6drc867bbd2a10c50c2@mail.gmail.com> References: <42456.207.188.248.157.1207836698.squirrel@www.cpcug.org> <79990c6b0804100747s1005ca6drc867bbd2a10c50c2@mail.gmail.com> Message-ID: <58044.207.188.248.157.1207841781.squirrel@www.cpcug.org> On Thu, April 10, 2008 10:47 am, Paul Moore wrote: > On 10/04/2008, Stanley A. Klein wrote: >> In summary, Python is being used on systems that have very different >> underlying OS use cases. To some extent, the natural use case for >> Python >> is closest to that of Linux/Unix. Running Python on Windows/Mac >> requires >> adapting for those platforms some of the kinds of tools that simplify >> operations on Linux/Unix systems. This discussion is essentially about >> how far that goes, how to accomplish it, and how to remain compatible >> with >> the existing tools on Linux/Unix. > > Thanks, that's a good summary. I would dispute your comment that "the > natural use case for Python is closest to that of Linux/Unix", > however. I think Python is perfectly adaptable to both environments, > and from *my* point of view, the issue is that Python is currently > well adapted to a Windows environment. It seems that the Unix/Linux > users find it less well adapted, and need changes as a result - but in > doing so, their changes are disrupting the Windows situation. > > However, this is from the POV of a Windows developer, who has no > sysadmin experience on Windows, and little experience with Unix. So > it's certainly biased. But from where I sit, there's no Windows issue > to solve, and while I'm happy for the Unix people to address the > problems they have, I'd be unhappy if in doing so they *make* problems > for Windows. > > A windows sysadmin may have a different perspective, though. > > Paul. > The reason I say that the natural use case for Python is closest to Linux/Unix is that Python is FOSS and its natural approaches encourage dependencies that are not hidden from the user. It is natural in Unix/Linux to install dependencies that are not compiled in as part of a monolithic application and are not bundled with the OS. Although Python is designed to be cross-platform, porting a "FOSS software with dependencies" environment to Windows is outside the natural Windows use case. The natural case in Windows is that all dependencies are either compiled in as part of a monolithic application or are bundled with the OS. One problem is that an excessive focus on tools that solve the "natural use case" issue tends to break the already existing solutions and distribution methods for Unix/Linux. Stan Klein From pje at telecommunity.com Thu Apr 10 18:54:32 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 10 Apr 2008 12:54:32 -0400 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <58044.207.188.248.157.1207841781.squirrel@www.cpcug.org> References: <42456.207.188.248.157.1207836698.squirrel@www.cpcug.org> <79990c6b0804100747s1005ca6drc867bbd2a10c50c2@mail.gmail.com> <58044.207.188.248.157.1207841781.squirrel@www.cpcug.org> Message-ID: <20080410165422.175DD3A406A@sparrow.telecommunity.com> At 11:36 AM 4/10/2008 -0400, Stanley A. Klein wrote: >The reason I say that the natural use case for Python is closest to >Linux/Unix is that Python is FOSS and its natural approaches encourage >dependencies that are not hidden from the user. It is natural in >Unix/Linux to install dependencies that are not compiled in as part of a >monolithic application and are not bundled with the OS. It's natural for sysadmins... not application developers. Application developers can't rely on system packagers and sysadmins to ship their applications as packages for the OS, *even if* the application *and* the OS are open source. All that's necessary for there to be issues, is that the application and some of its dependencies be in active development. So, all this chatter about what's natural for a given OS is just more provincialism -- i.e., believing that the way things are in one village is The One True Way That Everyone Should Follow. Meanwhile, the real-life use cases exist, and they won't be argued out of existence, any more than the people in other countries cease to exist because they're past the edge of the flat earth. :) From dpeterson at enthought.com Thu Apr 10 20:51:17 2008 From: dpeterson at enthought.com (Dave Peterson) Date: Thu, 10 Apr 2008 13:51:17 -0500 Subject: [Distutils] [Enthought-dev] Installation woes - again.. In-Reply-To: <200804101339.17812.meine@informatik.uni-hamburg.de> References: <200804091807.18050.meine@informatik.uni-hamburg.de> <200804100857.47438.meine@informatik.uni-hamburg.de> <200804100919.31681.meine@informatik.uni-hamburg.de> <200804101339.17812.meine@informatik.uni-hamburg.de> Message-ID: <47FE61A5.3040208@enthought.com> NOTE: cc'ing distutils-sig for help debugging potential problem with setuptools on gentoo. Hans Meine wrote: > Am Donnerstag, 10. April 2008 09:19:31 schrieb Hans Meine: > >> The same errors occur without virtualenv, so you're obviously right that >> this is setuptools-related. I could try again and really deinstall >> Gentoo's setuptools (instead of just locally installing it the official >> way), but I really doubt now that this would make a difference. >> > > So, I have even tried this - deinstalling setuptools altogether, and > installing everything from scratch. I used "script" to create a log file of > the process with all input and resulting output + errors: > http://kogs-www.informatik.uni-hamburg.de/~meine/clean-ets-install.log > Thanks for providing that full log! From looking at that, I'm really starting to suspect a bug in setuptools 0.6c8 as I can't make any sense of the following sequence of events. While it's possible we've done something wrong in our setup.py scripts, I have my doubts as this release appears to install fine for users on Windows, OSX, and RH. (For distutils-sig readers, Hans is using Gentoo.) 1) During the first install attempt of "ets", the log shows two lines reporting success with installing enthought.tvtk-2.0.2-py2.4-linux-i686.egg. One right after downloading and building enthought.tvtk from the source tarball, and one in the middle of the enthought.kiva build. 2) Regarding that enthought.kiva build, the log indicates that a dependency caused a search to be made for "enthought.kiva>=2.0.3.dev,<=2.0.3" and a match was found (enthought.kiva-2.0.3.tar.gz) The log then shows this building, apparently a successful build (no error messages, just warnings) up until the normally reported "Installed /path/to/egg" line where it mentions success with installing enthought.tvtk rather enthought.kiva. After that, it reports being unable to find the enthought.kiva distribution. 3) You then re-run the "easy_install -f ... ets" command which has no problem building enthought.kiva this time using the exact same source tarball. I have no idea what might be causing this sequence of events. Has anyone seen anything similar with setuptools? -- Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080410/bc074a06/attachment-0001.htm From jjl at pobox.com Thu Apr 10 22:12:26 2008 From: jjl at pobox.com (John J Lee) Date: Thu, 10 Apr 2008 21:12:26 +0100 (BST) Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <47FD667B.9040602@canterbury.ac.nz> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> <79990c6b0804091546p5cf67f2ta800c675e7baa256@mail.gmail.com> <47FD667B.9040602@canterbury.ac.nz> Message-ID: On Thu, 10 Apr 2008, Greg Ewing wrote: > Paul Moore wrote: [...] > I keep hoping that someday Linux will support something > like MacOSX application bundles and frameworks, but I > haven't seen any sign of it yet. Not the same, but "something like": http://0install.net/ John From dpeterson at enthought.com Thu Apr 10 22:48:12 2008 From: dpeterson at enthought.com (Dave Peterson) Date: Thu, 10 Apr 2008 15:48:12 -0500 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <43869.71.178.83.244.1207792737.squirrel@www.cpcug.org> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> <47FD4E9D.8060600@enthought.com> <43869.71.178.83.244.1207792737.squirrel@www.cpcug.org> Message-ID: <47FE7D0C.9090609@enthought.com> Stanley A. Klein wrote: > On Wed, 2008-04-09 at 18:17 -0500, Dave Peterson wrote: > >> I think I can sum up any further points by simply asking: "Should it >> be safe to assume I can distribute my application via eggs / >> easy_install just because it is written in Python?" >> > > > I think that based on this discussion the bottom line answer to this > question is "No". > I agree that it seems like that's where things are headed in the discussion. But the discussion doesn't always coincide with the reality, right? I guess I'm trolling more for a statement from the setuptools maintainer here. Particularly since I'm looking for an answer to my question about should Enthought continue to invest time into a setuptools patch that lets developers include docs, config files, etc. in eggs for installation in a FHS-approved location at install time? -- Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080410/8c14362f/attachment.htm From pje at telecommunity.com Fri Apr 11 00:06:29 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 10 Apr 2008 18:06:29 -0400 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <47FE7D0C.9090609@enthought.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> <47FD4E9D.8060600@enthought.com> <43869.71.178.83.244.1207792737.squirrel@www.cpcug.org> <47FE7D0C.9090609@enthought.com> Message-ID: <20080410220622.8A94F3A406A@sparrow.telecommunity.com> At 03:48 PM 4/10/2008 -0500, Dave Peterson wrote: >Stanley A. Klein wrote: >>On Wed, 2008-04-09 at 18:17 -0500, Dave Peterson wrote: >>>I think I can sum up any further points by simply asking: "Should it >>>be safe to assume I can distribute my application via eggs / >>>easy_install just because it is written in Python?" >> >>I think that based on this discussion the bottom line answer to this >>question is "No". > >I agree that it seems like that's where things are headed in the >discussion. But the discussion doesn't always coincide with the >reality, right? I guess I'm trolling more for a statement from the >setuptools maintainer here. > >Particularly since I'm looking for an answer to my question about >should Enthought continue to invest time into a setuptools patch >that lets developers include docs, config files, etc. in eggs for >installation in a FHS-approved location at install time? I think it's more than reasonable to define a standard for including such data. I'm somewhat more skeptical about doing that installation automatically within easy_install. Likewise, I'm skeptical about doing other sorts of non-package, non-script installation. I'd like to see proposals that show due care to cross-platformness, uninstallability, etc. In other words, when it comes to a "patch" -- the documentation is going to count for a lot more than the code, and I'd rather see a concrete proposal well in advance of the patch. Sooner would be better than later, too, because it's likely that the plan for "non-egg installs" is going to be affected by the plan as well. From pje at telecommunity.com Fri Apr 11 00:33:31 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 10 Apr 2008 18:33:31 -0400 Subject: [Distutils] setuptools and pkg registration In-Reply-To: <20080410213707.GF51167@nexus.in-nomine.org> References: <20080410213707.GF51167@nexus.in-nomine.org> Message-ID: <20080410223629.33CBF3A409E@sparrow.telecommunity.com> At 11:37 PM 4/10/2008 +0200, Jeroen Ruigrok van der Werven wrote: >Phillip, > >I am not sure (haven't verified) if my email ever got through to the list, >but with FreeBSD's Perl installation an additional module gets installed >that registers every CPAN installed module as a pkg in FreeBSD's pkg system. > >Given the recent bruhaha about the setuptools and eggs (hey, from a >programmer and sysadmin point of view I like 'm) I wonder if this would be >easier for the discussion if you could dump a prototype on the table. > >If it would help I can see if I can cook up something. At this point, it has become clear to me that the package database isn't really necessary for other systems. easy_install could simply not uninstall things that it doesn't find listed in its *own* metadata. Really, the only technical hurdle remaining to solving egg-free installation is setting up a namespace package's __path__ correctly, in a situation where: 1. The user has system packages contributing to the namespace in a "site" directory, 2. There is no __init__.py in the namespace under that site directory, 3. There are non-egg packages on PYTHONPATH that contribute to the same directory, and 4. Two versions of the same module within that namespace have been installed, one in site-packages and the other on PYTHONPATH Actually, there are various other criteria involved, such as whether pkg_resources is imported or not, and whether the setuptools site.py hack is employed in the PYTHONPATH directory. For example, if pkg_resources were imported (which normally happens when setuptools-installed scripts are run), it would be possible to have it do the sorting, and thus ensure that there's never a problem like that. What it would not fix, is the circumstance where you just pop into the Python interpreter with your PYTHONPATH set, if you have the above situation. And the above situation is going to occur anytime you're using PYTHONPATH to get a later version of something that uses a namespace package. So, the other alternative is to keep the 'site.py' hack in effect, but either change the contents of that site.py or add extra information to a .pth file so that the namespaces get sorted at that point. There would still be a hacked site module at that point, but at least upgrading the site.py would let us get rid of the need to load any .pth files from the directory, put eggs ahead of other things, etc. For that matter, the site.py wouldn't be needed unless you were installing things with namespace packages in it. (Unfortunately, looking at site-packages to see if the hack is needed won't help, since you might install a conflicting system package *later*.) From dpeterson at enthought.com Fri Apr 11 01:16:03 2008 From: dpeterson at enthought.com (Dave Peterson) Date: Thu, 10 Apr 2008 18:16:03 -0500 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080410220622.8A94F3A406A@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> <47FD4E9D.8060600@enthought.com> <43869.71.178.83.244.1207792737.squirrel@www.cpcug.org> <47FE7D0C.9090609@enthought.com> <20080410220622.8A94F3A406A@sparrow.telecommunity.com> Message-ID: <47FE9FB3.4040609@enthought.com> Phillip J. Eby wrote: > At 03:48 PM 4/10/2008 -0500, Dave Peterson wrote: >> Stanley A. Klein wrote: >>> On Wed, 2008-04-09 at 18:17 -0500, Dave Peterson wrote: >>>> I think I can sum up any further points by simply asking: "Should it >>>> be safe to assume I can distribute my application via eggs / >>>> easy_install just because it is written in Python?" >>> >>> I think that based on this discussion the bottom line answer to this >>> question is "No". >> >> I agree that it seems like that's where things are headed in the >> discussion. But the discussion doesn't always coincide with the >> reality, right? I guess I'm trolling more for a statement from the >> setuptools maintainer here. >> >> Particularly since I'm looking for an answer to my question about >> should Enthought continue to invest time into a setuptools patch that >> lets developers include docs, config files, etc. in eggs for >> installation in a FHS-approved location at install time? > > I think it's more than reasonable to define a standard for including > such data. I'm somewhat more skeptical about doing that installation > automatically within easy_install. Likewise, I'm skeptical about > doing other sorts of non-package, non-script installation. I'd like > to see proposals that show due care to cross-platformness, > uninstallability, etc. > > In other words, when it comes to a "patch" -- the documentation is > going to count for a lot more than the code, and I'd rather see a > concrete proposal well in advance of the patch. > > Sooner would be better than later, too, because it's likely that the > plan for "non-egg installs" is going to be affected by the plan as well. I believe I understand, and agree, with your concerns. Let me be clear on the status of where we are in our work: we've internally talked through a number of design possibilities, and are now trying out (via hacking on setuptools) how the one we thought was "best" worked out. In particular, we're concerned about the difficulty of use in terms of even just the use-cases we have for ETS projects. Also, since we do a bit of cross-platform deployment, we're also investigating those effects on the design as well. That being said, I don't think we're ready to put forward a proposal that would withstand too much public scrutiny quite yet - at least if the resulting discussion implied a significant time or effort commitment on our part to carry the conversation forward. If it sounded like we'd already figured it all out, I apologize for getting people's hopes up! I just wanted to make sure further pursuit in this direction on our part wasn't completely wasted. The above not withstanding, if anyone is interested in talking about it / helping us, we certainly wouldn't ignore you. I just can't promise immediate responses due to pretty pressing customer commitments on our part. Regarding the mention of 'uninstallability' above, I assume it would be sufficient if the installed files were simply included in the generated list of files provided by the "--record" option since there is currently no uninstall command at all? Or is there something else you'd like to see as an intermediate measure? I'd love to add an uninstall option to easy_install as well (it's something we get hit about alot by our user community) but there's only so many hours in a day. ;-) -- Dave From klaus_zimmermann at gmx.de Fri Apr 11 08:30:33 2008 From: klaus_zimmermann at gmx.de (Klaus Zimmermann) Date: Fri, 11 Apr 2008 08:30:33 +0200 Subject: [Distutils] [PATCH][setuptools] Add test_runner support to setup In-Reply-To: <1207482058.5957.0.camel@localhost> References: <1207408509.6012.9.camel@localhost> <20080405235300.795DE3A4042@sparrow.telecommunity.com> <1207482058.5957.0.camel@localhost> Message-ID: <1207895433.5996.3.camel@localhost> Hi Phillip, *, Am Sonntag, den 06.04.2008, 13:40 +0200 schrieb Klaus Zimmermann: > Am Samstag, den 05.04.2008, 19:53 -0400 schrieb Phillip J. Eby: > > At 05:15 PM 4/5/2008 +0200, Klaus Zimmermann wrote: > > >Hi all, > > > > > >for our nightly build system I needed the capability to use a different > > >test runner, namely one that outputs junit compatible xml files. > > >In order to support that through the setup.py test command I hacked on > > >setuptools. > > > > > >Is there a simpler way to do this? > > >If not you might want to consider attached patch for addition > > >(Should be against latest svn.) > > > > Add documentation to the patch and you've got yourself a deal. :) > > Updated patch attached. > Have fun, > Klaus was the patch satisfactory or is it still lacking? Or do I need to do anything more for it to be applied? On a side note: In the test.py command I noticed another option, test-module, which, if I understand it correctly, is somewhat a stripped-down, undocumented version of test-suite. Should it be deprecated or removed? Cheers, Klaus From elazarl at gmail.com Fri Apr 11 08:45:06 2008 From: elazarl at gmail.com (Elazar Leibovich) Date: Fri, 11 Apr 2008 09:45:06 +0300 Subject: [Distutils] Batch installation of python extentions Message-ID: <2d6bb6c0804102345i73cd51e2w63a205bb8556bb3d@mail.gmail.com> Hi, I need to install a big bunch of python extentions in a couple of offline windows XPcomputer. Is there any standard way to do that on the win32 platform? The best options I thought of is installing ez_install manually on every computer, and then install all the egg-files in a certain directory. Is there any better way to do that? Is there a standard way to distribute python extentions. Example extentions: scipy, pyserial, wxpython -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080411/785e2773/attachment.htm From pje at telecommunity.com Fri Apr 11 18:01:33 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 11 Apr 2008 12:01:33 -0400 Subject: [Distutils] [PATCH][setuptools] Add test_runner support to setup In-Reply-To: <1207895433.5996.3.camel@localhost> References: <1207408509.6012.9.camel@localhost> <20080405235300.795DE3A4042@sparrow.telecommunity.com> <1207482058.5957.0.camel@localhost> <1207895433.5996.3.camel@localhost> Message-ID: <20080411160124.646923A40AE@sparrow.telecommunity.com> At 08:30 AM 4/11/2008 +0200, Klaus Zimmermann wrote: >Hi Phillip, *, > >Am Sonntag, den 06.04.2008, 13:40 +0200 schrieb Klaus Zimmermann: > > Am Samstag, den 05.04.2008, 19:53 -0400 schrieb Phillip J. Eby: > > > At 05:15 PM 4/5/2008 +0200, Klaus Zimmermann wrote: > > > >Hi all, > > > > > > > >for our nightly build system I needed the capability to use a different > > > >test runner, namely one that outputs junit compatible xml files. > > > >In order to support that through the setup.py test command I hacked on > > > >setuptools. > > > > > > > >Is there a simpler way to do this? > > > >If not you might want to consider attached patch for addition > > > >(Should be against latest svn.) > > > > > > Add documentation to the patch and you've got yourself a deal. :) > > > > Updated patch attached. > > Have fun, > > Klaus > >was the patch satisfactory or is it still lacking? >Or do I need to do anything more for it to be applied? It's sitting in my folder of patches to eventually be applied. :) >On a side note: In the test.py command I noticed another option, >test-module, which, if I understand it correctly, is somewhat a >stripped-down, undocumented version of test-suite. Should it be >deprecated or removed? Probably, but not in the 0.6 line. From ziade.tarek at gmail.com Fri Apr 11 18:07:32 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 11 Apr 2008 18:07:32 +0200 Subject: [Distutils] [PATCH][setuptools] Add test_runner support to setup In-Reply-To: <20080411160124.646923A40AE@sparrow.telecommunity.com> References: <1207408509.6012.9.camel@localhost> <20080405235300.795DE3A4042@sparrow.telecommunity.com> <1207482058.5957.0.camel@localhost> <1207895433.5996.3.camel@localhost> <20080411160124.646923A40AE@sparrow.telecommunity.com> Message-ID: <94bdd2610804110907v5dd7b558r208146b22ff4df38@mail.gmail.com> On Fri, Apr 11, 2008 at 6:01 PM, Phillip J. Eby wrote: > > > >was the patch satisfactory or is it still lacking? > >Or do I need to do anything more for it to be applied? > > It's sitting in my folder of patches to eventually be applied. :) What about this one ? http://mail.python.org/pipermail/distutils-sig/2008-April/009187.html Do you have a public tracker where we can submit patches ? Thanks Tarek From pje at telecommunity.com Fri Apr 11 19:33:24 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 11 Apr 2008 13:33:24 -0400 Subject: [Distutils] [PATCH][setuptools] Add test_runner support to setup In-Reply-To: <94bdd2610804110907v5dd7b558r208146b22ff4df38@mail.gmail.co m> References: <1207408509.6012.9.camel@localhost> <20080405235300.795DE3A4042@sparrow.telecommunity.com> <1207482058.5957.0.camel@localhost> <1207895433.5996.3.camel@localhost> <20080411160124.646923A40AE@sparrow.telecommunity.com> <94bdd2610804110907v5dd7b558r208146b22ff4df38@mail.gmail.com> Message-ID: <20080411173314.0684E3A406A@sparrow.telecommunity.com> At 06:07 PM 4/11/2008 +0200, Tarek Ziad? wrote: >On Fri, Apr 11, 2008 at 6:01 PM, Phillip J. Eby wrote: > > > > > >was the patch satisfactory or is it still lacking? > > >Or do I need to do anything more for it to be applied? > > > > It's sitting in my folder of patches to eventually be applied. :) > >What about this one ? > >http://mail.python.org/pipermail/distutils-sig/2008-April/009187.html Yes. >Do you have a public tracker where we can submit patches ? No -- but I heard it's being worked on. My first preference would be a python.org hosted Roundup instance, especially if I can use my existing login. If that can't be arranged, I'll probably look into throwing up a Trac or Roundup myself -- eventually. From sklein at cpcug.org Fri Apr 11 21:05:39 2008 From: sklein at cpcug.org (Stanley A. Klein) Date: Fri, 11 Apr 2008 15:05:39 -0400 (EDT) Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: References: Message-ID: <55083.207.188.248.157.1207940739.squirrel@www.cpcug.org> On Fri, April 11, 2008 6:00 am, Dave Peterson wrote: > > Message: 5 > Date: Thu, 10 Apr 2008 18:16:03 -0500 > From: Dave Peterson > Subject: Re: [Distutils] how to easily consume just the parts of eggs > that are good for you > To: "Phillip J. Eby" > Cc: distutils-sig > > Phillip J. Eby wrote: >> At 03:48 PM 4/10/2008 -0500, Dave Peterson wrote: >>> Stanley A. Klein wrote: >>>> On Wed, 2008-04-09 at 18:17 -0500, Dave Peterson wrote: >>>>> I think I can sum up any further points by simply asking: "Should it >>>>> be safe to assume I can distribute my application via eggs / >>>>> easy_install just because it is written in Python?" >>>> >>>> I think that based on this discussion the bottom line answer to this >>>> question is "No". >>> >>> I agree that it seems like that's where things are headed in the >>> discussion. But the discussion doesn't always coincide with the >>> reality, right? I guess I'm trolling more for a statement from the >>> setuptools maintainer here. >>> >>> Particularly since I'm looking for an answer to my question about >>> should Enthought continue to invest time into a setuptools patch that >>> lets developers include docs, config files, etc. in eggs for >>> installation in a FHS-approved location at install time? >> >> I think it's more than reasonable to define a standard for including >> such data. I'm somewhat more skeptical about doing that installation >> automatically within easy_install. Likewise, I'm skeptical about >> doing other sorts of non-package, non-script installation. I'd like >> to see proposals that show due care to cross-platformness, >> uninstallability, etc. >> >> In other words, when it comes to a "patch" -- the documentation is >> going to count for a lot more than the code, and I'd rather see a >> concrete proposal well in advance of the patch. >> >> Sooner would be better than later, too, because it's likely that the >> plan for "non-egg installs" is going to be affected by the plan as well. > > I believe I understand, and agree, with your concerns. Let me be clear > on the status of where we are in our work: we've internally talked > through a number of design possibilities, and are now trying out (via > hacking on setuptools) how the one we thought was "best" worked out. In > particular, we're concerned about the difficulty of use in terms of even > just the use-cases we have for ETS projects. Also, since we do a bit > of cross-platform deployment, we're also investigating those effects on > the design as well. That being said, I don't think we're ready to put > forward a proposal that would withstand too much public scrutiny quite > yet - at least if the resulting discussion implied a significant time or > effort commitment on our part to carry the conversation forward. If it > sounded like we'd already figured it all out, I apologize for getting > people's hopes up! I just wanted to make sure further pursuit in this > direction on our part wasn't completely wasted. > > The above not withstanding, if anyone is interested in talking about it > / helping us, we certainly wouldn't ignore you. I just can't promise > immediate responses due to pretty pressing customer commitments on our > part. > > > Regarding the mention of 'uninstallability' above, I assume it would be > sufficient if the installed files were simply included in the generated > list of files provided by the "--record" option since there is currently > no uninstall command at all? Or is there something else you'd like to > see as an intermediate measure? I'd love to add an uninstall option to > easy_install as well (it's something we get hit about alot by our user > community) but there's only so many hours in a day. ;-) > Here are a few principles I think I've gleaned from this discussion: 1. Eggs are not intended to serve as substitutes for rpms, debs, and other platform package distribution systems. They are primarily intended as a convenience for developers during development. They can be used in distribution packaging in a manner analogous to .jar files in java. 2. Every platform wants to put the files somewhere different. Accordingly, Setuptools and Distutils need to support mapping the files to the preferred locations for the various platforms. 3. Setuptools and Distutils support some platform packaging systems, such as rpm. Packagers should distribute using those systems. If eggs are included in a package, they should be treated as any other file for that packaging system and not as a form of distribution. For example, an rpm should reasonably be able to contain egg files along with other distribution files. (Note that java-related rpms often contain .jar files.) Consideration should be given to adding features to existing support for platform packaging systems. Also, if egg files are to be included in a distribution package, it needs to be a two-step process (first make the eggs, then make the package). 4. It would probably help to have a style guide or automated converter for naming and placement of files in a source distribution to facilitate file location mappings and platform naming issues. For example, Windows accepts file names with spaces in them but Linux generally doesn't (and bdist_rpm produces errors if it sees them). For a multiplatform distribution either a style guide or a lint-type tool needs to address issues like this. Stan Klein From greg.ewing at canterbury.ac.nz Sat Apr 12 04:31:06 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 12 Apr 2008 14:31:06 +1200 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> <79990c6b0804091546p5cf67f2ta800c675e7baa256@mail.gmail.com> <47FD667B.9040602@canterbury.ac.nz> Message-ID: <48001EEA.3030209@canterbury.ac.nz> John J Lee wrote: >> I keep hoping that someday Linux will support something >> like MacOSX application bundles and frameworks, > > Not the same, but "something like": > > http://0install.net/ That looks interesting, but I'm not sure I'd quite call it "something like". It looks like another case of adding more complexity to fight existing complexity, rather than removing the original complexity. In other words, it seems to be just another package manager, albeit a particulary nice-sounding one. -- Greg From zooko at zooko.com Sat Apr 12 05:42:32 2008 From: zooko at zooko.com (zooko) Date: Fri, 11 Apr 2008 20:42:32 -0700 Subject: [Distutils] an example of setuptools being used to good effect -- allmydata.org Tahoe In-Reply-To: <47FD667B.9040602@canterbury.ac.nz> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> <79990c6b0804091546p5cf67f2ta800c675e7baa256@mail.gmail.com> <47FD667B.9040602@canterbury.ac.nz> Message-ID: Folks: I'm sorry, but I am not caught up on the current conversation about packaging. I'm very busy with exciting Python development -- http:// allmydata.com and http://allmydata.org . I understand from PJE's message that he thinks I misunderstand some things about PEP 262; this is entirely possible. I intend to catch up on reading the emails of this conversation and to read carefully PJE's messages about PEP 262 in the coming days. Meanwhile, here is the last message that I wrote before I got swamped with the aforementioned excitement: On Apr 9, 2008, at 5:59 PM, Greg Ewing wrote: > Paul Moore wrote: > >> I believe that Mac OS X goes for an even simpler structure - >> applications store *everything* in the one directory, so that >> install/uninstall is simply a directory copy/remove. > > Yep, and thereby cuts the whole gordian knot, throws the > pieces on the fire and burns them. :-) > > Package managers have always seemed to me to be an > excessively complex solution to a problem that needn't > have existed in the first place. Yes! I love the Zen of the Mac OS X packaging approach. The best install is none at all! (Of course, I also love apt.) > I keep hoping that someday Linux will support something > like MacOSX application bundles and frameworks, but I > haven't seen any sign of it yet. We're slowly approaching this level of simplicity in packaging of the *source code* of Allmydata.org "Tahoe", using setuptools. http://allmydata.org/source/tahoe/trunk/docs/install.html The list of dependencies which are automatically resolved by setuptools is visible here: [1]. It currently includes zfec, foolscap, simplejson, pycryptopp, nevow, zope.interface, twisted, and pywin32. This automatic resolution of dependencies works while fully preserving the user's ability to use their own packages and their own packaging tools. That is: 1. If any of these dependencies are already installed, such as in a Debian package or if the user has installed them by hand, then installing Tahoe will use the already-installed versions and not install anything new, and 2. For any of these dependencies that are not already installed, installing Tahoe will *not* write these dependencies into your standard system directory (which is potentially a sacred place where only your own packaging tool or your root account is allowed to tread) but will instead write them into a local, newly-created install directory from which you can then run Tahoe. (This part is similar in spirit to the Mac OS packaging technique.) Also, this install process never downloads anything from the Internet at install time, per our policy [2, 3], which also happens to be a policy some other people have, e.g. [4, 5]. This works on all of our supported platforms, which includes Linux, Solaris, Windows, Cygwin, and Mac OS X. Oh yes, we also have our buildbot [6] automatically produce Debian packages for edgy, feisty, etch, and gutsy. As far as I know, all of this is accomplished without breaking any of the use cases traditionally associated with setuptools / easy_install, for example "easy_install allmydata-tahoe" works, and if you want "setup.py install" to install into your standard system directory, it will. The reason that I am posting this is to let other programmers know that setuptools is actually a pretty useful tool, even if the use cases that you want to support are incompatible with certain easy_install traditions such as fetching new packages from the internet at buildtime or installing into your system directory. Regards, Zooko P.S. Two days ago I was able to remove twisted from the list of "Manual Dependencies" that people have to be aware of in order to try out Allmydata Tahoe from the source tarball. I think I can safely remove pyOpenSSL now, but that remains to be properly tested by our buildbot. I will be able to remove Crypto++ soon, due to the pycryptopp [7] library. If I can figure out a hack to work-around one of the major frustrations of setuptools (that you can't simply run "./setup.py install --prefix=$FOO"), and if I finish my setuptools plugin to run Twisted trial instead pyunit when "./setup.py test", then I'll be able to remove GNU make from the dependencies. That will leave only g++, Python, and OpenSSL as packages that a programmer has to manually deal with in order to try out Allmydata Tahoe from source. [1] http://allmydata.org/trac/tahoe/browser/_auto_deps.py [2] http://allmydata.org/trac/tahoe/wiki/Packaging [3] http://allmydata.org/trac/tahoe/ticket/229 [4] http://bytes.com/forum/thread666455.html [5] http://fedoraproject.org/wiki/Packaging/Python/Eggs [6] http://allmydata.org/buildbot/waterfall?show_events=false [7] http://allmydata.org/trac/pycryptopp From jjl at pobox.com Sat Apr 12 16:11:02 2008 From: jjl at pobox.com (John J Lee) Date: Sat, 12 Apr 2008 15:11:02 +0100 (BST) Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <48001EEA.3030209@canterbury.ac.nz> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> <79990c6b0804091546p5cf67f2ta800c675e7baa256@mail.gmail.com> <47FD667B.9040602@canterbury.ac.nz> <48001EEA.3030209@canterbury.ac.nz> Message-ID: On Sat, 12 Apr 2008, Greg Ewing wrote: > John J Lee wrote: >>> I keep hoping that someday Linux will support something >>> like MacOSX application bundles and frameworks, >> >> Not the same, but "something like": >> >> http://0install.net/ > > That looks interesting, but I'm not sure I'd quite > call it "something like". It looks like another case > of adding more complexity to fight existing complexity, > rather than removing the original complexity. > > In other words, it seems to be just another package > manager, albeit a particulary nice-sounding one. It allows you to think about "uninstallation" as "delete the app == delete the file" ("the file" might be different in different systems -- e.g. with ROX, it seems very similar to how I imagine Mac OS applications look, and certainly very similar to how RISC OS apps used to look). But it also (plausibly) claims to allow sharing of the data that comprises an application and its dependencies between users who don't trust each other (while preserving other desirable properties like the one in the previous paragraph, such as avoidance of version conflicts, etc.). I think that property probably justifies the added implementation complexity over unshared directories of files. And the desire for that property isn't a consequence of fighting existing complexity, right? John From pje at telecommunity.com Sat Apr 12 18:19:28 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 12 Apr 2008 12:19:28 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <1207973227.3212.73.camel@portableevil.develix.com> References: <1207973227.3212.73.camel@portableevil.develix.com> Message-ID: <20080412161922.6B3A33A406A@sparrow.telecommunity.com> At 09:07 PM 4/11/2008 -0700, Cliff Wells wrote: >Hi Phillip, > >I recently upgraded a hosting server from Python 2.4 to Python 2.5. >Many of the sites on this server couldn't easily be upgraded to 2.5 (at >least not in a timely fashion), so my interim solution was to simply >symlink ~/bin/python -> /usr/bin/python2.4 (~/bin is first in the user's >PATH). This worked swimmingly except for applications which relied on >Paste (I should note I use the method outlined here: >http://peak.telecommunity.com/DevCenter/EasyInstall#administrator-installation). > >The reason the Paste apps had an issue is because apparently setuptools >modifies the shebang line of paster to be "#!/usr/bin/python". This >hardcoded path overrides any PATH settings. I understand that this is >done at the time Paste is installed to tie it to the installed version >of Python. This makes sense, except that it doesn't do what it's >intended to when Python is upgraded (/usr/bin/python is no longer the >version it was installed with). > >It seems the correct solution to this is to use "#!/usr/bin/env >python" (or rather, evaluate `which env` to account for some systems >which have /bin/env rather that /usr/bin/env) which allows the sys admin >to override the Python binary much more easily. No, it isn't the correct solution. The correct solution is to re-run easy_install on the target -- it will leave the existing installed code in place, but regenerate the scripts with headers pointing to the currently-in-use Python. Even that is often not as good an idea as reinstalling for a new version of Python; some packages (notably, setuptools itself, but I have others) include different code and data depending on the Python version they are built for. For example, a package built for Python 2.5 might not include dependency specs for ctypes, sqlite, or wsgiref, but the same package might need to specify those dependencies on earlier versions of Python. By the way, please don't send me unsolicited private emails about setuptools unless you're a client or potential client. There's a reason that every single official web page about setuptools directs you to email the distutils-sig. I don't do private email for ANY of my open source projects, unless you're a client or looking to be one. Mailing lists have searchable archives, meaning that my time answering a question is being offset by its potential utility to a much larger audience. Thanks. From cliff at develix.com Sat Apr 12 18:58:28 2008 From: cliff at develix.com (Cliff Wells) Date: Sat, 12 Apr 2008 09:58:28 -0700 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080412161922.6B3A33A406A@sparrow.telecommunity.com> References: <1207973227.3212.73.camel@portableevil.develix.com> <20080412161922.6B3A33A406A@sparrow.telecommunity.com> Message-ID: <1208019508.3212.169.camel@portableevil.develix.com> On Sat, 2008-04-12 at 12:19 -0400, Phillip J. Eby wrote: > At 09:07 PM 4/11/2008 -0700, Cliff Wells wrote: > >It seems the correct solution to this is to use "#!/usr/bin/env > >python" (or rather, evaluate `which env` to account for some systems > >which have /bin/env rather that /usr/bin/env) which allows the sys admin > >to override the Python binary much more easily. > > No, it isn't the correct solution. The correct solution is to re-run > easy_install on the target -- it will leave the existing installed > code in place, but regenerate the scripts with headers pointing to > the currently-in-use Python. This is good to know, but still not as good as using env and the reason is simple: every Unix user on the planet knows the env solution and what it means if they see it in a script (they can override using PATH). The same cannot be said for the above. Unless I'm missing something, I can't see the advantage your way has over the traditional way. I'll be glad to hear how I'm wrong. > Even that is often not as good an idea as reinstalling for a new > version of Python; some packages (notably, setuptools itself, but I > have others) include different code and data depending on the Python > version they are built for. For example, a package built for Python > 2.5 might not include dependency specs for ctypes, sqlite, or > wsgiref, but the same package might need to specify those > dependencies on earlier versions of Python. Absolutely. Except I was specifically *not* upgrading many applications. Rather I wanted them to remain at 2.4. These are the ones I had to change the shebang line for. Now I know I could have run easy_install instead, but that information wasn't easily located at the time. > By the way, please don't send me unsolicited private emails about > setuptools unless you're a client or potential client. There's a > reason that every single official web page about setuptools directs > you to email the distutils-sig. I don't do private email for ANY of > my open source projects, unless you're a client or looking to be > one. Mailing lists have searchable archives, meaning that my time > answering a question is being offset by its potential utility to a > much larger audience. Thanks. My apologies. The link for the mailing list on the page that covers this on the peak site is quite small and follows a lot of text. Regards, Cliff Wells From gael.varoquaux at normalesup.org Sat Apr 12 19:08:49 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 12 Apr 2008 19:08:49 +0200 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <1208019508.3212.169.camel@portableevil.develix.com> References: <1207973227.3212.73.camel@portableevil.develix.com> <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> Message-ID: <20080412170849.GB8360@phare.normalesup.org> On Sat, Apr 12, 2008 at 09:58:28AM -0700, Cliff Wells wrote: > On Sat, 2008-04-12 at 12:19 -0400, Phillip J. Eby wrote: > > At 09:07 PM 4/11/2008 -0700, Cliff Wells wrote: > > >It seems the correct solution to this is to use "#!/usr/bin/env > > >python" (or rather, evaluate `which env` to account for some systems > > >which have /bin/env rather that /usr/bin/env) which allows the sys admin > > >to override the Python binary much more easily. > > No, it isn't the correct solution. The correct solution is to re-run > > easy_install on the target -- it will leave the existing installed > > code in place, but regenerate the scripts with headers pointing to > > the currently-in-use Python. > This is good to know, but still not as good as using env and the reason > is simple: every Unix user on the planet knows the env solution and what > it means if they see it in a script (they can override using PATH). The > same cannot be said for the above. +1. Ga?l From pje at telecommunity.com Sat Apr 12 19:49:37 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 12 Apr 2008 13:49:37 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <1208019508.3212.169.camel@portableevil.develix.com> References: <1207973227.3212.73.camel@portableevil.develix.com> <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> Message-ID: <20080412175146.51B3E3A409E@sparrow.telecommunity.com> At 09:58 AM 4/12/2008 -0700, Cliff Wells wrote: >Unless I'm missing something, I can't see the advantage your way has >over the traditional way. Well for one, it isn't affected by changes in PATH. From cliff at develix.com Sat Apr 12 21:30:04 2008 From: cliff at develix.com (Cliff Wells) Date: Sat, 12 Apr 2008 12:30:04 -0700 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080412175146.51B3E3A409E@sparrow.telecommunity.com> References: <1207973227.3212.73.camel@portableevil.develix.com> <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> Message-ID: <1208028604.3212.193.camel@portableevil.develix.com> On Sat, 2008-04-12 at 13:49 -0400, Phillip J. Eby wrote: > At 09:58 AM 4/12/2008 -0700, Cliff Wells wrote: > >Unless I'm missing something, I can't see the advantage your way has > >over the traditional way. > > Well for one, it isn't affected by changes in PATH. I'd call this a distinct disadvantage: the shebang line is a Unixism, not a Pythonism. Python applications shouldn't be an aberation on a Unix platform. PATH is *supposed* to affect applications. Trying to break this isn't helping anyone. It's also worth noting that this a deployment-specific consideration. Many deployers are not Python experts (or at least, we shouldn't expect them to be), but they ought to be fairly expert in their deployment platform. I would not expect a sysadmin to know where to start looking to know that they should re-run easy_install, but I would certainly expect them to know how to fix a PATH variable for an account. Regards, Cliff Wells From pje at telecommunity.com Sat Apr 12 23:53:31 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 12 Apr 2008 17:53:31 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <1208028604.3212.193.camel@portableevil.develix.com> References: <1207973227.3212.73.camel@portableevil.develix.com> <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> Message-ID: <20080412215326.B6CB03A406A@sparrow.telecommunity.com> At 12:30 PM 4/12/2008 -0700, Cliff Wells wrote: >PATH is *supposed* to affect applications. It affects which application you should run, not which interpreter you run the application with. From cliff at develix.com Sun Apr 13 00:26:52 2008 From: cliff at develix.com (Cliff Wells) Date: Sat, 12 Apr 2008 15:26:52 -0700 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080412215326.B6CB03A406A@sparrow.telecommunity.com> References: <1207973227.3212.73.camel@portableevil.develix.com> <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> Message-ID: <1208039212.3212.214.camel@portableevil.develix.com> On Sat, 2008-04-12 at 17:53 -0400, Phillip J. Eby wrote: > At 12:30 PM 4/12/2008 -0700, Cliff Wells wrote: > >PATH is *supposed* to affect applications. > > It affects which application you should run, not which interpreter > you run the application with. I think that's splitting hairs and pretty much just made up to defend your position. Regardless, ultimately it's dead wrong. Of *course* it should affect which interpreter you run. If I install a patched version of Python (or Perl, or PHP, or whatever), into /usr/local/bin and put that before /usr/bin in the PATH, I most absolutely expect that to be the interpreter that is used, regardless of what was used at the time the app was installed. If this breaks the application, it's fairly straightforward to adjust the PATH to fix it. Most importantly, it's something that a Unix admin (vs a Python programmer) can be expected to know. Anyway, regardless of who's correct concerning this issue (we can agree to disagree), I'm cannot understand why you'd want Python to behave differently (from a deployment standpoint) than other languages. If you want sysadmins to hate deploying Python apps, you are on the right road (make it unusual and require special Python knowledge to do "right"). In fact, "rightness" has little to do with it. Many conventions and standards aren't "right", but they continue to exist because that's what is understood. Being "right" at the expense of being non-standard isn't necessarily a good choice. Regards, Cliff From barry at python.org Sun Apr 13 00:38:56 2008 From: barry at python.org (Barry Warsaw) Date: Sat, 12 Apr 2008 18:38:56 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <1208039212.3212.214.camel@portableevil.develix.com> References: <1207973227.3212.73.camel@portableevil.develix.com> <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 12, 2008, at 6:26 PM, Cliff Wells wrote: > > On Sat, 2008-04-12 at 17:53 -0400, Phillip J. Eby wrote: >> At 12:30 PM 4/12/2008 -0700, Cliff Wells wrote: >>> PATH is *supposed* to affect applications. >> >> It affects which application you should run, not which interpreter >> you run the application with. > > I think that's splitting hairs and pretty much just made up to defend > your position. Regardless, ultimately it's dead wrong. Of *course* > it > should affect which interpreter you run. If I install a patched > version > of Python (or Perl, or PHP, or whatever), into /usr/local/bin and put > that before /usr/bin in the PATH, I most absolutely expect that to be > the interpreter that is used, regardless of what was used at the time > the app was installed. If this breaks the application, it's fairly > straightforward to adjust the PATH to fix it. Most importantly, it's > something that a Unix admin (vs a Python programmer) can be expected > to > know. Many operating systems and distributions now use Python extensively. Those should absolutely hardcode the #! line to use the system Python and should not in any way be affected by my $PATH. I've seen too many situations where changing $PATH breaks some aspect of the OS. I happen to think that setuptools is doing the right thing here, but it would be nice to have an override. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBSAE6AXEjvBPtnXfVAQJKuQP+JfqPDweFzQ1hN50XFQbi6InQEqL+b+dY M4UIdl5aS48ozoYIKGgM1aItgt/dfe0imCQkl2/cY9vzjmRRVVzK0RurPHjxyAWT CSFi+FuG54rJV4M0xwnoAZaEHQqo65g35iGz+mMe68ytKaSEt1TgukRzyF6RudBf 3Up+IgC8Kqc= =7t0k -----END PGP SIGNATURE----- From gael.varoquaux at normalesup.org Sun Apr 13 00:50:06 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 13 Apr 2008 00:50:06 +0200 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <1208039212.3212.214.camel@portableevil.develix.com> References: <1207973227.3212.73.camel@portableevil.develix.com> <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> Message-ID: <20080412225006.GN8360@phare.normalesup.org> On Sat, Apr 12, 2008 at 03:26:52PM -0700, Cliff Wells wrote: > Anyway, regardless of who's correct concerning this issue (we can agree > to disagree), I'm cannot understand why you'd want Python to behave > differently (from a deployment standpoint) than other languages. Numbers ! Only numbers can speak. On my Ubuntu box (standard, nothing special done to it): varoquau at epsilon:~$ grep '#!/usr/bin/env python' /usr/bin/* 2>/dev/null wc -l 33 varoquau at epsilon:~$ grep '#!/usr/bin/python' /usr/bin/* 2>/dev/null | wc -l 113 And if we don't look at the language type: varoquau at epsilon:~$ grep '#!/usr/bin/env' /usr/bin/* 2>/dev/null | wc -l 39 (the rest is pretty much perl, by the way). On a server I have an account on (Debian, I don't administer it): varoquau at quatramaran:~$ grep '#!/usr/bin/env' /usr/bin/* 2>/dev/null | wc -l 52 (this one has a much higher ratio of perl to python). On my mailhost: varoquau at phare:~$ grep '#!/usr/bin/python' /usr/bin/* 2>/dev/null | wc -l 10 varoquau at phare:~$ grep '#!/usr/bin/env' /usr/bin/* 2>/dev/null /usr/bin/raggle:#!/usr/bin/env ruby /usr/bin/web2png:#!/usr/bin/env python On my hosted and managed webserver (these guys seem to like ruby): (uiserver):u39018796:~ > grep '#!/usr/bin/python' /usr/bin/* 2>/dev/null | wc -l 0 (uiserver):u39018796:~ > grep '#!/usr/bin/env' /usr/bin/* 2>/dev/null /usr/bin/gem:#!/usr/bin/env ruby /usr/bin/gem_mirror:#!/usr/bin/env ruby /usr/bin/gem_server:#!/usr/bin/env ruby /usr/bin/gemlock:#!/usr/bin/env ruby /usr/bin/gemri:#!/usr/bin/env ruby /usr/bin/gemwhich:#!/usr/bin/env ruby /usr/bin/index_gem_repository:#!/usr/bin/env ruby (uiserver):u39018796:~ > grep '#!/usr/bin/ruby' /usr/bin/* 2>/dev/null /usr/bin/erb1.8:#!/usr/bin/ruby1.8 /usr/bin/irb1.8:#!/usr/bin/ruby1.8 /usr/bin/rdoc1.8:#!/usr/bin/ruby1.8 /usr/bin/testrb1.8:#!/usr/bin/ruby1.8 (uiserver):u39018796:~ > In conclusion, I think there is no real rule. It is more on a case by case basis. I prefer the "env". I could yell to the top of my voice on this mailing list to back my opinion. But the numbers are there, and they don't help me make my point :-). Ga?l From pje at telecommunity.com Sun Apr 13 01:54:19 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 12 Apr 2008 19:54:19 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <1208039212.3212.214.camel@portableevil.develix.com> References: <1207973227.3212.73.camel@portableevil.develix.com> <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> Message-ID: <20080412235549.8FBE13A409E@sparrow.telecommunity.com> At 03:26 PM 4/12/2008 -0700, Cliff Wells wrote: >On Sat, 2008-04-12 at 17:53 -0400, Phillip J. Eby wrote: > > At 12:30 PM 4/12/2008 -0700, Cliff Wells wrote: > > >PATH is *supposed* to affect applications. > > > > It affects which application you should run, not which interpreter > > you run the application with. > >I think that's splitting hairs and pretty much just made up to defend >your position. Anybody who thinks they can globally change which Python interpreter they're running their scripts with is cruising for a serious bruising. Better they discover their error sooner than later. Have you ever wondered why Red Hat has that stupid 'python2' executable? It's because they used #! lines pointing to 'python' rather than referencing a specific *version* of Python... and had to pay for it for years thereafter. (That's also, by the way, why easy_install also always installs a versioned executable name for itself.) From barry at python.org Sun Apr 13 01:58:24 2008 From: barry at python.org (Barry Warsaw) Date: Sat, 12 Apr 2008 19:58:24 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080412225006.GN8360@phare.normalesup.org> References: <1207973227.3212.73.camel@portableevil.develix.com> <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> Message-ID: <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 12, 2008, at 6:50 PM, Gael Varoquaux wrote: > On Sat, Apr 12, 2008 at 03:26:52PM -0700, Cliff Wells wrote: >> Anyway, regardless of who's correct concerning this issue (we can >> agree >> to disagree), I'm cannot understand why you'd want Python to behave >> differently (from a deployment standpoint) than other languages. > > Numbers ! Only numbers can speak. On my Ubuntu box (standard, nothing > special done to it): > > varoquau at epsilon:~$ grep '#!/usr/bin/env python' /usr/bin/* 2>/dev/ > null wc -l > 33 > > And if we don't look at the language type: > > varoquau at epsilon:~$ grep '#!/usr/bin/env' /usr/bin/* 2>/dev/null | > wc -l > 39 These are all broken and you should report bugs on them. I have reported many for Ubuntu. A system application should only ever depend on the system Python (or interpreter), never on the whims of your $PATH. If you want an application to depend on $PATH, grab a development release and install that in your favorite location. That's when /usr/bin/env is appropriate. Cheers, - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBSAFMoXEjvBPtnXfVAQKC2gQAgiFFH9KGinw3yy9wyAI6p1Hn2Sy+Sb6e x5z3B9kkT99zOl5AxdsaCEPWEIB+/Q2tRJypZXPVe9o7GPMIZETF9RzGIFV/tsmh LQF+lIPbdvXHsK1f1CibuRt3P+ij+EuSWH81FI7uE3SLXine51WRW+Pp+m3+74Xp kEP1dWn1GvE= =NFBv -----END PGP SIGNATURE----- From cliff at develix.com Sun Apr 13 02:28:22 2008 From: cliff at develix.com (Cliff Wells) Date: Sat, 12 Apr 2008 17:28:22 -0700 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080412235549.8FBE13A409E@sparrow.telecommunity.com> References: <1207973227.3212.73.camel@portableevil.develix.com> <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412235549.8FBE13A409E@sparrow.telecommunity.com> Message-ID: <1208046502.3212.227.camel@portableevil.develix.com> On Sat, 2008-04-12 at 19:54 -0400, Phillip J. Eby wrote: > At 03:26 PM 4/12/2008 -0700, Cliff Wells wrote: > > >On Sat, 2008-04-12 at 17:53 -0400, Phillip J. Eby wrote: > > > At 12:30 PM 4/12/2008 -0700, Cliff Wells wrote: > > > >PATH is *supposed* to affect applications. > > > > > > It affects which application you should run, not which interpreter > > > you run the application with. > > > >I think that's splitting hairs and pretty much just made up to defend > >your position. > > Anybody who thinks they can globally change which Python interpreter > they're running their scripts with is cruising for a serious > bruising. Better they discover their error sooner than later. If you are on RedHat, yes. On Gentoo, SourceMage, Foresight, etc the system is expected to do a "rolling" upgrade. So what you call an "error" is what is thought of as a "superior method" by many others. > Have you ever wondered why Red Hat has that stupid 'python2' > executable? It's because they used #! lines pointing to 'python' > rather than referencing a specific *version* of Python... and had to > pay for it for years thereafter. But these are system apps we are discussing, not user apps. System apps are not managed by setuptools, rather they are managed by the system's package manager. I agree they should point to the "system" python, whatever the system decides that should be, and the system should be responsible any upgrading to be done. In fact, it's specifically *because* of many distros depending so heavily on a particular version of Python that userspace Python apps need more flexibility in this area. /usr/bin/python is quite likely going to be out-of-date before the system as a whole is (or in my case, the other way around, as Gentoo moved to 2.5 before all my applications had). Either way, if they system Python and the userspace Python requirements get out of sync, it's clearly the userspace that's going to have to be flexible. > (That's also, by the way, why easy_install also always installs a > versioned executable name for itself.) Actually, if easy_install did something like "#!/usr/bin/python2.4" I think I'd have less of an issue. It would allow me to upgrade the system Python (which on Gentoo, for instance, you can safely do) while allowing legacy apps to continue to function until they can be upgraded. In fact, this would almost incidentally lead people down the right road to fixing it the way you suggest since they will certainly have to easy_install their required libraries under the new interpreter. Regards, Cliff From greg.ewing at canterbury.ac.nz Sun Apr 13 03:03:21 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 13 Apr 2008 13:03:21 +1200 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> <79990c6b0804091546p5cf67f2ta800c675e7baa256@mail.gmail.com> <47FD667B.9040602@canterbury.ac.nz> <48001EEA.3030209@canterbury.ac.nz> Message-ID: <48015BD9.5000502@canterbury.ac.nz> John J Lee wrote: > It allows you to think about "uninstallation" as "delete the app == > delete the file" But 0install doesn't do that, as far as I can tell -- it still keeps the data in some mysterious form and location known only to itself, and requires you to use special tools to install/remove apps. > with ROX, it seems very similar to how I imagine Mac OS > applications look Yes, ROX is very MacOSX-like, but I don't think it has anything to do with 0install. > But it also (plausibly) claims to allow sharing of the data that > comprises an application and its dependencies between users who don't > trust each other If ROX apps included a checksum, and the system verified it before running the app, that would give you the same thing trust-wise, I think. Dependency management is the part I agree is lacking in a MacOSX-like approach. Some tool for helping with that would be good to have. But I don't think it's necessary to make the components whose dependencies are being managed into anything complicated or mysterious in order to get that. They should just be files or directories that I can put into place myself, and look at to find out what I have. -- Greg From cliff at develix.com Sun Apr 13 03:40:41 2008 From: cliff at develix.com (Cliff Wells) Date: Sat, 12 Apr 2008 18:40:41 -0700 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080412235549.8FBE13A409E@sparrow.telecommunity.com> References: <1207973227.3212.73.camel@portableevil.develix.com> <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412235549.8FBE13A409E@sparrow.telecommunity.com> Message-ID: <1208050841.3212.240.camel@portableevil.develix.com> On Sat, 2008-04-12 at 19:54 -0400, Phillip J. Eby wrote: > (That's also, by the way, why easy_install also always installs a > versioned executable name for itself.) So if setuptools can rely on this name, why doesn't it use it in the shebang line? Regards, Cliff From waterbug at pangalactic.us Sun Apr 13 03:41:08 2008 From: waterbug at pangalactic.us (Stephen Waterbury) Date: Sat, 12 Apr 2008 21:41:08 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> References: <1207973227.3212.73.camel@portableevil.develix.com> <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> Message-ID: <480164B4.5080901@pangalactic.us> Barry Warsaw wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Apr 12, 2008, at 6:50 PM, Gael Varoquaux wrote: >> On Sat, Apr 12, 2008 at 03:26:52PM -0700, Cliff Wells wrote: >>> Anyway, regardless of who's correct concerning this issue (we can >>> agree >>> to disagree), I'm cannot understand why you'd want Python to behave >>> differently (from a deployment standpoint) than other languages. >> Numbers ! Only numbers can speak. On my Ubuntu box (standard, nothing >> special done to it): >> >> varoquau at epsilon:~$ grep '#!/usr/bin/env python' /usr/bin/* 2>/dev/ >> null wc -l >> 33 >> >> And if we don't look at the language type: >> >> varoquau at epsilon:~$ grep '#!/usr/bin/env' /usr/bin/* 2>/dev/null | >> wc -l >> 39 > > These are all broken and you should report bugs on them. I have > reported many for Ubuntu. A system application should only ever > depend on the system Python (or interpreter), never on the whims of > your $PATH. I agree with Barry here -- I would even go further and say that the Python used by system apps (i.e., apps that the OS depends on) should be separate and should not be on the default PATH (so the user can't use it accidentally), and no non-system apps should use the system Python unless specifically installed by the user or sysadmin into the system Python's area. I used to always set up my own Python[s] in /usr/local and put that first in my PATH, but I have gotten lazy lately, and sometimes it will bite me. ;) Steve From gael.varoquaux at normalesup.org Sun Apr 13 11:50:16 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 13 Apr 2008 11:50:16 +0200 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <480164B4.5080901@pangalactic.us> References: <1207973227.3212.73.camel@portableevil.develix.com> <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> Message-ID: <20080413095016.GE10092@phare.normalesup.org> On Sat, Apr 12, 2008 at 09:41:08PM -0400, Stephen Waterbury wrote: > > These are all broken and you should report bugs on them. I have > > reported many for Ubuntu. A system application should only ever > > depend on the system Python (or interpreter), never on the whims of > > your $PATH. > I agree with Barry here -- I would even go further and say that > the Python used by system apps (i.e., apps that the OS depends on) > should be separate and should not be on the default PATH (so the user > can't use it accidentally), and no non-system apps should use the > system Python unless specifically installed by the user or sysadmin > into the system Python's area. I agree with you, and not with Barry that all these are broken. If you look at the apps that use "#!/usr/bin/env", they are all pretty non-system apps, like for instance "f2py". I think it is good that these apps use the default python: I don't want to have f2py to be running for another python than my default one. I am actually quite surprised to see that ipython is installed with a hardcoded path to /usr/bin/python. Anyway, I am not going to bug report anything, as my system works just fine, thank you. Ga?l From barry at python.org Sun Apr 13 13:59:21 2008 From: barry at python.org (Barry Warsaw) Date: Sun, 13 Apr 2008 07:59:21 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <480164B4.5080901@pangalactic.us> References: <1207973227.3212.73.camel@portableevil.develix.com> <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> Message-ID: <1006AECE-3273-418A-A804-228177D4DCBB@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 12, 2008, at 9:41 PM, Stephen Waterbury wrote: > > I used to always set up my own Python[s] in /usr/local > and put that first in my PATH, but I have gotten lazy lately, and > sometimes it will bite me. ;) On Debian and derivatives (e.g. Ubuntu) you might have even more fun. They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of the system python*! This means that you can break your system Python by installing a version of Python from source and then distutil'ing things into there. Astoundingly, this is promoted as a feature. I've reported bugs on this and had discussions with some of the Debian Python packaging folks. I'm hoping that we'll find a solution that doesn't collide with a from-source default Python build. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBSAH1mnEjvBPtnXfVAQJAvAP/S1p03Jlq73fZB4H6Qt7cH/qXpInQ4G7i VDMdH0yxZ3Fih8LG/eFcvpu0YCI7iHWtct+h2GXA/gUVPlgWxwJDE2Ahixg9MNS3 faETU1DNqJb4oUbwjZD/FzwLCGRXp/17s4+Q2JX4YaOvShsVeS+F3Kr+3zwfDL0m hrMRfaJTYSE= =a8MZ -----END PGP SIGNATURE----- From gael.varoquaux at normalesup.org Sun Apr 13 14:04:05 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 13 Apr 2008 14:04:05 +0200 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <1006AECE-3273-418A-A804-228177D4DCBB@python.org> References: <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> Message-ID: <20080413120405.GB24381@phare.normalesup.org> On Sun, Apr 13, 2008 at 07:59:21AM -0400, Barry Warsaw wrote: > On Apr 12, 2008, at 9:41 PM, Stephen Waterbury wrote: > > I used to always set up my own Python[s] in /usr/local > > and put that first in my PATH, but I have gotten lazy lately, and > > sometimes it will bite me. ;) > On Debian and derivatives (e.g. Ubuntu) you might have even more fun. > They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of > the system python*! This means that you can break your system Python > by installing a version of Python from source and then distutil'ing > things into there. Astoundingly, this is promoted as a feature. I want it like that. You are confusing /opt and /usr/local. This is the way I expect things to work. I do not want to install my own packages in "/usr/lib/pythonX.Y/site-packages". This is for apt to deal with, not for me. Ga?l From barry at python.org Sun Apr 13 14:07:38 2008 From: barry at python.org (Barry Warsaw) Date: Sun, 13 Apr 2008 08:07:38 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080413095016.GE10092@phare.normalesup.org> References: <1207973227.3212.73.camel@portableevil.develix.com> <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <20080413095016.GE10092@phare.normalesup.org> Message-ID: <8EF71D63-7975-439B-9C87-1046D52F2BAC@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 13, 2008, at 5:50 AM, Gael Varoquaux wrote: > On Sat, Apr 12, 2008 at 09:41:08PM -0400, Stephen Waterbury wrote: >>> These are all broken and you should report bugs on them. I have >>> reported many for Ubuntu. A system application should only ever >>> depend on the system Python (or interpreter), never on the whims of >>> your $PATH. > >> I agree with Barry here -- I would even go further and say that >> the Python used by system apps (i.e., apps that the OS depends on) >> should be separate and should not be on the default PATH (so the user >> can't use it accidentally), and no non-system apps should use the >> system Python unless specifically installed by the user or sysadmin >> into the system Python's area. > > I agree with you, and not with Barry that all these are broken. If you > look at the apps that use "#!/usr/bin/env", they are all pretty > non-system apps, like for instance "f2py". I think it is good that > these > apps use the default python: I don't want to have f2py to be running > for > another python than my default one. I am actually quite surprised to > see > that ipython is installed with a hardcoded path to /usr/bin/python. > > Anyway, I am not going to bug report anything, as my system works just > fine, thank you. I'm not sure how you can disagree with me and not with Steve since he and I agree the system apps are broken. ;) Oh well, doesn't matter. Steve has a very interesting idea though, and it makes sense to me. I'm not sure where in the FHS you'd put that, but it makes sense for Python-written system apps to use the hidden Python and for /usr/bin/ python to be one that users can play with. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBSAH3jHEjvBPtnXfVAQKIJAP/WWE6i1pjpFJU3Z0mT3AiYeIKZjTfVXjV jC0DAgGj9AWS9TS1eGYWntb65C3ibZK2vsFjkr4Zg32yx1Dl2qvOfQ3yPD5+z7k5 pLP+LfP1U80HDl2poPRJCetnwV/F29IJpEHgNnWZboqrRrPxJlD3WI8L9D4gECP5 I19vl6e/IVo= =3/Pm -----END PGP SIGNATURE----- From waterbug at pangalactic.us Sun Apr 13 16:45:58 2008 From: waterbug at pangalactic.us (Stephen Waterbury) Date: Sun, 13 Apr 2008 10:45:58 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080413094930.GD10092@phare.normalesup.org> References: <1207973227.3212.73.camel@portableevil.develix.com> <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <20080413094930.GD10092@phare.normalesup.org> Message-ID: <48021CA6.70401@pangalactic.us> Gael Varoquaux wrote: > On Sat, Apr 12, 2008 at 09:41:08PM -0400, Stephen Waterbury wrote: >>> These are all broken and you should report bugs on them. I have >>> reported many for Ubuntu. A system application should only ever >>> depend on the system Python (or interpreter), never on the whims of >>> your $PATH. > >> I agree with Barry here -- I would even go further and say that >> the Python used by system apps (i.e., apps that the OS depends on) >> should be separate and should not be on the default PATH (so the user >> can't use it accidentally), and no non-system apps should use the >> system Python unless specifically installed by the user or sysadmin >> into the system Python's area. > > I agree with you, and not with Barry that all these are broken. If you > look at the apps that use "#!/usr/bin/env", they are all pretty > non-system apps, like for instance "f2py". I think it is good that these > apps use the default python: I don't want to have f2py to be running for > another python than my default one. I am actually quite surprised to see > that ipython is installed with a hardcoded path to /usr/bin/python. > > Anyway, I am not going to bug report anything, as my system works just > fine, thank you. Yes, your luck holds -- as does mine, for now ... but since Barry works for Canonical (besides being Python's new Release Manager), he has a civic responsibility to report Ubuntu Python bugs even if his own system works ... and since that can be quite time-consuming, I can't blame him for trying to recruit some help ... ;) Steve From waterbug at pangalactic.us Sun Apr 13 17:03:20 2008 From: waterbug at pangalactic.us (Stephen Waterbury) Date: Sun, 13 Apr 2008 11:03:20 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080413120405.GB24381@phare.normalesup.org> References: <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <20080413120405.GB24381@phare.normalesup.org> Message-ID: <480220B8.2080704@pangalactic.us> Gael Varoquaux wrote: > On Sun, Apr 13, 2008 at 07:59:21AM -0400, Barry Warsaw wrote: >> On Apr 12, 2008, at 9:41 PM, Stephen Waterbury wrote: > >>> I used to always set up my own Python[s] in /usr/local >>> and put that first in my PATH, but I have gotten lazy lately, and >>> sometimes it will bite me. ;) > >> On Debian and derivatives (e.g. Ubuntu) you might have even more fun. >> They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of >> the system python*! This means that you can break your system Python >> by installing a version of Python from source and then distutil'ing >> things into there. Astoundingly, this is promoted as a feature. > > I want it like that. You are confusing /opt and /usr/local. Heh ... that shows my Berkeley roots -- I'm not confused, I'm just an old SunOS guy still in denial about that damn SysV stuff that Solaris brought in, like using /opt instead of /usr/local -- bah, humbug! The OS has /usr; /usr/local is *mine* dammit! ;) Steve From waterbug at pangalactic.us Sun Apr 13 17:16:57 2008 From: waterbug at pangalactic.us (Stephen Waterbury) Date: Sun, 13 Apr 2008 11:16:57 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <1006AECE-3273-418A-A804-228177D4DCBB@python.org> References: <1207973227.3212.73.camel@portableevil.develix.com> <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> Message-ID: <480223E9.5040902@pangalactic.us> Barry Warsaw wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Apr 12, 2008, at 9:41 PM, Stephen Waterbury wrote: >> >> I used to always set up my own Python[s] in /usr/local >> and put that first in my PATH, but I have gotten lazy lately, and >> sometimes it will bite me. ;) > > On Debian and derivatives (e.g. Ubuntu) you might have even more fun. > They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of the > system python*! This means that you can break your system Python by > installing a version of Python from source and then distutil'ing things > into there. Astoundingly, this is promoted as a feature. That *is* astounding. I've been using Ubuntu for several years now, and I blush to admit I never noticed that until just the other day when I tried installing Python 2.6 from source (into the default /usr/local location, of course) and it broke all my running web stuff, so I hastily removed it -- sheesh! As I mentioned, I've been "lazy" about using the system Python for my own apps and development, but I would feel much more comfortable if the system Python had its own space. > I've reported bugs on this and had discussions with some of the Debian > Python packaging folks. I'm hoping that we'll find a solution that > doesn't collide with a from-source default Python build. Thanks for fighting the good fight, Barry -- hang in there! Maybe you can push for my "radical" solution. :) Steve From gael.varoquaux at normalesup.org Sun Apr 13 17:29:25 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 13 Apr 2008 17:29:25 +0200 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <480223E9.5040902@pangalactic.us> References: <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> Message-ID: <20080413152925.GF24381@phare.normalesup.org> On Sun, Apr 13, 2008 at 11:16:57AM -0400, Stephen Waterbury wrote: > > On Debian and derivatives (e.g. Ubuntu) you might have even more fun. > > They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of the > > system python*! This means that you can break your system Python by > > installing a version of Python from source and then distutil'ing things > > into there. Astoundingly, this is promoted as a feature. > That *is* astounding. I've been using Ubuntu for several years now, > and I blush to admit I never noticed that until just the other day when > I tried installing Python 2.6 from source (into the default /usr/local > location, of course) and it broke all my running web stuff, so I hastily > removed it -- sheesh! > As I mentioned, I've been "lazy" about using the system Python for my > own apps and development, but I would feel much more comfortable if > the system Python had its own space. > > I've reported bugs on this and had discussions with some of the Debian > > Python packaging folks. I'm hoping that we'll find a solution that > > doesn't collide with a from-source default Python build. Well, I am not too happy about what I hear here. It seems to me as if Ubuntu is headed the macOS X way, where (it seems) that a second Python needs to be installed on top of the system Python to add modules to it. I know it is not exactly the case, but this change is _really_ going to confuse people and I can this the mailing lists of module developper pilling up with requests for help. There needs to be a way for the user to install his own modules and keep using the system Python. I think it is a very bad idea to do it in /usr. Yes, you can teach your users to add things to their PYTHONPATH, but this put one additional thing to learn for people who just want to get things done. I think users are going to hate this. Maybe something less drastic can be done, by making sure that the /usr/local is far in the PYTHONPATH, that way packages installed in /usr/local do not override the system packages. Of course setuptools will break this, because it adds itsef to front of the PYTHONPATH, and totally break the PYTHONPATH semantics (Grrr). I still think you guys have it wrong, and if you want a new Python install, you do it in /opt. For me /usr/local is for user programs that cooperate for system programs. My 2 cents, Ga?l From waterbug at pangalactic.us Sun Apr 13 18:31:08 2008 From: waterbug at pangalactic.us (Stephen Waterbury) Date: Sun, 13 Apr 2008 12:31:08 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080413152925.GF24381@phare.normalesup.org> References: <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> Message-ID: <4802354C.5010103@pangalactic.us> Gael Varoquaux wrote: > On Sun, Apr 13, 2008 at 11:16:57AM -0400, Stephen Waterbury wrote: >>> On Debian and derivatives (e.g. Ubuntu) you might have even more fun. >>> They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of the >>> system python*! This means that you can break your system Python by >>> installing a version of Python from source and then distutil'ing things >>> into there. Astoundingly, this is promoted as a feature. > >> That *is* astounding. I've been using Ubuntu for several years now, >> and I blush to admit I never noticed that until just the other day when >> I tried installing Python 2.6 from source (into the default /usr/local >> location, of course) and it broke all my running web stuff, so I hastily >> removed it -- sheesh! > >> As I mentioned, I've been "lazy" about using the system Python for my >> own apps and development, but I would feel much more comfortable if >> the system Python had its own space. > >>> I've reported bugs on this and had discussions with some of the Debian >>> Python packaging folks. I'm hoping that we'll find a solution that >>> doesn't collide with a from-source default Python build. > Well, I am not too happy about what I hear here. It seems to me as if > Ubuntu is headed the macOS X way, where (it seems) that a second Python > needs to be installed on top of the system Python to add modules to it. I > know it is not exactly the case, but this change is _really_ going to > confuse people and I can this the mailing lists of module developper > pilling up with requests for help. There needs to be a way for the user > to install his own modules and keep using the system Python. I think you misunderstand my proposal, Gael ... You and I are using "system Python" to mean different things. In my proposal, *the* "system Python" and a packaged Python installed using the system's package manager would be separate things (and may or may not be the same version of Python)! What I am proposing: 1) the OS comes with its own "system Python", which is installed not as the "python" package, but as some OS-required package (maybe call it "system-python" or something) and it goes into /usr/system/bin/python or whatever -- it doesn't matter what the path is as long as it's not /usr/bin or anything on the default path. And system utilities that are python scripts should have their own system-specific, hard-coded shebang line. 2) separately from the "system Python", the available packages shown by the system's package manager include one or more "python.x" packages which are python interpreters that the user or sysadmin can optionally install, and which go into /usr. And the system package manager -- e.g., apt on Debian/Ubuntu systems) would have all its usual nicely-packaged python apps (python-this, python-that, ...) that would also install into /usr and use the nicely-packaged python (not to be confused with the "system Python" of 1). Steve From waterbug at pangalactic.us Sun Apr 13 18:36:13 2008 From: waterbug at pangalactic.us (Stephen Waterbury) Date: Sun, 13 Apr 2008 12:36:13 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <4802354C.5010103@pangalactic.us> References: <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802354C.5010103@pangalactic.us> Message-ID: <4802367D.6000300@pangalactic.us> Stephen Waterbury wrote: Oops, my email client doesn't detect unbalanced parens -- typo here (what I get for too-complicated paren/double-dash-mixing ;): > optionally install, and which go into /usr. And the system package > manager -- e.g., apt on Debian/Ubuntu systems) would have all its usual ^ should be "--" > nicely-packaged python apps (python-this, python-that, ...) that would > also install into /usr and use the nicely-packaged python (not to be > confused with the "system Python" of 1). Steve From barry at python.org Sun Apr 13 18:42:43 2008 From: barry at python.org (Barry Warsaw) Date: Sun, 13 Apr 2008 12:42:43 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080413120405.GB24381@phare.normalesup.org> References: <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <20080413120405.GB24381@phare.normalesup.org> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 13, 2008, at 8:04 AM, Gael Varoquaux wrote: > On Sun, Apr 13, 2008 at 07:59:21AM -0400, Barry Warsaw wrote: >> On Apr 12, 2008, at 9:41 PM, Stephen Waterbury wrote: > >>> I used to always set up my own Python[s] in /usr/local >>> and put that first in my PATH, but I have gotten lazy lately, and >>> sometimes it will bite me. ;) > >> On Debian and derivatives (e.g. Ubuntu) you might have even more fun. >> They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of >> the system python*! This means that you can break your system Python >> by installing a version of Python from source and then distutil'ing >> things into there. Astoundingly, this is promoted as a feature. > > I want it like that. You are confusing /opt and /usr/local. > > This is the way I expect things to work. I do not want to install my > own > packages in "/usr/lib/pythonX.Y/site-packages". This is for apt to > deal > with, not for me. Actually, the solution that I believe is a good compromise is to find some other path in /usr/local to augment the system Python's sys.path with. All I'm saying is that Python itself uses /usr/local/lib/ pythonX.Y/site-packages as the default from-source path, so if Debian wants a /usr/local path, it can pick some other subdirectory path and still meet its goals. BTW, I believe Debian is unique here. I talked to the Fedora guys and I've talked to a few Gentoo guys and both seemed surprised by Debian's policy here. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBSAI4A3EjvBPtnXfVAQLeagP/aCKr6WL+qvvM8vhABFxJDBbOJw1hQUQT fCUwL3bZ21a6iKiHLv5mdVBfaFjcqEmnbGqizUmTgnTuG+SaUXHNqBBG2lDCTZUF fNwWThN/TRtqO5OdknqFcfTQARtBE+YN52CpugjSmRrudZtAAdslWds8HnUP3nlG kN9J0TsEo1E= =DDbJ -----END PGP SIGNATURE----- From gael.varoquaux at normalesup.org Sun Apr 13 18:43:25 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 13 Apr 2008 18:43:25 +0200 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <4802354C.5010103@pangalactic.us> References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802354C.5010103@pangalactic.us> Message-ID: <20080413164325.GI24381@phare.normalesup.org> On Sun, Apr 13, 2008 at 12:31:08PM -0400, Stephen Waterbury wrote: > What I am proposing: > 1) the OS comes with its own "system Python", which is installed > not as the "python" package, but as some OS-required package > (maybe call it "system-python" or something) and it goes into > /usr/system/bin/python or whatever -- it doesn't matter what the > path is as long as it's not /usr/bin or anything on the default > path. And system utilities that are python scripts should have > their own system-specific, hard-coded shebang line. > 2) separately from the "system Python", the available packages > shown by the system's package manager include one or more "python.x" > packages which are python interpreters that the user or sysadmin can > optionally install, and which go into /usr. And the system package > manager -- e.g., apt on Debian/Ubuntu systems) would have all its usual > nicely-packaged python apps (python-this, python-that, ...) that would > also install into /usr and use the nicely-packaged python (not to be > confused with the "system Python" of 1). OK. I am starting to see what you mean. I agree it does make sense. It seems to me that you are bringing in a distinction between "system Python scripts" and user Python script. For me the system Python scripts should live in "/bin" and use the system Python, and the users should live in "/usr/bin" and use "/usr/bin/env python". But that's just me. Inspecting my boxes did show that this is quite close to the way it is already on Debian systems. I don't have accounts on other kind of Unix, so I can't see how it is done elsewhere. Cheers, Ga?l From barry at python.org Sun Apr 13 18:46:09 2008 From: barry at python.org (Barry Warsaw) Date: Sun, 13 Apr 2008 12:46:09 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <48021CA6.70401@pangalactic.us> References: <1207973227.3212.73.camel@portableevil.develix.com> <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <20080413094930.GD10092@phare.normalesup.org> <48021CA6.70401@pangalactic.us> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 13, 2008, at 10:45 AM, Stephen Waterbury wrote: > Yes, your luck holds -- as does mine, for now ... but since Barry > works for Canonical (besides being Python's new Release Manager), > he has a civic responsibility to report Ubuntu Python bugs even if > his own system works ... and since that can be quite time-consuming, > I can't blame him for trying to recruit some help ... ;) I should clarify a few things though: I'm not on the Ubuntu release team (I work on Launchpad) and AFAIK Ubuntu simply inherits Debian's policy here. Speaking purely as a user, I think Debian is wrong and I've reported bugs on it, and I think they are open to a compromise that can work for everyone. That's as much as I can do (short of turning off Python's svn notifications and sneaking a change in -- but if I had that power as Python's RM, I'd use it for much more important things[1] :). - -Barry [1] http://www.artima.com/weblogs/viewpost.jsp?thread=173477 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBSAI40nEjvBPtnXfVAQL7BgP9HWIajzqzLq0y4UZmSemHooqoukKnQA9O sxKAwzXthZPt5P9CUAduZKZqvYSQvSTmtwSMqoBjLGmfJo4WQIQ5BLZrq2s1cO6I SPPUWQ1RSfHAeczqTKMnlQpmHC5C0VBwNQUoq7+JWlbQVjr76CMsGcrHV4Cq3xPK xiEzKr/0HJg= =8R5Q -----END PGP SIGNATURE----- From barry at python.org Sun Apr 13 18:47:11 2008 From: barry at python.org (Barry Warsaw) Date: Sun, 13 Apr 2008 12:47:11 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <480220B8.2080704@pangalactic.us> References: <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <20080413120405.GB24381@phare.normalesup.org> <480220B8.2080704@pangalactic.us> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 13, 2008, at 11:03 AM, Stephen Waterbury wrote: > Gael Varoquaux wrote: >> On Sun, Apr 13, 2008 at 07:59:21AM -0400, Barry Warsaw wrote: >>> On Apr 12, 2008, at 9:41 PM, Stephen Waterbury wrote: >> >>>> I used to always set up my own Python[s] in /usr/local >>>> and put that first in my PATH, but I have gotten lazy lately, and >>>> sometimes it will bite me. ;) >> >>> On Debian and derivatives (e.g. Ubuntu) you might have even more >>> fun. >>> They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of >>> the system python*! This means that you can break your system >>> Python >>> by installing a version of Python from source and then distutil'ing >>> things into there. Astoundingly, this is promoted as a feature. >> >> I want it like that. You are confusing /opt and /usr/local. > > Heh ... that shows my Berkeley roots -- I'm not confused, > I'm just an old SunOS guy still in denial about that damn SysV stuff > that Solaris brought in, like using /opt instead of /usr/local -- > bah, humbug! The OS has /usr; /usr/local is *mine* dammit! ;) You and me both Steve! Of course, we're old farts so when we retire the kids can have their /usr/local. Until then, they'll have to pry it out of hands. :) - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBSAI5EXEjvBPtnXfVAQKU3wP+LnbcfTiE9AwDmIdNGrbnijGxWM5xGNgF Zg42Zs9RWWjjCABoOkZyHw1k3f0mp5DZtgA/XR8joPiVwyq8VV2diIQI4pw/c2No AGP0XpTOeqjHSm84jEDIT1YaFfp5NTuDzN360zxM4neLi9nG7pVD+OGebL4n/OPh aSB08rP/vKM= =cqEh -----END PGP SIGNATURE----- From floris.bruynooghe at gmail.com Sun Apr 13 19:05:08 2008 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Sun, 13 Apr 2008 18:05:08 +0100 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <1208039212.3212.214.camel@portableevil.develix.com> References: <1207973227.3212.73.camel@portableevil.develix.com> <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> Message-ID: <20080413170508.GA5703@laurie.devork> On Sat, Apr 12, 2008 at 03:26:52PM -0700, Cliff Wells wrote: > > On Sat, 2008-04-12 at 17:53 -0400, Phillip J. Eby wrote: > > At 12:30 PM 4/12/2008 -0700, Cliff Wells wrote: > > >PATH is *supposed* to affect applications. > > > > It affects which application you should run, not which interpreter > > you run the application with. > > I think that's splitting hairs and pretty much just made up to defend > your position. Nope, his statement is very correct. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From jjl at pobox.com Sun Apr 13 19:52:41 2008 From: jjl at pobox.com (John J Lee) Date: Sun, 13 Apr 2008 18:52:41 +0100 (BST) Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <48015BD9.5000502@canterbury.ac.nz> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> <79990c6b0804091546p5cf67f2ta800c675e7baa256@mail.gmail.com> <47FD667B.9040602@canterbury.ac.nz> <48001EEA.3030209@canterbury.ac.nz> <48015BD9.5000502@canterbury.ac.nz> Message-ID: On Sun, 13 Apr 2008, Greg Ewing wrote: > John J Lee wrote: > >> It allows you to think about "uninstallation" as "delete the app == delete >> the file" > > But 0install doesn't do that, as far as I can tell -- it > still keeps the data in some mysterious form and location > known only to itself, and requires you to use special tools > to install/remove apps. If you have a network connection, about the only reason for not wanting an app to be "installed" is that it has changed the behaviour of your system somehow, just by being in the "installed" state. But the presence of an app in the 0install cache -- which is what you mean here by "installed" -- doesn't change the behaviour of your system. One other reason, of course, is to free up disk space. You're correct that special tools are needed to manage the cache, and that that complicates the UI. I think that's a fair price to pay for safe sharing of data between users. >> with ROX, it seems very similar to how I imagine Mac OS >> applications look > > Yes, ROX is very MacOSX-like, but I don't think it has > anything to do with 0install. 0install provides one way of implementing that kind of system. If you want to share data, it's a better way than unshared directories of files. That's how 0install and ROX are related, from this perspective. >> But it also (plausibly) claims to allow sharing of the data that comprises >> an application and its dependencies between users who don't trust each >> other > > If ROX apps included a checksum, and the system verified it > before running the app, that would give you the same thing > trust-wise, I think. [...] That's an interesting idea, but how would the system find the app? If it doesn't, the data won't be shared. John From pje at telecommunity.com Sun Apr 13 20:04:56 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 13 Apr 2008 14:04:56 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080413152925.GF24381@phare.normalesup.org> References: <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> Message-ID: <20080413180536.B87DF3A40C2@sparrow.telecommunity.com> At 05:29 PM 4/13/2008 +0200, Gael Varoquaux wrote: >Of course setuptools will >break this, because it adds itsef to front of the PYTHONPATH, and totally >break the PYTHONPATH semantics (Grrr). The entire point of PYTHONPATH is to be able to put things at the front of it, so I don't see how that's breaking anything. When you ask easy_install to install something as the default importable version of that package, the only way to ensure that is by placing that thing at the front of sys.path, to override any other version of it that might be installed elsewhere on sys.path. Well, the only way short of going through all of sys.path and deleting anything that might potentially be in the way. That approach was tried in I think the 0.4 line of setuptools, however, and it wasn't very popular. :) From gael.varoquaux at normalesup.org Sun Apr 13 20:26:09 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 13 Apr 2008 20:26:09 +0200 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080413180536.B87DF3A40C2@sparrow.telecommunity.com> References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <20080413180536.B87DF3A40C2@sparrow.telecommunity.com> Message-ID: <20080413182609.GN24381@phare.normalesup.org> On Sun, Apr 13, 2008 at 02:04:56PM -0400, Phillip J. Eby wrote: > At 05:29 PM 4/13/2008 +0200, Gael Varoquaux wrote: >> Of course setuptools will >> break this, because it adds itsef to front of the PYTHONPATH, and totally >> break the PYTHONPATH semantics (Grrr). > > The entire point of PYTHONPATH is to be able to put things at the front of > it, so I don't see how that's breaking anything. Well actually, my phrasing in the above was catastrophic. I have mixed PYTHONPATH and sys.path. Anyhow, I have a problem with how setuptools deals with both: * sys.path > When you ask easy_install to install something as the default importable > version of that package, the only way to ensure that is by placing that > thing at the front of sys.path, to override any other version of it that > might be installed elsewhere on sys.path. What I would want is setuptools to respect the priorities for packages depending on where they are installed. In other words, if I install a package in /usr/local/lib and this is further down the priorities than /usr/lib, then I don't expect this package to override module imports happening from /usr/lib. You can justify what setuptools is doing from an implementation point of view, but it doesn't change the fact that it is doing the wrong thing. Wrong because it goes against what Python does. Thus I have to learn some completely new import semantics to deal with the fact that some of the packages on my system where installed by setuptools. In addition there is a good reason behind the standard Python behavior. * PYTHONPATH With standard Python, I can override the import mechanism using the PYTHONPATH. This way I can make sure that the modules I want get imported. And pretty much all the programs I can think of with a path environment variable behave like this. Now if I have a module installed with setuptools, I no longer have the possibility to do this, because setuptools inserts it self before the PYTHONPATH. First of all this really breaks the users expectancies. Second it puts people in a dead end. Actually it forces them to use setuptools, or to monkey-patch sys.path themselves, to override the import of a module. Once again you are expecting your users to know way more than a beginner's knowledge of Python. Python is simple and does what you expect... no longer. This is way worse than not respecting sys.path priorities. I have lost hours on that, and when I found out why, ... well I won't copy my remarks about setuptools on this mailing list. And what has happened to me has probably happened to many. This second problem could probably easily be fixed, though, by some more clever monkey patching of the sys.path. Actually both problems are solvable, if you care about not changing the way Python works. If you don't care, than make it obvious everywhere. But even if you do, I think it is a very bad thing, because people will still fall for the trap (eg I copy my scripts to a new server, which uses setuptools, although I don't know it, and I actually don't know what setuptools are). Don't call it Python, if you are going to change fundamentally the way it works. My 2 cents, Ga?l From setuptools at bugs.python.org Sun Apr 13 23:11:38 2008 From: setuptools at bugs.python.org (=?utf-8?q?Martin_v._L=C3=B6wis?=) Date: Sun, 13 Apr 2008 21:11:38 +0000 Subject: [Distutils] [issue7] Test 1 Message-ID: <1208121098.91.0.387513311865.issue7@psf.upfronthosting.co.za> New submission from Martin v. L?wis : Testing the installation ---------- messages: 7 nosy: loewis priority: bug status: unread title: Test 1 _______________________________________________ Setuptools tracker _______________________________________________ From pje at telecommunity.com Sun Apr 13 23:12:48 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 13 Apr 2008 17:12:48 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080413182609.GN24381@phare.normalesup.org> References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <20080413180536.B87DF3A40C2@sparrow.telecommunity.com> <20080413182609.GN24381@phare.normalesup.org> Message-ID: <20080413211243.1C1AF3A409D@sparrow.telecommunity.com> At 08:26 PM 4/13/2008 +0200, Gael Varoquaux wrote: >On Sun, Apr 13, 2008 at 02:04:56PM -0400, Phillip J. Eby wrote: > > At 05:29 PM 4/13/2008 +0200, Gael Varoquaux wrote: > >> Of course setuptools will > >> break this, because it adds itsef to front of the PYTHONPATH, and totally > >> break the PYTHONPATH semantics (Grrr). > > > > The entire point of PYTHONPATH is to be able to put things at the front of > > it, so I don't see how that's breaking anything. > >Well actually, my phrasing in the above was catastrophic. I have mixed >PYTHONPATH and sys.path. Anyhow, I have a problem with how setuptools >deals with both: > >* sys.path > > > When you ask easy_install to install something as the default importable > > version of that package, the only way to ensure that is by placing that > > thing at the front of sys.path, to override any other version of it that > > might be installed elsewhere on sys.path. > >What I would want is setuptools to respect the priorities for packages >depending on where they are installed. In other words, if I install a >package in /usr/local/lib and this is further down the priorities than >/usr/lib, then I don't expect this package to override module imports >happening from /usr/lib. And you're saying that setuptools does this? Since when? [pause to investigate] Hm... that's a bug, actually. If there are multiple 'site' directories, then what you describe will indeed happen; eggs in the later 'site' directory will indeed override those in the former, which should not happen. The issue is that the postamble of easy-install.pth is simply inserting the eggs at the head of sys.path, even in cases where they could simply be inserted immediately before the directory the .pth file is in. One minor issue, however, is that eggs in the systemwide site-packages must also be able to override the stdlib - so simply locating the correct insertion point isn't going to fix the whole problem. However, now that it's a known bug, it's merely an engineering problem to fix it. :) >With standard Python, I can override the import mechanism using the >PYTHONPATH. This way I can make sure that the modules I want get >imported. And pretty much all the programs I can think of with a path >environment variable behave like this. Now if I have a module installed >with setuptools, I no longer have the possibility to do this, because >setuptools inserts it self before the PYTHONPATH. I'm not following you here. Are you trying to specify the use of a different version of setuptools? What do you mean by "setuptools inserts itself before the PYTHONPATH"? >This second problem could probably easily be fixed, though, by some more >clever monkey patching of the sys.path. In general, setuptools' path management could be improved by tweaking its path hacks so that an egg is positioned only in front of the directory that contains it (except for lib/site-packages eggs, which need to also override lib). Patches welcome. From gael.varoquaux at normalesup.org Sun Apr 13 23:24:20 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 13 Apr 2008 23:24:20 +0200 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080413211243.1C1AF3A409D@sparrow.telecommunity.com> References: <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <20080413180536.B87DF3A40C2@sparrow.telecommunity.com> <20080413182609.GN24381@phare.normalesup.org> <20080413211243.1C1AF3A409D@sparrow.telecommunity.com> Message-ID: <20080413212420.GU24381@phare.normalesup.org> On Sun, Apr 13, 2008 at 05:12:48PM -0400, Phillip J. Eby wrote: > However, now that it's a known bug, it's merely an engineering problem to > fix it. :) Fantastic! I thought it was a feature, so I was unhappy, but if it's only a bug, well, I know too well that these things happen, and if nobody reports them, they don't get fixed. >> With standard Python, I can override the import mechanism using the >> PYTHONPATH. This way I can make sure that the modules I want get >> imported. And pretty much all the programs I can think of with a path >> environment variable behave like this. Now if I have a module installed >> with setuptools, I no longer have the possibility to do this, because >> setuptools inserts it self before the PYTHONPATH. > > I'm not following you here. Are you trying to specify the use of a > different version of setuptools? What do you mean by "setuptools inserts > itself before the PYTHONPATH"? Here is how things are happening (I simplified a bit the example, because I have a lot of things in my sys.path): varoquau at epsilon:~$ python -c "import sys; print sys.path" ['', '/usr/local/lib/python2.5/site-packages/configobj-4.4.0-py2.5.egg', '/usr/lib/python2.5/site-packages', '/usr/lib/python25.zip', '/usr/lib/python2.5', '/usr/lib/python2.5/plat-linux2', '/usr/lib/python2.5/lib-tk', '/usr/lib/python2.5/lib-dynload', '/usr/local/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages/wx-2.8-gtk2-unicode'] varoquau at epsilon:~$ export PYTHONPATH="/home/varoquau/python:" varoquau at epsilon:~$ python -c "import sys; print sys.path" ['', '/usr/local/lib/python2.5/site-packages/configobj-4.4.0-py2.5.egg', '/usr/lib/python2.5/site-packages', '/home/varoquau/python', '/home/varoquau', '/usr/lib/python25.zip', '/usr/lib/python2.5', '/usr/lib/python2.5/plat-linux2', '/usr/lib/python2.5/lib-tk', '/usr/lib/python2.5/lib-dynload', '/usr/local/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages/wx-2.8-gtk2-unicode'] Notice how what I added in my PYTHONPATH (the environment variable) is behind the egg. Well, what if I wanted to override configobj by my own version, in "/home/varoquau/python"? I am not able to do this without writing some Python code to monkey patch sys.path. The "$PYTHONPATH" should always come in front of everything. Are my explanations clear this time? Maybe this was not a feature, and another bug. In which case, please excuse the tone of my previous message. I should have reported the problem. I thought it was a know one, because it seems so natural to do the following in a Unix world. Cheers, Ga?l From pje at telecommunity.com Mon Apr 14 00:13:14 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 13 Apr 2008 18:13:14 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080413212420.GU24381@phare.normalesup.org> References: <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <20080413180536.B87DF3A40C2@sparrow.telecommunity.com> <20080413182609.GN24381@phare.normalesup.org> <20080413211243.1C1AF3A409D@sparrow.telecommunity.com> <20080413212420.GU24381@phare.normalesup.org> Message-ID: <20080413221437.6CDF13A409D@sparrow.telecommunity.com> At 11:24 PM 4/13/2008 +0200, Gael Varoquaux wrote: >Here is how things are happening (I simplified a bit the example, because >I have a lot of things in my sys.path): > >varoquau at epsilon:~$ python -c "import sys; print sys.path" >['', '/usr/local/lib/python2.5/site-packages/configobj-4.4.0-py2.5.egg', >'/usr/lib/python2.5/site-packages', '/usr/lib/python25.zip', >'/usr/lib/python2.5', '/usr/lib/python2.5/plat-linux2', >'/usr/lib/python2.5/lib-tk', '/usr/lib/python2.5/lib-dynload', >'/usr/local/lib/python2.5/site-packages', >'/usr/lib/python2.5/site-packages', >'/usr/lib/python2.5/site-packages/wx-2.8-gtk2-unicode'] >varoquau at epsilon:~$ export PYTHONPATH="/home/varoquau/python:" >varoquau at epsilon:~$ python -c "import sys; print sys.path" >['', '/usr/local/lib/python2.5/site-packages/configobj-4.4.0-py2.5.egg', >'/usr/lib/python2.5/site-packages', '/home/varoquau/python', >'/home/varoquau', '/usr/lib/python25.zip', '/usr/lib/python2.5', >'/usr/lib/python2.5/plat-linux2', '/usr/lib/python2.5/lib-tk', >'/usr/lib/python2.5/lib-dynload', >'/usr/local/lib/python2.5/site-packages', >'/usr/lib/python2.5/site-packages', >'/usr/lib/python2.5/site-packages/wx-2.8-gtk2-unicode'] > >Notice how what I added in my PYTHONPATH (the environment variable) is >behind the egg. Well, what if I wanted to override configobj by my own >version, in "/home/varoquau/python"? I am not able to do this without >writing some Python code to monkey patch sys.path. The "$PYTHONPATH" >should always come in front of everything. Are my explanations clear this >time? > >Maybe this was not a feature, and another bug. It's the same bug, actually. Ideally, the configobj.egg should be in front of the /usr/lib/python25.zip, but not any further forward than that. The last line of easy-install.pth is essentially too crude in its insertion method: it defaults to inserting at position zero, when it should default to a location that is either directly before the directory containing the .pth, or if that directory is the primary site-packages directory, then it should insert in a position that is after PYTHONPATH and the script directory, but before the stdlib. Now, if someone will contribute an alternative postamble for easy-install.pth that can figure all that out in a cross-platform/cross-python-version way -- and within the constraints of the limited code you can jam into a .pth file -- we can get it fixed. :) (The cross-python-version comment is a reference to the fact that the 'site' module was heavily rewritten in either 2.4 or 2.5; I forget which. So the tools available from inside a .pth file vary somewhat between Python versions.) From david at ar.media.kyoto-u.ac.jp Mon Apr 14 02:25:32 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 14 Apr 2008 09:25:32 +0900 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409075416.GA1761@phare.normalesup.org> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> <87bq4jolks.fsf@benfinney.id.au> <20080409075416.GA1761@phare.normalesup.org> Message-ID: <4802A47C.7020907@ar.media.kyoto-u.ac.jp> Gael Varoquaux wrote: > > 90 % (at least) of the world does not have such database. I, and probably > you, have such a very nice database. I works well, and we can choose to > forget the problems our users are facing. It does not solve them though. > > In addition, packaging is system-specific. I recently had to learn some > Debian packaging, because I wanted my Ubuntu and Debian users to be able > to use my projects seamlessly. What about RPMs for RHEL, Fedora, > Mandriva? ... and coronary packages? and MSIs? ... When do I find time to > do development if I have to learn all this packaging. There is no way around it: you have to learn about them. It is a PITA, but there is a limit of what a tool can do, specially for things like installers. I agree it would be fantastic to have an abstraction on all those packaging systems, but I don't think it is possible without a huge amount of work if at all. Deploying softwares is simply a very big problem that won't go away magically, even if eggs were perfect and nobody would complain about it. I strongly believe it is one of the reason why windows has been so popular: instead of targetting many combinations, you target windows, which does everything you ever need to do, and MS kind of guarantee that something which works today will still work in ten years. The only reliable way to handle dependencies if you don't have huge ressources is to bundle everything; IOW, not handling them. That's how most softwares work anyway on mac os X and windows: matlab for example is a huge thing with hundred of MB; most softwares do not depend on something else except the OS, which is a much more known thing on mac os X and windows than on linux. If you want to update a small part of it, life is tough, you upgrade everything. Of course, part of the thing is that it brings more revenue to mathworks, but I don't think it is the only reason. I also know some open source projects which do the same because they simply cannot track api changes (ardour, for example: if you build ardour from sources, you will get a private copy of the whole gtk stack). cheers, David From david at ar.media.kyoto-u.ac.jp Mon Apr 14 02:36:19 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 14 Apr 2008 09:36:19 +0900 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <48001EEA.3030209@canterbury.ac.nz> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> <79990c6b0804091546p5cf67f2ta800c675e7baa256@mail.gmail.com> <47FD667B.9040602@canterbury.ac.nz> <48001EEA.3030209@canterbury.ac.nz> Message-ID: <4802A703.9040602@ar.media.kyoto-u.ac.jp> Greg Ewing wrote: > > That looks interesting, but I'm not sure I'd quite > call it "something like". It looks like another case > of adding more complexity to fight existing complexity, > rather than removing the original complexity. > You won't be able to remove the initial complexity, because it is a feature. Honestly, one of the thing which annoys me the most when I have to use mac os X or windows is the lack of package management. Now, I don't think the situation on linux is ideal either. There are some technical issues, and some social issues; the worst thing to do is to find a technical solution to a social problem, so it is important to separate the two kinds, I think. On windows, most windows developers, as I understand it, do not have a strong need for package manager because they have almost everything they want with visual studio and MS dev tools. On a new linux machine, I may do apt-get install devtools. On windows, I run setup.exe for VS, plus the full Windows SDK. In a way, they do the same thing: providing everything a developer may need with as little hassle as possible for the developer (compilers, api, sdk, docs, etc... in a way which such as all the disparate things work together). > In other words, it seems to be just another package > manager, albeit a particulary nice-sounding one. There are two ways of looking at it, I think. One is to think that linux FHS (and unix in general) is totally broken. I personally really like how gobo linux tries to go around that: http://www.gobolinux.org/ It is like stow on steroids: I try to avoid installing anything from sources which is not handled through stow, and gobolinux just go one step further (quite a big step). The other one is to say disk space is cheap, just bundle everything (ala mac os X). 0install is a partial solution. There are also projects like klik or glick (done by a Red Had employer), which may be more similar to what you are after: Note that mac os X is a combination of the two in some ways. cheers, David From greg.ewing at canterbury.ac.nz Mon Apr 14 03:30:14 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 14 Apr 2008 13:30:14 +1200 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080413152925.GF24381@phare.normalesup.org> References: <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> Message-ID: <4802B3A6.5020606@canterbury.ac.nz> Gael Varoquaux wrote: > a second Python > needs to be installed on top of the system Python to add modules to it. Maybe the system should come with two pythons installed, one for use by the system and the other for users to add things to. Or at least be set up so that it appears that way -- they might share files under the hood. -- Greg From waterbug at pangalactic.us Mon Apr 14 03:39:15 2008 From: waterbug at pangalactic.us (Stephen Waterbury) Date: Sun, 13 Apr 2008 21:39:15 -0400 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <4802B3A6.5020606@canterbury.ac.nz> References: <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> Message-ID: <4802B5C3.5060405@pangalactic.us> Greg Ewing wrote: > Gael Varoquaux wrote: >> a second Python >> needs to be installed on top of the system Python to add modules to it. > > Maybe the system should come with two pythons installed, > one for use by the system and the other for users to add > things to. Or at least be set up so that it appears that > way -- they might share files under the hood. That was exactly my proposal -- see my message in this thread at 12:31 PM today. Steve From greg.ewing at canterbury.ac.nz Mon Apr 14 03:48:12 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 14 Apr 2008 13:48:12 +1200 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> <79990c6b0804091546p5cf67f2ta800c675e7baa256@mail.gmail.com> <47FD667B.9040602@canterbury.ac.nz> <48001EEA.3030209@canterbury.ac.nz> <48015BD9.5000502@canterbury.ac.nz> Message-ID: <4802B7DC.7040408@canterbury.ac.nz> John J Lee wrote: > If you have a network connection, about the only reason for not wanting > an app to be "installed" is that it has changed the behaviour of your > system somehow, just by being in the "installed" state. If you have a continuous high-speed network connection and aren't concerned about cost or bandwidth use or disk space taken up, it might make sense to have apps downloaded on demand, cached, etc. But not everyone works that way. I don't, much of the time. I prefer it when downloading an app and putting it on my system is an explicit step. I know you can use 0install that way, but I do it already on my MacOSX system without needing any special tool. :-) >> Yes, ROX is very MacOSX-like, but I don't think it has >> anything to do with 0install. > > 0install provides one way of implementing that kind of system. But it doesn't, if by "that kind of system" you mean one where an app or library is just an ordinary filesystem object. A 0install app appears to be very far from ordinary. > If you > want to share data, it's a better way than unshared directories of > files. There's nothing to stop a MacOSX user from putting an app in a publically-readable place and letting other people use it. I don't see what the big deal is there. >> If ROX apps included a checksum, and the system verified it >> before running the app, that would give you the same thing >> trust-wise, I think. > > That's an interesting idea, but how would the system find the app? The system doesn't have to find the app -- the user finds the app, using the same techniques he uses to find anything else in the filesystem he's interested in. -- Greg From greg.ewing at canterbury.ac.nz Mon Apr 14 04:16:13 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 14 Apr 2008 14:16:13 +1200 Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <4802A703.9040602@ar.media.kyoto-u.ac.jp> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> <79990c6b0804091546p5cf67f2ta800c675e7baa256@mail.gmail.com> <47FD667B.9040602@canterbury.ac.nz> <48001EEA.3030209@canterbury.ac.nz> <4802A703.9040602@ar.media.kyoto-u.ac.jp> Message-ID: <4802BE6D.6060706@canterbury.ac.nz> David Cournapeau wrote: > There are two ways of looking at it, I think. One is to think that linux > FHS (and unix in general) is totally broken. I don't think it's *totally* broken. I do think it goes overboard with splitting things up and scattering them around. I understand that there are reasons for some of that, but I don't see why e.g. includes, library files and other resources used by a package can't be kept together. > I personally really like > how gobo linux tries to go around that: > > http://www.gobolinux.org/ Hadn't heard of that before -- it sounds quite nice! Rather like MacOSX might look if it had dependency management. -- Greg From bignose+hates-spam at benfinney.id.au Mon Apr 14 04:54:09 2008 From: bignose+hates-spam at benfinney.id.au (Ben Finney) Date: Mon, 14 Apr 2008 12:54:09 +1000 Subject: [Distutils] desirability of multiple, divergent Python instances (was: shebang line modified by setuptools) References: <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> Message-ID: <87od8dtacu.fsf_-_@benfinney.id.au> Greg Ewing writes: > Gael Varoquaux wrote: > > a second Python needs to be installed on top of the system Python > > to add modules to it. > > Maybe the system should come with two pythons installed, one for use > by the system and the other for users to add things to. Or at least > be set up so that it appears that way -- they might share files > under the hood. I can't see how this is a good idea. I've seen it mentioned multiple times in this thread, but without justification. It's my position that the Python instance one uses for development should diverge as little as possible from the default system instance. Otherwise one is actively pursuing a recipe for dependency failures when one eventually deploys the result. Yes, there should be a way to deploy the code one is actively developing and testing to some location that will not affect the rest of the operation of the system. That is a far cry from asking that there should be multiple, divergent Python instances on the system. The further step of saying that the operating system should not use the same Python instance as a "user" on the system (and note that the line between those is far from clear), but instead should use some Python instance that behaves differently from every other use of Python on the system, seems like an even worse proposition, for the above reasons. Help me understand, please. -- \ "Most people don't realize that large pieces of coral, which | `\ have been painted brown and attached to the skull by common | _o__) wood screws, can make a child look like a deer." -- Jack Handey | Ben Finney From waterbug at pangalactic.us Mon Apr 14 06:59:41 2008 From: waterbug at pangalactic.us (Stephen Waterbury) Date: Mon, 14 Apr 2008 00:59:41 -0400 Subject: [Distutils] desirability of multiple, divergent Python instances In-Reply-To: <87od8dtacu.fsf_-_@benfinney.id.au> References: <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <87od8dtacu.fsf_-_@benfinney.id.au> Message-ID: <4802E4BD.4090408@pangalactic.us> Ben Finney wrote: > Greg Ewing writes: > >> Gael Varoquaux wrote: >>> a second Python needs to be installed on top of the system Python >>> to add modules to it. >> Maybe the system should come with two pythons installed, one for use >> by the system and the other for users to add things to. Or at least >> be set up so that it appears that way -- they might share files >> under the hood. > > I can't see how this is a good idea. I've seen it mentioned multiple > times in this thread, but without justification. > > It's my position that the Python instance one uses for development > should diverge as little as possible from the default system instance. > Otherwise one is actively pursuing a recipe for dependency failures > when one eventually deploys the result. > Yes, there should be a way to deploy the code one is actively > developing and testing to some location that will not affect the rest > of the operation of the system. That is a far cry from asking that > there should be multiple, divergent Python instances on the system. > > The further step of saying that the operating system should not use > the same Python instance as a "user" on the system (and note that the > line between those is far from clear), but instead should use some > Python instance that behaves differently from every other use of > Python on the system, seems like an even worse proposition, for the > above reasons. > > Help me understand, please. Have you read my proposal? It's in the message posted by me to this thread at 12:31 PM today. My main motivation in proposing it is to give the user complete control over the Python environment that is in their path -- which is not easy to do on Debian/Ubuntu at the moment -- by giving the Python utilities and packages that OS-related functions depend on a completely insular Python environment that the OS can manage however it sees fit. In my view, both as a developer and a user, but especially as a developer, it would make life easier. Steve From waterbug at pangalactic.us Mon Apr 14 07:16:10 2008 From: waterbug at pangalactic.us (Stephen Waterbury) Date: Mon, 14 Apr 2008 01:16:10 -0400 Subject: [Distutils] desirability of multiple, divergent Python instances In-Reply-To: <4802E4BD.4090408@pangalactic.us> References: <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <87od8dtacu.fsf_-_@benfinney.id.au> <4802E4BD.4090408@pangalactic.us> Message-ID: <4802E89A.6050700@pangalactic.us> Stephen Waterbury wrote: > Have you read my proposal? It's in the message posted by me to this > thread at 12:31 PM today. That's not a good way to reference it, of course, so here it is: Definitions: "system Python" -- the Python (and its site-pkgs, etc.) that any Python scripts used by the OS depend on, which would of course by solely determined by the OS. This would be distinct, in my proposal, from any packages named "python" or "python.x" in the package namespace of the system package manager (e.g. apt). What I am proposing: 1) that the OS comes with its own "system Python", which is installed not as the "python" package, but as some OS-required package (maybe call it "system-python" or something) and it goes into /usr/system/bin/python or whatever -- it doesn't matter what the path is as long as it's not /usr/bin or anything on the default path. And system utilities that are python scripts should have their own system-specific, hard-coded shebang line. 2) that separately from the "system Python", the available packages managed by the system's package manager include one or more "python.x" packages which are python interpreters that the user or sysadmin can optionally install, and which go into /usr, and which would share *nothing* with the "system Python". And the system package manager -- e.g., apt on Debian/Ubuntu systems) would have all its usual nicely-packaged python apps (python-this, python-that, ...) that would also install into /usr and use the nicely-packaged python (not to be confused with the "system Python" of 1). Fire away! :) Steve From eikeon at eikeon.com Mon Apr 14 07:37:39 2008 From: eikeon at eikeon.com (Daniel Krech) Date: Mon, 14 Apr 2008 01:37:39 -0400 Subject: [Distutils] desirability of multiple, divergent Python instances In-Reply-To: <4802E4BD.4090408@pangalactic.us> References: <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <87od8dtacu.fsf_-_@benfinney.id.au> <4802E4BD.4090408@pangalactic.us> Message-ID: On Apr 14, 2008, at 12:59 AM, Stephen Waterbury wrote: > Ben Finney wrote: >> Greg Ewing writes: >> >>> Gael Varoquaux wrote: >>>> a second Python needs to be installed on top of the system Python >>>> to add modules to it. >>> Maybe the system should come with two pythons installed, one for use >>> by the system and the other for users to add things to. Or at least >>> be set up so that it appears that way -- they might share files >>> under the hood. >> >> I can't see how this is a good idea. I've seen it mentioned multiple >> times in this thread, but without justification. >> >> It's my position that the Python instance one uses for development >> should diverge as little as possible from the default system >> instance. >> Otherwise one is actively pursuing a recipe for dependency failures >> when one eventually deploys the result. > >> Yes, there should be a way to deploy the code one is actively >> developing and testing to some location that will not affect the rest >> of the operation of the system. That is a far cry from asking that >> there should be multiple, divergent Python instances on the system. >> >> The further step of saying that the operating system should not use >> the same Python instance as a "user" on the system (and note that the >> line between those is far from clear), but instead should use some >> Python instance that behaves differently from every other use of >> Python on the system, seems like an even worse proposition, for the >> above reasons. >> >> Help me understand, please. > > Have you read my proposal? It's in the message posted by me to this > thread at 12:31 PM today. > > My main motivation in proposing it is to give the user complete > control > over the Python environment that is in their path -- which is not > easy to > do on Debian/Ubuntu at the moment -- by giving the Python utilities > and > packages that OS-related functions depend on a completely insular > Python > environment that the OS can manage however it sees fit. In my view, > both > as a developer and a user, but especially as a developer, it would > make > life easier. This seems related to the use case I was going after in 'autoinstall'. Which is to move the control of the Python environment to the application. This way each application can have its own insular Python environment. A first baby step in that direction: http://pypi.python.org/pypi/autoinstall/0.1a2 Currently it only supports binding dotted packages names to URLs (relative or absolute) that point to zip importable versions of the package. But I've been using it some and it's been making my life easier indeed. Curious what people think of a per application approach? The use case seems to be mentioned a fair bit in the various threads. From gael.varoquaux at normalesup.org Mon Apr 14 12:10:03 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 14 Apr 2008 12:10:03 +0200 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <4802B3A6.5020606@canterbury.ac.nz> References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> Message-ID: <20080414101003.GB27572@phare.normalesup.org> On Mon, Apr 14, 2008 at 01:30:14PM +1200, Greg Ewing wrote: > Gael Varoquaux wrote: > > a second Python > > needs to be installed on top of the system Python to add modules to it. > Maybe the system should come with two pythons installed, > one for use by the system and the other for users to add > things to. Or at least be set up so that it appears that > way -- they might share files under the hood. I am quite wary about these proposals, as well as the one environment per application ones. What you propose resembles very much to what MacOSX does, and MacOSX seems just so broken for Python. I don't use it, but I see on the different scientific-Python-related mailing lists how users have difficulties with MacOSX, and I have heard many people complain about this "feature". As a per-application environment, I think it is bad, because it limits reuse. As I see things, applications should be able to have access to all the Python modules installed, to be able to use them, if they need. I get definitely see applications having more feature if some modules are installed (eg. Sphinx, which does syntax highlighting if pygments is installed). I think this discussion is really going on because Python does not have good library-versioning support. What it needs is to get this, and not to get complete isolation, apart for some system-critical tasks, and only for these. AFAIK this is one of the problems setuptools is trying to solve. Not everybody is happy with the technical solution setuptools has come up, this only means it has to be improved. I think this library-versioning issue is very important, and needs to get support from the core developers, but I know this is not everybody's opinion :->. My 2 cents, Ga?l From david at ar.media.kyoto-u.ac.jp Mon Apr 14 12:51:54 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 14 Apr 2008 19:51:54 +0900 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080414101003.GB27572@phare.normalesup.org> References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <20080414101003.GB27572@phare.normalesup.org> Message-ID: <4803374A.6020106@ar.media.kyoto-u.ac.jp> Gael Varoquaux wrote: > > > I think this discussion is really going on because Python does not have > good library-versioning support. I personally think that the very idea to have several side by side versions of the same package is doomed to failure: what is needed is a stable API for the used packages. Library versioning is really only (a small) part of the problem, and supporting it won't magically solve the issue of complex deployments. cheers, David From dpeterson at enthought.com Mon Apr 14 14:56:43 2008 From: dpeterson at enthought.com (Dave Peterson) Date: Mon, 14 Apr 2008 07:56:43 -0500 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080414101003.GB27572@phare.normalesup.org> References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <20080414101003.GB27572@phare.normalesup.org> Message-ID: <4803548B.5030108@enthought.com> Gael Varoquaux wrote: > On Mon, Apr 14, 2008 at 01:30:14PM +1200, Greg Ewing wrote: > >> Gael Varoquaux wrote: >> >>> a second Python >>> needs to be installed on top of the system Python to add modules to it. >>> > > >> Maybe the system should come with two pythons installed, >> one for use by the system and the other for users to add >> things to. Or at least be set up so that it appears that >> way -- they might share files under the hood. >> > > I am quite wary about these proposals, as well as the one environment per > application ones. > In my experience, it is a fairly standard practice across all platforms to install independent application environments for any large, complex applications that come with a vendor support agreement. Admittedly, that isn't alot of Python applications, especially of the kind you're referring to where users are expected to add, remove, and intermix modules. However, that doesn't mean there aren't any Python apps that do fit the bill -- in fact, Enthought does alot of that. We want our apps to work even if the sysadmin upgrades the system libraries, Python, or what have you, or if the user installs multiple applications that have competing library requirements. > What you propose resembles very much to what MacOSX does, and MacOSX > seems just so broken for Python. I don't use it, but I see on the > different scientific-Python-related mailing lists how users have > difficulties with MacOSX, and I have heard many people complain about > this "feature". > > As a per-application environment, I think it is bad, because it limits > reuse. As I see things, applications should be able to have access to all > the Python modules installed, to be able to use them, if they need. I get > definitely see applications having more feature if some modules are > installed (eg. Sphinx, which does syntax highlighting if pygments is > installed). > > I think this discussion is really going on because Python does not have > good library-versioning support. What it needs is to get this, and not to > get complete isolation, apart for some system-critical tasks, and only > for these. AFAIK this is one of the problems setuptools is trying to > solve. Not everybody is happy with the technical solution setuptools has > come up, this only means it has to be improved. > IMO, this sort of multi-version thing only works if *everything* in the stack participates fully. I think the reason more things don't do this is because of the maintenance load it puts on developers / release managers to bump and synchronize versions apporpriately. -- Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080414/629feb3b/attachment.htm From sklein at cpcug.org Mon Apr 14 16:58:42 2008 From: sklein at cpcug.org (Stanley A. Klein) Date: Mon, 14 Apr 2008 10:58:42 -0400 (EDT) Subject: [Distutils] Distutils-SIG Digest, Vol 60, Issue 29 In-Reply-To: References: Message-ID: <52954.207.188.248.157.1208185122.squirrel@www.cpcug.org> On Mon, April 14, 2008 1:37 am, distutils-sig-request at python.org wrote: > Message: 4 > Date: Mon, 14 Apr 2008 14:16:13 +1200 > From: Greg Ewing > Subject: Re: [Distutils] how to easily consume just the parts of eggs > that are good for you > To: distutils-sig at python.org > > David Cournapeau wrote: >> There are two ways of looking at it, I think. One is to think that linux >> FHS (and unix in general) is totally broken. > > I don't think it's *totally* broken. I do think it goes > overboard with splitting things up and scattering them > around. I understand that there are reasons for some of > that, but I don't see why e.g. includes, library files > and other resources used by a package can't be kept > together. > Linux and the FHS aren't broken at all. They are just designed around a different concept than the single-user, proprietary software concept that Windows and Mac OS X are based on. They are based on multi-user systems with reusable software and multi-use software tools. They provide a natural environment for community developed and maintained Free/Open-Source Software (FOSS). You can think of Linux splitting things up and scattering them around. I think of Windows putting things that don't belong together in the same place just because they happen to be supplied by the same provider. Every Windows application is monolithic, because a proprietary provider who depends on something other than the OS buys it, pays the royalties for its use, hides it from the user by compiling it into the application binary, and includes it in the software package supplied. This doesn't usually happen in Unix or Linux. Providers depend on each other. That's the reason for dependencies. Trying to use FOSS on Windows creates issues that have to be addressed. BTW, if eggs are analogous to jars (as Eby states), eggs are absolutely not a complete packaging system and were never intended to be so. If you look at a java-related rpm package, you are likely to see a number of jar files along with a lot of other files. Stan Klein From meine at informatik.uni-hamburg.de Mon Apr 14 18:32:39 2008 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Mon, 14 Apr 2008 18:32:39 +0200 Subject: [Distutils] virtual env with exec prefix In-Reply-To: <200804101335.47020.meine@informatik.uni-hamburg.de> References: <200804101335.47020.meine@informatik.uni-hamburg.de> Message-ID: <200804141832.40294.meine@informatik.uni-hamburg.de> Hi Ian, recently I reported this problem to you: > [...] here at our university, python (as > well as hundreds of other software packages) is installed in a shared NFS > tree with a clear separation of --prefix and --exec-prefix, i.e. all > platform-specific stuff goes into the according subdirectories. [...] > [2] meine at kogspc33:~ -> virtualenv enthought-inst > New python executable in enthought-inst/bin/python > Installing setuptools.... > Complete output from command enthought-inst/bin/python -c "#!python > \"\"\"Bootstrap setuptoo... > > > > > " /software/python-2.4.4/SuSE-10...4.egg: > error: invalid Python installation: unable to > open /software/python-2.4.4/lib/python2.4/config/Makefile (No such file or > directory) > > [The message] is right, because it should be > /software/python-2.4.4/SuSE-10.2/lib/python2.4/config/Makefile > which exists (in the SuSE-10.2 directory). I have now debugged this a little bit (using virtualenv from SVN), and it seems to be a problem with the virtualenv'd python and setuptools. I have written the EZ_SETUP_PY to disk, and when I mimick virtualenv's command for installing setuptools, I get the error: [2] meine at kogspc33:~/Programming/enthought -> inst/bin/python ../ez_setup.py -v /home/meine/Programming/virtualenv-svn/support-files/setuptools-0.6c8-py2.4.egg error: invalid Python installation: unable to open /software/python-2.4.4/lib/python2.4/config/Makefile (No such file or directory) (NB: inst is the virtual env) OTOH, when I run this with the python binary from its original location, it works fine. Since the error comes from deep within setuptools (AFAICS), should this be discussed on distutils-sig? Ciao, / / /--/ / / ANS From floris.bruynooghe at gmail.com Mon Apr 14 18:53:11 2008 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Mon, 14 Apr 2008 17:53:11 +0100 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080414101003.GB27572@phare.normalesup.org> References: <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <20080414101003.GB27572@phare.normalesup.org> Message-ID: <20080414165311.GA18918@laurie.devork> On Mon, Apr 14, 2008 at 12:10:03PM +0200, Gael Varoquaux wrote: > I think this discussion is really going on because Python does not have > good library-versioning support. What it needs is to get this +1 From meine at informatik.uni-hamburg.de Mon Apr 14 19:34:50 2008 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Mon, 14 Apr 2008 19:34:50 +0200 Subject: [Distutils] virtual env with exec prefix In-Reply-To: <200804141832.40294.meine@informatik.uni-hamburg.de> References: <200804101335.47020.meine@informatik.uni-hamburg.de> <200804141832.40294.meine@informatik.uni-hamburg.de> Message-ID: <200804141934.51374.meine@informatik.uni-hamburg.de> Am Montag, 14. April 2008 18:32:39 schrieb Hans Meine: > > error: invalid Python installation: unable to > > open /software/python-2.4.4/lib/python2.4/config/Makefile (No such file > > or directory) I have found and fixed this bug; it's because virtualenv overwrites distutils.sysconfig.get_python_lib and does not respect the 'plat_specific' parameter properly. I have modified virtualenv by adding sys.real_exec_prefix along sys.real_prefix and letting orig-prefix.txt be a file with two lines. Now, I can use virtualenv here, too. :-) @distutils-sig: Sorry for the noise, it turned out to be virtualenv's fault. -- Ciao, / / /--/ / / ANS -------------- next part -------------- A non-text attachment was scrubbed... Name: virtualenv-respect-exec-prefix.diff Type: text/x-diff Size: 16685 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20080414/16fb135e/attachment-0001.diff From ianb at colorstudy.com Mon Apr 14 20:37:10 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 14 Apr 2008 13:37:10 -0500 Subject: [Distutils] virtual env with exec prefix In-Reply-To: <200804141832.40294.meine@informatik.uni-hamburg.de> References: <200804101335.47020.meine@informatik.uni-hamburg.de> <200804141832.40294.meine@informatik.uni-hamburg.de> Message-ID: <4803A456.2010103@colorstudy.com> Hans Meine wrote: > Hi Ian, > > recently I reported this problem to you: >> [...] here at our university, python (as >> well as hundreds of other software packages) is installed in a shared NFS >> tree with a clear separation of --prefix and --exec-prefix, i.e. all >> platform-specific stuff goes into the according subdirectories. [...] > >> [2] meine at kogspc33:~ -> virtualenv enthought-inst >> New python executable in enthought-inst/bin/python >> Installing setuptools.... >> Complete output from command enthought-inst/bin/python -c "#!python >> \"\"\"Bootstrap setuptoo... >> >> >> >> >> " /software/python-2.4.4/SuSE-10...4.egg: >> error: invalid Python installation: unable to >> open /software/python-2.4.4/lib/python2.4/config/Makefile (No such file or >> directory) >> >> [The message] is right, because it should be >> /software/python-2.4.4/SuSE-10.2/lib/python2.4/config/Makefile >> which exists (in the SuSE-10.2 directory). > > I have now debugged this a little bit (using virtualenv from SVN), and it > seems to be a problem with the virtualenv'd python and setuptools. I have > written the EZ_SETUP_PY to disk, and when I mimick virtualenv's command for > installing setuptools, I get the error: > > [2] meine at kogspc33:~/Programming/enthought -> > inst/bin/python ../ez_setup.py -v /home/meine/Programming/virtualenv-svn/support-files/setuptools-0.6c8-py2.4.egg > error: invalid Python installation: unable to > open /software/python-2.4.4/lib/python2.4/config/Makefile (No such file or > directory) > > (NB: inst is the virtual env) OTOH, when I run this with the python binary > from its original location, it works fine. Since the error comes from deep > within setuptools (AFAICS), should this be discussed on distutils-sig? Hmm... well, does this error only happen when you create a virtualenv from within a virtualenv? Otherwise, I think there's several things in distutils.sysconfig that need to be adjusted. That module isn't really documented, however, and I suspect there's just a bunch of things that will have to be added to over time, not a good general fix. Or... maybe there's some clever way to fix all these, but I'm not sure. -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From jan.matejek at novell.com Mon Apr 14 20:59:32 2008 From: jan.matejek at novell.com (Jan Matejek) Date: Mon, 14 Apr 2008 20:59:32 +0200 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: References: <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <20080413120405.GB24381@phare.normalesup.org> Message-ID: <4803A994.9070202@novell.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Barry Warsaw napsal(a): | On Apr 13, 2008, at 8:04 AM, Gael Varoquaux wrote: |> On Sun, Apr 13, 2008 at 07:59:21AM -0400, Barry Warsaw wrote: |>> On Apr 12, 2008, at 9:41 PM, Stephen Waterbury wrote: |>>> I used to always set up my own Python[s] in /usr/local |>>> and put that first in my PATH, but I have gotten lazy lately, and |>>> sometimes it will bite me. ;) |>> On Debian and derivatives (e.g. Ubuntu) you might have even more fun. |>> They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of |>> the system python*! This means that you can break your system Python |>> by installing a version of Python from source and then distutil'ing |>> things into there. Astoundingly, this is promoted as a feature. |> I want it like that. You are confusing /opt and /usr/local. | |> This is the way I expect things to work. I do not want to install my |> own |> packages in "/usr/lib/pythonX.Y/site-packages". This is for apt to |> deal |> with, not for me. | | Actually, the solution that I believe is a good compromise is to find | some other path in /usr/local to augment the system Python's sys.path | with. All I'm saying is that Python itself uses /usr/local/lib/ | pythonX.Y/site-packages as the default from-source path, so if Debian | wants a /usr/local path, it can pick some other subdirectory path and | still meet its goals. BTW, I believe Debian is unique here. I talked | to the Fedora guys and I've talked to a few Gentoo guys and both | seemed surprised by Debian's policy here. Actually, openSUSE does this too, and i've even gone as far as making distutils install into /usr/local by default. Several reasons to do this: a) FHS says, "user installed apps go into /usr/local, period" b) the user should have a clean site-packages directory for his needs c) we don't want the user to mess with package-managed modules in /usr and finally, the reason i'm not very happy to "pick some other subdirectory path": d) it is simple to tell distutils that default prefix is /usr/local. It is complicated and ugly to tell it that by default it should go to /usr/local/some-directory-of-my-own, and still be able to revert to system-ish installation by supplying --prefix=/usr (following the same semantics that autoconf/automake is using). Of course we could introduce SUSE-specific --install-like-system-does switch, but if the user wants, for whatever reason, to install to a system-ish location, he would need to find out about this. And furthermore, it would make cross-distro packaging even harder (openSUSE Buildservice anyone?) Plus, /usr/local/lib/pythonX.Y/site-packages still looks like a reasonably standard path, whereas /usr/local/weird-place/nobody-ever-heard-of is just your garden variety nonsensical and unsupported distro-specific customization. I never realized that this would be a bad thing to do, and i still dot think so - /usr/local is at the end of sys.path anyway, so user can override system-installed packages and break his python only if he specifically tries to do that. I will be happy to abandon this practice as soon as there is some kind of official policy that solves all the mentioned points. (Of course, things would be much simpler if python, in perl's fashion, by default supported site-packages and vendor-packages directories specifically for this purpose - still violating FHS, but in a way nobody really cares about anymore. Nudge nudge, wink wink ;) ) regards jan matejek | | -Barry | _______________________________________________ Distutils-SIG maillist - Distutils-SIG at python.org http://mail.python.org/mailman/listinfo/distutils-sig -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4-svn0 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFIA6mUjBrWA+AvBr8RAsK6AJ9lo7foixt2/LmV2RDb3TCM6e46igCeKB4f R2mWEX8hQ4qG2UaeescJS24= =u88R -----END PGP SIGNATURE----- From floris.bruynooghe at gmail.com Mon Apr 14 23:46:41 2008 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Mon, 14 Apr 2008 22:46:41 +0100 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <4803A994.9070202@novell.com> References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <20080413120405.GB24381@phare.normalesup.org> <4803A994.9070202@novell.com> Message-ID: <20080414214641.GA6087@laurie.devork> On Mon, Apr 14, 2008 at 08:59:32PM +0200, Jan Matejek wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Barry Warsaw napsal(a): > | On Apr 13, 2008, at 8:04 AM, Gael Varoquaux wrote: > |> On Sun, Apr 13, 2008 at 07:59:21AM -0400, Barry Warsaw wrote: > |>> On Apr 12, 2008, at 9:41 PM, Stephen Waterbury wrote: > |>>> I used to always set up my own Python[s] in /usr/local > |>>> and put that first in my PATH, but I have gotten lazy lately, and > |>>> sometimes it will bite me. ;) > |>> On Debian and derivatives (e.g. Ubuntu) you might have even more fun. > |>> They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of > |>> the system python*! This means that you can break your system Python > |>> by installing a version of Python from source and then distutil'ing > |>> things into there. Astoundingly, this is promoted as a feature. > |> I want it like that. You are confusing /opt and /usr/local. > | > |> This is the way I expect things to work. I do not want to install my > |> own > |> packages in "/usr/lib/pythonX.Y/site-packages". This is for apt to > |> deal > |> with, not for me. > | > | Actually, the solution that I believe is a good compromise is to find > | some other path in /usr/local to augment the system Python's sys.path > | with. All I'm saying is that Python itself uses /usr/local/lib/ > | pythonX.Y/site-packages as the default from-source path, so if Debian > | wants a /usr/local path, it can pick some other subdirectory path and > | still meet its goals. BTW, I believe Debian is unique here. I talked > | to the Fedora guys and I've talked to a few Gentoo guys and both > | seemed surprised by Debian's policy here. > > Actually, openSUSE does this too, and i've even gone as far as making > distutils install into /usr/local by default. Several reasons to do this: > a) FHS says, "user installed apps go into /usr/local, period" > b) the user should have a clean site-packages directory for his needs > c) we don't want the user to mess with package-managed modules in /usr > > and finally, the reason i'm not very happy to "pick some other > subdirectory path": > d) it is simple to tell distutils that default prefix is /usr/local. It > is complicated and ugly to tell it that by default it should go to > /usr/local/some-directory-of-my-own, and still be able to revert to > system-ish installation by supplying --prefix=/usr (following the same > semantics that autoconf/automake is using). I entirely agree with this (from Debian's POV). This is the way many things in the system works, you install shared libraries to /usr/local/lib etc. I think Python needs 3 clear site-packges directories: 1) system 2) local-admin 3) user. Personally I'd like to see PEP 370, which is trying to introduce 3) above, move in that direction. No idea how I go about that though. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From jjl at pobox.com Tue Apr 15 00:10:36 2008 From: jjl at pobox.com (John J Lee) Date: Mon, 14 Apr 2008 23:10:36 +0100 (BST) Subject: [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <4802B7DC.7040408@canterbury.ac.nz> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> <79990c6b0804091546p5cf67f2ta800c675e7baa256@mail.gmail.com> <47FD667B.9040602@canterbury.ac.nz> <48001EEA.3030209@canterbury.ac.nz> <48015BD9.5000502@canterbury.ac.nz> <4802B7DC.7040408@canterbury.ac.nz> Message-ID: On Mon, 14 Apr 2008, Greg Ewing wrote: > John J Lee wrote: >> If you have a network connection, about the only reason for not wanting an >> app to be "installed" is that it has changed the behaviour of your system >> somehow, just by being in the "installed" state. > > If you have a continuous high-speed network connection and > aren't concerned about cost or bandwidth use or disk space > taken up, it might make sense to have apps downloaded on demand, http://0install.net/faq.html#id2324452 Practically, I suspect the sharing and caching will result in lower network bandwidth usage. I guess practically, that's a matter to be answered mostly by measurement in common usage patterns, rather than by argument. > cached, etc. But not everyone works that way. I don't, much of > the time. I prefer it when downloading an app and putting it > on my system is an explicit step. You'll be the first against the wall when the revolution comes ;-) >>> Yes, ROX is very MacOSX-like, but I don't think it has >>> anything to do with 0install. >> >> 0install provides one way of implementing that kind of system. > > But it doesn't, if by "that kind of system" you mean one where > an app or library is just an ordinary filesystem object. A > 0install app appears to be very far from ordinary. Of course, I understand exactly what you mean. But since the answer to those kinds of questions depends on our different ideas of how "an app" or "installed" can most usefully be defined, I guess debating the words here is less profitable than the concepts and their consequences. I genuinely do suspect that the 0install model is simpler to understand than the "unshared directories of files" model (I won't really be confident unless and until I actually use the thing a lot, of course). [...] >>> If ROX apps included a checksum, and the system verified it >>> before running the app, that would give you the same thing >>> trust-wise, I think. >> >> That's an interesting idea, but how would the system find the app? > > The system doesn't have to find the app -- the user finds the > app, using the same techniques he uses to find anything else in > the filesystem he's interested in. In somebody else's user account, right? And the dependencies? And what app is that, anyway? http://0install.net/survey.html """If you don't know the hash, you can't trust it! Making it easy to browse the cache "Hey look - there's the Gimp! Let's run it!" is therefore an anti-goal.""" Of course, you could specify both the app (== URL, or hash, or pet name for it, or something like that) *and* where its data is on the disk, but that's a more complicated and less useful interface. John From cliff at develix.com Tue Apr 15 01:23:31 2008 From: cliff at develix.com (Cliff Wells) Date: Mon, 14 Apr 2008 16:23:31 -0700 Subject: [Distutils] shebang line modified by setuptools Message-ID: <1208215411.25403.25.camel@portableevil.develix.com> Sorry for breaking up the thread. I wasn't subscribed to the list (now I am) and apparently I stopped being CC'd at some point, so I'll have to sum up several things and address them here. 1) I agree that "system" scripts should use the "system" python (whatever that is defined to mean - for now it means /usr/bin/python) and that path should be hardcoded into the shebang line. 2) I disagree with Ben's position that: """ It's my position that the Python instance one uses for development should diverge as little as possible from the default system instance. Otherwise one is actively pursuing a recipe for dependency failures when one eventually deploys the result. """ The reason I disagree is that assumes the system is single user and/or single purpose. In my case (and many others) I'm concerned about a traditional shared hosting environment (not VPS). This means that there may be several versions of Python (and many, many ~/python2.x/site-packages directories). Diverging from the default system instance *in every single case* is the only way to make this workable. Not only do different accounts require a particular version of Python, but they require particular versions of installed packages (one might rely on Pylons 0.9.5 another on Pylons 0.9.7). 3) Since there is so much opposition to using /usr/bin/env in the shebang line, an acceptable alternative would be to hardcode the version-specific python instead (e.g. /usr/bin/python2.4). This seems to solve my particular issue while respecting Phillip's concerns about accidentally running an incorrect interpreter, and actually does a better job since Phillip's method (using /usr/bin/python) falsely assumes that the system Python will never change (untrue on Gentoo, Foresight, SourceMage and any other distro that supports "rolling upgrades"). 4) As far as the discussion about having three separate install areas, I think this is not necessary since it's already possible to have unlimited install areas (one per user) using features already built into setuptools. /usr/bin/python/ and /usr/local/bin/pythonX.X/ should suffice. If setuptools used versioning info along with a hardcoded path in the shebang line, this would become even more solid. I should also mention that some of this discussion is being made moot by virtualenv. I'll be using that for future stuff, but I have legacy installs to worry about as well. Regards, Cliff From greg.ewing at canterbury.ac.nz Tue Apr 15 02:43:22 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 15 Apr 2008 12:43:22 +1200 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <20080414101003.GB27572@phare.normalesup.org> References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <20080414101003.GB27572@phare.normalesup.org> Message-ID: <4803FA2A.9060101@canterbury.ac.nz> Gael Varoquaux wrote: > On Mon, Apr 14, 2008 at 01:30:14PM +1200, Greg Ewing wrote: > > > Maybe the system should come with two pythons installed, > > What you propose resembles very much to what MacOSX does... I see on the > different scientific-Python-related mailing lists how users have > difficulties with MacOSX It differs from MacOSX, because MacOSX only comes with one Python installed. A lot of the problems go away if you install another one, but things aren't set up that way by default. > I think this discussion is really going on because Python does not have > good library-versioning support. That's probably true. I was wondering recently whether there should be some way to specify which version of a module you want when you import it, e.g. import fooble[3] which means you want a fooble with API version 3 or something compatible with it. -- Greg From greg.ewing at canterbury.ac.nz Tue Apr 15 03:01:09 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 15 Apr 2008 13:01:09 +1200 Subject: [Distutils] Simple idea to resolve versioning problems In-Reply-To: <4803374A.6020106@ar.media.kyoto-u.ac.jp> References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <20080414101003.GB27572@phare.normalesup.org> <4803374A.6020106@ar.media.kyoto-u.ac.jp> Message-ID: <4803FE55.2000805@canterbury.ac.nz> David Cournapeau wrote: > what is needed is a stable API for the used packages. That's a nice ideal to aim for, but it's only achievable for old and mature packages. One could change the package name every time the API changes, but then *any* change to the API would make the package unusable by apps expecting an earlier version, even if the new API is backwards compatible. Having said that, I just realised you can address that by putting in symlinks for the previous versions. So, I hereby propose the following convention for naming of Python packages: packagename_ where is a number or other identifier that is changed whenever the API changes in any way. If we all follow this convention, we will be able to install and use multiple versions of a package side-by-side just as easily as we can Python versions, with no need for setuptools or any other fancy technology -- just plain standard Python the way it currently is! -- Greg From david at ar.media.kyoto-u.ac.jp Tue Apr 15 04:04:28 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 15 Apr 2008 11:04:28 +0900 Subject: [Distutils] Simple idea to resolve versioning problems In-Reply-To: <4803FE55.2000805@canterbury.ac.nz> References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <20080414101003.GB27572@phare.normalesup.org> <4803374A.6020106@ar.media.kyoto-u.ac.jp> <4803FE55.2000805@canterbury.ac.nz> Message-ID: <48040D2C.4010506@ar.media.kyoto-u.ac.jp> Greg Ewing wrote: > David Cournapeau wrote: >> what is needed is a stable API for the used packages. > > That's a nice ideal to aim for, but it's only achievable > for old and mature packages. > Yes, many packages can't have a stable API, at least for some time. But then, should you rely on them for deployment ? I am really not convinced that you can have both stable systems and packages which keep changing whenever they want. I don't believe that library versioning will give you that, that's just a really minor part of it. When you think about it for a second, what is library versioning ? A convention to track API changes. Library versioning without API stability just does not make sense to me. What would happen if with apt-get you could install gtk 2.0, 2.2, etc... up to gtk-2.12, and gtk broke some API between minor versions ? Nothing would work, because you would quickly have one package B which work with gtk 2.8 but not above, package C which needs gtk 2.10, and package D which needs B and C. That's why you have gtk-2.n, and debian project put huge effort to be sure that with some n, many projects will work on it. If gtk is not stable, this cannot happen. I am not aware of a single big open source project which have many dependencies without strong API versioning policies (which means not breaking between minor changes). For small projects, it is no problem. For big ones, you need either to rely on strongly versioned api, or to cut dependencies (by copying a private copy: see mozilla/firefox which embed their own gecko, ardour which embeds its own gtk, sndfile which embeds its own copy of ogg, etc...) > > If we all follow this convention, we will be able to > install and use multiple versions of a package side-by-side > just as easily as we can Python versions, with no need > for setuptools or any other fancy technology -- just > plain standard Python the way it currently is! I don't see how this could work. How do you do if you have a package D which depends on both C and B, and C needs one version of one package A, and B another one ? If you cannot use a common A, this cannot work (how one python instance would use different versions of the same package ?) cheers, David From meine at informatik.uni-hamburg.de Mon Apr 14 17:57:08 2008 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Mon, 14 Apr 2008 17:57:08 +0200 Subject: [Distutils] [Enthought-dev] Installation woes - again.. Message-ID: <200804141757.08586.meine@informatik.uni-hamburg.de> Hi, Dave P. has recently cross-posted here a mail from the enthought-dev thread I started, in which I describe serious problems installing the ETS with easy_install: https://mail.enthought.com/pipermail/enthought-dev/2008-April/013913.html I have tried setuptools 0.6_rc8 as distributed with Gentoo, 0.6_rc7 as installed by virtualenv, 0.6_rc8 as installed with ez_setup.py, and finally setuptools from SVN. I am getting really strange errors which others confirmed not to make much sense; please see the above description or the follow-up posts. (A detailed log is here: http://kogs-www.informatik.uni-hamburg.de/~meine/clean-ets-install.log ) I have worked around this specific problem for me, but I herewith offer to provide additional information in case anyone wants to get this fixed. Have a nice day, Hans From cliff at develix.com Tue Apr 15 06:57:34 2008 From: cliff at develix.com (Cliff Wells) Date: Mon, 14 Apr 2008 21:57:34 -0700 Subject: [Distutils] Simple idea to resolve versioning problems In-Reply-To: <4803FE55.2000805@canterbury.ac.nz> References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <20080414101003.GB27572@phare.normalesup.org> <4803374A.6020106@ar.media.kyoto-u.ac.jp> <4803FE55.2000805@canterbury.ac.nz> Message-ID: <1208235454.332.20.camel@portableevil.develix.com> On Tue, 2008-04-15 at 13:01 +1200, Greg Ewing wrote: > David Cournapeau wrote: > > what is needed is a stable API for the used packages. > > That's a nice ideal to aim for, but it's only achievable > for old and mature packages. I don't think so. It requires vigil on the part of the maintainer, but even young packages can have a sane versioning policy. > One could change the package name every time the API > changes, but then *any* change to the API would make the > package unusable by apps expecting an earlier version, > even if the new API is backwards compatible. > > Having said that, I just realised you can address that > by putting in symlinks for the previous versions. I don't think that's supported on Windows. > So, I hereby propose the following convention for naming > of Python packages: > > packagename_ I don't know that this is an actual replacement for setuptools (rather a way to simplify setuptools), but I like the idea in general. However what I'd rather see is: site-packages/package/major_version It would keep the site-packages directory from becoming a giant mess and be more portable. > where is a number or other identifier that is > changed whenever the API changes in any way. I think the convention is major.minor where minor releases are backwards-compatible and major releases aren't expected to be (but might be). Sometimes this is confusing as projects will do a major version release due to say, major code-refactoring that doesn't necessarily affect the API, so the API "version" isn't necessarily reflected in the package version. On the other hand, not using the higher version certainly won't break existing code either, so it's rather moot whether you *could* use the higher versioned software if your concern is not breaking an existing system. Regards, Cliff From david at ar.media.kyoto-u.ac.jp Tue Apr 15 07:06:11 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 15 Apr 2008 14:06:11 +0900 Subject: [Distutils] Simple idea to resolve versioning problems In-Reply-To: <1208235454.332.20.camel@portableevil.develix.com> References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <20080414101003.GB27572@phare.normalesup.org> <4803374A.6020106@ar.media.kyoto-u.ac.jp> <4803FE55.2000805@canterbury.ac.nz> <1208235454.332.20.camel@portableevil.develix.com> Message-ID: <480437C3.5000403@ar.media.kyoto-u.ac.jp> Cliff Wells wrote: > > I think the convention is major.minor where minor releases are > backwards-compatible and major releases aren't expected to be (but might > be). > AFAIK, that's the general rule, but python itself does not respect this convention, so I don't see this happening for python packages. cheers, David From p.f.moore at gmail.com Tue Apr 15 10:28:14 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 15 Apr 2008 09:28:14 +0100 Subject: [Distutils] Simple idea to resolve versioning problems In-Reply-To: <4803FE55.2000805@canterbury.ac.nz> References: <1208028604.3212.193.camel@portableevil.develix.com> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <20080414101003.GB27572@phare.normalesup.org> <4803374A.6020106@ar.media.kyoto-u.ac.jp> <4803FE55.2000805@canterbury.ac.nz> Message-ID: <79990c6b0804150128i61b2db55j1f17413355059ce6@mail.gmail.com> On 15/04/2008, Greg Ewing wrote: > That's a nice ideal to aim for, but it's only achievable > for old and mature packages. > > One could change the package name every time the API > changes, but then *any* change to the API would make the > package unusable by apps expecting an earlier version, > even if the new API is backwards compatible. Is nobody but me seeing shades of Windows "DLL hell" in all of this? Seriously, the problems being discussed here seem very similar to those encountered in the past by Windows with differing versions of DLLs. A number of solutions have been tried, with varying degrees of success. Anyone looking to address this problem for Python, really should take a look at the DLL hell history in Windows, if only to see what types of approach didn't work. Paul. From p.f.moore at gmail.com Tue Apr 15 10:57:52 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 15 Apr 2008 09:57:52 +0100 Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module) In-Reply-To: References: <20080317151637.532D23A409D@sparrow.telecommunity.com> <20080317161305.22B183A409D@sparrow.telecommunity.com> <47DEC0EB.3000404@palladion.com> <440CD260-F6F6-4484-8326-0EA1D7B83132@zooko.com> <79990c6b0803191134had5def2uff0742cc5188b125@mail.gmail.com> Message-ID: <79990c6b0804150157n558c7572x17f33582e6a585a9@mail.gmail.com> On 19/03/2008, Guido van Rossum wrote: > > I'm currently working on an addition to pkgutil to provide this type > > of function. I'm considering going a little further (adding functions > > to get a file-like object, test for existence, and list available > > resources, modelled on the pkg_resources functions - but these extra > > ones are not supported by the loader protocol, so I'm undecided as to > > whether it's worth it, I'll see how complex the code gets). > > I'd only do what __loader__ offers. People can always wrap a StringIO around it. > > > Once I have a patch, I'll post it to the tracker. What's the best > > approach? Code a patch for 3.0 and backport, or code for 2.6 and let > > the merging process do its stuff? > > Code for 2.6, let the merge do its thing. I've had a patch in http://bugs.python.org/issue2439 for a few weeks now. It just adds a get_data function, on the basis that that's all the loader API offers. I think it's complete - Phillip Eby helped thrash out a couple of problems. Is there anything I can do to get it applied, or should I just leave it now for someone to decide if they care enough? (As it's a library change, I don't know to what extent the "no API changes after the next alpha" rule will apply). Paul. From david at ar.media.kyoto-u.ac.jp Tue Apr 15 11:13:37 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 15 Apr 2008 18:13:37 +0900 Subject: [Distutils] Simple idea to resolve versioning problems In-Reply-To: <79990c6b0804150128i61b2db55j1f17413355059ce6@mail.gmail.com> References: <1208028604.3212.193.camel@portableevil.develix.com> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <20080414101003.GB27572@phare.normalesup.org> <4803374A.6020106@ar.media.kyoto-u.ac.jp> <4803FE55.2000805@canterbury.ac.nz> <79990c6b0804150128i61b2db55j1f17413355059ce6@mail.gmail.com> Message-ID: <480471C1.4080508@ar.media.kyoto-u.ac.jp> Paul Moore wrote: > > Is nobody but me seeing shades of Windows "DLL hell" in all of this? Dll hell was caused because there was no versioning, and new dll overwrote older ones, while not being compatible. If we add a versioning checking, we won't have dll hell problem, but dependency hell, actually, which is not much better. As far as I understand, with .net, MS started to support multiple version (several runtimes could be installed at the same time), but it did not really solve the deployment issues. > > Seriously, the problems being discussed here seem very similar to > those encountered in the past by Windows with differing versions of > DLLs. A number of solutions have been tried, with varying degrees of > success. Anyone looking to address this problem for Python, really > should take a look at the DLL hell history in Windows, if only to see > what types of approach didn't work. One of the recent approach I am aware of is the GAC (global assembly cache) for .net support. It enable multiple versions of the same assembly (more or less equivalent to a python module) to be installed at the same time: http://www.mono-project.com/Assemblies_and_the_GAC This document is actually very interesting (I've just read it while looking for a link for the GAC). Although aimed at .net development, it is almost entirely applicable to the problems we are talking about here. Interestingly, it recommends against having multiple versions of the same assembly in the GAC without committment to stable API. cheers, David From ncoghlan at gmail.com Tue Apr 15 11:57:27 2008 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 15 Apr 2008 19:57:27 +1000 Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module) In-Reply-To: <79990c6b0804150157n558c7572x17f33582e6a585a9@mail.gmail.com> References: <20080317151637.532D23A409D@sparrow.telecommunity.com> <20080317161305.22B183A409D@sparrow.telecommunity.com> <47DEC0EB.3000404@palladion.com> <440CD260-F6F6-4484-8326-0EA1D7B83132@zooko.com> <79990c6b0803191134had5def2uff0742cc5188b125@mail.gmail.com> <79990c6b0804150157n558c7572x17f33582e6a585a9@mail.gmail.com> Message-ID: <48047C07.1020807@gmail.com> Paul Moore wrote: > Is there anything I can do to get it applied, or should I just leave > it now for someone to decide if they care enough? (As it's a library > change, I don't know to what extent the "no API changes after the next > alpha" rule will apply). I'm looking into it now - assuming it applies cleanly and the tests run happily afterwards (and I don't see any reason it won't after reading the patch), it should be checked in later tonight. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From greg.ewing at canterbury.ac.nz Wed Apr 16 00:21:25 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 16 Apr 2008 10:21:25 +1200 Subject: [Distutils] Simple idea to resolve versioning problems In-Reply-To: <48040D2C.4010506@ar.media.kyoto-u.ac.jp> References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <20080414101003.GB27572@phare.normalesup.org> <4803374A.6020106@ar.media.kyoto-u.ac.jp> <4803FE55.2000805@canterbury.ac.nz> <48040D2C.4010506@ar.media.kyoto-u.ac.jp> Message-ID: <48052A65.7070702@canterbury.ac.nz> David Cournapeau wrote: > Library versioning without API > stability just does not make sense to me. Yes, obviously if you have library versioning you need to use it properly in order for it to be any use. > How do you do if you have a package D > which depends on both C and B, and C needs one version of one package A, > and B another one? If they really do need different versions, this is insoluble. If they can actually use the same version because it's compatible enough, it may be necessary to make Python smart enough to notice that they're symlinks to the same file and only import it once. Or maybe something does need to be added to the language that understands version numbering, compatibility rules, etc. as I suggested in another post. -- Greg From david at ar.media.kyoto-u.ac.jp Wed Apr 16 01:43:08 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 16 Apr 2008 08:43:08 +0900 Subject: [Distutils] Simple idea to resolve versioning problems In-Reply-To: <48052A65.7070702@canterbury.ac.nz> References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <20080414101003.GB27572@phare.normalesup.org> <4803374A.6020106@ar.media.kyoto-u.ac.jp> <4803FE55.2000805@canterbury.ac.nz> <48040D2C.4010506@ar.media.kyoto-u.ac.jp> <48052A65.7070702@canterbury.ac.nz> Message-ID: <48053D8C.5000101@ar.media.kyoto-u.ac.jp> Greg Ewing wrote: > > If they really do need different versions, this is insoluble. > But that's by far the most significant problem of packages with a lot of dependencies ! And by enabling multiple versions, side by side, you will make this problem more pervasive. I don't know if you have experience with packaging on linux systems, but if it was allowed multiple versions of the same library (say gtk 2.4, gtk 2.6, etc...) with incompatible changes of API in between, it would be a nightmare. What will happen if multiple versions of the same package are allowed in python ? Everyone will start to say :I need version == N, and quickly, it will be unmanageable, because the above diamond diagram A <- (B <- D1, C <- D2) will happen all the time. You really need to be able to say I need version >= N, which means maintaining API. It is ok to have a few major, side by side versions for fundamental packages (on my ubuntu system, I do have system installed qt 3 and qt 4, for example). This would be good to have in python, I agree. So my point is that package versioning only work if you can depends on one version and all revision after, which means side by side installation of several minor revisions does not help. Having 2,3 major different versions of the same package is OK, if the package changes say every two years (like python). But it won't work if it changes every month (imagine what would happen if python 2.3.1 was not compatible with 2.3.2 ?) IOW, enabling version requirement without strong API compatibility is the path to dependency hell. If you enable side by side installation of many versions of the same package, you will make the problem worse, not better. If you need to use a package without strong API compatibility commitment, then the only real solution is to make your own copy. cheers, David From greg.ewing at canterbury.ac.nz Wed Apr 16 05:03:51 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 16 Apr 2008 15:03:51 +1200 Subject: [Distutils] Simple idea to resolve versioning problems In-Reply-To: <79990c6b0804150128i61b2db55j1f17413355059ce6@mail.gmail.com> References: <1208028604.3212.193.camel@portableevil.develix.com> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <20080414101003.GB27572@phare.normalesup.org> <4803374A.6020106@ar.media.kyoto-u.ac.jp> <4803FE55.2000805@canterbury.ac.nz> <79990c6b0804150128i61b2db55j1f17413355059ce6@mail.gmail.com> Message-ID: <48056C97.80402@canterbury.ac.nz> Paul Moore wrote: > Is nobody but me seeing shades of Windows "DLL hell" in all of this? I think DLL Hell occurred partly because people *didn't* distinguish different API versions clearly with different file names. Also because programs were allowed to put shared stuff in the system directory with no form of reference counting. -- Greg From greg.ewing at canterbury.ac.nz Wed Apr 16 06:09:02 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 16 Apr 2008 16:09:02 +1200 Subject: [Distutils] Simple idea to resolve versioning problems In-Reply-To: <48053D8C.5000101@ar.media.kyoto-u.ac.jp> References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <20080414101003.GB27572@phare.normalesup.org> <4803374A.6020106@ar.media.kyoto-u.ac.jp> <4803FE55.2000805@canterbury.ac.nz> <48040D2C.4010506@ar.media.kyoto-u.ac.jp> <48052A65.7070702@canterbury.ac.nz> <48053D8C.5000101@ar.media.kyoto-u.ac.jp> Message-ID: <48057BDE.1080802@canterbury.ac.nz> David Cournapeau wrote: > Greg Ewing wrote: > > > If they really do need different versions, this is insoluble. > > But that's by far the most significant problem of packages with a lot of > dependencies ! But if your application really does depend on two libraries that have conflicting requirements like this, the application itself is screwed to begin with. There's *no* way of making it work on *any* system, whether that system has library versioning or not. Consequently, the developer will be unable to make it work on his own machine, and will therefore never attempt to deploy it! > It is ok to have a few > major, side by side versions for fundamental packages (on my ubuntu > system, I do have system installed qt 3 and qt 4, for example). This > would be good to have in python, I agree. That's what I'm talking about! As long as the API remains backward compatible, there is no need to have anything but the latest version installed -- and in fact, in the kind of system I have in mind, the earlier versions would *never be used* even if they were installed. That's because the program would say something like "Please give me gtk-2.3", and the system would know that anything called gtk-2.x with x >= 3 is compatible with that, and would pick the one with the greatest x out of those available. However, if a gtk-3.0 ever comes out, it implies that the API is *not* backward compatible. So then it makes sense to have both a gtk-2.x and a gtk-3.y installed, and the versioning system will keep them from getting confused. This relies on a couple of things: (a) library developers sticking rigidly to the version numbering semantics; (b) applications always requesting the version they actually require. > IOW, enabling version requirement without strong API compatibility is > the path to dependency hell. I fully agree. However, this is a social issue, not a technical one. Everyone using the versioning system would need to understand the rules and be willing to follow them. -- Greg From david at ar.media.kyoto-u.ac.jp Wed Apr 16 06:49:26 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 16 Apr 2008 13:49:26 +0900 Subject: [Distutils] Simple idea to resolve versioning problems In-Reply-To: <48057BDE.1080802@canterbury.ac.nz> References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <20080414101003.GB27572@phare.normalesup.org> <4803374A.6020106@ar.media.kyoto-u.ac.jp> <4803FE55.2000805@canterbury.ac.nz> <48040D2C.4010506@ar.media.kyoto-u.ac.jp> <48052A65.7070702@canterbury.ac.nz> <48053D8C.5000101@ar.media.kyoto-u.ac.jp> <48057BDE.1080802@canterbury.ac.nz> Message-ID: <48058556.6020202@ar.media.kyoto-u.ac.jp> Greg Ewing wrote: > > But if your application really does depend on two libraries that > have conflicting requirements like this, the application itself > is screwed to begin with. There's *no* way of making it work > on *any* system, whether that system has library versioning or > not. Of course. I was just stressing that versioning does not solve the deployment problem, but merely is a small part of the solution. > > Consequently, the developer will be unable to make it work on > his own machine, and will therefore never attempt to deploy it! It can happen easily once you have different systems: maybe on my os it is not a problem, but maybe it is on yours. Python is not exactly the same everywhere, the version of python is not the same everywhere, etc... > > That's what I'm talking about! As long as the API remains > backward compatible, there is no need to have anything but the > latest version installed -- and in fact, in the kind of system > I have in mind, the earlier versions would *never be used* > even if they were installed. What's the point of having it installed, then ? I am confused. >> IOW, enabling version requirement without strong API compatibility is >> the path to dependency hell. > > I fully agree. However, this is a social issue, not a > technical one. Yes, that's mainly a social issue, which is why I wanted to stress that versioning won't magically solve it (just to be clear, I am not saying that that's what you said, but I felt like one general idea in this thread was that once you have versioning, you solve deployment issues). All I am saying is: - If you use a package without a strong API commitment, and if you use versioning to use it, you will make problems worse, not better. - If you use versioning for packages with strong API commitment, then it is definitely useful. cheers, David From pjenvey at underboss.org Wed Apr 16 20:00:57 2008 From: pjenvey at underboss.org (Philip Jenvey) Date: Wed, 16 Apr 2008 11:00:57 -0700 Subject: [Distutils] Shebang lines on Jython Message-ID: <5E0A31A3-0C32-4DA1-9D71-0BE3A5B9B288@underboss.org> Speaking of shebang lines, there's problems with setuptools shebang lines on Jython. Unfortunately it's going to require another patch or two to setuptools. The problem being that Jython's executable is a .sh or .bat file (that'll invoke java), and interpreters in shebang lines can't be interpreter files, i.e. another script. Running a .py script that shebangs the Jython .sh executable directly will result in something along the lines of a "test.py: line 2 import command not found" sh error. For posix, we'll just need to special case Jython on posix to make a shebang along the lines of: #!/usr/bin/env /opt/local/bin/jython Windows isn't as easy. setuptools' launcher.c similarly can't use a .bat file as the interpreter. A workaround would be to use cmd.exe / c in the same way we'll use /usr/bin/env on posix. launcher.c currently can't handle this, though, the problem being it can't correctly quote the cmd.exe /c line correctly. With: #!c:\windows\system32\cmd.exe /c c:\jython2.2\jython.bat launcher.c ends up calling: "c:\windows\system32\cmd.exe" "/c" "c:\jython2.2\jython.bat" "c: \scripts\myscript-script.py" when we need: "c:\windows\system32\cmd.exe" /c ""c:\jython2.2\jython.bat" "c: \scripts\myscript-script.py"" which is another set of quotes around the entire /c arg, and yes no quotes around /c (I have no idea why "/c" doesn't work). I guess we would have to patch launcher.c to deal with this special situation or make a custom version for Jython. That's pretty awful -- might anyone have a simpler way of handling this? -- Philip Jenvey From pje at telecommunity.com Wed Apr 16 20:39:03 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 16 Apr 2008 14:39:03 -0400 Subject: [Distutils] Shebang lines on Jython In-Reply-To: <5E0A31A3-0C32-4DA1-9D71-0BE3A5B9B288@underboss.org> References: <5E0A31A3-0C32-4DA1-9D71-0BE3A5B9B288@underboss.org> Message-ID: <20080416183851.1B3C13A40AC@sparrow.telecommunity.com> At 11:00 AM 4/16/2008 -0700, Philip Jenvey wrote: >Speaking of shebang lines, there's problems with setuptools shebang >lines on Jython. Unfortunately it's going to require another patch or >two to setuptools. > >The problem being that Jython's executable is a .sh or .bat file >(that'll invoke java), and interpreters in shebang lines can't be >interpreter files, i.e. another script. Running a .py script that >shebangs the Jython .sh executable directly will result in something >along the lines of a "test.py: line 2 import command not found" sh >error. > >For posix, we'll just need to special case Jython on posix to make a >shebang along the lines of: > >#!/usr/bin/env /opt/local/bin/jython > >Windows isn't as easy. setuptools' launcher.c similarly can't use >a .bat file as the interpreter. A workaround would be to use cmd.exe / >c in the same way we'll use /usr/bin/env on posix. launcher.c >currently can't handle this, though, the problem being it can't >correctly quote the cmd.exe /c line correctly. > >With: > >#!c:\windows\system32\cmd.exe /c c:\jython2.2\jython.bat > >launcher.c ends up calling: > >"c:\windows\system32\cmd.exe" "/c" "c:\jython2.2\jython.bat" "c: >\scripts\myscript-script.py" > >when we need: > >"c:\windows\system32\cmd.exe" /c ""c:\jython2.2\jython.bat" "c: >\scripts\myscript-script.py"" > >which is another set of quotes around the entire /c arg, and yes no >quotes around /c (I have no idea why "/c" doesn't work). > >I guess we would have to patch launcher.c to deal with this special >situation or make a custom version for Jython. That's pretty awful -- >might anyone have a simpler way of handling this? Make a jython.exe that either runs the batch file, or does the same things itself. As you can see from launcher.c, it's not that hard to make one, and trivial to build it with the cygwin/MinGW tools. And then make jython's sys.executable on Windows be that Jython.exe. You can pretty much do the same thing for other platforms. Yes, I know it means you'll have to compile something, but it beats the living tar out of trying to handle parameters and options in shebang lines. Have a look at the chart on this page: http://www.in-ulm.de/~mascheck/various/shebang/ for some horror stories. From deron.meranda at gmail.com Wed Apr 16 21:42:12 2008 From: deron.meranda at gmail.com (Deron Meranda) Date: Wed, 16 Apr 2008 15:42:12 -0400 Subject: [Distutils] ez_setup.py fails with ImportError: No module named core Message-ID: <5c06fa770804161242j2547f4d9s13e5ceaf60acd751@mail.gmail.com> For some reason I can not bootstrap setuptools 0.6c8 into a python. This is Python 2.5 running under AIX 5.2, and the site-packages directory is completely empty. I've tried to use just ez_setup.py with no options, as well as downloading the egg file locally into a temporary directory and using: python ez_setup.py -v -H "" setuptools-0.6c8-py2.5.egg I get this strange traceback every time: Downloading http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c8-py2.5.egg Traceback (most recent call last): File "ez_setup.py", line 267, in main(sys.argv[1:]) File "ez_setup.py", line 200, in main from setuptools.command.easy_install import main File "build/bdist.linux-i686/egg/setuptools/__init__.py", line 2, in File "build/bdist.linux-i686/egg/setuptools/extension.py", line 1, in ImportError: No module named core Furthermore, the local egg file setuptools-0.6c8-py2.5.egg is removed from my working directory! My working directory is completely outside of the python path or site-packages dir. I also tried using the -d option to explicitly point to the site-packages dir, but the results are the same. I can not figure out what exactly is wrong. Any ideas? -- Deron Meranda From pje at telecommunity.com Wed Apr 16 21:53:36 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 16 Apr 2008 15:53:36 -0400 Subject: [Distutils] ez_setup.py fails with ImportError: No module named core In-Reply-To: <5c06fa770804161242j2547f4d9s13e5ceaf60acd751@mail.gmail.co m> References: <5c06fa770804161242j2547f4d9s13e5ceaf60acd751@mail.gmail.com> Message-ID: <20080416195423.481123A40AC@sparrow.telecommunity.com> At 03:42 PM 4/16/2008 -0400, Deron Meranda wrote: >For some reason I can not bootstrap setuptools 0.6c8 into a python. > >This is Python 2.5 running under AIX 5.2, and the site-packages >directory is completely empty. I've tried to use just ez_setup.py >with no options, as well as downloading the egg file locally into >a temporary directory and using: > python ez_setup.py -v -H "" setuptools-0.6c8-py2.5.egg > >I get this strange traceback every time: > >Downloading >http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c8-py2.5.egg >Traceback (most recent call last): > File "ez_setup.py", line 267, in > main(sys.argv[1:]) > File "ez_setup.py", line 200, in main > from setuptools.command.easy_install import main > File "build/bdist.linux-i686/egg/setuptools/__init__.py", line 2, > in > File "build/bdist.linux-i686/egg/setuptools/extension.py", line > 1, in >ImportError: No module named core This is an indication that your distutils installation is broken and/or missing. Make sure that if your OS has a 'python-dev' or 'python-devel' or 'python-distutils' package, that it's installed. From deron.meranda at gmail.com Wed Apr 16 21:55:35 2008 From: deron.meranda at gmail.com (Deron Meranda) Date: Wed, 16 Apr 2008 15:55:35 -0400 Subject: [Distutils] ez_setup.py fails with ImportError: No module named core In-Reply-To: <5c06fa770804161242j2547f4d9s13e5ceaf60acd751@mail.gmail.com> References: <5c06fa770804161242j2547f4d9s13e5ceaf60acd751@mail.gmail.com> Message-ID: <5c06fa770804161255j7ce7982ax3dad6a6a5d940086@mail.gmail.com> On Wed, Apr 16, 2008 at 3:42 PM, Deron Meranda wrote: > For some reason I can not bootstrap setuptools 0.6c8 into a python. > ... > Traceback (most recent call last): > File "ez_setup.py", line 267, in > main(sys.argv[1:]) > File "ez_setup.py", line 200, in main > from setuptools.command.easy_install import main > File "build/bdist.linux-i686/egg/setuptools/__init__.py", line 2, in > File "build/bdist.linux-i686/egg/setuptools/extension.py", line 1, in > ImportError: No module named core I think I figured this out. It's nothing to do with setuptools. The Python I'm using has a missing file, .../lib/python2.5/distutils/core.py I think something scanning the system must have thought it was a core dump file instead and removed it. -- Deron Meranda From pjenvey at underboss.org Wed Apr 16 23:40:47 2008 From: pjenvey at underboss.org (Philip Jenvey) Date: Wed, 16 Apr 2008 14:40:47 -0700 Subject: [Distutils] Shebang lines on Jython In-Reply-To: <20080416183851.1B3C13A40AC@sparrow.telecommunity.com> References: <5E0A31A3-0C32-4DA1-9D71-0BE3A5B9B288@underboss.org> <20080416183851.1B3C13A40AC@sparrow.telecommunity.com> Message-ID: <3C96C700-915A-42DE-A5C2-0B223F929985@underboss.org> On Apr 16, 2008, at 11:39 AM, Phillip J. Eby wrote: > Make a jython.exe that either runs the batch file, or does the same > things itself. As you can see from launcher.c, it's not that hard to > make one, and trivial to build it with the cygwin/MinGW tools. And > then make jython's sys.executable on Windows be that Jython.exe. > > You can pretty much do the same thing for other platforms. Yes, I > know it means you'll have to compile something, but it beats the > living tar out of trying to handle parameters and options in shebang > lines. Have a look at the chart on this page: > > http://www.in-ulm.de/~mascheck/various/shebang/ > > for some horror stories. Yea I'm trying not to worry about the arg parsing evilness on posix. Since it's not too common to pass an argument to python in the shebang line anyway, without going out of our way to support that we could get away with #!/usr/bin/env /opt/local/bin/jython Your point definitely applies to our Windows situation, though, thanks. Maybe we could ship a windows .exe and just keep posix the way it is for now. In the future maybe we could have our installer attempt to compile a runner on posix if a C compiler is installed, otherwise fallback to the .sh script, but I'm not going to deal with that at the moment. With that I'll have to create a patch to setuptools to use #!/usr/bin/ env if on jython and open(sys.executable).read(2) == '#!' and not sys.executable.endswith('.bat') (or some kind of windows check). -- Philip Jenvey From ianb at colorstudy.com Thu Apr 17 01:43:09 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 16 Apr 2008 18:43:09 -0500 Subject: [Distutils] Setuptools: problem with setuptools.command.bdist_egg:write_stub Message-ID: <48068F0D.3060006@colorstudy.com> In setuptools.command.bdist_egg:write_stub() Setuptools writes this function out alongside .so files: def __bootstrap__(): global __bootstrap__, __loader__, __file__ import sys, pkg_resources, imp __file__ = pkg_resources.resource_filename(__name__,) del __bootstrap__, __loader__ imp.load_dynamic(__name__,__file__) __bootstrap__() This is problematic on App Engine, as the .py file is loaded because .so files cannot be imported. However, unlike a zip file (where I assume this function is intended to be used) there is no __loader__. As a result "del __loader__" fails. I submitted this to App Engine (http://code.google.com/p/googleappengine/issues/detail?id=229), but I also think Setuptools is creating an unnecessarily fragile function here. -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From greg.ewing at canterbury.ac.nz Thu Apr 17 02:54:58 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 17 Apr 2008 12:54:58 +1200 Subject: [Distutils] Simple idea to resolve versioning problems In-Reply-To: <48058556.6020202@ar.media.kyoto-u.ac.jp> References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <20080414101003.GB27572@phare.normalesup.org> <4803374A.6020106@ar.media.kyoto-u.ac.jp> <4803FE55.2000805@canterbury.ac.nz> <48040D2C.4010506@ar.media.kyoto-u.ac.jp> <48052A65.7070702@canterbury.ac.nz> <48053D8C.5000101@ar.media.kyoto-u.ac.jp> <48057BDE.1080802@canterbury.ac.nz> <48058556.6020202@ar.media.kyoto-u.ac.jp> Message-ID: <48069FE2.9070200@canterbury.ac.nz> David Cournapeau wrote: > Greg Ewing wrote: > > > the earlier versions would *never be used* > > even if they were installed. > > What's the point of having it installed, then ? I am confused. I never said there was a point. When I mentioned multiple installed versions, I was talking about multiple *incompatible* versions. That part can be handled purely by convention -- just be disciplined enough to change the package name whenever an incompatible API change occurs. But the other part is what happens when there is a backward-compatible change -- apps need some way to say what minimum version they need, without having to exactly match the version installed. I've already mentioned one way of dealing with that -- install symlinks or .pth files or some such thing pointing from all the supported minor versions to the one that's currently installed. But that would cause difficulties due to different paths to the same module not being recognised as such. I think that Python would benefit from having some standard mechanism added to deal with this. One possibility I thought of is to extend the import statement so you can say things like import gtk[2,3] which means "give me gtk with major version == 2 and minor version >= 3". Any thoughts on that idea? -- Greg From pje at telecommunity.com Thu Apr 17 05:13:17 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 16 Apr 2008 23:13:17 -0400 Subject: [Distutils] Setuptools: problem with setuptools.command.bdist_egg:write_stub In-Reply-To: <48068F0D.3060006@colorstudy.com> References: <48068F0D.3060006@colorstudy.com> Message-ID: <20080417031306.8A8FD3A4096@sparrow.telecommunity.com> At 06:43 PM 4/16/2008 -0500, Ian Bicking wrote: >In setuptools.command.bdist_egg:write_stub() Setuptools writes this >function out alongside .so files: > >def __bootstrap__(): > global __bootstrap__, __loader__, __file__ > import sys, pkg_resources, imp > __file__ = pkg_resources.resource_filename(__name__,) > del __bootstrap__, __loader__ > imp.load_dynamic(__name__,__file__) >__bootstrap__() > >This is problematic on App Engine, as the .py file is loaded because .so >files cannot be imported. However, unlike a zip file (where I assume >this function is intended to be used) there is no __loader__. As a >result "del __loader__" fails. > >I submitted this to App Engine >(http://code.google.com/p/googleappengine/issues/detail?id=229), but I >also think Setuptools is creating an unnecessarily fragile function here. Until now, the only place the .py file would be imported in this scenario is in a zipfile... which ensures the loader would exist. If you're simply unzipping an egg here, you'll need to delete the .py as well as the .so, since neither is of any use here. Note that unless App Engine supports imp.load_dynamic(), all you're going to do by changing this is move the error one line down. I'm not sure I see how that's helpful. Certainly, though, if you want to provide a setuptools patch to add "__loader__ = None" before the "del" line, that would work. (I presume you are patching imp's load_dynamic to raise an ImportError, instead of the NameError or whatever you're currently getting from the del operation.) Of course, you'll have to rebuild any eggs you're using this way, since the bootstrap code is placed there by bdist_egg, not easy_install. From david at ar.media.kyoto-u.ac.jp Thu Apr 17 06:36:24 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 17 Apr 2008 13:36:24 +0900 Subject: [Distutils] [Python-3000] sizeof(size_t) < sizeof(long) In-Reply-To: <4806D2E9.3080905@canterbury.ac.nz> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22D20148@EXMBX04.exchhosting.com> <5c6f2a5d0804141538q2e5a777dp60807879a56105fe@mail.gmail.com> <87D3F9C72FBF214DB39FA4E3FE618CDC6E22E74C2C@EXMBX04.exchhosting.com> <5c6f2a5d0804141729t63dec56an4c53176249cee9b5@mail.gmail.com> <87D3F9C72FBF214DB39FA4E3FE618CDC6E22E74CB6@EXMBX04.exchhosting.com> <4804B83F.5010504@gmail.com> <48056F4A.4030102@canterbury.ac.nz> <48059966.4010102@v.loewis.de> <4806A3DE.6030807@canterbury.ac.nz> <4806B922.9070202@ar.media.kyoto-u.ac.jp> <4806BD48.6030908@canterbury.ac.nz> <4806C0C5.3090703@ar.media.kyoto-u.ac.jp> <4806D2E9.3080905@canterbury.ac.nz> Message-ID: <4806D3C8.1000508@ar.media.kyoto-u.ac.jp> Greg Ewing wrote: > When you go into a computer store and ask for > 256MB of RAM, you don't expect to be asked "What size > bytes would that be, then, sir?" > I ask for "256 Mo", Mo for Mega-octet: French (and most non English languages I am aware of) does not have this ambiguity :) And anyway, in a computer store, you find memory for personal computers, where one byte always has 8 bits. > So it's a de facto standard, and one that works perfectly > well. Going against it is both futile and unnecessary, > as far as I can see. > Going against the C standard seems pretty futile to me. cheers, David From greg.ewing at canterbury.ac.nz Thu Apr 17 07:15:38 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 17 Apr 2008 17:15:38 +1200 Subject: [Distutils] [Python-3000] sizeof(size_t) < sizeof(long) In-Reply-To: <4806D3C8.1000508@ar.media.kyoto-u.ac.jp> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22D20148@EXMBX04.exchhosting.com> <5c6f2a5d0804141538q2e5a777dp60807879a56105fe@mail.gmail.com> <87D3F9C72FBF214DB39FA4E3FE618CDC6E22E74C2C@EXMBX04.exchhosting.com> <5c6f2a5d0804141729t63dec56an4c53176249cee9b5@mail.gmail.com> <87D3F9C72FBF214DB39FA4E3FE618CDC6E22E74CB6@EXMBX04.exchhosting.com> <4804B83F.5010504@gmail.com> <48056F4A.4030102@canterbury.ac.nz> <48059966.4010102@v.loewis.de> <4806A3DE.6030807@canterbury.ac.nz> <4806B922.9070202@ar.media.kyoto-u.ac.jp> <4806BD48.6030908@canterbury.ac.nz> <4806C0C5.3090703@ar.media.kyoto-u.ac.jp> <4806D2E9.3080905@canterbury.ac.nz> <4806D3C8.1000508@ar.media.kyoto-u.ac.jp> Message-ID: <4806DCFA.2080904@canterbury.ac.nz> David Cournapeau wrote: > I ask for "256 Mo", Mo for Mega-octet: French (and most non English > languages I am aware of) does not have this ambiguity :) But would you ask for it that way in an English-speaking country? Would they have a clue what you were talking about? Anyway, I don't think it really is an ambiguity in practice -- only in the minds of those that have too much time to read standards documents. > And anyway, in > a computer store, you find memory for personal computers, where one byte > always has 8 bits. Does it? Or... > Going against the C standard seems pretty futile to me. Hmmm. So as long as I program my computer in C, it has 8 bit bytes... or at least a C implementation with 8-bit chars. But if I use a special Unicode-C with 16-bit chars, I've then got half as many bytes of memory as I had before... So I have to tell the guy in the computer shop what programming language and implementation I'm using, too! :-) Seems to me that disregarding this particular quirk of the C standard is a lesser evil than confusing everybody by changing the meaning of byte all the time. [In attempt to pull this very slightly closer to being on-topic... what does this imply for the Python bytes type? Are they always 8-bit bytes, or are they whatever size your C compiler thinks bytes are? Do the Python docs need to clarify this?] -- Greg From skip at pobox.com Fri Apr 18 04:20:08 2008 From: skip at pobox.com (skip at pobox.com) Date: Thu, 17 Apr 2008 21:20:08 -0500 Subject: [Distutils] How to test setup.py with easy_install? Message-ID: <18440.1368.358825.77976@montanaro-dyndns-org.local> So I've just added install_requires=["lockfile>=0.2"] to my setup() call. How do I test that from within my local source repository? Running "python setup.py install" doesn't seem to do anything with it (I didn't expect it to). I assume I have to get easy_install to process the setup.py file in the current directory somehow, but I don't see how. Thanks, Skip From skip at pobox.com Fri Apr 18 05:24:44 2008 From: skip at pobox.com (skip at pobox.com) Date: Thu, 17 Apr 2008 22:24:44 -0500 Subject: [Distutils] How to test setup.py with easy_install? In-Reply-To: <18440.1368.358825.77976@montanaro-dyndns-org.local> References: <18440.1368.358825.77976@montanaro-dyndns-org.local> Message-ID: <18440.5244.360484.65592@montanaro-dyndns-org.local> >>>>> "skip" == skip writes: skip> So I've just added skip> install_requires=["lockfile>=0.2"] skip> to my setup() call. How do I test that from within my local skip> source repository? Okay, I figured it out. I wasn't thinking with all three of my neurons. Two of them have already fallen asleep. It's been a long day... Sorry for the noise... Skip From bignose+hates-spam at benfinney.id.au Fri Apr 18 08:33:31 2008 From: bignose+hates-spam at benfinney.id.au (Ben Finney) Date: Fri, 18 Apr 2008 16:33:31 +1000 Subject: [Distutils] desirability of multiple, divergent Python instances References: <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <87od8dtacu.fsf_-_@benfinney.id.au> <4802E4BD.4090408@pangalactic.us> Message-ID: <87zlrr4qpw.fsf@benfinney.id.au> Stephen Waterbury writes: > Ben Finney wrote: > > It's my position that the Python instance one uses for development > > should diverge as little as possible from the default system > > instance. Otherwise one is actively pursuing a recipe for > > dependency failures when one eventually deploys the result. > > > [...] > > Help me understand, please. > > Have you read my proposal? I'd read it, yes. Thanks for bringing it explicitly back into the thread. > My main motivation in proposing it is to give the user complete > control over the Python environment that is in their path -- which > is not easy to do on Debian/Ubuntu at the moment This is a laudable goal. It is met, for just about every other user-configurable application, by allowing user-specific *overrides* to the defaults to be set in configuration files. > by giving the Python utilities and packages that OS-related > functions depend on a completely insular Python environment that the > OS can manage however it sees fit. In my view, both as a developer > and a user, but especially as a developer, it would make life > easier. I can't see how you come to this conclusion. Surely one wants the possibility of divergence from the default to be explicit and easily discoverable? This is entirely lost if the only assumption that can be made is "the entire Python environment has been replicated and utterly isolated from the system default Python". Stephen Waterbury writes: > "system Python" -- the Python (and its site-pkgs, etc.) that any > Python scripts used by the OS depend on, which would of course by > solely determined by the OS. > > This would be distinct, in my proposal, from any packages named > "python" or "python.x" in the package namespace of the system > package manager (e.g. apt). It seems clear to me that divergence between these two should be kept to a minimum; and that such divergence is best minimalised if they are *the same instance*, merely operating under explicit customisation for each use that needs it. -- \ "Pinky, are you pondering what I'm pondering?" "I think so, | `\ Brain, but isn't that why they invented tube socks?" -- _Pinky | _o__) and The Brain_ | Ben Finney From bignose+hates-spam at benfinney.id.au Fri Apr 18 08:39:16 2008 From: bignose+hates-spam at benfinney.id.au (Ben Finney) Date: Fri, 18 Apr 2008 16:39:16 +1000 Subject: [Distutils] shebang line modified by setuptools References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <20080414101003.GB27572@phare.normalesup.org> Message-ID: <87skxj4qgb.fsf@benfinney.id.au> Gael Varoquaux writes: > On Mon, Apr 14, 2008 at 01:30:14PM +1200, Greg Ewing wrote: > > Maybe the system should come with two pythons installed, one for > > use by the system and the other for users to add things to. Or at > > least be set up so that it appears that way -- they might share > > files under the hood. > > I am quite wary about these proposals, as well as the one > environment per application ones. > > What you propose resembles very much to what MacOSX does, and MacOSX > seems just so broken for Python. I don't use it, but I see on the > different scientific-Python-related mailing lists how users have > difficulties with MacOSX, and I have heard many people complain > about this "feature". > > As a per-application environment, I think it is bad, because it > limits reuse. As I see things, applications should be able to have > access to all the Python modules installed, to be able to use them, > if they need. I get definitely see applications having more feature > if some modules are installed (eg. Sphinx, which does syntax > highlighting if pygments is installed). Thank you. This is a major part of my concern. The Python environment should be reliably the same when invoked on a given system configuration, with egregious application-specific differences to *the instance itself* kept to a minimum. That's not to say that applications can't *override* the default instance; but such customisation should be kept to a minimum in order that dependencies, special cases, etc. are minimised. -- \ "What's another word for Thesaurus?" -- Steven Wright | `\ | _o__) | Ben Finney From floris.bruynooghe at gmail.com Fri Apr 18 10:05:32 2008 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Fri, 18 Apr 2008 09:05:32 +0100 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <1006AECE-3273-418A-A804-228177D4DCBB@python.org> References: <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> Message-ID: <20080418080532.GA2610@laurie.devork> On Sun, Apr 13, 2008 at 07:59:21AM -0400, Barry Warsaw wrote: > On Apr 12, 2008, at 9:41 PM, Stephen Waterbury wrote: > > > > I used to always set up my own Python[s] in /usr/local > > and put that first in my PATH, but I have gotten lazy lately, and > > sometimes it will bite me. ;) > > On Debian and derivatives (e.g. Ubuntu) you might have even more fun. > They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of > the system python*! This means that you can break your system Python > by installing a version of Python from source and then distutil'ing > things into there. Astoundingly, this is promoted as a feature. > > I've reported bugs on this and had discussions with some of the Debian > Python packaging folks. I'm hoping that we'll find a solution that > doesn't collide with a from-source default Python build. I think it would be really good if we could agree on a simple solution for this problem. However I haven't really seen an agreement come out of this thread. So far what (I think) some proposed solutions are: 1 Have a different path in /usr/local (not /usr/local/lib/pythonX.Y/site-packages) to augment the system sys.path with. 2 Make sure /usr/local is near the end of the sys.path so system apps won't break due to it (currently /usr/local/lib/pythonX.Y/site-packages appears before /usr/lib/pythonX.Y/site-packages on my system, not sure if this is normal and if not who's fault that is). 3 Don't support compiling your own python in /usr/local, do that in /opt. 4 Introduce a hidden python on the system which the system uses to run system applications that use python. The user would be unable to change that environment unless they really tried. (But surely then there are still discussions about which paths end up on sys.path for the user python and which paths are for locally compiled pythons, I might have missed something in this discussion). Personally I think the solution should be as simple as possible so I'd go for 1, 2 or 3. 3 will be difficult to teach people (definitely if they disagree) so that won't work, which leaves 1 and 2. Opinions? Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From fairwinds at eastlink.ca Fri Apr 18 06:46:21 2008 From: fairwinds at eastlink.ca (David Pratt) Date: Fri, 18 Apr 2008 01:46:21 -0300 Subject: [Distutils] programatically making eggs Message-ID: <4808279D.9000802@eastlink.ca> Hi. What would be the best way of programatically making eggs as opposed to python setup.py bdist_egg sdist on the command line. Many thanks. From chris at simplistix.co.uk Fri Apr 18 12:19:52 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 18 Apr 2008 11:19:52 +0100 Subject: [Distutils] zip_safe=False not being respected? Message-ID: <480875C8.6090309@simplistix.co.uk> Hi All, I'm trying to install a package into a Zope INSTANCE_HOME with: python2.4 setup.py install --home=$INSTANCE_HOME ...however, this is leaving me with an easy-install.pth and a .egg file in $INSTANCE_HOME, even though the package has zip_safe=False in its setup.py. What gives? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Fri Apr 18 12:21:00 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 18 Apr 2008 11:21:00 +0100 Subject: [Distutils] zip_safe=False not being respected? [correction] In-Reply-To: <480875C8.6090309@simplistix.co.uk> References: <480875C8.6090309@simplistix.co.uk> Message-ID: <4808760C.1050601@simplistix.co.uk> Chris Withers wrote: > ...however, this is leaving me with an easy-install.pth and a .egg file > in $INSTANCE_HOME, ...should have been "in $INSTANCE_HOME/lib/python". cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Fri Apr 18 12:30:37 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 18 Apr 2008 11:30:37 +0100 Subject: [Distutils] zip_safe=False not being respected? [it gets worse] In-Reply-To: <4808760C.1050601@simplistix.co.uk> References: <480875C8.6090309@simplistix.co.uk> <4808760C.1050601@simplistix.co.uk> Message-ID: <4808784D.5000504@simplistix.co.uk> Chris Withers wrote: > Chris Withers wrote: >> ...however, this is leaving me with an easy-install.pth and a .egg file >> in $INSTANCE_HOME, Okay, so the .egg is actually a directory and not a file. ...and in there is lurking the package I'm trying install. Why all this convolution? Why not install the package where I actually asked it to go? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From waterbug at pangalactic.us Fri Apr 18 13:57:31 2008 From: waterbug at pangalactic.us (Stephen Waterbury) Date: Fri, 18 Apr 2008 07:57:31 -0400 Subject: [Distutils] desirability of multiple, divergent Python instances In-Reply-To: <87zlrr4qpw.fsf@benfinney.id.au> References: <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <87od8dtacu.fsf_-_@benfinney.id.au> <4802E4BD.4090408@pangalactic.us> <87zlrr4qpw.fsf@benfinney.id.au> Message-ID: <48088CAB.2080209@pangalactic.us> Ben Finney wrote: > Stephen Waterbury writes: > >> "system Python" -- the Python (and its site-pkgs, etc.) that any >> Python scripts used by the OS depend on, which would of course be >> solely determined by the OS. >> >> This would be distinct, in my proposal, from any packages named >> "python" or "python.x" in the package namespace of the system >> package manager (e.g. apt). > > It seems clear to me that divergence between these two should be kept > to a minimum ... Why? Steve From jim at zope.com Fri Apr 18 14:28:35 2008 From: jim at zope.com (Jim Fulton) Date: Fri, 18 Apr 2008 08:28:35 -0400 Subject: [Distutils] zip_safe=False not being respected? In-Reply-To: <480875C8.6090309@simplistix.co.uk> References: <480875C8.6090309@simplistix.co.uk> Message-ID: <200804181228.m3ICSphL007939@mail116c25.carrierzone.com> On Apr 18, 2008, at 6:19 AM, Chris Withers wrote: > Hi All, > > I'm trying to install a package into a Zope INSTANCE_HOME with: > > python2.4 setup.py install --home=$INSTANCE_HOME > > ...however, this is leaving me with an easy-install.pth and a .egg > file > in $INSTANCE_HOME, even though the package has zip_safe=False in its > setup.py. > > What gives? What platform? --home is ignored on windows. Note that zope-safe has nothing to do with install location. Jim -- Jim Fulton Zope Corporation From jim at zope.com Fri Apr 18 14:33:47 2008 From: jim at zope.com (Jim Fulton) Date: Fri, 18 Apr 2008 08:33:47 -0400 Subject: [Distutils] zip_safe=False not being respected? In-Reply-To: <200804181228.m3ICSphL007939@mail116c25.carrierzone.com> References: <480875C8.6090309@simplistix.co.uk> <200804181228.m3ICSphL007939@mail116c25.carrierzone.com> Message-ID: <200804181234.m3ICY3ck010548@mail107c25.carrierzone.com> On Apr 18, 2008, at 8:28 AM, Jim Fulton wrote: > > On Apr 18, 2008, at 6:19 AM, Chris Withers wrote: >> Hi All, >> >> I'm trying to install a package into a Zope INSTANCE_HOME with: >> >> python2.4 setup.py install --home=$INSTANCE_HOME >> >> ...however, this is leaving me with an easy-install.pth and a .egg >> file >> in $INSTANCE_HOME, even though the package has zip_safe=False in its >> setup.py. >> >> What gives? > > > What platform? --home is ignored on windows. Sorry, I didn't read your note carefully enough. --home says to use a unix-style layout rooted at the given directory. I believe you want --install-lib Jim -- Jim Fulton Zope Corporation From lxander.m at gmail.com Fri Apr 18 14:34:34 2008 From: lxander.m at gmail.com (Alexander Michael) Date: Fri, 18 Apr 2008 08:34:34 -0400 Subject: [Distutils] zip_safe=False not being respected? [it gets worse] In-Reply-To: <4808784D.5000504@simplistix.co.uk> References: <480875C8.6090309@simplistix.co.uk> <4808760C.1050601@simplistix.co.uk> <4808784D.5000504@simplistix.co.uk> Message-ID: <525f23e80804180534g27f1efe6te723f0b03fcc885d@mail.gmail.com> On Fri, Apr 18, 2008 at 6:30 AM, Chris Withers wrote: > Chris Withers wrote: > > Chris Withers wrote: > >> ...however, this is leaving me with an easy-install.pth and a .egg file > >> in $INSTANCE_HOME, > > Okay, so the .egg is actually a directory and not a file. > ...and in there is lurking the package I'm trying install. > > Why all this convolution? Why not install the package where I actually > asked it to go? By default, setuptools enabled setup.py's install eggs (either zipped or unzipped) so that you can multiple version installed (but only one active, of course). If you want to install just a single version as a standard python package directory, then use the --single-version-externally-managed option as described in the documentation: . From fairwinds at eastlink.ca Fri Apr 18 15:11:23 2008 From: fairwinds at eastlink.ca (David Pratt) Date: Fri, 18 Apr 2008 10:11:23 -0300 Subject: [Distutils] programatically making eggs In-Reply-To: <4808279D.9000802@eastlink.ca> References: <4808279D.9000802@eastlink.ca> Message-ID: <48089DFB.7060405@eastlink.ca> It looks like what I should be doing is subclassing Command from setuptools, then passing in user options. I'll be trying a few things out this way but any additional info would be helpful. Many thanks. Regards, David David Pratt wrote: > Hi. What would be the best way of programatically making eggs as opposed > to python setup.py bdist_egg sdist on the command line. Many thanks. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From ziade.tarek at gmail.com Fri Apr 18 15:36:03 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 18 Apr 2008 15:36:03 +0200 Subject: [Distutils] programatically making eggs In-Reply-To: <48089DFB.7060405@eastlink.ca> References: <4808279D.9000802@eastlink.ca> <48089DFB.7060405@eastlink.ca> Message-ID: <94bdd2610804180636g13a9bcc2s9734056e91be2670@mail.gmail.com> On Fri, Apr 18, 2008 at 3:11 PM, David Pratt wrote: > It looks like what I should be doing is subclassing Command from > setuptools, then passing in user options. I'll be trying a few things > out this way but any additional info would be helpful. Many thanks. or you can use distutils.core.run_setup, which allows you to call setup() with your arguments, in a realistic way. in any case, it just consists of calling setup() with a sys.argv correctly filled That would be simpler imho instead of creating a new command Tarek > > Regards, > David > > > > David Pratt wrote: > > Hi. What would be the best way of programatically making eggs as opposed > > to python setup.py bdist_egg sdist on the command line. Many thanks. > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > http://mail.python.org/mailman/listinfo/distutils-sig > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From fairwinds at eastlink.ca Fri Apr 18 15:42:01 2008 From: fairwinds at eastlink.ca (David Pratt) Date: Fri, 18 Apr 2008 10:42:01 -0300 Subject: [Distutils] programatically making eggs In-Reply-To: <94bdd2610804180636g13a9bcc2s9734056e91be2670@mail.gmail.com> References: <4808279D.9000802@eastlink.ca> <48089DFB.7060405@eastlink.ca> <94bdd2610804180636g13a9bcc2s9734056e91be2670@mail.gmail.com> Message-ID: <4808A529.1030504@eastlink.ca> Your right, Tarek. This is much more sane. Many thanks for the advice. Regards, David Tarek Ziad? wrote: > On Fri, Apr 18, 2008 at 3:11 PM, David Pratt wrote: >> It looks like what I should be doing is subclassing Command from >> setuptools, then passing in user options. I'll be trying a few things >> out this way but any additional info would be helpful. Many thanks. > > or you can use distutils.core.run_setup, which allows you to call setup() > with your arguments, in a realistic way. > > in any case, it just consists of calling setup() with a sys.argv > correctly filled > > That would be simpler imho instead of creating a new command > > Tarek > > >> Regards, >> David >> >> >> >> David Pratt wrote: >> > Hi. What would be the best way of programatically making eggs as opposed >> > to python setup.py bdist_egg sdist on the command line. Many thanks. >> > _______________________________________________ >> > Distutils-SIG maillist - Distutils-SIG at python.org >> > http://mail.python.org/mailman/listinfo/distutils-sig >> > >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig >> > > > From chris at simplistix.co.uk Fri Apr 18 16:41:06 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 18 Apr 2008 15:41:06 +0100 Subject: [Distutils] zip_safe=False not being respected? In-Reply-To: <200804181234.m3ICY3ck010548@mail107c25.carrierzone.com> References: <480875C8.6090309@simplistix.co.uk> <200804181228.m3ICSphL007939@mail116c25.carrierzone.com> <200804181234.m3ICY3ck010548@mail107c25.carrierzone.com> Message-ID: <4808B302.1020806@simplistix.co.uk> Jim Fulton wrote: > >> What platform? --home is ignored on windows. > > Sorry, I didn't read your note carefully enough. > > --home says to use a unix-style layout rooted at the given directory. > > I believe you want --install-lib python2.4 setup.py install --install-lib=$INSTANCE/lib/python ...still gives me a .pth and .egg dir in $INSTANCE/lib/python, which isn't what I want as .pth files don't work in $INSTANCE/lib/python cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From lxander.m at gmail.com Fri Apr 18 18:02:00 2008 From: lxander.m at gmail.com (Alexander Michael) Date: Fri, 18 Apr 2008 12:02:00 -0400 Subject: [Distutils] zip_safe=False not being respected? [it gets worse] In-Reply-To: <4808B266.9030608@simplistix.co.uk> References: <480875C8.6090309@simplistix.co.uk> <4808760C.1050601@simplistix.co.uk> <4808784D.5000504@simplistix.co.uk> <525f23e80804180532pab84ecqf65db167073eaebe@mail.gmail.com> <4808B266.9030608@simplistix.co.uk> Message-ID: <525f23e80804180902j5310344dwa316aebd87c9cc5f@mail.gmail.com> On Fri, Apr 18, 2008 at 10:38 AM, Chris Withers wrote: > Alexander Michael wrote: > > > By default, setuptools enabled setup.py's install eggs (either zipped > > or unzipped) so that you can multiple version installed (but only one > > active, of course). If you want to install just a single version as a > > standard python package directory, then use the > > --single-version-externally-managed option as described in the > > documentation: > > > > $ python2.4 setup.py install --home=$INSTANCE \ > --single-version-externally-managed > running install > error: You must specify --record or --root when building system packages > > Sadly neither the --record or --root options appear to be documented > anywhere... That is sad. There is a hint from "python setup.py install --help": --root install everything relative to this alternate root directory --record filename in which to record list of installed files Maybe they're provided by standard distutils? I know the seuptools documentation doesn't reiterate anything about distutils, just the additions. From pje at telecommunity.com Fri Apr 18 19:58:41 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 18 Apr 2008 13:58:41 -0400 Subject: [Distutils] zip_safe=False not being respected? [it gets worse] In-Reply-To: <525f23e80804180902j5310344dwa316aebd87c9cc5f@mail.gmail.co m> References: <480875C8.6090309@simplistix.co.uk> <4808760C.1050601@simplistix.co.uk> <4808784D.5000504@simplistix.co.uk> <525f23e80804180532pab84ecqf65db167073eaebe@mail.gmail.com> <4808B266.9030608@simplistix.co.uk> <525f23e80804180902j5310344dwa316aebd87c9cc5f@mail.gmail.com> Message-ID: <20080418175841.255F33A409D@sparrow.telecommunity.com> At 12:02 PM 4/18/2008 -0400, Alexander Michael wrote: >On Fri, Apr 18, 2008 at 10:38 AM, Chris Withers > wrote: > > Alexander Michael wrote: > > > > > By default, setuptools enabled setup.py's install eggs (either zipped > > > or unzipped) so that you can multiple version installed (but only one > > > active, of course). If you want to install just a single version as a > > > standard python package directory, then use the > > > --single-version-externally-managed option as described in the > > > documentation: > > > > > > > $ python2.4 setup.py install --home=$INSTANCE \ > > --single-version-externally-managed > > running install > > error: You must specify --record or --root when building system packages > > > > Sadly neither the --record or --root options appear to be documented > > anywhere... > >That is sad. There is a hint from "python setup.py install --help": > --root install everything relative to this > alternate root directory > --record filename in which to record list of > installed files > >Maybe they're provided by standard distutils? I know the seuptools >documentation doesn't reiterate anything about distutils, just the >additions. Correct: --root and --record are standard distutils options. In this case, you want --record, not --root. The reason this option requires either --root or --record, is so that you have a way to know what was installed and uninstall it. In the --root case, everything installed will be under a single directory tree, and in the --record case, you'll have a list of the files. From lists at zopyx.com Fri Apr 18 20:45:24 2008 From: lists at zopyx.com (Andreas Jung) Date: Fri, 18 Apr 2008 20:45:24 +0200 Subject: [Distutils] [zc.buildout] Accessing part configuration from other recipe? Message-ID: Hi, is it possible to get hold of the buildout configuration of part A from within the recipe configured for part B? Usecase: the configuration of part A should work as blueprint for actions performed through the recipe configured for part B. Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20080418/5ef3b2ac/attachment.pgp From jim at zope.com Fri Apr 18 20:54:10 2008 From: jim at zope.com (Jim Fulton) Date: Fri, 18 Apr 2008 14:54:10 -0400 Subject: [Distutils] [zc.buildout] Accessing part configuration from other recipe? In-Reply-To: References: Message-ID: <0E2895E6-9CB8-4867-A828-B61EF9CD5B0D@zope.com> On Apr 18, 2008, at 2:45 PM, Andreas Jung wrote: > is it possible to get hold of the buildout configuration of part A > from within the recipe configured for part B? Yes. The __init__ method of a recipe can read other section options using buildout[section_name][option_name]. Note that any data read should be reflected in the reading parts options so that buildout can know if a part's configuration has changed. Jim -- Jim Fulton Zope Corporation From bignose+hates-spam at benfinney.id.au Sat Apr 19 01:05:12 2008 From: bignose+hates-spam at benfinney.id.au (Ben Finney) Date: Sat, 19 Apr 2008 09:05:12 +1000 Subject: [Distutils] shebang line modified by setuptools References: <20080412161922.6B3A33A406A@sparrow.telecommunity.com> <1208019508.3212.169.camel@portableevil.develix.com> <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <20080418080532.GA2610@laurie.devork> Message-ID: <87hcdysr13.fsf@benfinney.id.au> Floris Bruynooghe writes: > I think it would be really good if we could agree on a simple > solution for this problem. However I haven't really seen an > agreement come out of this thread. So far what (I think) some > proposed solutions are: > > 1 Have a different path in /usr/local (not > /usr/local/lib/pythonX.Y/site-packages) to augment the system > sys.path with. > > 2 Make sure /usr/local is near the end of the sys.path so system apps > won't break due to it (currently > /usr/local/lib/pythonX.Y/site-packages appears before > /usr/lib/pythonX.Y/site-packages on my system, not sure if this is > normal and if not who's fault that is). These both appear to be addressing the issue that customisations (of the form "install a new Python package") in '/usr/local' can disrupt the behaviour of system tools using Python. So, here's my proposed solution: Provide a simple way for a Python program or instance to specify "don't have '/usr/local' in the 'sys.path'". Recommend that the system Python tools use this feature to ignore '/usr/local'. Continue using '/usr/local' for custom site-local packages. That is, if we're talking about the system Python tools being the unusual case ("don't want site-local custom packages"), then that's where the focus should be on allowing for that special case. Yes, I'm aware of 'sys.path.remove("/usr/local")'. Is that the simplest way to ensure the above behaviour? -- \ "If you wish to strive for peace of soul, then believe; if you | `\ wish to be a devotee of truth, then inquire." -- Friedrich | _o__) Nietzsche | Ben Finney ? From languitar at semipol.de Sun Apr 20 00:18:35 2008 From: languitar at semipol.de (Johannes Wienke) Date: Sun, 20 Apr 2008 00:18:35 +0200 Subject: [Distutils] unit testing and stub c shared libraries Message-ID: <480A6FBB.1030502@semipol.de> Hi, I'm currently struggling to find a good way for running unit tests in my project. Adding a custom Command to my setup.py for running the tests wouldn't be a big problem but the unit tests need a few additional shared libraries. These libraries are simple stub files used for testing the plugin loader I'm developing. So is there a way to tell distutils to build those shared libraries only if want to run the unit tests? Otherwise those libs should be ignored because I don't want them to be installed generally. Thanks for the help Johannes -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/distutils-sig/attachments/20080420/f9a4ef6f/attachment.pgp From floris.bruynooghe at gmail.com Sun Apr 20 19:15:15 2008 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Sun, 20 Apr 2008 18:15:15 +0100 Subject: [Distutils] shebang line modified by setuptools In-Reply-To: <87hcdysr13.fsf@benfinney.id.au> References: <20080412175146.51B3E3A409E@sparrow.telecommunity.com> <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <20080418080532.GA2610@laurie.devork> <87hcdysr13.fsf@benfinney.id.au> Message-ID: <20080420171515.GA28569@laurie.devork> On Sat, Apr 19, 2008 at 09:05:12AM +1000, Ben Finney wrote: > Floris Bruynooghe writes: > > > I think it would be really good if we could agree on a simple > > solution for this problem. However I haven't really seen an > > agreement come out of this thread. So far what (I think) some > > proposed solutions are: > > > > 1 Have a different path in /usr/local (not > > /usr/local/lib/pythonX.Y/site-packages) to augment the system > > sys.path with. > > > > 2 Make sure /usr/local is near the end of the sys.path so system apps > > won't break due to it (currently > > /usr/local/lib/pythonX.Y/site-packages appears before > > /usr/lib/pythonX.Y/site-packages on my system, not sure if this is > > normal and if not who's fault that is). > > These both appear to be addressing the issue that customisations (of > the form "install a new Python package") in '/usr/local' can disrupt > the behaviour of system tools using Python. > > So, here's my proposed solution: > > Provide a simple way for a Python program or instance to specify > "don't have '/usr/local' in the 'sys.path'". Recommend that the system > Python tools use this feature to ignore '/usr/local'. Continue using > '/usr/local' for custom site-local packages. Seems hard. While it might be feasable for apps (who will live in known directories) it becomes a lot harder when random applications ship python scripts in some random directory, e.g. cgi-bin or so. It basically means that a large amount of packages in a distro will always be broken to this effect until somone stumbles over this particular bug for this particular package and happens to be good enough to file that bug. It's the same problem as with requiring system packages to use a separate python installation with a more strictly pinned sys.path. > That is, if we're talking about the system Python tools being the > unusual case ("don't want site-local custom packages"), then that's > where the focus should be on allowing for that special case. So most people here don't like Debian's and SUSE's behaviour, that's sad but anyway, which means 1 above --introducing a new Python endorsed site-packages dir in /usr/local-- seems unlikely. Since 2 above --making sure /usr/local is at the end of sys.path-- can be done entirely by the distributions who like /usr/local on sys.path this seems like the best option. In Debian this is currently implemented by adding /usr/local/ to the prefixes of site.py, ensuring that .pth files in /usr/local are still usable. So just to err on the side of safety: can anyone think of a reason that having /usr/local/lib/pythonX.Y/site-packages/ at the end of sys.path in this way could cause problems? Thanks Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From setuptools at bugs.python.org Mon Apr 21 04:28:30 2008 From: setuptools at bugs.python.org (Jeff Rush) Date: Mon, 21 Apr 2008 02:28:30 +0000 Subject: [Distutils] [issue8] A Test Issue In-Reply-To: <1208744910.36.0.637775752197.issue8@psf.upfronthosting.co.za> Message-ID: <1208744910.36.0.637775752197.issue8@psf.upfronthosting.co.za> New submission from Jeff Rush : Testing a bug post. ---------- assignedto: jrush messages: 13 nosy: jrush priority: urgent status: deferred title: A Test Issue _______________________________________________ Setuptools tracker _______________________________________________ From setuptools at bugs.python.org Mon Apr 21 04:34:05 2008 From: setuptools at bugs.python.org (Jeff Rush) Date: Mon, 21 Apr 2008 02:34:05 +0000 Subject: [Distutils] [issue9] Emailed Bug Report re Setuptools In-Reply-To: <480BF9B9.3010504@taupro.com> Message-ID: <480BF9B9.3010504@taupro.com> New submission from Jeff Rush : a test bug report re setuptools of the new tracker instance. ---------- messages: 14 nosy: jrush status: unread title: Emailed Bug Report re Setuptools _______________________________________________ Setuptools tracker _______________________________________________ From pjenvey at underboss.org Mon Apr 21 08:24:31 2008 From: pjenvey at underboss.org (Philip Jenvey) Date: Sun, 20 Apr 2008 23:24:31 -0700 Subject: [Distutils] Chicken and egg_info.writers problem with setup.py egg_info Message-ID: <237BBE36-01A1-4F80-A74C-4F51214FF7F3@underboss.org> Say you have custom egg_info.writers defined in a project's setup.py. When you use the egg_info command to restore an egg-info directory that was previously wiped out, the project's egg_info.writers aren't used. Since they aren't registered in your non-existent egg-info directory, they aren't picked up. Is it possible for the egg_info command do some boot strapping to make them work, since is has the entry_points string on hand? -- Philip Jenvey From pje at telecommunity.com Mon Apr 21 16:20:28 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 21 Apr 2008 10:20:28 -0400 Subject: [Distutils] Chicken and egg_info.writers problem with setup.py egg_info In-Reply-To: <237BBE36-01A1-4F80-A74C-4F51214FF7F3@underboss.org> References: <237BBE36-01A1-4F80-A74C-4F51214FF7F3@underboss.org> Message-ID: <20080421142018.0954E3A4070@sparrow.telecommunity.com> At 11:24 PM 4/20/2008 -0700, Philip Jenvey wrote: >Say you have custom egg_info.writers defined in a project's setup.py. >When you use the egg_info command to restore an egg-info directory >that was previously wiped out, the project's egg_info.writers aren't >used. > >Since they aren't registered in your non-existent egg-info directory, >they aren't picked up. Is it possible for the egg_info command do >some boot strapping to make them work, since is has the entry_points >string on hand? Patches welcome, though I personally just run the egg_info command a second time. From pfein at pobox.com Mon Apr 21 21:24:21 2008 From: pfein at pobox.com (Pete) Date: Mon, 21 Apr 2008 15:24:21 -0400 Subject: [Distutils] Setuptools Bug: all files installed +x Message-ID: <2024A24F-7BEA-437B-96F4-AB8A6D61AFC6@pobox.com> On both linux & OS X, Setuptools installs all .py/.pyc files with mode a+x (executable for all users). This occurs regardless of original the permissions in the source tarball. Doing so breaks nosetests, which by default refuses to import executable files for test-discovery purposes as a safety measure. This behavior is broken & dangerous. -- Pete pfein at pobox.com From pje at telecommunity.com Mon Apr 21 21:42:53 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 21 Apr 2008 15:42:53 -0400 Subject: [Distutils] Setuptools Bug: all files installed +x In-Reply-To: <2024A24F-7BEA-437B-96F4-AB8A6D61AFC6@pobox.com> References: <2024A24F-7BEA-437B-96F4-AB8A6D61AFC6@pobox.com> Message-ID: <20080421194238.DD2AB3A4070@sparrow.telecommunity.com> At 03:24 PM 4/21/2008 -0400, Pete wrote: >On both linux & OS X, Setuptools installs all .py/.pyc files with mode >a+x (executable for all users). This occurs regardless of original >the permissions in the source tarball. Doing so breaks nosetests, >which by default refuses to import executable files for test-discovery >purposes as a safety measure. > >This behavior is broken & dangerous. I don't see how it's either one. An explanation would be helpful. Note, by the way, that setuptools is not particularly designed to support running tests against an installed package; I myself have stopped distributing tests in installed packages and require a source installation (e.g. using easy_install --editable) to run tests. This pattern is particularly important for reducing runtime installation requirements, as tests often require additional packages (such as nose itself) in order to run. From asmodai at in-nomine.org Mon Apr 21 22:18:19 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Mon, 21 Apr 2008 22:18:19 +0200 Subject: [Distutils] setuptools on 64 bit Windows broken? Message-ID: <20080421201819.GQ64048@nexus.in-nomine.org> [Apologies if this was discussed in the past, I did some searching but couldn't find much, but then again my attention is spread a little too thing lately.] Right now setuptools on Windows x64 with 64-bits Python 2.5 is (partially) broken. The error shown is: Cannot find Python executable C:\Python25\python.exe when running easy_install. The phrase comes from launcher.c which is compiled via MingW to cli.exe and gui.exe. Both of these are present in the setuptools repository and are 32-bits only for all I can determine (using file). I have not thoroughly checked it yet (due to being written in Unix-C, even though it's aimed at Windows), but my guess is it's using some API calls which fail on 64-bits Windows. I see there's a mingw-64 toolkit on the 'net nowadays. Has anyone tried this yet? Is it indeed an API issue? Furthermore, when creating an egg on Windows x64, it will create it with an identifier of win32, even though C code in a project is compiled with an AMD64 version of the compiler. For all I know 32-bits Windows systems will also pull down this version of the egg when given the chance. I guess this is due to Python 2.5 64 bit reporting win32 for sys.platform? So don't we miss something to properly distinguish between 64 bits Windows and 32 bits? Especially with x64 and Vista 64 on the market nowadays? Thanks for any pointers/clues... -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Whoever undertakes to set himself up as judge in the field of truth and knowledge is shipwrecked by the laughter of the Gods. From pfein at pobox.com Mon Apr 21 22:23:09 2008 From: pfein at pobox.com (Pete) Date: Mon, 21 Apr 2008 16:23:09 -0400 Subject: [Distutils] Setuptools Bug: all files installed +x In-Reply-To: <20080421194238.DD2AB3A4070@sparrow.telecommunity.com> References: <2024A24F-7BEA-437B-96F4-AB8A6D61AFC6@pobox.com> <20080421194238.DD2AB3A4070@sparrow.telecommunity.com> Message-ID: <100418B3-CA0B-45EE-AE0A-F6C25D685C02@pobox.com> On Apr 21, 2008, at 3:42 PM, Phillip J. Eby wrote: > At 03:24 PM 4/21/2008 -0400, Pete wrote: >> On both linux & OS X, Setuptools installs all .py/.pyc files with >> mode >> a+x (executable for all users). This occurs regardless of original >> the permissions in the source tarball. Doing so breaks nosetests, >> which by default refuses to import executable files for test- >> discovery >> purposes as a safety measure. >> >> This behavior is broken & dangerous. > > I don't see how it's either one. An explanation would be helpful. It's broken in that the source tarball includes per-file permissions and setuptools is blindly overriding them. I realize that's simply restating my original complaint, but seeing as setuptools must be *explicitly* changing the permissions on the installed files, perhaps the onus is on you to explain why that's a good idea in the first place. In any event, a motivating example: Some non-script modules are intended to be executable - think doctest, or anything else that does a `if __name__ == __main__:`. As a developer, I purposely set such modules executable (including setting svn:executable) and leave the others as r-w. And there lies the danger. The executable bit is an indication that a file is intended to be executable. Unix-like systems will treat running a file without a leading #! as a shell script. This can cause arbitrary commands to be executed - for example, this is valid python: rm -f /usr Perhaps contrived, but should demonstrate the point. As a more realistic example, `import` is an imagelib command that takes over the X cursor (for taking a screenshot IIRC). > Note, by the way, that setuptools is not particularly designed to > support running tests against an installed package; I myself have > stopped distributing tests in installed packages and require a > source installation (e.g. using easy_install --editable) to run tests. I'm not looking for explicit testing support from setuptools for testing here - I'm just asking that a bug that breaks a 3rd party testing package be fixed. -- Pete pfein at pobox.com From pje at telecommunity.com Tue Apr 22 00:01:39 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 21 Apr 2008 18:01:39 -0400 Subject: [Distutils] Setuptools Bug: all files installed +x In-Reply-To: <100418B3-CA0B-45EE-AE0A-F6C25D685C02@pobox.com> References: <2024A24F-7BEA-437B-96F4-AB8A6D61AFC6@pobox.com> <20080421194238.DD2AB3A4070@sparrow.telecommunity.com> <100418B3-CA0B-45EE-AE0A-F6C25D685C02@pobox.com> Message-ID: <20080421220124.8C9283A4070@sparrow.telecommunity.com> At 04:23 PM 4/21/2008 -0400, Pete wrote: >I'm not looking for explicit testing support from setuptools for >testing here - I'm just asking that a bug that breaks a 3rd party >testing package be fixed. You haven't stated anything yet that sounds like an actual bug to me. What nose is doing seems like the bug to me, in fact, since modules exist "in the wild" that are both module and script. (You yourself gave this as an example.) It certainly might be reasonable... in a posix-only + svn-only + tar-only world... to make the assumption that installed permissions could be based in part on source file permissions. However, setuptools has to support scenarios where none of those environmental conditions apply. (e.g. Windows, a half-dozen revision control systems, and zip files.) That means that existing executable permissions are not -- and as far as I can see, *cannot* -- be used as a cross-platform basis for determining the permissions to give to installed files. Some other mechanism to explicitly specify the permissions would need to exist. From bignose+hates-spam at benfinney.id.au Tue Apr 22 01:02:03 2008 From: bignose+hates-spam at benfinney.id.au (Ben Finney) Date: Tue, 22 Apr 2008 09:02:03 +1000 Subject: [Distutils] Setuptools Bug: all files installed +x References: <2024A24F-7BEA-437B-96F4-AB8A6D61AFC6@pobox.com> <20080421194238.DD2AB3A4070@sparrow.telecommunity.com> Message-ID: <87bq42stg4.fsf@benfinney.id.au> "Phillip J. Eby" writes: > At 03:24 PM 4/21/2008 -0400, Pete wrote: > >On both linux & OS X, Setuptools installs all .py/.pyc files with mode > >a+x (executable for all users). This occurs regardless of original > >the permissions in the source tarball. Doing so breaks nosetests, > >which by default refuses to import executable files for test-discovery > >purposes as a safety measure. > > > >This behavior is broken & dangerous. > > I don't see how it's either one. An explanation would be helpful. Broken: The source tarball has specific permissions set for those files, and Setupttols is breaking that by overriding the permissions. A feature (preserve desired permission mode of the file) is lost. Dangerous: The tests will run fine on the developer's machine, but will skip over the modules entirely when they are installed via Setuptools, because of the above broken behaviour. Test output will indicate no problems, because the permission modes (as incorrectly set by Setuptools) indicate explicitly to the testing tool that those modules should be skipped. This results in tests being omitted silently, which is the danger. (good sigmonster, have a cookie) -- \ ?Ignorance more frequently begets confidence than does | `\ knowledge.? ?Charles Darwin, _The Descent of Man_, 1871 | _o__) | Ben Finney From pfein at pobox.com Tue Apr 22 17:49:05 2008 From: pfein at pobox.com (Pete) Date: Tue, 22 Apr 2008 11:49:05 -0400 Subject: [Distutils] Setuptools Bug: all files installed +x In-Reply-To: <20080421220124.8C9283A4070@sparrow.telecommunity.com> References: <2024A24F-7BEA-437B-96F4-AB8A6D61AFC6@pobox.com> <20080421194238.DD2AB3A4070@sparrow.telecommunity.com> <100418B3-CA0B-45EE-AE0A-F6C25D685C02@pobox.com> <20080421220124.8C9283A4070@sparrow.telecommunity.com> Message-ID: <4D80DB69-BB12-457A-B060-E95793A79CD9@pobox.com> On Apr 21, 2008, at 6:01 PM, Phillip J. Eby wrote: > At 04:23 PM 4/21/2008 -0400, Pete wrote: >> I'm not looking for explicit testing support from setuptools for >> testing here - I'm just asking that a bug that breaks a 3rd party >> testing package be fixed. > > You haven't stated anything yet that sounds like an actual bug to me. What about the dangerous & broken complaint? > What nose is doing seems like the bug to me, in fact, since modules > exist "in the wild" that are both module and script. (You yourself > gave this as an example.) No, it's a safety mechanism. From the nose docs for the --exe flag (which forcibly disables such checks): --exe Look for tests in python modules that are executable. Normal behavior is to exclude executable modules, since they may not be import-safe [NOSE_INCLUDE_EXE] > It certainly might be reasonable... in a posix-only + svn-only + > tar-only world... to make the assumption that installed permissions > could be based in part on source file permissions. However, > setuptools has to support scenarios where none of those > environmental conditions apply. (e.g. Windows, a half-dozen > revision control systems, and zip files.) I fail to see how setting +x on all non-script .py's addresses the environments that don't support permissions. Given the complete lack of permissions information in non-tar archives, it'd arguably be equally valid to set -x (non-executable) on all .py's or to set them at random. What problem does the current behavior solve? If authors need support for permissions, they clearly need to use an archive file format that supports that. If they don't use such a format, they should expect what they get. For those of us using tar/ posix, the fact that setuptools goes out its way to override developer/ distributor intent is problematic. I know what I'm doing, and setuptools doesn't let me do it. > That means that existing executable permissions are not -- and as > far as I can see, *cannot* -- be used as a cross-platform basis for > determining the permissions to give to installed files. Some other > mechanism to explicitly specify the permissions would need to exist. If there's a strong reason for +x'ing non-tar archives, perhaps setuptools could disable this behavior for archive formats that support (ie, tar). -- Pete pfein at pobox.com From pje at telecommunity.com Tue Apr 22 18:19:11 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Apr 2008 12:19:11 -0400 Subject: [Distutils] Setuptools Bug: all files installed +x In-Reply-To: <4D80DB69-BB12-457A-B060-E95793A79CD9@pobox.com> References: <2024A24F-7BEA-437B-96F4-AB8A6D61AFC6@pobox.com> <20080421194238.DD2AB3A4070@sparrow.telecommunity.com> <100418B3-CA0B-45EE-AE0A-F6C25D685C02@pobox.com> <20080421220124.8C9283A4070@sparrow.telecommunity.com> <4D80DB69-BB12-457A-B060-E95793A79CD9@pobox.com> Message-ID: <20080422161916.D33BE3A4070@sparrow.telecommunity.com> At 11:49 AM 4/22/2008 -0400, Pete wrote: >On Apr 21, 2008, at 6:01 PM, Phillip J. Eby wrote: > >>At 04:23 PM 4/21/2008 -0400, Pete wrote: >>>I'm not looking for explicit testing support from setuptools for >>>testing here - I'm just asking that a bug that breaks a 3rd party >>>testing package be fixed. >> >>You haven't stated anything yet that sounds like an actual bug to me. > >What about the dangerous & broken complaint? Which I don't yet understand, let alone agree with. Simply asserting over and over that it's bad and dangerous doesn't help. One thing that you particularly seem to be missing is that the distutils also ignore a Python module's source permissions -- whether they come from a tarball or not. From pfein at pobox.com Tue Apr 22 18:52:33 2008 From: pfein at pobox.com (Pete) Date: Tue, 22 Apr 2008 12:52:33 -0400 Subject: [Distutils] Setuptools Bug: all files installed +x In-Reply-To: <20080422161916.D33BE3A4070@sparrow.telecommunity.com> References: <2024A24F-7BEA-437B-96F4-AB8A6D61AFC6@pobox.com> <20080421194238.DD2AB3A4070@sparrow.telecommunity.com> <100418B3-CA0B-45EE-AE0A-F6C25D685C02@pobox.com> <20080421220124.8C9283A4070@sparrow.telecommunity.com> <4D80DB69-BB12-457A-B060-E95793A79CD9@pobox.com> <20080422161916.D33BE3A4070@sparrow.telecommunity.com> Message-ID: <24C59DA6-6470-4285-8519-4B8E7ABFB79C@pobox.com> On Apr 22, 2008, at 12:19 PM, Phillip J. Eby wrote: > At 11:49 AM 4/22/2008 -0400, Pete wrote: >> On Apr 21, 2008, at 6:01 PM, Phillip J. Eby wrote: >> >>> At 04:23 PM 4/21/2008 -0400, Pete wrote: >>>> I'm not looking for explicit testing support from setuptools for >>>> testing here - I'm just asking that a bug that breaks a 3rd party >>>> testing package be fixed. >>> >>> You haven't stated anything yet that sounds like an actual bug to >>> me. >> >> What about the dangerous & broken complaint? > > Which I don't yet understand, let alone agree with. Simply > asserting over and over that it's bad and dangerous doesn't help. This bit, from my email on April 21, 2008 4:23:09; Ben Finney's point about tests being silently skipped is also valid, and was how I originally came across this problem. In any event, a motivating example: Some non-script modules are intended to be executable - think doctest, or anything else that does a `if __name__ == __main__:`. As a developer, I purposely set such modules executable (including setting svn:executable) and leave the others as r-w. And there lies the danger. The executable bit is an indication that a file is intended to be executable. Unix-like systems will treat running a file without a leading #! as a shell script. This can cause arbitrary commands to be executed - for example, this is valid python: rm -f /usr Perhaps contrived, but should demonstrate the point. As a more realistic example, `import` is an imagelib command that takes over the X cursor (for taking a screenshot IIRC). > One thing that you particularly seem to be missing is that the > distutils also ignore a Python module's source permissions -- > whether they come from a tarball or not. Ok, but AFAIK distutils doesn't then +x everything, which is the problem here. -- Pete pfein at pobox.com From klaus_zimmermann at gmx.de Tue Apr 22 19:11:59 2008 From: klaus_zimmermann at gmx.de (Klaus Zimmermann) Date: Tue, 22 Apr 2008 19:11:59 +0200 Subject: [Distutils] Setuptools Bug: all files installed +x In-Reply-To: <20080421220124.8C9283A4070@sparrow.telecommunity.com> References: <2024A24F-7BEA-437B-96F4-AB8A6D61AFC6@pobox.com> <20080421194238.DD2AB3A4070@sparrow.telecommunity.com> <100418B3-CA0B-45EE-AE0A-F6C25D685C02@pobox.com> <20080421220124.8C9283A4070@sparrow.telecommunity.com> Message-ID: <1208884319.3686.3.camel@localhost> Am Montag, den 21.04.2008, 18:01 -0400 schrieb Phillip J. Eby: > It certainly might be reasonable... in a posix-only + svn-only + > tar-only world... to make the assumption that installed permissions > could be based in part on source file permissions. However, > setuptools has to support scenarios where none of those environmental > conditions apply. (e.g. Windows, a half-dozen revision control > systems, and zip files.) Neither Windows nor zip files support permissions at all, correct? Thus neither one is a reason to mess with the permissions, right? As for the revision control systems: I'd like to learn about those that don't support permissions, because I am not aware of any and I'd like to avoid them in the future. Thanks for your help in this regard, Phillip! Klaus From mhearne at usgs.gov Tue Apr 22 20:34:45 2008 From: mhearne at usgs.gov (Michael Hearne) Date: Tue, 22 Apr 2008 12:34:45 -0600 Subject: [Distutils] easy_install error Message-ID: I just tried to install the egg for Pyx, and got the error pasted below. I've contacted the mailing list for that project, and they seem to be unsure of where they can put shared data for their egg. The error message below says that I should contact the maintainers of the package (which I have done) and the EasyInstall maintainers for a workaround. So, is there a workaround for this? Thanks, Mike Hearne Running PyX-0.10/setup.py -q bdist_egg --dist-dir /tmp/easy_install- _D_bmc/PyX-0.10/egg-dist-tmp-r1bHsH error: Setup script exited with error: SandboxViolation: mkdir('/usr/ local/share/pyx',) {} The package setup script has attempted to modify files on your system that are not within the EasyInstall build area, and has been aborted. This package cannot be safely installed by EasyInstall, and may not support alternate installation locations even if you run its setup script by hand. Please inform the package's author and the EasyInstall maintainers to find out if a fix or workaround is available. ------------------------------------------------------ Michael Hearne mhearne at usgs.gov (303) 273-8620 USGS National Earthquake Information Center 1711 Illinois St. Golden CO 80401 Senior Software Engineer Synergetics, Inc. ------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Tue Apr 22 21:47:45 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Apr 2008 15:47:45 -0400 Subject: [Distutils] easy_install error In-Reply-To: References: Message-ID: <20080422194731.4C42D3A40D8@sparrow.telecommunity.com> At 12:34 PM 4/22/2008 -0600, Michael Hearne wrote: >I just tried to install the egg for Pyx, and got the error pasted >below. I've contacted the mailing list for that project, and they >seem to be unsure of where they can put shared data for their egg. Eggs do not have shared data -- they only have runtime-accessible data, because they're not a distribution format in the way that system packages are. However, the problem in this case is that PyX isn't using any of the standard distutils ways to install these files. If they were using the standard install_data command, the data would at least be shipped inside the egg. If they were using the package_data facility, however (which available in Python 2.4 and up), they would be able to include the files inside their package (and thus the egg) and be able to access them using either __file__-relative paths or the pkg_resources API. Any of these approaches would work, and unless they need to support Python 2.3, would probably be easier to use than their current method of creating a siteconfig.py file. Their setup.py would then not need to include any custom distutils commands, as it does now. (And such a change would not require any reliance on setuptools, either.) From cliff at develix.com Tue Apr 22 21:56:48 2008 From: cliff at develix.com (Cliff Wells) Date: Tue, 22 Apr 2008 12:56:48 -0700 Subject: [Distutils] Setuptools Bug: all files installed +x In-Reply-To: <20080422161916.D33BE3A4070@sparrow.telecommunity.com> References: <2024A24F-7BEA-437B-96F4-AB8A6D61AFC6@pobox.com> <20080421194238.DD2AB3A4070@sparrow.telecommunity.com> <100418B3-CA0B-45EE-AE0A-F6C25D685C02@pobox.com> <20080421220124.8C9283A4070@sparrow.telecommunity.com> <4D80DB69-BB12-457A-B060-E95793A79CD9@pobox.com> <20080422161916.D33BE3A4070@sparrow.telecommunity.com> Message-ID: <1208894208.15868.186.camel@portableevil.develix.com> On Tue, 2008-04-22 at 12:19 -0400, Phillip J. Eby wrote: > At 11:49 AM 4/22/2008 -0400, Pete wrote: > >On Apr 21, 2008, at 6:01 PM, Phillip J. Eby wrote: > > > >>At 04:23 PM 4/21/2008 -0400, Pete wrote: > >>>I'm not looking for explicit testing support from setuptools for > >>>testing here - I'm just asking that a bug that breaks a 3rd party > >>>testing package be fixed. > >> > >>You haven't stated anything yet that sounds like an actual bug to me. > > > >What about the dangerous & broken complaint? > > Which I don't yet understand, let alone agree with. Simply asserting > over and over that it's bad and dangerous doesn't help. I thought his explanation was rather clear. Here's a valid Python file that will make you unhappy if you execute it from the shell: rm = 0 # shell error, no effect rf = 1 # another shell error home = 8 # etc... result=`rm -rf /home` In Python you get '0', in Bash you get a wiped out home directory. Yes, it's quite contrived. But his point about "import" also being a typical command on *nix (runs an Imagemagick binary) points out that there's probably even more concerns that we aren't thinking of. This is especially a concern because shell scripts don't abort on error (so you could conceivably make it quite a ways through a Python file until you hit some random name that happens to be a valid Unix command). It's probably unlikely that someone would accidentally run one of these modules inadvertently. But "highly unlikely" is also not "secure". I've certainly run the wrong thing on accident because Bash's tab-completion caught me off-guard. He's also right in that arbitrarily setting the execute bit apparently serves no explainable purpose (otherwise I assume you'd have provided an explanation by now), so the onus of explaining why this is a desirable behavior comes back to setuptools. > One thing that you particularly seem to be missing is that the > distutils also ignore a Python module's source permissions -- whether > they come from a tarball or not. I'm not sure why any file in site-packages would need the execute bit set. Executable scripts are (or should be) installed elsewhere. If the bit *must* be set one way or the other, then it should be turned off by default. Regards, Cliff From mhearne at usgs.gov Wed Apr 23 00:27:03 2008 From: mhearne at usgs.gov (Michael Hearne) Date: Tue, 22 Apr 2008 16:27:03 -0600 Subject: [Distutils] questions on setuptools Message-ID: <3B886397-6371-4705-A23D-BF146B9B469F@usgs.gov> I'm creating a software package for in-house consumption, and will be using setuptools to create an egg that I can easily install, upgrade, etc. I'm able to answer most of my own questions from trolling through the documentation, but I have two that I can't find the answers for: 1) If I depend on packages that are not in PyPI and are not available through some other simple means (svn, simple .py, etc.), can I note these dependencies in such a way that easy_install will complain if they are not present and _stop_ installation? 2) If I already have a "main" script, is it possible to have that installed in /usr/local/bin (assuming my install path is "/usr/ local")? If so, how? I read this section (http://peak.telecommunity.com/DevCenter/setuptools#automatic-script-creation ) but it doesn't seem to address my issue. Thanks, Mike ------------------------------------------------------ Michael Hearne mhearne at usgs.gov (303) 273-8620 USGS National Earthquake Information Center 1711 Illinois St. Golden CO 80401 Senior Software Engineer Synergetics, Inc. ------------------------------------------------------ From pje at telecommunity.com Wed Apr 23 01:40:13 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 22 Apr 2008 19:40:13 -0400 Subject: [Distutils] questions on setuptools In-Reply-To: <3B886397-6371-4705-A23D-BF146B9B469F@usgs.gov> References: <3B886397-6371-4705-A23D-BF146B9B469F@usgs.gov> Message-ID: <20080422234000.545873A4070@sparrow.telecommunity.com> At 04:27 PM 4/22/2008 -0600, Michael Hearne wrote: >I'm creating a software package for in-house consumption, and will be >using setuptools to create an egg that I can easily install, upgrade, >etc. > >I'm able to answer most of my own questions from trolling through the >documentation, but I have two that I can't find the answers for: > >1) If I depend on packages that are not in PyPI and are not available >through some other simple means (svn, simple .py, etc.), can I note >these dependencies in such a way that easy_install will complain if >they are not present and _stop_ installation? No. After all, if the package isn't available in such a way that easy_install can install it, then there is also no way for easy_install to detect its presence or absence, anyway. >2) If I already have a "main" script, is it possible to have that >installed in /usr/local/bin (assuming my install path is "/usr/ >local")? If so, how? I read this section >(http://peak.telecommunity.com/DevCenter/setuptools#automatic-script-creation >) but it doesn't seem to address my issue. Just use the regular distutils 'scripts' option. From pje at telecommunity.com Wed Apr 23 21:41:31 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 23 Apr 2008 15:41:31 -0400 Subject: [Distutils] questions on setuptools In-Reply-To: <410E2F7E-B67C-4297-B1AE-351D2B6E755B@usgs.gov> References: <3B886397-6371-4705-A23D-BF146B9B469F@usgs.gov> <20080422234000.545873A4070@sparrow.telecommunity.com> <410E2F7E-B67C-4297-B1AE-351D2B6E755B@usgs.gov> Message-ID: <20080423194117.DEFD13A40AC@sparrow.telecommunity.com> At 01:01 PM 4/23/2008 -0600, Michael Hearne wrote: >Phillip - Thanks for the quick response. One feature suggestion for >the future: > > From my perhaps naive perspective, it seems like it would be easy to >add non-PyPI dependencies that the installer could complain about. >For example, if the setup() function accepted a keyword list that >contained just the name of the package (whatever you use to import >it), and then have the installer do something like this: > >deps = ['foo','bar'] (this is the keyword list passed to the setup >function) >notfound = [] >for dep in deps: > try: > __import__(dep) > except: > notfound.append(dep) > if notfound: > print 'The following required packages are not > installed. Exiting.' >% (str(notfound)) > sys.exit(1) I don't think this is a significant enough improvement over the package aborting at runtime, to add the complexity. Plus, it removes the incentive for the depender to lobby the dependee to make their package easy_install-able. :) (And, it's actually more complex if you're trying to do version detection as well as presence detection. There's some old code in setuptools.depends that tries to do this sort of thing, but I abandoned that approach many years ago.) From jeff at drinktomi.com Thu Apr 24 01:43:17 2008 From: jeff at drinktomi.com (Jeff Younker) Date: Wed, 23 Apr 2008 16:43:17 -0700 Subject: [Distutils] Simple idea to resolve versioning problems In-Reply-To: <48057BDE.1080802@canterbury.ac.nz> References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <20080414101003.GB27572@phare.normalesup.org> <4803374A.6020106@ar.media.kyoto-u.ac.jp> <4803FE55.2000805@canterbury.ac.nz> <48040D2C.4010506@ar.media.kyoto-u.ac.jp> <48052A65.7070702@canterbury.ac.nz> <48053D8C.5000101@ar.media.kyoto-u.ac.jp> <48057BDE.1080802@canterbury.ac.nz> Message-ID: <2D496882-78DA-4DDD-871B-7D5A2E473D5A@drinktomi.com> On Apr 15, 2008, at 9:09 PM, Greg Ewing wrote: > David Cournapeau wrote: >> Greg Ewing wrote: >> >>> If they really do need different versions, this is insoluble. >> >> But that's by far the most significant problem of packages with a >> lot of >> dependencies ! > > But if your application really does depend on two libraries that > have conflicting requirements like this, the application itself > is screwed to begin with. There's *no* way of making it work > on *any* system, whether that system has library versioning or > not. > > Consequently, the developer will be unable to make it work on > his own machine, and will therefore never attempt to deploy it! That's good and fine for the situation where one application is being deployed on a Python installation, but that isn't a very realistic situation. There is usually more than one application/script/whatever running off of a given Python installation. Here's an example of how you can get into an unworkable situation: Application A depends on: library B v 1.1 library C v 2.3 Library B v 1.1 has no dependencies. Now application D comes along. It depends upon: library B v 1.4 But library B v 1.4 depends upon library C v 2.6. Application A *breaks* with library C v 2.6. Which does the user give up? Application A (their storage management client) or application D (their monitoring agent)? This sort of stuff happens all the time. This is what a Versioning (with a capital V attempts to solve.) > That's because the program would say something like "Please > give me gtk-2.3", and the system would know that anything > called gtk-2.x with x >= 3 is compatible with that, and would > pick the one with the greatest x out of those available. Thats a great theory, but that's not how the real world works. Python packages are an ecology where there will be inconsistencies between different minor versions of the same package. A legitimate bug fix may break behavior that other packages depend upon while correcting the behavior for others. That's just the way stuff happens. We can encourage 10,000 disparate developers to adhere to best practices, but it's fantasy to imagine that they will. Most of the people putting together packages are donating their 80% solutions to the rest of the world. They will develop these packages over time. The packages will be changed to suit their *own* needs as time goes on. Those changes will be released back to the public eventually (we hope). Some of those changes are going to break compatibility with the old version for some reason or another. Perhaps 100% backwards compatibility either wasn't important to the author. Perhaps because they didn't have the time to do exhaustive backwards compatibility testing. Why didn't they expend the effort? Because she's working on her thesis or maybe because his wife just had triplets. Perhaps people came to depend on functionality that the author didn't ever expect them to use, or never even anticipated, and has never even heard' of. What we have to figure out his how to allow *those* packages to work with *these* applications no matter what. >> IOW, enabling version requirement without strong API compatibility is >> the path to dependency hell. > > I fully agree. However, this is a social issue, not a > technical one. Everyone using the versioning system would > need to understand the rules and be willing to follow them. What is needed is a Real Versioning System. A system in which a library or application consists of: 1) files 2) a dependency link map When the system loads the package it links in the dependent modules. Then you can get a system where: Application A imports B v 1.1 and C v 2.3 and Application D imports B v 1.4 and C v 2.6. At least that's the holy grail as I see it. -jeff From greg.ewing at canterbury.ac.nz Thu Apr 24 02:24:03 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 24 Apr 2008 12:24:03 +1200 Subject: [Distutils] Simple idea to resolve versioning problems In-Reply-To: <2D496882-78DA-4DDD-871B-7D5A2E473D5A@drinktomi.com> References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <20080414101003.GB27572@phare.normalesup.org> <4803374A.6020106@ar.media.kyoto-u.ac.jp> <4803FE55.2000805@canterbury.ac.nz> <48040D2C.4010506@ar.media.kyoto-u.ac.jp> <48052A65.7070702@canterbury.ac.nz> <48053D8C.5000101@ar.media.kyoto-u.ac.jp> <48057BDE.1080802@canterbury.ac.nz> <2D496882-78DA-4DDD-871B-7D5A2E473D5A@drinktomi.com> Message-ID: <480FD323.1030201@canterbury.ac.nz> Jeff Younker wrote: > Thats a great theory, but that's not how the real world works. Python > packages are an ecology where there will be inconsistencies between > different minor versions of the same package. I'm not sure what you're arguing here. If you're saying that having a version management system won't make these sorts of problems go away, then I fully agree. But that's no reason not to have a verion management system. If you have one, then at least it provides a clear set of rules for people to try to adhere to, and helps show up when they aren't being adhered to, so that the problem can be fixed. > A legitimate bug fix > may break behavior that other packages depend upon while correcting > the behavior for others. If it does that, then it will break other packages anyway, whether a versioning system is being used or not. So I would say it's introduced another bug which in turn needs to be fixed. The versioning system is not the cause of this and can't be blamed for it. > What is needed is a Real Versioning System. A system in which a library > or application consists of: > > 1) files > 2) a dependency link map That sound like a dependency management system to me, not a version management system. They're different things, although dependency management can benefit from having a clear way of labelling versions. > When the system loads the package it links in the dependent modules. Then > you can get a system where: > > Application A imports B v 1.1 and C v 2.3 > > and > > Application D imports B v 1.4 and C v 2.6. Ummm... how is that any different from what I suggested? If in the Python context "loads" means "uses an import statement". -- Greg From david at ar.media.kyoto-u.ac.jp Thu Apr 24 04:46:31 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 24 Apr 2008 11:46:31 +0900 Subject: [Distutils] Simple idea to resolve versioning problems In-Reply-To: <2D496882-78DA-4DDD-871B-7D5A2E473D5A@drinktomi.com> References: <1208028604.3212.193.camel@portableevil.develix.com> <20080412215326.B6CB03A406A@sparrow.telecommunity.com> <1208039212.3212.214.camel@portableevil.develix.com> <20080412225006.GN8360@phare.normalesup.org> <23BDD366-BB0D-488E-B9CE-3C4A55573E09@python.org> <480164B4.5080901@pangalactic.us> <1006AECE-3273-418A-A804-228177D4DCBB@python.org> <480223E9.5040902@pangalactic.us> <20080413152925.GF24381@phare.normalesup.org> <4802B3A6.5020606@canterbury.ac.nz> <20080414101003.GB27572@phare.normalesup.org> <4803374A.6020106@ar.media.kyoto-u.ac.jp> <4803FE55.2000805@canterbury.ac.nz> <48040D2C.4010506@ar.media.kyoto-u.ac.jp> <48052A65.7070702@canterbury.ac.nz> <48053D8C.5000101@ar.media.kyoto-u.ac.jp> <48057BDE.1080802@canterbury.ac.nz> <2D496882-78DA-4DDD-871B-7D5A2E473D5A@drinktomi.com> Message-ID: <480FF487.9000106@ar.media.kyoto-u.ac.jp> Jeff Younker wrote: > > That's good and fine for the situation where one application is > being deployed on a Python installation, but that isn't a very realistic > situation. There is usually more than one application/script/whatever > running off of a given Python installation. > > Here's an example of how you can get into an unworkable situation: > > Application A depends on: > library B v 1.1 > library C v 2.3 > > Library B v 1.1 has no dependencies. > > Now application D comes along. It depends upon: > library B v 1.4 > > But library B v 1.4 depends upon library C v 2.6. > > Application A *breaks* with library C v 2.6. > > Which does the user give up? Application A (their > storage management client) or application D (their > monitoring agent)? > > This sort of stuff happens all the time. > > This is what a Versioning (with a capital V attempts > to solve.) No, versioning does not solve this at all, this has nothing to do with versioning. Dependency management does, and the two issues are mostly orthogonal. Installing multiple versions of the same package with versioning as an attempt to solve dependencies is broken. It has been used before, without any success. > Thats a great theory, but that's not how the real world works. Python > packages are an ecology where there will be inconsistencies between > different minor versions of the same package. A legitimate bug fix > may break behavior that other packages depend upon while correcting > the behavior for others. That's just the way stuff happens. > > We can encourage 10,000 disparate developers to adhere to best practices, > but it's fantasy to imagine that they will. Most of the people > putting together > packages are donating their 80% solutions to the rest of the world. But most packages won't depend one on each other; most of the 80 % packages you are talking about are not meant to be used by other. Thinking versioning + side by side installation will solve the problem of breaking API is totally wrong. It just does not work. Please take a look at this page I mentioned before: http://www.mono-project.com/Assemblies_and_the_GAC It explains the problem very well, much better that I will ever be able to. Look at the paragraph "What should be installer in the GAC", the GAC being the system used in the .net ecosystem to maintain multiple version side by side and provide a system to serve the right library (assembly in .net terms) at the right application. Again, having a few packages with several versions is Ok. But enabling this as the general rule is path to hell from a maintenance/support point of view. cheers, David From info at zopyx.com Thu Apr 24 17:40:51 2008 From: info at zopyx.com (Andreas Jung) Date: Thu, 24 Apr 2008 17:40:51 +0200 Subject: [Distutils] Providing binary packages for Python versions with different internal unicode representation Message-ID: Hi, Python interpreters can be compiled by different internal unicode representations (UCS2 vs. UCS4). Extension modules using unicode methods of Python internally can not be used with a Python interpreter with a different internal unicode representation. What's the deal when you want to provide binary packages of an extension module for Python interpreters with different unicode representations....is there some support for doing this using setuptools? Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available URL: From pje at telecommunity.com Thu Apr 24 18:20:14 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 24 Apr 2008 12:20:14 -0400 Subject: [Distutils] Providing binary packages for Python versions with different internal unicode representation In-Reply-To: References: Message-ID: <20080424161959.6BD5F3A40AC@sparrow.telecommunity.com> At 05:40 PM 4/24/2008 +0200, Andreas Jung wrote: >Hi, > >Python interpreters can be compiled by different internal unicode >representations (UCS2 vs. UCS4). Extension modules using unicode >methods of Python internally can not be used with a Python >interpreter with a different >internal unicode representation. What's the deal when you want to provide >binary packages of an extension module for Python interpreters with >different unicode representations....is there some support for doing this >using setuptools? Nope. I've posted before, asking about trying to resolve this and other issues with the distutils "platform" string format, but there's never been anybody with sufficient interest+knowledge to really do anything about it for distutils or setuptools. From tseaver at palladion.com Thu Apr 24 18:42:53 2008 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 24 Apr 2008 12:42:53 -0400 Subject: [Distutils] Providing binary packages for Python versions with different internal unicode representation In-Reply-To: References: Message-ID: <4810B88D.5080902@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Andreas Jung wrote: > Python interpreters can be compiled by different internal unicode > representations (UCS2 vs. UCS4). Extension modules using unicode methods of > Python internally can not be used with a Python interpreter with a different > internal unicode representation. What's the deal when you want to provide > binary packages of an extension module for Python interpreters with > different unicode representations....is there some support for doing this > using setuptools? Nope. A better strategy is not to provide binaries at all. Instead, provide only 'sdist' distributions, and let people build compatible extensions themselves. For tool-deficient Windows users, provide a binary built using the "default" Python interpreter for that release. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIELiN+gerLs4ltQ4RAiRoAKCh86yd1VmErKiKXTdFPXSlnhStOwCgskNi baiiT0Qdv/1Tu63F3tLt33M= =C0Y4 -----END PGP SIGNATURE----- From lists at informa.tiker.net Thu Apr 24 18:42:55 2008 From: lists at informa.tiker.net (Andreas =?utf-8?q?Kl=C3=B6ckner?=) Date: Thu, 24 Apr 2008 12:42:55 -0400 Subject: [Distutils] setuptools "install" writes to build directory Message-ID: <200804241242.57019.lists@informa.tiker.net> Hi there, when you install a setuptools-equipped package with python setup.py build sudo python setup.py install the "install" stage writes to the build directory, as root. This is not great behavior, and will fail on NFS-mounted build roots. Can something be done to avoid this? Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From pje at telecommunity.com Thu Apr 24 19:49:04 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 24 Apr 2008 13:49:04 -0400 Subject: [Distutils] setuptools "install" writes to build directory In-Reply-To: <200804241242.57019.lists@informa.tiker.net> References: <200804241242.57019.lists@informa.tiker.net> Message-ID: <20080424174917.475E83A40AC@sparrow.telecommunity.com> At 12:42 PM 4/24/2008 -0400, Andreas Kl??ckner wrote: >Hi there, > >when you install a setuptools-equipped package with > >python setup.py build >sudo python setup.py install > >the "install" stage writes to the build directory, as root. This is not great >behavior, and will fail on NFS-mounted build roots. > >Can something be done to avoid this? With the normal distutils, you can use the --skip-build option to install. For setuptools, this also works, but only if you're using --single-version-externally-managed and/or --root (i.e., the distutils-compatible mode). If you're not using the distutils-compatible mode, then do this: python setup.py bdist_egg sudo python setup.py easy_install dist/projectname-version-whatever.egg To separate the build and install steps. From lists at informa.tiker.net Thu Apr 24 20:10:02 2008 From: lists at informa.tiker.net (Andreas =?iso-8859-1?q?Kl=F6ckner?=) Date: Thu, 24 Apr 2008 14:10:02 -0400 Subject: [Distutils] setuptools "install" writes to build directory In-Reply-To: <20080424174917.475E83A40AC@sparrow.telecommunity.com> References: <200804241242.57019.lists@informa.tiker.net> <20080424174917.475E83A40AC@sparrow.telecommunity.com> Message-ID: <200804241410.03639.lists@informa.tiker.net> On Donnerstag 24 April 2008, Phillip J. Eby wrote: > If you're not using the distutils-compatible mode, then do this: > > python setup.py bdist_egg > sudo python setup.py easy_install dist/projectname-version-whatever.egg > > To separate the build and install steps. Thank you! :) Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From lists at informa.tiker.net Thu Apr 24 20:04:38 2008 From: lists at informa.tiker.net (Andreas =?utf-8?q?Kl=C3=B6ckner?=) Date: Thu, 24 Apr 2008 14:04:38 -0400 Subject: [Distutils] Fwd: Re: [Numpy-discussion] numpy as a setuptools dependency Message-ID: <200804241404.40485.lists@informa.tiker.net> Ok, forwarding this to distutils-sig. :) Andreas -------------- next part -------------- An embedded message was scrubbed... From: "Brian Granger" Subject: Re: [Numpy-discussion] numpy as a setuptools dependency Date: Thu, 24 Apr 2008 11:30:42 -0600 Size: 3881 URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From tseaver at palladion.com Thu Apr 24 20:01:03 2008 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 24 Apr 2008 14:01:03 -0400 Subject: [Distutils] setuptools "install" writes to build directory In-Reply-To: <200804241242.57019.lists@informa.tiker.net> References: <200804241242.57019.lists@informa.tiker.net> Message-ID: <4810CADF.8070606@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Andreas Kl?ckner wrote: > Hi there, > > when you install a setuptools-equipped package with > > python setup.py build > sudo python setup.py install > > the "install" stage writes to the build directory, as root. This is not great > behavior, and will fail on NFS-mounted build roots. > > Can something be done to avoid this? Something like: /path/to/python setup.py install_lib --skip-build works. I don't know why '--skip-build' doesn't keep the 'install' command from running 'build_py'. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIEMrf+gerLs4ltQ4RAkfXAJ9rUV+w/BXI8nvGpc7MGw1y5yY5EgCgsW4J 700ubjkHN9lztWB3yr/vsVw= =alvY -----END PGP SIGNATURE----- From mustapha at headnet.dk Fri Apr 25 11:06:39 2008 From: mustapha at headnet.dk (mustapha) Date: Fri, 25 Apr 2008 11:06:39 +0200 Subject: [Distutils] Can't generate console script with buildout - even when using zc.recipe.egg In-Reply-To: <195B1D75-B453-4B1E-9485-354FFBDB27BB@zope.com> References: <195B1D75-B453-4B1E-9485-354FFBDB27BB@zope.com> Message-ID: <48119F1F.7000901@headnet.dk> Hi Jim, I wrote a little patch to add support for old style scripts. I don't know if it is a good solution (maybe I missed something) Can you please review it and help me to get it done if I misunderstood. Many thanks, /Mustapha Jim Fulton wrote: > On Apr 1, 2008, at 11:07 AM, Anton Stonor wrote: >> Hi there, >> >> I was hoping a buildout guru could give a hand on this one: >> >> The "RelStorage" package comes as an egg and installing it in a >> buildout >> environment works fine. >> >> However I can't find a way to let buildout generate the zodbconvert >> console script that comes with RelStorage. >> >> easy_install RelStorage adds the script fine. >> >> I know you should use zc.recipe.egg in this case, but no result. >> >> The RelStorage setup.py comes with this: >> >> scripts=['scripts/zodbconvert.py'] >> >> but no "console_scripts" entry points (maybe that is needed?) > > Yes, buildout needs that. buildout only installs scripts declared as > entry points. As you discovered, you can specify the entry points > yourself. Buildout also needs the entry point to be importable from > the egg. Unfortunately, old-style distutils script handling doesn't > put the scripts in a place where they are importable. > > >> >> I found a kind-of workaround: >> >> entry-points = zodbconvert=zodbconvert:main >> extra-paths = >> /home/stonor/projects/mybuildout/eggs/RelStorage-1.0.1-py2.4.egg/EGG- >> INFO/scripts >> >> But it is annoying to hard code the path to >> RelStorage-1.0.1-py2.4.egg/EGG-INFO/scripts. >> >> Is there a way to dynamically look up the path to an egg with bulidout >> or a another way to solve my issue? > > No, because the scripts aren't included in the RelStorage Python > package and aren't in the egg path. You should get Shane to move this > script into a module that can be imported from the egg. This is what I > did with the ZODB scripts a while back. > > Perhaps buildout should support the traditional script-installation > mechanism, but that mechanism assumes that the modules needed by the > script are installed into site packages and that's not a model that > buildout provides. Virtualenv supports this use case because it > emulates a Python install. > > Jim > > -- > Jim Fulton > Zope Corporation > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- A non-text attachment was scrubbed... Name: patch.diff Type: text/x-diff Size: 2403 bytes Desc: not available URL: From mustapha at headnet.dk Fri Apr 25 11:06:39 2008 From: mustapha at headnet.dk (mustapha) Date: Fri, 25 Apr 2008 11:06:39 +0200 Subject: [Distutils] Can't generate console script with buildout - even when using zc.recipe.egg In-Reply-To: <195B1D75-B453-4B1E-9485-354FFBDB27BB@zope.com> References: <195B1D75-B453-4B1E-9485-354FFBDB27BB@zope.com> Message-ID: <48119F1F.7000901@headnet.dk> Hi Jim, I wrote a little patch to add support for old style scripts. I don't know if it is a good solution (maybe I missed something) Can you please review it and help me to get it done if I misunderstood. Many thanks, /Mustapha Jim Fulton wrote: > On Apr 1, 2008, at 11:07 AM, Anton Stonor wrote: >> Hi there, >> >> I was hoping a buildout guru could give a hand on this one: >> >> The "RelStorage" package comes as an egg and installing it in a >> buildout >> environment works fine. >> >> However I can't find a way to let buildout generate the zodbconvert >> console script that comes with RelStorage. >> >> easy_install RelStorage adds the script fine. >> >> I know you should use zc.recipe.egg in this case, but no result. >> >> The RelStorage setup.py comes with this: >> >> scripts=['scripts/zodbconvert.py'] >> >> but no "console_scripts" entry points (maybe that is needed?) > > Yes, buildout needs that. buildout only installs scripts declared as > entry points. As you discovered, you can specify the entry points > yourself. Buildout also needs the entry point to be importable from > the egg. Unfortunately, old-style distutils script handling doesn't > put the scripts in a place where they are importable. > > >> >> I found a kind-of workaround: >> >> entry-points = zodbconvert=zodbconvert:main >> extra-paths = >> /home/stonor/projects/mybuildout/eggs/RelStorage-1.0.1-py2.4.egg/EGG- >> INFO/scripts >> >> But it is annoying to hard code the path to >> RelStorage-1.0.1-py2.4.egg/EGG-INFO/scripts. >> >> Is there a way to dynamically look up the path to an egg with bulidout >> or a another way to solve my issue? > > No, because the scripts aren't included in the RelStorage Python > package and aren't in the egg path. You should get Shane to move this > script into a module that can be imported from the egg. This is what I > did with the ZODB scripts a while back. > > Perhaps buildout should support the traditional script-installation > mechanism, but that mechanism assumes that the modules needed by the > script are installed into site packages and that's not a model that > buildout provides. Virtualenv supports this use case because it > emulates a Python install. > > Jim > > -- > Jim Fulton > Zope Corporation > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- A non-text attachment was scrubbed... Name: patch.diff Type: text/x-diff Size: 2403 bytes Desc: not available URL: From strawman at astraw.com Sat Apr 26 12:49:34 2008 From: strawman at astraw.com (Andrew Straw) Date: Sat, 26 Apr 2008 03:49:34 -0700 Subject: [Distutils] [ANN] stdeb 0.2 Message-ID: <481308BE.60108@astraw.com> I would like to announce the release of stdeb 0.2. = What is it? = stdeb http://stdeb.python-hosting.com/ ("setuptools debian") produces Debian source packages from Python packages via a new distutils command, sdist_dsc, which produces a Debian source package of a Python package. Automatic defaults are provided for the Debian package, but many aspects of the resulting package can be customized via a configuration file. = What's new? = This is a primarily a bug-fix release cleaning up after the alpha series which introduced many new features. Here is the detailed changelog since the last alpha release. 008-03-27 Add ability to pass environment variables to setup.py script. 2008-03-18 Do not allow '.' in source package names. 2008-01-20 Allows a user to build every dependency a package has stated on it's setup.py, recursively. 2007-10-29 Allow upstream tarball to have different name from debian .orig.tar.gz but keep md5sum. 2007-05-28 Fix bug where python distribution name contained '-' but setuptools renamed this to '_'. 2007-05-11 Fix py2dsc script to properly set __file__ and __name__. 2007-04-18 Fix bug where .egg-info renaming failed when upstream version contained '-'. = Thanks = A special thanks to Pedro Algarvio, aka s0undt3ch, for helping with this release. From superdupont at gmail.com Sat Apr 26 16:46:48 2008 From: superdupont at gmail.com (Carlos Valiente) Date: Sat, 26 Apr 2008 15:46:48 +0100 Subject: [Distutils] PATCH: Multiple test suites Message-ID: This patch adds support for running more than one test suite, as in: $ ./setup.py test -s foo.tests.suite,bar.tests.suite Enjoy, C -------------- next part -------------- A non-text attachment was scrubbed... Name: patch-multiple-test-suites.diff Type: application/octet-stream Size: 4859 bytes Desc: not available URL: From pfein at pobox.com Sat Apr 26 17:55:57 2008 From: pfein at pobox.com (Peter Fein) Date: Sat, 26 Apr 2008 11:55:57 -0400 Subject: [Distutils] Ignore ~/.pydistutils.cfg ? Message-ID: is there a way to tell distutils/setuptools to ignore ~/.pydistutils.cfg? I use a config to keep easy_install'd packages in my home directory but doing so confuses MacPorts; they're apparently passing --exec/-- exec-prefix, but it seems to be getting overriden by my config. See: http://trac.macports.org/projects/macports/ticket/9831#comment :4 From ziade.tarek at gmail.com Sat Apr 26 18:24:40 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 26 Apr 2008 18:24:40 +0200 Subject: [Distutils] Ignore ~/.pydistutils.cfg ? In-Reply-To: References: Message-ID: <94bdd2610804260924r19ad9eabv68668619808fccd5@mail.gmail.com> Hi Peter, not yet, but some work is being done at this time: http://bugs.python.org/issue1180 Regards On Sat, Apr 26, 2008 at 5:55 PM, Peter Fein wrote: > is there a way to tell distutils/setuptools to ignore ~/.pydistutils.cfg? > > I use a config to keep easy_install'd packages in my home directory but > doing so confuses MacPorts; they're apparently passing --exec/--exec-prefix, > but it seems to be getting overriden by my config. See: > http://trac.macports.org/projects/macports/ticket/9831#comment:4 > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From exarkun at divmod.com Sat Apr 26 18:46:10 2008 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Sat, 26 Apr 2008 12:46:10 -0400 Subject: [Distutils] [ANN] stdeb 0.2 In-Reply-To: <481308BE.60108@astraw.com> Message-ID: <20080426164610.6859.110254766.divmod.quotient.35549@ohm> On Sat, 26 Apr 2008 03:49:34 -0700, Andrew Straw wrote: >I would like to announce the release of stdeb 0.2. > >= What is it? = > >stdeb http://stdeb.python-hosting.com/ ("setuptools debian") produces Debian source packages from Python packages via a new distutils command, sdist_dsc, which produces a Debian source package of a Python package. Automatic defaults are provided for the Debian package, but many aspects of the resulting package can be customized via a configuration file. > >= What's new? = > Cool. I gave this a try on Twisted. Here's what I found: * dpkg-buildpackage failed on the result when I ran sdist_dsc from a "development" version of Twisted - ie, one with a version string like "8.0.1+r23437". This seemed to be caused by directories in the source package being named inconsistently, resulting in commands failing with "no such directory" errors. * After I switched to a released version of Twisted, I was able to create a package. When installing the package, I encountered this failure: Compiling /usr/lib/python2.4/site-packages/twisted/test/generator_failure_tests.py ... File "/usr/lib/python2.4/site-packages/twisted/test/generator_failure_tests.py", line 66 yield ^ SyntaxError: invalid syntax pycentral: pycentral pkginstall: error byte-compiling files (991) pycentral pkginstall: error byte-compiling files (991) dpkg: error processing python-twisted (--install): subprocess post-installation script returned error exit status 1 Errors were encountered while processing: python-twisted Indeed, there is a file in Twisted which is not valid Python 2.4 syntax. It is only used by the test suite when running on Python 2.5. Is there some way to have this excluded or compensated for? * Perhaps because of the previous error, the installed package doesn't look quite right. With Python 2.4, the `twisted? package is importable but with Python 2.5 it can't be found. Compared to the twisted dir in the Python 2.4 site-packages directory, the twisted dir in the Python 2.5 site-packages directory is very sparsely populated, with many files and directories missing. Thanks for your work on this. Jean-Paul From strawman at astraw.com Sat Apr 26 22:02:46 2008 From: strawman at astraw.com (Andrew Straw) Date: Sat, 26 Apr 2008 13:02:46 -0700 Subject: [Distutils] [ANN] stdeb 0.2 In-Reply-To: <20080426164610.6859.110254766.divmod.quotient.35549@ohm> References: <20080426164610.6859.110254766.divmod.quotient.35549@ohm> Message-ID: <48138A66.7000607@astraw.com> Jean-Paul Calderone wrote: > On Sat, 26 Apr 2008 03:49:34 -0700, Andrew Straw > wrote: >> I would like to announce the release of stdeb 0.2. > Cool. I gave this a try on Twisted. Here's what I found: Thanks for testing and, better, reporting your feedback. > * dpkg-buildpackage failed on the result when I ran sdist_dsc from a > "development" version of Twisted - ie, one with a version string > like "8.0.1+r23437". This seemed to be caused by directories in the > source package being named inconsistently, resulting in commands > failing with "no such directory" errors. OK, I added a couple of packages with such version numbers to the test suite and fixed a few bugs that cropped up. I've just released 0.2.1 which includes these fixes. > Indeed, there is a file in Twisted which is not valid Python 2.4 syntax. > It is only used by the test suite when running on Python 2.5. Is there > some way to have this excluded or compensated for? Yes, this can be handled with the XS-Python-Version: config option. (See the section "Customizing the produced Debian source package (config options)" on the web page.) Specifically, you want "XS-Python-Version: >= 2.5". > > * Perhaps because of the previous error, the installed package doesn't > look quite right. With Python 2.4, the `twisted? package is > importable > but with Python 2.5 it can't be found. Compared to the twisted dir > in the Python 2.4 site-packages directory, the twisted dir in the > Python > 2.5 site-packages directory is very sparsely populated, with many > files > and directories missing. Let me know if this still persists. Most of the files get installed to /usr/share/pycentral, and then a post-install script creates symlinks to the /usr/lib/python2.x directories. (This is standard python-central behavior.) From exarkun at divmod.com Sat Apr 26 22:29:55 2008 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Sat, 26 Apr 2008 16:29:55 -0400 Subject: [Distutils] [ANN] stdeb 0.2 In-Reply-To: <48138A66.7000607@astraw.com> Message-ID: <20080426202955.6859.494780034.divmod.quotient.55317@ohm> On Sat, 26 Apr 2008 13:02:46 -0700, Andrew Straw wrote: >Jean-Paul Calderone wrote: >> On Sat, 26 Apr 2008 03:49:34 -0700, Andrew Straw >> wrote: >>> I would like to announce the release of stdeb 0.2. >> Cool. I gave this a try on Twisted. Here's what I found: >Thanks for testing and, better, reporting your feedback. > [snip] Great! The version issue seems to be resolved now. I also edited the generated XS-Python-Version to indicate the Python 2.5 dependency and that seems to have fixed the installation issue in the resulting .deb. With a bit more work and a custom config file or three this may be creating some nice debs. This is going to be very useful, thanks. :) Jean-Paul From marius at pov.lt Sun Apr 27 00:08:04 2008 From: marius at pov.lt (Marius Gedminas) Date: Sun, 27 Apr 2008 01:08:04 +0300 Subject: [Distutils] Ignore ~/.pydistutils.cfg ? In-Reply-To: References: Message-ID: <20080426220804.GA25075@platonas> On Sat, Apr 26, 2008 at 11:55:57AM -0400, Peter Fein wrote: > is there a way to tell distutils/setuptools to ignore > ~/.pydistutils.cfg? > > I use a config to keep easy_install'd packages in my home directory but > doing so confuses MacPorts; they're apparently passing --exec/-- > exec-prefix, but it seems to be getting overriden by my config. See: > http://trac.macports.org/projects/macports/ticket/9831#comment:4 For a similar reason I removed ~/.pydistutils.cfg and instead created a very short easy_install wrapper script to come first in my $PATH. It invokes the real easy_install and passes a couple of options that I previously specified in .pydistutils.cfg. Marius Gedminas -- Press any key to continue, or any other key to cancel. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From doolagarl2002 at yahoo.com.au Mon Apr 28 05:04:32 2008 From: doolagarl2002 at yahoo.com.au (Scott Brown) Date: Sun, 27 Apr 2008 20:04:32 -0700 (PDT) Subject: [Distutils] setuptools easy_install upgrade to 0.6c8 problem Message-ID: <931481.90148.qm@web56406.mail.re3.yahoo.com> I have upgraded setuptools from http://peak.telecommunity.com/DevCenter/EasyInstall to 0.6c8, but my system keep trying to use the older version 0.6c7. If I remove the older version, then I get the error message: File "/Library/Python/2.5/site-packages/setuptools-0.6c8-py2.5.egg/pkg_resources.py", line 524, in resolve pkg_resources.DistributionNotFound: setuptools==0.6c7 when I try to run easy install. When I check the installation of the new version it says that it has already been installed: sudo python ez_setup.py Setuptools version 0.6c8 or greater has been installed. (Run "ez_setup.py -U setuptools" to reinstall or upgrade.) I am using Mac OS X 10.5. How do I completely remove all traces of the old version of setuptools and get the new version 0.6c8 recognised? Get the name you always wanted with the new y7mail email address. www.yahoo7.com.au/y7mail From pje at telecommunity.com Mon Apr 28 06:07:34 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 28 Apr 2008 00:07:34 -0400 Subject: [Distutils] setuptools easy_install upgrade to 0.6c8 problem In-Reply-To: <931481.90148.qm@web56406.mail.re3.yahoo.com> References: <931481.90148.qm@web56406.mail.re3.yahoo.com> Message-ID: <20080428040819.200183A410E@sparrow.telecommunity.com> At 08:04 PM 4/27/2008 -0700, Scott Brown wrote: >I have upgraded setuptools from >http://peak.telecommunity.com/DevCenter/EasyInstall to 0.6c8, but my >system keep trying to use the older version 0.6c7. If I remove the >older version, then I get the error message: > > File > "/Library/Python/2.5/site-packages/setuptools-0.6c8-py2.5.egg/pkg_resources.py", > line 524, in resolve >pkg_resources.DistributionNotFound: setuptools==0.6c7 > >when I try to run easy install. What's the full traceback? I can't tell from just those two lines what's actually going on. >When I check the installation of the new version it says that it has >already been installed: > >sudo python ez_setup.py >Setuptools version 0.6c8 or greater has been installed. >(Run "ez_setup.py -U setuptools" to reinstall or upgrade.) > >I am using Mac OS X 10.5. How do I completely remove all traces of >the old version of setuptools and get the new version 0.6c8 recognised? Offhand, my guess is that the script location you're installing the new version to is later on your PATH than the old version. It looks like your PYTHONPATH is correct, but that the 'easy_install' executable wasn't overwritten when you installed the newer version -- i.e., the new version installed its script to a different location that's either not on your PATH or is after the location where the old easy_install script is. So, I would probably do a 'which easy_install' to find out which one you're running (I'm assuming OS X includes a 'which' command, which it may not) and then run 'ez_setup -U --script-dir=/whatever setuptools' where '/whatever' is the directory where your current 'easy_install' executable is. From doolagarl2002 at yahoo.com.au Mon Apr 28 11:41:45 2008 From: doolagarl2002 at yahoo.com.au (Scott Brown) Date: Mon, 28 Apr 2008 02:41:45 -0700 (PDT) Subject: [Distutils] setuptools easy_install upgrade to 0.6c8 problem In-Reply-To: <20080428040819.200183A410E@sparrow.telecommunity.com> Message-ID: <185414.10607.qm@web56414.mail.re3.yahoo.com> That fixed it, thank you. --- On Mon, 28/4/08, Phillip J. Eby wrote: > From: Phillip J. Eby > Subject: Re: [Distutils] setuptools easy_install upgrade to 0.6c8 problem > To: doolagarl2002 at yahoo.com.au, distutils-sig at python.org > Received: Monday, 28 April, 2008, 2:07 PM > At 08:04 PM 4/27/2008 -0700, Scott Brown wrote: > >I have upgraded setuptools from > >http://peak.telecommunity.com/DevCenter/EasyInstall to > 0.6c8, but my > >system keep trying to use the older version 0.6c7. If I > remove the > >older version, then I get the error message: > > > > File > > > "/Library/Python/2.5/site-packages/setuptools-0.6c8-py2.5.egg/pkg_resources.py", > > > line 524, in resolve > >pkg_resources.DistributionNotFound: setuptools==0.6c7 > > > >when I try to run easy install. > > What's the full traceback? I can't tell from just > those two lines > what's actually going on. > > > >When I check the installation of the new version it > says that it has > >already been installed: > > > >sudo python ez_setup.py > >Setuptools version 0.6c8 or greater has been installed. > >(Run "ez_setup.py -U setuptools" to reinstall > or upgrade.) > > > >I am using Mac OS X 10.5. How do I completely remove > all traces of > >the old version of setuptools and get the new version > 0.6c8 recognised? > > Offhand, my guess is that the script location you're > installing the > new version to is later on your PATH than the old version. > It looks > like your PYTHONPATH is correct, but that the > 'easy_install' > executable wasn't overwritten when you installed the > newer version -- > i.e., the new version installed its script to a different > location > that's either not on your PATH or is after the location > where the old > easy_install script is. > > So, I would probably do a 'which easy_install' to > find out which one > you're running (I'm assuming OS X includes a > 'which' command, which > it may not) and then run 'ez_setup -U > --script-dir=/whatever > setuptools' where '/whatever' is the directory > where your current > 'easy_install' executable is. Get the name you always wanted with the new y7mail email address. www.yahoo7.com.au/y7mail From meine at informatik.uni-hamburg.de Mon Apr 28 16:00:16 2008 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Mon, 28 Apr 2008 16:00:16 +0200 Subject: [Distutils] virtual env with exec prefix In-Reply-To: <4803A456.2010103@colorstudy.com> References: <200804101335.47020.meine@informatik.uni-hamburg.de> <200804141832.40294.meine@informatik.uni-hamburg.de> <4803A456.2010103@colorstudy.com> Message-ID: <200804281600.17015.meine@informatik.uni-hamburg.de> Hi Ian! Am Montag, 14. April 2008 20:37:10 schrieb Ian Bicking: > Hmm... well, does this error only happen when you create a virtualenv > from within a virtualenv? No, absolutely not. It arises whenever python is installed with --exec-prefix != --prefix. > Otherwise, I think there's several things in > distutils.sysconfig that need to be adjusted. That module isn't really > documented, however, and I suspect there's just a bunch of things that > will have to be added to over time, not a good general fix. Or... maybe > there's some clever way to fix all these, but I'm not sure. AFAICS, my patch is quite clean - it just extends virtualenv slightly by handling exec_prefix and prefix separately. This affects the same entrypoints as before (get_python_inc / get_python_lib) and thus promotes to the same changes in higher level functions. Here's how I tested it: import distutils.sysconfig, os for p in range(2): for s in range(2): print distutils.sysconfig.get_python_lib(p, s) print "\nMakefile:", distutils.sysconfig.get_makefile_filename() print " exists:", os.path.exists(distutils.sysconfig.get_makefile_filename()) You can see the results before/after my virtualenv change here: *** Without virtualenv: *** [2] meine at kogspc33:~ -> python test_python_inst.py /software/python-2.4.4/lib/python2.4/site-packages /software/python-2.4.4/lib/python2.4 /software/python-2.4.4/SuSE-10.2/lib/python2.4/site-packages /software/python-2.4.4/SuSE-10.2/lib/python2.4 Makefile: /software/python-2.4.4/SuSE-10.2/lib/python2.4/config/Makefile exists: True *** Trying to create a virtualenv: *** [2] meine at kogspc33:~ -> virtualenv tmp/ve1 New python executable in tmp/ve1/bin/python Installing setuptools.... Complete output from command tmp/ve1/bin/python -c "#!python \"\"\"Bootstrap setuptoo... " /software/python-2.4.4/lib/pyt...4.egg: error: invalid Python installation: unable to open /software/python-2.4.4/lib/python2.4/config/Makefile (No such file or directory) ---------------------------------------- ...Installing setuptools...done. Traceback (most recent call last): File "/software/python-2.4.4/bin/virtualenv", line 7, in ? sys.exit( File "/software/python-2.4.4/lib/python2.4/site-packages/virtualenv-1.0-py2.4.egg/virtualenv.py", line 337, in main create_environment(home_dir, site_packages=not options.no_site_packages, clear=options.clear) File "/software/python-2.4.4/lib/python2.4/site-packages/virtualenv-1.0-py2.4.egg/virtualenv.py", line 536, in create_environment install_setuptools(py_executable) File "/software/python-2.4.4/lib/python2.4/site-packages/virtualenv-1.0-py2.4.egg/virtualenv.py", line 260, in install_setuptools extra_env=env) File "/software/python-2.4.4/lib/python2.4/site-packages/virtualenv-1.0-py2.4.egg/virtualenv.py", line 396, in call_subprocess raise OSError( OSError: Command tmp/ve1/bin/python -c "#!python \"\"\"Bootstrap setuptoo... " /software/python-2.4.4/lib/pyt...4.egg failed with error code 1 *** Unpatched virtualenv: *** As illustrated above, the installation of setuptools failed, but one can use the already-installed python interpreter to run my test program and gets: [2] meine at kogspc33:~ -> tmp/ve1/bin/python test_python_inst.py /home/meine/tmp/ve1/lib/python2.4/site-packages /software/python-2.4.4/lib/python2.4 /home/meine/tmp/ve1/lib/python2.4/site-packages /software/python-2.4.4/lib/python2.4 Makefile: /software/python-2.4.4/lib/python2.4/config/Makefile exists: False This explains why the setuptools installation failed. *** Patched virtualenv with real_exec_prefix *** [2] meine at kogspc33:~ -> . tmp/ve2/bin/activate (ve2)[2] meine at kogspc33:~ -> python test_python_inst.py /home/meine/tmp/ve2/lib/python2.4/site-packages /software/python-2.4.4/lib/python2.4 /home/meine/tmp/ve2/lib/python2.4/site-packages /software/python-2.4.4/SuSE-10.2/lib/python2.4 Makefile: /software/python-2.4.4/SuSE-10.2/lib/python2.4/config/Makefile exists: True I hope I could convince you to include this crucial fix in virtualenv. -- Ciao, / / /--/ / / ANS From optilude at gmx.net Tue Apr 29 02:00:12 2008 From: optilude at gmx.net (Martin Aspeli) Date: Tue, 29 Apr 2008 01:00:12 +0100 Subject: [Distutils] buildout and mod_wsgi Message-ID: Hi all, I'm trying to deploy a Repoze-based application via mod_wsgi. I'm building Repoze in a buildout. The problem is that I need to get mod_wsgi to execute the WSGI script with buildout's working set of eggs. The mod_wsgi configuration looks like this: WSGIDaemonProcess tmp threads=1 processes=4 maximum-requests=10000 python-path=/usr/lib/python2.4/site-packages ServerName my.machine.local WSGIScriptAlias / /path/to/bin/zope2.wsgi WSGIProcessGroup tmp WSGIPassAuthorization On Now, this says to create a process pool of Python processes with the given python-path. This is really geared towards the virtualenv use case, where you'd have a custom virtualenv python-path for each project. In the case of buildout, however, the pythonpath is explicitly for console scripts by munging console scripts and doing sys.path manipulation. Unfortunately, zope2.wsgi is not a console script, it's just a script that contains: from paste.deploy import loadapp ini = '/path/to/pasteconfig.ini' application = loadapp('config:%s' % ini) The key here is that this is a script that needs to define a global variable 'application'. The only way I can make this work, is to paste a block of sys.path manipulation from a console script that buildout has had the opportunity to munge, but that's not exactly a stable solution. I can see the following possible solutions: 1) Make a buildout recipe that creates a directory with a .pth file for all the eggs in the working set. This would then be able to serve as a python-path above. 2) Make a buildout recipe that generates the zope2.wsgi script as above, but also generates the sys.path munging. Does one of these exist already? Is there a better way? For reference: http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGIDaemonProcess Cheers, Martin From jim at zope.com Tue Apr 29 14:28:31 2008 From: jim at zope.com (Jim Fulton) Date: Tue, 29 Apr 2008 08:28:31 -0400 Subject: [Distutils] buildout and mod_wsgi In-Reply-To: References: Message-ID: <9F1A2A98-EB09-4E3E-B647-4A0666AA0060@zope.com> On Apr 28, 2008, at 8:00 PM, Martin Aspeli wrote: ... > I'm trying to deploy a Repoze-based application via mod_wsgi. I'm > building Repoze in a buildout. The problem is that I need to get > mod_wsgi to execute the WSGI script with buildout's working set of > eggs. > > The mod_wsgi configuration looks like this: > > WSGIDaemonProcess tmp threads=1 processes=4 maximum- > requests=10000 python-path=/usr/lib/python2.4/site-packages > > > ServerName my.machine.local > WSGIScriptAlias / /path/to/bin/zope2.wsgi > WSGIProcessGroup tmp > WSGIPassAuthorization On > > > > Now, this says to create a process pool of Python processes with the > given python-path. This is really geared towards the virtualenv use > case, where you'd have a custom virtualenv python-path for each > project. > In the case of buildout, however, the pythonpath is explicitly for > console scripts by munging console scripts and doing sys.path > manipulation. This is imprecise at best. Buildout, like easy_install, generate console scripts. They do so with different assumptions. Easy_install makes the assumption that eggs are in the Python path, typically in site-packages, wheres buildout doesn't. > Unfortunately, zope2.wsgi is not a console script, it's just a > script that contains: > > from paste.deploy import loadapp > > ini = '/path/to/pasteconfig.ini' > application = loadapp('config:%s' % ini) > > The key here is that this is a script that needs to define a global > variable 'application'. Interesting. So is it execfiled? Imported? > The only way I can make this work, is to paste a block of sys.path > manipulation from a console script that buildout has had the > opportunity to munge, but that's not exactly a stable solution. > > > I can see the following possible solutions: > > 1) Make a buildout recipe that creates a directory with a .pth file > for all the eggs in the working set. This would then be able to > serve as a python-path above. Nah > 2) Make a buildout recipe that generates the zope2.wsgi script as > above, but also generates the sys.path munging. Yes, that's the right solution. This should be straightforward. All you really need is the working_set that you can get from calling documented APIs in the zc.buildout.easy_install module. Then you can iterate over that getting the distribution locations to add to the front of sys.path. > Does one of these exist already? Is there a better way? 2 is the better way. Jim -- Jim Fulton Zope Corporation From skip at pobox.com Tue Apr 29 17:11:56 2008 From: skip at pobox.com (skip at pobox.com) Date: Tue, 29 Apr 2008 10:11:56 -0500 Subject: [Distutils] Two versions of SQLAlchemy side-by-side? Message-ID: <18455.15036.578777.273580@montanaro-dyndns-org.local> We have SQLAlchemy 0.3.3 installed via setuptools. I upgraded to 0.4.5 today but had to back that out (by editing easy-install.pth) because of API changes. Now I have these two installs .../site-packages/SQLAlchemy-0.3.3-py2.4.egg .../site-packages/SQLAlchemy-0.4.5-py2.4.egg Is there a way to import the 0.4.5 version for testing without disturbing the 0.3.3 version? I tried placing the 0.4.5 egg directory in PYTHONPATH but I still get the 0.3.3 version when I "import sqlalchemy". What's a fella to do? Thanks, Skip From lxander.m at gmail.com Tue Apr 29 17:29:34 2008 From: lxander.m at gmail.com (Alexander Michael) Date: Tue, 29 Apr 2008 11:29:34 -0400 Subject: [Distutils] Two versions of SQLAlchemy side-by-side? In-Reply-To: <18455.15036.578777.273580@montanaro-dyndns-org.local> References: <18455.15036.578777.273580@montanaro-dyndns-org.local> Message-ID: <525f23e80804290829r3b2bfd2du4cf4cb27380ee4da@mail.gmail.com> The quick hack to request a version, good to use in the interpreter or a "throw-away" script: import pkg_resources pkg_resources.require('SQLAlchemy==0.4.5') import sqlalchemy This works when the directory that SQLAlchemy-0.4.5-py2.4.egg is in is listed in sys.path. On Tue, Apr 29, 2008 at 11:11 AM, wrote: > We have SQLAlchemy 0.3.3 installed via setuptools. I upgraded to 0.4.5 > today but had to back that out (by editing easy-install.pth) because of API > changes. Now I have these two installs > > .../site-packages/SQLAlchemy-0.3.3-py2.4.egg > .../site-packages/SQLAlchemy-0.4.5-py2.4.egg > > Is there a way to import the 0.4.5 version for testing without disturbing > the 0.3.3 version? I tried placing the 0.4.5 egg directory in PYTHONPATH > but I still get the 0.3.3 version when I "import sqlalchemy". > > What's a fella to do? > > Thanks, > > Skip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From skip at pobox.com Tue Apr 29 17:42:56 2008 From: skip at pobox.com (skip at pobox.com) Date: Tue, 29 Apr 2008 10:42:56 -0500 Subject: [Distutils] Two versions of SQLAlchemy side-by-side? In-Reply-To: <525f23e80804290829r3b2bfd2du4cf4cb27380ee4da@mail.gmail.com> References: <18455.15036.578777.273580@montanaro-dyndns-org.local> <525f23e80804290829r3b2bfd2du4cf4cb27380ee4da@mail.gmail.com> Message-ID: <18455.16896.997650.539149@montanaro-dyndns-org.local> Alexander> The quick hack to request a version, good to use in the Alexander> interpreter or a "throw-away" script: Alexander> import pkg_resources Alexander> pkg_resources.require('SQLAlchemy==0.4.5') Alexander> import sqlalchemy Alexander> This works when the directory that SQLAlchemy-0.4.5-py2.4.egg Alexander> is in is listed in sys.path. Thanks, however that raises a VersionConflict exception: >>> pkg_resources.require('SQLAlchemy==0.4.5') Traceback (most recent call last): File "", line 1, in ? File "/opt/app/g++lib6/python-2.4.5/lib/python2.4/site-packages/setuptools-0.6c7-py2.4.egg/pkg_resources.py", line 626, in require needed = self.resolve(parse_requirements(requirements)) File "/opt/app/g++lib6/python-2.4.5/lib/python2.4/site-packages/setuptools-0.6c7-py2.4.egg/pkg_resources.py", line 528, in resolve raise VersionConflict(dist,req) # XXX put more info here pkg_resources.VersionConflict: (SQLAlchemy 0.3.3 (/opt/app/g++lib6/python-2.4.5/lib/python2.4/site-packages/SQLAlchemy-0.3.3-py2.4.egg), Requirement.parse('SQLAlchemy==0.4.5')) I need 0.3.3 to be the default version, but need to be able to import 0.4.5 for testing. Thx, Skip From lxander.m at gmail.com Tue Apr 29 18:03:12 2008 From: lxander.m at gmail.com (Alexander Michael) Date: Tue, 29 Apr 2008 12:03:12 -0400 Subject: [Distutils] Two versions of SQLAlchemy side-by-side? In-Reply-To: <18455.16896.997650.539149@montanaro-dyndns-org.local> References: <18455.15036.578777.273580@montanaro-dyndns-org.local> <525f23e80804290829r3b2bfd2du4cf4cb27380ee4da@mail.gmail.com> <18455.16896.997650.539149@montanaro-dyndns-org.local> Message-ID: <525f23e80804290903j6047d390v99b732a458c35a3a@mail.gmail.com> On Tue, Apr 29, 2008 at 11:42 AM, wrote: > > Alexander> The quick hack to request a version, good to use in the > Alexander> interpreter or a "throw-away" script: > > Alexander> import pkg_resources > Alexander> pkg_resources.require('SQLAlchemy==0.4.5') > Alexander> import sqlalchemy > > Alexander> This works when the directory that SQLAlchemy-0.4.5-py2.4.egg > Alexander> is in is listed in sys.path. > > Thanks, however that raises a VersionConflict exception: > > > >>> pkg_resources.require('SQLAlchemy==0.4.5') > Traceback (most recent call last): > File "", line 1, in ? > File "/opt/app/g++lib6/python-2.4.5/lib/python2.4/site-packages/setuptools-0.6c7-py2.4.egg/pkg_resources.py", line 626, in require > needed = self.resolve(parse_requirements(requirements)) > File "/opt/app/g++lib6/python-2.4.5/lib/python2.4/site-packages/setuptools-0.6c7-py2.4.egg/pkg_resources.py", line 528, in resolve > raise VersionConflict(dist,req) # XXX put more info here pkg_resources.VersionConflict: (SQLAlchemy 0.3.3 > (/opt/app/g++lib6/python-2.4.5/lib/python2.4/site-packages/SQLAlchemy-0.3.3-py2.4.egg), Requirement.parse('SQLAlchemy==0.4.5')) > > I need 0.3.3 to be the default version, but need to be able to import 0.4.5 > for testing. Ah. Yes. I install everything --multi-version so I have forgotten about this issue with the default working set. Maybe __requires__ = ['SQLAlchemy==0.4.5'] import pkg_resources import sqlalchemy will help pkg_resources create the desired WorkingSet. From skip at pobox.com Tue Apr 29 20:18:44 2008 From: skip at pobox.com (skip at pobox.com) Date: Tue, 29 Apr 2008 13:18:44 -0500 Subject: [Distutils] Two versions of SQLAlchemy side-by-side? In-Reply-To: <525f23e80804290903j6047d390v99b732a458c35a3a@mail.gmail.com> References: <18455.15036.578777.273580@montanaro-dyndns-org.local> <525f23e80804290829r3b2bfd2du4cf4cb27380ee4da@mail.gmail.com> <18455.16896.997650.539149@montanaro-dyndns-org.local> <525f23e80804290903j6047d390v99b732a458c35a3a@mail.gmail.com> Message-ID: <18455.26244.777807.424677@montanaro-dyndns-org.local> >> Thanks, however that raises a VersionConflict exception: Alexander> Ah. Yes. I install everything --multi-version so I have Alexander> forgotten about this issue with the default working Alexander> set. Maybe Alexander> __requires__ = ['SQLAlchemy==0.4.5'] Alexander> import pkg_resources Alexander> import sqlalchemy Thanks. Worked like a charm. I'll have to check out --multi-version. I've never used it before. Is there any downside to using it? Skip From pje at telecommunity.com Tue Apr 29 20:39:06 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 29 Apr 2008 14:39:06 -0400 Subject: [Distutils] Two versions of SQLAlchemy side-by-side? In-Reply-To: <18455.26244.777807.424677@montanaro-dyndns-org.local> References: <18455.15036.578777.273580@montanaro-dyndns-org.local> <525f23e80804290829r3b2bfd2du4cf4cb27380ee4da@mail.gmail.com> <18455.16896.997650.539149@montanaro-dyndns-org.local> <525f23e80804290903j6047d390v99b732a458c35a3a@mail.gmail.com> <18455.26244.777807.424677@montanaro-dyndns-org.local> Message-ID: <20080429183849.7AD4F3A4080@sparrow.telecommunity.com> At 01:18 PM 4/29/2008 -0500, skip at pobox.com wrote: >Thanks. Worked like a charm. I'll have to check out --multi-version. I've >never used it before. Is there any downside to using it? The only downside is that you won't be able to "import sqlalchemy" at all, without first importing pkg_resources and require()-ing the desired version. And in your current scenario, you'll want to keep 0.3.3 as your default since other packages with an open-ended version requirement would end up selecting 0.4.5. From optilude at gmx.net Tue Apr 29 22:31:24 2008 From: optilude at gmx.net (Martin Aspeli) Date: Tue, 29 Apr 2008 21:31:24 +0100 Subject: [Distutils] buildout and mod_wsgi In-Reply-To: <9F1A2A98-EB09-4E3E-B647-4A0666AA0060@zope.com> References: <9F1A2A98-EB09-4E3E-B647-4A0666AA0060@zope.com> Message-ID: Jim Fulton wrote: > On Apr 28, 2008, at 8:00 PM, Martin Aspeli wrote: > ... >> I'm trying to deploy a Repoze-based application via mod_wsgi. I'm >> building Repoze in a buildout. The problem is that I need to get >> mod_wsgi to execute the WSGI script with buildout's working set of >> eggs. >> >> The mod_wsgi configuration looks like this: >> >> WSGIDaemonProcess tmp threads=1 processes=4 maximum- >> requests=10000 python-path=/usr/lib/python2.4/site-packages >> >> >> ServerName my.machine.local >> WSGIScriptAlias / /path/to/bin/zope2.wsgi >> WSGIProcessGroup tmp >> WSGIPassAuthorization On >> >> >> >> Now, this says to create a process pool of Python processes with the >> given python-path. This is really geared towards the virtualenv use >> case, where you'd have a custom virtualenv python-path for each >> project. > >> In the case of buildout, however, the pythonpath is explicitly for >> console scripts by munging console scripts and doing sys.path >> manipulation. > > This is imprecise at best. Buildout, like easy_install, generate > console scripts. They do so with different assumptions. Easy_install > makes the assumption that eggs are in the Python path, typically in > site-packages, wheres buildout doesn't. It wasn't meant as a criticism at all - in general, I prefer this way of working. It's just made things a bit complex in this particular case. :) >> Unfortunately, zope2.wsgi is not a console script, it's just a >> script that contains: >> >> from paste.deploy import loadapp >> >> ini = '/path/to/pasteconfig.ini' >> application = loadapp('config:%s' % ini) >> >> The key here is that this is a script that needs to define a global >> variable 'application'. > > Interesting. So is it execfiled? Imported? Imported, I believe. I'm not an expert on wsgi or mod_wsgi. See http://code.google.com/p/modwsgi/wiki/ConfigurationGuidelines >> The only way I can make this work, is to paste a block of sys.path >> manipulation from a console script that buildout has had the >> opportunity to munge, but that's not exactly a stable solution. >> >> >> I can see the following possible solutions: >> >> 1) Make a buildout recipe that creates a directory with a .pth file >> for all the eggs in the working set. This would then be able to >> serve as a python-path above. > > Nah Can you elaborate on why this is the "wrong" solution? >> 2) Make a buildout recipe that generates the zope2.wsgi script as >> above, but also generates the sys.path munging. > > Yes, that's the right solution. > > This should be straightforward. All you really need is the > working_set that you can get from calling documented APIs in the > zc.buildout.easy_install module. Then you can iterate over that > getting the distribution locations to add to the front of sys.path. I find the idea of writing out Python code to a file fairly daunting and error-prone. It's obviously doable, but I'd be happier with some way of looping over the working set from within the zope2.wsgi script itself and adding it to sys.path. Is there some clean way to "serialise" the working set and pull it out again in a different script? Cheers, Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book From mhearne at usgs.gov Wed Apr 30 17:15:35 2008 From: mhearne at usgs.gov (Michael Hearne) Date: Wed, 30 Apr 2008 09:15:35 -0600 Subject: [Distutils] problems bundling data with setuptools Message-ID: <48E1E04C-C2E0-4352-97F4-34896C05E09D@usgs.gov> I'm having trouble bundling any kind of data (non-Python code) in my egg. Here are the file contents of my source directory: hello.py (cheesy script) README.txt MANIFEST.in (has one line, "include *.txt") setup.py, which looks like: from setuptools import setup, find_packages setup( name = "HelloWorld", version = "0.1", scripts=['hello.py'], include_package_data = True ) I'm creating the egg with python setup.py bdist_egg. I would expect that README.txt should appear in the egg somewhere, but it doesn't. What am I missing? Thanks, Mike ------------------------------------------------------ Michael Hearne mhearne at usgs.gov (303) 273-8620 USGS National Earthquake Information Center 1711 Illinois St. Golden CO 80401 Senior Software Engineer Synergetics, Inc. ------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Wed Apr 30 17:27:30 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 30 Apr 2008 11:27:30 -0400 Subject: [Distutils] problems bundling data with setuptools In-Reply-To: <48E1E04C-C2E0-4352-97F4-34896C05E09D@usgs.gov> References: <48E1E04C-C2E0-4352-97F4-34896C05E09D@usgs.gov> Message-ID: <20080430152713.447CD3A4070@sparrow.telecommunity.com> At 09:15 AM 4/30/2008 -0600, Michael Hearne wrote: >I'm having trouble bundling any kind of data (non-Python code) in my egg. > >Here are the file contents of my source directory: >hello.py (cheesy script) >README.txt >MANIFEST.in (has one line, "include *.txt") >setup.py, which looks like: > >from setuptools import setup, find_packages > >setup( > name = "HelloWorld", > version = "0.1", > scripts=['hello.py'], > include_package_data = True >) > >I'm creating the egg with python setup.py bdist_egg. > >I would expect that README.txt should appear in the egg somewhere, >but it doesn't. > >What am I missing? README.txt isn't in a package, so it's not "package data". And MANIFEST.in determines what goes in your *source* distribution, not your binary distribution. From mhearne at usgs.gov Wed Apr 30 17:33:04 2008 From: mhearne at usgs.gov (Michael Hearne) Date: Wed, 30 Apr 2008 09:33:04 -0600 Subject: [Distutils] problems bundling data with setuptools In-Reply-To: <20080430152713.447CD3A4070@sparrow.telecommunity.com> References: <48E1E04C-C2E0-4352-97F4-34896C05E09D@usgs.gov> <20080430152713.447CD3A4070@sparrow.telecommunity.com> Message-ID: Oh, ok. So is there a way to include data (in a package or not) in my binary distribution? --Mike On Apr 30, 2008, at 9:27 AM, Phillip J. Eby wrote: > At 09:15 AM 4/30/2008 -0600, Michael Hearne wrote: >> I'm having trouble bundling any kind of data (non-Python code) in >> my egg. >> >> Here are the file contents of my source directory: >> hello.py (cheesy script) >> README.txt >> MANIFEST.in (has one line, "include *.txt") >> setup.py, which looks like: >> >> from setuptools import setup, find_packages >> >> setup( >> name = "HelloWorld", >> version = "0.1", >> scripts=['hello.py'], >> include_package_data = True >> ) >> >> I'm creating the egg with python setup.py bdist_egg. >> >> I would expect that README.txt should appear in the egg somewhere, >> but it doesn't. >> >> What am I missing? > > README.txt isn't in a package, so it's not "package data". And > MANIFEST.in determines what goes in your *source* distribution, not > your binary distribution. ------------------------------------------------------ Michael Hearne mhearne at usgs.gov (303) 273-8620 USGS National Earthquake Information Center 1711 Illinois St. Golden CO 80401 Senior Software Engineer Synergetics, Inc. ------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Wed Apr 30 17:59:57 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 30 Apr 2008 11:59:57 -0400 Subject: [Distutils] problems bundling data with setuptools In-Reply-To: References: <48E1E04C-C2E0-4352-97F4-34896C05E09D@usgs.gov> <20080430152713.447CD3A4070@sparrow.telecommunity.com> Message-ID: <20080430155940.34A3C3A4080@sparrow.telecommunity.com> At 09:33 AM 4/30/2008 -0600, Michael Hearne wrote: >Oh, ok. So is there a way to include data (in a package or not) in >my binary distribution? See the distutils docs on that. There has been some discussion around creating a standard for including docs and other things in egg metadata, but no concrete proposals have emerged yet. In the meantime, it's important to understand that "eggs are to Python as jars are to Java" -- As far as I know, nobody includes README files, documentation, or any other kind of data in a .jar file, unless it will be used by the code at runtime. In other words, the source distribution is the canonical method of distributing anything that's not used at execution time. And "easy_install --editable" will fetch and unpack source for you, assuming that a source distribution is available. (Unfortunately, many people are only uploading eggs to PyPI, when they should be uploading source distributions, and optionally including eggs if relevant.) Truth be told, there's little reason to upload an .egg to PyPI unless: 1. it contains C code built for OS X or Windows 2. you want it to be able to work with a bootstrapper that lets you import from URLs on sys.path, 3. it's a plugin for a system that allows you to upload or "drop in" eggs to use as plugins. If your project doesn't match these descriptions, there's not really a need for an .egg to be uploaded; easy_install will build one just fine from a source distribution.