From jorge.vargas at gmail.com Mon Jul 2 06:07:33 2007 From: jorge.vargas at gmail.com (Jorge Vargas) Date: Mon, 2 Jul 2007 00:07:33 -0400 Subject: [Distutils] setup.py develop for extras? Message-ID: <32822fe60707012107v6bb6542as965f674b16267afd@mail.gmail.com> Hi how could I trigger installing extras with develop? for example easy_install somePackage equals cd somePackage python setup.py develop easy_install somePackage[someExtras] equals??? From pje at telecommunity.com Mon Jul 2 06:44:34 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 02 Jul 2007 00:44:34 -0400 Subject: [Distutils] setup.py develop for extras? In-Reply-To: <32822fe60707012107v6bb6542as965f674b16267afd@mail.gmail.co m> References: <32822fe60707012107v6bb6542as965f674b16267afd@mail.gmail.com> Message-ID: <20070702044224.DBB4A3A4046@sparrow.telecommunity.com> At 12:07 AM 7/2/2007 -0400, Jorge Vargas wrote: >Hi how could I trigger installing extras with develop? > >for example > >easy_install somePackage >equals >cd somePackage >python setup.py develop > >easy_install somePackage[someExtras] >equals??? You just said it. First put the package in develop mode, then run easy_install somePackage[extras]. From lists at zopyx.com Mon Jul 2 13:07:13 2007 From: lists at zopyx.com (Andreas Jung) Date: Mon, 02 Jul 2007 13:07:13 +0200 Subject: [Distutils] Looking for a presentation on Egg Message-ID: <1B03F79999E9B184F0A42993@[192.168.0.24]> Hi, I have to give a 30 minute presentation on Python egg at our next company developer meeting. Does anyone know of a presentation or slides on this topic to be used as a basis? Thanks, Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 186 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070702/d1bb2a4d/attachment.pgp From gns24 at sphinx.mythic-beasts.com Mon Jul 2 14:08:00 2007 From: gns24 at sphinx.mythic-beasts.com (Graham Stratton) Date: Mon, 2 Jul 2007 13:08:00 +0100 (BST) Subject: [Distutils] Buildout not updating setuptools Message-ID: Hi, I'm having a problem with buildout. My system has setuptools 0.6c5 installed. The buildout bootstrap.py script doesn't install the latest version, and nor does running bin/buildout. The result is that the Routes install fails with The required version of setuptools (>=0.6c6) is not available, ... The problem seems to be that bootstrap.py creates a link for setuptools in develop-eggs. Why would that be? Thanks, Graham From ianb at colorstudy.com Mon Jul 2 16:45:00 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 02 Jul 2007 09:45:00 -0500 Subject: [Distutils] Looking for a presentation on Egg In-Reply-To: <1B03F79999E9B184F0A42993@[192.168.0.24]> References: <1B03F79999E9B184F0A42993@[192.168.0.24]> Message-ID: <46890F6C.3020704@colorstudy.com> It's pretty old, but I don't think much from it has changed: http://svn.colorstudy.com/home/ianb/setuptools-presentation Feel free to take from it as you want Andreas Jung wrote: > Hi, > > I have to give a 30 minute presentation on Python egg at our next > company developer meeting. Does anyone know of a presentation or slides > on this topic to be used as a basis? > > Thanks, > Andreas > > > ------------------------------------------------------------------------ > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org | Write code, do good | http://topp.openplans.org/careers From jim at zope.com Mon Jul 2 17:56:17 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 2 Jul 2007 11:56:17 -0400 Subject: [Distutils] Performance implications of having large numbers of eggs? In-Reply-To: <20070628160503.64AC21DF4F5@mail.enthought.com> References: <4683D81F.9030606@enthought.com> <20070628160503.64AC21DF4F5@mail.enthought.com> Message-ID: <0C3CB51F-714D-4323-B922-C2C1D8097438@zope.com> On Jun 28, 2007, at 12:05 PM, Rick Ratzel wrote: > >> Date: Thu, 28 Jun 2007 10:47:43 -0500 >> From: Dave Peterson >> >> Has anyone done any investigation into the performance >> implications of >> having large numbers of eggs installed? Is there any sort of >> performance hit? >> >> >> It seems to me that having a really large path might slow down >> imports a >> bit, though I suspect this is in C code so probably not a >> significant >> problem. It also seems like there might be some startup >> penalties due >> to the overhead of setting up the path when using eggs, but >> this is a >> one-time cost during python startup, so probably not too bad >> either. > > Another option to avoid a startup penalty is to have all eggs > installed with > -m (not in the easy-install.pth file, or "deactivated") and have > code require() > the specific dependency. This has the obvious disadvantage of > having to change > a fair amount of code. But, the advantages are only adding the > eggs that are > needed to the path when they're needed (instead of every egg in > every PYTHONPATH > dir), and your code will be sure that it's using the version that it's > compatible with. > > I should mention that I don't have any metrics on the startup > penalty, so a > change like this may not be worth it if you're only trying to > improve that. Note that buildout takes this a step further by determining what eggs are needed at install time, rather than run time. I imagine that this could speed startup further, but I don't have any metrics either. :) Of course, you could use the same approach without using buildout. Note that sooner or later, I'm pretty sure we're going to need a more clever algorithm, likely with some sort of backtracking, to determining working sets. At that point, it will become very important to not do this at run time. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Mon Jul 2 18:30:11 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 02 Jul 2007 12:30:11 -0400 Subject: [Distutils] Performance implications of having large numbers of eggs? In-Reply-To: <0C3CB51F-714D-4323-B922-C2C1D8097438@zope.com> References: <4683D81F.9030606@enthought.com> <20070628160503.64AC21DF4F5@mail.enthought.com> <0C3CB51F-714D-4323-B922-C2C1D8097438@zope.com> Message-ID: <20070702162804.4C20F3A4046@sparrow.telecommunity.com> At 11:56 AM 7/2/2007 -0400, Jim Fulton wrote: >Note that sooner or later, I'm pretty sure we're going to >need a more clever algorithm, likely with some sort of backtracking, >to determining working sets. At that point, it will become very >important to not do this at run time. pkg_resources already backtracks if it needs to, although only once (when pkg_resources is imported) and only if the start script's requirements conflict with packages that are activated by default. From pythonhead at gentoo.org Mon Jul 2 18:43:53 2007 From: pythonhead at gentoo.org (Rob Cakebread) Date: Mon, 02 Jul 2007 09:43:53 -0700 Subject: [Distutils] setuptools --no-dependencies option needed? Message-ID: <46892B49.2060300@gentoo.org> Hello, I'm wondering if a --no-dependencies option would be useful. In Gentoo we edit every setup.py file that has install_requires, setup_requires, or extra_requires and remove that with sed or a patch because our package manager takes care of installing all dependencies and because users may already have packages installed previously without setuptools (in python 2.4). We've found that can cause problems[1] and are considering using --single-version-externally-managed, but ideally we'd like to use the full features of setuptools (without losing the ability to install multiple versions, require-ing specific versions etc.). Would an option similar to --single-version-externally-managed that simply doesn't check for installed dependencies and does not try to download and install any dependencies be possible? Rob [1] http://bugs.gentoo.org/show_bug.cgi?id=183977 From pje at telecommunity.com Mon Jul 2 18:58:39 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 02 Jul 2007 12:58:39 -0400 Subject: [Distutils] setuptools --no-dependencies option needed? In-Reply-To: <46892B49.2060300@gentoo.org> References: <46892B49.2060300@gentoo.org> Message-ID: <20070702165628.E4EB93A4046@sparrow.telecommunity.com> At 09:43 AM 7/2/2007 -0700, Rob Cakebread wrote: >Hello, > >I'm wondering if a --no-dependencies option would be useful. In Gentoo >we edit every setup.py file that has install_requires, setup_requires, >or extra_requires and remove that with sed or a patch because our >package manager takes care of installing all dependencies and because >users may already have packages installed previously without setuptools >(in python 2.4). > >We've found that can cause problems[1] and are considering using > --single-version-externally-managed, but ideally we'd like to use the >full features of setuptools (without losing the ability to install >multiple versions, require-ing specific versions etc.). > >Would an option similar to --single-version-externally-managed that >simply doesn't check for installed dependencies and does not try to >download and install any dependencies be possible? use "setup.py easy_install . --no-deps" From jim at zope.com Mon Jul 2 19:32:08 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 2 Jul 2007 13:32:08 -0400 Subject: [Distutils] Missing options for setuptools develop In-Reply-To: References: Message-ID: <14D8711E-0B84-4716-A718-12A5E3AE7E10@zope.com> I agree that it would be nice to be able to pass build_ext arguments to develop. You can put build_ext options in a setup.cfg file. And, FWIW, buildout has a recipe that lets you specify custom build_ext options when creating develop eggs. Jim On Jun 28, 2007, at 2:58 PM, Jamie Kirkpatrick wrote: > Couple of things. > > Currently there is no way to pass custom options through setuptools > "develop" to distutils build_ext. What this means in practice is > that you cannot do the following things: > > ? pass the --debug flag to build_ext so as to have your module build > against debug libraries. > ? pass any of the SWIG options through. > ? I'm sure there are other limitations. > > Because of this I ended up abandoning the swig functionality of > distutils and running that as a pre distuils phase of my build > scripts - it would be better If I could use it through distutils > though. The show stopper is the fact I can't build against debug > libraries - this means I cannot mix the --debug type builds and the > develop functionality, something that seems like a logical thing to > do. > > Is there any chance that someone could find a way to allow options to > be passed through to build_ext as it is run? I'm sure its just a > case of adding another option to setuptools develop and then passing > it through at the relevant point. If noone wants to take it on I'm > happy to write a patch if we all agree its the way to go. > > Thanks in advance > > Jamie > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jkp at kirkconsulting.co.uk Mon Jul 2 23:21:42 2007 From: jkp at kirkconsulting.co.uk (Jamie Kirkpatrick) Date: Mon, 2 Jul 2007 22:21:42 +0100 Subject: [Distutils] Missing options for setuptools develop In-Reply-To: <14D8711E-0B84-4716-A718-12A5E3AE7E10@zope.com> References: <14D8711E-0B84-4716-A718-12A5E3AE7E10@zope.com> Message-ID: <71BD4D0D-3062-4F89-82A3-2040B5A6C271@kirkconsulting.co.uk> Actually Jim, I've found I am able to call with options (as suggested by Phillip) if I put build_ext in the command line first. So the options do exist they just aren't very obvious - I think maybe just improving the docs would be enough. Thanks for the replies. Jamie On 2 Jul 2007, at 18:32, Jim Fulton wrote: > > I agree that it would be nice to be able to pass build_ext > arguments to develop. > > You can put build_ext options in a setup.cfg file. > > And, FWIW, buildout has a recipe that lets you specify custom > build_ext options when creating develop eggs. > > Jim > > On Jun 28, 2007, at 2:58 PM, Jamie Kirkpatrick wrote: > >> Couple of things. >> >> Currently there is no way to pass custom options through setuptools >> "develop" to distutils build_ext. What this means in practice is >> that you cannot do the following things: >> >> ? pass the --debug flag to build_ext so as to have your module build >> against debug libraries. >> ? pass any of the SWIG options through. >> ? I'm sure there are other limitations. >> >> Because of this I ended up abandoning the swig functionality of >> distutils and running that as a pre distuils phase of my build >> scripts - it would be better If I could use it through distutils >> though. The show stopper is the fact I can't build against debug >> libraries - this means I cannot mix the --debug type builds and the >> develop functionality, something that seems like a logical thing >> to do. >> >> Is there any chance that someone could find a way to allow options to >> be passed through to build_ext as it is run? I'm sure its just a >> case of adding another option to setuptools develop and then passing >> it through at the relevant point. If noone wants to take it on I'm >> happy to write a patch if we all agree its the way to go. >> >> Thanks in advance >> >> Jamie >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > -- > Jim Fulton mailto:jim at zope.com Python Powered! > CTO (540) 361-1714 http://www.python.org > Zope Corporation http://www.zope.com http://www.zope.org > > > From pythonhead at gentoo.org Tue Jul 3 17:33:35 2007 From: pythonhead at gentoo.org (Rob Cakebread) Date: Tue, 03 Jul 2007 08:33:35 -0700 Subject: [Distutils] setuptools --no-dependencies option needed? In-Reply-To: <20070702165628.E4EB93A4046@sparrow.telecommunity.com> References: <46892B49.2060300@gentoo.org> <20070702165628.E4EB93A4046@sparrow.telecommunity.com> Message-ID: <468A6C4F.4090607@gentoo.org> Phillip J. Eby wrote: > > use "setup.py easy_install . --no-deps" > We've tried that but it fails because we need to install to our sandbox directory first. Currently we use this, so we need to be able to specify the root with your example: python setup.py install --root=${D} I've tried every combination of setuptools options I can think of (-d, -s, --prefix, -S) but it always fails with permission errors trying to write to the root filesystem: error: can't create or remove files in install directory [Errno 13] Permission denied: '/usr/lib/python2.5/site-packages/test-easy-install-4486.write-test' Thanks, Rob From pje at telecommunity.com Tue Jul 3 17:50:33 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 03 Jul 2007 11:50:33 -0400 Subject: [Distutils] [PEAK] Dependencies before calling setup() In-Reply-To: <468A5A48.6090101@oilspace.com> References: <468A5A48.6090101@oilspace.com> Message-ID: <20070703155307.F225D3A40A9@sparrow.telecommunity.com> At 06:16 PM 7/3/2007 +0400, Timur Izhbulatov wrote: >Hi all, > >I use Cheetah in my project and want to precompile templates in my >project's setup.py. So, I need to have Cheetah installed *before* >setup() is called. Don't do that. setup scripts must *never* perform such actions before setup() is called. What if someone is just running "setup.py --help", after all? NEVER do this. easy_install's sandboxing code will also reject your package as unsafe to run if you attempt something like this. >I'm trying to use setuptools.command.easy_install.main(['Cheetah']) >but I can't find any way to update sys.path after that. Is there any >'official' way to do this? The official way to require that a package is available during setup() execution is viat the 'setup_requires' keyword, similar to the 'install_requires' keyword. If you need the same package in both situations, you must list it in both keywords. To actually do the precompile, you should subclass the install_data or install_lib command, and override its run() method to do the compilation, as well as doing whatever it did before. You use the 'cmdclass' keyword to setup() to pass in a dictionary mapping command names to your subclass(es). Your subclasses *must* honor the options they are given, and not attempt to install anything except where they are told. The options will be available as attributes of the instance created by the distutils. (Oh, and speaking of the distutils, the correct place for discussion/support of setuptools is the distutils-SIG. Please direct future correspondence there. Thanks.) From pje at telecommunity.com Tue Jul 3 19:14:35 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 03 Jul 2007 13:14:35 -0400 Subject: [Distutils] setuptools --no-dependencies option needed? In-Reply-To: <468A6C4F.4090607@gentoo.org> References: <46892B49.2060300@gentoo.org> <20070702165628.E4EB93A4046@sparrow.telecommunity.com> <468A6C4F.4090607@gentoo.org> Message-ID: <20070703171224.D1C553A40A3@sparrow.telecommunity.com> At 08:33 AM 7/3/2007 -0700, Rob Cakebread wrote: >Phillip J. Eby wrote: >>use "setup.py easy_install . --no-deps" > >We've tried that but it fails because we need to install to our >sandbox directory first. Currently we use this, so we need to be >able to specify the root with your example: > >python setup.py install --root=${D} I'm confused. If you're specifying "install --root", you should be getting --single-version-externally-managed, which means that dependencies shouldn't be being installed in the first place. What exactly is the problem again? From pythonhead at gentoo.org Tue Jul 3 21:46:23 2007 From: pythonhead at gentoo.org (Rob Cakebread) Date: Tue, 03 Jul 2007 12:46:23 -0700 Subject: [Distutils] setuptools --no-dependencies option needed? In-Reply-To: <20070703171224.D1C553A40A3@sparrow.telecommunity.com> References: <46892B49.2060300@gentoo.org> <20070702165628.E4EB93A4046@sparrow.telecommunity.com> <468A6C4F.4090607@gentoo.org> <20070703171224.D1C553A40A3@sparrow.telecommunity.com> Message-ID: <468AA78F.8000503@gentoo.org> Phillip J. Eby wrote: > At 08:33 AM 7/3/2007 -0700, Rob Cakebread wrote: >> Phillip J. Eby wrote: >>> use "setup.py easy_install . --no-deps" >> >> We've tried that but it fails because we need to install to our >> sandbox directory first. Currently we use this, so we need to be able >> to specify the root with your example: >> >> python setup.py install --root=${D} > > I'm confused. If you're specifying "install --root", you should be > getting --single-version-externally-managed, which means that > dependencies shouldn't be being installed in the first place. > Ah ha, I didn't know that was changed (in setuptools 0.6a11). The problem was for packages like elementtree and python 2.4, when their setup.py used distutils. So, that is not an issue since 0.6a11, great. There is one minor problem which will only happen when we miss a dependency. If a package has test_requires it will try to download and install packages when we run setup.py test. For instance in rdflib it will try to install nose. It would be nice to be able to make it abort instead, but that's not a big deal like install_requires pre-0.6a11 Thanks again, Rob From pje at telecommunity.com Tue Jul 3 22:25:44 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 03 Jul 2007 16:25:44 -0400 Subject: [Distutils] setuptools --no-dependencies option needed? In-Reply-To: <468AA78F.8000503@gentoo.org> References: <46892B49.2060300@gentoo.org> <20070702165628.E4EB93A4046@sparrow.telecommunity.com> <468A6C4F.4090607@gentoo.org> <20070703171224.D1C553A40A3@sparrow.telecommunity.com> <468AA78F.8000503@gentoo.org> Message-ID: <20070703202357.6C05A3A40A3@sparrow.telecommunity.com> At 12:46 PM 7/3/2007 -0700, Rob Cakebread wrote: >There is one minor problem which will only happen when we miss a >dependency. If a package has test_requires it will try to download >and install packages when we run setup.py test. For instance in >rdflib it will try to install nose. > >It would be nice to be able to make it abort instead, but that's >not a big deal like install_requires pre-0.6a11 Well, you can configure "--allow-hosts=localhost" for easy_install, e.g. by putting this in an appropriate distutils .cfg file: [easy_install] allow_hosts = localhost And then it won't download. However, I'm not sure if this is really a problem. Any eggs that get downloaded for tests aren't actually installed; they're just dumped in the setup.py directory and added to sys.path during the test run. They don't get installed with the package. From lists at zopyx.com Wed Jul 4 13:49:30 2007 From: lists at zopyx.com (Andreas Jung) Date: Wed, 04 Jul 2007 13:49:30 +0200 Subject: [Distutils] Looking for a presentation on Egg In-Reply-To: <46890F6C.3020704@colorstudy.com> References: <1B03F79999E9B184F0A42993@[192.168.0.24]> <46890F6C.3020704@colorstudy.com> Message-ID: --On 2. Juli 2007 09:45:00 -0500 Ian Bicking wrote: > It's pretty old, but I don't think much from it has changed: > http://svn.colorstudy.com/home/ianb/setuptools-presentation > > Feel free to take from it as you want > Thanks. This is an excellent starting point. Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 186 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070704/81356a99/attachment.pgp From TimurIzhbulatov at oilspace.com Wed Jul 4 15:14:45 2007 From: TimurIzhbulatov at oilspace.com (Timur Izhbulatov) Date: Wed, 04 Jul 2007 17:14:45 +0400 Subject: [Distutils] [PEAK] Dependencies before calling setup() In-Reply-To: <20070703155307.F225D3A40A9@sparrow.telecommunity.com> References: <468A5A48.6090101@oilspace.com> <20070703155307.F225D3A40A9@sparrow.telecommunity.com> Message-ID: <468B9D45.2040000@oilspace.com> Phillip J. Eby wrote: > At 06:16 PM 7/3/2007 +0400, Timur Izhbulatov wrote: >> Hi all, >> >> I use Cheetah in my project and want to precompile templates in my >> project's setup.py. So, I need to have Cheetah installed *before* >> setup() is called. > > Don't do that. setup scripts must *never* perform such actions before > setup() is called. What if someone is just running "setup.py --help", > after all? > > NEVER do this. easy_install's sandboxing code will also reject your > package as unsafe to run if you attempt something like this. Oops. A dirty hack :( >> I'm trying to use setuptools.command.easy_install.main(['Cheetah']) >> but I can't find any way to update sys.path after that. Is there any >> 'official' way to do this? > > The official way to require that a package is available during setup() > execution is viat the 'setup_requires' keyword, similar to the > 'install_requires' keyword. If you need the same package in both > situations, you must list it in both keywords. > > To actually do the precompile, you should subclass the install_data or > install_lib command, and override its run() method to do the > compilation, as well as doing whatever it did before. You use the > 'cmdclass' keyword to setup() to pass in a dictionary mapping command > names to your subclass(es). > > Your subclasses *must* honor the options they are given, and not attempt > to install anything except where they are told. The options will be > available as attributes of the instance created by the distutils. Thanks for the exhaustive explanation; it's very helpful. Subclassing install_lib worked just fine. Regards, -- Timur Izhbulatov OILspace, 26 Leninskaya sloboda, bld. 2, 2nd floor, 115280 Moscow, Russia P:+7 495 105 7245 + ext.205 F:+7 495 105 7246 E:TimurIzhbulatov at oilspace.com Building Successful Supply Chains - One Solution At A Time. www.oilspace.com From noah.gift at gmail.com Wed Jul 4 17:17:23 2007 From: noah.gift at gmail.com (Noah Gift) Date: Wed, 4 Jul 2007 11:17:23 -0400 Subject: [Distutils] Looking for a presentation on Egg In-Reply-To: References: <1B03F79999E9B184F0A42993@192.168.0.24> <46890F6C.3020704@colorstudy.com> Message-ID: That is a good presentation! I found this too: http://us.pycon.org/TX2007/PythonEggsCreation . I remember going to the egg talk at PyCon this year, but I don't remember if there were slides or not? On 7/4/07, Andreas Jung wrote: > > > > --On 2. Juli 2007 09:45:00 -0500 Ian Bicking wrote: > > > It's pretty old, but I don't think much from it has changed: > > http://svn.colorstudy.com/home/ianb/setuptools-presentation > > > > Feel free to take from it as you want > > > > Thanks. This is an excellent starting point. > > Andreas > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > > -- http://www.blog.noahgift.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070704/c5dc3650/attachment.htm From jtk at yahoo.com Wed Jul 4 23:18:57 2007 From: jtk at yahoo.com (Jeff Kowalczyk) Date: Wed, 04 Jul 2007 17:18:57 -0400 Subject: [Distutils] disable register and upload commands for a setup.py Message-ID: What would be the most reliable way to disable, with error output, the PyPI 'register' and/or 'upload' commands for a given setup.py? This is for developing certain python egg/packages having setup.py:setup() including full package metadata, but which should never (or not yet) accidentally be registered with PyPI. Thanks. From pje at telecommunity.com Thu Jul 5 01:09:02 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 04 Jul 2007 19:09:02 -0400 Subject: [Distutils] disable register and upload commands for a setup.py In-Reply-To: References: Message-ID: <20070704230653.DFC923A4046@sparrow.telecommunity.com> At 05:18 PM 7/4/2007 -0400, Jeff Kowalczyk wrote: >What would be the most reliable way to disable, with error output, the >PyPI 'register' and/or 'upload' commands for a given setup.py? See the 'cmdclass' argument to distutils.core.setup(). (You can use it to specify alternative implementations of 'register' and 'upload', that do whatever you want.) From mhammond at skippinet.com.au Thu Jul 5 05:12:00 2007 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 5 Jul 2007 13:12:00 +1000 Subject: [Distutils] bdist_wininst and Vista User Account Control (UAC) Message-ID: <052101c7beb2$41883490$0200a8c0@enfoldsystems.local> The pywin32 extensions require (well, prefer) administrative access during installation - certain files are copied to the System32 directory and the registry at HKEY_LOCAL_MACHINE is written to. Also, if I understand correctly, if Python happened to be installed into "\Program Files", admin access would be required to create any files in that directory tree - I'm not sure what permissions the \PythonXX directory are created with, but its not unreasable to assume that some shops might choose to secure that directory similarly to "\Program Files". The simplest way to achieve this for bdist_wininst installations is to include some magic in a "manifest". I've confirmed that once this magic is added, programs created by bdist_wininst get the little "shield" icon overlay and prompt for elevation before starting the executable. A problem here is that not all installations will require admin access - eg, a user who installed Python just for themselves will not need elevation to install an extension. A solution here would be for the installer to *not* be marked as requiring elevation, then sniffing the registry to make an educated guess (eg, HKLM\Software\Python\PythonCore\2.5 existing could indicate admin access is required). If it finds elevation is required, it would spawn another copy of itself with elevation requested, and terminate. This will have a side-effect of meaning the installer never gets the "shield" overlay, meaning the user will not be expecting to be prompted - but that is something we can live with. However, there is a complication here. Any "pure python" packages are not tied to a particular Python version, so the user can choose what installed Python version to use. Hence, in the general case, we can only determine if admin is required after the UI has opened and the user has selected the Python version. Arranging for the new "elevated" child process at this point will be (a) ugly, as the UI will vanish before the child process displays its GUI and (b) would require additional command-line processing logic - ie, passing the target directory to the child process. If we could make the determination *before* the GUI opens, it would appear seamless and would not require special command-line handling (the child process just does everything) So I see a few alternatives, but none are very desirable: * Only make this "admin required" check if a specific version of Python is necessary (ie, the package contains extension modules). This would leave pure-python packages out in the cold. * Live with the ugly UI that would result in performing that check after the Python version has been selected, and add the command-line processing necessary to make this work. * Ignore the issue and try to educate people that they must explicitly use "Run as Administrator" for such packages on Vista. I'm wondering if anyone has any opinions or thoughts on how we should handle this? Cheers, Mark From jim at zope.com Wed Jul 4 14:29:03 2007 From: jim at zope.com (Jim Fulton) Date: Wed, 4 Jul 2007 08:29:03 -0400 Subject: [Distutils] Preferring final releases Message-ID: <54BEA581-A262-42C5-9E2F-2DB00874CC99@zope.com> Currently, when looking for a new release, easy_install and buildout look for the newest distributions that satisfy a requirement. They will, for example, prefer a newer development release or preliminary release (dev, alpha, beta, or rc) over a final release. I don't think this is a good default policy. In buildout, I'm going to add an option, prefer-final, to control this. If this option is true, then buildout will prefer final distributions over non-final distributions. It will choose the latest final version that meets a requirement if there is one. Maybe buildout should make this the default. (Thoughts?) In any case, since buildout lets people create their own defaults for buildout options, people will be able to choose any default they wish. With this option enabled, you'll be able to get a non-final distribution by explicitly requiring it. I think easy_install and setuptools should provide a similar option and provide a way (presumably through setup.cfg) to make it the default. For backward compatibility reasons, it may not be possible to make this the default policy for setuptools, although, arguably, it is a better default. Jim Note that for buildout, what I said above isn't accurate. :) Buildout prefers develop distributions over non-develop distributions. It will currently choose a develop distribution that meets a requirement, if there is one. If prefer-final is enabled, then buildout will choose the latest develop distribution that satisfies the requirement, if there is one, otherwise, it will pick the latest final distribution that meets a requirement, if there is one, otherwise, it will pick the latest distribution that meets a requirement. -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Thu Jul 5 14:51:40 2007 From: jim at zope.com (Jim Fulton) Date: Thu, 5 Jul 2007 08:51:40 -0400 Subject: [Distutils] Buildout not updating setuptools In-Reply-To: References: Message-ID: <7481034A-8513-45F7-90E8-BBDE7B64BABA@zope.com> On Jul 2, 2007, at 8:08 AM, Graham Stratton wrote: Sorry to take so long to reply. I needed to find time to try to reproduce this. > Hi, I'm having a problem with buildout. > > My system has setuptools 0.6c5 installed. The buildout bootstrap.py > script > doesn't install the latest version, No, It will use the existing setuptools installation if there is one. > and nor does running bin/buildout. The > result is that the Routes install fails with > The required version of setuptools (>=0.6c6) is not available, ... I tried to reproduce this, but was unable to. > The problem seems to be that bootstrap.py creates a link for > setuptools in > develop-eggs. Why would that be? Is your existing installation of setuptools a develop egg? Buildout always prefers develop eggs that satisfies a requirement, so if you started with a develop egg of setuptools, it won't be upgraded to a later version unless you specifially request one. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From fmcingvale at gmail.com Thu Jul 5 16:26:12 2007 From: fmcingvale at gmail.com (Frank McIngvale) Date: Thu, 5 Jul 2007 09:26:12 -0500 Subject: [Distutils] setuptools: Problem with auto-scripts and standalone scripts Message-ID: <3c55fdbb0707050726l2ad1437ct30b0eb801bdb593f@mail.gmail.com> Hi, I'm trying to use setuptools auto-script feature with standalone scripts and am having problems. Say I have a script "mypkg/scripts/foo.py": --------------------- import sys print "Hello foo! My args are:",sys.argv --------------------- When I try wrapping it like this: 'console_scripts': [ 'foo = mypkg.scripts.foo', ] Then run like this: $ foo aa bb cc It appears to start OK (and it gets the cmdline args) but crashes with a traceback: ------------ Hello foo! My args are: ['c:\\frank\\py25\\Scripts\\foo-script.py', 'aa', 'bb', 'cc'] Traceback (most recent call last): File "c:\frank\py25\Scripts\foo-script.py", line 8, in load_entry_point('mypkg==0.99.50-rc1', 'console_scripts', 'foo')() TypeError: 'module' object is not callable ------------ Am I setting up 'console_scripts' incorrectly, or is this kind of usage not supported? thanks, Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070705/ac7dcc14/attachment.html From nathan at yergler.net Thu Jul 5 16:57:45 2007 From: nathan at yergler.net (Nathan R. Yergler) Date: Thu, 5 Jul 2007 07:57:45 -0700 Subject: [Distutils] setuptools: Problem with auto-scripts and standalone scripts In-Reply-To: <3c55fdbb0707050726l2ad1437ct30b0eb801bdb593f@mail.gmail.com> References: <3c55fdbb0707050726l2ad1437ct30b0eb801bdb593f@mail.gmail.com> Message-ID: Scripts need to address a callable, not a module. See http://peak.telecommunity.com/DevCenter/setuptools#automatic-script-creation and inline below for corrections. On 7/5/07, Frank McIngvale wrote: > Hi, I'm trying to use setuptools auto-script feature with standalone scripts > and am having problems. > > Say I have a script "mypkg/scripts/foo.py": > --------------------- > import sys > print "Hello foo! My args are:", sys.argv import sys def main(): print "Hello foo! My args are:", sys.argv" if __name__ == '__main__': # here for compatibility main() > --------------------- > > When I try wrapping it like this: > > 'console_scripts': [ > 'foo = mypkg.scripts.foo', > ] 'console_scripts': [ 'foo = mypkg.scripts.foo:main', ] > > Then run like this: > $ foo aa bb cc > > It appears to start OK (and it gets the cmdline args) but crashes with a > traceback: > ------------ > Hello foo! My args are: > ['c:\\frank\\py25\\Scripts\\foo-script.py', 'aa', 'bb', > 'cc'] > Traceback (most recent call last): > File "c:\frank\py25\Scripts\foo-script.py", line 8, in > load_entry_point('mypkg==0.99.50-rc1', 'console_scripts', 'foo')() > TypeError: 'module' object is not callable > ------------ > > Am I setting up 'console_scripts' incorrectly, or is this kind of usage not > supported? > > thanks, > Frank > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > From fmcingvale at gmail.com Thu Jul 5 17:55:20 2007 From: fmcingvale at gmail.com (Frank McIngvale) Date: Thu, 5 Jul 2007 10:55:20 -0500 Subject: [Distutils] setuptools: Problem with auto-scripts and standalone scripts In-Reply-To: References: <3c55fdbb0707050726l2ad1437ct30b0eb801bdb593f@mail.gmail.com> Message-ID: <3c55fdbb0707050855s4ba00660i3f28170d62912d0b@mail.gmail.com> Thanks! That did the trick. frank On 7/5/07, Nathan R. Yergler wrote: > > Scripts need to address a callable, not a module. See > > http://peak.telecommunity.com/DevCenter/setuptools#automatic-script-creation > and inline below for corrections. > > On 7/5/07, Frank McIngvale wrote: > > Hi, I'm trying to use setuptools auto-script feature with standalone > scripts > > and am having problems. > > > > Say I have a script "mypkg/scripts/foo.py": > > --------------------- > > import sys > > print "Hello foo! My args are:", sys.argv > > import sys > > def main(): > print "Hello foo! My args are:", sys.argv" > > if __name__ == '__main__': > # here for compatibility > main() > > > --------------------- > > > > When I try wrapping it like this: > > > > 'console_scripts': [ > > 'foo = mypkg.scripts.foo', > > ] > > 'console_scripts': [ > 'foo = mypkg.scripts.foo:main', > ] > > > > > Then run like this: > > $ foo aa bb cc > > > > It appears to start OK (and it gets the cmdline args) but crashes with a > > traceback: > > ------------ > > Hello foo! My args are: > > ['c:\\frank\\py25\\Scripts\\foo-script.py', 'aa', 'bb', > > 'cc'] > > Traceback (most recent call last): > > File "c:\frank\py25\Scripts\foo-script.py", line 8, in > > load_entry_point('mypkg==0.99.50-rc1', 'console_scripts', 'foo')() > > TypeError: 'module' object is not callable > > ------------ > > > > Am I setting up 'console_scripts' incorrectly, or is this kind of usage > not > > supported? > > > > thanks, > > Frank > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > http://mail.python.org/mailman/listinfo/distutils-sig > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070705/347dbf3c/attachment.htm From theller at ctypes.org Thu Jul 5 20:57:49 2007 From: theller at ctypes.org (Thomas Heller) Date: Thu, 05 Jul 2007 20:57:49 +0200 Subject: [Distutils] bdist_wininst and Vista User Account Control (UAC) In-Reply-To: <052101c7beb2$41883490$0200a8c0@enfoldsystems.local> References: <052101c7beb2$41883490$0200a8c0@enfoldsystems.local> Message-ID: Mark Hammond schrieb: > The pywin32 extensions require (well, prefer) administrative access during > installation - certain files are copied to the System32 directory and the > registry at HKEY_LOCAL_MACHINE is written to. Also, if I understand > correctly, if Python happened to be installed into "\Program Files", admin > access would be required to create any files in that directory tree - I'm > not sure what permissions the \PythonXX directory are created with, but its > not unreasable to assume that some shops might choose to secure that > directory similarly to "\Program Files". [...] > So I see a few alternatives, but none are very desirable: I have not yet even used vista, so I fear I cannot answer your questions, or even offer an opinion... > * Only make this "admin required" check if a specific version of Python is > necessary (ie, the package contains extension modules). This would leave > pure-python packages out in the cold. > > * Live with the ugly UI that would result in performing that check after the > Python version has been selected, and add the command-line processing > necessary to make this work. This sound like the most universal option to me. > * Ignore the issue and try to educate people that they must explicitly use > "Run as Administrator" for such packages on Vista. What if the user does not have or cannot get admin rights? Can he not install the package then? > I'm wondering if anyone has any opinions or thoughts on how we should handle > this? bdist_wininst has its own problems anyway - most severe the MSVC runtime dll issue. Another one is that it won't work for 64-bit installations. bdist_msi solves the 64-bit and MSVCRT issue. I would expect that it also solves the UAC problems. OTOH it is not possible (AFAIK) to build bdist_msi installers on Linux systems. Thomas From mhammond at skippinet.com.au Fri Jul 6 01:10:59 2007 From: mhammond at skippinet.com.au (Mark Hammond) Date: Fri, 6 Jul 2007 09:10:59 +1000 Subject: [Distutils] bdist_wininst and Vista User Account Control (UAC) In-Reply-To: Message-ID: <00a701c7bf59$c0ad5cd0$090a0a0a@enfoldsystems.local> Hi Thomas, > > So I see a few alternatives, but none are very desirable: > > I have not yet even used vista, so I fear I cannot answer > your questions, or even offer an opinion... > > > * Only make this "admin required" check if a specific > version of Python is > > necessary (ie, the package contains extension modules). > This would leave > > pure-python packages out in the cold. > > > > * Live with the ugly UI that would result in performing > that check after the > > Python version has been selected, and add the command-line > processing > > necessary to make this work. > > This sound like the most universal option to me. > > > * Ignore the issue and try to educate people that they must > explicitly use > > "Run as Administrator" for such packages on Vista. > > What if the user does not have or cannot get admin rights? > Can he not install the package then? That is correct - but that would be true regardless of how distutils tries to support it. Assuming we correctly detect that admin is required, my proposed changes would cause failure at the start of the install, rather than *trying* to install stuff and some stuff (but not necessarily all) failing, as would happen now. > > > I'm wondering if anyone has any opinions or thoughts on how > we should handle > > this? > > bdist_wininst has its own problems anyway - most severe the > MSVC runtime dll issue. What issue is this? Could it be solved by adding a manifest to the executable? It seems to me that Python itself built with VC2005 has this same issue. There is the problem of *installing* this library though, but if we assume Python itself has already done this, I don't see the problem for bdist_wininst. > Another one is that it won't work for 64-bit installations. My experiments show that very nearly works! I haven't actually got such a binary to install, but a 64bit bdist_wininst based installer correctly runs all the way to asking me to select an installed Python (where it stops as I don't have such an install yet). IIUC, to get this far, bdist_wininst has already sniffed out most data it needs. I could be wrong here, but I'm wondering what problems you had in mind? > bdist_msi solves the 64-bit and MSVCRT issue. I would expect > that it also solves the > UAC problems. OTOH it is not possible (AFAIK) to build > bdist_msi installers on Linux systems. Yeah, that may be what I need to use going forward - which basically would leave bdist_wininst to slowly die - but I guess that is good to know now, rather than after I spend more time trying to apply life-support to it... Cheers, Mark From jim at zope.com Fri Jul 6 13:50:00 2007 From: jim at zope.com (Jim Fulton) Date: Fri, 6 Jul 2007 07:50:00 -0400 Subject: [Distutils] pywin32 egg? In-Reply-To: <052101c7beb2$41883490$0200a8c0@enfoldsystems.local> References: <052101c7beb2$41883490$0200a8c0@enfoldsystems.local> Message-ID: Since you happen to be paying attention to distributions atm ... :) I would love to see a pywin32 egg, which by definition, didn't need to modify anything when it was installed. I guess there are some features in pywin32 that wouldn't work with such a setup, but I'm guessing that most features would. Even if it wasn't called pywin32, I'd really like to see an egg that contained the parts of pywin32 that can be used without modifying anything on installation. Please. Pretty please. :) It would make pywin32 much mroe usable for people using setuptools. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Fri Jul 6 20:09:34 2007 From: jim at zope.com (Jim Fulton) Date: Fri, 6 Jul 2007 14:09:34 -0400 Subject: [Distutils] Setuptools feature request: simplified version specification Message-ID: <40F297ED-A54E-4558-9854-EEE5126C5B5C@zope.com> We (me and a bunch of my Zope friends) find that we'll often want version specifications of the form: project_name >=Vdev, =1dev, <2dev") We think this is so common that we'd like a short-hand way of spelling it. I'll note that I'm not even sure I got the spelling above right. The intent is to request version 1, meaning any release of version 1. I think that's what I spelled above, although I'm not sure. If I got it wrong, maybe someone will correct me. Aside from the verbosity of the spelling above, I think the difficulty in spelling it is a strike against it. Note that a naive spelling: "foo >=1, <2" is wrong because it excludes pre-releases of 1 and includes pre-releases of 2. I propose that a valid version that ends in a number and that isn't preceded by an operator be a valid version specifier and be interpreted as a range. So, assuming that I know how to spell the range, a specification of: project_name V (e.g. "foo 1") would be equivalent to: project_name >=Vdev, =1dev, <2dev") This would work with multi-part versions, so "foo 1.2" would be equivalent to "foo >=1.2dev <1.3dev". Note that this could be combined with other version specifiers. For example, to require any version 1 or 2 of foo or versions 3.2 final or later: foo 1, 2, >=3.2 Also note that any version will do, so: foo 1.2a1 would be equivalent to: foo >=1.2a1dev <1.2a2dev And note that versions that don't end in numbers wouldn't be valid version specifiers, so: foo 1a would not be a valid specifier. Thoughts? Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From dpeterson at enthought.com Fri Jul 6 20:17:18 2007 From: dpeterson at enthought.com (Dave Peterson) Date: Fri, 06 Jul 2007 13:17:18 -0500 Subject: [Distutils] Setuptools feature request: simplified version specification In-Reply-To: <40F297ED-A54E-4558-9854-EEE5126C5B5C@zope.com> References: <40F297ED-A54E-4558-9854-EEE5126C5B5C@zope.com> Message-ID: <468E872E.3050006@enthought.com> Jim Fulton wrote: > We (me and a bunch of my Zope friends) find that we'll often want > version specifications of the form: > > project_name >=Vdev, =1dev, <2dev") > > We think this is so common that we'd like a short-hand way of > spelling it. > > I'll note that I'm not even sure I got the spelling above right. The > intent is to request version 1, meaning any release of version 1. I > think that's what I spelled above, although I'm not sure. If I got > it wrong, maybe someone will correct me. Aside from the verbosity of > the spelling above, I think the difficulty in spelling it is a strike > against it. Note that a naive spelling: "foo >=1, <2" is wrong > because it excludes pre-releases of 1 and includes pre-releases of 2. > > I hope you got it right because that's exactly the syntax we're using here at Enthought. Well, except that we use a '.dev' instead of a 'dev' :-) And I agree with you that getting it right isn't trivial. It tooks us a couple of attempts and exploring a couple of usecases to figure it out. > I propose that a valid version that ends in a number and that isn't > preceded by an operator be a valid version specifier and be > interpreted as a range. So, assuming that I know how to spell the > range, a specification of: > > project_name V (e.g. "foo 1") > > would be equivalent to: > > project_name >=Vdev, =1dev, <2dev") > > This would work with multi-part versions, so "foo 1.2" would be > equivalent to "foo >=1.2dev <1.3dev". > > Note that this could be combined with other version specifiers. For > example, to require any version 1 or 2 of foo or versions 3.2 final > or later: > > foo 1, 2, >=3.2 > > Also note that any version will do, so: > > foo 1.2a1 > > would be equivalent to: > > foo >=1.2a1dev <1.2a2dev > > And note that versions that don't end in numbers wouldn't be valid > version specifiers, so: > > foo 1a > > would not be a valid specifier. > I think this all sounds great, except that I'd made the last example valid by assuming it ended with a 0. i.e. foo 1a ==> foo >=1adev, <1a1dev This would be somewhat inline with the current pattern of expanding with zeros to say version 1 is equivalent to 1.0.0.0....0. -- Dave From ct at gocept.com Fri Jul 6 20:19:21 2007 From: ct at gocept.com (Christian Theune) Date: Fri, 06 Jul 2007 20:19:21 +0200 Subject: [Distutils] Setuptools feature request: simplified version specification In-Reply-To: <40F297ED-A54E-4558-9854-EEE5126C5B5C@zope.com> References: <40F297ED-A54E-4558-9854-EEE5126C5B5C@zope.com> Message-ID: <1183745961.9781.33.camel@mindy> Am Freitag, den 06.07.2007, 14:09 -0400 schrieb Jim Fulton: > We (me and a bunch of my Zope friends) find that we'll often want > version specifications of the form: > > project_name >=Vdev, =1dev, <2dev") > > We think this is so common that we'd like a short-hand way of > spelling it. > > [...] > > Thoughts? I've been using eggs for a while now and being able to specify a release series would be very helpful to me. I'm one of his Zope friends and I'm fond of the syntax he proposes. Christian From jim at zope.com Fri Jul 6 20:26:51 2007 From: jim at zope.com (Jim Fulton) Date: Fri, 6 Jul 2007 14:26:51 -0400 Subject: [Distutils] Setuptools feature request: simplified version specification In-Reply-To: <468E872E.3050006@enthought.com> References: <40F297ED-A54E-4558-9854-EEE5126C5B5C@zope.com> <468E872E.3050006@enthought.com> Message-ID: On Jul 6, 2007, at 2:17 PM, Dave Peterson wrote: ... > I think this all sounds great, except that I'd made the last example > valid by assuming it ended with a 0. i.e. > > foo 1a ==> foo >=1adev, <1a1dev > > This would be somewhat inline with the current pattern of expanding > with > zeros to say version 1 is equivalent to 1.0.0.0....0. Ah, good point, however, we'd still be treating versions ending in numbers differently from versions ending in letters, so it doesn't really simplify anything. After all, we wouldn't treat "1" as "1.0" and thus ">=1.0, <1.1". IMO, this is such an edge case that I don't really think it needs to be accommodated. OTOH, it's such an edge case, I don't care what we do with it. :) Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Fri Jul 6 20:38:00 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 06 Jul 2007 14:38:00 -0400 Subject: [Distutils] Setuptools feature request: simplified version specification In-Reply-To: <40F297ED-A54E-4558-9854-EEE5126C5B5C@zope.com> References: <40F297ED-A54E-4558-9854-EEE5126C5B5C@zope.com> Message-ID: <20070706184303.823BC3A4046@sparrow.telecommunity.com> At 02:09 PM 7/6/2007 -0400, Jim Fulton wrote: >We (me and a bunch of my Zope friends) find that we'll often want >version specifications of the form: > > project_name >=Vdev, =1dev, <2dev") > >We think this is so common that we'd like a short-hand way of >spelling it. > >I'll note that I'm not even sure I got the spelling above right. The >intent is to request version 1, meaning any release of version 1. I >think that's what I spelled above, although I'm not sure. If I got >it wrong, maybe someone will correct me. Aside from the verbosity of >the spelling above, I think the difficulty in spelling it is a strike >against it. Note that a naive spelling: "foo >=1, <2" is wrong >because it excludes pre-releases of 1 and includes pre-releases of 2. > >I propose that a valid version that ends in a number and that isn't >preceded by an operator be a valid version specifier and be >interpreted as a range. So, assuming that I know how to spell the >range, a specification of: > > project_name V (e.g. "foo 1") > >would be equivalent to: > > project_name >=Vdev, =1dev, <2dev") > >This would work with multi-part versions, so "foo 1.2" would be >equivalent to "foo >=1.2dev <1.3dev". > >Note that this could be combined with other version specifiers. For >example, to require any version 1 or 2 of foo or versions 3.2 final >or later: > > foo 1, 2, >=3.2 > >Also note that any version will do, so: > > foo 1.2a1 > >would be equivalent to: > > foo >=1.2a1dev <1.2a2dev > >And note that versions that don't end in numbers wouldn't be valid >version specifiers, so: > > foo 1a > >would not be a valid specifier. > >Thoughts? I'm not sure what I think of ' ' as the operator. This change would also have to be an 0.7 feature, as any package that uses it will be dependent on having a setuptools version that understands it, at least in order to build an egg. (We could have egg_info write the expanded forms, so that the built eggs' dependencies would be comprehensible to older versions.) I'm also not sure that '1a' can't/shouldn't be a valid specifier, although I suppose it would mean being equal to either ">= 1a.dev, <1b.dev" or ">= 1a0dev, <1a1dev", both of which are a little weird. :) From ianb at colorstudy.com Fri Jul 6 20:51:02 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 06 Jul 2007 13:51:02 -0500 Subject: [Distutils] Setuptools feature request: simplified version specification In-Reply-To: <20070706184303.823BC3A4046@sparrow.telecommunity.com> References: <40F297ED-A54E-4558-9854-EEE5126C5B5C@zope.com> <20070706184303.823BC3A4046@sparrow.telecommunity.com> Message-ID: <468E8F16.2030905@colorstudy.com> Phillip J. Eby wrote: > I'm not sure what I think of ' ' as the operator. This change would > also have to be an 0.7 feature, as any package that uses it will be > dependent on having a setuptools version that understands it, at > least in order to build an egg. (We could have egg_info write the > expanded forms, so that the built eggs' dependencies would be > comprehensible to older versions.) How about foo~=1.3, meaning foo>=1.3a,<1.4a ? It reads well to me, since it's like "about equal". -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org : Write code, do good : http://topp.openplans.org/careers From jim at zope.com Fri Jul 6 20:54:44 2007 From: jim at zope.com (Jim Fulton) Date: Fri, 6 Jul 2007 14:54:44 -0400 Subject: [Distutils] Setuptools feature request: simplified version specification In-Reply-To: <20070706184303.823BC3A4046@sparrow.telecommunity.com> References: <40F297ED-A54E-4558-9854-EEE5126C5B5C@zope.com> <20070706184303.823BC3A4046@sparrow.telecommunity.com> Message-ID: On Jul 6, 2007, at 2:38 PM, Phillip J. Eby wrote: > At 02:09 PM 7/6/2007 -0400, Jim Fulton wrote: >> We (me and a bunch of my Zope friends) find that we'll often want >> version specifications of the form: >> >> project_name >=Vdev, =1dev, <2dev") >> >> We think this is so common that we'd like a short-hand way of >> spelling it. >> >> I'll note that I'm not even sure I got the spelling above right. The >> intent is to request version 1, meaning any release of version 1. I >> think that's what I spelled above, although I'm not sure. If I got >> it wrong, maybe someone will correct me. Aside from the verbosity of >> the spelling above, I think the difficulty in spelling it is a strike >> against it. Note that a naive spelling: "foo >=1, <2" is wrong >> because it excludes pre-releases of 1 and includes pre-releases of 2. >> >> I propose that a valid version that ends in a number and that isn't >> preceded by an operator be a valid version specifier and be >> interpreted as a range. So, assuming that I know how to spell the >> range, a specification of: >> >> project_name V (e.g. "foo 1") >> >> would be equivalent to: >> >> project_name >=Vdev, =1dev, <2dev") >> >> This would work with multi-part versions, so "foo 1.2" would be >> equivalent to "foo >=1.2dev <1.3dev". >> >> Note that this could be combined with other version specifiers. For >> example, to require any version 1 or 2 of foo or versions 3.2 final >> or later: >> >> foo 1, 2, >=3.2 >> >> Also note that any version will do, so: >> >> foo 1.2a1 >> >> would be equivalent to: >> >> foo >=1.2a1dev <1.2a2dev >> >> And note that versions that don't end in numbers wouldn't be valid >> version specifiers, so: >> >> foo 1a >> >> would not be a valid specifier. >> >> Thoughts? > > I'm not sure what I think of ' ' as the operator. I wasn't suggesting adding '' as an operator. :) > This change would also have to be an 0.7 feature, as any package > that uses it will be dependent on having a setuptools version that > understands it, at least in order to build an egg. Yup. > (We could have egg_info write the expanded forms, so that the > built eggs' dependencies would be comprehensible to older versions.) That makes sense. Hm. That's interesting. I hadn't thought through the implication of that. I guess there are all sorts of things that can be done in a setup script to make this work now without changing setuptools. In the long term, I'd like to setuptools support this syntax, however, I suppose there are lots of opportunities to support it in the short term. I'll have to see what I can think of. > I'm also not sure that '1a' can't/shouldn't be a valid specifier, > although I suppose it would mean being equal to either ">= 1a.dev, > <1b.dev" or ">= 1a0dev, <1a1dev", both of which are a little > weird. :) As I mentioned in my response to Dave, I'm OK with anything since I think this is an extreme edge case, however I think it's best not to guess. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Fri Jul 6 20:58:55 2007 From: jim at zope.com (Jim Fulton) Date: Fri, 6 Jul 2007 14:58:55 -0400 Subject: [Distutils] Setuptools feature request: simplified version specification In-Reply-To: <468E8F16.2030905@colorstudy.com> References: <40F297ED-A54E-4558-9854-EEE5126C5B5C@zope.com> <20070706184303.823BC3A4046@sparrow.telecommunity.com> <468E8F16.2030905@colorstudy.com> Message-ID: <6C0ED629-7638-4296-B416-48C14D921EDB@zope.com> On Jul 6, 2007, at 2:51 PM, Ian Bicking wrote: > Phillip J. Eby wrote: >> I'm not sure what I think of ' ' as the operator. This change >> would also have to be an 0.7 feature, as any package that uses it >> will be dependent on having a setuptools version that understands >> it, at least in order to build an egg. (We could have egg_info >> write the expanded forms, so that the built eggs' dependencies >> would be comprehensible to older versions.) > > How about foo~=1.3, meaning foo>=1.3a,<1.4a ? It reads well to me, > since it's like "about equal". I think that "foo 1.3" is more readable than "~=1.3". Do we absolutely need an operator? If we absolutely need an operator, or if the majority of people here would prefer one, then I can live with one and would find it to be a big improvement to the current situation. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From mhammond at skippinet.com.au Sat Jul 7 07:20:18 2007 From: mhammond at skippinet.com.au (Mark Hammond) Date: Sat, 7 Jul 2007 15:20:18 +1000 Subject: [Distutils] pywin32 egg? In-Reply-To: Message-ID: <022d01c7c056$828bd0f0$090a0a0a@enfoldsystems.local> Hi Jim, > Since you happen to be paying attention to distributions atm ... :) > > I would love to see a pywin32 egg, which by definition, didn't need > to modify anything when it was installed. I guess there are some > features in pywin32 that wouldn't work with such a setup, but I'm > guessing that most features would. Even if it wasn't called > pywin32, > I'd really like to see an egg that contained the parts of pywin32 > that can be used without modifying anything on installation. > > Please. Pretty please. :) It would make pywin32 much mroe > usable for > people using setuptools. I'm sure you know the drill - patches are always welcome :) I've never modified a setup script to support eggs, and at this stage I'm more interested in getting existing functionality ported to x64 and Vista than creating a new distribution with reduced functionality. Also, if part of the functionality we had to drop was implementing COM objects (and we would if we couldn't install stuff into system32 or modify the HKLM part of the registry), then I'd be reluctant to call this a suitable distribution for anyone other than people wishing to package pywin32 with their own custom application, and I'd struggle to justify spending much time creating and supporting a distribution just for them. IOW, I'm certainly not opposed to the concept, but struggling to justify squeezing it into the time I have available for this stuff... Cheers, Mark From pje at telecommunity.com Sat Jul 7 14:29:35 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 07 Jul 2007 08:29:35 -0400 Subject: [Distutils] pywin32 egg? In-Reply-To: <022d01c7c056$828bd0f0$090a0a0a@enfoldsystems.local> References: <022d01c7c056$828bd0f0$090a0a0a@enfoldsystems.local> Message-ID: <20070707122727.4A95D3A4046@sparrow.telecommunity.com> At 03:20 PM 7/7/2007 +1000, Mark Hammond wrote: >Hi Jim, > > > Since you happen to be paying attention to distributions atm ... :) > > > > I would love to see a pywin32 egg, which by definition, didn't need > > to modify anything when it was installed. I guess there are some > > features in pywin32 that wouldn't work with such a setup, but I'm > > guessing that most features would. Even if it wasn't called > > pywin32, > > I'd really like to see an egg that contained the parts of pywin32 > > that can be used without modifying anything on installation. > > > > Please. Pretty please. :) It would make pywin32 much mroe > > usable for > > people using setuptools. > >I'm sure you know the drill - patches are always welcome :) I've never >modified a setup script to support eggs, and at this stage I'm more >interested in getting existing functionality ported to x64 and Vista than >creating a new distribution with reduced functionality. > >Also, if part of the functionality we had to drop was implementing COM >objects (and we would if we couldn't install stuff into system32 or modify >the HKLM part of the registry), then I'd be reluctant to call this a >suitable distribution for anyone other than people wishing to package >pywin32 with their own custom application, and I'd struggle to justify >spending much time creating and supporting a distribution just for them. I'm trying to remember for sure, but ISTR that I've successfully installed pywin32 in egg form. Ah yes, I did, and it's working for me in one of my Python installations. Don't remember how I did it though! [pause to try and install it with easy_install in another Python installation] Ah. Okay, first, pywin32 isn't registered on PyPI, so you need a -f link to find the files: easy_install -f http://sourceforge.net/project/showfiles.php?group_id=78018 pywin32 This is easily fixed by adding a PyPI listing with the above URL registered as the "Download URL". easy_install successfully installs the package, but in order to do the registry and copying stuff, the postinstall script has to be run: $ Scripts/pywin32_postinstall.py -install Traceback (most recent call last): File "Scripts\pywin32_postinstall.py", line 5, in ? pkg_resources.run_script('pywin32==210', 'pywin32_postinstall.py') File "c:\cygwin\home\pje\projects\setuptools-0.6\pkg_resources.py", line 448, in run_script self.require(requires)[0].run_script(script_name, ns) File "c:\cygwin\home\pje\projects\setuptools-0.6\pkg_resources.py", line 1166, in run_script execfile(script_filename, namespace, namespace) File "c:\cygwin\home\pje\chandlerstuff\chandler\release\bin\lib\site-packages\pywin32-210-py2.4-win32.egg\EGG-INFO\scripts\pywin32_postinstall.py", line 365, in ? install() File "c:\cygwin\home\pje\chandlerstuff\chandler\release\bin\lib\site-packages\pywin32-210-py2.4-win32.egg\EGG-INFO\scripts\pywin32_postinstall.py", line 155, in install LoadSystemModule(lib_dir, "pywintypes") File "c:\cygwin\home\pje\chandlerstuff\chandler\release\bin\lib\site-packages\pywin32-210-py2.4-win32.egg\EGG-INFO\scripts\pywin32_postinstall.py", line 94, in LoadSystemModule ('.dll', 'rb', imp.C_EXTENSION)) ImportError: DLL load failed: The specified module could not be found. The problem here is that lib_dir is calculated using the system configuration, instead of by locating where the package is actually installed. I changed the install() function like this: import timer lib_dir = os.path.dirname(timer.__file__) Importing some other pywin32 module than 'timer' would be okay; it just looked like something small and unlikely to depend on anything else. With this change, the install script produced this output:: $ Scripts/pywin32_postinstall.py` -install Copied pythoncom24.dll to C:\WINDOWS\system32\pythoncom24.dll Copied pywintypes24.dll to C:\WINDOWS\system32\pywintypes24.dll Registered: Python.Interpreter Registered: Python.Dictionary Registered: Python -> Software\Python\PythonCore\2.4\Help[None]=None -> Software\Python\PythonCore\2.4\Help\Pythonwin Reference[None]='C:\\...\\lib\\site-packages\\pywin32-210-py2.4-win32.egg\\PyWin32.chm' Creating directory C:\...\bin\lib\site-packages\pywin32-210-py2.4-win32.egg\win32com\gen_py The pywin32 extensions were successfully installed. Is there anything I can do to verify that this did in fact work correctly? It appears that all would be needed for pywin32 to be fully easy_install-able would be this change to the postinstall script (which shouldn't affect its functionality for direct installation), and an entry on PyPI. No setup script changed required -- or so it appears. Anyone wishing to use the features enabled by the postinstall script would just need to run it with -install. One question of course is, what features do you lose if you don't run the postinstall? If I hadn't run it, would I still have been able to use Python to access COM objects exposed by other applications, or would I have been unable to do COM at all? From faassen at startifact.com Sat Jul 7 14:22:28 2007 From: faassen at startifact.com (Martijn Faassen) Date: Sat, 07 Jul 2007 14:22:28 +0200 Subject: [Distutils] Preferring final releases In-Reply-To: <54BEA581-A262-42C5-9E2F-2DB00874CC99@zope.com> References: <54BEA581-A262-42C5-9E2F-2DB00874CC99@zope.com> Message-ID: Hey, Jim Fulton wrote: > I think easy_install and setuptools should provide a similar option > and provide a way (presumably through setup.cfg) to make it the > default. For backward compatibility reasons, it may not be possible > to make this the default policy for setuptools, although, arguably, > it is a better default. +1 for adding this option. Without such an option, it's pretty hard to specify you only want actual final releases, unless you tie down the release number entirely. A project has a harder time releasing betas of their next iteration as well, as people might be pulling it in before they are ready. A way to prevent this from happening would be to spell out exact release numbers for all dependencies in all the setup.py's. It's tricky to specify this with <, by the way: foo > 4.0 and foo < 4.1 will still get you 4.1b, which is what you frequently don't want. Regards, Martijn From ct at gocept.com Sat Jul 7 15:19:17 2007 From: ct at gocept.com (Christian Theune) Date: Sat, 07 Jul 2007 15:19:17 +0200 Subject: [Distutils] Preferring final releases In-Reply-To: References: <54BEA581-A262-42C5-9E2F-2DB00874CC99@zope.com> Message-ID: <1183814357.6121.14.camel@mindy> Am Samstag, den 07.07.2007, 14:22 +0200 schrieb Martijn Faassen: > +1 for adding this option. Without such an option, it's pretty hard to > specify you only want actual final releases, unless you tie down the > release number entirely. A project has a harder time releasing betas of > their next iteration as well, as people might be pulling it in before > they are ready. A way to prevent this from happening would be to spell > out exact release numbers for all dependencies in all the setup.py's. > [...] Me too. :) From jim at zope.com Sat Jul 7 17:24:32 2007 From: jim at zope.com (Jim Fulton) Date: Sat, 7 Jul 2007 11:24:32 -0400 Subject: [Distutils] pywin32 egg? In-Reply-To: <022d01c7c056$828bd0f0$090a0a0a@enfoldsystems.local> References: <022d01c7c056$828bd0f0$090a0a0a@enfoldsystems.local> Message-ID: <49BA7755-6B30-4214-B6BF-97D53C178DE9@zope.com> On Jul 7, 2007, at 1:20 AM, Mark Hammond wrote: > Hi Jim, > >> Since you happen to be paying attention to distributions atm ... :) >> >> I would love to see a pywin32 egg, which by definition, didn't need >> to modify anything when it was installed. I guess there are some >> features in pywin32 that wouldn't work with such a setup, but I'm >> guessing that most features would. Even if it wasn't called >> pywin32, >> I'd really like to see an egg that contained the parts of pywin32 >> that can be used without modifying anything on installation. >> >> Please. Pretty please. :) It would make pywin32 much mroe >> usable for >> people using setuptools. > > I'm sure you know the drill - patches are always welcome :) Unfortunately, I don't have sufficient Windows knowledge. > I've never > modified a setup script to support eggs, and at this stage You generally don't need to unless you want to be able to specify extra meta data. A standard distutils setup file should do. The main difference is that eggs are "zero installation", so someone installing an egg won't get any post-installation done automatically. ... > Also, if part of the functionality we had to drop was implementing COM > objects (and we would if we couldn't install stuff into system32 or > modify > the HKLM part of the registry), then I'd be reluctant to call this a > suitable distribution for anyone other than people wishing to package > pywin32 with their own custom application, and I'd struggle to justify > spending much time creating and supporting a distribution just for > them. I'm sorry to hear that. > IOW, I'm certainly not opposed to the concept, but struggling to > justify > squeezing it into the time I have available for this stuff... I might be willing to create such distributions, at least for our own use, and make them publicly available, much as I've done for Twisted, but I probably wouldn't do a great job. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From zooko at zooko.com Sat Jul 7 18:12:16 2007 From: zooko at zooko.com (Zooko O'Whielacronx) Date: Sat, 7 Jul 2007 10:12:16 -0600 Subject: [Distutils] .py vs .pyc (was: Newbie questions about setuptools) In-Reply-To: <20070619163156.CB7023A40A5@sparrow.telecommunity.com> References: <002c01c7b280$89df76b0$6a00a8c0@your4dacd0ea75> <20070619163156.CB7023A40A5@sparrow.telecommunity.com> Message-ID: <97DEE0D1-80EF-4D74-A783-1D66BDB6106F@zooko.com> On Jun 19, 2007, at 10:33 AM, Phillip J. Eby wrote: >> 2. The resulting egg file is much larger than the .zip file >> created by >> 'python setup.py sdist' in spite of exclude_package_data = { '': >> ['*.pyc','*.pyo']}, >> >> Is there any way to exclude .pyc and .pyo files? > > No; you can exclude the source if you like, though, with > --exclude-source-files. Eggs are a binary distribution format, > originally developed to support user-installed plugins for systems > like Chandler, Zope, etc. They aren't a source distribution format; > sdist works well enough for that and for easy_install if you have a > pure-Python package (or your users have C compilers). I would prefer to ship compiled C modules and source .py files. For one thing, compressed source .py files are usually around 25% smaller than compressed .pyc files. For another thing, I value making source available to the end-user, and I am loathe to ship non-source unless I really have to (which I pretty much do for C extension modules). So, please accept this as a feature request to make the above- mentioned behavior configurable for users such as me. Regards, Zooko From pje at telecommunity.com Sat Jul 7 19:04:47 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 07 Jul 2007 13:04:47 -0400 Subject: [Distutils] .py vs .pyc (was: Newbie questions about setuptools) In-Reply-To: <97DEE0D1-80EF-4D74-A783-1D66BDB6106F@zooko.com> References: <002c01c7b280$89df76b0$6a00a8c0@your4dacd0ea75> <20070619163156.CB7023A40A5@sparrow.telecommunity.com> <97DEE0D1-80EF-4D74-A783-1D66BDB6106F@zooko.com> Message-ID: <20070707170235.2194D3A405F@sparrow.telecommunity.com> At 10:12 AM 7/7/2007 -0600, Zooko O'Whielacronx wrote: >On Jun 19, 2007, at 10:33 AM, Phillip J. Eby wrote: > > >> 2. The resulting egg file is much larger than the .zip file > >> created by > >> 'python setup.py sdist' in spite of exclude_package_data = { '': > >> ['*.pyc','*.pyo']}, > >> > >> Is there any way to exclude .pyc and .pyo files? > > > > No; you can exclude the source if you like, though, with > > --exclude-source-files. Eggs are a binary distribution format, > > originally developed to support user-installed plugins for systems > > like Chandler, Zope, etc. They aren't a source distribution format; > > sdist works well enough for that and for easy_install if you have a > > pure-Python package (or your users have C compilers). > >I would prefer to ship compiled C modules and source .py files. For >one thing, compressed source .py files are usually around 25% smaller >than compressed .pyc files. For another thing, I value making source >available to the end-user, and I am loathe to ship non-source unless >I really have to (which I pretty much do for C extension modules). > >So, please accept this as a feature request to make the above- >mentioned behavior configurable for users such as me. For the reasons I already explained above, this is a no-go. If you want to ship binaries plus source, ship eggs. If you want to ship without .pyc/.pyo files, ship an sdist. This distinction is not about your preferences as a distributor, but about your users. As a user, I don't want to pay for .py-compilation every time I import your modules, just to save you a few extra bytes of bandwidth. If the option existed to ship without compiled modules, people would use it. Following which, other people would begin claiming that "eggs are slow". And all this, for what actual benefit? Absolutely zip! ...no pun intended. From gns24 at mythic-beasts.com Sat Jul 7 20:07:57 2007 From: gns24 at mythic-beasts.com (Graham Stratton) Date: Sat, 7 Jul 2007 19:07:57 +0100 Subject: [Distutils] Buildout not updating setuptools In-Reply-To: <7481034A-8513-45F7-90E8-BBDE7B64BABA@zope.com> References: <7481034A-8513-45F7-90E8-BBDE7B64BABA@zope.com> Message-ID: <37CA9D48-278C-44DF-AA4F-90C70F1AB349@mythic-beasts.com> Hi Jim, thanks for your time. >> and nor does running bin/buildout. The >> result is that the Routes install fails with >> The required version of setuptools (>=0.6c6) is not available, ... > > I tried to reproduce this, but was unable to. Here's what I did on a clean ubuntu server from the VMWare appliance site: #Install the system setuptools (0.6c5): $ sudo apt-get install subversion python-setuptools $ svn co svn://svn.zope.org/repos/main/zc.buildout/trunk/bootstrap/ . $ vi buildout.cfg ***** [buildout] parts = foxtrot [foxtrot] recipe = zc.recipe.egg interpreter = python2.5 eggs = routes ****** $ python2.5 bootstrap.py $ ls develop-eggs/ setuptools.egg-link $ cat develop-eggs/setuptools.egg-link /usr/lib/python2.5/site-packages $ ls eggs/ zc.buildout-1.0.0b28-py2.5.egg $./bin/buildout -v Installing 'zc.buildout', 'setuptools'. We have the best distribution that satisfies 'zc.buildout'. Picked: zc.buildout = 1.0.0b28 We have a develop egg: setuptools 0.6c5 Installing 'zc.recipe.egg'. We have the best distribution that satisfies 'zc.recipe.egg'. Picked: zc.recipe.egg = 1.0.0b6 Uninstalling foxtrot. Installing foxtrot. Installing 'routes'. We have no distributions for routes that satisfies 'routes'. Couldn't find index page for 'routes' (maybe misspelled?) Getting distribution for 'routes'. We have a develop egg: setuptools 0.6c5 Running easy_install: /usr/bin/python2.5 "-c" "from setuptools.command.easy_install import main; main()" "-mUNxd" "/home/notroot/eggs/tmp_IqDV_" "-q" "/tmp/ tmpDtCTlqget_dist/Routes-1.7.tar.gz" path=/usr/lib/python2.5/site-packages The required version of setuptools (>=0.6c6) is not available, and can't be installed while this script is running. Please install a more recent version first. (Currently using setuptools 0.6c5 (/usr/lib/python2.5/site-packages)) error: Setup script exited with 2 An error occured when trying to install Routes 1.7.Look above this message for any errors thatwere output by easy_install. While: Installing foxtrot. Getting distribution for 'routes'. Error: Couldn't install: Routes 1.7 I hope that's useful output/input, Graham From ianb at colorstudy.com Sat Jul 7 21:12:11 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Sat, 07 Jul 2007 14:12:11 -0500 Subject: [Distutils] [Web-SIG] entry points, etc In-Reply-To: <346386E9-FB03-4713-BCFB-35E95EC8DD99@zope.com> References: <468F0E9F.2030904@colorstudy.com> <346386E9-FB03-4713-BCFB-35E95EC8DD99@zope.com> Message-ID: <468FE58B.7070309@colorstudy.com> Jim Fulton wrote: > > On Jul 6, 2007, at 11:55 PM, Ian Bicking wrote: > >> Incidentally, something that would be nice with wsgiconfig is if we >> could all agree on how to specify things like entry points and objects. >> Specifically Paste Deploy uses egg:Distribution#ep_name, and >> zc.buildout uses Distribution:ep_name. And Paste Deploy defaults to >> ep_name=main while zc.buildout defaults to ep_name=default. > > Yup. > > Some notes. # is unattractive to me because it looks like a comment. > ConfigParser is a bit odd in it's treatment of #s. It treates them as > comments after empty lines and after section names, but not after comments. If ConfigParser did pay attention to comments it would make everything much more complicated. But that's another issue. > I used ":" because setuptools uses module:name when defining entry > points. That may not have been a good reason. Well, it could be good or bad overlap. Mostly it has to be something illegal in a distribution name, and ideally in any specifier (so you can do Package==0.5:ep_name). > If we agree on some standard, I'll support it. That should happen over > on the distutils-sig list. OK, copied over. >> Paste Deploy uses "entry_point_type = object.name" when you aren't using >> an entry point, but I'd like to switch to just "object.name" with an >> optional "object.name [ep_type]". This helps out those people who have >> some hangup with writing their own setup.py. So having a clear way to >> distinguish between an object reference and an entry point reference >> would be ideal. I still would prefer the entry points, as they make it >> easier to search the system for providing objects and easier to handle >> backward compatibility, but I don't have any reason to *require* entry >> points in my code generally. > > It is hard to assess this out of context. I will note that setuptools > uses dotted_module_name:dotted_object_name to name objects and I'd be > inclined to be consistent with that. If we used a different delimiter > between eggs and entry points and between modules and names, then we'd > be able to tell the refrecnes apart. Again, I think this is more > general than web applications. Another case came up as I was thinking about this, which is referring to an object in a specific Python file (not module). I find this very useful when I'm doing some kind of configuration that involves deployment-specific code. I really don't want to create a package and module and fiddle with sys.path, for a bit of code that is purely deployment specific. So I'd like to also allow for a specifier like filename:object_name. We could use prefixes for all of these. I'm fine supporting : instead of # for entry point names. We could use prefixes, e.g., "egg:Package:ep". Somehow that reads a little funny to me, too many colons. We could also just use "egg Package:ep", which maybe reads better. Then there'd also be "python module:object", and "file filename.py:object". Maybe "python " would be the default, I'm not sure. I encourage people to use entry points, but I don't mind including "egg " -- if anything, I think the name is helpful to people who aren't always clear on the distinction between distributions and packages and modules. The default entry point name is also important. I prefer "main" because I think it better describes the purpose of the entry-point-that-need-not-be-named. "default" feels like it's just describing the mechanism of the entry point loader. -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org : Write code, do good : http://topp.openplans.org/careers From fdrake at gmail.com Sat Jul 7 21:41:39 2007 From: fdrake at gmail.com (Fred Drake) Date: Sat, 7 Jul 2007 15:41:39 -0400 Subject: [Distutils] [Web-SIG] entry points, etc In-Reply-To: <468FE58B.7070309@colorstudy.com> References: <468F0E9F.2030904@colorstudy.com> <346386E9-FB03-4713-BCFB-35E95EC8DD99@zope.com> <468FE58B.7070309@colorstudy.com> Message-ID: <9cee7ab80707071241k2f4c2fb6u9cefe9e5980a27b9@mail.gmail.com> On 7/7/07, Ian Bicking wrote: > Then there'd also be "python module:object", and "file > filename.py:object". Maybe "python " would be the default, I'm not > sure. I'd be in favor of not having a default expression type for this, but to require it to be spelled out every time. -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From jim at zope.com Sun Jul 8 13:17:51 2007 From: jim at zope.com (Jim Fulton) Date: Sun, 8 Jul 2007 07:17:51 -0400 Subject: [Distutils] Buildout not updating setuptools In-Reply-To: <37CA9D48-278C-44DF-AA4F-90C70F1AB349@mythic-beasts.com> References: <7481034A-8513-45F7-90E8-BBDE7B64BABA@zope.com> <37CA9D48-278C-44DF-AA4F-90C70F1AB349@mythic-beasts.com> Message-ID: <66366D92-429D-42B0-88B7-34DAA231337E@zope.com> Oh system Pythons. I can only reproduce this using the ubuntu python-setuptools package. Installing setuptools myself using ez_setup doesn't give this behavior. The Unbuntu python-setuptools package installation is rather odd. It doesn't install setuptools as a normal egg. Rather it installs it in such a way that it shows up as a develop egg. jim at starbase14:~/tmp$ python Python 2.5.1 (r251:54863, May 2 2007, 16:56:35) [GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import pkg_resources >>> pkg_resources.working_set.find(pkg_resources.Requirement.parse ('setuptools')) setuptools 0.6c5 (/usr/lib/python2.5/site-packages) Note that the location of the setuptools egg is site packages. >>> d = pkg_resources.working_set.find (pkg_resources.Requirement.parse('setuptools')) >>> d.precedence == pkg_resources.DEVELOP_DIST True So, the Ubuntu packager for setuptools decided not to install setuptools as an actual egg, but as a wonky sort-of develop egg with a less than ideal location. IMO, this is just wrong and broken. Does anyone disagree? I could probably add logic to the bootstrap script to get around this, but I'm not very motivated to do so. I'm inclined to say that if you choose to use this version of setuptools, by using your system Python and a system package of setuptools, then you'll have to live with its limitations. Of course, your system easy_install won't install routs either: jim at starbase14:~/tmp$ sudo easy_install routes Password: Searching for routes Reading http://cheeseshop.python.org/pypi/routes/ Couldn't find index page for 'routes' (maybe misspelled?) Scanning index of all packages (this may take a while) Reading http://cheeseshop.python.org/pypi/ Reading http://cheeseshop.python.org/pypi/Routes/1.7 Reading http://routes.groovie.org/ Best match: Routes 1.7 Downloading http://cheeseshop.python.org/packages/source/R/Routes/ Routes- 1.7.tar.gz#md5=6b8892b8aef7495228376f4e8b6c747b Processing Routes-1.7.tar.gz Running Routes-1.7/setup.py -q bdist_egg --dist-dir /tmp/ easy_install-goT506/Routes-1.7/egg-dist-tmp-fT_EqK The required version of setuptools (>=0.6c6) is not available, and can't be installed while this script is running. Please install a more recent version first. (Currently using setuptools 0.6c5 (/usr/lib/python2.5/site-packages)) error: Setup script exited with 2 The good news is that setuptools can repair itself: sudo easy_install -U setuptools Doing this causes setuptools to upgrade itself to a regular egg-based install. At this point (at least on my system), I can install routes and, when I run buildout, it doesn't create an egg link in the develo- eggs directory. Of course, at that point, you've probably messed up your system packaging I will probably do something about this in the long run though just to avoid the question coming up. Jim On Jul 7, 2007, at 2:07 PM, Graham Stratton wrote: > Hi Jim, thanks for your time. > >>> and nor does running bin/buildout. The >>> result is that the Routes install fails with >>> The required version of setuptools (>=0.6c6) is not available, ... >> >> I tried to reproduce this, but was unable to. > > Here's what I did on a clean ubuntu server from the VMWare appliance > site: > > #Install the system setuptools (0.6c5): > $ sudo apt-get install subversion python-setuptools > $ svn co svn://svn.zope.org/repos/main/zc.buildout/trunk/bootstrap/ . > > $ vi buildout.cfg > ***** > [buildout] > parts = foxtrot > > [foxtrot] > recipe = zc.recipe.egg > interpreter = python2.5 > eggs = routes > ****** > > $ python2.5 bootstrap.py > > $ ls develop-eggs/ > setuptools.egg-link > $ cat develop-eggs/setuptools.egg-link > /usr/lib/python2.5/site-packages > $ ls eggs/ > zc.buildout-1.0.0b28-py2.5.egg > > $./bin/buildout -v > Installing 'zc.buildout', 'setuptools'. > We have the best distribution that satisfies 'zc.buildout'. > Picked: zc.buildout = 1.0.0b28 > We have a develop egg: setuptools 0.6c5 > Installing 'zc.recipe.egg'. > We have the best distribution that satisfies 'zc.recipe.egg'. > Picked: zc.recipe.egg = 1.0.0b6 > Uninstalling foxtrot. > Installing foxtrot. > Installing 'routes'. > We have no distributions for routes that satisfies 'routes'. > Couldn't find index page for 'routes' (maybe misspelled?) > Getting distribution for 'routes'. > We have a develop egg: setuptools 0.6c5 > Running easy_install: > /usr/bin/python2.5 "-c" "from setuptools.command.easy_install import > main; main()" "-mUNxd" "/home/notroot/eggs/tmp_IqDV_" "-q" "/tmp/ > tmpDtCTlqget_dist/Routes-1.7.tar.gz" > path=/usr/lib/python2.5/site-packages > > The required version of setuptools (>=0.6c6) is not available, and > can't be installed while this script is running. Please install > a more recent version first. > > (Currently using setuptools 0.6c5 (/usr/lib/python2.5/site-packages)) > error: Setup script exited with 2 > An error occured when trying to install Routes 1.7.Look above this > message for any errors thatwere output by easy_install. > While: > Installing foxtrot. > Getting distribution for 'routes'. > Error: Couldn't install: Routes 1.7 > > > > I hope that's useful output/input, > > Graham > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Sun Jul 8 18:56:20 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 08 Jul 2007 12:56:20 -0400 Subject: [Distutils] Buildout not updating setuptools In-Reply-To: <66366D92-429D-42B0-88B7-34DAA231337E@zope.com> References: <7481034A-8513-45F7-90E8-BBDE7B64BABA@zope.com> <37CA9D48-278C-44DF-AA4F-90C70F1AB349@mythic-beasts.com> <66366D92-429D-42B0-88B7-34DAA231337E@zope.com> Message-ID: <20070708165407.6209D3A404D@sparrow.telecommunity.com> At 07:17 AM 7/8/2007 -0400, Jim Fulton wrote: >So, the Ubuntu packager for setuptools decided not to install >setuptools as an actual egg, but as a wonky sort-of develop egg with >a less than ideal location. > >IMO, this is just wrong and broken. Does anyone disagree? Yes, because it's fine for them to install it as a develop egg; that's how system packages are *supposed* to be installed. In other words, it's not wonky, unless they also changed something besides just installing it .egg-info style. Thus, if buildout is assuming that "DEVELOP_DIST" egg paths contain only modules or packages that are part of that egg, it is broken. Note, for example, that as of Python 2.5, the distutils install *all* packages with an .egg-info file, which is detected by pkg_resources as a DEVELOP_DIST. A plain old Python 2.5 install with stock system packages will be chock full of develop-style eggs, in other words, even if the packages in question didn't use setuptools at all. This is by design. If you want to be able to figure out if something is really a "develop" installation, you need to look for an .egg-link file, not the mere existence of a DEVELOP_DIST egg. Only the "develop" command writes .egg-link files. >The good news is that setuptools can repair itself: > > sudo easy_install -U setuptools > >Doing this causes setuptools to upgrade itself to a regular egg-based >install. At this point (at least on my system), I can install routes >and, when I run buildout, it doesn't create an egg link in the develo- >eggs directory. Of course, at that point, you've probably messed up >your system packaging Installing setuptools to a $PYTHONPATH directory should work fine... but might not, depending on whether the system setuptools has a 'site.py' included, or whether it was removed by some busy-body. The next 0.6 version will have a workaround for this so that you can still do $PYTHONPATH installation even if the system setuptools was hacked on by an over-eager packager thinking that a 'site.py' shouldn't be in site-packages. :) From jim at zope.com Mon Jul 9 13:33:50 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 9 Jul 2007 07:33:50 -0400 Subject: [Distutils] Buildout not updating setuptools In-Reply-To: <20070708165407.6209D3A404D@sparrow.telecommunity.com> References: <7481034A-8513-45F7-90E8-BBDE7B64BABA@zope.com> <37CA9D48-278C-44DF-AA4F-90C70F1AB349@mythic-beasts.com> <66366D92-429D-42B0-88B7-34DAA231337E@zope.com> <20070708165407.6209D3A404D@sparrow.telecommunity.com> Message-ID: <8EFFAD5D-A571-47D3-A5E4-A242466B5F55@zope.com> On Jul 8, 2007, at 12:56 PM, Phillip J. Eby wrote: > At 07:17 AM 7/8/2007 -0400, Jim Fulton wrote: >> So, the Ubuntu packager for setuptools decided not to install >> setuptools as an actual egg, but as a wonky sort-of develop egg with >> a less than ideal location. >> >> IMO, this is just wrong and broken. Does anyone disagree? > > Yes, because it's fine for them to install it as a develop egg; > that's how system packages are *supposed* to be installed. Really? Why? If this is true, then "develop" seems to be a misnomer. > In other words, it's not wonky, unless they also changed something > besides just installing it .egg-info style. > > Thus, if buildout is assuming that "DEVELOP_DIST" egg paths contain > only modules or packages that are part of that egg, it is broken. > > Note, for example, that as of Python 2.5, the distutils install > *all* packages with an .egg-info file, which is detected by > pkg_resources as a DEVELOP_DIST. A plain old Python 2.5 install > with stock system packages will be chock full of develop-style > eggs, in other words, even if the packages in question didn't use > setuptools at all. > > This is by design. > > If you want to be able to figure out if something is really a > "develop" installation, you need to look for an .egg-link file, not > the mere existence of a DEVELOP_DIST egg. Only the "develop" > command writes .egg-link files. But you can't look for an egg-link file if all you have is a distribution. The dist location points to the contents of the egg link file. How is one to determine, given a DEVELOP_DIS egg in an environment whether it was created by the develop command? Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Mon Jul 9 17:42:48 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 09 Jul 2007 11:42:48 -0400 Subject: [Distutils] Buildout not updating setuptools In-Reply-To: <8EFFAD5D-A571-47D3-A5E4-A242466B5F55@zope.com> References: <7481034A-8513-45F7-90E8-BBDE7B64BABA@zope.com> <37CA9D48-278C-44DF-AA4F-90C70F1AB349@mythic-beasts.com> <66366D92-429D-42B0-88B7-34DAA231337E@zope.com> <20070708165407.6209D3A404D@sparrow.telecommunity.com> <8EFFAD5D-A571-47D3-A5E4-A242466B5F55@zope.com> Message-ID: <20070709154035.CB9083A404D@sparrow.telecommunity.com> At 07:33 AM 7/9/2007 -0400, Jim Fulton wrote: >On Jul 8, 2007, at 12:56 PM, Phillip J. Eby wrote: > >>At 07:17 AM 7/8/2007 -0400, Jim Fulton wrote: >>>So, the Ubuntu packager for setuptools decided not to install >>>setuptools as an actual egg, but as a wonky sort-of develop egg with >>>a less than ideal location. >>> >>>IMO, this is just wrong and broken. Does anyone disagree? >> >>Yes, because it's fine for them to install it as a develop egg; >>that's how system packages are *supposed* to be installed. > >Really? Why? If this is true, then "develop" seems to be a misnomer. Yep - which is why I've been referring to them as "egg-info" or "single version, externally managed" eggs. :) The code still uses one symbol for DEVELOP_DIST because the selection precedence of such eggs is the same, regardless of whether they're generated by "install" or "develop". >>If you want to be able to figure out if something is really a >>"develop" installation, you need to look for an .egg-link file, not >>the mere existence of a DEVELOP_DIST egg. Only the "develop" >>command writes .egg-link files. > >But you can't look for an egg-link file if all you have is a >distribution. The dist location points to the contents of the egg >link file. How is one to determine, given a DEVELOP_DIS egg in an >environment whether it was created by the develop command? I didn't say it was easy. ;) You'd have to check for .egg-link files along the environment path, and see if there's one that points to the distribution in question. One possible distinction is that distutils generates .egg-info *files* with version information in the file name, and setuptools "install" and bdist packages include the version information in the directory name by default. So you can rule out any egg for which a versioned filename exists as not being "develop" targets. Unfortunately, you can't assume the reverse: just because it's an .egg-info directory and *doesn't* have any version information, you don't know for sure that it was generated by develop. It could be a system package produced by one of those people who likes to tinker, and decided to remove the version info from the directory name for some crazy reason. OTOH, you could just say the heck with 'em. :) You can easily check whether the .egg-info is a file: if it's a DEVELOP_DIST, and ``dist.list_metadata('')`` contains 1 or 0 entries, it's a metadata file, and definitely not a develop. In order to check the directory version info, you'll have to steal a peak at the ``dist._provider.egg_info`` path and see if it's got a '-' in it before the '.'. If yes, it's not a "develop"-generated egg. From jim at zope.com Mon Jul 9 19:10:33 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 9 Jul 2007 13:10:33 -0400 Subject: [Distutils] Buildout not updating setuptools In-Reply-To: <20070709154035.CB9083A404D@sparrow.telecommunity.com> References: <7481034A-8513-45F7-90E8-BBDE7B64BABA@zope.com> <37CA9D48-278C-44DF-AA4F-90C70F1AB349@mythic-beasts.com> <66366D92-429D-42B0-88B7-34DAA231337E@zope.com> <20070708165407.6209D3A404D@sparrow.telecommunity.com> <8EFFAD5D-A571-47D3-A5E4-A242466B5F55@zope.com> <20070709154035.CB9083A404D@sparrow.telecommunity.com> Message-ID: On Jul 9, 2007, at 11:42 AM, Phillip J. Eby wrote: > At 07:33 AM 7/9/2007 -0400, Jim Fulton wrote: > >> On Jul 8, 2007, at 12:56 PM, Phillip J. Eby wrote: >> >>> At 07:17 AM 7/8/2007 -0400, Jim Fulton wrote: >>>> So, the Ubuntu packager for setuptools decided not to install >>>> setuptools as an actual egg, but as a wonky sort-of develop egg >>>> with >>>> a less than ideal location. >>>> >>>> IMO, this is just wrong and broken. Does anyone disagree? >>> >>> Yes, because it's fine for them to install it as a develop egg; >>> that's how system packages are *supposed* to be installed. >> >> Really? Why? If this is true, then "develop" seems to be a misnomer. > > Yep - which is why I've been referring to them as "egg-info" or > "single version, externally managed" eggs. :) > > The code still uses one symbol for DEVELOP_DIST because the > selection precedence of such eggs is the same, regardless of > whether they're generated by "install" or "develop". IMO, they shouldn't have the same precedence. IMO installations made with the develop command should have the highest precedence. After all, why would someone use the develop command if they didn't want to see the effects of changes they're making in development. Could we can change the develop command to generate a different symbol, like REALLY_DEVELOP_DIST and give it the highest precedence? (Tempted to wink, but not really kidding...) It appears that it doesn't actually generate anything atm, but I think it should. >>> If you want to be able to figure out if something is really a >>> "develop" installation, you need to look for an .egg-link file, not >>> the mere existence of a DEVELOP_DIST egg. Only the "develop" >>> command writes .egg-link files. >> >> But you can't look for an egg-link file if all you have is a >> distribution. The dist location points to the contents of the egg >> link file. How is one to determine, given a DEVELOP_DIS egg in an >> environment whether it was created by the develop command? > > I didn't say it was easy. ;) You'd have to check for .egg-link > files along the environment path, and see if there's one that > points to the distribution in question. > > One possible distinction is that distutils generates .egg-info > *files* with version information in the file name, and setuptools > "install" and bdist packages include the version information in the > directory name by default. So you can rule out any egg for which a > versioned filename exists as not being "develop" targets. > > Unfortunately, you can't assume the reverse: just because it's > an .egg-info directory and *doesn't* have any version information, > you don't know for sure that it was generated by develop. It could > be a system package produced by one of those people who likes to > tinker, and decided to remove the version info from the directory > name for some crazy reason. OTOH, you could just say the heck with > 'em. :) > > You can easily check whether the .egg-info is a file: if it's a > DEVELOP_DIST, and ``dist.list_metadata('')`` contains 1 or 0 > entries, it's a metadata file, and definitely not a develop. > > In order to check the directory version info, you'll have to steal > a peak at the ``dist._provider.egg_info`` path and see if it's got > a '-' in it before the '.'. If yes, it's not a "develop"-generated > egg. So I guess there's nothing actually stored in the egg info do indicate that it was built with develop. I'll have to think about what this means for buildout. I suppose I should simply treat anything in the buildout's develop-eggs directory to be deserving of special treatment and ignore the precedence. Things will only get put there when buildout invokes the develop command or by the custom egg builder, whos output should get special treatment that it''s not getting now. Thanks for, um, clarifying this for me. :) Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Mon Jul 9 19:45:08 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 09 Jul 2007 13:45:08 -0400 Subject: [Distutils] Buildout not updating setuptools In-Reply-To: References: <7481034A-8513-45F7-90E8-BBDE7B64BABA@zope.com> <37CA9D48-278C-44DF-AA4F-90C70F1AB349@mythic-beasts.com> <66366D92-429D-42B0-88B7-34DAA231337E@zope.com> <20070708165407.6209D3A404D@sparrow.telecommunity.com> <8EFFAD5D-A571-47D3-A5E4-A242466B5F55@zope.com> <20070709154035.CB9083A404D@sparrow.telecommunity.com> Message-ID: <20070709174254.788833A404D@sparrow.telecommunity.com> At 01:10 PM 7/9/2007 -0400, Jim Fulton wrote: >IMO, they shouldn't have the same precedence. IMO installations made >with the develop command should have the highest precedence. After >all, why would someone use the develop command if they didn't want to >see the effects of changes they're making in development. Could we >can change the develop command to generate a different symbol, like >REALLY_DEVELOP_DIST and give it the highest precedence? (Tempted to >wink, but not really kidding...) It appears that it doesn't actually >generate anything atm, but I think it should. I could have .egg-link files use a different precedence, but I wouldn't make it the highest. Develop eggs have the lowest precedence for two reasons: one, easy_install prefers to use eggs it can copy, and two, develop eggs have lower precedence to prevent you thinking that your version requirements are correct when they aren't. That is, if you depend on 1.9.dev-r1201, and that picks up your development version in place of an .egg file with the same version, you won't be able to tell that your requirement is wrong. This can occur if you are working on an SVN checkout - your develop egg has the same version as the built one, but you could have changed it. This was actually more of a happy coincidence than something I planned ahead of time, but it does occur in practice. :) Note that in any event, if we make platform-specific eggs have higher precedence, then platform-specific built eggs will have to take precedence over develop eggs, because "true" develop eggs are never viewed as "platform-specific" (because they have no platform name in their .egg-info directory name). From jim at zope.com Mon Jul 9 20:09:46 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 9 Jul 2007 14:09:46 -0400 Subject: [Distutils] Buildout not updating setuptools In-Reply-To: <20070709174254.788833A404D@sparrow.telecommunity.com> References: <7481034A-8513-45F7-90E8-BBDE7B64BABA@zope.com> <37CA9D48-278C-44DF-AA4F-90C70F1AB349@mythic-beasts.com> <66366D92-429D-42B0-88B7-34DAA231337E@zope.com> <20070708165407.6209D3A404D@sparrow.telecommunity.com> <8EFFAD5D-A571-47D3-A5E4-A242466B5F55@zope.com> <20070709154035.CB9083A404D@sparrow.telecommunity.com> <20070709174254.788833A404D@sparrow.telecommunity.com> Message-ID: <041C24EC-4234-4CBD-9673-C34470030366@zope.com> On Jul 9, 2007, at 1:45 PM, Phillip J. Eby wrote: > At 01:10 PM 7/9/2007 -0400, Jim Fulton wrote: >> IMO, they shouldn't have the same precedence. IMO installations made >> with the develop command should have the highest precedence. After >> all, why would someone use the develop command if they didn't want to >> see the effects of changes they're making in development. Could we >> can change the develop command to generate a different symbol, like >> REALLY_DEVELOP_DIST and give it the highest precedence? (Tempted to >> wink, but not really kidding...) It appears that it doesn't actually >> generate anything atm, but I think it should. > > I could have .egg-link files use a different precedence, but I > wouldn't make it the highest. Develop eggs have the lowest > precedence for two reasons: one, easy_install prefers to use eggs > it can copy, Why does this matter to users? > and two, develop eggs have lower precedence to prevent you thinking > that your version requirements are correct when they aren't. > > That is, if you depend on 1.9.dev-r1201, and that picks up your > development version in place of an .egg file with the same version, > you won't be able to tell that your requirement is wrong. This can > occur if you are working on an SVN checkout - your develop egg has > the same version as the built one, but you could have changed it. That's a fair argument. I think it is just as bad to be making changes to an svn checkout and having those changes have no effect. I suppose that any time we select among two distributions with the same version based on precedence, we should issue a warning, at least when one of them is a develop dist. > This was actually more of a happy coincidence than something I > planned ahead of time, but it does occur in practice. :) It made me very unhappy when I first started working with setuptools. I use develop eggs a lot and I definately expect them to be used. > Note that in any event, if we make platform-specific eggs have > higher precedence, then platform-specific built eggs will have to > take precedence over develop eggs, because "true" develop eggs are > never viewed as "platform-specific" (because they have no platform > name in their .egg-info directory name). This argument sounds overly pedantic to me. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Mon Jul 9 20:41:02 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 09 Jul 2007 14:41:02 -0400 Subject: [Distutils] Buildout not updating setuptools In-Reply-To: <041C24EC-4234-4CBD-9673-C34470030366@zope.com> References: <7481034A-8513-45F7-90E8-BBDE7B64BABA@zope.com> <37CA9D48-278C-44DF-AA4F-90C70F1AB349@mythic-beasts.com> <66366D92-429D-42B0-88B7-34DAA231337E@zope.com> <20070708165407.6209D3A404D@sparrow.telecommunity.com> <8EFFAD5D-A571-47D3-A5E4-A242466B5F55@zope.com> <20070709154035.CB9083A404D@sparrow.telecommunity.com> <20070709174254.788833A404D@sparrow.telecommunity.com> <041C24EC-4234-4CBD-9673-C34470030366@zope.com> Message-ID: <20070709183849.6E89D3A404D@sparrow.telecommunity.com> At 02:09 PM 7/9/2007 -0400, Jim Fulton wrote: >On Jul 9, 2007, at 1:45 PM, Phillip J. Eby wrote: > >>At 01:10 PM 7/9/2007 -0400, Jim Fulton wrote: >>>IMO, they shouldn't have the same precedence. IMO installations made >>>with the develop command should have the highest precedence. After >>>all, why would someone use the develop command if they didn't want to >>>see the effects of changes they're making in development. Could we >>>can change the develop command to generate a different symbol, like >>>REALLY_DEVELOP_DIST and give it the highest precedence? (Tempted to >>>wink, but not really kidding...) It appears that it doesn't actually >>>generate anything atm, but I think it should. >> >>I could have .egg-link files use a different precedence, but I >>wouldn't make it the highest. Develop eggs have the lowest >>precedence for two reasons: one, easy_install prefers to use eggs >>it can copy, > >Why does this matter to users? It doesn't! -- that's precisely why easy_install is free to pick the one that's easier for it. :) >I suppose that any time we select among two distributions with the >same version based on precedence, we should issue a warning, at least >when one of them is a develop dist. That might indeed be doable... at least for tools like buildout and easy_install. I'm not sure it's worth doing so in pkg_resources. >>This was actually more of a happy coincidence than something I >>planned ahead of time, but it does occur in practice. :) > >It made me very unhappy when I first started working with >setuptools. I use develop eggs a lot and I definately expect them to >be used. If you're not installing them with -m, they will be used, because they'll be the one in easy-install.pth. >>Note that in any event, if we make platform-specific eggs have >>higher precedence, then platform-specific built eggs will have to >>take precedence over develop eggs, because "true" develop eggs are >>never viewed as "platform-specific" (because they have no platform >>name in their .egg-info directory name). > >This argument sounds overly pedantic to me. It's actually a comment about the *implementation*. Develop eggs *can't* include version or platform information in their .egg-info directory names, or it makes them useless. From jim at zope.com Mon Jul 9 21:14:55 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 9 Jul 2007 15:14:55 -0400 Subject: [Distutils] Buildout not updating setuptools In-Reply-To: <20070709183849.6E89D3A404D@sparrow.telecommunity.com> References: <7481034A-8513-45F7-90E8-BBDE7B64BABA@zope.com> <37CA9D48-278C-44DF-AA4F-90C70F1AB349@mythic-beasts.com> <66366D92-429D-42B0-88B7-34DAA231337E@zope.com> <20070708165407.6209D3A404D@sparrow.telecommunity.com> <8EFFAD5D-A571-47D3-A5E4-A242466B5F55@zope.com> <20070709154035.CB9083A404D@sparrow.telecommunity.com> <20070709174254.788833A404D@sparrow.telecommunity.com> <041C24EC-4234-4CBD-9673-C34470030366@zope.com> <20070709183849.6E89D3A404D@sparrow.telecommunity.com> Message-ID: <4F6BA19F-0FD9-4C47-A71A-DA8771EB0C83@zope.com> On Jul 9, 2007, at 2:41 PM, Phillip J. Eby wrote: > At 02:09 PM 7/9/2007 -0400, Jim Fulton wrote: > >> On Jul 9, 2007, at 1:45 PM, Phillip J. Eby wrote: >> >>> At 01:10 PM 7/9/2007 -0400, Jim Fulton wrote: >>>> IMO, they shouldn't have the same precedence. IMO installations >>>> made >>>> with the develop command should have the highest precedence. After >>>> all, why would someone use the develop command if they didn't >>>> want to >>>> see the effects of changes they're making in development. Could we >>>> can change the develop command to generate a different symbol, like >>>> REALLY_DEVELOP_DIST and give it the highest precedence? >>>> (Tempted to >>>> wink, but not really kidding...) It appears that it doesn't >>>> actually >>>> generate anything atm, but I think it should. >>> >>> I could have .egg-link files use a different precedence, but I >>> wouldn't make it the highest. Develop eggs have the lowest >>> precedence for two reasons: one, easy_install prefers to use eggs >>> it can copy, >> >> Why does this matter to users? > > It doesn't! -- that's precisely why easy_install is free to pick > the one that's easier for it. :) The this I was refering to was: "easy_install prefers to use eggs it can copy". Users don't care about this. I think users do care how develop distributions are handled. ... >>> This was actually more of a happy coincidence than something I >>> planned ahead of time, but it does occur in practice. :) >> >> It made me very unhappy when I first started working with >> setuptools. I use develop eggs a lot and I definately expect them to >> be used. > > If you're not installing them with -m, they will be used, because > they'll be the one in easy-install.pth. I install *everything* with -m. When creating the working set for a particular application, I need to decide what should go in the working set. setuptools, left to it's own devices, will choose a non- develop egg over a develop egg. This made me sad. >>> Note that in any event, if we make platform-specific eggs have >>> higher precedence, then platform-specific built eggs will have to >>> take precedence over develop eggs, because "true" develop eggs are >>> never viewed as "platform-specific" (because they have no platform >>> name in their .egg-info directory name). >> >> This argument sounds overly pedantic to me. > > It's actually a comment about the *implementation*. Develop eggs > *can't* include version or platform information in their .egg-info > directory names, or it makes them useless. Yes, but I don't see why that should effect their precedence. BTW, at this point, this discussion is pedantic, so we can stop. I can see that, for buildout, I'm not going to be able to use precedence. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From mhammond at skippinet.com.au Tue Jul 10 00:49:06 2007 From: mhammond at skippinet.com.au (Mark Hammond) Date: Tue, 10 Jul 2007 08:49:06 +1000 Subject: [Distutils] pywin32 egg? In-Reply-To: <20070707122727.4A95D3A4046@sparrow.telecommunity.com> Message-ID: <040e01c7c27b$5b8bc100$090a0a0a@enfoldsystems.local> > Ah. Okay, first, pywin32 isn't registered on PyPI, so you need a -f > link to find the files: > > easy_install -f > http://sourceforge.net/project/showfiles.php?group_id=78018 pywin32 > > This is easily fixed by adding a PyPI listing with the above URL > registered as the "Download URL". Cool - done! > > easy_install successfully installs the package, but in order to do > the registry and copying stuff, the postinstall script has to be run: > > $ Scripts/pywin32_postinstall.py -install > Traceback (most recent call last): > File "Scripts\pywin32_postinstall.py", line 5, in ? > pkg_resources.run_script('pywin32==210', > 'pywin32_postinstall.py') > File > "c:\cygwin\home\pje\projects\setuptools-0.6\pkg_resources.py", line > 448, in run_script > self.require(requires)[0].run_script(script_name, ns) > File > "c:\cygwin\home\pje\projects\setuptools-0.6\pkg_resources.py", line > 1166, in run_script > execfile(script_filename, namespace, namespace) > File > "c:\cygwin\home\pje\chandlerstuff\chandler\release\bin\lib\sit > e-packages\pywin32-210-py2.4-win32.egg\EGG-INFO\scripts\pywin3 > 2_postinstall.py", > line 365, in ? > install() > File > "c:\cygwin\home\pje\chandlerstuff\chandler\release\bin\lib\sit > e-packages\pywin32-210-py2.4-win32.egg\EGG-INFO\scripts\pywin3 > 2_postinstall.py", > line 155, in install > LoadSystemModule(lib_dir, "pywintypes") > File > "c:\cygwin\home\pje\chandlerstuff\chandler\release\bin\lib\sit > e-packages\pywin32-210-py2.4-win32.egg\EGG-INFO\scripts\pywin3 > 2_postinstall.py", > line 94, in LoadSystemModule > ('.dll', 'rb', imp.C_EXTENSION)) > ImportError: DLL load failed: The specified module could not be found. > > The problem here is that lib_dir is calculated using the system > configuration, instead of by locating where the package is actually > installed. I changed the install() function like this: > > import timer > lib_dir = os.path.dirname(timer.__file__) > Importing some other pywin32 module than 'timer' would be okay; it > just looked like something small and unlikely to depend on anything > else. With this change, the install script produced this output:: hrm - normally lib_dir is pointing at site-packages - ie, the parent of 'win32', 'win32com' etc - whereas the code above would give site-packages\win32. That function should be fairly easy to make egg-friendly though. > Is there anything I can do to verify that this did in fact > work correctly? Simplest would be to paste the following 2 lines into a file: set py = CreateObject("Python.Interpreter") WScript.Echo("1+1=" & py.Eval("1+1")) then execute: cscript.exe //E:VBScript filename I suspect it will work though given the output you demonstrated. post_install.py creates a .pth file, which I suspect is redundant, or just plain wrong, when using eggs - it might be work looking for that pywin32.pth file and seeing what is in it. > > It appears that all would be needed for pywin32 to be fully > easy_install-able would be this change to the postinstall script > (which shouldn't affect its functionality for direct installation), > and an entry on PyPI. No setup script changed required -- or so it > appears. Anyone wishing to use the features enabled by the > postinstall script would just need to run it with -install. Excellent! > One question of course is, what features do you lose if you don't run > the postinstall? If I hadn't run it, would I still have been able to > use Python to access COM objects exposed by other applications, or > would I have been unable to do COM at all? You probably would have only lost the ability to implement COM objects - consuming them should be fine. You don't get the help file registered where pythonwin can find it and don't get a bogus .pth file , but that is about it. Cheers, Mark From pje at telecommunity.com Tue Jul 10 02:46:48 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 09 Jul 2007 20:46:48 -0400 Subject: [Distutils] pywin32 egg? In-Reply-To: <040e01c7c27b$5b8bc100$090a0a0a@enfoldsystems.local> References: <20070707122727.4A95D3A4046@sparrow.telecommunity.com> <040e01c7c27b$5b8bc100$090a0a0a@enfoldsystems.local> Message-ID: <20070710004435.1AF873A404D@sparrow.telecommunity.com> At 08:49 AM 7/10/2007 +1000, Mark Hammond wrote: >hrm - normally lib_dir is pointing at site-packages - ie, the parent of >'win32', 'win32com' etc - whereas the code above would give >site-packages\win32. Ah. Well, in the egg case, there is no win32 directory, nor win32/lib subdirectory thereof; everything that's not a package just goes right into the site-packages\pywin32-210-py2.4-win32.egg directory. If you'd like to see what's happening, download http://peak.telecommunity.com/dist/ez_setup.py and run: ez_setup.py pywin32 or alternately, install the Windows installer for setuptools, then run "python -m easy_install pywin32". At that point, you'll be able to see the site-packages layout the way it ends up under setuptools. If setuptools' directory layout is wrong in some way, let me know and I'll see what I can do. > > Is there anything I can do to verify that this did in fact > > work correctly? > >Simplest would be to paste the following 2 lines into a file: > >set py = CreateObject("Python.Interpreter") >WScript.Echo("1+1=" & py.Eval("1+1")) > >then execute: >cscript.exe //E:VBScript filename Microsoft (R) Windows Script Host Version 5.6 Copyright (C) Microsoft Corporation 1996-2001. All rights reserved. C:\...\testfile(1, 1) (null): The specified module could not be found. >You probably would have only lost the ability to implement COM objects - >consuming them should be fine. You don't get the help file registered where >pythonwin can find it and don't get a bogus .pth file , but that is >about it. As it happens, the bogus .pth file gets installed inside of the pywin32*.egg directory, so it doesn't actually get read by Python. By design, easy_install wants everything for a package to be contained in its .egg, and refuses to put anything outside of it except wrapper scripts and a .pth file to put the .egg on sys.path. From arve.knudsen at gmail.com Tue Jul 10 10:32:31 2007 From: arve.knudsen at gmail.com (Arve Knudsen) Date: Tue, 10 Jul 2007 10:32:31 +0200 Subject: [Distutils] Regeneration of source manifest Message-ID: Hello As far as I can tell setuptools does not always regenerate the source manifest when MANIFEST.in changes, contrary to what the documentation says. That is, if I add a directive to exclude files with the .txt extension to MANIFEST.in (after having previously including them), and run the sdist command, such files are still included. I have to remove the .egg-info for things to work as expected. This must clearly be a bug? Regards, Arve Knudsen -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070710/c34e70eb/attachment.html From jim at zope.com Tue Jul 10 15:52:42 2007 From: jim at zope.com (Jim Fulton) Date: Tue, 10 Jul 2007 09:52:42 -0400 Subject: [Distutils] Merge catalog and distutils sigs Message-ID: <6C0A5EEC-7E01-4C25-BC09-E0B595C8109A@zope.com> Is there are good reason for the distutils and catalog sigs to be separate? Now, that PyPI is an integral part of the distribution system, I find most topics are really of of interested to both sigs, and I bet that the overlap between the sigs is significant. Would anyone object to combining them? Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Tue Jul 10 16:32:10 2007 From: jim at zope.com (Jim Fulton) Date: Tue, 10 Jul 2007 10:32:10 -0400 Subject: [Distutils] Why so many zc.buildout versions? In-Reply-To: <46937F10.3070201@weitershausen.de> References: <46937F10.3070201@weitershausen.de> Message-ID: <73FE055E-D4F4-44E7-9DEE-353601795EC2@zope.com> You raise a really good point, which is especially relevant in light of pypi performance issues and discussions. I'm copying the distutils and catalog sigs to get some wider discussion. I apologize for the cross posting. I'm beginning to wonder about the strategy that setuptools uses, or maybe about the way we are using the index. It's important to note that there is nothing specific about the buildout package here. It is very important to make multiple versions available to support requirements for specific package versions. It make builds/installs repeatable, whether talking about buildout or other systems built on setuptools. When someone has tested and wants to release an application built from a collection of distributions, they will want to specify those *specific* versions for future builds or installs. This means that we need to retain any versions published indefinitely in a way that can be found by setuptools. Currently, the only way to support multiple versions with the cheeseshop is to unhide past releases. This has a fairly severe effect on performance. As the example below shows, setuptools will fetch the package page and then fetch the pages for each release. That's a lot of requests. What makes it worse is that the individual package pages can be fairly long. I've gotten in the habit of including full documentation on every release page. For example, recent release pages for zc.buildout are around 200K. This is a fairly significant amount of data to transfer. This will certainly make the scanning process take a long time for clients. (Obviously, if we keep doing things the way we are, I'll need to stop doing that.) All of this aggravates any performance problems we might have. Up to now, setuptools has tried hard to use existing systems without change. This means that it reuses systems designed primarily for people, not software. I think that setuptools rightly took the approach it has up to now so that progress could be made without making people change other systems. This was appropriate when setuptools was evolving and people were figuring out ways to use it. I think it is time to take a step back and think a lot harder about how we'd want to structure an index to support setuptools. IMO, a setuptools-aware index would have a single page for each package: - The single page would be published in a case-insensitive way. It would be nice to find a way to avoid this, or maybe we should use a windows-based web server. :) It would also be served very cheaply, for example statically. - The single page would list links for all available distributions, which should include all distributions published. It would also list any other URLs that should be scanned for releases, when releases aren't all uploaded to PyPI. - The single page would contain very little additional information. It would be for use by software, not humans. In addition, the root page with a trailing / would be empty and very cheap. There are a lot of ways we could achieve this pretty cheaply while keeping the existing system pretty much as it is. For example, the current effort to bake static pages could bake these pages instead. We could make the new index available at a different URL for people to play with while we worked the kinks out of the process. Of course, those of us who use the cheesehop and setuptools extensively can also achieve much of this by changing the way we work. Thoughts? Jim On Jul 10, 2007, at 8:44 AM, Philipp von Weitershausen wrote: > When easy_installing zc.buildout I realized that the CheeseShop > still lists a gazillion old versions of zc.buildout. That makes it > take quite some time to install zc.buildout (see below), and I > reckon the same sort of check has to happen each time it looks for > a new version of that egg... > > Is there any reason for having so many old versions around? > > > $ easy_install zc.buildout > Searching for zc.buildout > Reading http://cheeseshop.python.org/pypi/zc.buildout/ > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b19 > Reading http://svn.zope.org/zc.buildout > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b22 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b23 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b20 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b21 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b26 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b27 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b24 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b25 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b28 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b17 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b16 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b18 > Best match: zc.buildout 1.0.0b28 > ... -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Tue Jul 10 17:56:42 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 10 Jul 2007 11:56:42 -0400 Subject: [Distutils] Why so many zc.buildout versions? In-Reply-To: <73FE055E-D4F4-44E7-9DEE-353601795EC2@zope.com> References: <46937F10.3070201@weitershausen.de> <73FE055E-D4F4-44E7-9DEE-353601795EC2@zope.com> Message-ID: <20070710155535.4D8CC3A40A9@sparrow.telecommunity.com> At 10:32 AM 7/10/2007 -0400, Jim Fulton wrote: >Currently, the only way to support multiple versions with the >cheeseshop is to unhide past releases. This has a fairly severe >effect on performance. As the example below shows, setuptools will >fetch the package page and then fetch the pages for each release. >That's a lot of requests. This could potentially be fixed in setuptools, so that it only looks at release pages that match its requirements, in highest-to-lowest version order, stopping as soon as a suitable match is found. That would eliminate the current issue -- but only for new versions of setuptools. So I do like your idea better, since it can be made to work for already-deployed clients as well. >I think it is time to take a step back and think a lot harder about >how we'd want to structure an index to support setuptools. +1, as long as somebody's willing to build and host the thing. Please see my earlier comments on the Catalog-Sig about this. >IMO, a setuptools-aware index would have a single page for each package: > >- The single page would be published in a case-insensitive way. It >would be nice to find a way to avoid this, or maybe we should use a >windows-based web server. :) It would also be served very cheaply, >for example statically. Apache's CheckSpelling directive does case-insensitivity and approximate matching. Combine that with making the directories be based on "safe_name" values to begin with, and you should be all set. >- The single page would list links for all available distributions, >which should include all distributions published. It would also list >any other URLs that should be scanned for releases, when releases >aren't all uploaded to PyPI. The piece you're missing here is direct links to other downloads, such as "#egg=project-dev" subversion links. However, if you extracted these from all of the relevant PyPI HTML pages, you could certainly do that. >In addition, the root page with a trailing / would be empty and very >cheap. As long as the individual package directories are safe_name based, this would work. >There are a lot of ways we could achieve this pretty cheaply while >keeping the existing system pretty much as it is. Of course, there are still other reasons to want to improve the Cheeseshop's performance, such as search engines and other bots. >For example, the current effort to bake static pages could bake these >pages instead. We could make the new index available at a different >URL for people to play with while we worked the kinks out of the >process. ...and then use a User-Agent rewrite rule to redirect setuptools clients to the static piece, as soon as we're satisfied that it works. From jim at zope.com Tue Jul 10 18:04:01 2007 From: jim at zope.com (Jim Fulton) Date: Tue, 10 Jul 2007 12:04:01 -0400 Subject: [Distutils] Why so many zc.buildout versions? In-Reply-To: <20070710155535.4D8CC3A40A9@sparrow.telecommunity.com> References: <46937F10.3070201@weitershausen.de> <73FE055E-D4F4-44E7-9DEE-353601795EC2@zope.com> <20070710155535.4D8CC3A40A9@sparrow.telecommunity.com> Message-ID: On Jul 10, 2007, at 11:56 AM, Phillip J. Eby wrote: > At 10:32 AM 7/10/2007 -0400, Jim Fulton wrote: >> Currently, the only way to support multiple versions with the >> cheeseshop is to unhide past releases. This has a fairly severe >> effect on performance. As the example below shows, setuptools will >> fetch the package page and then fetch the pages for each release. >> That's a lot of requests. > > This could potentially be fixed in setuptools, so that it only > looks at release pages that match its requirements, in highest-to- > lowest version order, stopping as soon as a suitable match is > found. That would eliminate the current issue No, it will mitigate the current issue somewhat, but it will still involve multiple requests per package, while a simpler index structure would allow a single request per package. > -- but only for new versions of setuptools. So I do like your idea > better, since it can be made to work for already-deployed clients > as well. Yup. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From theller at ctypes.org Tue Jul 10 19:43:06 2007 From: theller at ctypes.org (Thomas Heller) Date: Tue, 10 Jul 2007 19:43:06 +0200 Subject: [Distutils] bdist_wininst and Vista User Account Control (UAC) In-Reply-To: <00a701c7bf59$c0ad5cd0$090a0a0a@enfoldsystems.local> References: <00a701c7bf59$c0ad5cd0$090a0a0a@enfoldsystems.local> Message-ID: <4693C52A.9040705@ctypes.org> [sorry for the late reply] Mark Hammond schrieb: > Hi Thomas, > >> >> > I'm wondering if anyone has any opinions or thoughts on how >> we should handle >> > this? >> >> bdist_wininst has its own problems anyway - most severe the >> MSVC runtime dll issue. > > What issue is this? Could it be solved by adding a manifest to the > executable? It seems to me that Python itself built with VC2005 has this > same issue. There is the problem of *installing* this library though, but > if we assume Python itself has already done this, I don't see the problem > for bdist_wininst. The usual C runtime lib problems as it embeds Python, and also calls problematic functions in the rumtime lib, so it should use the same lib as Python itself. If you build a pure Python bdist_wininst installer, which can be used to install into Python 2.3, 2.4, 2.5, and more versions, the postinstall script (if any) does not run correctly if Python uses the wrong runtime lib. > >> Another one is that it won't work for 64-bit installations. > > My experiments show that very nearly works! I haven't actually got such a > binary to install, but a 64bit bdist_wininst based installer correctly runs > all the way to asking me to select an installed Python (where it stops as I > don't have such an install yet). IIUC, to get this far, bdist_wininst has > already sniffed out most data it needs. I could be wrong here, but I'm > wondering what problems you had in mind? I have not tried to build a 64bit runtime stub for bdist_wininst, that's the only reason so far. It could probably be solved. >> bdist_msi solves the 64-bit and MSVCRT issue. I would expect >> that it also solves the >> UAC problems. OTOH it is not possible (AFAIK) to build >> bdist_msi installers on Linux systems. > > Yeah, that may be what I need to use going forward - which basically would > leave bdist_wininst to slowly die - but I guess that is good to know now, > rather than after I spend more time trying to apply life-support to it... Well, bdist_wininst had a sucessful live, I see no problems if it is replaced with something else - be it eggs or bdist_msi. Thomas From pje at telecommunity.com Tue Jul 10 20:29:39 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 10 Jul 2007 14:29:39 -0400 Subject: [Distutils] bdist_wininst and Vista User Account Control (UAC) In-Reply-To: <4693C52A.9040705@ctypes.org> References: <00a701c7bf59$c0ad5cd0$090a0a0a@enfoldsystems.local> <4693C52A.9040705@ctypes.org> Message-ID: <20070710183310.4F5DA3A40A4@sparrow.telecommunity.com> At 07:43 PM 7/10/2007 +0200, Thomas Heller wrote: >Well, bdist_wininst had a sucessful live, I see no problems if it is >replaced with something else - be it eggs or bdist_msi. Perhaps someone could make an egg-to-msi translator, so that non-Windows users could still build and distribute eggs, but Windows users could still install them with built-in uninstallability. Eggs certainly don't have a complex internal format -- indeed, they don't offer as much installation flexibility as bdist_wininst does. easy_install can convert most (but not all) bdist_wininst .exe's to eggs, so a similar conversion from eggs to MSI should be possible. From martin at v.loewis.de Tue Jul 10 23:36:28 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 10 Jul 2007 23:36:28 +0200 Subject: [Distutils] [Catalog-sig] Why so many zc.buildout versions? In-Reply-To: <73FE055E-D4F4-44E7-9DEE-353601795EC2@zope.com> References: <46937F10.3070201@weitershausen.de> <73FE055E-D4F4-44E7-9DEE-353601795EC2@zope.com> Message-ID: <4693FBDC.2060201@v.loewis.de> > For example, the current effort to bake static pages could bake these > pages instead. Certainly not instead; in addition, if there are volunteers to implement that. > We could make the new index available at a different > URL for people to play with while we worked the kinks out of the > process. I have been thinking about the same thing. I think it would be good to have, however, it will surely take some time until all setuptools implementations learn to use it. > Of course, those of us who use the cheesehop and setuptools > extensively can also achieve much of this by changing the way we work. Hmm. How about those using them extensively start contributing to them also? Regards, Martin From martin at v.loewis.de Tue Jul 10 23:39:28 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 10 Jul 2007 23:39:28 +0200 Subject: [Distutils] [Catalog-sig] Why so many zc.buildout versions? In-Reply-To: References: <46937F10.3070201@weitershausen.de> <73FE055E-D4F4-44E7-9DEE-353601795EC2@zope.com> <20070710155535.4D8CC3A40A9@sparrow.telecommunity.com> Message-ID: <4693FC90.9060001@v.loewis.de> > No, it will mitigate the current issue somewhat, but it will still > involve multiple requests per package, while a simpler index > structure would allow a single request per package. I don't understand. If setuptools would always look /pypi/package/version first, it would immediately find the right page if that version is indeed stored in the cheeseshop. Why would that require multiple requests per package? Regards, Martin From jim at zope.com Tue Jul 10 23:49:43 2007 From: jim at zope.com (Jim Fulton) Date: Tue, 10 Jul 2007 17:49:43 -0400 Subject: [Distutils] [Catalog-sig] Why so many zc.buildout versions? In-Reply-To: <4693FBDC.2060201@v.loewis.de> References: <46937F10.3070201@weitershausen.de> <73FE055E-D4F4-44E7-9DEE-353601795EC2@zope.com> <4693FBDC.2060201@v.loewis.de> Message-ID: <4D7FD5E2-7460-4A48-A1B0-C1247B0A3FB8@zope.com> On Jul 10, 2007, at 5:36 PM, Martin v. L?wis wrote: >> For example, the current effort to bake static pages could bake these >> pages instead. > > Certainly not instead; in addition, if there are volunteers to > implement > that. Sure, > >> We could make the new index available at a different >> URL for people to play with while we worked the kinks out of the >> process. > > I have been thinking about the same thing. I think it would be good > to have, however, it will surely take some time until all setuptools > implementations learn to use it. No, not at all. You can tell setuptools to use a different index than the current one. For example, this is a command-line option for easy_install and a configuration option for buildout. >> Of course, those of us who use the cheesehop and setuptools >> extensively can also achieve much of this by changing the way we >> work. > > Hmm. How about those using them extensively start contributing to > them also? I like to think that I am by participating in this discussion. Actually changing the cheeseshop software has a very high learning curve. I don't think that I can make that kind of time any time soon. I'm very grateful that you and Ren? are doing what you're doing. I also suspect that, given your and Ren?'s activity, it would be counter productive for someone else to get involved at that level, but maybe I'm wrong about that. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Tue Jul 10 23:55:28 2007 From: jim at zope.com (Jim Fulton) Date: Tue, 10 Jul 2007 17:55:28 -0400 Subject: [Distutils] [Catalog-sig] Why so many zc.buildout versions? In-Reply-To: <4693FC90.9060001@v.loewis.de> References: <46937F10.3070201@weitershausen.de> <73FE055E-D4F4-44E7-9DEE-353601795EC2@zope.com> <20070710155535.4D8CC3A40A9@sparrow.telecommunity.com> <4693FC90.9060001@v.loewis.de> Message-ID: On Jul 10, 2007, at 5:39 PM, Martin v. L?wis wrote: >> No, it will mitigate the current issue somewhat, but it will still >> involve multiple requests per package, while a simpler index >> structure would allow a single request per package. > > I don't understand. If setuptools would always look > /pypi/package/version first, it would immediately find the right > page if that version is indeed stored in the cheeseshop. > > Why would that require multiple requests per package? It usually doesn't have a single required version. It usually has just a package name or a name and a range of versions. It has to scan the package page to find out what versions are available, and *then* it can load the release page for the highest version that satisfies the requirement. It can usually read that one page, however, there may be additional filtering needed that would cause it to search multiple releases. For example, it might be looking for a source distribution, or a platform-specific distribution that isn't available for the most recent release. In any case, the best case is that it has to scan the package page to find the most recent release, and then scan that release page. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From martin at v.loewis.de Wed Jul 11 07:06:59 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 11 Jul 2007 07:06:59 +0200 Subject: [Distutils] [Catalog-sig] Why so many zc.buildout versions? In-Reply-To: <4D7FD5E2-7460-4A48-A1B0-C1247B0A3FB8@zope.com> References: <46937F10.3070201@weitershausen.de> <73FE055E-D4F4-44E7-9DEE-353601795EC2@zope.com> <4693FBDC.2060201@v.loewis.de> <4D7FD5E2-7460-4A48-A1B0-C1247B0A3FB8@zope.com> Message-ID: <46946573.2070400@v.loewis.de> >> I have been thinking about the same thing. I think it would be good >> to have, however, it will surely take some time until all setuptools >> implementations learn to use it. > > No, not at all. You can tell setuptools to use a different index than > the current one. For example, this is a command-line option for > easy_install and a configuration option for buildout. Yes. However, that will make the feature only available to those who know about it. I have very shallow knowledge of setuptools and easy_install only (I nearly never use them at all), and I surely would miss such an option, and miss why it's relevant. It's true that the Apache installation could also redirect existing installations to the new pages, but I doubt that they would be otherwise widely used until setuptools changes its hard-coded default. >> Hmm. How about those using them extensively start contributing to >> them also? > > I like to think that I am by participating in this discussion. Actually > changing the cheeseshop software has a very high learning curve. I don't > think that I can make that kind of time any time soon. I'm very > grateful that you and Ren? are doing what you're doing. I also suspect > that, given your and Ren?'s activity, it would be counter productive for > someone else to get involved at that level, but maybe I'm wrong about that. I strongly think you are. There are many things that could be improved, and I would not mind leaving the cheeseshop alone if some other maintainer came along - I also have other things to do. Regards, Martin From renesd at gmail.com Wed Jul 11 14:20:22 2007 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Wed, 11 Jul 2007 22:20:22 +1000 Subject: [Distutils] [Catalog-sig] Why so many zc.buildout versions? In-Reply-To: <46946573.2070400@v.loewis.de> References: <46937F10.3070201@weitershausen.de> <73FE055E-D4F4-44E7-9DEE-353601795EC2@zope.com> <4693FBDC.2060201@v.loewis.de> <4D7FD5E2-7460-4A48-A1B0-C1247B0A3FB8@zope.com> <46946573.2070400@v.loewis.de> Message-ID: <64ddb72c0707110520j42bb8f27nb676bcf4de39d14c@mail.gmail.com> I have to say the cheeseshop code was pretty easy to get into. I think I was able to make most of my changes within the first reading of it. It quite clearly separates things like the templates, the database functionality and the 'webui'. There definitely are a huge amount of things that I would love to change with it over time, and I hope other people begin to develop it more - it can only help the python community as a whole. The amount of people doing releases has increased quite a lot even in the last two months, so I think the releases will get more frequent. As it grows it will continue to need different changes - optimizations to the database/webserver, and also optimizations to the user interface. On 7/11/07, "Martin v. L?wis" wrote: > >> I have been thinking about the same thing. I think it would be good > >> to have, however, it will surely take some time until all setuptools > >> implementations learn to use it. > > > > No, not at all. You can tell setuptools to use a different index than > > the current one. For example, this is a command-line option for > > easy_install and a configuration option for buildout. > > Yes. However, that will make the feature only available to those who > know about it. I have very shallow knowledge of setuptools and > easy_install only (I nearly never use them at all), and I surely would > miss such an option, and miss why it's relevant. > > It's true that the Apache installation could also redirect existing > installations to the new pages, but I doubt that they would be otherwise > widely used until setuptools changes its hard-coded default. > > >> Hmm. How about those using them extensively start contributing to > >> them also? > > > > I like to think that I am by participating in this discussion. Actually > > changing the cheeseshop software has a very high learning curve. I don't > > think that I can make that kind of time any time soon. I'm very > > grateful that you and Ren? are doing what you're doing. I also suspect > > that, given your and Ren?'s activity, it would be counter productive for > > someone else to get involved at that level, but maybe I'm wrong about that. > > I strongly think you are. There are many things that could be improved, > and I would not mind leaving the cheeseshop alone if some other > maintainer came along - I also have other things to do. > > Regards, > Martin > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > From jim at zope.com Wed Jul 11 14:56:22 2007 From: jim at zope.com (Jim Fulton) Date: Wed, 11 Jul 2007 08:56:22 -0400 Subject: [Distutils] [Catalog-sig] Why so many zc.buildout versions? In-Reply-To: <46946573.2070400@v.loewis.de> References: <46937F10.3070201@weitershausen.de> <73FE055E-D4F4-44E7-9DEE-353601795EC2@zope.com> <4693FBDC.2060201@v.loewis.de> <4D7FD5E2-7460-4A48-A1B0-C1247B0A3FB8@zope.com> <46946573.2070400@v.loewis.de> Message-ID: <7E6E8D05-9669-4765-B61D-254835DDA553@zope.com> On Jul 11, 2007, at 1:06 AM, Martin v. L?wis wrote: >>> I have been thinking about the same thing. I think it would be good >>> to have, however, it will surely take some time until all setuptools >>> implementations learn to use it. >> >> No, not at all. You can tell setuptools to use a different index >> than >> the current one. For example, this is a command-line option for >> easy_install and a configuration option for buildout. > > Yes. However, that will make the feature only available to those who > know about it. I have very shallow knowledge of setuptools and > easy_install only (I nearly never use them at all), and I surely would > miss such an option, and miss why it's relevant. That's fine. I don't care if most people can find it. While it is an *experimental* index, it is fine if only a few people play with it. If it is proven to work properly, then we could arrange that other people get it by default. > It's true that the Apache installation could also redirect existing > installations to the new pages, but I doubt that they would be > otherwise > widely used until setuptools changes its hard-coded default. Right, that's why, if the experiment works, we should then change the Apache config to rediect setuptools to it. Changing the apache config is much easier than updating the setuptools installed base. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Thu Jul 12 01:46:45 2007 From: jim at zope.com (Jim Fulton) Date: Wed, 11 Jul 2007 19:46:45 -0400 Subject: [Distutils] [Catalog-sig] start on static generation, and caching - apache config. In-Reply-To: <200707120811.49824.richardjones@optushome.com.au> References: <64ddb72c0707062124r30aa1b06k88838e06d73c45bd@mail.gmail.com> <7605F808-8C05-4735-A8E9-F2663083F4F5@zope.com> <469522D6.1070706@v.loewis.de> <200707120811.49824.richardjones@optushome.com.au> Message-ID: On Jul 11, 2007, at 6:11 PM, Richard Jones wrote: > On Thu, 12 Jul 2007, Martin v. L?wis wrote: >>> The questions for us is, how much effort we are willing to make to >>> prevent people from shooting themselves in the foot. I can >>> understand >>> why Phillip would like the package index to prevent people from >>> choosing >>> problematic package names. >> >> That's not my understanding - the issue isn't with "problematic >> package >> names", but with conflicting package names. IOW, any single name is >> fine - it's a pair of names that would cause a problem (and only if >> you wanted to install both packages on the same system). > > A big issue that's not been raised is that *distutils* have no > package name > rules, but it's being proposed that PyPI does - thus a package > author will > potentially get an error when uploading their package, and also the > name that > appears in the index may be quite different to the name of their > package. Maybe distutils should have more package name rules than it does now. We (the Community) should be free to change things based on experience. We now have a lot more experience with this stuff than we had a few years ago. Maybe we should consider a reset. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Thu Jul 12 21:02:10 2007 From: jim at zope.com (Jim Fulton) Date: Thu, 12 Jul 2007 15:02:10 -0400 Subject: [Distutils] Case sensitivity of package names In-Reply-To: <20070712184056.F219A3A40B0@sparrow.telecommunity.com> References: <64ddb72c0707062124r30aa1b06k88838e06d73c45bd@mail.gmail.com> <468F3CD4.1070501@v.loewis.de> <64ddb72c0707070038j3970e379gabae0060f48036ea@mail.gmail.com> <64ddb72c0707070203q39247d4axf183292e0b00225@mail.gmail.com> <468FC2BB.7030607@v.loewis.de> <6B8CDD2C-4F02-4836-89FA-0D00EAFE0F74@zope.com> <468FF69B.2090503@v.loewis.de> <057B56A0-CA3D-4EF3-B34D-A3174FE3B72C@zope.com> <46910BBF.3010308@v.loewis.de> <4692B3A3.5030209@v.loewis.de> <20070710003214.A2EA83A404D@sparrow.telecommunity.com> <46931A3A.5000703@v.loewis.de> <20070710141304.BC6903A40A4@sparrow.telecommunity.com> <4693FA2A.3020107@v.loewis.de> <20070710221547.4A3043A40A4@sparrow.telecommunity.com> <469467AA.7070409@v.loewis.de> <7605F808-8C05-4735-A8E9-F2663083F4F5@zope.com> <469522D6.1070706@v.loewis.de> <20070712180539.3BFB43A40D7@sparrow.telecommunity.com> <0F4183B4-D60B-4715-A75C-531332C0CE2B@zope.com> <20070712184056.F219A3A40B0@sparrow.telecommunity.com> Message-ID: <068C982C-701B-45D6-BC34-C48B217B80E8@zope.com> On Jul 12, 2007, at 2:43 PM, Phillip J. Eby wrote: > At 02:15 PM 7/12/2007 -0400, Jim Fulton wrote: >> I want to make sure I understand this. I would hope that searching >> would be case insensitive and otherwise flexible wrt names. > > PyPI's searching is indeed case insensitive, and is a substring/ > keyword search as well. > > >> Is there >> any reason we can't expect URLs and requirement specifications to be >> precisely spelled? That is, if someone names their package "sPaM", I >> see no reason why PyPI needs to support anything other than >> http:// www.python.org/pypi/sPaM as the one URL of the package. >> Someone >> should be able to use the search UI to search for "spam" and see a >> result that includes "sPaM". From then on, they should be able to >> type the name "sPaM". Or am I missing something? > > You're missing that the subject is about similarity of names. > A typo of say, 'SPam' shouldn't return me some package *other* > than the one I'm looking for. I No, I understand that part. I understand the desire to avoid conflicts that cause problems down the road. I would prefer to "disallow" this by rejecting new package names that are too similar to already-registered packages. > t'd be nice if the resulting page said something besides "Not > Found", too... like "there's no SPam, but here are a bunch of > packages whose name contains 'spam'". I think this would be fine in a human interface. > If it did that, setuptools would be able to find the right page > without hitting the main index, too. But redirection, as proposed > by Martin, also accomplishes the same thing. I really don't like this for setuptools. My preference is that setuptools should be required to ask for a package with precise spelling. > And again, all this helps human direct users of the index, too. I think it encourages humans to do bad things. Is someone misspells ZODB3 as zodb3 and is able to install it with easy_install, then they'll be tempted to use the name "zodb3" in their requirements specifications. That is a bad thing IMO. We're talking about technical users and I think it is reasonable to expect them to be precise in their specifications. I could live with case-insensitive package names if we (for some definition of we, possibly being Guido) decided we want them, but I'd prefer they be case sensitive. I'd still be in favor of avoiding confusing duplicates. If we stick with case-sentitive package names, then I'd prefer that the interaction of setuptools with the index be case sensitive. I wouldn't object to setuptools giving people help. So, for example, if I type "zodb3", I wouldn't object to setuptools letting the user know that maybe they should use ZODB3. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Thu Jul 12 21:26:02 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 12 Jul 2007 15:26:02 -0400 Subject: [Distutils] Case sensitivity of package names In-Reply-To: <068C982C-701B-45D6-BC34-C48B217B80E8@zope.com> References: <64ddb72c0707062124r30aa1b06k88838e06d73c45bd@mail.gmail.com> <468F3CD4.1070501@v.loewis.de> <64ddb72c0707070038j3970e379gabae0060f48036ea@mail.gmail.com> <64ddb72c0707070203q39247d4axf183292e0b00225@mail.gmail.com> <468FC2BB.7030607@v.loewis.de> <6B8CDD2C-4F02-4836-89FA-0D00EAFE0F74@zope.com> <468FF69B.2090503@v.loewis.de> <057B56A0-CA3D-4EF3-B34D-A3174FE3B72C@zope.com> <46910BBF.3010308@v.loewis.de> <4692B3A3.5030209@v.loewis.de> <20070710003214.A2EA83A404D@sparrow.telecommunity.com> <46931A3A.5000703@v.loewis.de> <20070710141304.BC6903A40A4@sparrow.telecommunity.com> <4693FA2A.3020107@v.loewis.de> <20070710221547.4A3043A40A4@sparrow.telecommunity.com> <469467AA.7070409@v.loewis.de> <7605F808-8C05-4735-A8E9-F2663083F4F5@zope.com> <469522D6.1070706@v.loewis.de> <20070712180539.3BFB43A40D7@sparrow.telecommunity.com> <0F4183B4-D60B-4715-A75C-531332C0CE2B@zope.com> <20070712184056.F219A3A40B0@sparrow.telecommunity.com> <068C982C-701B-45D6-BC34-C48B217B80E8@zope.com> Message-ID: <20070712192350.347B13A40B0@sparrow.telecommunity.com> At 03:02 PM 7/12/2007 -0400, Jim Fulton wrote: >We're talking about >technical users and I think it is reasonable to expect them to be >precise in their specifications. IMO, "technical users" is a wider range of people than you seem to be thinking of. In any case, this is a separate topic from disallowing too-similar names -- which you agree we should do. Whether to then also introduce case-sensitivity into various parts of easy_install is another subject that doesn't really matter to the catalog-sig. Please note, however, that it is not a minor change by any means -- case-insensitivity exists throughout pkg_resources and setuptools to handle operating system filename case-insensitivity, not just for index lookups. In fact, I believe the index lookups *are* case-sensitive; IIRC it's only link parsing that is case-insensitive. From jim at zope.com Thu Jul 12 21:31:45 2007 From: jim at zope.com (Jim Fulton) Date: Thu, 12 Jul 2007 15:31:45 -0400 Subject: [Distutils] Case sensitivity of package names In-Reply-To: <20070712192350.347B13A40B0@sparrow.telecommunity.com> References: <64ddb72c0707062124r30aa1b06k88838e06d73c45bd@mail.gmail.com> <468F3CD4.1070501@v.loewis.de> <64ddb72c0707070038j3970e379gabae0060f48036ea@mail.gmail.com> <64ddb72c0707070203q39247d4axf183292e0b00225@mail.gmail.com> <468FC2BB.7030607@v.loewis.de> <6B8CDD2C-4F02-4836-89FA-0D00EAFE0F74@zope.com> <468FF69B.2090503@v.loewis.de> <057B56A0-CA3D-4EF3-B34D-A3174FE3B72C@zope.com> <46910BBF.3010308@v.loewis.de> <4692B3A3.5030209@v.loewis.de> <20070710003214.A2EA83A404D@sparrow.telecommunity.com> <46931A3A.5000703@v.loewis.de> <20070710141304.BC6903A40A4@sparrow.telecommunity.com> <4693FA2A.3020107@v.loewis.de> <20070710221547.4A3043A40A4@sparrow.telecommunity.com> <469467AA.7070409@v.loewis.de> <7605F808-8C05-4735-A8E9-F2663083F4F5@zope.com> <469522D6.1070706@v.loewis.de> <20070712180539.3BFB43A40D7@sparrow.telecommunity.com> <0F4183B4-D60B-4715-A75C-531332C0CE2B@zope.com> <20070712184056.F219A3A40B0@sparrow.telecommunity.com> <068C982C-701B-45D6-BC34-C48B217B80E8@zope.com> <20070712192350.347B13A40B0@sparrow.telecommunity.com> Message-ID: <4CD1A7D8-1911-45C9-AB08-C4DC3E1CDFA9@zope.com> On Jul 12, 2007, at 3:26 PM, Phillip J. Eby wrote: ... > Whether to then also introduce case-sensitivity into various parts > of easy_install is another subject that doesn't really matter to > the catalog-sig. I'm not sure we agree on what matters to the catalog sig. :) (I still need to respond to your note on that topic.) > Please note, however, that it is not a minor change by any means -- > case-insensitivity exists throughout pkg_resources and setuptools > to handle operating system filename case-insensitivity, not just > for index lookups. In fact, I believe the index lookups *are* case- > sensitive; IIRC it's only link parsing that is case-insensitive. I'm not suggesting that you shouldn't deal with file-system case insensitivity. If I were to change setuptools to match my opinion, I would probably just change the code that tries to get a package listing to look for close matches to print a suggestion and stop rather than guessing a package name and continuing. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From martin at v.loewis.de Thu Jul 12 23:38:50 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 12 Jul 2007 23:38:50 +0200 Subject: [Distutils] [Catalog-sig] Case sensitivity of package names In-Reply-To: <068C982C-701B-45D6-BC34-C48B217B80E8@zope.com> References: <64ddb72c0707062124r30aa1b06k88838e06d73c45bd@mail.gmail.com> <64ddb72c0707070203q39247d4axf183292e0b00225@mail.gmail.com> <468FC2BB.7030607@v.loewis.de> <6B8CDD2C-4F02-4836-89FA-0D00EAFE0F74@zope.com> <468FF69B.2090503@v.loewis.de> <057B56A0-CA3D-4EF3-B34D-A3174FE3B72C@zope.com> <46910BBF.3010308@v.loewis.de> <4692B3A3.5030209@v.loewis.de> <20070710003214.A2EA83A404D@sparrow.telecommunity.com> <46931A3A.5000703@v.loewis.de> <20070710141304.BC6903A40A4@sparrow.telecommunity.com> <4693FA2A.3020107@v.loewis.de> <20070710221547.4A3043A40A4@sparrow.telecommunity.com> <469467AA.7070409@v.loewis.de> <7605F808-8C05-4735-A8E9-F2663083F4F5@zope.com> <469522D6.1070706@v.loewis.de> <20070712180539.3BFB43A40D7@sparrow.telecommunity.com> <0F4183B4-D60B-4715-A75C-531332C0CE2B@zope.com> <20070712184056.F219A3A40B0@sparrow.telecommunity.com> <068C982C-701B-45D6-BC34-C48B217B80E8@zope.com> Message-ID: <46969F6A.8030904@v.loewis.de> > I really don't like this for setuptools. My preference is that > setuptools should be required to ask for a package with precise > spelling. I think the way setuptools currently works is this: Every name gets converted to its lower-case safe-name equivalent. All dependencies, file names, resource identifications etc are based on that version of the name, *not* the "true" name of the package. Then, when setuptools tries to find a package whose "true" name is in mixed-case, it uses the lower-cased safe-named version, and PyPI reports that the package does not exist. Then, setuptools queries the entire package list, trying to find out the original spelling of the package. I'm sure Phillip will correct me if I'm wrong. > I could live with case-insensitive package names if we (for some > definition of we, possibly being Guido) decided we want them, but I'd > prefer they be case sensitive. I'd still be in favor of avoiding > confusing duplicates. If we stick with case-sentitive package names, > then I'd prefer that the interaction of setuptools with the index be > case sensitive. See above - I believe setuptools package names are case insensitive today. Regards, Martin From pje at telecommunity.com Fri Jul 13 01:35:05 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 12 Jul 2007 19:35:05 -0400 Subject: [Distutils] [Catalog-sig] Case sensitivity of package names In-Reply-To: <46969F6A.8030904@v.loewis.de> References: <64ddb72c0707062124r30aa1b06k88838e06d73c45bd@mail.gmail.com> <64ddb72c0707070203q39247d4axf183292e0b00225@mail.gmail.com> <468FC2BB.7030607@v.loewis.de> <6B8CDD2C-4F02-4836-89FA-0D00EAFE0F74@zope.com> <468FF69B.2090503@v.loewis.de> <057B56A0-CA3D-4EF3-B34D-A3174FE3B72C@zope.com> <46910BBF.3010308@v.loewis.de> <4692B3A3.5030209@v.loewis.de> <20070710003214.A2EA83A404D@sparrow.telecommunity.com> <46931A3A.5000703@v.loewis.de> <20070710141304.BC6903A40A4@sparrow.telecommunity.com> <4693FA2A.3020107@v.loewis.de> <20070710221547.4A3043A40A4@sparrow.telecommunity.com> <469467AA.7070409@v.loewis.de> <7605F808-8C05-4735-A8E9-F2663083F4F5@zope.com> <469522D6.1070706@v.loewis.de> <20070712180539.3BFB43A40D7@sparrow.telecommunity.com> <0F4183B4-D60B-4715-A75C-531332C0CE2B@zope.com> <20070712184056.F219A3A40B0@sparrow.telecommunity.com> <068C982C-701B-45D6-BC34-C48B217B80E8@zope.com> <46969F6A.8030904@v.loewis.de> Message-ID: <20070712233252.3C2913A40A9@sparrow.telecommunity.com> At 11:38 PM 7/12/2007 +0200, Martin v. L?wis wrote: > > I really don't like this for setuptools. My preference is that > > setuptools should be required to ask for a package with precise > > spelling. > >I think the way setuptools currently works is this: > >Every name gets converted to its lower-case safe-name equivalent. >All dependencies, file names, resource identifications etc >are based on that version of the name, *not* the "true" >name of the package. Object comparisons are done case-insensitively, but the objects themselves keep the case-insensitive forms ('key' attributes) separate from the originally-input names ('project_name' attributes). >Then, when setuptools tries to find a package whose "true" >name is in mixed-case, it uses the lower-cased safe-named >version, and PyPI reports that the package does not exist. >Then, setuptools queries the entire package list, trying >to find out the original spelling of the package. This is almost correct, except that it actually tries to lookup whatever the user actually input, then the safe_name() form of that. For index lookups, it does not actually change the case of what was entered, so if the user enters something that exactly matches what's on PyPI, they'll have a better chance of getting everything in one request.... unless there are multiple versions listed, of course. From sklein at cpcug.org Fri Jul 13 21:33:40 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Fri, 13 Jul 2007 15:33:40 -0400 (EDT) Subject: [Distutils] In using bdist_rpm, how do I get documentation files to go where they belong? Message-ID: <44255.207.188.248.157.1184355220.squirrel@www.cpcug.org> Now that I understand how to prepare the setup.cfg for Fedora, I have been able to generate some rpms for Python packages using bdist_rpm. However, using only find_packages() causes only the executable part to be packaged. I have made various attempts to get the documentation packaged without success. 1. The first issue is that I get a peculiar error that may be an incorrect setup.py/setup.cfg or may be a bug. The result of the attempt is an error in copying a file called "d" that I didn't declare and doesn't exist. I did declare a directory called "docs". 2. The second issue is that Fedora wants to put documentation files in /usr/share/doc/package_name_and_version and I can't figure out how to make that happen. This placement of documentation is specified under the Unix Filesystem Hierarchy Standard referenced under the Linux Standards Base. The same standard specifies configuration files going under /etc, and the same issue applies if there are any such files. Below is an example from an attempt to package MojoView (from the Python Cheeseshop) with a modified setup.py. I would appreciate any help. Thanks. Stan Klein ------------------------- MojoView packaging attempt ----------------- My setup.py is: from setuptools import setup, find_packages setup(name="MojoView", version="1.0", description="Simple framwork for building PyGTK database applications", author="Jamey Cribbs", author_email="jcribbs at twmi.rr.com", url="http://www.netpromi.com/mojoview.html", packages = find_packages(), data_files = [("docs", "docs/*"), ("examples", "examples/*"), ("gtk", "gtk/*"), ("readme.txt", "readme.txt")] ) The setup.cfg is: #setup.cfg to get proper rpms [install] optimize = 1 [bdist_rpm] doc_files = readme.txt docs/ gtk/ examples/ The output from the attempt is: [stan at localhost MojoView-1.0]$ python setup.py bdist_rpm running bdist_rpm running egg_info writing MojoView.egg-info/PKG-INFO writing top-level names to MojoView.egg-info/top_level.txt writing dependency_links to MojoView.egg-info/dependency_links.txt reading manifest file 'MojoView.egg-info/SOURCES.txt' writing manifest file 'MojoView.egg-info/SOURCES.txt' writing 'build/bdist.linux-i686/rpm/SPECS/MojoView.spec' running sdist warning: sdist: standard file not found: should have one of README, README.txt creating MojoView-1.0 creating MojoView-1.0/MojoView.egg-info creating MojoView-1.0/mojoview making hard links in MojoView-1.0... hard linking setup.cfg -> MojoView-1.0 hard linking setup.py -> MojoView-1.0 hard linking MojoView.egg-info/PKG-INFO -> MojoView-1.0/MojoView.egg-info hard linking MojoView.egg-info/SOURCES.txt -> MojoView-1.0/MojoView.egg-info hard linking MojoView.egg-info/dependency_links.txt -> MojoView-1.0/MojoView.egg-info hard linking MojoView.egg-info/top_level.txt -> MojoView-1.0/MojoView.egg-info hard linking mojoview/__init__.py -> MojoView-1.0/mojoview hard linking mojoview/aboutwin.py -> MojoView-1.0/mojoview hard linking mojoview/browse.py -> MojoView-1.0/mojoview hard linking mojoview/busycursor.py -> MojoView-1.0/mojoview hard linking mojoview/clist.py -> MojoView-1.0/mojoview hard linking mojoview/editdaterange.py -> MojoView-1.0/mojoview hard linking mojoview/editrec.py -> MojoView-1.0/mojoview hard linking mojoview/editwin.py -> MojoView-1.0/mojoview hard linking mojoview/errorwin.py -> MojoView-1.0/mojoview hard linking mojoview/guierror.py -> MojoView-1.0/mojoview hard linking mojoview/mainmenu.py -> MojoView-1.0/mojoview hard linking mojoview/mv.py -> MojoView-1.0/mojoview hard linking mojoview/notebookwin.py -> MojoView-1.0/mojoview hard linking mojoview/toolbar.py -> MojoView-1.0/mojoview copying setup.cfg -> MojoView-1.0 Writing MojoView-1.0/setup.cfg tar -cf dist/MojoView-1.0.tar MojoView-1.0 gzip -f9 dist/MojoView-1.0.tar removing 'MojoView-1.0' (and everything under it) copying dist/MojoView-1.0.tar.gz -> build/bdist.linux-i686/rpm/SOURCES building RPMs rpmbuild -ba --define _topdir /home/stan/exper/mojoview/MojoView-1.0/build/bdist.linux-i686/rpm --clean build/bdist.linux-i686/rpm/SPECS/MojoView.spec Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.61946 + umask 022 + cd /home/stan/exper/mojoview/MojoView-1.0/build/bdist.linux-i686/rpm/BUILD + LANG=C + export LANG + unset DISPLAY + cd /home/stan/exper/mojoview/MojoView-1.0/build/bdist.linux-i686/rpm/BUILD + rm -rf MojoView-1.0 + /bin/gzip -dc /home/stan/exper/mojoview/MojoView-1.0/build/bdist.linux-i686/rpm/SOURCES/MojoView-1.0.tar.gz + tar -xvvf - drwxrwxr-x stan/stan 0 2007-07-13 14:37 MojoView-1.0/ drwxrwxr-x stan/stan 0 2007-07-13 14:37 MojoView-1.0/MojoView.egg-info/ -rw-rw-r-- stan/stan 9 2007-07-13 14:37 MojoView-1.0/MojoView.egg-info/top_level.txt -rw-rw-r-- stan/stan 437 2007-07-13 14:37 MojoView-1.0/MojoView.egg-info/SOURCES.txt -rw-rw-r-- stan/stan 276 2007-07-13 14:37 MojoView-1.0/MojoView.egg-info/PKG-INFO -rw-rw-r-- stan/stan 1 2007-07-13 14:37 MojoView-1.0/MojoView.egg-info/dependency_links.txt -rw-rw-rw- stan/stan 519 2007-07-12 17:00 MojoView-1.0/setup.py drwxrwxr-x stan/stan 0 2007-07-13 14:37 MojoView-1.0/mojoview/ -rw-rw-rw- stan/stan 2034 2003-08-21 14:17 MojoView-1.0/mojoview/editdaterange.py -rw-rw-rw- stan/stan 23415 2003-09-02 16:51 MojoView-1.0/mojoview/browse.py -rw-rw-rw- stan/stan 2212 2003-09-02 16:38 MojoView-1.0/mojoview/notebookwin.py -rw-rw-rw- stan/stan 1477 2003-08-08 15:41 MojoView-1.0/mojoview/editwin.py-rw-rw-rw- stan/stan 593 2003-08-08 15:41 MojoView-1.0/mojoview/busycursor.py -rw-rw-rw- stan/stan 1829 2003-09-02 16:29 MojoView-1.0/mojoview/toolbar.py-rw-rw-rw- stan/stan 16470 2003-09-02 16:50 MojoView-1.0/mojoview/clist.py -rw-rw-rw- stan/stan 373 2003-08-08 15:42 MojoView-1.0/mojoview/errorwin.py -rw-rw-rw- stan/stan 26 2003-08-08 15:39 MojoView-1.0/mojoview/__init__.py -rw-rw-rw- stan/stan 520 2003-08-12 08:56 MojoView-1.0/mojoview/aboutwin.py -rw-rw-rw- stan/stan 3675 2003-09-02 16:39 MojoView-1.0/mojoview/mainmenu.py -rw-rw-rw- stan/stan 317 2003-08-28 14:31 MojoView-1.0/mojoview/mv.py -rw-rw-rw- stan/stan 1983 2003-09-03 13:18 MojoView-1.0/mojoview/guierror.py -rw-rw-rw- stan/stan 2849 2003-09-02 16:27 MojoView-1.0/mojoview/editrec.py-rw-rw-r-- stan/stan 143 2007-07-13 14:37 MojoView-1.0/setup.cfg -rw-rw-r-- stan/stan 276 2007-07-13 14:37 MojoView-1.0/PKG-INFO + STATUS=0 + '[' 0 -ne 0 ']' + cd MojoView-1.0 ++ /usr/bin/id -u + '[' 500 = 0 ']' ++ /usr/bin/id -u + '[' 500 = 0 ']' + /bin/chmod -Rf a+rX,u+w,g-w,o-w . + exit 0 Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.61946 + umask 022 + cd /home/stan/exper/mojoview/MojoView-1.0/build/bdist.linux-i686/rpm/BUILD + cd MojoView-1.0 + LANG=C + export LANG + unset DISPLAY + python setup.py build running build running build_py creating build creating build/lib creating build/lib/mojoview copying mojoview/editdaterange.py -> build/lib/mojoview copying mojoview/browse.py -> build/lib/mojoview copying mojoview/notebookwin.py -> build/lib/mojoview copying mojoview/editwin.py -> build/lib/mojoview copying mojoview/busycursor.py -> build/lib/mojoview copying mojoview/toolbar.py -> build/lib/mojoview copying mojoview/clist.py -> build/lib/mojoview copying mojoview/errorwin.py -> build/lib/mojoview copying mojoview/__init__.py -> build/lib/mojoview copying mojoview/aboutwin.py -> build/lib/mojoview copying mojoview/mainmenu.py -> build/lib/mojoview copying mojoview/mv.py -> build/lib/mojoview copying mojoview/guierror.py -> build/lib/mojoview copying mojoview/editrec.py -> build/lib/mojoview + exit 0 Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.75576 + umask 022 + cd /home/stan/exper/mojoview/MojoView-1.0/build/bdist.linux-i686/rpm/BUILD + cd MojoView-1.0 + LANG=C + export LANG + unset DISPLAY + python setup.py install --single-version-externally-managed --root=/var/tmp/MojoView-1.0-1-buildroot --record=INSTALLED_FILES running install running build running build_py running install_lib writing byte-compilation script '/tmp/tmp0s5onf.py' /usr/bin/python -O /tmp/tmp0s5onf.py removing /tmp/tmp0s5onf.py running install_data error: can't copy 'd': doesn't exist or not a regular file error: Bad exit status from /var/tmp/rpm-tmp.75576 (%install) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.75576 (%install) error: command 'rpmbuild' failed with exit status 1 From steve at holdenweb.com Sat Jul 14 15:32:42 2007 From: steve at holdenweb.com (Steve Holden) Date: Sat, 14 Jul 2007 09:32:42 -0400 Subject: [Distutils] Cygwin build of PIL 1.1.6 hangs Message-ID: <4698D07A.6090400@holdenweb.com> Has anyone noticed or reported Cygwin problems? The ordinarily well-behaved PIL install hangs on Cygwin. I put debug prints inside spawn.py and it appears that the os.waitpid(pid, 0) inside spawn_posix is failing to return when the gcc task completes. Nothing on Google about this that I can find, and no response from either the PIL or Cygwin lists. This seems a pretty fundamental breakage to not get noticed. What have I done wrong? This is using Python 2.5.2 (same thing happens with 2.5.1) on Cygwin 1.5.24, using gcc 3.4.4-3. The task manager shows 100% CPU (with kernel usage hovering about 35%) and a python.exe process gobbling CPU like there's no tomorrow. If I CTRL/C the build I get an "interrupted" diagnostic but the python.exe (presumably the child) continues to gobble CPU until killed from the task manager. I haven't yet started debugging the child process, as I am hoping someone knows of a fix. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden --------------- Asciimercial ------------------ Get on the web: Blog, lens and tag the Internet Many services currently offer free registration ----------- Thank You for Reading ------------- From optilude at gmx.net Sun Jul 15 21:46:41 2007 From: optilude at gmx.net (Martin Aspeli) Date: Sun, 15 Jul 2007 21:46:41 +0200 Subject: [Distutils] Buildout error reporting - connection refused Message-ID: <469A79A1.7020603@gmx.net> Hi Jim & co, I just ran a buildout and got this error. It went away again (I guess it was a server going down), but it annoyed me that I couldn't work out *which* URL it was trying to connect to that got refused. Could buildout produce a bit more information in this case? Also - I had a find-link at this point that I know was down. It is not the link for python-openid, so it shouldn't matter, it's only a backup mirror (which atm is even empty). But would setuptools/buildout bomb out regardless, if one of the find-links was down? Thanks, Martin $ ./bin/buildout -vv Installing 'zc.buildout', 'setuptools'. We have the best distribution that satisfies 'zc.buildout'. Picked: zc.buildout = 1.0.0b28 We have the best distribution that satisfies 'setuptools'. Picked: setuptools = 0.6c6 Installing 'plone.recipe.plone'. We have the best distribution that satisfies 'plone.recipe.plone'. Adding find link 'http://download.zope.org/distribution/' from plone.recipe.plone 3.0rc1 Picked: plone.recipe.plone = 3.0rc1 Getting required 'plone.recipe.distros' required by plone.recipe.plone 3.0rc1. We have the best distribution that satisfies 'plone.recipe.distros'. Picked: plone.recipe.distros = 0.2 Getting required 'zc.recipe.egg' required by plone.recipe.plone 3.0rc1. We have the best distribution that satisfies 'zc.recipe.egg'. Picked: zc.recipe.egg = 1.0.0b6 Installing 'plone.recipe.zope2install'. We have the best distribution that satisfies 'plone.recipe.zope2install'. Adding find link 'http://download.zope.org/distribution/' from plone.recipe.zope2install 0.1 Picked: plone.recipe.zope2install = 0.1 Installing 'plone.recipe.zope2instance'. We have the best distribution that satisfies 'plone.recipe.zope2instance'. Adding find link 'http://download.zope.org/distribution/' from plone.recipe.zope2instance 0.6.1 Picked: plone.recipe.zope2instance = 0.6.1 Configuration data: [zope2] executable = /opt/local/Library/Frameworks/Python.framework/Versions/2.4/bin/python location = /Users/optilude/tmp/eggs/zopeskel/rc1/parts/zope2 recipe = plone.recipe.zope2install url = http://www.zope.org/Products/Zope/2.10.4/Zope-2.10.4-final.tgz [buildout] bin-directory = /Users/optilude/tmp/eggs/zopeskel/rc1/bin develop = develop-eggs-directory = /Users/optilude/tmp/eggs/zopeskel/rc1/develop-eggs directory = /Users/optilude/tmp/eggs/zopeskel/rc1 download-directory = /Users/optilude/tmp/eggs/zopeskel/rc1/downloads eggs = eggs-directory = /Users/optilude/.buildout/eggs executable = /opt/local/Library/Frameworks/Python.framework/Versions/2.4/bin/python find-links = installed = /Users/optilude/tmp/eggs/zopeskel/rc1/.installed.cfg log-format = log-level = INFO newest = true offline = false parts = %(__buildout_space_n__)splone zope2 productdistros instance zopepy parts-directory = /Users/optilude/tmp/eggs/zopeskel/rc1/parts python = buildout verbosity = 20 [productdistros] location = /Users/optilude/tmp/eggs/zopeskel/rc1/parts/productdistros nested-packages = prefix = /Users/optilude/tmp/eggs/zopeskel/rc1/parts/productdistros recipe = plone.recipe.distros urls = version-suffix-packages = [instance] _b = /Users/optilude/tmp/eggs/zopeskel/rc1/bin _d = /Users/optilude/tmp/eggs/zopeskel/rc1/develop-eggs _e = /Users/optilude/.buildout/eggs bin-directory = /Users/optilude/tmp/eggs/zopeskel/rc1/bin debug-mode = on develop-eggs-directory = /Users/optilude/tmp/eggs/zopeskel/rc1/develop-eggs eggs = %(__buildout_space_n__)s%(__buildout_space_n__)sarchetypes.kss==1.2-rc1.1 kss.core==1.2-rc1.1 plone.app.content==1.0rc1 plone.app.contentmenu==1.0rc1 plone.app.contentrules==1.0rc1 plone.app.controlpanel==1.0rc1 plone.app.customerize==1.0rc1 plone.app.form==1.0rc1 plone.app.i18n==1.0rc1 plone.app.iterate==1.0rc1 plone.app.kss==1.2-rc1.1 plone.app.layout==1.0rc1 plone.app.linkintegrity==1.0rc1.1 plone.app.openid==1.0rc1 plone.app.portlets==1.0rc1 plone.app.redirector==1.0rc1 plone.app.viewletmanager==1.0rc1 plone.app.vocabularies==1.0rc1 plone.app.workflow==1.0rc1 plone.contentrules==1.0rc1 plone.fieldsets==1.0rc1 plone.i18n==1.0rc1 plone.intelligenttext==1.0rc1 plone.locking==1.0rc1 plone.memoize==1.0rc1 plone.openid==1.0rc1 plone.portlets==1.0rc1 plone.session==1.0rc1 plone.theme==1.0rc1 wicked==1.1.3 five.customerize==0.1.3 five.localsitemanager==0.2 eggs-directory = /Users/optilude/.buildout/eggs executable = /opt/local/Library/Frameworks/Python.framework/Versions/2.4/bin/python http-address = 8080 location = /Users/optilude/tmp/eggs/zopeskel/rc1/parts/instance products = %(__buildout_space_n__)s/Users/optilude/tmp/eggs/zopeskel/rc1/products /Users/optilude/tmp/eggs/zopeskel/rc1/parts/productdistros /Users/optilude/tmp/eggs/zopeskel/rc1/parts/plone recipe = plone.recipe.zope2instance scripts = user = admin:admin verbose-security = on zcml = zope2-location = /Users/optilude/tmp/eggs/zopeskel/rc1/parts/zope2 [zopepy] _b = /Users/optilude/tmp/eggs/zopeskel/rc1/bin _d = /Users/optilude/tmp/eggs/zopeskel/rc1/develop-eggs _e = /Users/optilude/.buildout/eggs bin-directory = /Users/optilude/tmp/eggs/zopeskel/rc1/bin develop-eggs-directory = /Users/optilude/tmp/eggs/zopeskel/rc1/develop-eggs eggs = %(__buildout_space_n__)s%(__buildout_space_n__)sarchetypes.kss==1.2-rc1.1 kss.core==1.2-rc1.1 plone.app.content==1.0rc1 plone.app.contentmenu==1.0rc1 plone.app.contentrules==1.0rc1 plone.app.controlpanel==1.0rc1 plone.app.customerize==1.0rc1 plone.app.form==1.0rc1 plone.app.i18n==1.0rc1 plone.app.iterate==1.0rc1 plone.app.kss==1.2-rc1.1 plone.app.layout==1.0rc1 plone.app.linkintegrity==1.0rc1.1 plone.app.openid==1.0rc1 plone.app.portlets==1.0rc1 plone.app.redirector==1.0rc1 plone.app.viewletmanager==1.0rc1 plone.app.vocabularies==1.0rc1 plone.app.workflow==1.0rc1 plone.contentrules==1.0rc1 plone.fieldsets==1.0rc1 plone.i18n==1.0rc1 plone.intelligenttext==1.0rc1 plone.locking==1.0rc1 plone.memoize==1.0rc1 plone.openid==1.0rc1 plone.portlets==1.0rc1 plone.session==1.0rc1 plone.theme==1.0rc1 wicked==1.1.3 five.customerize==0.1.3 five.localsitemanager==0.2 eggs-directory = /Users/optilude/.buildout/eggs executable = /opt/local/Library/Frameworks/Python.framework/Versions/2.4/bin/python extra-paths = /Users/optilude/tmp/eggs/zopeskel/rc1/parts/zope2/lib/python interpreter = zopepy recipe = zc.recipe.egg scripts = zopepy [plone] _b = /Users/optilude/tmp/eggs/zopeskel/rc1/bin _d = /Users/optilude/tmp/eggs/zopeskel/rc1/develop-eggs _e = /Users/optilude/.buildout/eggs bin-directory = /Users/optilude/tmp/eggs/zopeskel/rc1/bin develop-eggs-directory = /Users/optilude/tmp/eggs/zopeskel/rc1/develop-eggs eggs = archetypes.kss==1.2-rc1.1 kss.core==1.2-rc1.1 plone.app.content==1.0rc1 plone.app.contentmenu==1.0rc1 plone.app.contentrules==1.0rc1 plone.app.controlpanel==1.0rc1 plone.app.customerize==1.0rc1 plone.app.form==1.0rc1 plone.app.i18n==1.0rc1 plone.app.iterate==1.0rc1 plone.app.kss==1.2-rc1.1 plone.app.layout==1.0rc1 plone.app.linkintegrity==1.0rc1.1 plone.app.openid==1.0rc1 plone.app.portlets==1.0rc1 plone.app.redirector==1.0rc1 plone.app.viewletmanager==1.0rc1 plone.app.vocabularies==1.0rc1 plone.app.workflow==1.0rc1 plone.contentrules==1.0rc1 plone.fieldsets==1.0rc1 plone.i18n==1.0rc1 plone.intelligenttext==1.0rc1 plone.locking==1.0rc1 plone.memoize==1.0rc1 plone.openid==1.0rc1 plone.portlets==1.0rc1 plone.session==1.0rc1 plone.theme==1.0rc1 wicked==1.1.3 five.customerize==0.1.3 five.localsitemanager==0.2 eggs-directory = /Users/optilude/.buildout/eggs executable = /opt/local/Library/Frameworks/Python.framework/Versions/2.4/bin/python find-links = http://dist.plone.org location = /Users/optilude/tmp/eggs/zopeskel/rc1/parts/plone nested-packages = CMF-2.1.0-beta2.tar.gz prefix = /Users/optilude/tmp/eggs/zopeskel/rc1/parts/plone products = /Users/optilude/tmp/eggs/zopeskel/rc1/parts/plone python-url = http://www.python.org/ftp/python/2.4.4/Python-2.4.4.tgz recipe = plone.recipe.plone urls = http://www.zope.org/Products/CMF/CMF-2.1.0-beta2/CMF-2.1.0-beta2.tar.gz http://plone.org/products/atcontenttypes/releases/1.2/atcontenttypes-1-2-0-rc1.tgz http://plone.org/products/atreferencebrowserwidget/releases/2.0/ATReferenceBrowserWidget-2.0rc1.tar.gz http://plone.org/products/cmfdynamicviewfti/releases/3.0/CMFDynamicViewFTI-3.0rc1.tar.gz http://plone.org/products/cmfformcontroller/releases/2.1/CMFFormController-2.1rc1.tar.gz http://plone.org/products/cmfplacefulworkflow/releases/1.2.0-rc1/cmfplacefulworkflow-1-2-0-rc1.tgz http://plone.googlecode.com/files/PloneBase-3.0-rc1.tar.gz http://plone.org/products/cmfquickinstallertool/releases/2.0.3/CMFQuickInstallerTool-2.0.3.tar.gz http://plone.org/products/cmfdifftool/releases/0.3.4/cmfdifftool-0-3-4.tgz http://plone.org/products/cmftestcase/releases/0.9.6/CMFTestCase-0.9.6.tar.gz http://plone.org/products/extendedpathindex/releases/2.4/ExtendedPathIndex-2.4.tgz http://plone.org/products/external-editor/releases/0.9.3/ExternalEditor-0.9.3-src.tgz http://plone.org/products/groupuserfolder/releases/3.54.4/GroupUserFolder-3.54.4.tgz http://plone.org/products/kupu/releases/1.4/kupu-1-4rc1.tgz http://plone.org/products/pts/releases/1.4.4/PlacelessTranslationService-1.4.4.tar.gz http://plone.org/products/plonetestcase/releases/0.9.6/PloneTestCase-0.9.6.tar.gz http://plone.org/products/plonetranslations/releases/3.0.3/PloneTranslations-3.0.3rc1.tar.gz http://plone.org/products/plonelanguagetool/releases/2.0/PloneLanguageTool-2.0rc1.tar.gz http://plone.org/products/securemailhost/releases/1.1/SecureMailHost-1.1.tar.gz http://plone.org/products/resourceregistries/releases/1.4.0/resourceregistries-1.4.0-rc1.tgz http://plone.org/products/statusmessages/releases/3.0/statusmessages-3.0rc1.tar.gz http://plone.org/products/plonepas/releases/3.0/plonepas-3-0rc1-tar.gz http://www.zope.org/Products/PluggableAuthService/PluggableAuthService-1.5/PluggableAuthService-1.5.tar.gz http://plone.org/products/passwordresettool/releases/1.0/PasswordResetTool-1.0rc1.tar.gz http://www.zope.org/Products/PluginRegistry/PluginRegistry-1.1.2/PluginRegistry-1.1.2.tar.gz http://antiloop.plone.org/download/ZopeVersionControl-0.3.4.tar.gz http://plone.org/products/cmfeditions/releases/1.1/cmfeditions-1-1-rc1.tgz http://plone.org/products/nuplone/releases/0.8/NuPlone-0.8.tgz http://www.dieter.handshake.de/pyprojects/zope/AdvancedQuery.tgz http://plone.org/products/archetypes/releases/1.5/Archetypes-1.5.0-rc2.tar.gz version-suffix-packages = PluggableAuthService-1.5.tar.gz PluginRegistry-1.1.2.tar.gz zope2-url = http://www.zope.org/Products/Zope/2.10.4/Zope-2.10.4-final.tgz Installing plone. Installing 'archetypes.kss==1.2-rc1.1', 'kss.core==1.2-rc1.1', 'plone.app.content==1.0rc1', 'plone.app.contentmenu==1.0rc1', 'plone.app.contentrules==1.0rc1', 'plone.app.controlpanel==1.0rc1', 'plone.app.customerize==1.0rc1', 'plone.app.form==1.0rc1', 'plone.app.i18n==1.0rc1', 'plone.app.iterate==1.0rc1', 'plone.app.kss==1.2-rc1.1', 'plone.app.layout==1.0rc1', 'plone.app.linkintegrity==1.0rc1.1', 'plone.app.openid==1.0rc1', 'plone.app.portlets==1.0rc1', 'plone.app.redirector==1.0rc1', 'plone.app.viewletmanager==1.0rc1', 'plone.app.vocabularies==1.0rc1', 'plone.app.workflow==1.0rc1', 'plone.contentrules==1.0rc1', 'plone.fieldsets==1.0rc1', 'plone.i18n==1.0rc1', 'plone.intelligenttext==1.0rc1', 'plone.locking==1.0rc1', 'plone.memoize==1.0rc1', 'plone.openid==1.0rc1', 'plone.portlets==1.0rc1', 'plone.session==1.0rc1', 'plone.theme==1.0rc1', 'wicked==1.1.3', 'five.customerize==0.1.3', 'five.localsitemanager==0.2'. We have the distribution that satisfies 'archetypes.kss==1.2-rc1.1'. We have the distribution that satisfies 'kss.core==1.2-rc1.1'. We have the distribution that satisfies 'plone.app.content==1.0rc1'. We have the distribution that satisfies 'plone.app.contentmenu==1.0rc1'. We have the distribution that satisfies 'plone.app.contentrules==1.0rc1'. We have the distribution that satisfies 'plone.app.controlpanel==1.0rc1'. We have the distribution that satisfies 'plone.app.customerize==1.0rc1'. We have the distribution that satisfies 'plone.app.form==1.0rc1'. We have the distribution that satisfies 'plone.app.i18n==1.0rc1'. We have the distribution that satisfies 'plone.app.iterate==1.0rc1'. We have the distribution that satisfies 'plone.app.kss==1.2-rc1.1'. We have the distribution that satisfies 'plone.app.layout==1.0rc1'. We have the distribution that satisfies 'plone.app.linkintegrity==1.0rc1.1'. We have the distribution that satisfies 'plone.app.openid==1.0rc1'. We have the distribution that satisfies 'plone.app.portlets==1.0rc1'. We have the distribution that satisfies 'plone.app.redirector==1.0rc1'. We have the distribution that satisfies 'plone.app.viewletmanager==1.0rc1'. We have the distribution that satisfies 'plone.app.vocabularies==1.0rc1'. We have the distribution that satisfies 'plone.app.workflow==1.0rc1'. We have the distribution that satisfies 'plone.contentrules==1.0rc1'. We have the distribution that satisfies 'plone.fieldsets==1.0rc1'. We have the distribution that satisfies 'plone.i18n==1.0rc1'. We have the distribution that satisfies 'plone.intelligenttext==1.0rc1'. We have the distribution that satisfies 'plone.locking==1.0rc1'. We have the distribution that satisfies 'plone.memoize==1.0rc1'. We have the distribution that satisfies 'plone.openid==1.0rc1'. We have the distribution that satisfies 'plone.portlets==1.0rc1'. We have the distribution that satisfies 'plone.session==1.0rc1'. We have the distribution that satisfies 'plone.theme==1.0rc1'. We have the distribution that satisfies 'wicked==1.1.3'. We have the distribution that satisfies 'five.customerize==0.1.3'. We have the distribution that satisfies 'five.localsitemanager==0.2'. Getting required 'setuptools' required by five.customerize 0.1.3. required by five.localsitemanager 0.2. required by plone.app.content 1.0rc1. required by plone.app.contentmenu 1.0rc1. required by plone.app.contentrules 1.0rc1. required by plone.app.controlpanel 1.0rc1. required by plone.app.customerize 1.0rc1. required by plone.app.form 1.0rc1. required by plone.app.i18n 1.0rc1. required by plone.app.iterate 1.0rc1. required by plone.app.layout 1.0rc1. required by plone.app.openid 1.0rc1. required by plone.app.portlets 1.0rc1. required by plone.app.redirector 1.0rc1. required by plone.app.viewletmanager 1.0rc1. required by plone.app.vocabularies 1.0rc1. required by plone.app.workflow 1.0rc1. required by plone.contentrules 1.0rc1. required by plone.fieldsets 1.0rc1. required by plone.i18n 1.0rc1. required by plone.intelligenttext 1.0rc1. required by plone.locking 1.0rc1. required by plone.memoize 1.0rc1. required by plone.openid 1.0rc1. required by plone.portlets 1.0rc1. required by plone.session 1.0rc1. required by plone.theme 1.0rc1. required by plone.app.linkintegrity 1.0rc1.1. required by wicked 1.1.3. required by archetypes.kss 1.2-rc1.1. required by kss.core 1.2-rc1.1. required by plone.app.kss 1.2-rc1.1. Download error: (61, 'Connection refused') -- Some packages may not be found! We have the best distribution that satisfies 'setuptools'. Picked: setuptools = 0.6c6 Getting required 'python-openid>=2.0.0,<2.0.999' required by plone.openid 1.0rc1. We have the best distribution that satisfies 'python-openid>=2.0.0,<2.0.999'. Picked: python-openid = 2.0.1 While: Installing plone. An internal error occured due to a bug in either zc.buildout or in a recipe being used: URLError: -- Acquisition is a jealous mistress From optilude at gmx.net Sun Jul 15 23:12:58 2007 From: optilude at gmx.net (Martin Aspeli) Date: Sun, 15 Jul 2007 23:12:58 +0200 Subject: [Distutils] Empty directories in eggs Message-ID: Hi, I tried to build an egg from this tag, using bdist_egg and sdist: http://svn.plone.org/svn/collective/ZopeSkel/tags/0.5/ However, the packaged egg lost several directories, such as: http://svn.plone.org/svn/collective/ZopeSkel/tags/0.5/zopeskel/templates/plone3_buildout/products/ These are empty, but under version control. To get them into the egg (I need it because it's part of a Paste Script template which expects the empty directories to be there), I had to add some mostly-empty (if slightly useful) README.txt files (on the 0.6 tag). Bug or feature? ;-) Martin -- Acquisition is a jealous mistress From miles at jamkit.com Mon Jul 16 13:07:03 2007 From: miles at jamkit.com (Miles) Date: Mon, 16 Jul 2007 12:07:03 +0100 Subject: [Distutils] zc.buildout and zc.recipe.egg Message-ID: Hi, Apologies for the length of this post. I am using zc.buildout to compile a specific python version with various extra libraries (e.g. aspell, mysql). This is all working fine, and I have it creating an interpreter with all the required libraries via the "interpreter" parameter. However, I then want to compile/make/make install zope2 against this specific python interpreter via ./configure --with-python=./bin/mypython etc... I am using the zc.recipe.cmmi recipe to do this. This fails (it works if I use the python executable as the --with-python argument). I've tracked it down to a small difference between the standard python interpreter and the one generated by the zc.recipe.egg recipe: - the standard python interpreter seems to add the directory of the file you are running to the path - the generated one does not add the directory to the path This means that ./Python2.3/bin/python inst/configure.py runs fine but ./bin/mypython inst/configure.py fails with an import error as it cannot import anything relative to the file being run I don't want to add it in as an "extra_path" because it is not needed after this step and I don't want it to be part of the standard environment. My question is: - is this a desirable difference in behaviour between the two interpreters? I expected them to work exactly the same... - what's the cleanest way to get this working consistently/in an automated way through the buildout? At the moment, I have manually added the following into the generated script to recreate the behaviour: if _args: + import os + sys.argv[:] = _args + f = sys.argv[0] + p = os.path.split( os.path.abspath(f) )[:-1] + sys.path[0:0] = os.path.join( p ) execfile(f) Thanks, Miles From zooko at zooko.com Mon Jul 16 22:07:58 2007 From: zooko at zooko.com (Zooko O'Whielacronx) Date: Mon, 16 Jul 2007 14:07:58 -0600 Subject: [Distutils] .py vs .pyc (was: Newbie questions about setuptools) In-Reply-To: <20070707170235.2194D3A405F@sparrow.telecommunity.com> References: <002c01c7b280$89df76b0$6a00a8c0@your4dacd0ea75> <20070619163156.CB7023A40A5@sparrow.telecommunity.com> <97DEE0D1-80EF-4D74-A783-1D66BDB6106F@zooko.com> <20070707170235.2194D3A405F@sparrow.telecommunity.com> Message-ID: <190DB14F-EBE3-4726-A99E-464E943BB00E@zooko.com> On Jul 7, 2007, at 11:04 AM, Phillip J. Eby wrote: > At 10:12 AM 7/7/2007 -0600, Zooko O'Whielacronx wrote: >> >> I would prefer to ship compiled C modules and source .py files. ... >> So, please accept this as a feature request to make the above- >> mentioned behavior configurable for users such as me. > > For the reasons I already explained above, this is a no-go. If you > want to ship binaries plus source, ship eggs. If you want to ship > without .pyc/.pyo files, ship an sdist. > > This distinction is not about your preferences as a distributor, > but about your users. As a user, I don't want to pay for .py- > compilation every time I import your modules, just to save you a > few extra bytes of bandwidth. > > If the option existed to ship without compiled modules, people > would use it. Following which, other people would begin claiming > that "eggs are slow". And all this, for what actual benefit? > Absolutely zip! ...no pun intended. My motivations are to benefit my users, not to reduce my bandwidth usage (which is negligible). Perhaps some users have different values than others with respect to download time versus launch time. Have you measured the relative launch time for .py's vs. .pyc's? I certainly wouldn't want to impose an excessive increase in launch time, but if it is sufficiently small then I may choose to ship just-.py instead of just-.pyc or both-.py-and-.pyc. Let's see... Hm. After experimenting with a bit here, it seems to take just as long to launch with .pyc's-and-.py's as with .py's alone, but it seems to be faster with .pyo's-and-.py's. However, I don't trust these results -- I only ran each test once, and this machine isn't idle. So it remains an open question in my mind as to how much, if any, .pyc's accelerates launch time. Perhaps I'll do some better benchmarking next time I find myself packaging up a release of my major project (http://allmydata.org ). If anyone else has measured this behavior more carefully I'd like to hear about it. Just to reiterate, I value making source code easily available to users, so I would prefer to ship .py files unless it is too costly. (And I would prefer not to ship .pyc or .pyo files unless they help with launch time more than they hurt with download time.) Regards, Zooko From lists at zopyx.com Tue Jul 17 12:15:35 2007 From: lists at zopyx.com (Andreas Jung) Date: Tue, 17 Jul 2007 12:15:35 +0200 Subject: [Distutils] [easy_install] won't upgrade egg from 1.0.9 -> 1.0.10 Message-ID: I uploaded z3c.sqlalchemy 1.0.10 to Cheeseshop. easy_install -U z3c.sqlalchemy won't upgrade an installed 1.0.9 egg to the new version unless I pass the current version as "z3c.sqlalchemy==1.0.10". Bug or feature? Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 186 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070717/992e36b2/attachment.pgp From pje at telecommunity.com Tue Jul 17 17:46:24 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 17 Jul 2007 11:46:24 -0400 Subject: [Distutils] [easy_install] won't upgrade egg from 1.0.9 -> 1.0.10 In-Reply-To: References: Message-ID: <20070717155111.2109F3A403A@sparrow.telecommunity.com> At 12:15 PM 7/17/2007 +0200, Andreas Jung wrote: >I uploaded z3c.sqlalchemy 1.0.10 to Cheeseshop. >easy_install -U z3c.sqlalchemy won't upgrade an installed 1.0.9 egg >to the new version unless I pass the current version as >"z3c.sqlalchemy==1.0.10". > >Bug or feature? Sounds like a bug, except that 1.0.10 > 1.0.9: >>> from pkg_resources import parse_version as pv >>> pv('1.0.9')>> pv('1.0.10')>pv('1.0.9') True So it's not in the version testing, or shouldn't be. Try using the -v option to easy_install and see what URLs it's picking up. Also, is the problem reproducible? i.e., if you get rid of the new version and put back the old, do you still have the same problem? From staellar at gmail.com Tue Jul 17 20:31:58 2007 From: staellar at gmail.com (vadim suvorov) Date: Tue, 17 Jul 2007 13:31:58 -0500 Subject: [Distutils] unattended install Message-ID: I have a customer who has many (Windows) computers on which Python and related packages must be installed/updated on regular bases. One of their concerns is wininst based packages, which do not allow unattended installation. Newer and greater MSI helps, but there are legacy packages. Can you give me opinion: 1) Why pywin32 (Mark Hammond) is continued to be distributed in wininst form? 2) If one prepares MSI package, are there foreseeable snags to distribute it in 2.4 environment? 3) I offered to consider development of wininst to msi (ready package) converter. Is it stupid suggestion, or is it feasable (and easier than silencing of wininst)? Would there be interest for such converter (my customer is willing to contribute to Python project)? Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070717/038bf274/attachment.htm From mhammond at skippinet.com.au Tue Jul 17 23:58:35 2007 From: mhammond at skippinet.com.au (Mark Hammond) Date: Wed, 18 Jul 2007 07:58:35 +1000 Subject: [Distutils] unattended install In-Reply-To: Message-ID: <0a0601c7c8bd$a00e89f0$090a0a0a@enfoldsystems.local> > I have a customer who has many (Windows) computers on which Python and related > packages must be installed/updated on regular bases. > One of their concerns is wininst based packages, which do not allow > unattended installation. Newer and greater MSI helps, but there > are legacy packages. > Can you give me opinion: > 1) Why pywin32 (Mark Hammond) is continued to be distributed > in wininst form? The bdist_msi packager is missing some features of bdist_wininst; things like adding of shortcuts, a "post_install" script that is capable of recording files to be uninstalled, communication of setup options to the install scripts, etc. > 2) If one prepares MSI package, are there foreseeable snags to distribute it in > 2.4 environment? See above - although I am currently working on bdist_msi so I can package an x64 version - this is likely to make it easier for me to use bdist_msi - although at this stage I'm not sure about moving all releases to bdist_wininst. > 3) I offered to consider development of wininst to msi (ready package) converter. What would such a converter do? For simple packages with an existing distutils setup script, you simply use a different command-line to create an MSI instead of an EXE, so a converter seems unnecessary. A converter could not address the issues I listed above either (ie, functionality available to the post-install script) > Is it stupid suggestion, or is it feasable (and easier than silencing of wininst)? > Would there be interest for such converter (my customer is willing to contribute to Python project)? Other alternatives include: * Open the .exe installer as a .zip file and simply copy the files into the target * Enhance bdist_wininst such that it supports args for silent install, then ask people maintaining the packages to use this new executable. I'd have no problem doing that. Cheers, Mark From staellar at gmail.com Wed Jul 18 00:38:00 2007 From: staellar at gmail.com (vadim suvorov) Date: Tue, 17 Jul 2007 17:38:00 -0500 Subject: [Distutils] unattended install In-Reply-To: <0a0601c7c8bd$a00e89f0$090a0a0a@enfoldsystems.local> References: <0a0601c7c8bd$a00e89f0$090a0a0a@enfoldsystems.local> Message-ID: Thank you very much for clarifications. >> > 3) I offered to consider development of wininst to msi (ready package) > >> converter. > > >> What would such a converter do? For simple packages with an existing > >> distutils setup script, you simply use a different command-line to > create an > >> MSI instead of an EXE, so a converter seems unnecessary. A converter > could > >> not address the issues I listed above either (ie, functionality > available to > >> the post-install script) > The proposed converter would handle third-party modules, which came in as ready-to-use .exe files. An idea to read distribution.exe (as zip) and write as msi distribution, which would be installed than on all computers in need. That was based on presumption that wininst is superseded by msi, which is clearly incorrect, as you've shown. Therefore conversion would allow to use 'legacy' distributions (without using wininst). Can you comment on one more idea: instead of modifying bdist_wininst, to modify wininst.exe to support silent installation. This change could be obviously submitted as well, and as an interim solution simple replacement of stub (in distribution.exe) could be used. Vadim -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070717/bb947046/attachment.htm From mhammond at skippinet.com.au Wed Jul 18 01:08:54 2007 From: mhammond at skippinet.com.au (Mark Hammond) Date: Wed, 18 Jul 2007 09:08:54 +1000 Subject: [Distutils] unattended install In-Reply-To: Message-ID: <0a1001c7c8c7$72c775b0$090a0a0a@enfoldsystems.local> > That was based on presumption that wininst is superseded by msi, which is > clearly incorrect, as you've shown. Well, all I really meant to say was that bdist_msi doesn't (yet) have all the features of bdist_wininst - but bdist_msi does appear to be the "future" (ie, there is not much interest on this list towards keeping bdist_wininst alive going forward) > Can you comment on one more idea: instead of modifying bdist_wininst, to modify > wininst.exe to support silent installation. This change could be obviously > submitted as well, and as an interim solution simple replacement of stub (in > distribution.exe) could be used. Yes, that is what I meant - modify the stub executable bdist_wininst uses to perform its magic. However, keep in mind the comments above regarding the (lack of) future for bdist_wininst - eg, even with this modification, its not clear it will ever be suitable for installing 64bit Python extensions. So while it may be suitable for your specific problem, you may need to revise things when x64 moves into your radar, for example. Mark From staellar at gmail.com Wed Jul 18 01:21:42 2007 From: staellar at gmail.com (vadim suvorov) Date: Tue, 17 Jul 2007 18:21:42 -0500 Subject: [Distutils] unattended install In-Reply-To: <0a1001c7c8c7$72c775b0$090a0a0a@enfoldsystems.local> References: <0a1001c7c8c7$72c775b0$090a0a0a@enfoldsystems.local> Message-ID: Thank you. It was my impression also. I guess, I need to do more research regarding available options. Vadim -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070717/f9e3e3c8/attachment.html From mhammond at skippinet.com.au Wed Jul 18 07:29:23 2007 From: mhammond at skippinet.com.au (Mark Hammond) Date: Wed, 18 Jul 2007 15:29:23 +1000 Subject: [Distutils] cross-compiling using distutils Message-ID: <0a3c01c7c8fc$9a082f90$090a0a0a@enfoldsystems.local> Hi all, I'm making good progress on using bdist_msi to package up x64 versions of the pywin32 extensions. On the way I've struck a number of issues. In an effort to try and keey the discussions focussed, I'll create new threads for each of them, and the first cab off the rank is cross-compilation. IIUC, the only way to build Python or Python extensions using VS2003 is to cross-compile from a 32bit platform. Although this requirement is only short term, in the longer term we should support cross-compilation to prevent a Windows developer needing access to all architectures just to create a build. Here are the problems I had: * The dev box needs access to both a 32bit and a 64bit version of Python - the 32bit one to execute Python itself, and the 64bit one to link against. * PYTHONHOME had to be set to the x64 version of Python when running the build, otherwise the link would fail as it tries to use the 32bit python.lib * '/MACHINE:AMD64' or '/MACHINE:Itanium' manually needs to be added to the link commandline. * '/D_M_X64' had to be passed to the compiler (which seems unusual, but I didn't dig too deep into this) One way to approach this might be to add a "global" option to distutils. It might be possible to call it --python_bin_dir, which would point at the libraries for the target system (and we could attempt to sniff the architecture from them). It would need to be a 'global' option as many commands may need to know the target (eg, bdist_msi would also need to know) and distutils.util.get_platform() would need to change to reflect the target platform, not the platform being used to perform the build. msvccompiler.get_build_architecture() could also change to take advantage of this and grow support for /MACHINE etc. Does anyone have any thoughts on this? I'm happy to come up with a patch, but I'm looking for direction (or even just encouragement ) Thanks, Mark From mhammond at skippinet.com.au Wed Jul 18 08:00:28 2007 From: mhammond at skippinet.com.au (Mark Hammond) Date: Wed, 18 Jul 2007 16:00:28 +1000 Subject: [Distutils] distutils.util.get_platform() for Windows Message-ID: <0a4001c7c900$f1ba2640$090a0a0a@enfoldsystems.local> Many of the distutils "commands" use distutils.util.get_platform() as the basis for file and directory names used to package up extensions. On Windows, this returns the value of sys.platform. On all (desktop) Windows versions, this currently returns 'win32'. This causes a problem when trying to create a 64bit version of an extension. For example, using bdist_msi, the pywin32 extensions end up with a filename of 'pywin32-211.win32-py2.5.msi' for both 32bit and 64bit versions. This is not desirable for (hopefully) obvious reasons. I'd like to propose that an (untested and against 2.5) patch similar to the following be adopted in distutils: Index: util.py =================================================================== --- util.py (revision 56286) +++ util.py (working copy) @@ -29,8 +29,19 @@ irix-5.3 irix64-6.2 - For non-POSIX platforms, currently just returns 'sys.platform'. + For Windows, the result will be one of 'win32', 'amd64' or 'itanium' + + For other non-POSIX platforms, currently just returns 'sys.platform'. """ + if os.name == 'nt': + # copied from msvccompiler - find the processor architecture + prefix = " bit (" + i = string.find(sys.version, prefix) + if i == -1: + return sys.platform + j = string.find(sys.version, ")", i) + return sys.version[i+len(prefix):j].lower() + if os.name != "posix" or not hasattr(os, 'uname'): # XXX what about the architecture? NT is Intel or Alpha, # Mac OS is M68k or PPC, etc. This will result in both the final version of most bdist_* installations having the architecture in the filename. It also has the nice side effect of having the temp directories used by these commands include the architecture in their names, meaning its possible to build multiple Windows architectures from the same build tree, although that is not the primary motivation. Also note that bdist_msi has 'win32' hard-coded in one place, where a call to get_platform() would be more appropriate, but I'm assuming that is a bug (ie, bdist_msi should use get_platform() regardless of the outcome of this discussion about what get_platform() should return) Note that this issue is quite different than, but ultimately impacted by, the cross-compiling issue. Its quite different as even when building x64 natively on x64, 'win32' is used in the generated filename and this patch fixes that. It is impacted by cross-compiling, as it assumes the host environment is the target environment - but so does the rest of that function on all platforms. Any objections? Cheers, Mark From martin.field at ibs.fr Wed Jul 18 14:12:49 2007 From: martin.field at ibs.fr (Martin Field) Date: Wed, 18 Jul 2007 14:12:49 +0200 Subject: [Distutils] Question concerning dist_utils build_clib command Message-ID: <469E03C1.10800@ibs.fr> Hello, I'm writing with what is probably a very simple question about the build_clib command in distutils. My problem is that I have part of a library that I am building which should be built with no optimization (O0) instead of the default O2 (with gcc). After various attempts, I ended up subclassing the "build_clib" class so that I could pass the "extra_preargs" argument in the call to "self.compiler.compile" in the class's "build_libraries" method. Unfortunately, both O2 (the default) and O0 now appear in the compilation options and the first (O2!) takes precedence it seems. I wonder if you have come across this problem and whether you have a solution? I would really appreciate any help that you can give me. Many thanks. Best regards, Martin Field. From pje at telecommunity.com Wed Jul 18 19:31:21 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 18 Jul 2007 13:31:21 -0400 Subject: [Distutils] distutils.util.get_platform() for Windows In-Reply-To: <0a4001c7c900$f1ba2640$090a0a0a@enfoldsystems.local> References: <0a4001c7c900$f1ba2640$090a0a0a@enfoldsystems.local> Message-ID: <20070718172910.C1AC23A40A7@sparrow.telecommunity.com> At 04:00 PM 7/18/2007 +1000, Mark Hammond wrote: >This will result in both the final version of most bdist_* installations >having the architecture in the filename. It also has the nice side effect >of having the temp directories used by these commands include the >architecture in their names, meaning its possible to build multiple Windows >architectures from the same build tree, although that is not the primary >motivation. I presume the intention of this is to have it end up as either 'win32' or 'win64', yes? >Also note that bdist_msi has 'win32' hard-coded in one place, >where a call to get_platform() would be more appropriate, but I'm assuming >that is a bug (ie, bdist_msi should use get_platform() regardless of the >outcome of this discussion about what get_platform() should return) Well, if it becomes possible to build msi's on other platforms, they're still going to target Windows. Currently you can build a bdist_wininst on Linux, for example, especially if it's only pure Python contents. From dpeterson at enthought.com Wed Jul 18 22:44:01 2007 From: dpeterson at enthought.com (Dave Peterson) Date: Wed, 18 Jul 2007 15:44:01 -0500 Subject: [Distutils] Is there an official stable release of setuptools? Message-ID: <469E7B91.90900@enthought.com> Hello, Is there a blessed-as-stable, official, release of setuptools? Perhaps it's just me but a version number of the form '0.6c6' or '0.7a1' just doesn't seem like the developers think it is stable yet, even if it is being widely used. :-) The reason I'm asking is that there is some debate going on at Enthought about whether we should make available an egg of setuptools in our 'stable' egg repository. My personal opinion is that we shouldn't be publishing binaries of sources and calling them 'stable' unless the developers of that source have said 'this version is stable' in some form. Said with a bit more detail, I think for us to call something stable, we need to start with stable source and then do testing to verify that our builds are being done correctly. So, I'd want someone to say publicly that 0.6c6 (or whatever version) is a stable release of the source before we say our build of that should go in our stable repository. However, there are others who think 'stable' just means that we've found, in our own testing, that things generally work as advertised and that really, our build process is building correctly. (I may be paraphrasing incorrectly, but since other Enthought-ers read this list, I'll trust them to correct me!) This would mean that we could put a binary of setuptools 0.7a1 up in our stable repo. Any thoughts? -- Dave From gentoodev at gmail.com Wed Jul 18 23:04:37 2007 From: gentoodev at gmail.com (Rob Cakebread) Date: Wed, 18 Jul 2007 14:04:37 -0700 Subject: [Distutils] Is there an official stable release of setuptools? Message-ID: <9b06ffb10707181404j62a63507hecb8262091c454e6@mail.gmail.com> Dave Peterson wrote: > However, there are others who think 'stable' just means that we've > found, in our own testing, that things generally work as advertised and > that really, our build process is building correctly. > I think you'll find that this is how most software packagers (check any Linux distribution) do it, after the package has been in the testing or unstable branch for a reasonable amount of time. Also, you may find you have to mark a pre-alpha package stable the day it's released if the previous stable version has a serious security issue. From pje at telecommunity.com Wed Jul 18 23:48:52 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 18 Jul 2007 17:48:52 -0400 Subject: [Distutils] Is there an official stable release of setuptools? In-Reply-To: <469E7B91.90900@enthought.com> References: <469E7B91.90900@enthought.com> Message-ID: <20070718214637.3A8BB3A403A@sparrow.telecommunity.com> At 03:44 PM 7/18/2007 -0500, Dave Peterson wrote: >Hello, > >Is there a blessed-as-stable, official, release of setuptools? >Perhaps it's just me but a version number of the form '0.6c6' or '0.7a1' >just doesn't seem like the developers think it is stable yet, even if it >is being widely used. :-) It depends on your definition of "stable", of course. :) There will be an 0.6c7, fixing a couple of outstanding issues. I will then at some point simply bless an 0.6 final, regardless of whether there are any more bugs found, because having an 0.6c8 is just ridiculous. ;-) 0.6c6 is nonetheless an official *release* of setuptools. We do not have a "stable/unstable" distinction, only a "release/development" distinction. 0.7a1 has not been released, but 0.6c6 has. Releases are uploaded to the CheeseShop, and do not have 'dev-r####' tags in their version. >However, there are others who think 'stable' just means that we've >found, in our own testing, that things generally work as advertised and >that really, our build process is building correctly. (I may be >paraphrasing incorrectly, but since other Enthought-ers read this list, >I'll trust them to correct me!) This would mean that we could put a >binary of setuptools 0.7a1 up in our stable repo. Since 0.7a1 does not exist yet (only unreleased 0.7a1dev-r#### versions), you won't be able to do that just yet. :) From dpeterson at enthought.com Thu Jul 19 00:18:05 2007 From: dpeterson at enthought.com (Dave Peterson) Date: Wed, 18 Jul 2007 17:18:05 -0500 Subject: [Distutils] Is there an official stable release of setuptools? In-Reply-To: <20070718214637.3A8BB3A403A@sparrow.telecommunity.com> References: <469E7B91.90900@enthought.com> <20070718214637.3A8BB3A403A@sparrow.telecommunity.com> Message-ID: <469E919D.8090506@enthought.com> Phillip J. Eby wrote: > At 03:44 PM 7/18/2007 -0500, Dave Peterson wrote: >> Hello, >> >> Is there a blessed-as-stable, official, release of setuptools? >> Perhaps it's just me but a version number of the form '0.6c6' or '0.7a1' >> just doesn't seem like the developers think it is stable yet, even if it >> is being widely used. :-) > > It depends on your definition of "stable", of course. :) There will > be an 0.6c7, fixing a couple of outstanding issues. I will then at > some point simply bless an 0.6 final, regardless of whether there are > any more bugs found, because having an 0.6c8 is just ridiculous. ;-) We're in the same boat with some of our components. ;-) > 0.6c6 is nonetheless an official *release* of setuptools. We do not > have a "stable/unstable" distinction, only a "release/development" > distinction. Okay. That's good enough for me. Sounds like we can effectively think of any non-dev release of setuptools as what I was terming a 'stable' release. Perhaps in general we should stop using the term 'stable' and change it to 'released'? I'll use that in the rest of this e-mail. > 0.7a1 has not been released, but 0.6c6 has. Releases are uploaded to > the CheeseShop, and do not have 'dev-r####' tags in their version. > > >> However, there are others who think 'stable' just means that we've >> found, in our own testing, that things generally work as advertised and >> that really, our build process is building correctly. (I may be >> paraphrasing incorrectly, but since other Enthought-ers read this list, >> I'll trust them to correct me!) This would mean that we could put a >> binary of setuptools 0.7a1 up in our stable repo. > > Since 0.7a1 does not exist yet (only unreleased 0.7a1dev-r#### > versions), you won't be able to do that just yet. :) Yeah, here is where it gets tricky. Apparently some of us got overzealous (my term, not theirs) and modified code in *our* component source (that we now want to 'release') such that it depends on the API in 0.7a1dev-r####. We are currently building dev builds of your dev source such that they have a version specified like '0.7a1dev-r####.devYYYYMMDDHHMM' which, beyond being quite a mouthful, distinguishes any of our numerous attempts at building your dev source from one another. They each get tagged with the date of the build since they may be built against the same svn revision. (Not such a big deal with setuptools, but the principal is meant for other components where getting the build right may take several attempts over days or weeks.) Anyway, when we want to 'release' our component, we want a corresponding release of the dependency -- but you're not there yet -- so we're talking about putting out a version called '0.7a1dev-r####' in our 'releases' dir. This seems to be the only solution given where we are (besides backing out the implementation that relies on the 0.7a1 features.) Since people can already get an install of 0.7a1dev-r#### this probably isn't the end of the world, but we are now calling it a release when you're not -- which bothers me. :-) I've never had to think like a 'packager' before trying to set this up. (Thanks for that term by the way Rob!) Perhaps its time for more googling of how they do it. Thanks for your responses! -- Dave From mhammond at skippinet.com.au Thu Jul 19 01:13:56 2007 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 19 Jul 2007 09:13:56 +1000 Subject: [Distutils] distutils.util.get_platform() for Windows In-Reply-To: <20070718172910.C1AC23A40A7@sparrow.telecommunity.com> Message-ID: <0aca01c7c991$516bf920$090a0a0a@enfoldsystems.local> > At 04:00 PM 7/18/2007 +1000, Mark Hammond wrote: > >This will result in both the final version of most bdist_* > installations > >having the architecture in the filename. It also has the > nice side effect > >of having the temp directories used by these commands include the > >architecture in their names, meaning its possible to build > multiple Windows > >architectures from the same build tree, although that is not > the primary > >motivation. > > I presume the intention of this is to have it end up as either > 'win32' or 'win64', yes? Probably 'win32', 'amd64' or 'itanium' - I'm not worried about the specific strings, but there would need to be different ones for each of the 64bit architectures. > >Also note that bdist_msi has 'win32' hard-coded in one place, > >where a call to get_platform() would be more appropriate, > but I'm assuming > >that is a bug (ie, bdist_msi should use get_platform() > regardless of the > >outcome of this discussion about what get_platform() should return) > > Well, if it becomes possible to build msi's on other platforms, > they're still going to target Windows. Currently you can build a > bdist_wininst on Linux, for example, especially if it's only pure > Python contents. If the feature you refer to was capable of making packages with extension modules (ie, a kind of cross-compile environment), it seems to me that the same problem would exist - regardless of the architecture, the generated filename would always be identical. There would be no way to identify the architecture from just the name of the file. On the other hand, if this ability to create an MSI on Linux will forever be limited to *only* pure-python packages, I'd think the needs of people who need to package multiple architecures on Windows trumps this feature. Its not clear from your reply, but do you believe that all architectures having identical filenames is a problem? If so, how do you think we should approach it? Cheers, Mark From pje at telecommunity.com Thu Jul 19 01:29:23 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 18 Jul 2007 19:29:23 -0400 Subject: [Distutils] distutils.util.get_platform() for Windows In-Reply-To: <0aca01c7c991$516bf920$090a0a0a@enfoldsystems.local> References: <20070718172910.C1AC23A40A7@sparrow.telecommunity.com> <0aca01c7c991$516bf920$090a0a0a@enfoldsystems.local> Message-ID: <20070718233733.CFD3B3A403A@sparrow.telecommunity.com> At 09:13 AM 7/19/2007 +1000, Mark Hammond wrote: >Its not clear from your reply, but do you believe that all architectures >having identical filenames is a problem? If so, how do you think we should >approach it? Mostly, I'm just interested in understanding how to update setuptools' platform API functions: http://peak.telecommunity.com/DevCenter/PkgResources#platform-utilities In particular, the difference here between the supported platform vs. build platform, and how to tell whether two platform strings are compatible. Setuptools uses these functions to know what eggs will work on the current system, as well as what filenames to build eggs with. From tseaver at palladion.com Thu Jul 19 02:03:32 2007 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 18 Jul 2007 20:03:32 -0400 Subject: [Distutils] distutils.util.get_platform() for Windows In-Reply-To: <0aca01c7c991$516bf920$090a0a0a@enfoldsystems.local> References: <20070718172910.C1AC23A40A7@sparrow.telecommunity.com> <0aca01c7c991$516bf920$090a0a0a@enfoldsystems.local> Message-ID: <469EAA54.8060307@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Mark Hammond wrote: >> At 04:00 PM 7/18/2007 +1000, Mark Hammond wrote: >>> This will result in both the final version of most bdist_* >> installations >>> having the architecture in the filename. It also has the >> nice side effect >>> of having the temp directories used by these commands include the >>> architecture in their names, meaning its possible to build >> multiple Windows >>> architectures from the same build tree, although that is not >> the primary >>> motivation. >> I presume the intention of this is to have it end up as either >> 'win32' or 'win64', yes? > > Probably 'win32', 'amd64' or 'itanium' - I'm not worried about the specific > strings, but there would need to be different ones for each of the 64bit > architectures. Why would you use processor type IDs to indicate Windows-specifc platforms? Lots of systems running on AMD64 boxen don't run windows (can't say "lots" and "Itanium" in the same sentence, I guess, but I know for a fact that OpenVMS is running on Itanium, at least). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD4DBQFGnqpU+gerLs4ltQ4RAj5PAJ4jKxm7Lqxx9hwXRaRZl69CLysQ6ACVEKAb ql/iXfN18/e9wmdYWKxnAQ== =8r02 -----END PGP SIGNATURE----- From mhammond at skippinet.com.au Thu Jul 19 02:06:49 2007 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 19 Jul 2007 10:06:49 +1000 Subject: [Distutils] distutils.util.get_platform() for Windows In-Reply-To: <20070718233733.CFD3B3A403A@sparrow.telecommunity.com> Message-ID: <0ad301c7c998$b604ac40$090a0a0a@enfoldsystems.local> > At 09:13 AM 7/19/2007 +1000, Mark Hammond wrote: > >Its not clear from your reply, but do you believe that all > architectures > >having identical filenames is a problem? If so, how do you > think we should > >approach it? I'm still not sure what the answer to the question is though :) > > Mostly, I'm just interested in understanding how to update > setuptools' platform API functions: > > http://peak.telecommunity.com/DevCenter/PkgResources#platform-utilities That sounds worthwhile - but I'm unsure why we wouldn't simply update distutils in similar ways, and then have setuptools borrow that implementation, especially if the requirements are similar - which they seem to be. Does setuptools have unique requirements in this regard, or is there some other reason I'm missing why we can't kill multiple birds with a single stone? I assume that 'distutils' is still the 'officially preferred' way of building extensions? Or maybe distutils used directly really is considered dead, so I'm wasting my time even discussing changes to distutils itself? I'm really just trying to make it simple for the next person trying to build for 64bit Windows platforms (I've got a build - it just uses lots of hacks that may not be obvious to others), but I'm no longer sure what we expect this next person to be using when they build their extensions... Mark From tseaver at palladion.com Thu Jul 19 02:03:32 2007 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 18 Jul 2007 20:03:32 -0400 Subject: [Distutils] distutils.util.get_platform() for Windows In-Reply-To: <0aca01c7c991$516bf920$090a0a0a@enfoldsystems.local> References: <20070718172910.C1AC23A40A7@sparrow.telecommunity.com> <0aca01c7c991$516bf920$090a0a0a@enfoldsystems.local> Message-ID: <469EAA54.8060307@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Mark Hammond wrote: >> At 04:00 PM 7/18/2007 +1000, Mark Hammond wrote: >>> This will result in both the final version of most bdist_* >> installations >>> having the architecture in the filename. It also has the >> nice side effect >>> of having the temp directories used by these commands include the >>> architecture in their names, meaning its possible to build >> multiple Windows >>> architectures from the same build tree, although that is not >> the primary >>> motivation. >> I presume the intention of this is to have it end up as either >> 'win32' or 'win64', yes? > > Probably 'win32', 'amd64' or 'itanium' - I'm not worried about the specific > strings, but there would need to be different ones for each of the 64bit > architectures. Why would you use processor type IDs to indicate Windows-specifc platforms? Lots of systems running on AMD64 boxen don't run windows (can't say "lots" and "Itanium" in the same sentence, I guess, but I know for a fact that OpenVMS is running on Itanium, at least). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD4DBQFGnqpU+gerLs4ltQ4RAj5PAJ4jKxm7Lqxx9hwXRaRZl69CLysQ6ACVEKAb ql/iXfN18/e9wmdYWKxnAQ== =8r02 -----END PGP SIGNATURE----- From mhammond at skippinet.com.au Thu Jul 19 02:33:56 2007 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 19 Jul 2007 10:33:56 +1000 Subject: [Distutils] distutils.util.get_platform() for Windows In-Reply-To: <469EAA54.8060307@palladion.com> Message-ID: <0ade01c7c99c$7dcfccc0$090a0a0a@enfoldsystems.local> Tres writes: > Mark Hammond wrote: > >> At 04:00 PM 7/18/2007 +1000, Mark Hammond wrote: > >>> This will result in both the final version of most bdist_* > >> installations > >>> having the architecture in the filename. It also has the > >> nice side effect > >>> of having the temp directories used by these commands include the > >>> architecture in their names, meaning its possible to build > >> multiple Windows > >>> architectures from the same build tree, although that is not > >> the primary > >>> motivation. > >> I presume the intention of this is to have it end up as either > >> 'win32' or 'win64', yes? > > > > Probably 'win32', 'amd64' or 'itanium' - I'm not worried > about the specific > > strings, but there would need to be different ones for each > of the 64bit > > architectures. > > Why would you use processor type IDs to indicate Windows-specifc > platforms? Lots of systems running on AMD64 boxen don't run windows > (can't say "lots" and "Itanium" in the same sentence, I guess, but I > know for a fact that OpenVMS is running on Itanium, at least). Yes, I agree with that. I'm not too worried about what the specific strings are, and I agree they should include the OS *and* the architecture (eg, 'win32', 'win-amdx64' or 'win-itanium' might be suitable, or maybe win64-amd/win64-itanium). However, I'm just trying to take things one step at a time - if we can agree that having the same string for all architectures is bad, we can then move forward into a "bike-shed" discussion of what the new strings should be :) I'm yet to hear anyone explicitly agree with me that the current situation needs changing though. Cheers, Mark From pje at telecommunity.com Thu Jul 19 03:02:06 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 18 Jul 2007 21:02:06 -0400 Subject: [Distutils] distutils.util.get_platform() for Windows In-Reply-To: <0ad301c7c998$b604ac40$090a0a0a@enfoldsystems.local> References: <20070718233733.CFD3B3A403A@sparrow.telecommunity.com> <0ad301c7c998$b604ac40$090a0a0a@enfoldsystems.local> Message-ID: <20070719005953.73E633A403A@sparrow.telecommunity.com> At 10:06 AM 7/19/2007 +1000, Mark Hammond wrote: > > At 09:13 AM 7/19/2007 +1000, Mark Hammond wrote: > > >Its not clear from your reply, but do you believe that all > > architectures > > >having identical filenames is a problem? If so, how do you > > think we should > > >approach it? > >I'm still not sure what the answer to the question is though :) I just want to make sure that I understand your proposal well enough to ensure that 1) it won't cause trouble for eggs and 2) I can implement the necessary changes, if any, to setuptools. > > Mostly, I'm just interested in understanding how to update > > setuptools' platform API functions: > > > > http://peak.telecommunity.com/DevCenter/PkgResources#platform-utilities > >That sounds worthwhile - but I'm unsure why we wouldn't simply update >distutils in similar ways, Because distutils' get_platform() isn't really clear about the distinctions that setuptools makes explicit, about version compatibility. In a sense, distutils' get_platform assumes that no platform can run any other platform's code, even though this isn't so for at least Mac OS. For example, if a 'win32' package can be used on 'win64', setuptools needs to know this -- but distutils has no clue and doesn't care, because distutils doesn't *do* anything with platform information except generate filenames (including build directory file names). Notice that setuptools has *two* different get_*_platform APIs, because of this distinction, plus a "compatible_platforms" function; distutils doesn't have any of that. So if we move away from win32 as a platform designator, I need to understand *precisely* what values will replace it, so that compatible_platforms will be able to support them correctly. In general, platform strings are an underspecified area of the distutils, so if we change anything about them, I'd like to improve the specification. I'm not happy with adding more code whose behavior is implementation-defined, as it leaves me with no way to figure out what platforms are compatible with what. From mhammond at skippinet.com.au Thu Jul 19 03:44:37 2007 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 19 Jul 2007 11:44:37 +1000 Subject: [Distutils] distutils.util.get_platform() for Windows In-Reply-To: <20070719005953.73E633A403A@sparrow.telecommunity.com> Message-ID: <0ae501c7c9a6$5e71d120$090a0a0a@enfoldsystems.local> Phillip writes: > Because distutils' get_platform() isn't really clear about the > distinctions that setuptools makes explicit, about version > compatibility. In a sense, distutils' get_platform assumes that no > platform can run any other platform's code, even though this isn't so > for at least Mac OS. > > For example, if a 'win32' package can be used on 'win64', setuptools > needs to know this -- but distutils has no clue and doesn't care, > because distutils doesn't *do* anything with platform information > except generate filenames (including build directory file names). IIUC, this is a little tricky - in this specific example, it depends on what Python architecture is installed. On an x64 system, a 32bit Python and 32bit extensions can be installed - but an x64 extension can not. However, if an x64 version of Python itself is installed, 32bit extensions can not be. Note that both x64 and x32 builds of Python on Windows report 'win32' for sys.platform and 'nt' for os.name. So from this perspective, it seems no compatibility can be recorded between the architectures, as they can never be mixed. > Notice that setuptools has *two* different get_*_platform APIs, > because of this distinction, plus a "compatible_platforms" function; > distutils doesn't have any of that. I understand that is how distutils works now. I'm questioning if that is the way distutils should keep working in the future. Is there some reason I'm missing why we can't *add* this functionality to distutils and have setuptools consume it? Or is the (defacto) intent that cross-compilation only be supported via setuptools? I've no objection to this being the case, but "explicit is better than implicit" > So if we move away from win32 as a platform designator, I need to > understand *precisely* what values will replace it, so that > compatible_platforms will be able to support them correctly. Yes, part of this process would be deciding the precise strings that would be returned. > In general, platform strings are an underspecified area of the > distutils, so if we change anything about them, I'd like to improve > the specification. I'm not happy with adding more code whose > behavior is implementation-defined, as it leaves me with no way to > figure out what platforms are compatible with what. I've no problem with that - but best I can tell, no compatibility could be recorded for this specific example, so this problem doesn't really depend on a better way of capturing compatibility between platforms. Cheers, Mark From jim at zope.com Thu Jul 19 13:06:34 2007 From: jim at zope.com (Jim Fulton) Date: Thu, 19 Jul 2007 07:06:34 -0400 Subject: [Distutils] Prototype setuptools-specific PyPI index. Message-ID: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> Over the past few months, we've struggled quite a bit with Python Package Index (PyPI) performance and stability. Thanks to the heroic efforts of Martin v. L?wis and others, performance and especially stability have improved quite a bit. Martin has demonstrated that, at least when running well, PyPI seems to answer most requests on the order of 7 miliseconds (around 150 requests per second) internally. That's not bad. Unfortunately for users, actual times can be quite a bit longer. For me at work, request take around 300 milliseconds. For Martin, they seem to take somewhat longer. 300 milliseconds isn't so bad for a request or two, however, easy install can easily make 10s or even hundreds of requests to satisfy a user request for a package. zc.buildout, when verifying that a large system with many tens of packages has the most up to date versions of each package can easily make thousands of requests. Why do setuptools and buildout make so many requests? If a package exposes more than one release, then setuptools checks the package's main PyPI page and the pages for each release. We need to be able to easily use older releases, so we can't hide old releases. Typical projects of ours have many old releases exposed. If setuptools was more clever in the way it searched PyPI, but it would still have to make a minimum of 2 requests per package for packages with multiple versions exposed. Another potential issue is that PyPI pages can be large. I've found it convenient to use PyPI package pages as the home page for many of my projects. I like to include package documentation in my project pages. Perhaps this is an abuse of PyPI, but it is very convenient for me and no one has complained. :) The zc.buildout pages are around 200K. That's a fair bit of data for setuptools to download and scan for download URLs. In the course of this discussion, I've realized that it doesn't make sense for setuptools to use the same interface that humans use. setuptools doesn't need to see all of the data that is useful to humans. Similarly, humans generally don't need to see all of the historical releases for a project. I suggested a simple page format designed just for setuptools. An alternative would be an xmlrpc API. I prefer pages because I think that, over time, the amount of requests from automated tools like easy_install and zc.buildout will increase substantially and ultimately, will overwhelm dynamic servers, even ones like PyPI that are reasonably fast. I also think that a simple static collection of pages will be easier to mirror and I think some number of geographic mirrors is likely to help some people. I promised to prototype the format I suggested. I've created and experimental prototype setuptools-specific package index at http://download.zope.org/ppix Going to that page gives brief instructions for using it with easy_install and zc.buildout. To see an individual package page, add the package name to the URL, as in: http://download.zope.org/ppix/setuptools/ A few things to note about this: - I don't expose a long package list at http://download.zope.org/ ppix/. The long package list would be expensive to download and supports a use case that I consider to be of negative value, which is installing packages with case-insensitive package names, I think it is important for humans to be able to search for packages using case- insensitive search terms, but I think that, after identifying a package, precise package names should be used. I think it is especially important that precise package names be used in package requirements. - There is a single page per package. This can greatly reduce the number of requests. Packages that store all of their distributions in PyPI and that don't have off-site home pages or download URLs can be scanned with a single request. Note that I excluded home page and download URLs that pointed back to the packages PyPI page, as that wouldn't provide any new information to setuptools. - Download URLs for *hidden* packages are included. Humans don't need to see old revisions, but setuptools-based tools do. If we used an index like this for setuptools, we could stop unhiding old releases when we created new releases in PyPI. This would make PyPI more useful to humans and less of a pain for developers. - Download URLs are the same as they are in PyPI. Using this new index, distributions are still downloaded from PyPI, so the index doesn't affect PyPI download statistics. To see the impact of this, it's interesting to look at installing zc.buildout using easy_install from PyPI and from the experimental index: Installing using PyPI looks like this: (env)jim at ds9:~/tmp$ time easy_install zc.buildout Searching for zc.buildout Reading http://cheeseshop.python.org/pypi/zc.buildout/ Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b19 Reading http://svn.zope.org/zc.buildout Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b22 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b23 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b20 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b21 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b26 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b27 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b24 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b25 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b28 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b17 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b16 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b18 Best match: zc.buildout 1.0.0b28 Downloading http://cheeseshop.python.org/packages/2.5/z/ zc.buildout/zc.buildout-1.0.0b28- py2.5.egg#md5=4e37e53f010ed7984555a029732f479d Processing zc.buildout-1.0.0b28-py2.5.egg creating /home/jim/tmp/env/lib/python2.5/zc.buildout-1.0.0b28- py2.5.egg Extracting zc.buildout-1.0.0b28-py2.5.egg to /home/jim/tmp/env/lib/ python2.5 Adding zc.buildout 1.0.0b28 to easy-install.pth file Installing buildout script to /home/jim/tmp/env/bin/ Installed /home/jim/tmp/env/lib/python2.5/zc.buildout-1.0.0b28- py2.5.egg Processing dependencies for zc.buildout Searching for setuptools==0.6c6 Best match: setuptools 0.6c6 Processing setuptools-0.6c6-py2.5.egg Adding setuptools 0.6c6 to easy-install.pth file Installing easy_install script to /home/jim/tmp/env/bin/ Installing easy_install-2.5 script to /home/jim/tmp/env/bin/ Installed /home/jim/tmp/env/lib/python2.5/setuptools-0.6c6-py2.5.egg Processing dependencies for setuptools==0.6c6 Finished processing dependencies for setuptools==0.6c6 Finished installing setuptools==0.6c6 Finished processing dependencies for zc.buildout Finished installing zc.buildout real 0m31.360s user 0m1.136s sys 0m0.060s Note the large number of pages read. Here I was installing a single package with one dependency, setuptools, that was already installed. Let's look at this again using the experimental index: (env)jim at ds9:~/tmp$ time easy_install -i http://download.zope.org/ ppix zc.buildout Searching for zc.buildout Reading http://download.zope.org/ppix/zc.buildout/ Best match: zc.buildout 1.0.0b28 Downloading http://cheeseshop.python.org/packages/2.5/z/ zc.buildout/zc.buildout-1.0.0b28- py2.5.egg#md5=4e37e53f010ed7984555a029732f479d Processing zc.buildout-1.0.0b28-py2.5.egg creating /home/jim/tmp/env/lib/python2.5/zc.buildout-1.0.0b28- py2.5.egg Extracting zc.buildout-1.0.0b28-py2.5.egg to /home/jim/tmp/env/lib/ python2.5 Adding zc.buildout 1.0.0b28 to easy-install.pth file Installing buildout script to /home/jim/tmp/env/bin/ Installed /home/jim/tmp/env/lib/python2.5/zc.buildout-1.0.0b28- py2.5.egg Processing dependencies for zc.buildout Searching for setuptools==0.6c6 Best match: setuptools 0.6c6 Processing setuptools-0.6c6-py2.5.egg Adding setuptools 0.6c6 to easy-install.pth file Installing easy_install script to /home/jim/tmp/env/bin/ Installing easy_install-2.5 script to /home/jim/tmp/env/bin/ Installed /home/jim/tmp/env/lib/python2.5/setuptools-0.6c6-py2.5.egg Processing dependencies for setuptools==0.6c6 Finished processing dependencies for setuptools==0.6c6 Finished installing setuptools==0.6c6 Finished processing dependencies for zc.buildout Finished installing zc.buildout real 0m7.006s user 0m0.244s sys 0m0.040s Note: - We made far fewer requests with the new index - Most of the time in the second example was spent actually downloading the buildout distribution. Most of the time in the first example was spent reading the index. - I used workingenv to create clean environments for each of the examples above. WRT zc.buildout, refreshing a buildout with just ZODB installed in it takes about 45 seconds for me using PyPI and about 5 seconds using the experimental index. Some of the speed improvements is due to the fact that the experimental index is much closer to me (on the net) than PyPI. ATM, requests to PyPI take *me* around 500 milliseconds, while requests to the experimental index are taking between 100 and 300 milliseconds. (I'm at home and this seems to be somewhat variable.) Most of the speed improvements are from reducing the number of requests. I'm polling PyPI once a minute to get and apply updates. Thanks to the new XML-RPC method that Martin added, this is very efficient to do. I encourage people to check this out and even try using it with easy_install and especially buildout. AFAIK, aside from being much faster and showing download files for hidden releases it is completely equivalent to PyPI for setuptools use. My intension is to keep this experimental index going and up to date for the foreseeable future and plan to use it for all my work. My primary goal is to prototype the new index format. If this seems useful, then I think that www.python.org should expose an index in this format to setuptools, either at a different URL or by satisfying setuptools requests from the index based on client information. I'd love to see this index populated via a baking mechanism that updates package pages when they change, rather than through polling as I'm doing. There would be some benefit to having geographic mirrors. I suspect that having such mirrors available would improve performance further, at least for some folks. It might also be useful to have some mirrors for redundancy purposes. Note though that what I'm doing is mirroring the only index data. I'm not mirroring distributions. Of course, I'd be happy to make my software available. (It already is via our subversion repository.) I hope this effort spurs useful discussion and progress. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From fdrake at acm.org Thu Jul 19 16:01:27 2007 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu, 19 Jul 2007 10:01:27 -0400 Subject: [Distutils] zc.buildout and zc.recipe.egg In-Reply-To: References: Message-ID: <200707191001.27881.fdrake@acm.org> On Monday 16 July 2007, Miles wrote: > - the standard python interpreter seems to add the directory of the > file you are running to the path > - the generated one does not add the directory to the path That's correct. This is a limitation in zc.recipe.egg; we're open to patches that improve the generated interpreter script. -Fred -- Fred L. Drake, Jr. From miles at jamkit.com Thu Jul 19 23:21:59 2007 From: miles at jamkit.com (Miles) Date: Thu, 19 Jul 2007 22:21:59 +0100 Subject: [Distutils] zc.buildout and zc.recipe.egg In-Reply-To: <200707191001.27881.fdrake@acm.org> References: <200707191001.27881.fdrake@acm.org> Message-ID: <469FD5F7.5070607@jamkit.com> Hi, > That's correct. This is a limitation in zc.recipe.egg; we're open to patches > that improve the generated interpreter script. Thanks for confirming, I wasn't sure if it was like that for a reason. I attach a patch that fixes up some other parameters so the generated interpreter script's behaviour is less distinguis matches what you get when you start the python interpreter more exactly. Works for me, but I'd appreciate some more eyes... Miles -------------- next part -------------- A non-text attachment was scrubbed... Name: interpreter.patch Type: text/x-patch Size: 879 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070719/c697d630/attachment.bin From jim at zope.com Thu Jul 19 23:55:51 2007 From: jim at zope.com (Jim Fulton) Date: Thu, 19 Jul 2007 17:55:51 -0400 Subject: [Distutils] zc.buildout and zc.recipe.egg In-Reply-To: <469FD5F7.5070607@jamkit.com> References: <200707191001.27881.fdrake@acm.org> <469FD5F7.5070607@jamkit.com> Message-ID: <65F87547-59A1-42C2-8B48-151B8FBDAD9E@zope.com> Much thanks. Can you also supply a test? Better yet, since you are a zope contributor, you could just check in the change, including the test. Jim On Jul 19, 2007, at 5:21 PM, Miles wrote: > Hi, > >> That's correct. This is a limitation in zc.recipe.egg; we're open >> to patches that improve the generated interpreter script. > > Thanks for confirming, I wasn't sure if it was like that for a reason. > > I attach a patch that fixes up some other parameters so the > generated interpreter script's behaviour is less distinguis matches > what you get when you start the python interpreter more exactly. > Works for me, but I'd appreciate some more eyes... > > Miles > > Index: zc.buildout/src/zc/buildout/easy_install.py > =================================================================== > --- zc.buildout/src/zc/buildout/easy_install.py (revision 78190) > +++ zc.buildout/src/zc/buildout/easy_install.py (working copy) > @@ -896,11 +896,17 @@ > py_script_template = '''\ > #!%(python)s > import sys > - > +import os > + > sys.path[0:0] = [ > + '', > %(path)s, > ] > > +abs__file__ = os.path.abspath( __file__ ) > +sys.executable = abs__file__ > +del ( __file__ ) > + > _interactive = True > if len(sys.argv) > 1: > import getopt > @@ -914,7 +920,12 @@ > > if _args: > sys.argv[:] = _args > - execfile(sys.argv[0]) > + f = sys.argv[0] > + absf = os.path.abspath(f) > + p = os.path.split( absf )[0] > + sys.path[0] = os.path.join( p ) > + __file__ = absf > + execfile( f ) > > if _interactive: > import code > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From miles at jamkit.com Fri Jul 20 00:04:39 2007 From: miles at jamkit.com (Miles Waller) Date: Thu, 19 Jul 2007 23:04:39 +0100 Subject: [Distutils] zc.buildout and zc.recipe.egg In-Reply-To: <65F87547-59A1-42C2-8B48-151B8FBDAD9E@zope.com> References: <200707191001.27881.fdrake@acm.org> <469FD5F7.5070607@jamkit.com> <65F87547-59A1-42C2-8B48-151B8FBDAD9E@zope.com> Message-ID: <469FDFF7.20200@jamkit.com> Actually, I'm a bit unclear as to how to test this in a useful way: do you think it is sufficient just to test that the content of the script generated contains specific text (that doesn't guarantee the script that is generated actually works!)? Thanks Miles Jim Fulton wrote: > Much thanks. Can you also supply a test? Better yet, since you are a > zope contributor, you could just check in the change, including the test. > > Jim > > On Jul 19, 2007, at 5:21 PM, Miles wrote: > >> Hi, >> >>> That's correct. This is a limitation in zc.recipe.egg; we're open >>> to patches that improve the generated interpreter script. >> >> Thanks for confirming, I wasn't sure if it was like that for a reason. >> >> I attach a patch that fixes up some other parameters so the generated >> interpreter script's behaviour is less distinguis matches what you >> get when you start the python interpreter more exactly. Works for >> me, but I'd appreciate some more eyes... >> >> Miles >> >> Index: zc.buildout/src/zc/buildout/easy_install.py >> =================================================================== >> --- zc.buildout/src/zc/buildout/easy_install.py (revision 78190) >> +++ zc.buildout/src/zc/buildout/easy_install.py (working copy) >> @@ -896,11 +896,17 @@ >> py_script_template = '''\ >> #!%(python)s >> import sys >> - >> +import os >> + >> sys.path[0:0] = [ >> + '', >> %(path)s, >> ] >> >> +abs__file__ = os.path.abspath( __file__ ) >> +sys.executable = abs__file__ >> +del ( __file__ ) >> + >> _interactive = True >> if len(sys.argv) > 1: >> import getopt >> @@ -914,7 +920,12 @@ >> >> if _args: >> sys.argv[:] = _args >> - execfile(sys.argv[0]) >> + f = sys.argv[0] >> + absf = os.path.abspath(f) >> + p = os.path.split( absf )[0] >> + sys.path[0] = os.path.join( p ) >> + __file__ = absf >> + execfile( f ) >> >> if _interactive: >> import code >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > -- > Jim Fulton mailto:jim at zope.com Python Powered! > CTO (540) 361-1714 http://www.python.org > Zope Corporation http://www.zope.com http://www.zope.org > > > > From miles at jamkit.com Thu Jul 19 23:21:59 2007 From: miles at jamkit.com (Miles) Date: Thu, 19 Jul 2007 22:21:59 +0100 Subject: [Distutils] zc.buildout and zc.recipe.egg In-Reply-To: <200707191001.27881.fdrake@acm.org> References: <200707191001.27881.fdrake@acm.org> Message-ID: <469FD5F7.5070607@jamkit.com> Hi, > That's correct. This is a limitation in zc.recipe.egg; we're open to patches > that improve the generated interpreter script. Thanks for confirming, I wasn't sure if it was like that for a reason. I attach a patch that fixes up some other parameters so the generated interpreter script's behaviour is less distinguis matches what you get when you start the python interpreter more exactly. Works for me, but I'd appreciate some more eyes... Miles -------------- next part -------------- A non-text attachment was scrubbed... Name: interpreter.patch Type: text/x-patch Size: 879 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070719/c697d630/attachment-0003.bin From jim at zope.com Fri Jul 20 01:22:57 2007 From: jim at zope.com (Jim Fulton) Date: Thu, 19 Jul 2007 19:22:57 -0400 Subject: [Distutils] zc.buildout and zc.recipe.egg In-Reply-To: <469FDFF7.20200@jamkit.com> References: <200707191001.27881.fdrake@acm.org> <469FD5F7.5070607@jamkit.com> <65F87547-59A1-42C2-8B48-151B8FBDAD9E@zope.com> <469FDFF7.20200@jamkit.com> Message-ID: <4AE62DDC-4A60-40F7-ABFE-AE1FF994F4DB@zope.com> On Jul 19, 2007, at 6:04 PM, Miles Waller wrote: > Actually, I'm a bit unclear as to how to test this in a useful > way: do you think it is sufficient just to test that the content > of the script generated contains specific text (that doesn't > guarantee the script that is generated actually works!)? You could use the generated script to run another script imports and uses a module that is in the current working directory. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From martin at v.loewis.de Fri Jul 20 10:21:18 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 20 Jul 2007 10:21:18 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> Message-ID: <46A0707E.6000103@v.loewis.de> > I've created and experimental prototype setuptools-specific package > index at > > http://download.zope.org/ppix Cool! If this proves useful, people are encouraged to contribute the proper patches to PyPI to regenerate the page directly on each log change. There is a slight transactional trickiness to doing so: If you regenerate before the commit, it might be that the commit fails; then you would have to rollback the page update, too. If you regenerate after commit, it might be that you run into race conditions if the same package sees two updates in two transactions very quickly, and the second regeneration completes before the first one. If people would find it easier to make these pages dynamic, such patches would also be kindly accepted. Generating the pages on access should be fairly cheap; the SQL is select filename,md5_digest from release_files where name='setuptools'; and putting the result of that into an ppix-like HTML page should be much faster than invoking ZPT. Regards, Martin From jodok at lovelysystems.com Fri Jul 20 10:50:40 2007 From: jodok at lovelysystems.com (Jodok Batlogg) Date: Fri, 20 Jul 2007 10:50:40 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> Message-ID: <6000B516-6593-4A98-AA08-B6C7B329BC62@lovelysystems.com> thanks jim. you save our day. we'll send some austrian cheese over :) jodok On 19.07.2007, at 13:06, Jim Fulton wrote: > Over the past few months, we've struggled quite a bit with Python > Package Index (PyPI) performance and stability. Thanks to the heroic > efforts of Martin v. L?wis and others, performance and especially > stability have improved quite a bit. Martin has demonstrated that, at > least when running well, PyPI seems to answer most requests on the > order of 7 miliseconds (around 150 requests per second) internally. > That's not bad. Unfortunately for users, actual times can be quite a > bit longer. For me at work, request take around 300 milliseconds. > For Martin, they seem to take somewhat longer. 300 milliseconds > isn't so bad for a request or two, however, easy install can easily > make 10s or even hundreds of requests to satisfy a user request for a > package. zc.buildout, when verifying that a large system with many > tens of packages has the most up to date versions of each package can > easily make thousands of requests. > > Why do setuptools and buildout make so many requests? If a package > exposes more than one release, then setuptools checks the package's > main PyPI page and the pages for each release. We need to be able to > easily use older releases, so we can't hide old releases. Typical > projects of ours have many old releases exposed. If setuptools was > more clever in the way it searched PyPI, but it would still have to > make a minimum of 2 requests per package for packages with multiple > versions exposed. > > Another potential issue is that PyPI pages can be large. I've found > it convenient to use PyPI package pages as the home page for many of > my projects. I like to include package documentation in my project > pages. Perhaps this is an abuse of PyPI, but it is very convenient > for me and no one has complained. :) The zc.buildout pages are > around 200K. That's a fair bit of data for setuptools to download > and scan for download URLs. > > In the course of this discussion, I've realized that it doesn't make > sense for setuptools to use the same interface that humans use. > setuptools doesn't need to see all of the data that is useful to > humans. Similarly, humans generally don't need to see all of the > historical releases for a project. I suggested a simple page format > designed just for setuptools. An alternative would be an xmlrpc > API. I prefer pages because I think that, over time, the amount of > requests from automated tools like easy_install and zc.buildout will > increase substantially and ultimately, will overwhelm dynamic > servers, even ones like PyPI that are reasonably fast. I also think > that a simple static collection of pages will be easier to mirror and > I think some number of geographic mirrors is likely to help some > people. I promised to prototype the format I suggested. > > I've created and experimental prototype setuptools-specific package > index at > > http://download.zope.org/ppix > > Going to that page gives brief instructions for using it with > easy_install and zc.buildout. To see an individual package page, add > the package name to the URL, as in: > > http://download.zope.org/ppix/setuptools/ > > A few things to note about this: > > - I don't expose a long package list at http://download.zope.org/ > ppix/. The long package list would be expensive to download and > supports a use case that I consider to be of negative value, which is > installing packages with case-insensitive package names, I think it > is important for humans to be able to search for packages using case- > insensitive search terms, but I think that, after identifying a > package, precise package names should be used. I think it is > especially important that precise package names be used in package > requirements. > > - There is a single page per package. This can greatly reduce the > number of requests. Packages that store all of their distributions > in PyPI and that don't have off-site home pages or download URLs can > be scanned with a single request. Note that I excluded home page and > download URLs that pointed back to the packages PyPI page, as that > wouldn't provide any new information to setuptools. > > - Download URLs for *hidden* packages are included. Humans don't > need to see old revisions, but setuptools-based tools do. If we used > an index like this for setuptools, we could stop unhiding old > releases when we created new releases in PyPI. This would make PyPI > more useful to humans and less of a pain for developers. > > - Download URLs are the same as they are in PyPI. Using this new > index, distributions are still downloaded from PyPI, so the index > doesn't affect PyPI download statistics. > > To see the impact of this, it's interesting to look at installing > zc.buildout using easy_install from PyPI and from the experimental > index: > Installing using PyPI looks like this: > > (env)jim at ds9:~/tmp$ time easy_install zc.buildout > Searching for zc.buildout > Reading http://cheeseshop.python.org/pypi/zc.buildout/ > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b19 > Reading http://svn.zope.org/zc.buildout > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b22 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b23 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b20 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b21 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b26 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b27 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b24 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b25 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b28 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b17 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b16 > Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b18 > Best match: zc.buildout 1.0.0b28 > Downloading http://cheeseshop.python.org/packages/2.5/z/ > zc.buildout/zc.buildout-1.0.0b28- > py2.5.egg#md5=4e37e53f010ed7984555a029732f479d > Processing zc.buildout-1.0.0b28-py2.5.egg > creating /home/jim/tmp/env/lib/python2.5/zc.buildout-1.0.0b28- > py2.5.egg > Extracting zc.buildout-1.0.0b28-py2.5.egg to /home/jim/tmp/env/lib/ > python2.5 > Adding zc.buildout 1.0.0b28 to easy-install.pth file > Installing buildout script to /home/jim/tmp/env/bin/ > > Installed /home/jim/tmp/env/lib/python2.5/zc.buildout-1.0.0b28- > py2.5.egg > Processing dependencies for zc.buildout > Searching for setuptools==0.6c6 > Best match: setuptools 0.6c6 > Processing setuptools-0.6c6-py2.5.egg > Adding setuptools 0.6c6 to easy-install.pth file > Installing easy_install script to /home/jim/tmp/env/bin/ > Installing easy_install-2.5 script to /home/jim/tmp/env/bin/ > > Installed /home/jim/tmp/env/lib/python2.5/setuptools-0.6c6- > py2.5.egg > Processing dependencies for setuptools==0.6c6 > Finished processing dependencies for setuptools==0.6c6 > Finished installing setuptools==0.6c6 > Finished processing dependencies for zc.buildout > Finished installing zc.buildout > > real 0m31.360s > user 0m1.136s > sys 0m0.060s > > Note the large number of pages read. Here I was installing a single > package with one dependency, setuptools, that was already installed. > Let's look at this again using the experimental index: > > (env)jim at ds9:~/tmp$ time easy_install -i http://download.zope.org/ > ppix zc.buildout > Searching for zc.buildout > Reading http://download.zope.org/ppix/zc.buildout/ > Best match: zc.buildout 1.0.0b28 > Downloading http://cheeseshop.python.org/packages/2.5/z/ > zc.buildout/zc.buildout-1.0.0b28- > py2.5.egg#md5=4e37e53f010ed7984555a029732f479d > Processing zc.buildout-1.0.0b28-py2.5.egg > creating /home/jim/tmp/env/lib/python2.5/zc.buildout-1.0.0b28- > py2.5.egg > Extracting zc.buildout-1.0.0b28-py2.5.egg to /home/jim/tmp/env/lib/ > python2.5 > Adding zc.buildout 1.0.0b28 to easy-install.pth file > Installing buildout script to /home/jim/tmp/env/bin/ > > Installed /home/jim/tmp/env/lib/python2.5/zc.buildout-1.0.0b28- > py2.5.egg > Processing dependencies for zc.buildout > Searching for setuptools==0.6c6 > Best match: setuptools 0.6c6 > Processing setuptools-0.6c6-py2.5.egg > Adding setuptools 0.6c6 to easy-install.pth file > Installing easy_install script to /home/jim/tmp/env/bin/ > Installing easy_install-2.5 script to /home/jim/tmp/env/bin/ > > Installed /home/jim/tmp/env/lib/python2.5/setuptools-0.6c6- > py2.5.egg > Processing dependencies for setuptools==0.6c6 > Finished processing dependencies for setuptools==0.6c6 > Finished installing setuptools==0.6c6 > Finished processing dependencies for zc.buildout > Finished installing zc.buildout > > real 0m7.006s > user 0m0.244s > sys 0m0.040s > > Note: > > - We made far fewer requests with the new index > > - Most of the time in the second example was spent actually > downloading the buildout distribution. Most of the time in the first > example was spent reading the index. > > - I used workingenv to create clean environments for each of the > examples above. > > WRT zc.buildout, refreshing a buildout with just ZODB installed in it > takes about 45 seconds for me using PyPI and about 5 seconds using > the experimental index. > > Some of the speed improvements is due to the fact that the > experimental index is much closer to me (on the net) than PyPI. ATM, > requests to PyPI take *me* around 500 milliseconds, while requests to > the experimental index are taking between 100 and 300 milliseconds. > (I'm at home and this seems to be somewhat variable.) Most of the > speed improvements are from reducing the number of requests. > > I'm polling PyPI once a minute to get and apply updates. Thanks to > the new XML-RPC method that Martin added, this is very efficient to > do. > > I encourage people to check this out and even try using it with > easy_install and especially buildout. AFAIK, aside from being much > faster and showing download files for hidden releases it is > completely equivalent to PyPI for setuptools use. My intension is to > keep this experimental index going and up to date for the foreseeable > future and plan to use it for all my work. > > My primary goal is to prototype the new index format. If this seems > useful, then I think that www.python.org should expose an index in > this format to setuptools, either at a different URL or by satisfying > setuptools requests from the index based on client information. I'd > love to see this index populated via a baking mechanism that updates > package pages when they change, rather than through polling as I'm > doing. > > There would be some benefit to having geographic mirrors. I suspect > that having such mirrors available would improve performance further, > at least for some folks. It might also be useful to have some > mirrors for redundancy purposes. Note though that what I'm doing is > mirroring the only index data. I'm not mirroring distributions. Of > course, I'd be happy to make my software available. (It already is > via our subversion repository.) > > I hope this effort spurs useful discussion and progress. > > Jim > > -- > Jim Fulton mailto:jim at zope.com Python Powered! > CTO (540) 361-1714 http://www.python.org > Zope Corporation http://www.zope.com http://www.zope.org > > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -- "Although never is often better than *right* now." -- The Zen of Python, by Tim Peters Jodok Batlogg, Lovely Systems Schmelzh?tterstra?e 26a, 6850 Dornbirn, Austria phone: +43 5572 908060, fax: +43 5572 908060-77 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2454 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070720/26d28f4b/attachment.bin From martin.field at ibs.fr Fri Jul 20 11:40:15 2007 From: martin.field at ibs.fr (Martin Field) Date: Fri, 20 Jul 2007 11:40:15 +0200 Subject: [Distutils] Question concerning dist_utils build_clib command Message-ID: <46A082FF.4000402@ibs.fr> Hi again, As a follow up to my last posting, I have found a not very elegant workaround which consists in modifying the appropriate compiler flags in the compiler's compiler_so list. These ultimately come from Python's make and include files via the function "customize_compiler" in "distutils.sysconfig". Is there any convenient way of altering these flags between calls to "setup" which would avoid having to change or extend the distutils code? Any help would be appreciated. Martin. From ct at gocept.com Fri Jul 20 12:02:45 2007 From: ct at gocept.com (Christian Theune) Date: Fri, 20 Jul 2007 12:02:45 +0200 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> Message-ID: <1184925765.6519.3.camel@mindy> Am Donnerstag, den 19.07.2007, 07:06 -0400 schrieb Jim Fulton: > I promised to prototype the format I suggested. > > I've created and experimental prototype setuptools-specific package > index at > > http://download.zope.org/ppix Yay! This works like a charme! > There would be some benefit to having geographic mirrors. I suspect > that having such mirrors available would improve performance further, > at least for some folks. It might also be useful to have some > mirrors for redundancy purposes. Note though that what I'm doing is > mirroring the only index data. I'm not mirroring distributions. Of > course, I'd be happy to make my software available. (It already is > via our subversion repository.) I'd be happy to support mirroring once all this is sorted out/ I can offer a server in Germany/Europe. Christian From jim at zope.com Fri Jul 20 13:45:57 2007 From: jim at zope.com (Jim Fulton) Date: Fri, 20 Jul 2007 07:45:57 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A0707E.6000103@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A0707E.6000103@v.loewis.de> Message-ID: <5105308E-F651-438B-8C3D-F5FCAF8A8351@zope.com> On Jul 20, 2007, at 4:21 AM, Martin v. L?wis wrote: >> I've created and experimental prototype setuptools-specific package >> index at >> >> http://download.zope.org/ppix > > Cool! If this proves useful, people are encouraged to contribute the > proper patches to PyPI to regenerate the page directly on each log > change. > > There is a slight transactional trickiness to doing so: If you > regenerate before the commit, it might be that the commit fails; > then you would have to rollback the page update, too. If you > regenerate after commit, it might be that you run into race > conditions if the same package sees two updates in two > transactions very quickly, and the second regeneration completes > before the first one. > > If people would find it easier to make these pages dynamic, > such patches would also be kindly accepted. Generating the > pages on access should be fairly cheap; the SQL is > > select filename,md5_digest from release_files where name='setuptools'; > > and putting the result of that into an ppix-like HTML page > should be much faster than invoking ZPT. A few notes. It is important to show files from hidden releases as well as unhidden releases. I suspect the select statement above does that. I parse long descriptions to get #egg= links. I also give some special care to urls that point back to PyPI to avoid having setuptools go back to the human interface. It might be easiest to just trigger the existing ppix sw to poll after a change. Thanks to your xmlrpc addition, polling is quite cheap. Alternatively, we could install the existing software in a way that polls more or less continuously. This would be quite trivial. What you suggest is probably cleaner but requires some expertise with the current software. :) I'd much rather generate static files (as I'm doing now) than serve these dynamically. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Fri Jul 20 13:48:39 2007 From: jim at zope.com (Jim Fulton) Date: Fri, 20 Jul 2007 07:48:39 -0400 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: <1184925765.6519.3.camel@mindy> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <1184925765.6519.3.camel@mindy> Message-ID: <465B76C9-D7D2-420E-BBBB-E7F24F6FA710@zope.com> On Jul 20, 2007, at 6:02 AM, Christian Theune wrote: ... > I'd be happy to support mirroring once all this is sorted out/ I can > offer a server in Germany/Europe. If we decide that mirrors would be a good idea, it will be important, imo, to select mirror sites bases on their connectivity. The goal of the mirrors should be to try to give people options with short network distances. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From ct at gocept.com Fri Jul 20 13:52:12 2007 From: ct at gocept.com (Christian Theune) Date: Fri, 20 Jul 2007 13:52:12 +0200 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: <465B76C9-D7D2-420E-BBBB-E7F24F6FA710@zope.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <1184925765.6519.3.camel@mindy> <465B76C9-D7D2-420E-BBBB-E7F24F6FA710@zope.com> Message-ID: <1184932332.6519.11.camel@mindy> Am Freitag, den 20.07.2007, 07:48 -0400 schrieb Jim Fulton: > On Jul 20, 2007, at 6:02 AM, Christian Theune wrote: > ... > > I'd be happy to support mirroring once all this is sorted out/ I can > > offer a server in Germany/Europe. > > If we decide that mirrors would be a good idea, it will be important, > imo, to select mirror sites bases on their connectivity. The goal of > the mirrors should be to try to give people options with short > network distances. Right, however, do you have any specific parameters that can be measured in mind? (Our server is reasonably well connected, reachable with about 5 hops from within Germany with latency around 40ms on a DSL line. Multiple GBit lines to the hosting center.) Christian From jim at zope.com Fri Jul 20 15:42:37 2007 From: jim at zope.com (Jim Fulton) Date: Fri, 20 Jul 2007 09:42:37 -0400 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: <1184932332.6519.11.camel@mindy> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <1184925765.6519.3.camel@mindy> <465B76C9-D7D2-420E-BBBB-E7F24F6FA710@zope.com> <1184932332.6519.11.camel@mindy> Message-ID: <5686B35D-34DD-49FE-A8E7-37397A4AE808@zope.com> On Jul 20, 2007, at 7:52 AM, Christian Theune wrote: > Am Freitag, den 20.07.2007, 07:48 -0400 schrieb Jim Fulton: >> On Jul 20, 2007, at 6:02 AM, Christian Theune wrote: >> ... >>> I'd be happy to support mirroring once all this is sorted out/ I can >>> offer a server in Germany/Europe. >> >> If we decide that mirrors would be a good idea, it will be important, >> imo, to select mirror sites bases on their connectivity. The goal of >> the mirrors should be to try to give people options with short >> network distances. > > Right, however, do you have any specific parameters that can be > measured > in mind? I'm not enough of a network expert. Hopefully, someone more knowledgeable will make a suggestion. BTW, with the current PyPI performance, I'm guessing we could have 10s of mirrors poll once a minute without affecting other users. > (Our server is reasonably well connected, reachable with about 5 hops > from within Germany with latency around 40ms on a DSL line. Multiple > GBit lines to the hosting center.) I didn't mean to suggest that you weren't well connected. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From dpeterson at enthought.com Fri Jul 20 20:50:44 2007 From: dpeterson at enthought.com (Dave Peterson) Date: Fri, 20 Jul 2007 13:50:44 -0500 Subject: [Distutils] Are there best practices on creating/using egg extras? Message-ID: <46A10404.6030601@enthought.com> Hello, I'm not sure if I've painted Enthought into a corner or not, but I can't figure out if there is a way to help users of a library, delivered as an egg, to know and/or maintain dependencies on extras declared in that egg. By which I mean that if my library, called X, declares extras 1, 2, and 3, and the source for X includes all the code that implements the features in extras 1, 2, and 3, and the extras document the external dependencies component X has to get those extras to work, how does someone importing from the API in X know whether their dependency on it should be 'X', 'X[1]', X[1,2]', etc. ? After all, all the API methods are already there even if they didn't install the extras. So far the only mechanisms I can see are (a) a manual one which depends on people (everyone using X!) knowing the internals of X such that they can tell that if they import symbols a, b, or c, then that means they need extra 1, etc., and (b) trial and error iteration driven by unit / integration tests. Am I just misusing or misunderstanding extras here? BTW, what I'm trying to do is convert Enthought's old monolithic distribution of ETS into components that could be installed individually. The first step of this was to package each component as an egg; the second was to ensure that cross-component dependencies were put in place (simply declaring A requires B if code in component A imported a symbol from packages in component B); the third was to try to minimize these cross-component dependencies by minor refactoring and introduction of extras to represent non-core functionality; and the fourth is to ensure that cross-component dependencies properly include extras as needed. It's this last step that has got me thinking about this problem. I've been able to mostly automate the previous steps, or at least write tools to help me, but for this fourth step, I can't figure out how to do it consistently. It is quite possible that I've missed some best practice of when to create or use extras. Any advice would be greatly appreciated! -- Dave From pje at telecommunity.com Fri Jul 20 21:58:09 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 20 Jul 2007 15:58:09 -0400 Subject: [Distutils] Are there best practices on creating/using egg extras? In-Reply-To: <46A10404.6030601@enthought.com> References: <46A10404.6030601@enthought.com> Message-ID: <20070720200235.20A2F3A40A8@sparrow.telecommunity.com> At 01:50 PM 7/20/2007 -0500, Dave Peterson wrote: >Hello, > >I'm not sure if I've painted Enthought into a corner or not, but I can't >figure out if there is a way to help users of a library, delivered as an >egg, to know and/or maintain dependencies on extras declared in that >egg. By which I mean that if my library, called X, declares extras 1, >2, and 3, and the source for X includes all the code that implements the >features in extras 1, 2, and 3, and the extras document the external >dependencies component X has to get those extras to work, how does >someone importing from the API in X know whether their dependency on it >should be 'X', 'X[1]', X[1,2]', etc. ? After all, all the API methods >are already there even if they didn't install the extras. So far the >only mechanisms I can see are (a) a manual one which depends on people >(everyone using X!) knowing the internals of X such that they can tell >that if they import symbols a, b, or c, then that means they need extra >1, etc., and (b) trial and error iteration driven by unit / integration >tests. > >Am I just misusing or misunderstanding extras here? Note that if you have code that imports from your extras, you can always include: pkg_resources.require("X[1]") right before the imports. Then, instead of an obscure import error, users will get an obscure DistributionNotFound error that at least tells them one of the packages they'll need to install. ;-) From dpeterson at enthought.com Fri Jul 20 22:06:45 2007 From: dpeterson at enthought.com (Dave Peterson) Date: Fri, 20 Jul 2007 15:06:45 -0500 Subject: [Distutils] Are there best practices on creating/using egg extras? In-Reply-To: <20070720200235.20A2F3A40A8@sparrow.telecommunity.com> References: <46A10404.6030601@enthought.com> <20070720200235.20A2F3A40A8@sparrow.telecommunity.com> Message-ID: <46A115D5.3030603@enthought.com> Phillip J. Eby wrote: > At 01:50 PM 7/20/2007 -0500, Dave Peterson wrote: >> Hello, >> >> I'm not sure if I've painted Enthought into a corner or not, but I can't >> figure out if there is a way to help users of a library, delivered as an >> egg, to know and/or maintain dependencies on extras declared in that >> egg. By which I mean that if my library, called X, declares extras 1, >> 2, and 3, and the source for X includes all the code that implements the >> features in extras 1, 2, and 3, and the extras document the external >> dependencies component X has to get those extras to work, how does >> someone importing from the API in X know whether their dependency on it >> should be 'X', 'X[1]', X[1,2]', etc. ? After all, all the API methods >> are already there even if they didn't install the extras. So far the >> only mechanisms I can see are (a) a manual one which depends on people >> (everyone using X!) knowing the internals of X such that they can tell >> that if they import symbols a, b, or c, then that means they need extra >> 1, etc., and (b) trial and error iteration driven by unit / integration >> tests. >> >> Am I just misusing or misunderstanding extras here? > > Note that if you have code that imports from your extras, you can > always include: > > pkg_resources.require("X[1]") > > right before the imports. Then, instead of an obscure import error, > users will get an obscure DistributionNotFound error that at least > tells them one of the packages they'll need to install. ;-) Right. Always better to have an obscure error that is more informative about how to solve it. :-) But someone still needs to know when to insert pkg_resources.require("X[1]") in their code. Which means knowing which symbols in X's API need a given extra. So I'm back at my original problem, no? -- Dave From pje at telecommunity.com Fri Jul 20 22:09:39 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 20 Jul 2007 16:09:39 -0400 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> Message-ID: <20070720200721.88E1D3A403A@sparrow.telecommunity.com> At 07:06 AM 7/19/2007 -0400, Jim Fulton wrote: >I've created and experimental prototype setuptools-specific package >index at > > http://download.zope.org/ppix > >Going to that page gives brief instructions for using it with >easy_install and zc.buildout. FYI, the handling of homepage and download links is broken. You have e.g. 'meta="homepage"' instead of 'rel="homepage"', so easy_install doesn't pick these up and look for links there, meaning that ppix fails to find downloads for e.g. pywin32 which is hosted at Sourceforge. (On a perhaps not entirely unrelated note, the Cheeseshop appears to be down at the moment: """Error... There's been a problem with your request psycopg.OperationalError: no connection to the server""") By the way, I'd suggest explaining (or linking to an explanation) on the ppix main page describing how to configure easy_install such that the '-i' option isn't necessary. Perhaps we could add an example to the EasyInstall docs somewhere near: http://peak.telecommunity.com/DevCenter/EasyInstall#creating-your-own-package-index and then link to it from the ppix page. From jim at zope.com Fri Jul 20 22:18:55 2007 From: jim at zope.com (Jim Fulton) Date: Fri, 20 Jul 2007 16:18:55 -0400 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: <20070720200721.88E1D3A403A@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <20070720200721.88E1D3A403A@sparrow.telecommunity.com> Message-ID: <24B11DD1-DD79-4171-A38F-06B642EC354B@zope.com> On Jul 20, 2007, at 4:09 PM, Phillip J. Eby wrote: > At 07:06 AM 7/19/2007 -0400, Jim Fulton wrote: >> I've created and experimental prototype setuptools-specific package >> index at >> >> http://download.zope.org/ppix >> >> Going to that page gives brief instructions for using it with >> easy_install and zc.buildout. > > FYI, the handling of homepage and download links is broken. You > have e.g. 'meta="homepage"' instead of 'rel="homepage"', so > easy_install doesn't pick these up and look for links there, > meaning that ppix fails to find downloads for e.g. pywin32 which is > hosted at Sourceforge. Doh! Fixed. > (On a perhaps not entirely unrelated note, the Cheeseshop appears > to be down at the moment: > > """Error... > > There's been a problem with your request > > psycopg.OperationalError: no connection to the server""") > > > By the way, I'd suggest explaining (or linking to an explanation) > on the ppix main page describing how to configure easy_install such > that the '-i' option isn't necessary. If you send me some text, I'd be happy to add it to the ppix main page. > Perhaps we could add an example to the EasyInstall docs somewhere > near: > > http://peak.telecommunity.com/DevCenter/EasyInstall#creating-your- > own-package-index > > and then link to it from the ppix page. +1 Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From barry at python.org Fri Jul 20 22:32:19 2007 From: barry at python.org (Barry Warsaw) Date: Fri, 20 Jul 2007 16:32:19 -0400 Subject: [Distutils] Bazaar support for setuptools Message-ID: <63CA092B-CCC2-4491-BE43-87C4B4E4381B@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Well, I hacked this together pretty quickly but it seems to work for my particular use case. The Mailman source code is now kept under the Bazaar revision control system, and I'm about to merge a branch that converts the Mailman 3 branch from autoconf-based builds to setuptools. So I needed Bazaar support in order to build my sdist files and such. Writing the plugin code was pretty easy after Michael Hudson gave me the key bzrlib clues. Maybe this could be included by default in the next version of setuptools? I've added it as an attachment to SF patch #1757782 https://sourceforge.net/tracker/index.php? func=detail&aid=1757782&group_id=5470&atid=305470 Side note: I found it interesting and a bit annoying that setuptools doesn't call find_files_for_bzr() recursively. It would seem like the framework should do the recursion instead of the plugin, as I'd think it would be the most common use case. In any event, the setuptools documentation should probably be clear that it's up to the plugin to recurse into subdirectories. Cheers, - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRqEb03EjvBPtnXfVAQLqAAP+PHgTq5B8pJR20DrbaLr3xx+vZAUJN2TK o4s9+5dPKrsm7YCoMkGla2xongmZcFqy9KlMuNJ3FT/5JgeneC4Og/52L+/A0HU0 4nFgBaXd1tKLQRVFKxr6CzqLqt9jnNfzXmNNNqweYvoa/wXU44WxL0sdiLxjljKT 1yzjcd3D1aA= =9uBA -----END PGP SIGNATURE----- From pje at telecommunity.com Fri Jul 20 23:18:57 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 20 Jul 2007 17:18:57 -0400 Subject: [Distutils] Bazaar support for setuptools In-Reply-To: <63CA092B-CCC2-4491-BE43-87C4B4E4381B@python.org> References: <63CA092B-CCC2-4491-BE43-87C4B4E4381B@python.org> Message-ID: <20070720211640.5677A3A403A@sparrow.telecommunity.com> At 04:32 PM 7/20/2007 -0400, Barry Warsaw wrote: >Well, I hacked this together pretty quickly but it seems to work for >my particular use case. The Mailman source code is now kept under >the Bazaar revision control system, and I'm about to merge a branch >that converts the Mailman 3 branch from autoconf-based builds to >setuptools. So I needed Bazaar support in order to build my sdist >files and such. > >Writing the plugin code was pretty easy after Michael Hudson gave me >the key bzrlib clues. Maybe this could be included by default in the >next version of setuptools? No, since it relies on a package (bzrlib) that's not in the stdlib or bundled with setuptools. You can, however, just add a setup.py to your module that declares the entry point, and upload the whole thing to Cheeseshop. You can even declare its dependency on bzrlib and have that automatically installed too, assuming that it's either on the Cheeseshop or you can supply a download link or page with download links in your setup(find_links=[...]). >Side note: I found it interesting and a bit annoying that setuptools >doesn't call find_files_for_bzr() recursively. It would seem like >the framework should do the recursion instead of the plugin, as I'd >think it would be the most common use case. Well, the assumption is that only the plugin knows whether a directory is a candidate for recursion. Also, some revision control systems only keep path information in one place, so trying to call the plugin recursively would be redundant. >In any event, the >setuptools documentation should probably be clear that it's up to the >plugin to recurse into subdirectories. Fair enough; patches welcome. In fact, feel free to just go ahead and check documentation additions right in, just drop me a line and let me know what you did so I can double-check and edit if necessary. From barry at python.org Sat Jul 21 01:31:53 2007 From: barry at python.org (Barry Warsaw) Date: Fri, 20 Jul 2007 19:31:53 -0400 Subject: [Distutils] Bazaar support for setuptools In-Reply-To: <20070720211640.5677A3A403A@sparrow.telecommunity.com> References: <63CA092B-CCC2-4491-BE43-87C4B4E4381B@python.org> <20070720211640.5677A3A403A@sparrow.telecommunity.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Phillip, thanks for the quick response. On Jul 20, 2007, at 5:18 PM, Phillip J. Eby wrote: > At 04:32 PM 7/20/2007 -0400, Barry Warsaw wrote: >> Well, I hacked this together pretty quickly but it seems to work for >> my particular use case. The Mailman source code is now kept under >> the Bazaar revision control system, and I'm about to merge a branch >> that converts the Mailman 3 branch from autoconf-based builds to >> setuptools. So I needed Bazaar support in order to build my sdist >> files and such. >> >> Writing the plugin code was pretty easy after Michael Hudson gave me >> the key bzrlib clues. Maybe this could be included by default in the >> next version of setuptools? > > No, since it relies on a package (bzrlib) that's not in the stdlib > or bundled with setuptools. Would it help if the plugin could fall back to the command line bzr (1) program if bzrlib wasn't installed in your Python? I've modified the code to do this and it seems to work fine either way. I haven't looked, but ISTM that both CVS and Subversion support have to work somewhat similarly. How does setuptools handle things if either cvs (1) or svn(1) weren't available? > You can, however, just add a setup.py to your module that declares > the entry point, and upload the whole thing to Cheeseshop. I could if the Cheeseshop was up right now. ;) Great idea. The code lives here: https://code.launchpad.net/~barry/setuptoolsbzr/trunk and I'll upload it to the Cheeseshop as soon as that service gets repaired. > You can even declare its dependency on bzrlib and have that > automatically installed too, assuming that it's either on the > Cheeseshop or you can supply a download link or page with download > links in your setup(find_links=[...]). I don't quite understand this last bit. Is there documentation on the find_links keyword argument? I couldn't find it. What I did find was dependency_links so I added the following to my setup.py: entry_points = { 'setuptools.file_finders': [ 'bzr = setuptoolsbzr:find_files_for_bzr', ], }, # Optionally grab bzr, which provides bzrlib extras_require = { 'bzrlib': ['bzr'], }, dependency_links = [ 'http://bazaar-vcs.org', ] I haven't actually tested this part because I can't find documentation on instructing 'python setup.py install' to install optional packages (is that even possible from the command line?). Note further that while bzrlib itself isn't a Cheeseshop package, there's a bzr tarball on that site that should -- I think -- install bzrlib as part of its setup.py. >> Side note: I found it interesting and a bit annoying that setuptools >> doesn't call find_files_for_bzr() recursively. It would seem like >> the framework should do the recursion instead of the plugin, as I'd >> think it would be the most common use case. > > Well, the assumption is that only the plugin knows whether a > directory is a candidate for recursion. Also, some revision > control systems only keep path information in one place, so trying > to call the plugin recursively would be redundant. Make sense, thanks. >> In any event, the >> setuptools documentation should probably be clear that it's up to the >> plugin to recurse into subdirectories. > > Fair enough; patches welcome. In fact, feel free to just go ahead > and check documentation additions right in, just drop me a line and > let me know what you did so I can double-check and edit if necessary. The code's in the sandbox, right? I may indeed do that. Cheers, - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRqFF6nEjvBPtnXfVAQJvhwP/T0rXxGARQDrQ2OdVcu3/3IMNY+MCUDj3 B8hYflHST/XgxVBnQGtlhROsKsMxvsqqVePuGlj9F0ykViaTu5dMcZ6XFdHw47QJ wZ7S++nQ1lN/DsRN1UJsIXpnrNkLkxaAloiTU8omkS7xJ0MVVRSkespt8iyBWaiz AZ7aCkWWO6M= =LswG -----END PGP SIGNATURE----- From pje at telecommunity.com Sat Jul 21 02:15:55 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 20 Jul 2007 20:15:55 -0400 Subject: [Distutils] Bazaar support for setuptools In-Reply-To: References: <63CA092B-CCC2-4491-BE43-87C4B4E4381B@python.org> <20070720211640.5677A3A403A@sparrow.telecommunity.com> Message-ID: <20070721001338.93AC63A403A@sparrow.telecommunity.com> At 07:31 PM 7/20/2007 -0400, Barry Warsaw wrote: >Would it help if the plugin could fall back to the command line bzr >(1) program if bzrlib wasn't installed in your Python? I've modified >the code to do this and it seems to work fine either way. I haven't >looked, but ISTM that both CVS and Subversion support have to work >somewhat similarly. How does setuptools handle things if either cvs >(1) or svn(1) weren't available? It doesn't use them; it just reads CVS/Entries and .svn/entries. >>You can even declare its dependency on bzrlib and have that >>automatically installed too, assuming that it's either on the >>Cheeseshop or you can supply a download link or page with download >>links in your setup(find_links=[...]). > >I don't quite understand this last bit. Is there documentation on >the find_links keyword argument? I couldn't find it. What I did >find was dependency_links so I added the following to my setup.py: Eep, I meant dependency_links. >I haven't actually tested this part because I can't find >documentation on instructing 'python setup.py install' to install >optional packages (is that even possible from the command line?). Try 'install_requires' instead of 'extras_require'. From barry at python.org Sat Jul 21 05:24:29 2007 From: barry at python.org (Barry Warsaw) Date: Fri, 20 Jul 2007 23:24:29 -0400 Subject: [Distutils] Bazaar support for setuptools In-Reply-To: <20070721001338.93AC63A403A@sparrow.telecommunity.com> References: <63CA092B-CCC2-4491-BE43-87C4B4E4381B@python.org> <20070720211640.5677A3A403A@sparrow.telecommunity.com> <20070721001338.93AC63A403A@sparrow.telecommunity.com> Message-ID: <0BA455A0-8BBA-40BD-911A-03981B76479E@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jul 20, 2007, at 8:15 PM, Phillip J. Eby wrote: > At 07:31 PM 7/20/2007 -0400, Barry Warsaw wrote: >> Would it help if the plugin could fall back to the command line >> bzr (1) program if bzrlib wasn't installed in your Python? I've >> modified >> the code to do this and it seems to work fine either way. I haven't >> looked, but ISTM that both CVS and Subversion support have to work >> somewhat similarly. How does setuptools handle things if either >> cvs (1) or svn(1) weren't available? > > It doesn't use them; it just reads CVS/Entries and .svn/entries. Ah, clever! >>> You can even declare its dependency on bzrlib and have that >>> automatically installed too, assuming that it's either on the >>> Cheeseshop or you can supply a download link or page with download >>> links in your setup(find_links=[...]). >> >> I don't quite understand this last bit. Is there documentation on >> the find_links keyword argument? I couldn't find it. What I did >> find was dependency_links so I added the following to my setup.py: > > Eep, I meant dependency_links. Turns out not to be necessary. Now that the cheeseshop is back up, I was able to check and find out that bzr is in there. It doesn't exactly help though because the tarball is broken. There's an open bug on this: https://bugs.launchpad.net/launchpad/+bug/125521/+index >> I haven't actually tested this part because I can't find >> documentation on instructing 'python setup.py install' to install >> optional packages (is that even possible from the command line?). > > Try 'install_requires' instead of 'extras_require'. Great for testing, thanks. But um, how /do/ normal users tell setup.py to install those extra packages? Again, thanks Phillip. Now that the Cheeseshop is back up, you can grab the package from here: http://cheeseshop.python.org/pypi/setuptoolsbzr Cheers, - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRqF8bXEjvBPtnXfVAQKxEgP/a3xzia9QIljMzCYa5WTvt6mduOPAvigm iFkMNltxPsP08v6WmoGoFPuYfZOuDve89lw1hpW7JXk4Be7xpgUOC+2yG9rkJTp6 brWmoeH7o/AunfyrgqAkbc2zpIq/GUXDtF+Lx5/jXfClXvliBaR9G2X+chSjSlF1 8pT+D/Liv0w= =CUOD -----END PGP SIGNATURE----- From pje at telecommunity.com Sat Jul 21 07:02:03 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 21 Jul 2007 01:02:03 -0400 Subject: [Distutils] Bazaar support for setuptools In-Reply-To: <0BA455A0-8BBA-40BD-911A-03981B76479E@python.org> References: <63CA092B-CCC2-4491-BE43-87C4B4E4381B@python.org> <20070720211640.5677A3A403A@sparrow.telecommunity.com> <20070721001338.93AC63A403A@sparrow.telecommunity.com> <0BA455A0-8BBA-40BD-911A-03981B76479E@python.org> Message-ID: <20070721045946.4AD203A403A@sparrow.telecommunity.com> At 11:24 PM 7/20/2007 -0400, Barry Warsaw wrote: >Great for testing, thanks. But um, how /do/ normal users tell >setup.py to install those extra packages? Normally, people use install_requires, and the packages get installed automatically if you're using setuptools or easy_install. Meanwhile, if you want to install a package's extras, you just use "easy_install foo[bar,baz]" to install package foo with extras bar and baz. "setup.py install" can't and won't ever handle extras. Please keep in mind, though, that extras are for *optional* features. Your package won't work without bzrlib, so that's not really an "extra" and probably shouldn't be defined as such. From martin at v.loewis.de Sat Jul 21 08:05:13 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 21 Jul 2007 08:05:13 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <20070720200721.88E1D3A403A@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <20070720200721.88E1D3A403A@sparrow.telecommunity.com> Message-ID: <46A1A219.60906@v.loewis.de> > (On a perhaps not entirely unrelated note, the Cheeseshop appears to > be down at the moment: > > """Error... > > There's been a problem with your request > > psycopg.OperationalError: no connection to the server""") Around that time, the Postgres log has these entries: 2007-07-20 21:53:24 [14636] LOG: received fast shutdown request 2007-07-20 21:53:24 [14636] LOG: aborting any active transactions 2007-07-20 21:53:24 [26166] FATAL: terminating connection due to administrator command 2007-07-20 21:53:24 [15769] FATAL: terminating connection due to administrator command 2007-07-20 21:53:24 [10390] FATAL: terminating connection due to administrator command 2007-07-20 21:53:24 [31182] FATAL: terminating connection due to administrator command 2007-07-20 21:53:24 [30066] FATAL: terminating connection due to administrator command 2007-07-20 21:53:24 [10162] FATAL: terminating connection due to administrator command 2007-07-20 21:53:24 [17452] FATAL: terminating connection due to administrator command 2007-07-20 21:53:24 [17147] FATAL: terminating connection due to administrator command 2007-07-20 21:53:24 [1159] LOG: shutting down 2007-07-20 21:53:26 [1159] LOG: database system is shut down 2007-07-20 21:53:33 [1469] LOG: database system was shut down at 2007-07-20 21:53:26 CEST 2007-07-20 21:53:33 [1469] LOG: checkpoint record is at A/FD833F0 2007-07-20 21:53:33 [1469] LOG: redo record is at A/FD833F0; undo record is at 0/0; shutdown TRUE 2007-07-20 21:53:33 [1469] LOG: next transaction ID: 110977718; next OID: 61913929 2007-07-20 21:53:33 [1469] LOG: database system is ready and Sean Reifschneider was logged in, so I suspect he did some maintenance work. Sean? Regards, Martin From jafo at tummy.com Sat Jul 21 08:17:20 2007 From: jafo at tummy.com (Sean Reifschneider) Date: Sat, 21 Jul 2007 00:17:20 -0600 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A1A219.60906@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <20070720200721.88E1D3A403A@sparrow.telecommunity.com> <46A1A219.60906@v.loewis.de> Message-ID: <20070721061720.GB4489@tummy.com> On Sat, Jul 21, 2007 at 08:05:13AM +0200, "Martin v. L?wis" wrote: >Around that time, the Postgres log has these entries: There was an upgrade of Postgres done earlier, as far as I can see, pypi is running. It must have been resolved earlier. AMK mentioned there was a problem with the upgrade restart and Apache had to be restarted, that was like 6 hours ago though. Thanks, Sean -- "I not only use all the brains that I have, but all that I can borrow." -- Woodrow Wilson Sean Reifschneider, Member of Technical Staff tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability From barry at python.org Sat Jul 21 16:43:54 2007 From: barry at python.org (Barry Warsaw) Date: Sat, 21 Jul 2007 10:43:54 -0400 Subject: [Distutils] Bazaar support for setuptools In-Reply-To: <20070721045946.4AD203A403A@sparrow.telecommunity.com> References: <63CA092B-CCC2-4491-BE43-87C4B4E4381B@python.org> <20070720211640.5677A3A403A@sparrow.telecommunity.com> <20070721001338.93AC63A403A@sparrow.telecommunity.com> <0BA455A0-8BBA-40BD-911A-03981B76479E@python.org> <20070721045946.4AD203A403A@sparrow.telecommunity.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 [removed the bazaar list from the recipients] On Jul 21, 2007, at 1:02 AM, Phillip J. Eby wrote: > At 11:24 PM 7/20/2007 -0400, Barry Warsaw wrote: >> Great for testing, thanks. But um, how /do/ normal users tell >> setup.py to install those extra packages? > > Normally, people use install_requires, and the packages get > installed automatically if you're using setuptools or easy_install. > > Meanwhile, if you want to install a package's extras, you just use > "easy_install foo[bar,baz]" to install package foo with extras bar > and baz. "setup.py install" can't and won't ever handle extras. > > Please keep in mind, though, that extras are for *optional* > features. Your package won't work without bzrlib, so that's not > really an "extra" and probably shouldn't be defined as such. Agreed that in this particular example, extras aren't appropriate. But I have another use case where I definitely want to declare an optional dependency (e.g. munepy's optional dependence on the nose package for tests). So I'm wondering, why do you say setup.py can't and won't every support this? Is there a technical or philosophical limitation? I would think there should be a user friendlier way for a developer sitting there at the setup.py file to request that the optional packages get installed. I don't count using a different command as "user friendlier". Cheers, - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRqIbq3EjvBPtnXfVAQKKBwP+NIOV6bW6Pj55E2X76h3OYs1NT+OSP/ej pP8n6Vwmaq35fAfuwonYdQ6NErnf5SQqIrXLLmR8EgVQwoiGrzt4w/cd2AXEtn6I mZE87+P6wd4a96ts7ypUoh7keRdm8u/uX0knb2WBaZzT8AV67/kGknL08MlMWlRM NxXqn7qu9Mw= =r6Xh -----END PGP SIGNATURE----- From martin at v.loewis.de Sat Jul 21 19:00:30 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 21 Jul 2007 19:00:30 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> Message-ID: <46A23BAE.5090907@v.loewis.de> > I've created and experimental prototype setuptools-specific package > index at > > http://download.zope.org/ppix I've now added something similar as http://cheeseshop.python.org/simple/ It differs from your site in a few ways: - it does include a top-level index of all packages (but neither releases nor descriptions) - it's always current, due to being dynamically computed - it may differ in the precise list of URLs displayed; if there are important deviations, please let me know. Regards, Martin From jim at zope.com Sat Jul 21 19:12:48 2007 From: jim at zope.com (Jim Fulton) Date: Sat, 21 Jul 2007 13:12:48 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A23BAE.5090907@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> Message-ID: <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> On Jul 21, 2007, at 1:00 PM, Martin v. L?wis wrote: >> I've created and experimental prototype setuptools-specific package >> index at >> >> http://download.zope.org/ppix > > I've now added something similar as > > http://cheeseshop.python.org/simple/ Way cool! > > It differs from your site in a few ways: > > - it does include a top-level index of all packages (but neither > releases nor descriptions) Why? This is a relatively expensive page, due to it's size I assume, that really provides no value. This will slow down setuptools. > - it's always current, due to being dynamically computed And also unreliable, for the same reason. For example, it would have been inaccessible yesterday afternoon. And also puts more load on the server. It would be much better imo if static pages could be written on writes. > - it may differ in the precise list of URLs displayed; > if there are important deviations, please let me know. The download and homepage URL anchors need rel="download" or rel="homepage". They lack the #egg= links. Compare your page for setuptools to mine. Also, some packages use their pypi pages as their home page links. You want to exclude these, otherwise, setuptools will circle around to the human interface, which defeats point of the simple interface. Thanks for plugging away on this. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Sat Jul 21 19:48:16 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 21 Jul 2007 13:48:16 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A23BAE.5090907@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> Message-ID: <20070721174558.DDF923A403A@sparrow.telecommunity.com> At 07:00 PM 7/21/2007 +0200, Martin v. L?wis wrote: > > I've created and experimental prototype setuptools-specific package > > index at > > > > http://download.zope.org/ppix > >I've now added something similar as > >http://cheeseshop.python.org/simple/ It's very fast, thanks. >It differs from your site in a few ways: > >- it does include a top-level index of all packages (but neither > releases nor descriptions) Unfortunately, that doesn't help current versions of setuptools. See point #7 of: http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api Setuptools looks for release links, not package links on that page. Compare: $ easy_install -vvvi http://cheeseshop.python.org/simple Pywin32 Searching for Pywin32 Reading http://cheeseshop.python.org/simple/Pywin32/ Couldn't find index page for 'Pywin32' (maybe misspelled?) Scanning index of all packages (this may take a while) Reading http://cheeseshop.python.org/simple/ No local packages or download links found for Pywin32 error: Could not find suitable distribution for Requirement.parse('Pywin32') $ easy_install -vvvi http://cheeseshop.python.org/pypi Pywin32 Searching for Pywin32 Reading http://cheeseshop.python.org/pypi/Pywin32/ Couldn't find index page for 'Pywin32' (maybe misspelled?) Scanning index of all packages (this may take a while) Reading http://cheeseshop.python.org/pypi/ Reading http://cheeseshop.python.org/pypi/pywin32/210 Reading http://sf.net/projects/pywin32 ... >- it's always current, due to being dynamically computed >- it may differ in the precise list of URLs displayed; > if there are important deviations, please let me know. Jim's already mentioned these, but the rel="" info (per the index API spec's point #6), and the links embedded in the long_description field (per point #4) are missing. Without these, easy_install can't find sourceforge links, subversion checkouts, or any other embedded direct download links. For example: $ easy_install -vvvi http://cheeseshop.python.org/simple pywin32 Searching for pywin32 Reading http://cheeseshop.python.org/simple/pywin32/ No local packages or download links found for pywin32 error: Could not find suitable distribution for Requirement.parse('pywin32') $ easy_install -vvvi http://cheeseshop.python.org/pypi pywin32 Searching for pywin32 Reading http://cheeseshop.python.org/pypi/pywin32/ Reading http://sf.net/projects/pywin32 Reading http://sourceforge.net/project/showfiles.php?group_id=78018 Found link: http://downloads.sourceforge.net/pywin32/pywin32-210.win32-py2.2.exe?modtime=1159009204&big_mirror=0 ...[a dozen more links] $ easy_install -i http://cheeseshop.python.org/simple setuptools==dev Searching for setuptools==dev Reading http://cheeseshop.python.org/simple/setuptools/ No local packages or download links found for setuptools==dev error: Could not find suitable distribution for Requirement.parse('setuptools==dev') $ easy_install -i http://cheeseshop.python.org/pypi setuptools==dev Searching for setuptools==dev Reading http://cheeseshop.python.org/pypi/setuptools/ Reading http://cheeseshop.python.org/pypi/setuptools Reading http://cheeseshop.python.org/pypi/setuptools/0.6c6 Best match: setuptools dev Downloading http://svn.python.org/projects/sandbox/trunk/setuptools/#egg=setuptools-dev Doing subversion checkout from http://svn.python.org/projects/sandbox/trunk/setuptools/ to ... From martin at v.loewis.de Sat Jul 21 21:08:52 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 21 Jul 2007 21:08:52 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> Message-ID: <46A259C4.6090605@v.loewis.de> >> - it does include a top-level index of all packages (but neither >> releases nor descriptions) > > Why? This is a relatively expensive page, due to it's size I assume, > that really provides no value. This will slow down setuptools. IIUC, it won't slow down setuptools, as setuptools looks at it only if it cannot find the real package page due to a misspelling. So as long as everything is spelled correctly, it should not provide any slowdown. If people do misspell a package name when invoking easy_install, they get the feature that you consider of no value. As for performance - 30 downloads take 3.9s currently from nearby. >> - it's always current, due to being dynamically computed > > And also unreliable, for the same reason. For example, it would have > been inaccessible yesterday afternoon. The same could happen to Apache, too, of course. svn.python.org sometimes fails to restart when a restart is request on log rotation. Any software is unreliable; to reduce downtime, you need an operator that is available when something breaks. > And also puts more load on the server. It would be much better imo > if static pages could be written on writes. Contributions are welcome. In addition to me considering it futile, I also don't know how to implement it correctly. >> - it may differ in the precise list of URLs displayed; >> if there are important deviations, please let me know. > > The download and homepage URL anchors need rel="download" or > rel="homepage". Done. > They lack the #egg= links. How are these computed? > Also, some packages use their pypi pages as their home page links. Ok, done. Regards, Martin From g.brandl at gmx.net Sat Jul 21 21:23:16 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 21 Jul 2007 21:23:16 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A23BAE.5090907@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> Message-ID: Martin v. L?wis schrieb: >> I've created and experimental prototype setuptools-specific package >> index at >> >> http://download.zope.org/ppix > > I've now added something similar as > > http://cheeseshop.python.org/simple/ > > It differs from your site in a few ways: > > - it does include a top-level index of all packages (but neither > releases nor descriptions) > - it's always current, due to being dynamically computed > - it may differ in the precise list of URLs displayed; > if there are important deviations, please let me know. What I, as an outsider, can see: for the Pygments package, Jim's page lists the development link from the package description (http://trac.pocoo.org/repos/pygments/trunk#egg=Pygments-dev), but it looks like it's badly extracted (it has a trailing ">`__"), yours doesn't list it at all. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From martin at v.loewis.de Sat Jul 21 21:23:30 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 21 Jul 2007 21:23:30 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <20070721174558.DDF923A403A@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <20070721174558.DDF923A403A@sparrow.telecommunity.com> Message-ID: <46A25D32.4080606@v.loewis.de> > Unfortunately, that doesn't help current versions of setuptools. See > point #7 of: > > http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api > > Setuptools looks for release links, not package links on that page. I don't understand. What's a "release link"? The links on the index page *do* go to the "project's active version pages", as specified (there aren't any numbered version pages) Jim left out that page entirely - are you saying it is impossible to provide such an index page with the page structure that Jim proposed? > $ easy_install -vvvi http://cheeseshop.python.org/simple Pywin32 > Searching for Pywin32 > Reading http://cheeseshop.python.org/simple/Pywin32/ > Couldn't find index page for 'Pywin32' (maybe misspelled?) > Scanning index of all packages (this may take a while) > Reading http://cheeseshop.python.org/simple/ > No local packages or download links found for Pywin32 I see that it doesn't work, but I cannot understand why. On http://cheeseshop.python.org/simple/ "pywin32" is clearly linked, so it should be able to resolve the misspelling. > Jim's already mentioned these, but the rel="" info (per the index API > spec's point #6), This is fixed. > and the links embedded in the long_description field > (per point #4) are missing. I have to think about this more. Is it correct that you want all href attributes of all a elements in the long_description? And how do you know what the long_description is from just looking at the rendered page? Regards, Martin From pje at telecommunity.com Sat Jul 21 21:51:26 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 21 Jul 2007 15:51:26 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A25D32.4080606@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <20070721174558.DDF923A403A@sparrow.telecommunity.com> <46A25D32.4080606@v.loewis.de> Message-ID: <20070721194908.F16373A403A@sparrow.telecommunity.com> At 09:23 PM 7/21/2007 +0200, Martin v. L?wis wrote: > > Unfortunately, that doesn't help current versions of setuptools. See > > point #7 of: > > > > http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api > > > > Setuptools looks for release links, not package links on that page. > >I don't understand. What's a "release link"? The links on the index >page *do* go to the "project's active version pages", as specified >(there aren't any numbered version pages) See point #2: """2. Individual project version pages' URLs must be of the form base/projectname/version, where base is the package index's base URL.""" That's what's meant by "version pages" in point #7 -- i.e., they *must* be of that two-part form for setuptools to recognize them as such. >I see that it doesn't work, but I cannot understand why. >On > >http://cheeseshop.python.org/simple/ > >"pywin32" is clearly linked, so it should be able to resolve >the misspelling. It could perhaps be *changed* to do so, but at present it follows the spec's definition of "version page" URLs. > > Jim's already mentioned these, but the rel="" info (per the index API > > spec's point #6), > >This is fixed. Great; Sourceforge and other offsite download pages work now. > > and the links embedded in the long_description field > > (per point #4) are missing. > >I have to think about this more. Is it correct that you want all href >attributes of all a elements in the long_description? Yes; of course, the usual rendering needs to be applied, since long_description can contain reStructuredText. > And how do you >know what the long_description is from just looking at the rendered >page? You don't need to; easy_install discovers those links the same way it does any other Cheeseshop-provided download links. From easy_install's point of view, the entire page is just one big mass of links that might point to downloads: """4. ...It is explicitly permitted for a project's "long_description" to include URLs, and these should be formatted as HTML links by the package index, as EasyInstall does *no special processing* [emph. added] to identify what parts of a page are index-specific and which are part of the project's supplied description.""" In other words, the *only* links that are specially handled are the "rel" ones, which it follows unconditionally to look for additional direct download links. All other links are merely *inspected* to see if they obviously refer to a downloadable package (e.g. .tgz, .zip, .egg, .exe etc., or explicitly-marked #egg). As a side-effect, this means that links to perform Cheeseshop operations, links to other parts of python.org, etc. are simply ignored, as they are not links to downloadables nor marked as #egg. If a URL can be determined by inspection to be a download link, then easy_install extracts version and platform info from the URL and adds it as a candidate for download selection. When both the home page and download URL have been read, along with any detected "active version pages" (as defined above), then easy_install chooses the "best" download URL from all the candidates it has seen up to that point. From pje at telecommunity.com Sat Jul 21 21:53:40 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 21 Jul 2007 15:53:40 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> Message-ID: <20070721195122.CF2343A40D7@sparrow.telecommunity.com> At 09:23 PM 7/21/2007 +0200, Georg Brandl wrote: >What I, as an outsider, can see: for the Pygments package, Jim's page >lists the development link from the package description >(http://trac.pocoo.org/repos/pygments/trunk#egg=Pygments-dev), but >it looks like it's badly extracted (it has a trailing ">`__"), yours >doesn't list it at all. Hm, perhaps Jim is extracting it by looking for #egg URLs, rather than by actually processing the reST markup with docutils. That should probably be fixed, since there are many ways to specify URLs in reST and handling them all with regular expressions is unlikely to work as well as applying regular expressions to the resulting HTML. :) (Also, looking only for #egg links will miss non-#egg links embedded in the long_description, in the event that someone places direct download links there.) From martin at v.loewis.de Sun Jul 22 00:53:03 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 22 Jul 2007 00:53:03 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <20070721194908.F16373A403A@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <20070721174558.DDF923A403A@sparrow.telecommunity.com> <46A25D32.4080606@v.loewis.de> <20070721194908.F16373A403A@sparrow.telecommunity.com> Message-ID: <46A28E4F.5070905@v.loewis.de> > See point #2: > > """2. Individual project version pages' URLs must be of the form > base/projectname/version, where base is the package index's base URL.""" > > That's what's meant by "version pages" in point #7 -- i.e., they *must* > be of that two-part form for setuptools to recognize them as such. Ok, but I still cannot see how to fix that: there simply *is* no version part that I could point to. Does that mean that Jim's approach does not work? > Yes; of course, the usual rendering needs to be applied, since > long_description can contain reStructuredText. Ok, I now added these links as well. Regards, Martin From pje at telecommunity.com Sun Jul 22 01:20:04 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 21 Jul 2007 19:20:04 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A28E4F.5070905@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <20070721174558.DDF923A403A@sparrow.telecommunity.com> <46A25D32.4080606@v.loewis.de> <20070721194908.F16373A403A@sparrow.telecommunity.com> <46A28E4F.5070905@v.loewis.de> Message-ID: <20070721231808.2D5793A403A@sparrow.telecommunity.com> At 12:53 AM 7/22/2007 +0200, Martin v. L?wis wrote: > > See point #2: > > > > """2. Individual project version pages' URLs must be of the form > > base/projectname/version, where base is the package index's base URL.""" > > > > That's what's meant by "version pages" in point #7 -- i.e., they *must* > > be of that two-part form for setuptools to recognize them as such. > >Ok, but I still cannot see how to fix that: there simply *is* no >version part that I could point to. Actually, 'version' is allowed to be an empty string, so simply adding a trailing '/' to the links you're generating now should work. The only thing the version part of a version page URL is used for, is to handle links to .py files: setuptools uses the package version (if available) to synthesize a setup.py for installing standalone .py files. If the version is not available, it won't be able to do that, but that's a relatively minor feature, all things considered. Few packages are distributed via a single .py download URL, but the package index could actually tack on an #egg designator to such links in order to preserve 100% backward-compatibility. >Does that mean that Jim's approach does not work? Jim isn't providing the top-level index, and thus doesn't provide punctuation or case corrections. The "version pages" convention is only used by setuptools to discover additional index pages for crawling, anyway, and his whole design is intended to prevent crawling. > > Yes; of course, the usual rendering needs to be applied, since > > long_description can contain reStructuredText. > >Ok, I now added these links as well. Looks good, thanks! From martin at v.loewis.de Sun Jul 22 09:42:19 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 22 Jul 2007 09:42:19 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <20070721231808.2D5793A403A@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <20070721174558.DDF923A403A@sparrow.telecommunity.com> <46A25D32.4080606@v.loewis.de> <20070721194908.F16373A403A@sparrow.telecommunity.com> <46A28E4F.5070905@v.loewis.de> <20070721231808.2D5793A403A@sparrow.telecommunity.com> Message-ID: <46A30A5B.4020007@v.loewis.de> > Actually, 'version' is allowed to be an empty string, so simply adding a > trailing '/' to the links you're generating now should work. It does indeed. Regards, Martin From jim at zope.com Sun Jul 22 15:09:44 2007 From: jim at zope.com (Jim Fulton) Date: Sun, 22 Jul 2007 09:09:44 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A259C4.6090605@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> Message-ID: On Jul 21, 2007, at 3:08 PM, Martin v. L?wis wrote: >>> - it does include a top-level index of all packages (but neither >>> releases nor descriptions) >> >> Why? This is a relatively expensive page, due to it's size I assume, >> that really provides no value. This will slow down setuptools. > > IIUC, it won't slow down setuptools, as setuptools looks at it only > if it cannot find the real package page due to a misspelling. So > as long as everything is spelled correctly, it should not provide > any slowdown. > > If people do misspell a package name when invoking easy_install, > they get the feature that you consider of no value. That is not correct. Not all packages are in PyPI. Using a package that isn't in PyPI will trigger a fetch of that page. It isn't misspelled, it's just not there. People should *not* misspell pages when using setuptools. They should certainly not use misspelled package names in requirements. In my strongly help opinion, allowing imprecise names in requirements and setuptools command if of negative value. > As for performance - 30 downloads take 3.9s currently from nearby. That's nice. For me, that page takes 3 or 4 times as long as other pages. >>> - it's always current, due to being dynamically computed >> >> And also unreliable, for the same reason. For example, it would have >> been inaccessible yesterday afternoon. > > The same could happen to Apache, too, of course. svn.python.org > sometimes fails to restart when a restart is request on log rotation. > > Any software is unreliable; to reduce downtime, you need an operator > that is available when something breaks. Apache has a far better record than the cheeseshop. I give up. >> And also puts more load on the server. It would be much better imo >> if static pages could be written on writes. > > Contributions are welcome. In addition to me considering it futile, > I also don't know how to implement it correctly. I'd be happy to contribute my polling version. That solves my problems and I can't justify the additional effort to figure out the cheeseshop softtware. ... >> They lack the #egg= links. > > How are these computed? By parsing the description. Apparently, I'm going this incorrectly. I'll have to look into that. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Sun Jul 22 15:16:44 2007 From: jim at zope.com (Jim Fulton) Date: Sun, 22 Jul 2007 09:16:44 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <20070721195122.CF2343A40D7@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <20070721195122.CF2343A40D7@sparrow.telecommunity.com> Message-ID: On Jul 21, 2007, at 3:53 PM, Phillip J. Eby wrote: > At 09:23 PM 7/21/2007 +0200, Georg Brandl wrote: >> What I, as an outsider, can see: for the Pygments package, Jim's page >> lists the development link from the package description >> (http://trac.pocoo.org/repos/pygments/trunk#egg=Pygments-dev), but >> it looks like it's badly extracted (it has a trailing ">`__"), yours >> doesn't list it at all. > > Hm, perhaps Jim is extracting it by looking for #egg URLs, rather > than by actually processing the reST markup with docutils. Yup. > That should probably be fixed, since there are many ways to specify > URLs in reST and handling them all with regular expressions is > unlikely to work Yeah, I was hoping to get off easy. :) > as well as applying regular expressions to the resulting HTML. :) :) > (Also, looking only for #egg links will miss non-#egg links > embedded in the long_description, in the event that someone places > direct download links there.) By this, I assume you mean direct links to distributions. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Sun Jul 22 15:19:05 2007 From: jim at zope.com (Jim Fulton) Date: Sun, 22 Jul 2007 09:19:05 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <20070721231808.2D5793A403A@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <20070721174558.DDF923A403A@sparrow.telecommunity.com> <46A25D32.4080606@v.loewis.de> <20070721194908.F16373A403A@sparrow.telecommunity.com> <46A28E4F.5070905@v.loewis.de> <20070721231808.2D5793A403A@sparrow.telecommunity.com> Message-ID: On Jul 21, 2007, at 7:20 PM, Phillip J. Eby wrote: ... > Jim isn't providing the top-level index, and thus doesn't provide > punctuation or case corrections. Yup > The "version pages" convention is only used by setuptools to > discover additional index pages for crawling, anyway, and his whole > design is intended to prevent crawling. That's a secondary benefit. The main goal is to avoid the expense of that page for packages that aren't in PyPI, as some packages I use aren't. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From martin at v.loewis.de Sun Jul 22 18:24:41 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 22 Jul 2007 18:24:41 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> Message-ID: <46A384C9.8040404@v.loewis.de> >> If people do misspell a package name when invoking easy_install, >> they get the feature that you consider of no value. > > That is not correct. Not all packages are in PyPI. Using a package that > isn't in PyPI will trigger a fetch of that page. I don't understand. What page is fetched if the package is not in PyPI? > It isn't misspelled, > it's just not there. People should *not* misspell pages when using > setuptools. They should certainly not use misspelled package names in > requirements. In my strongly help opinion, allowing imprecise names in > requirements and setuptools command if of negative value. I cannot comment on. I don't use setuptools, and have no intuition what is good or bad when using it (for example, I consider .egg files and the notion of eggs inherently bad). My main motivation to provide that page is that the setuptools specification says it should be there. As this entire infrastructure is for the sake of setuptools, I find it pointless to not support setuptools fully. > I'd be happy to contribute my polling version. That solves my problems > and I can't justify the additional effort to figure out the cheeseshop > softtware. I'd like to hear other opinions here. Would people prefer if the index was always correct (and perhaps somewhat slow), or would they prefer instead that it is super-efficient (and somewhat out-of-date)? Regards, Martin From martin at v.loewis.de Sun Jul 22 18:26:14 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 22 Jul 2007 18:26:14 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <20070721174558.DDF923A403A@sparrow.telecommunity.com> <46A25D32.4080606@v.loewis.de> <20070721194908.F16373A403A@sparrow.telecommunity.com> <46A28E4F.5070905@v.loewis.de> <20070721231808.2D5793A403A@sparrow.telecommunity.com> Message-ID: <46A38526.2010308@v.loewis.de> > That's a secondary benefit. The main goal is to avoid the expense of > that page for packages that aren't in PyPI, as some packages I use aren't. I see. Shouldn't that be fixed by providing an option to setuptools that avoids going to the index for missing packages? Regards, Martin From tseaver at palladion.com Sun Jul 22 18:33:11 2007 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 22 Jul 2007 12:33:11 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A384C9.8040404@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <46A384C9.8040404@v.loewis.de> Message-ID: <46A386C7.5080203@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. L?wis wrote: >>> If people do misspell a package name when invoking easy_install, >>> they get the feature that you consider of no value. >> That is not correct. Not all packages are in PyPI. Using a package that >> isn't in PyPI will trigger a fetch of that page. > > I don't understand. What page is fetched if the package is not in PyPI? I think Jim was referring to a package which is *registered* in PyPI, but whose download location was elsewhere. >> I'd be happy to contribute my polling version. That solves my problems >> and I can't justify the additional effort to figure out the cheeseshop >> softtware. > > I'd like to hear other opinions here. Would people prefer if the index > was always correct (and perhaps somewhat slow), or would they prefer > instead that it is super-efficient (and somewhat out-of-date)? I would prefer the second, particularly as I think the caching solution lends itself to mirroring, which would also improve availability. - From my complete ignorance of the underlying architecture: the polling solution would stay pretty current if there were an extremely cheap way to ask for the latest "transaction ID" on the cheeseshop, or if the query could fetch only registrations newer than the last poll time. Are such queries possible over the XML-RPC interface? Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGo4bH+gerLs4ltQ4RAjiWAJ9/5TeOWAHdwL7PS5QAUnpyZWJzMQCeN5hT 5rRjOHzAu4cf+TKktNntWV8= =p59N -----END PGP SIGNATURE----- From tseaver at palladion.com Sun Jul 22 18:33:11 2007 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 22 Jul 2007 12:33:11 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A384C9.8040404@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <46A384C9.8040404@v.loewis.de> Message-ID: <46A386C7.5080203@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. L?wis wrote: >>> If people do misspell a package name when invoking easy_install, >>> they get the feature that you consider of no value. >> That is not correct. Not all packages are in PyPI. Using a package that >> isn't in PyPI will trigger a fetch of that page. > > I don't understand. What page is fetched if the package is not in PyPI? I think Jim was referring to a package which is *registered* in PyPI, but whose download location was elsewhere. >> I'd be happy to contribute my polling version. That solves my problems >> and I can't justify the additional effort to figure out the cheeseshop >> softtware. > > I'd like to hear other opinions here. Would people prefer if the index > was always correct (and perhaps somewhat slow), or would they prefer > instead that it is super-efficient (and somewhat out-of-date)? I would prefer the second, particularly as I think the caching solution lends itself to mirroring, which would also improve availability. - From my complete ignorance of the underlying architecture: the polling solution would stay pretty current if there were an extremely cheap way to ask for the latest "transaction ID" on the cheeseshop, or if the query could fetch only registrations newer than the last poll time. Are such queries possible over the XML-RPC interface? Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGo4bH+gerLs4ltQ4RAjiWAJ9/5TeOWAHdwL7PS5QAUnpyZWJzMQCeN5hT 5rRjOHzAu4cf+TKktNntWV8= =p59N -----END PGP SIGNATURE----- From pje at telecommunity.com Sun Jul 22 18:40:11 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 22 Jul 2007 12:40:11 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A38526.2010308@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <20070721174558.DDF923A403A@sparrow.telecommunity.com> <46A25D32.4080606@v.loewis.de> <20070721194908.F16373A403A@sparrow.telecommunity.com> <46A28E4F.5070905@v.loewis.de> <20070721231808.2D5793A403A@sparrow.telecommunity.com> <46A38526.2010308@v.loewis.de> Message-ID: <20070722163754.A78EF3A40A9@sparrow.telecommunity.com> At 06:26 PM 7/22/2007 +0200, Martin v. L?wis wrote: > > That's a secondary benefit. The main goal is to avoid the expense of > > that page for packages that aren't in PyPI, as some packages I use aren't. > >I see. Shouldn't that be fixed by providing an option to setuptools >that avoids going to the index for missing packages? There's already such an option; --find-links or -f lets you specify URLs that should be checked before *any* PyPI access occurs. If all dependencies can be met using those URLs without going to PyPI, and you haven't explicitly requested -U (--update), easy_install doesn't go to PyPI. You can also specify such links in a setup script using setup(dependency_links=[...]), which bakes them into the .egg. When searching for that egg's dependencies, easy_install will pick them up and use them. So, it's actually possible to install a package and all its dependencies without using PyPI at all, if the package author(s) bake the URLs in. From jim at zope.com Sun Jul 22 18:38:09 2007 From: jim at zope.com (Jim Fulton) Date: Sun, 22 Jul 2007 12:38:09 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A384C9.8040404@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <46A384C9.8040404@v.loewis.de> Message-ID: On Jul 22, 2007, at 12:24 PM, Martin v. L?wis wrote: >>> If people do misspell a package name when invoking easy_install, >>> they get the feature that you consider of no value. >> >> That is not correct. Not all packages are in PyPI. Using a >> package that >> isn't in PyPI will trigger a fetch of that page. > > I don't understand. What page is fetched if the package is not in > PyPI? We have lots of packages that aren't in PyPI. Some of them aren't ready for PyPI or are not of general interest. Some are proprietary. >> It isn't misspelled, >> it's just not there. People should *not* misspell pages when using >> setuptools. They should certainly not use misspelled package >> names in >> requirements. In my strongly help opinion, allowing imprecise >> names in >> requirements and setuptools command if of negative value. > > I cannot comment on. I don't use setuptools, and have no intuition > what > is good or bad when using it (for example, I consider .egg files and > the notion of eggs inherently bad). > > My main motivation to provide that page is that the setuptools > specification says it should be there. As this entire infrastructure > is for the sake of setuptools, I find it pointless to not support > setuptools fully. Fair enough. Theory beats practicality every time. ;) >> I'd be happy to contribute my polling version. That solves my >> problems >> and I can't justify the additional effort to figure out the >> cheeseshop >> softtware. > > I'd like to hear other opinions here. Yes. This has been a fairly limited discussion. Sigh. > Would people prefer if the index > was always correct (and perhaps somewhat slow), or would they prefer > instead that it is super-efficient (and somewhat out-of-date)? Where somewhat out of date could be a matter of seconds. IMO, a python.org index could poll every few seconds, given that local polling only takes a few milliseconds. I have a feeling that this discussion is going to annoy someone with PyPI software knowledge enough to add baking on write. :) For example, I had the impression that Rene' was planning to invoke scripts after updates. It would be easy to invoke my polling script or a script based on your work, BTW, I'm pretty sure that geographic mirrors are desirable, both for performance and redundancy reasons. I think that, for these, polling once a minute is plenty and puts negligible load on PyPI, assuming that there aren't hundreds of them. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Sun Jul 22 18:41:55 2007 From: jim at zope.com (Jim Fulton) Date: Sun, 22 Jul 2007 12:41:55 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A386C7.5080203@palladion.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <46A384C9.8040404@v.loewis.de> <46A386C7.5080203@palladion.com> Message-ID: <437D4304-ECF3-4240-8C33-F946128F8232@zope.com> On Jul 22, 2007, at 12:33 PM, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Martin v. L?wis wrote: >>>> If people do misspell a package name when invoking easy_install, >>>> they get the feature that you consider of no value. >>> That is not correct. Not all packages are in PyPI. Using a >>> package that >>> isn't in PyPI will trigger a fetch of that page. >> >> I don't understand. What page is fetched if the package is not in >> PyPI? > > I think Jim was referring to a package which is *registered* in PyPI, > but whose download location was elsewhere. No, I was referring to packages that aren't ready for or of interest to PyPI or to proprietary packages. ... > - From my complete ignorance of the underlying architecture: the > polling > solution would stay pretty current if there were an extremely cheap > way > to ask for the latest "transaction ID" on the cheeseshop, or if the > query could fetch only registrations newer than the last poll time. There is such an API thanks to Martin. > Are > such queries possible over the XML-RPC interface? Yup. I'm using them. Queries take only a few milliseconds per request on the server. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From fuzzyman at voidspace.org.uk Sun Jul 22 18:41:39 2007 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 22 Jul 2007 17:41:39 +0100 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A384C9.8040404@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <46A384C9.8040404@v.loewis.de> Message-ID: <46A388C3.7050503@voidspace.org.uk> Martin v. L?wis wrote: >>> If people do misspell a package name when invoking easy_install, >>> they get the feature that you consider of no value. >>> >> That is not correct. Not all packages are in PyPI. Using a package that >> isn't in PyPI will trigger a fetch of that page. >> > > I don't understand. What page is fetched if the package is not in PyPI? > > >> It isn't misspelled, >> it's just not there. People should *not* misspell pages when using >> setuptools. They should certainly not use misspelled package names in >> requirements. In my strongly help opinion, allowing imprecise names in >> requirements and setuptools command if of negative value. >> > > I cannot comment on. I don't use setuptools, and have no intuition what > is good or bad when using it (for example, I consider .egg files and > the notion of eggs inherently bad). > > Oh right. It makes using Python and packages *dramatically* easier. I guess you are resigned to their arrival though. ;-) > My main motivation to provide that page is that the setuptools > specification says it should be there. As this entire infrastructure > is for the sake of setuptools, I find it pointless to not support > setuptools fully. > > Great. >> I'd be happy to contribute my polling version. That solves my problems >> and I can't justify the additional effort to figure out the cheeseshop >> softtware. >> > > I'd like to hear other opinions here. Would people prefer if the index > was always correct (and perhaps somewhat slow), or would they prefer > instead that it is super-efficient (and somewhat out-of-date)? > I'd prefer accuracy over speed here. Michael Foord > Regards, > Martin > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > From pje at telecommunity.com Sun Jul 22 18:51:40 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 22 Jul 2007 12:51:40 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> Message-ID: <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> At 09:09 AM 7/22/2007 -0400, Jim Fulton wrote: >People should *not* misspell pages >when using setuptools. They should certainly not use misspelled >package names in requirements. People do all sorts of things they shouldn't. That doesn't stop them blaming other people for their mistakes. It's said that a 10% improvement in ease-of-use can double a product's users. Case sensitivity is a barrier to entry for new users, and setuptools can't afford any additional entry barriers. A significant part of setuptools' audience includes people who are new to Python, or at least new to installing or distributing Python modules, and quite a lot of setuptools features are aimed squarely at that audience. This happens to be one of them. > In my strongly help opinion, allowing >imprecise names in requirements and setuptools command if of negative >value. I understand that perspective. But practicality beats purity, and this is absolutely a "worse is better" type of situation. Setuptools has lots of features that are targeted at different audiences. There are plenty of features targeted at the group you're in, don't begrudge the other groups their features. :) (This is probably one reason that setuptools is so controversial; everybody can find *something* about it to hate, even if those very same things are quite loved by a different group of users. E.g. you and case-insensitivity, Martin and eggs, etc.) From martin at v.loewis.de Sun Jul 22 18:54:36 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 22 Jul 2007 18:54:36 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A386C7.5080203@palladion.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <46A384C9.8040404@v.loewis.de> <46A386C7.5080203@palladion.com> Message-ID: <46A38BCC.1000707@v.loewis.de> > I would prefer the second, particularly as I think the caching solution > lends itself to mirroring, which would also improve availability. I think this conclusion is wrong: Jim already has a mirror infrastructure that anybody can run, without the need of running that on the central server. > - From my complete ignorance of the underlying architecture: the polling > solution would stay pretty current if there were an extremely cheap way > to ask for the latest "transaction ID" on the cheeseshop, or if the > query could fetch only registrations newer than the last poll time. Are > such queries possible over the XML-RPC interface? Yes; you can ask for all changes since a certain UTC time. People shouldn't invoke that every UTC second, though - once a minute is fine. Regards, Martin From martin at v.loewis.de Sun Jul 22 19:03:49 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 22 Jul 2007 19:03:49 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <46A384C9.8040404@v.loewis.de> Message-ID: <46A38DF5.6010701@v.loewis.de> Jim Fulton schrieb: > On Jul 22, 2007, at 12:24 PM, Martin v. L?wis wrote: >>>> If people do misspell a package name when invoking easy_install, >>>> they get the feature that you consider of no value. >>> >>> That is not correct. Not all packages are in PyPI. Using a package that >>> isn't in PyPI will trigger a fetch of that page. >> >> I don't understand. What page is fetched if the package is not in PyPI? > > We have lots of packages that aren't in PyPI. Some of them aren't ready > for PyPI or are not of general interest. Some are proprietary. Ah, ok. So I stand to my original statement (the one you classified as incorrect): *If* I do misspell a package name, *then* setuptools will correct the spelling if the index page is available. >> Would people prefer if the index >> was always correct (and perhaps somewhat slow), or would they prefer >> instead that it is super-efficient (and somewhat out-of-date)? > > Where somewhat out of date could be a matter of seconds. And where somewhat slower could be "practically not noticable". > BTW, I'm pretty sure that geographic mirrors are desirable, both for > performance and redundancy reasons. I think that, for these, polling > once a minute is plenty and puts negligible load on PyPI, assuming that > there aren't hundreds of them. Sure: I don't mind at all if more people run your software on their machines. If people want it more official, we can have "cheeseshop0.python.org", "cheeseshop1.python.org", and so on, or "de.cheeseshop.python.org", "jp.cheeseshop.python.org", and so on. As I said before: if people also want to mirror the files, I'd ask them provide download statistics. Given the changelog, it would be easy to keep a file mirror up-to-date (of course, if a mirror downloads all files, these downloads also count towards the download statistics - which might confuse people). Regards, Martin From martin at v.loewis.de Sun Jul 22 20:40:05 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 22 Jul 2007 20:40:05 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> Message-ID: <46A3A485.7060602@v.loewis.de> > WRT zc.buildout, refreshing a buildout with just ZODB installed in it > takes about 45 seconds for me using PyPI and about 5 seconds using > the experimental index. Can you kindly provide a measurement for the index at http://cheeseshop.python.org/simple/ as well? Thanks, Martin From mhammond at skippinet.com.au Mon Jul 23 03:01:05 2007 From: mhammond at skippinet.com.au (Mark Hammond) Date: Mon, 23 Jul 2007 11:01:05 +1000 Subject: [Distutils] distutils.util.get_platform() for Windows In-Reply-To: <0a4001c7c900$f1ba2640$090a0a0a@enfoldsystems.local> Message-ID: <0d4901c7ccc5$00447ae0$090a0a0a@enfoldsystems.local> I wrote: > Many of the distutils "commands" use > distutils.util.get_platform() as the > basis for file and directory names used to package up extensions. On > Windows, this returns the value of sys.platform. On all > (desktop) Windows versions, this currently returns 'win32'. > > This causes a problem when trying to create a 64bit version > of an extension. > For example, using bdist_msi, the pywin32 extensions end up > with a filename > of 'pywin32-211.win32-py2.5.msi' for both 32bit and 64bit > versions. This is not desirable for (hopefully) obvious > reasons. > > I'd like to propose that an (untested and against 2.5) patch > similar to the following be adopted in distutils: Thanks to all for the comments so far. It seems that there are no real objections, nor any real barriers to making this change. While Philip's notes regarding dependencies are well founded, I believe its clear that no dependencies are possible between 32 and 64bit versions for Windows, so there are no broader architectural requirements that would block us moving ahead with this. Tres rightly points out that the name of the OS should be embedded in the name of the package - so rather than including a new patch, I will simply propose the specific strings that should be returned. Specifically, I propose that distutils.util.get_platform return the following: * On 32bit Windows, continue to return 'win32'. * On 64bit windows, return a string of the format 'win64-{architecture}' A good value for 'architecture' isn't clear. Either 'AMD' or 'Itanium' appeals at first glance, but I'm a little concerned that (say) a "casual" user with a new Intel Core Duo processor will not know they should use something labelled as "AMD" (eg, "I explicitly asked for an Intel chip, not an AMD one"). An alternative could be be 'x64' or 'i64', but I'm not sure that casual user would be any better off, and with only a single letter distinguishing them, there is scope for confusion. On my final (mutant) hand, it seems that Itanium will be a historical footnote and demand for Itanium 64bit packages will be tiny (I've had a reasonable number of requests for x64 versions of pywin32, but zero for i64), so I doubt many packages will bother with Itanium. So, I'm leaning towards 'win64-x64' and 'win64-i64', with the expectation that the 'i64' variant will be rarely seen in the wild. Alternatively, 'win-x64' and 'win-i64' appear reasonable - it doesn't seem necessary that '64' appear twice in the name. Depending on the feedback (or even lack of, which I'll take as meaning there are no objections to my proposal), I'll create a patch, solicit review, and assuming no further objections, check the changes in to the trunk. All comments welcome! Cheers, Mark From dpeterson at enthought.com Mon Jul 23 05:59:00 2007 From: dpeterson at enthought.com (Dave Peterson) Date: Sun, 22 Jul 2007 22:59:00 -0500 Subject: [Distutils] distutils.util.get_platform() for Windows In-Reply-To: <0d4901c7ccc5$00447ae0$090a0a0a@enfoldsystems.local> References: <0d4901c7ccc5$00447ae0$090a0a0a@enfoldsystems.local> Message-ID: <46A42784.5030405@enthought.com> Mark Hammond wrote: > A good value for 'architecture' isn't clear. Either 'AMD' or 'Itanium' > appeals at first glance, but I'm a little concerned that (say) a "casual" > user with a new Intel Core Duo processor will not know they should use > something labelled as "AMD" (eg, "I explicitly asked for an Intel chip, not > an AMD one"). An alternative could be be 'x64' or 'i64', but I'm not sure > that casual user would be any better off, and with only a single letter > distinguishing them, there is scope for confusion. On my final (mutant) > hand, it seems that Itanium will be a historical footnote and demand for > Itanium 64bit packages will be tiny (I've had a reasonable number of > requests for x64 versions of pywin32, but zero for i64), so I doubt many > packages will bother with Itanium. > > So, I'm leaning towards 'win64-x64' and 'win64-i64', with the expectation > that the 'i64' variant will be rarely seen in the wild. Alternatively, > 'win-x64' and 'win-i64' appear reasonable - it doesn't seem necessary that > '64' appear twice in the name. > I could be mistaken, but I believe the standard abbreviations for these architectures are 'x86_64' (intel/amd chipset) and 'ia64' (itanium) -- see the first two paragraphs of the wikipedia article on Itanium for an example: http://en.wikipedia.org/wiki/Itanium So how about something like: win-x86_64 win-ia64 -- Dave From fdrake at gmail.com Mon Jul 23 06:56:48 2007 From: fdrake at gmail.com (Fred Drake) Date: Mon, 23 Jul 2007 00:56:48 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> Message-ID: <9cee7ab80707222156o2bae8a32pdaf7767f8c167918@mail.gmail.com> On 7/22/07, Phillip J. Eby wrote: > Setuptools has lots of features that are targeted at different > audiences. There are plenty of features targeted at the group you're > in, don't begrudge the other groups their features. :) Actually, I suspect this is a substantial contributor to setuptools being considered controversial: it encompasses to many different features. That certainly keeps me feeling unhappy about depending on it. -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From pje at telecommunity.com Mon Jul 23 07:41:44 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 23 Jul 2007 01:41:44 -0400 Subject: [Distutils] setuptools controversy (was Re: Prototype setuptools-specific PyPI index.) In-Reply-To: <9cee7ab80707222156o2bae8a32pdaf7767f8c167918@mail.gmail.co m> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <9cee7ab80707222156o2bae8a32pdaf7767f8c167918@mail.gmail.com> Message-ID: <20070723053949.EBC3E3A40A9@sparrow.telecommunity.com> At 12:56 AM 7/23/2007 -0400, Fred Drake wrote: >On 7/22/07, Phillip J. Eby wrote: >>Setuptools has lots of features that are targeted at different >>audiences. There are plenty of features targeted at the group you're >>in, don't begrudge the other groups their features. :) > >Actually, I suspect this is a substantial contributor to setuptools >being considered controversial: it encompasses to many different >features. That's precisely my point, though: "too many" always means "has features I don't use", even though the features one person uses are considered superfluous by other parties, who consider the first person's "superfluous" features essential. However, I'm entirely reconciled to it being controversial. It is in fact impossible to have a significant impact on *anything* without creating controversy, as anyone involved with the history of Zope should be well aware. ;-) Usually, people think they can get away with making "simpler" versions of Zope because they don't understand the full range of requirements which it is intended to meet. The situation with setuptools is much the same, except that sometimes now it's the Zope folks making the accusations of too many features, excess complexity, evil, etc. :) From jim at zope.com Mon Jul 23 12:59:44 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 23 Jul 2007 06:59:44 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A38DF5.6010701@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <46A384C9.8040404@v.loewis.de> <46A38DF5.6010701@v.loewis.de> Message-ID: On Jul 22, 2007, at 1:03 PM, Martin v. L?wis wrote: > Jim Fulton schrieb: >> On Jul 22, 2007, at 12:24 PM, Martin v. L?wis wrote: >>>>> If people do misspell a package name when invoking easy_install, >>>>> they get the feature that you consider of no value. >>>> >>>> That is not correct. Not all packages are in PyPI. Using a >>>> package that >>>> isn't in PyPI will trigger a fetch of that page. >>> >>> I don't understand. What page is fetched if the package is not in >>> PyPI? >> >> We have lots of packages that aren't in PyPI. Some of them aren't >> ready >> for PyPI or are not of general interest. Some are proprietary. > > Ah, ok. So I stand to my original statement (the one you classified > as incorrect): *If* I do misspell a package name, *then* setuptools > will correct the spelling if the index page is available. Your full original statement was: On Jul 21, 2007, at 3:08 PM, Martin v. L?wis wrote: > IIUC, it won't slow down setuptools, as setuptools looks at it only > if it cannot find the real package page due to a misspelling. So > as long as everything is spelled correctly, it should not provide > any slowdown. > > If people do misspell a package name when invoking easy_install, > they get the feature that you consider of no value. I was referring to the part about not slowing things down when people didn't misspell. But it looks like I was mistaken. It was my understanding that setuptools always checked index/ when it couldn't find index/package_name/, but as Phillip pointed out, if it finds a package via find links, it won't look at index/. Basic tests seem to confirm this. >>> Would people prefer if the index >>> was always correct (and perhaps somewhat slow), or would they prefer >>> instead that it is super-efficient (and somewhat out-of-date)? >> >> Where somewhat out of date could be a matter of seconds. > > And where somewhat slower could be "practically not noticable". I wasn't arguing about speed. I agree that when PyPI is working well, the difference between the speed of the dynamic page and the speed of a static page wouldn't be noticeable. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Mon Jul 23 13:08:50 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 23 Jul 2007 07:08:50 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> Message-ID: <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> On Jul 22, 2007, at 12:51 PM, Phillip J. Eby wrote: > At 09:09 AM 7/22/2007 -0400, Jim Fulton wrote: >> People should *not* misspell pages >> when using setuptools. They should certainly not use misspelled >> package names in requirements. > > People do all sorts of things they shouldn't. That doesn't stop > them blaming other people for their mistakes. > > It's said that a 10% improvement in ease-of-use can double a > product's users. Case sensitivity is a barrier to entry for new > users, and setuptools can't afford any additional entry barriers. I totally don't buy this in a case like this. People installing packages with setuptools are technical users. We expect them to write Python scripts. > A significant part of setuptools' audience includes people who are > new to Python, or at least new to installing or distributing Python > modules, and quite a lot of setuptools features are aimed squarely > at that audience. This happens to be one of them. I don't think that encouraging use of case insensitive names by people who are about start learning a language that uses case sensitive names is doing them any favors. >> In my strongly help opinion, allowing >> imprecise names in requirements and setuptools command if of negative >> value. > > I understand that perspective. But practicality beats purity, and > this is absolutely a "worse is better" type of situation. Obviously we disagree. > Setuptools has lots of features that are targeted at different > audiences. There are plenty of features targeted at the group > you're in, don't begrudge the other groups their features. :) I don't think you are helping them. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Mon Jul 23 13:36:45 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 23 Jul 2007 07:36:45 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A3A485.7060602@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A3A485.7060602@v.loewis.de> Message-ID: <617C738B-BDB4-4EDE-900E-64B50EFC2ED6@zope.com> On Jul 22, 2007, at 2:40 PM, Martin v. L?wis wrote: >> WRT zc.buildout, refreshing a buildout with just ZODB installed in it >> takes about 45 seconds for me using PyPI and about 5 seconds using >> the experimental index. > > Can you kindly provide a measurement for the index at > http://cheeseshop.python.org/simple/ as well? Yup. So, ATM: Using old PyPI takes about 1m5s Using simple takes about 25s Using ppix takes about 8s Some notes: - ZODB isn't the best example as it has download links to www.zope.org, making it take longer than packages without offsite links (relative to PyPI). - I expect that the difference between simple and ppix *for me* is a matter of geography. Refreshing an empty buildout checks the zc.buildout and setuptools packages. For that: Old PyPI takes 25s Simple takes 8s and ppix takes .5s Again, I assume that the difference between simple and ppix has more to do with geography than the difference between serving statically and dynamically. The simple page has more links on it than the ppix page, because I haven't gotten around to scarf all links off of a restructured-text rendering of long description. I doubt that makes any difference. It will be interesting to try again after I fix that. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Mon Jul 23 17:22:30 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 23 Jul 2007 11:22:30 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> Message-ID: <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> At 07:08 AM 7/23/2007 -0400, Jim Fulton wrote: >On Jul 22, 2007, at 12:51 PM, Phillip J. Eby wrote: >>At 09:09 AM 7/22/2007 -0400, Jim Fulton wrote: >>>People should *not* misspell pages >>>when using setuptools. They should certainly not use misspelled >>>package names in requirements. >> >>People do all sorts of things they shouldn't. That doesn't stop >>them blaming other people for their mistakes. >> >>It's said that a 10% improvement in ease-of-use can double a >>product's users. Case sensitivity is a barrier to entry for new >>users, and setuptools can't afford any additional entry barriers. > >I totally don't buy this in a case like this. People installing >packages with setuptools are technical users. We expect them to >write Python scripts. No, "we" don't. Eggs were created to support application-level plugins, such as are used by Trac and Chandler. Trac and Chandler users are not necessarily programmers, let alone Python programmers. From tseaver at palladion.com Mon Jul 23 18:01:02 2007 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 23 Jul 2007 12:01:02 -0400 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> Message-ID: <46A4D0BE.4030706@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Phillip J. Eby wrote: > At 07:08 AM 7/23/2007 -0400, Jim Fulton wrote: >> On Jul 22, 2007, at 12:51 PM, Phillip J. Eby wrote: >>> At 09:09 AM 7/22/2007 -0400, Jim Fulton wrote: >>>> People should *not* misspell pages >>>> when using setuptools. They should certainly not use misspelled >>>> package names in requirements. >>> People do all sorts of things they shouldn't. That doesn't stop >>> them blaming other people for their mistakes. >>> >>> It's said that a 10% improvement in ease-of-use can double a >>> product's users. Case sensitivity is a barrier to entry for new >>> users, and setuptools can't afford any additional entry barriers. >> I totally don't buy this in a case like this. People installing >> packages with setuptools are technical users. We expect them to >> write Python scripts. > > No, "we" don't. Eggs were created to support application-level > plugins, such as are used by Trac and Chandler. Trac and Chandler > users are not necessarily programmers, let alone Python programmers. But by definition, the people typing the names of the dependencies into a 'setup.py' for such a plugin *are* Python programmers, and could be expected to know about case sensitivity. I don't think Jim was areguing that human-centric *search* should punish misspellings, but rather that encouraging such sloppiness in other packages is a misfeature, especially if supporting it induces a tax on *all* users of automated dependency resolution. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGpNC++gerLs4ltQ4RAr2HAJ9UdPIVdz36inTG7nkm8SnrWPpcOgCgjKPc sOqbuwOhUvlsSYpgxFSz1mg= =F1EY -----END PGP SIGNATURE----- From tseaver at palladion.com Mon Jul 23 18:01:02 2007 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 23 Jul 2007 12:01:02 -0400 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> Message-ID: <46A4D0BE.4030706@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Phillip J. Eby wrote: > At 07:08 AM 7/23/2007 -0400, Jim Fulton wrote: >> On Jul 22, 2007, at 12:51 PM, Phillip J. Eby wrote: >>> At 09:09 AM 7/22/2007 -0400, Jim Fulton wrote: >>>> People should *not* misspell pages >>>> when using setuptools. They should certainly not use misspelled >>>> package names in requirements. >>> People do all sorts of things they shouldn't. That doesn't stop >>> them blaming other people for their mistakes. >>> >>> It's said that a 10% improvement in ease-of-use can double a >>> product's users. Case sensitivity is a barrier to entry for new >>> users, and setuptools can't afford any additional entry barriers. >> I totally don't buy this in a case like this. People installing >> packages with setuptools are technical users. We expect them to >> write Python scripts. > > No, "we" don't. Eggs were created to support application-level > plugins, such as are used by Trac and Chandler. Trac and Chandler > users are not necessarily programmers, let alone Python programmers. But by definition, the people typing the names of the dependencies into a 'setup.py' for such a plugin *are* Python programmers, and could be expected to know about case sensitivity. I don't think Jim was areguing that human-centric *search* should punish misspellings, but rather that encouraging such sloppiness in other packages is a misfeature, especially if supporting it induces a tax on *all* users of automated dependency resolution. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGpNC++gerLs4ltQ4RAr2HAJ9UdPIVdz36inTG7nkm8SnrWPpcOgCgjKPc sOqbuwOhUvlsSYpgxFSz1mg= =F1EY -----END PGP SIGNATURE----- From noah.gift at gmail.com Mon Jul 23 18:37:47 2007 From: noah.gift at gmail.com (Noah Gift) Date: Mon, 23 Jul 2007 12:37:47 -0400 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: <46A4D0BE.4030706@palladion.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> Message-ID: > > > But by definition, the people typing the names of the dependencies into > a 'setup.py' for such a plugin *are* Python programmers, and could be > expected to know about case sensitivity. > > I don't think Jim was areguing that human-centric *search* should punish > misspellings, but rather that encouraging such sloppiness in other > packages is a misfeature, especially if supporting it induces a tax on > *all* users of automated dependency resolution. > > In my humble opinion, I for one completely agree with Phillip. I have had to sit down with quite a few new Python Programmers and show them how to use easy_install and I "thank God" easy_install is smart enough to figure out case sensitivity. This is a wonderful feature!!!! Please don't ever get rid of it :) Not being able to install a package as they couldn't figure out the exact name of the package could be the final straw for some new programmer to Python! Noah Gift -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070723/9d7ebe75/attachment.html From barry at python.org Mon Jul 23 18:46:24 2007 From: barry at python.org (Barry Warsaw) Date: Mon, 23 Jul 2007 12:46:24 -0400 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: <46A4D0BE.4030706@palladion.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jul 23, 2007, at 12:01 PM, Tres Seaver wrote: >>>> It's said that a 10% improvement in ease-of-use can double a >>>> product's users. Under that principle, can I renew my plea for a better name than "easy_install"? - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRqTbYHEjvBPtnXfVAQIHmgP+L5eDz3n4mrcPk5K6NEexQPLrOT9iSd+w cFYhn+FL5QoK6snRfxFp25KFmdz/raKDeGpQ4ZIy3nhpZTqxeQpPCsAg84rrw0lQ lflPXkMMmZJTi+3JmjXc2mhj2SlHZ+73XxRPcD2NKnqr14sxlunJMPe4/IX+y1Rf 9C5WVwoCiJ0= =b+zs -----END PGP SIGNATURE----- From pje at telecommunity.com Mon Jul 23 19:40:08 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 23 Jul 2007 13:40:08 -0400 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> Message-ID: <20070723173750.CBD083A403D@sparrow.telecommunity.com> At 12:46 PM 7/23/2007 -0400, Barry Warsaw wrote: >Under that principle, can I renew my plea for a better name than >"easy_install"? Do you have a specific name in mind? From barry at python.org Mon Jul 23 20:38:52 2007 From: barry at python.org (Barry Warsaw) Date: Mon, 23 Jul 2007 14:38:52 -0400 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: <20070723173750.CBD083A403D@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <20070723173750.CBD083A403D@sparrow.telecommunity.com> Message-ID: <70977DD1-18B5-4359-B847-BD21A86D4372@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jul 23, 2007, at 1:40 PM, Phillip J. Eby wrote: > At 12:46 PM 7/23/2007 -0400, Barry Warsaw wrote: >> Under that principle, can I renew my plea for a better name than >> "easy_install"? > > Do you have a specific name in mind? I knew you were going to ask that. :) My first requirement is "something without an underscore". My second would be "something that's evocative of snakes and eggs". A couple of Pycons ago I think I suggested 'hatch' which IIRC you didn't like because you wanted to reserve that for some other function, but I've forgotten what that was. OTOH, a quick search didn't reveal any collisions with hatch(1) on *nix. lay(1) is probably not a good choice . egg(1) is probably the best choice IMO. It doesn't conflict with any other existing *nix command that I can tell, and I can't think of a better command for dealing with Python eggs. It's immediately evocative. Now that I think about it, maybe it was 'egg' that you wanted to reserve. I like 'egg' too because Ruby gems are managed with the gem(1) command so there's a parallel that is easily remembered. I'd further recommend that Python itself come with an 'egg(1)' command, which should be a shortcut for 'python setup.py'. To be honest, the latter is not as user friendly as 'egg sdist bdist_egg upload -s' but if the two use cases conflict too much, then egg(1) should be reserved for the end user because developers can swallow the slight inconvenience more easily. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRqT1vXEjvBPtnXfVAQLjbAP/av1ONvVJLULNQNqrrkXyqlEPFmbNHAqH piS2Db+8CyU2JCEa7cpqrv04S69kaTp6DdPggdIJbbqMNH4WQFUjP15HESky+4qH ExkjbxdKCnxC0U6UO9DLng6NkMwJpeddYkzf+/XlgunCxW2EjatMsqZkYpSH/Zov dBDdKCylT0s= =im3z -----END PGP SIGNATURE----- From tseaver at palladion.com Mon Jul 23 20:48:40 2007 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 23 Jul 2007 14:48:40 -0400 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> Message-ID: <46A4F808.4050406@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Noah Gift wrote: >> >> But by definition, the people typing the names of the dependencies into >> a 'setup.py' for such a plugin *are* Python programmers, and could be >> expected to know about case sensitivity. >> >> I don't think Jim was areguing that human-centric *search* should punish >> misspellings, but rather that encouraging such sloppiness in other >> packages is a misfeature, especially if supporting it induces a tax on >> *all* users of automated dependency resolution. >> >> > In my humble opinion, I for one completely agree with Phillip. I have had > to sit down with quite a few new Python Programmers and show them how to use > easy_install and I "thank God" easy_install is smart enough to figure out > case sensitivity. This is a wonderful feature!!!! Please don't ever get > rid of it :) > Not being able to install a package as they couldn't figure out the exact > name of the package could be the final straw for some new programmer to > Python! There are two different use cases here: 1. User mis-types the name of a package on the command line, e.g.: $ easy_install Foo when it should be spelled: $ easy_install foo Being forgiving of case-mangling here ia a concern of the easy_install *application*, and is non-controversil. 2. Programmer mis-types the name of a package in the dependencies for his own pacakge, e.g.: setup(install_requires=['Foo']...) In this case, coddling the error causes it to *propagate*, becuase other programmers will copy it directly, or depend on the error- filled package. Worse, the cost of error correction is transferred to *all* users of the setuptools library, even if they never use 'easy_install' at all. I'm fine with leaving the newbie-friendly behavior in 'easy_install'; I just don't like the performance hit it induces on users of setuptools who *can* spell. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGpPgI+gerLs4ltQ4RApzMAJ0WP6gzaM8n99fxkyo0Se285Te3bQCg1vxF 6ihYIENH8GpsQ7/ZF062T4Q= =OuxU -----END PGP SIGNATURE----- From tseaver at palladion.com Mon Jul 23 20:48:40 2007 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 23 Jul 2007 14:48:40 -0400 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> Message-ID: <46A4F808.4050406@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Noah Gift wrote: >> >> But by definition, the people typing the names of the dependencies into >> a 'setup.py' for such a plugin *are* Python programmers, and could be >> expected to know about case sensitivity. >> >> I don't think Jim was areguing that human-centric *search* should punish >> misspellings, but rather that encouraging such sloppiness in other >> packages is a misfeature, especially if supporting it induces a tax on >> *all* users of automated dependency resolution. >> >> > In my humble opinion, I for one completely agree with Phillip. I have had > to sit down with quite a few new Python Programmers and show them how to use > easy_install and I "thank God" easy_install is smart enough to figure out > case sensitivity. This is a wonderful feature!!!! Please don't ever get > rid of it :) > Not being able to install a package as they couldn't figure out the exact > name of the package could be the final straw for some new programmer to > Python! There are two different use cases here: 1. User mis-types the name of a package on the command line, e.g.: $ easy_install Foo when it should be spelled: $ easy_install foo Being forgiving of case-mangling here ia a concern of the easy_install *application*, and is non-controversil. 2. Programmer mis-types the name of a package in the dependencies for his own pacakge, e.g.: setup(install_requires=['Foo']...) In this case, coddling the error causes it to *propagate*, becuase other programmers will copy it directly, or depend on the error- filled package. Worse, the cost of error correction is transferred to *all* users of the setuptools library, even if they never use 'easy_install' at all. I'm fine with leaving the newbie-friendly behavior in 'easy_install'; I just don't like the performance hit it induces on users of setuptools who *can* spell. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGpPgI+gerLs4ltQ4RApzMAJ0WP6gzaM8n99fxkyo0Se285Te3bQCg1vxF 6ihYIENH8GpsQ7/ZF062T4Q= =OuxU -----END PGP SIGNATURE----- From noah.gift at gmail.com Mon Jul 23 20:53:12 2007 From: noah.gift at gmail.com (Noah Gift) Date: Mon, 23 Jul 2007 14:53:12 -0400 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: <70977DD1-18B5-4359-B847-BD21A86D4372@python.org> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <20070723173750.CBD083A403D@sparrow.telecommunity.com> <70977DD1-18B5-4359-B847-BD21A86D4372@python.org> Message-ID: > > > egg(1) is probably the best choice IMO. It doesn't conflict with any > other existing *nix command that I can tell, and I can't think of a > better command for dealing with Python eggs. It's immediately > evocative. Now that I think about it, maybe it was 'egg' that you > wanted to reserve. I like 'egg' too because Ruby gems are managed > with the gem(1) command so there's a parallel that is easily remembered. I agree. I really like egg! easy_install is a bit of a pain as tab completion with underscores is troublesome. 3 characters is even better.. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070723/16d363ad/attachment.html From benji at benjiyork.com Mon Jul 23 20:54:27 2007 From: benji at benjiyork.com (Benji York) Date: Mon, 23 Jul 2007 14:54:27 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A384C9.8040404@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <46A384C9.8040404@v.loewis.de> Message-ID: <46A4F963.3040609@benjiyork.com> Martin v. L?wis wrote: > would they prefer instead that it is super-efficient (and somewhat > out-of-date)? Yes. At most a few minutes out of date and faster/more reliable would be my strong preference. -- Benji York http://benjiyork.com From jim at zope.com Mon Jul 23 20:55:16 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 23 Jul 2007 14:55:16 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A4F808.4050406@palladion.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <46A4F808.4050406@palladion.com> Message-ID: <9FFADEB3-0E83-417E-B6EE-AF9A172690D0@zope.com> On Jul 23, 2007, at 2:48 PM, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Noah Gift wrote: >>> >>> But by definition, the people typing the names of the >>> dependencies into >>> a 'setup.py' for such a plugin *are* Python programmers, and >>> could be >>> expected to know about case sensitivity. >>> >>> I don't think Jim was areguing that human-centric *search* should >>> punish >>> misspellings, but rather that encouraging such sloppiness in other >>> packages is a misfeature, especially if supporting it induces a >>> tax on >>> *all* users of automated dependency resolution. >>> >>> >> In my humble opinion, I for one completely agree with Phillip. I >> have had >> to sit down with quite a few new Python Programmers and show them >> how to use >> easy_install and I "thank God" easy_install is smart enough to >> figure out >> case sensitivity. This is a wonderful feature!!!! Please don't >> ever get >> rid of it :) >> Not being able to install a package as they couldn't figure out >> the exact >> name of the package could be the final straw for some new >> programmer to >> Python! > > There are two different use cases here: > > 1. User mis-types the name of a package on the command line, e.g.: > > $ easy_install Foo > > when it should be spelled: > > $ easy_install foo > > Being forgiving of case-mangling here ia a concern of the > easy_install *application*, and is non-controversil. For me this is potentially controversial because: > 2. Programmer mis-types the name of a package in the dependencies > for his own pacakge, e.g.: > > setup(install_requires=['Foo']...) Note that this might be intentional, as opposed to a typo. The programmer will think "Foo" is a valid name because it worked with easy_install. It's true that easy_install prints a warning, but it is buried in so much output that it is easily missed or ignored. > In this case, coddling the error causes it to *propagate*, becuase > other programmers will copy it directly, or depend on the error- > filled package. Worse, the cost of error correction is > transferred > to *all* users of the setuptools library, even if they never use > 'easy_install' at all. Well said. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From benji at benjiyork.com Mon Jul 23 20:58:44 2007 From: benji at benjiyork.com (Benji York) Date: Mon, 23 Jul 2007 14:58:44 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A38DF5.6010701@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <46A384C9.8040404@v.loewis.de> <46A38DF5.6010701@v.loewis.de> Message-ID: <46A4FA64.5050404@benjiyork.com> Martin v. L?wis wrote: > And where somewhat slower could be "practically not noticable". Perhaps it /could/ be, but isn't currently. For example, updating one piece of software I have with almost 150 dependencies takes 45 seconds with ppix, 4:45 without. I plan to do similar timings with the "simple" PyPI interface when I get a chance and report the results here. -- Benji York http://benjiyork.com From jim at zope.com Mon Jul 23 21:06:46 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 23 Jul 2007 15:06:46 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A4FA64.5050404@benjiyork.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <46A384C9.8040404@v.loewis.de> <46A38DF5.6010701@v.loewis.de> <46A4FA64.5050404@benjiyork.com> Message-ID: <4B15F81D-3980-47FD-AC61-47F8E1EED20F@zope.com> On Jul 23, 2007, at 2:58 PM, Benji York wrote: > Martin v. L?wis wrote: >> And where somewhat slower could be "practically not noticable". > > Perhaps it /could/ be, but isn't currently. For example, updating > one piece of software I have with almost 150 dependencies takes 45 > seconds with ppix, 4:45 without. I plan to do similar timings with > the "simple" PyPI interface when I get a chance and report the > results here. I suspect that this has more to do with network distance than with server speed. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From noah.gift at gmail.com Mon Jul 23 21:30:44 2007 From: noah.gift at gmail.com (Noah Gift) Date: Mon, 23 Jul 2007 15:30:44 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <4B15F81D-3980-47FD-AC61-47F8E1EED20F@zope.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <46A384C9.8040404@v.loewis.de> <46A38DF5.6010701@v.loewis.de> <46A4FA64.5050404@benjiyork.com> <4B15F81D-3980-47FD-AC61-47F8E1EED20F@zope.com> Message-ID: On 7/23/07, Jim Fulton wrote: > > > On Jul 23, 2007, at 2:58 PM, Benji York wrote: > > > Martin v. L?wis wrote: > >> And where somewhat slower could be "practically not noticable". > > > > Perhaps it /could/ be, but isn't currently. For example, updating > > one piece of software I have with almost 150 dependencies takes 45 > > seconds with ppix, 4:45 without. I plan to do similar timings with > > the "simple" PyPI interface when I get a chance and report the > > results here. > > I suspect that this has more to do with network distance than with > server speed. That is an interesting point. It is amazing how many directory type things get slammed, but the problem is really latency...such as a slow DNS lookup. I wonder how much quicker an easy_install would be will local DNS lookups,package names, etc. I had a problem with a LDAP server I setup that was really tricky to figure out until I wrote some scripts that ran continuously getting stats, and I realized that a DNS server would hang occasionally and it would grind everything to a halt. People kept telling me they would have an occasional 'ls -l' that would hang for 20 seconds. Caching DNS servers fixed it. Jim > > -- > Jim Fulton mailto:jim at zope.com Python > Powered! > CTO (540) 361-1714 > http://www.python.org > Zope Corporation http://www.zope.com > http://www.zope.org > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- http://www.blog.noahgift.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070723/af718f06/attachment.htm From benji at benjiyork.com Mon Jul 23 21:41:05 2007 From: benji at benjiyork.com (Benji York) Date: Mon, 23 Jul 2007 15:41:05 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <4B15F81D-3980-47FD-AC61-47F8E1EED20F@zope.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <46A384C9.8040404@v.loewis.de> <46A38DF5.6010701@v.loewis.de> <46A4FA64.5050404@benjiyork.com> <4B15F81D-3980-47FD-AC61-47F8E1EED20F@zope.com> Message-ID: <46A50451.5050908@benjiyork.com> Jim Fulton wrote: > On Jul 23, 2007, at 2:58 PM, Benji York wrote: > >> Martin v. L?wis wrote: >>> And where somewhat slower could be "practically not noticable". >> Perhaps it /could/ be, but isn't currently. For example, updating >> one piece of software I have with almost 150 dependencies takes 45 >> seconds with ppix, 4:45 without. I plan to do similar timings with >> the "simple" PyPI interface when I get a chance and report the >> results here. > > I suspect that this has more to do with network distance than with > server speed. That's actually my point. Geographically distributed mirrors that are a little out of sync are much more valuable (IMO) than a centralized service that is absolutely up to date, but "far" away. For me the static/dynamic argument is more about stability, and central/distributed is more about (network) speed. -- Benji York http://benjiyork.com From benji at benjiyork.com Mon Jul 23 21:02:08 2007 From: benji at benjiyork.com (Benji York) Date: Mon, 23 Jul 2007 15:02:08 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> Message-ID: <46A4FB30.2000304@benjiyork.com> Jim Fulton wrote: > On Jul 22, 2007, at 12:51 PM, Phillip J. Eby wrote: >> A significant part of setuptools' audience includes people who are >> new to Python, or at least new to installing or distributing Python >> modules, and quite a lot of setuptools features are aimed squarely >> at that audience. This happens to be one of them. > > I don't think that encouraging use of case insensitive names by > people who are about start learning a language that uses case > sensitive names is doing them any favors. Agreed. -- Benji York http://benjiyork.com From benji at benjiyork.com Mon Jul 23 21:05:42 2007 From: benji at benjiyork.com (Benji York) Date: Mon, 23 Jul 2007 15:05:42 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> Message-ID: <46A4FC06.3010109@benjiyork.com> Noah Gift wrote: > In my humble opinion, I for one completely agree with Phillip. I have had to > sit down with quite a few new Python Programmers and show them how to use > easy_install and I "thank God" easy_install is smart enough to figure out case > sensitivity. This is a wonderful feature!!!! Please don't ever get rid of it :) If easy_install had instead said "sorry, I can't find 'foo', perhaps you meant 'Foo'", then the user would be both spared frustration and enlightened. -- Benji York http://benjiyork.com From shutterbug at dwx.com Mon Jul 23 04:49:10 2007 From: shutterbug at dwx.com (Marcus) Date: Sun, 22 Jul 2007 21:49:10 -0500 Subject: [Distutils] Setuptools - Removing Old Packages Message-ID: <46A41726.2010002@dwx.com> I'm using easy_install for managing automatic updates to an application. I include the "--multi-version", "--upgrade", and "--install-dir" options, so that only an egg is copied to the directory I specify. I see that the current recommendation is to manually remove old packages, since there isn't an uninstall feature of setuptools. How would I go about detecting obsolete eggs myself, so that I can remove them programatically after downloading the latest version of an egg? Thanks, Marcus From martin at v.loewis.de Mon Jul 23 22:00:12 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 23 Jul 2007 22:00:12 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <617C738B-BDB4-4EDE-900E-64B50EFC2ED6@zope.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A3A485.7060602@v.loewis.de> <617C738B-BDB4-4EDE-900E-64B50EFC2ED6@zope.com> Message-ID: <46A508CC.5010706@v.loewis.de> > Yup. So, ATM: > > Using old PyPI takes about 1m5s > Using simple takes about 25s > Using ppix takes about 8s Thanks! > Again, I assume that the difference between simple and ppix has more to > do with geography than the difference between serving statically and > dynamically. The simple page has more links on it than the ppix page, > because I haven't gotten around to scarf all links off of a > restructured-text rendering of long description. I doubt that makes any > difference. It will be interesting to try again after I fix that. If you think that the /simple pages are correct, it might be easier to just mirror them instead of doing all the work yourself. I don't plan to take that service offline, unless experimentation shows it has serious flaws. Regards, Martin From martin at v.loewis.de Mon Jul 23 22:04:36 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 23 Jul 2007 22:04:36 +0200 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: <46A4D0BE.4030706@palladion.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> Message-ID: <46A509D4.3070108@v.loewis.de> > But by definition, the people typing the names of the dependencies into > a 'setup.py' for such a plugin *are* Python programmers, and could be > expected to know about case sensitivity. > > I don't think Jim was areguing that human-centric *search* should punish > misspellings, but rather that encouraging such sloppiness in other > packages is a misfeature, especially if supporting it induces a tax on > *all* users of automated dependency resolution. Right. I think Phillip is primarily talking about package names as specified on the command line of easy_install. So if your concern is about package names specified in dependencies, one solution could be that setuptools distinguishes whether to apply case corrections and normalization, depending on whether it was an end-user-typed name or a programmer-specified one. What I don't know is how difficult that would be to implement, and what volunteer is supposed to implement it if it were easy/possible, so I by no means propose that such a solution should be implemented, even if it would solve the problem. Regards, Martin From rlratzel at enthought.com Mon Jul 23 22:04:18 2007 From: rlratzel at enthought.com (Rick Ratzel) Date: Mon, 23 Jul 2007 15:04:18 -0500 (CDT) Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A4FC06.3010109@benjiyork.com> (message from Benji York on Mon, 23 Jul 2007 15:05:42 -0400) References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <46A4FC06.3010109@benjiyork.com> Message-ID: <20070723200419.058451DF4F6@mail.enthought.com> > Date: Mon, 23 Jul 2007 15:05:42 -0400 > From: Benji York > > Noah Gift wrote: > > In my humble opinion, I for one completely agree with Phillip. I have had to > > sit down with quite a few new Python Programmers and show them how to use > > easy_install and I "thank God" easy_install is smart enough to figure out case > > sensitivity. This is a wonderful feature!!!! Please don't ever get rid of it :) > > If easy_install had instead said "sorry, I can't find 'foo', perhaps you > meant 'Foo'", then the user would be both spared frustration and > enlightened. +1 -- Rick Ratzel - Enthought, Inc. 515 Congress Avenue, Suite 2100 - Austin, Texas 78701 512-536-1057 x229 - Fax: 512-536-1059 http://www.enthought.com From tseaver at palladion.com Mon Jul 23 22:05:23 2007 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 23 Jul 2007 16:05:23 -0400 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: <70977DD1-18B5-4359-B847-BD21A86D4372@python.org> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <20070723173750.CBD083A403D@sparrow.telecommunity.com> <70977DD1-18B5-4359-B847-BD21A86D4372@python.org> Message-ID: <46A50A03.9050008@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Barry Warsaw wrote: > On Jul 23, 2007, at 1:40 PM, Phillip J. Eby wrote: > >> At 12:46 PM 7/23/2007 -0400, Barry Warsaw wrote: >>> Under that principle, can I renew my plea for a better name than >>> "easy_install"? >> Do you have a specific name in mind? > > I knew you were going to ask that. :) My first requirement is > "something without an underscore". Why? That has to be a pure "de gustibus" argument AFAICT. > My second would be "something > that's evocative of snakes and eggs". You prefer cute, rather than self-explanatory? > A couple of Pycons ago I think I suggested 'hatch' which IIRC you > didn't like because you wanted to reserve that for some other > function, but I've forgotten what that was. OTOH, a quick search > didn't reveal any collisions with hatch(1) on *nix. > > lay(1) is probably not a good choice . > > egg(1) is probably the best choice IMO. It doesn't conflict with any > other existing *nix command that I can tell, and I can't think of a > better command for dealing with Python eggs. It's immediately > evocative. Now that I think about it, maybe it was 'egg' that you > wanted to reserve. I like 'egg' too because Ruby gems are managed > with the gem(1) command so there's a parallel that is easily remembered. > > I'd further recommend that Python itself come with an 'egg(1)' > command, which should be a shortcut for 'python setup.py'. To be > honest, the latter is not as user friendly as 'egg sdist bdist_egg > upload -s' but if the two use cases conflict too much, then egg(1) > should be reserved for the end user because developers can swallow > the slight inconvenience more easily. Why would a *command* invoke setup.py? I don't see the point: peopple who want that kind of convenience can write two-liner shell scripts, spelled the way they like, or even put an alias in their shell profile. The point of 'easy_install' is that it has been wired to a *specific* python installation, and knows how to fetch and install distributions into it. It doesn't even make sense to *use* easy_install if you are already in the unpacked source distribution; the intersection of the two sets of use cases is empty. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGpQoD+gerLs4ltQ4RAp1UAJ9xfqZtK62Y3LgDYuhsm3XJHRujgACeO8ta v8/OOznC6xWEbD5KzzHuG6M= =tRSS -----END PGP SIGNATURE----- From tseaver at palladion.com Mon Jul 23 22:05:23 2007 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 23 Jul 2007 16:05:23 -0400 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: <70977DD1-18B5-4359-B847-BD21A86D4372@python.org> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <20070723173750.CBD083A403D@sparrow.telecommunity.com> <70977DD1-18B5-4359-B847-BD21A86D4372@python.org> Message-ID: <46A50A03.9050008@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Barry Warsaw wrote: > On Jul 23, 2007, at 1:40 PM, Phillip J. Eby wrote: > >> At 12:46 PM 7/23/2007 -0400, Barry Warsaw wrote: >>> Under that principle, can I renew my plea for a better name than >>> "easy_install"? >> Do you have a specific name in mind? > > I knew you were going to ask that. :) My first requirement is > "something without an underscore". Why? That has to be a pure "de gustibus" argument AFAICT. > My second would be "something > that's evocative of snakes and eggs". You prefer cute, rather than self-explanatory? > A couple of Pycons ago I think I suggested 'hatch' which IIRC you > didn't like because you wanted to reserve that for some other > function, but I've forgotten what that was. OTOH, a quick search > didn't reveal any collisions with hatch(1) on *nix. > > lay(1) is probably not a good choice . > > egg(1) is probably the best choice IMO. It doesn't conflict with any > other existing *nix command that I can tell, and I can't think of a > better command for dealing with Python eggs. It's immediately > evocative. Now that I think about it, maybe it was 'egg' that you > wanted to reserve. I like 'egg' too because Ruby gems are managed > with the gem(1) command so there's a parallel that is easily remembered. > > I'd further recommend that Python itself come with an 'egg(1)' > command, which should be a shortcut for 'python setup.py'. To be > honest, the latter is not as user friendly as 'egg sdist bdist_egg > upload -s' but if the two use cases conflict too much, then egg(1) > should be reserved for the end user because developers can swallow > the slight inconvenience more easily. Why would a *command* invoke setup.py? I don't see the point: peopple who want that kind of convenience can write two-liner shell scripts, spelled the way they like, or even put an alias in their shell profile. The point of 'easy_install' is that it has been wired to a *specific* python installation, and knows how to fetch and install distributions into it. It doesn't even make sense to *use* easy_install if you are already in the unpacked source distribution; the intersection of the two sets of use cases is empty. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGpQoD+gerLs4ltQ4RAp1UAJ9xfqZtK62Y3LgDYuhsm3XJHRujgACeO8ta v8/OOznC6xWEbD5KzzHuG6M= =tRSS -----END PGP SIGNATURE----- From jim at zope.com Mon Jul 23 22:05:53 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 23 Jul 2007 16:05:53 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A508CC.5010706@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A3A485.7060602@v.loewis.de> <617C738B-BDB4-4EDE-900E-64B50EFC2ED6@zope.com> <46A508CC.5010706@v.loewis.de> Message-ID: On Jul 23, 2007, at 4:00 PM, Martin v. L?wis wrote: ... >> Again, I assume that the difference between simple and ppix has >> more to >> do with geography than the difference between serving statically and >> dynamically. The simple page has more links on it than the ppix page, >> because I haven't gotten around to scarf all links off of a >> restructured-text rendering of long description. I doubt that >> makes any >> difference. It will be interesting to try again after I fix that. > > If you think that the /simple pages are correct, it might be easier to > just mirror them instead of doing all the work yourself. Good point. I might just do that. > I don't plan to take that service offline, unless experimentation > shows it has serious flaws. Cool. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From martin at v.loewis.de Mon Jul 23 22:16:27 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 23 Jul 2007 22:16:27 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A4FA64.5050404@benjiyork.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <46A384C9.8040404@v.loewis.de> <46A38DF5.6010701@v.loewis.de> <46A4FA64.5050404@benjiyork.com> Message-ID: <46A50C9B.7060501@v.loewis.de> > Perhaps it /could/ be, but isn't currently. For example, updating one > piece of software I have with almost 150 dependencies takes 45 seconds > with ppix, 4:45 without. I plan to do similar timings with the "simple" > PyPI interface when I get a chance and report the results here. I was, of course, talking about the simple interface. The full index will certainly take much more time because setuptools has to request more pages, and each page contains a lot of unnecessary data. Regards, Martin From martin at v.loewis.de Mon Jul 23 22:21:05 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 23 Jul 2007 22:21:05 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A50451.5050908@benjiyork.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <46A384C9.8040404@v.loewis.de> <46A38DF5.6010701@v.loewis.de> <46A4FA64.5050404@benjiyork.com> <4B15F81D-3980-47FD-AC61-47F8E1EED20F@zope.com> <46A50451.5050908@benjiyork.com> Message-ID: <46A50DB1.3080207@v.loewis.de> >>>> And where somewhat slower could be "practically not noticable". >>> Perhaps it /could/ be, but isn't currently. For example, updating >>> one piece of software I have with almost 150 dependencies takes 45 >>> seconds with ppix, 4:45 without. I plan to do similar timings with >>> the "simple" PyPI interface when I get a chance and report the >>> results here. >> >> I suspect that this has more to do with network distance than with >> server speed. > > That's actually my point. Geographically distributed mirrors that are a > little out of sync are much more valuable (IMO) than a centralized > service that is absolutely up to date, but "far" away. Ok, but then your response didn't really answer my question. If people want to run distributed mirrors that are somewhat behind, by all means: start today (just remember to talk to me if you also want to mirror files - if not, just run Jim's software as-is). My question was about the "simple" interface on the central server, to which you seem to say "I don't need it at all - whether it's current and slow or behind and fast" (which, in a sense, is also a response to the question, namely "I don't care"). Regards, Martin From constant.beta at gmail.com Mon Jul 23 22:25:09 2007 From: constant.beta at gmail.com (=?ISO-8859-2?Q?Micha=B3_Kwiatkowski?=) Date: Mon, 23 Jul 2007 22:25:09 +0200 Subject: [Distutils] Setuptools - Removing Old Packages In-Reply-To: <46A41726.2010002@dwx.com> References: <46A41726.2010002@dwx.com> Message-ID: <5e8b0f6b0707231325r601be034le79928409899f586@mail.gmail.com> On 7/23/07, Marcus wrote: > I'm using easy_install for managing automatic updates to an application. > I include the "--multi-version", "--upgrade", and "--install-dir" > options, so that only an egg is copied to the directory I specify. > > I see that the current recommendation is to manually remove old > packages, since there isn't an uninstall feature of setuptools. How > would I go about detecting obsolete eggs myself, so that I can remove > them programatically after downloading the latest version of an egg? Check out yolk (http://cheeseshop.python.org/pypi/yolk). Cheers, mk From barry at python.org Mon Jul 23 22:37:34 2007 From: barry at python.org (Barry Warsaw) Date: Mon, 23 Jul 2007 16:37:34 -0400 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: <46A50A03.9050008@palladion.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <20070723173750.CBD083A403D@sparrow.telecommunity.com> <70977DD1-18B5-4359-B847-BD21A86D4372@python.org> <46A50A03.9050008@palladion.com> Message-ID: <01933B2A-4BB0-4C10-AF67-740D36BB53C0@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jul 23, 2007, at 4:05 PM, Tres Seaver wrote: > Barry Warsaw wrote: >> On Jul 23, 2007, at 1:40 PM, Phillip J. Eby wrote: >> >>> At 12:46 PM 7/23/2007 -0400, Barry Warsaw wrote: >>>> Under that principle, can I renew my plea for a better name than >>>> "easy_install"? >>> Do you have a specific name in mind? >> >> I knew you were going to ask that. :) My first requirement is >> "something without an underscore". > > Why? That has to be a pure "de gustibus" argument AFAICT. It's not. Few *nix commands use underscores, and for good reason. It's user unfriendly because it's on an uncommon keycap you have to search for *and* you have to shift to it. And by "you" I don't mean you and me (i.e. programmers) who type the silly things all day, I mean "users". Sometimes hyphens are used (e.g. 'apt-get') which is only slightly better but those are still rare. And this recommendation says (to me) that utility names should be between 2 and 9 characters and be composed of lower case letters and digits: http://www.opengroup.org/onlinepubs/009695399/basedefs/ xbd_chap12.html#tag_12_02 >> My second would be "something >> that's evocative of snakes and eggs". > > You prefer cute, rather than self-explanatory? I prefer something memorable and easy to type. BTW, it's only a claim that 'easy_install' is actually 'easy' -- a claim I happen to agree with, mostly, but it's more hopefully self-explanatory than objectively so. >> I'd further recommend that Python itself come with an 'egg(1)' >> command, which should be a shortcut for 'python setup.py'. To be >> honest, the latter is not as user friendly as 'egg sdist bdist_egg >> upload -s' but if the two use cases conflict too much, then egg(1) >> should be reserved for the end user because developers can swallow >> the slight inconvenience more easily. > > Why would a *command* invoke setup.py? I don't see the point: > peopple > who want that kind of convenience can write two-liner shell scripts, > spelled the way they like, or even put an alias in their shell > profile. Of course, and developers who dislike the verbosity are well equipped to do any kind of aliasing they want. I'm not going to belabor this point because it's not the use case I /really/ care about and I don't want to sidetrack the discussion about coming up with a better command than 'easy_install' for users. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRqURj3EjvBPtnXfVAQIFrgP/XbBgGxLLgKyIMCVeyXLsRArL2IrocLKn Uu5lykvaunN5Q6UgqJM9JzOvHldbd2Igs0hugNnGXOWOzEUUJITASgPkcetTmwAs 1eot2oa6ZAbHLo1bgntBPW9HLAK7mZES7/Py8VwSnksubCX03aCu4mR4WnCCfM+H ssFW37aEwDU= =M/XJ -----END PGP SIGNATURE----- From pje at telecommunity.com Mon Jul 23 22:47:04 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 23 Jul 2007 16:47:04 -0400 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: <46A509D4.3070108@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <46A509D4.3070108@v.loewis.de> Message-ID: <20070723204446.294223A40B2@sparrow.telecommunity.com> At 10:04 PM 7/23/2007 +0200, Martin v. L?wis wrote: > > But by definition, the people typing the names of the dependencies into > > a 'setup.py' for such a plugin *are* Python programmers, and could be > > expected to know about case sensitivity. > > > > I don't think Jim was areguing that human-centric *search* should punish > > misspellings, but rather that encouraging such sloppiness in other > > packages is a misfeature, especially if supporting it induces a tax on > > *all* users of automated dependency resolution. > >Right. I think Phillip is primarily talking about package names as >specified on the command line of easy_install. > >So if your concern is about package names specified in dependencies, >one solution could be that setuptools distinguishes whether to apply >case corrections and normalization, depending on whether it was an >end-user-typed name or a programmer-specified one. > >What I don't know is how difficult that would be to implement, and >what volunteer is supposed to implement it if it were easy/possible, >so I by no means propose that such a solution should be implemented, >even if it would solve the problem. Yes, especially since compatibility with the existing installation base requires case insensitivity, because on case-insensitive platforms easy_install already normalizes the case of filenames it creates. So, the question of what the "right thing" to do is in the abstract has already been moot for a year or two. From pje at telecommunity.com Mon Jul 23 22:48:40 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 23 Jul 2007 16:48:40 -0400 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: <46A50A03.9050008@palladion.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <20070723173750.CBD083A403D@sparrow.telecommunity.com> <70977DD1-18B5-4359-B847-BD21A86D4372@python.org> <46A50A03.9050008@palladion.com> Message-ID: <20070723204620.931993A40AA@sparrow.telecommunity.com> At 04:05 PM 7/23/2007 -0400, Tres Seaver wrote: >It doesn't even make sense to *use* easy_install if you are >already in the unpacked source distribution; the intersection of the >two sets of use cases is empty. Not true, actually. "setup.py install" doesn't give you the same flexibility of options as "easy_install .", since the latter lets you specify options like whether or not to unzip the resulting package, and whether to include dependencies or not. From lars at ibp.de Mon Jul 23 22:17:05 2007 From: lars at ibp.de (Lars Immisch) Date: Mon, 23 Jul 2007 22:17:05 +0200 Subject: [Distutils] Prototype setuptools-specific PyPI index. In-Reply-To: <70977DD1-18B5-4359-B847-BD21A86D4372@python.org> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <20070723173750.CBD083A403D@sparrow.telecommunity.com> <70977DD1-18B5-4359-B847-BD21A86D4372@python.org> Message-ID: <46A50CC1.7000304@ibp.de> > A couple of Pycons ago I think I suggested 'hatch' which IIRC you > didn't like because you wanted to reserve that for some other > function, but I've forgotten what that was. OTOH, a quick search > didn't reveal any collisions with hatch(1) on *nix. +1 on a rename. Both 'hatch' and 'egg' are excellent. I'm not fond of the name 'easy_install', because: - it is very generic - the bias in the name (easy) will backfire whenever it fails for any reason. Like: 'Hey, this *bleep* thing thinks its easy to use, but it doesn't *bleep* work'. - Lars From fdrake at gmail.com Mon Jul 23 23:08:11 2007 From: fdrake at gmail.com (Fred Drake) Date: Mon, 23 Jul 2007 17:08:11 -0400 Subject: [Distutils] setuptools controversy (was Re: Prototype setuptools-specific PyPI index.) In-Reply-To: <20070723053949.EBC3E3A40A9@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <9cee7ab80707222156o2bae8a32pdaf7767f8c167918@mail.gmail.com> <20070723053949.EBC3E3A40A9@sparrow.telecommunity.com> Message-ID: <9cee7ab80707231408t24c00125tee04af510724eaa3@mail.gmail.com> On 7/23/07, Phillip J. Eby wrote: > However, I'm entirely reconciled to it being controversial. It is in > fact impossible to have a significant impact on *anything* without > creating controversy, as anyone involved with the history of Zope > should be well aware. ;-) Our level of expertise in that is certainly substantial. :-) > Usually, people think they can get away with making "simpler" > versions of Zope because they don't understand the full range of > requirements which it is intended to meet. The situation with > setuptools is much the same, except that sometimes now it's the Zope > folks making the accusations of too many features, excess complexity, > evil, etc. :) I don't think I made an accusation myself. My concern is fed by my belief that setuptools tries to improve on distutils in many different ways, some of which seem better suited to separate tools. I've no problem with you disagreeing with me on this. :-) -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From martin at v.loewis.de Mon Jul 23 23:13:48 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 23 Jul 2007 23:13:48 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <20070723204446.294223A40B2@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <46A509D4.3070108@v.loewis.de> <20070723204446.294223A40B2@sparrow.telecommunity.com> Message-ID: <46A51A0C.2090800@v.loewis.de> > Yes, especially since compatibility with the existing installation > base requires case insensitivity, because on case-insensitive > platforms easy_install already normalizes the case of filenames it > creates. So, the question of what the "right thing" to do is in the > abstract has already been moot for a year or two. Can you elaborate a bit, please? Why does the case of filenames matter for the queries it makes? AFAIU, it gets package names either from the user or from setup.py, perhaps also from packages dependency inside .egg files (assuming those support dependencies); these should all be case-sensitive. Regards, Martin From pje at telecommunity.com Mon Jul 23 23:21:16 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 23 Jul 2007 17:21:16 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A51A0C.2090800@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <46A509D4.3070108@v.loewis.de> <20070723204446.294223A40B2@sparrow.telecommunity.com> <46A51A0C.2090800@v.loewis.de> Message-ID: <20070723211919.361E23A403D@sparrow.telecommunity.com> At 11:13 PM 7/23/2007 +0200, Martin v. L?wis wrote: > > Yes, especially since compatibility with the existing installation > > base requires case insensitivity, because on case-insensitive > > platforms easy_install already normalizes the case of filenames it > > creates. So, the question of what the "right thing" to do is in the > > abstract has already been moot for a year or two. > >Can you elaborate a bit, please? Why does the case of filenames >matter for the queries it makes? > >AFAIU, it gets package names either from the user or from setup.py, >perhaps also from packages dependency inside .egg files (assuming >those support dependencies); these should all be case-sensitive. In order to resolve dependencies, the system looks at installed .egg files and directories (and .egg-info direcories), and extracts package name and version info from the filenames. From benji at benjiyork.com Mon Jul 23 23:34:16 2007 From: benji at benjiyork.com (Benji York) Date: Mon, 23 Jul 2007 17:34:16 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A50DB1.3080207@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <46A384C9.8040404@v.loewis.de> <46A38DF5.6010701@v.loewis.de> <46A4FA64.5050404@benjiyork.com> <4B15F81D-3980-47FD-AC61-47F8E1EED20F@zope.com> <46A50451.5050908@benjiyork.com> <46A50DB1.3080207@v.loewis.de> Message-ID: <46A51ED8.4020502@benjiyork.com> Martin v. L?wis wrote: > My question was about the "simple" interface on the central > server Ah, I didn't realize. > to which you seem to say "I don't need it at all - whether > it's current and slow or behind and fast" (which, in a sense, > is also a response to the question, namely "I don't care"). I think it's a great idea to have both human- and machine-targeted versions available. It looks like setuptools is about twice as fast (in at least one instance) with the simple version. That seems like a pretty big win to me. -- Benji York http://benjiyork.com From jim at zope.com Mon Jul 23 23:56:37 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 23 Jul 2007 17:56:37 -0400 Subject: [Distutils] setuptools controversy (was Re: Prototype setuptools-specific PyPI index.) In-Reply-To: <9cee7ab80707231408t24c00125tee04af510724eaa3@mail.gmail.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <9cee7ab80707222156o2bae8a32pdaf7767f8c167918@mail.gmail.com> <20070723053949.EBC3E3A40A9@sparrow.telecommunity.com> <9cee7ab80707231408t24c00125tee04af510724eaa3@mail.gmail.com> Message-ID: On Jul 23, 2007, at 5:08 PM, Fred Drake wrote: > On 7/23/07, Phillip J. Eby wrote: ... >> Usually, people think they can get away with making "simpler" >> versions of Zope because they don't understand the full range of >> requirements which it is intended to meet. The situation with >> setuptools is much the same, except that sometimes now it's the Zope >> folks making the accusations of too many features, excess complexity, >> evil, etc. :) I can't find your message, so I'm responding via Fred's. I hope none of the "Zope folks" have said setuptools is evil. We're big fans and are adopting it whole heartedly. We want it to succeed. Speaking for myself, I think it would benefit from being simpler. It is a lot more complex that I suspect you realize. Of course, there are things I might do differently. But it is a great accomplishment. You should be very proud of it and we are very grateful. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From fuzzyman at voidspace.org.uk Tue Jul 24 00:01:36 2007 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 23 Jul 2007 23:01:36 +0100 Subject: [Distutils] setuptools controversy (was Re: Prototype setuptools-specific PyPI index.) In-Reply-To: References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <9cee7ab80707222156o2bae8a32pdaf7767f8c167918@mail.gmail.com> <20070723053949.EBC3E3A40A9@sparrow.telecommunity.com> <9cee7ab80707231408t24c00125tee04af510724eaa3@mail.gmail.com> Message-ID: <46A52540.1000308@voidspace.org.uk> Jim Fulton wrote: > On Jul 23, 2007, at 5:08 PM, Fred Drake wrote: > [snip...] > But it is a great accomplishment. > You should be very proud of it and we are very grateful. > > +1 Michael Foord http://www.voidspace.org.uk/python/articles.shtml From mhammond at skippinet.com.au Tue Jul 24 00:12:04 2007 From: mhammond at skippinet.com.au (Mark Hammond) Date: Tue, 24 Jul 2007 08:12:04 +1000 Subject: [Distutils] distutils.util.get_platform() for Windows In-Reply-To: <46A42784.5030405@enthought.com> Message-ID: <0df001c7cd76$80d70a40$090a0a0a@enfoldsystems.local> Dave writes: > I could be mistaken, but I believe the standard abbreviations > for these > architectures are 'x86_64' (intel/amd chipset) and 'ia64' > (itanium) -- > see the first two paragraphs of the wikipedia article on > Itanium for an > example: > http://en.wikipedia.org/wiki/Itanium > > So how about something like: > win-x86_64 > win-ia64 Good point - but I'm not that keen on 'x86_64' - it seems a little too long, especially as we will not be using 'x86_32', and the extra characters don't seem like they will help resolve confusion (ie, if someone doesn't know what architecture they are using, the extra '86' isn't going to help). http://en.wikipedia.org/wiki/X86-64 notes that 'x64' is a common name, so how does 'win-x64' and 'win-ia64' sound as a compromise? I'm happy to let any other informal "votes" make a final decision though... Cheers, Mark From pje at telecommunity.com Tue Jul 24 00:51:04 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 23 Jul 2007 18:51:04 -0400 Subject: [Distutils] setuptools controversy (was Re: Prototype setuptools-specific PyPI index.) In-Reply-To: References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <9cee7ab80707222156o2bae8a32pdaf7767f8c167918@mail.gmail.com> <20070723053949.EBC3E3A40A9@sparrow.telecommunity.com> <9cee7ab80707231408t24c00125tee04af510724eaa3@mail.gmail.com> Message-ID: <20070723224845.C2C173A403D@sparrow.telecommunity.com> At 05:56 PM 7/23/2007 -0400, Jim Fulton wrote: >On Jul 23, 2007, at 5:08 PM, Fred Drake wrote: > >>On 7/23/07, Phillip J. Eby wrote: >... >>>Usually, people think they can get away with making "simpler" >>>versions of Zope because they don't understand the full range of >>>requirements which it is intended to meet. The situation with >>>setuptools is much the same, except that sometimes now it's the Zope >>>folks making the accusations of too many features, excess complexity, >>>evil, etc. :) > >I can't find your message, so I'm responding via Fred's. > >I hope none of the "Zope folks" have said setuptools is evil. I was referring to your comments regarding case-insensitivity being considered harmful. >Speaking for myself, I think it would benefit from being simpler. Good lord yes, and *I* of course would have benefited more than most, had that been possible. (Since I would've needed to do a lot less work.) >It is a lot more complex that I suspect you realize. Right back at you. :) Except that in my case, the complexity I'm aware of is in the complexity of the requirements that had to be met to reach the goal of Total World Domination. If those requirements hadn't been met at the start, it would never have gotten so far as to allow for the luxury of discussing whether perhaps some of those requirements could have been overlooked or handled differently. (E.g., the recent progress on setuptools-specific indexing.) >Of course, there are things I might do differently. Again, right back at you. :) If I had the luxury of being able to build up some kind of community consensus to use whatever was "pure" (as opposed to practical), I most likely would've done a lot of things differently. I certainly don't mind disagreement, in any case. It's just that somebody's got to weigh the cost and benefits of a decision, viewed over the entire scope of setuptools' many audiences, which sometimes means I'm going to Pronounce against something even when I agree with your view (within the scope of your perspective). Other times, simple backward-compatibility or other practical issues will rule out changing things that even I might myself consider to be a mistake in retrospect. From pje at telecommunity.com Tue Jul 24 00:52:53 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 23 Jul 2007 18:52:53 -0400 Subject: [Distutils] Where in setuptools is the setup.cfg install optimize=1 option processed? In-Reply-To: <63913.207.188.248.157.1185229400.squirrel@www.cpcug.org> References: <63913.207.188.248.157.1185229400.squirrel@www.cpcug.org> Message-ID: <20070723225034.2E0813A403D@sparrow.telecommunity.com> At 06:23 PM 7/23/2007 -0400, Stanley A. Klein wrote: >I've been able to generate Fedora rpms for several Python packages since >you provided the info on the setup.cfg [install] optimize=1 option when >running bdist_rpm. > >I'm now having the same problem trying to generate rpms for particular >packages in the Enthought system that use some special setuptools in numpy >that import the regular setuptools and make changes. I tried the >setup.cfg [install] optimize=1 option and it appeared to have no effect >(i.e., I got the old error). > >Where in setuptools is this option processed? I would like to find the >place in the numpy-modified setuptools I need to fix to get the same >processing as the regular setuptools. It's not a setuptools option, it's a distutils option. It's applied when the spec file generated by bdist_rpm runs "setup.py install" to copy the files to the target directory. From pje at telecommunity.com Tue Jul 24 00:53:58 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 23 Jul 2007 18:53:58 -0400 Subject: [Distutils] distutils.util.get_platform() for Windows In-Reply-To: <0df001c7cd76$80d70a40$090a0a0a@enfoldsystems.local> References: <46A42784.5030405@enthought.com> <0df001c7cd76$80d70a40$090a0a0a@enfoldsystems.local> Message-ID: <20070723225138.28F0A3A403D@sparrow.telecommunity.com> At 08:12 AM 7/24/2007 +1000, Mark Hammond wrote: >Dave writes: > > > I could be mistaken, but I believe the standard abbreviations > > for these > > architectures are 'x86_64' (intel/amd chipset) and 'ia64' > > (itanium) -- > > see the first two paragraphs of the wikipedia article on > > Itanium for an > > example: > > http://en.wikipedia.org/wiki/Itanium > > > > So how about something like: > > win-x86_64 > > win-ia64 > >Good point - but I'm not that keen on 'x86_64' - it seems a little too long, >especially as we will not be using 'x86_32', and the extra characters don't >seem like they will help resolve confusion (ie, if someone doesn't know what >architecture they are using, the extra '86' isn't going to help). > >http://en.wikipedia.org/wiki/X86-64 notes that 'x64' is a common name, so >how does 'win-x64' and 'win-ia64' sound as a compromise? I'm happy to let >any other informal "votes" make a final decision though... win-x64 and win-ia64 sound fine to me, for whatever that's worth. :) From bwinton at latte.ca Tue Jul 24 01:45:24 2007 From: bwinton at latte.ca (Blake Winton) Date: Mon, 23 Jul 2007 19:45:24 -0400 Subject: [Distutils] distutils.util.get_platform() for Windows In-Reply-To: <20070723225138.28F0A3A403D@sparrow.telecommunity.com> References: <46A42784.5030405@enthought.com> <0df001c7cd76$80d70a40$090a0a0a@enfoldsystems.local> <20070723225138.28F0A3A403D@sparrow.telecommunity.com> Message-ID: <46A53D94.8020702@latte.ca> Phillip J. Eby wrote: >>> So how about something like: >>> win-x86_64 >>> win-ia64 >> http://en.wikipedia.org/wiki/X86-64 notes that 'x64' is a common name, so >> how does 'win-x64' and 'win-ia64' sound as a compromise? I'm happy to let >> any other informal "votes" make a final decision though... > win-x64 and win-ia64 sound fine to me, for whatever that's worth. :) Along with win-x32? (I suggest win32-x86, win64-x86 and win64-ia, but as I don't plan on using more than one of them, I don't particularly care. win32, (or was that win-32?,) win-x64 and win-ia64 seem fine, if that's what gets chosen.) Later, Blake. From JVinke at nassco.com Tue Jul 24 01:42:24 2007 From: JVinke at nassco.com (Vinke, Johan) Date: Mon, 23 Jul 2007 16:42:24 -0700 Subject: [Distutils] Problem with bdist_wininst option (trying to distribute a COM server, on Windows XP) Message-ID: <19FDA17B69C10C46AD6BE50906D05DC9026BA77B@EMAIL1.nassco.local> Preface: If this ends up being a real "Newbie" question, my apologies to all. But I have read "Distributing Python Modules" by Greg Ward, May 19, 2004, particularly Chapter 6 - Creating Built Distributions - and 6.3 - Creating Windows Installer. And I still can't see what I'm doing wrong. I'm having trouble using distutils with the bdist_wininst option. I've been able to create a COM server and package it in a .DLL, but I can't create an "installer" as a .exe My goal is to create a .exe that can be run on a client machine that does NOT have the standard Python environment, and this .exe would install all the modules?/.dll's/.exe's that are necessary for the COM server to run on the client machine. When I activate the "Run" command from within the PythonWin window and select the program setup_NaComponent_exe.py (attached), it nicely creates build and dist sub-directories. In the dist directory, there is a .exe and a .dll file: NaComponent.dll NaComponent.exe The interactive window has lots of informational messages such as: *** copy extensions *** . . *** copy dlls *** . . *** binary dependencies *** . . But the .exe and .dll created from the above (without the bdist_wininst argument) cannot be simply installed on a client machine and act as a COM server. (I tried registering the .dll with this command: regsvr32 /s \\Eng\prod\Group\TIS\Dev\Mfg\M3_New_PipeShop_COM\Current\Application\dis t\NaComponent.dll but that did not work. Next, I ran setup_NaComponent_exe.py, but I added the argument: bdist_wininst The messages returned in the interactive window were: >>> running bdist_wininst running build installing to build\bdist.win32\wininst >>> There was an error: ....\setup_NaComponent_exe.py' returned exit code error: build\bdist.win32\wininst: No such fi So I created the directory build\bdist.win32\wininst, reran the setup program, and got this: >>> running bdist_wininst running build installing to build\bdist.win32\wininst creating 'c:\docume~1\n50519\locals~1\temp\tmpaqvrsy.zip' and adding '.' to it removing 'build\bdist.win32\wininst' (and everything under it) >>> >>> But there was no zip file tmpaqvrsy.zip in the above-mentioned directory, and the wininst directory was removed (as described above). Can someone tell me what I'm doing wrong? Johan Vinke Nassco San Diego, CA _________________________________________________________________ Version info: Python Version: 2.3.3 PythonWin Version: win32all build 163 1st info line in PythonWin: PythonWin 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32. -------------- next part -------------- A non-text attachment was scrubbed... Name: setup_NaComponent_exe.py Type: application/octet-stream Size: 1773 bytes Desc: setup_NaComponent_exe.py Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070723/2ddd996e/attachment.obj From mhammond at skippinet.com.au Tue Jul 24 02:07:33 2007 From: mhammond at skippinet.com.au (Mark Hammond) Date: Tue, 24 Jul 2007 10:07:33 +1000 Subject: [Distutils] distutils.util.get_platform() for Windows In-Reply-To: <46A53D94.8020702@latte.ca> Message-ID: <0e0601c7cd86$a2e7a3f0$090a0a0a@enfoldsystems.local> > Phillip J. Eby wrote: > >>> So how about something like: > >>> win-x86_64 > >>> win-ia64 > >> http://en.wikipedia.org/wiki/X86-64 notes that 'x64' is a > common name, so > >> how does 'win-x64' and 'win-ia64' sound as a compromise? > I'm happy to let > >> any other informal "votes" make a final decision though... > > win-x64 and win-ia64 sound fine to me, for whatever that's > worth. :) > > Along with win-x32? The plan is to leave 32bit windows alone - it will continue to return 'win32' > (I suggest win32-x86, win64-x86 and win64-ia, but as I don't plan on > using more than one of them, I don't particularly care. > win32, (or was > that win-32?,) win-x64 and win-ia64 seem fine, if that's what > gets chosen.) I'd be inclined to agree if 32bit windows was also up for a change - but I see no good reason to do that, and a number of bad ones. It does mean there will be a slight inconsistency in the names, but I think we can live with that - ie: pywin32-211.win32-py2.5.msi pywin32-211.win-x64-py2.5.msi pywin32-211.win-ia64-py2.5.msi Would be the names pywin32 uses for the relevant platforms, and although there is an extra dash in the 64bit versions, I think it looks quite reasonable. Cheers, Mark From mhammond at skippinet.com.au Tue Jul 24 02:11:19 2007 From: mhammond at skippinet.com.au (Mark Hammond) Date: Tue, 24 Jul 2007 10:11:19 +1000 Subject: [Distutils] Problem with bdist_wininst option (trying to distributea COM server, on Windows XP) In-Reply-To: <19FDA17B69C10C46AD6BE50906D05DC9026BA77B@EMAIL1.nassco.local> Message-ID: <0e0701c7cd87$295e8d90$090a0a0a@enfoldsystems.local> Johan: > I'm having trouble using distutils with the bdist_wininst option. > I've been able to create a COM server and package it in a .DLL, > but I can't create an "installer" as a .exe My goal is to create > a .exe that can be run on a client machine that does NOT have the > standard Python environment, and this .exe would install all the > modules?/.dll's/.exe's that are necessary for the COM server to > run on the client machine. That is not the role of bdist_wininst. bdist_wininst is designed to create an installer for a python package, not a stand-alone executable created using Python. What you want is the py2exe tool (http://www.py2exe.org/) to create an executable, then a generic installer tool, such as NSIS, Inno or an MSI based tool to actually install the tool on the end-user's PC. py2exe has its own mailing list - see the link above. Cheers, Mark -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 2060 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070724/de94be77/attachment-0001.bin From martin at v.loewis.de Tue Jul 24 06:40:18 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 24 Jul 2007 06:40:18 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <20070723211919.361E23A403D@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <46A509D4.3070108@v.loewis.de> <20070723204446.294223A40B2@sparrow.telecommunity.com> <46A51A0C.2090800@v.loewis.de> <20070723211919.361E23A403D@sparrow.telecommunity.com> Message-ID: <46A582B2.3060105@v.loewis.de> Phillip J. Eby schrieb: > At 11:13 PM 7/23/2007 +0200, Martin v. L?wis wrote: >> > Yes, especially since compatibility with the existing installation >> > base requires case insensitivity, because on case-insensitive >> > platforms easy_install already normalizes the case of filenames it >> > creates. So, the question of what the "right thing" to do is in the >> > abstract has already been moot for a year or two. >> >> Can you elaborate a bit, please? Why does the case of filenames >> matter for the queries it makes? > > In order to resolve dependencies, the system looks at installed .egg > files and directories (and .egg-info direcories), and extracts package > name and version info from the filenames. Still - why does that require case-insensitive lookups to the index? Suppose a package specifies a dependency Foo. IIUC, you look on disk whether foo is already present, finding the version(s) of foo installed in that process. Then, this either is satisfying or not. If it is, you don't need the index at all. If it is not, you need to go to the index - but you still know that it is Foo that you were looking for, no? So lookups for dependencies in the index could always be case-sensitive; please correct me if I'm wrong. Regards, Martin From jim at zope.com Tue Jul 24 12:11:15 2007 From: jim at zope.com (Jim Fulton) Date: Tue, 24 Jul 2007 06:11:15 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <20070723211919.361E23A403D@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <46A509D4.3070108@v.loewis.de> <20070723204446.294223A40B2@sparrow.telecommunity.com> <46A51A0C.2090800@v.loewis.de> <20070723211919.361E23A403D@sparrow.telecommunity.com> Message-ID: <0CED8E91-F8C1-4951-A4C6-F7DDA81BE027@zope.com> On Jul 23, 2007, at 5:21 PM, Phillip J. Eby wrote: > At 11:13 PM 7/23/2007 +0200, Martin v. L?wis wrote: >>> Yes, especially since compatibility with the existing installation >>> base requires case insensitivity, because on case-insensitive >>> platforms easy_install already normalizes the case of filenames it >>> creates. So, the question of what the "right thing" to do is in the >>> abstract has already been moot for a year or two. >> >> Can you elaborate a bit, please? Why does the case of filenames >> matter for the queries it makes? >> >> AFAIU, it gets package names either from the user or from setup.py, >> perhaps also from packages dependency inside .egg files (assuming >> those support dependencies); these should all be case-sensitive. > > In order to resolve dependencies, the system looks at installed .egg > files and directories (and .egg-info direcories), and extracts > package name and version info from the filenames. But the package name and version are in the PKG-INFO files, so it certainly has access to non-normalized names. Why can't it double check a possible match against that file? Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From benji at benjiyork.com Tue Jul 24 16:42:37 2007 From: benji at benjiyork.com (Benji York) Date: Tue, 24 Jul 2007 10:42:37 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A4FA64.5050404@benjiyork.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <46A384C9.8040404@v.loewis.de> <46A38DF5.6010701@v.loewis.de> <46A4FA64.5050404@benjiyork.com> Message-ID: <46A60FDD.2030207@benjiyork.com> Benji York wrote: > I plan to do similar timings with the "simple" PyPI interface when I > get a chance and report the results here. Here are my non-scientific results: buildout times: regular: 4:52.86 simple: 3:15.57 ppix: 2:03.58 As everyone is aware, network latency has a large impact on this so here are the shortest round-trip packet times I got (with a small sample). cheeseshop.python.org: 93ms download.zope.org: 8ms I suspect the majority/entirety of the difference between ppix and simple is network related. -- Benji York http://benjiyork.com From pje at telecommunity.com Tue Jul 24 17:31:19 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 24 Jul 2007 11:31:19 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <0CED8E91-F8C1-4951-A4C6-F7DDA81BE027@zope.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <46A509D4.3070108@v.loewis.de> <20070723204446.294223A40B2@sparrow.telecommunity.com> <46A51A0C.2090800@v.loewis.de> <20070723211919.361E23A403D@sparrow.telecommunity.com> <0CED8E91-F8C1-4951-A4C6-F7DDA81BE027@zope.com> Message-ID: <20070724153349.CAF4B3A40AE@sparrow.telecommunity.com> At 06:11 AM 7/24/2007 -0400, Jim Fulton wrote: >On Jul 23, 2007, at 5:21 PM, Phillip J. Eby wrote: > >>At 11:13 PM 7/23/2007 +0200, Martin v. L?wis wrote: >>>>Yes, especially since compatibility with the existing installation >>>>base requires case insensitivity, because on case-insensitive >>>>platforms easy_install already normalizes the case of filenames it >>>>creates. So, the question of what the "right thing" to do is in the >>>>abstract has already been moot for a year or two. >>> >>>Can you elaborate a bit, please? Why does the case of filenames >>>matter for the queries it makes? >>> >>>AFAIU, it gets package names either from the user or from setup.py, >>>perhaps also from packages dependency inside .egg files (assuming >>>those support dependencies); these should all be case-sensitive. >> >>In order to resolve dependencies, the system looks at installed .egg >>files and directories (and .egg-info direcories), and extracts >>package name and version info from the filenames. > >But the package name and version are in the PKG-INFO files, so it >certainly has access to non-normalized names. Why can't it double >check a possible match against that file? Because if case actually made a difference, we couldn't have both packages installed in the same directory, could we? And why add an extra file open (which currently is only needed for "develop" eggs) to the process of building a working set or environment, in order to confirm something whose only purpose is to make requirements more difficult to specify? :) Note that if what's bothering you is the package index access time, use Apache's mod_speling to enable case-insensitive URLs for the static page tree. From jim at zope.com Tue Jul 24 17:39:38 2007 From: jim at zope.com (Jim Fulton) Date: Tue, 24 Jul 2007 11:39:38 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <20070724153349.CAF4B3A40AE@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <46A509D4.3070108@v.loewis.de> <20070723204446.294223A40B2@sparrow.telecommunity.com> <46A51A0C.2090800@v.loewis.de> <20070723211919.361E23A403D@sparrow.telecommunity.com> <0CED8E91-F8C1-4951-A4C6-F7DDA81BE027@zope.com> <20070724153349.CAF4B3A40AE@sparrow.telecommunity.com> Message-ID: <37BB633B-B2E6-43BB-AB16-CFE807CF8625@zope.com> On Jul 24, 2007, at 11:31 AM, Phillip J. Eby wrote: > At 06:11 AM 7/24/2007 -0400, Jim Fulton wrote: > >> On Jul 23, 2007, at 5:21 PM, Phillip J. Eby wrote: >> >>> At 11:13 PM 7/23/2007 +0200, Martin v. L?wis wrote: >>>>> Yes, especially since compatibility with the existing installation >>>>> base requires case insensitivity, because on case-insensitive >>>>> platforms easy_install already normalizes the case of filenames it >>>>> creates. So, the question of what the "right thing" to do is >>>>> in the >>>>> abstract has already been moot for a year or two. >>>> >>>> Can you elaborate a bit, please? Why does the case of filenames >>>> matter for the queries it makes? >>>> >>>> AFAIU, it gets package names either from the user or from setup.py, >>>> perhaps also from packages dependency inside .egg files (assuming >>>> those support dependencies); these should all be case-sensitive. >>> >>> In order to resolve dependencies, the system looks at installed .egg >>> files and directories (and .egg-info direcories), and extracts >>> package name and version info from the filenames. >> >> But the package name and version are in the PKG-INFO files, so it >> certainly has access to non-normalized names. Why can't it double >> check a possible match against that file? > > Because if case actually made a difference, we couldn't have both > packages installed in the same directory, could we? And why add an > extra file open (which currently is only needed for "develop" eggs) > to the process of building a working set or environment, in order > to confirm something whose only purpose is to make requirements > more difficult to specify? :) Currently, we allow packages to differ only in case. The fact that setuptools pretends we don't doesn't change the fact that we do. You said that "compatibility with the existing installation base requires case insensitivity, because on case-insensitive platforms easy_install already normalizes the case of filenames it creates". I'm merely pointing out that we don't have to rely soley on the file name. > Note that if what's bothering you is the package index access time, > use Apache's mod_speling to enable case-insensitive URLs for the > static page tree. *If* we decide that package names are case insensitive, then we should do this. We haven't decided this. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Tue Jul 24 17:54:32 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 24 Jul 2007 11:54:32 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <37BB633B-B2E6-43BB-AB16-CFE807CF8625@zope.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <46A509D4.3070108@v.loewis.de> <20070723204446.294223A40B2@sparrow.telecommunity.com> <46A51A0C.2090800@v.loewis.de> <20070723211919.361E23A403D@sparrow.telecommunity.com> <0CED8E91-F8C1-4951-A4C6-F7DDA81BE027@zope.com> <20070724153349.CAF4B3A40AE@sparrow.telecommunity.com> <37BB633B-B2E6-43BB-AB16-CFE807CF8625@zope.com> Message-ID: <20070724155212.E16A93A40A7@sparrow.telecommunity.com> At 11:39 AM 7/24/2007 -0400, Jim Fulton wrote: >On Jul 24, 2007, at 11:31 AM, Phillip J. Eby wrote: > >>At 06:11 AM 7/24/2007 -0400, Jim Fulton wrote: >> >>>On Jul 23, 2007, at 5:21 PM, Phillip J. Eby wrote: >>> >>>>At 11:13 PM 7/23/2007 +0200, Martin v. L?wis wrote: >>>>>>Yes, especially since compatibility with the existing installation >>>>>>base requires case insensitivity, because on case-insensitive >>>>>>platforms easy_install already normalizes the case of filenames it >>>>>>creates. So, the question of what the "right thing" to do is >>>>>>in the >>>>>>abstract has already been moot for a year or two. >>>>> >>>>>Can you elaborate a bit, please? Why does the case of filenames >>>>>matter for the queries it makes? >>>>> >>>>>AFAIU, it gets package names either from the user or from setup.py, >>>>>perhaps also from packages dependency inside .egg files (assuming >>>>>those support dependencies); these should all be case-sensitive. >>>> >>>>In order to resolve dependencies, the system looks at installed .egg >>>>files and directories (and .egg-info direcories), and extracts >>>>package name and version info from the filenames. >>> >>>But the package name and version are in the PKG-INFO files, so it >>>certainly has access to non-normalized names. Why can't it double >>>check a possible match against that file? >> >>Because if case actually made a difference, we couldn't have both >>packages installed in the same directory, could we? And why add an >>extra file open (which currently is only needed for "develop" eggs) >>to the process of building a working set or environment, in order >>to confirm something whose only purpose is to make requirements >>more difficult to specify? :) > >Currently, we allow packages to differ only in case. The fact that >setuptools pretends we don't doesn't change the fact that we do. I wasn't under the impression that we were discussing whether allowing project names to differ only in case was a good idea, since I haven't heard anybody give an argument that it's a *good* idea. In fact, it seems like an obviously bad idea on its face, whether setuptools is in the picture or not. >>Note that if what's bothering you is the package index access time, >>use Apache's mod_speling to enable case-insensitive URLs for the >>static page tree. > >*If* we decide that package names are case insensitive, then we >should do this. We haven't decided this. Well, so far the only argument *against* it that I recall seeing, is your argument that sloppy requirement specs slow everybody down by making them do the extra package index hit. So, if that's fixable, what other argument is there for treating the names case-sensitively? From jim at zope.com Tue Jul 24 18:36:29 2007 From: jim at zope.com (Jim Fulton) Date: Tue, 24 Jul 2007 12:36:29 -0400 Subject: [Distutils] We need to make a decision wrt distribution names Message-ID: <7AC9ED0E-FFAF-4493-9EBF-068538F2ABA9@zope.com> Obviously, we are having a debate about what forms distribution names can take. I think we need a decision. Does anyone know if there are existing rules for package names? I can't find them if there are. Up until now, I think we've been in somewhat of a prototyping mode, but I think it's time to move beyond that. I strongly suggest that we need an official specification that says: - what's a legal package name and - what the equivalence rules for package names are. Whatever we decide needs to be well supported by setuptools and PyPI. I can live with whatever we decide as long as we decide something and make sure it is well communicated and implemented. In particular, I could live with the equivalence rules that setuptools uses if they are documented and if they are supported correctly and efficiently by the index (including mirrors). IMO, a decision is extremely important. If we can't reach consensus, then we need to call in the BDFL. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Tue Jul 24 18:54:36 2007 From: jim at zope.com (Jim Fulton) Date: Tue, 24 Jul 2007 12:54:36 -0400 Subject: [Distutils] We need to make a decision wrt distribution names In-Reply-To: <7AC9ED0E-FFAF-4493-9EBF-068538F2ABA9@zope.com> References: <7AC9ED0E-FFAF-4493-9EBF-068538F2ABA9@zope.com> Message-ID: <41DE9981-BDAE-4A35-B989-DAD4749CA6BD@zope.com> On Jul 24, 2007, at 12:36 PM, Jim Fulton wrote: > > Obviously, we are having a debate about what forms distribution > names can take. I think we need a decision. > > Does anyone know if there are existing rules for package names? Doh. I meant "distribution names". Sorry. > I can't find them if there are. Up until now, I think we've been > in somewhat of a prototyping mode, but I think it's time to move > beyond that. > > I strongly suggest that we need an official specification that says: > > - what's a legal package name and Ditto > > - what the equivalence rules for package names are. Ditto. > > Whatever we decide needs to be well supported by setuptools and > PyPI. I can live with whatever we decide as long as we decide > something and make sure it is well communicated and implemented. In > particular, I could live with the equivalence rules that setuptools > uses if they are documented and if they are supported correctly and > efficiently by the index (including mirrors). > > IMO, a decision is extremely important. If we can't reach > consensus, then we need to call in the BDFL. -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Tue Jul 24 18:55:39 2007 From: jim at zope.com (Jim Fulton) Date: Tue, 24 Jul 2007 12:55:39 -0400 Subject: [Distutils] We need to make a decision wrt distribution names (second try) Message-ID: Obviously, we are having a debate about what forms distribution names can take. I think we need a decision. Does anyone know if there are existing rules for distribution names? I can't find them if there are. Up until now, I think we've been in somewhat of a prototyping mode, but I think it's time to move beyond that. I strongly suggest that we need an official specification that says: - what's a legal distribution name and - what the equivalence rules for distribution names are. Whatever we decide needs to be well supported by setuptools and PyPI. I can live with whatever we decide as long as we decide something and make sure it is well communicated and implemented. In particular, I could live with the equivalence rules that setuptools uses if they are documented and if they are supported correctly and efficiently by the index (including mirrors). IMO, a decision is extremely important. If we can't reach consensus, then we need to call in the BDFL. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Tue Jul 24 18:57:01 2007 From: jim at zope.com (Jim Fulton) Date: Tue, 24 Jul 2007 12:57:01 -0400 Subject: [Distutils] [Catalog-sig] We need to make a decision wrt distribution names In-Reply-To: <46A62DA7.9000304@pangalactic.us> References: <7AC9ED0E-FFAF-4493-9EBF-068538F2ABA9@zope.com> <46A62DA7.9000304@pangalactic.us> Message-ID: On Jul 24, 2007, at 12:49 PM, Stephen Waterbury wrote: > Jim Fulton wrote: >> Does anyone know if there are existing rules for package names? I >> can't find them if there are. ... > > Well, there is PEP 8, which has this to say on the subject: > > "Package and Module Names > > "Modules should have short, all-lowercase names. Underscores > can be used in the module name if it improves readability. > Python packages should also have short, all-lowercase names, > although the use of underscores is discouraged." Doh, I was sloppy in my terminology. I should have said "distribution name". We're talking about the names used in PyPI, the Python Distribution index. ;) Also the value passed to the "name" argument of setup. Sorry for the confusion. :) Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From waterbug at pangalactic.us Tue Jul 24 19:09:20 2007 From: waterbug at pangalactic.us (Stephen Waterbury) Date: Tue, 24 Jul 2007 13:09:20 -0400 Subject: [Distutils] [Catalog-sig] We need to make a decision wrt distribution names In-Reply-To: References: <7AC9ED0E-FFAF-4493-9EBF-068538F2ABA9@zope.com> <46A62DA7.9000304@pangalactic.us> Message-ID: <46A63240.7070003@pangalactic.us> Jim Fulton wrote: > > On Jul 24, 2007, at 12:49 PM, Stephen Waterbury wrote: > >> Jim Fulton wrote: >>> Does anyone know if there are existing rules for package names? I >>> can't find them if there are. ... >> >> Well, there is PEP 8, which has this to say on the subject: >> >> "Package and Module Names >> >> "Modules should have short, all-lowercase names. Underscores >> can be used in the module name if it improves readability. >> Python packages should also have short, all-lowercase names, >> although the use of underscores is discouraged." > > Doh, I was sloppy in my terminology. I should have said "distribution > name". We're talking about the names used in PyPI, the Python > Distribution index. ;) Also the value passed to the "name" argument of > setup. > > Sorry for the confusion. :) Actually, I wasn't confused. :) I'd suggest a convention that allows a distribution "title" (e.g., "Zope", "Twisted", etc.) and a distribution "name" that would simply be the name of the distribution's top-level package (e.g., "zope", "twisted", etc.), which should follow the PEP 8 suggestion for package names and should be what setuptools uses together with a version reference to uniquely identify a specific distribution/version (egg). Steve From waterbug at pangalactic.us Tue Jul 24 18:49:43 2007 From: waterbug at pangalactic.us (Stephen Waterbury) Date: Tue, 24 Jul 2007 12:49:43 -0400 Subject: [Distutils] We need to make a decision wrt distribution names In-Reply-To: <7AC9ED0E-FFAF-4493-9EBF-068538F2ABA9@zope.com> References: <7AC9ED0E-FFAF-4493-9EBF-068538F2ABA9@zope.com> Message-ID: <46A62DA7.9000304@pangalactic.us> Jim Fulton wrote: > Does anyone know if there are existing rules for package names? I > can't find them if there are. ... Well, there is PEP 8, which has this to say on the subject: "Package and Module Names "Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability. Python packages should also have short, all-lowercase names, although the use of underscores is discouraged." Steve From lars at ibp.de Tue Jul 24 19:28:59 2007 From: lars at ibp.de (Lars Immisch) Date: Tue, 24 Jul 2007 19:28:59 +0200 Subject: [Distutils] We need to make a decision wrt distribution names (second try) In-Reply-To: References: Message-ID: <46A636DB.50105@ibp.de> Hi, > Obviously, we are having a debate about what forms distribution names > can take. I think we need a decision. Thanks for bringing this up. > Does anyone know if there are existing rules for distribution names? > I can't find them if there are. Up until now, I think we've been in > somewhat of a prototyping mode, but I think it's time to move beyond > that. > > I strongly suggest that we need an official specification that says: > > - what's a legal distribution name and > > - what the equivalence rules for distribution names are. Comparison rules are also important: Is artin-1.2-rc2 < artin-1.2? IMO, it's perfectly fine to just state: comparisons are lexicographical (ASCII only). But I'd like to see this mentioned somewhere. - Lars From martin at v.loewis.de Tue Jul 24 19:29:06 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 24 Jul 2007 19:29:06 +0200 Subject: [Distutils] [Catalog-sig] We need to make a decision wrt distribution names In-Reply-To: <7AC9ED0E-FFAF-4493-9EBF-068538F2ABA9@zope.com> References: <7AC9ED0E-FFAF-4493-9EBF-068538F2ABA9@zope.com> Message-ID: <46A636E2.2090408@v.loewis.de> > I strongly suggest that we need an official specification that says: The process would then be to write a PEP. It will end with a BDFL pronouncement either way, but that might be easy to obtain if there is consensus up-front. Regards, Martin From jim at zope.com Tue Jul 24 19:33:43 2007 From: jim at zope.com (Jim Fulton) Date: Tue, 24 Jul 2007 13:33:43 -0400 Subject: [Distutils] We need to make a decision wrt distribution names (second try) In-Reply-To: <46A636DB.50105@ibp.de> References: <46A636DB.50105@ibp.de> Message-ID: On Jul 24, 2007, at 1:28 PM, Lars Immisch wrote: > Hi, > >> Obviously, we are having a debate about what forms distribution >> names can take. I think we need a decision. > > Thanks for bringing this up. > >> Does anyone know if there are existing rules for distribution >> names? I can't find them if there are. Up until now, I think >> we've been in somewhat of a prototyping mode, but I think it's >> time to move beyond that. >> I strongly suggest that we need an official specification that says: >> - what's a legal distribution name and >> - what the equivalence rules for distribution names are. > > Comparison rules are also important: > > Is artin-1.2-rc2 < artin-1.2? Note that these are not distribution names. Well, that depends on how you define "distribution names". Sigh. The dsitribution names I'm trying to talk about don't have version numbers. I don't see a particular reason why these distribution names have to be ordered, Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From martin at v.loewis.de Tue Jul 24 19:40:55 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 24 Jul 2007 19:40:55 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <20070724153349.CAF4B3A40AE@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <46A509D4.3070108@v.loewis.de> <20070723204446.294223A40B2@sparrow.telecommunity.com> <46A51A0C.2090800@v.loewis.de> <20070723211919.361E23A403D@sparrow.telecommunity.com> <0CED8E91-F8C1-4951-A4C6-F7DDA81BE027@zope.com> <20070724153349.CAF4B3A40AE@sparrow.telecommunity.com> Message-ID: <46A639A7.7090305@v.loewis.de> >> But the package name and version are in the PKG-INFO files, so it >> certainly has access to non-normalized names. Why can't it double >> check a possible match against that file? > > Because if case actually made a difference, we couldn't have both > packages installed in the same directory, could we? Right. However, there is a difference between case-insensitive, and case-preserving. > Note that if what's bothering you is the package index access time, use > Apache's mod_speling to enable case-insensitive URLs for the static page > tree. That won't help. If you look for a name of a non-registered package, setuptools will go to the index even if mod_speling corrects spelling errors. Such an approach is only possible if setuptools would stop using the entire index if the server has case-insensitive lookup (which it cannot determine). Regards, Martin From pje at telecommunity.com Tue Jul 24 19:45:08 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 24 Jul 2007 13:45:08 -0400 Subject: [Distutils] [Catalog-sig] We need to make a decision wrt distribution names In-Reply-To: <46A63240.7070003@pangalactic.us> References: <7AC9ED0E-FFAF-4493-9EBF-068538F2ABA9@zope.com> <46A62DA7.9000304@pangalactic.us> <46A63240.7070003@pangalactic.us> Message-ID: <20070724174248.F40AA3A40A7@sparrow.telecommunity.com> At 01:09 PM 7/24/2007 -0400, Stephen Waterbury wrote: >Actually, I wasn't confused. :) I'd suggest a convention that allows >a distribution "title" (e.g., "Zope", "Twisted", etc.) and a >distribution "name" that would simply be the name of the >distribution's top-level package (e.g., "zope", "twisted", etc.), This proposal would rule out namespace packages, in addition to being incompatible with existing distribution names. Note that package != distribution -- a distribution may contain zero or more packages (even top-level), *and* a single package (top-level or otherwise) may be spread over more than one distribution. Also note that this was true even with the distutils, long before setuptools existed. From pje at telecommunity.com Tue Jul 24 19:48:54 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 24 Jul 2007 13:48:54 -0400 Subject: [Distutils] [Catalog-sig] We need to make a decision wrt distribution names (second try) In-Reply-To: References: <46A636DB.50105@ibp.de> Message-ID: <20070724174634.2283E3A40A7@sparrow.telecommunity.com> At 01:33 PM 7/24/2007 -0400, Jim Fulton wrote: >Note that these are not distribution names. Well, that depends on >how you define "distribution names". Sigh. The dsitribution names >I'm trying to talk about don't have version numbers. Setuptools uses the term "project name" for what you're calling a distribution name, if that helps any. :) From pje at telecommunity.com Tue Jul 24 19:52:32 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 24 Jul 2007 13:52:32 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A639A7.7090305@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <46A509D4.3070108@v.loewis.de> <20070723204446.294223A40B2@sparrow.telecommunity.com> <46A51A0C.2090800@v.loewis.de> <20070723211919.361E23A403D@sparrow.telecommunity.com> <0CED8E91-F8C1-4951-A4C6-F7DDA81BE027@zope.com> <20070724153349.CAF4B3A40AE@sparrow.telecommunity.com> <46A639A7.7090305@v.loewis.de> Message-ID: <20070724175013.238323A40A7@sparrow.telecommunity.com> At 07:40 PM 7/24/2007 +0200, Martin v. L?wis wrote: > >> But the package name and version are in the PKG-INFO files, so it > >> certainly has access to non-normalized names. Why can't it double > >> check a possible match against that file? > > > > Because if case actually made a difference, we couldn't have both > > packages installed in the same directory, could we? > >Right. However, there is a difference between case-insensitive, >and case-preserving. I don't understand your statement here, nor what is supposed to follow from it. > > Note that if what's bothering you is the package index access time, use > > Apache's mod_speling to enable case-insensitive URLs for the static page > > tree. > >That won't help. If you look for a name of a non-registered package, >setuptools will go to the index even if mod_speling corrects spelling >errors. Jim's objection was that if it's possible to get case-correction from the index, people will declare setup.py dependencies with incorrect case, leading to other packages having indirect dependencies with incorrect case, leading to lots of package index lookups. This objection is relevant only to requirements which differ from the actual project name only by their case. A non-registered package lookup is going to fail no matter what, and thus isn't going to wind up in a setup.py without a dependency_links specifier that will prevent it being looked up in the package index to begin with. From jim at zope.com Tue Jul 24 19:51:31 2007 From: jim at zope.com (Jim Fulton) Date: Tue, 24 Jul 2007 13:51:31 -0400 Subject: [Distutils] [Catalog-sig] We need to make a decision wrt distribution names (second try) In-Reply-To: <20070724174634.2283E3A40A7@sparrow.telecommunity.com> References: <46A636DB.50105@ibp.de> <20070724174634.2283E3A40A7@sparrow.telecommunity.com> Message-ID: <0A0EDDEC-7BCC-41C1-822A-5D93AF20E1F7@zope.com> On Jul 24, 2007, at 1:48 PM, Phillip J. Eby wrote: > At 01:33 PM 7/24/2007 -0400, Jim Fulton wrote: >> Note that these are not distribution names. Well, that depends on >> how you define "distribution names". Sigh. The dsitribution names >> I'm trying to talk about don't have version numbers. > > Setuptools uses the term "project name" for what you're calling a > distribution name, if that helps any. :) Right. I'm happy to use that. Does anyone want to disagree? BTW, to up the ante, I volunteer to try to update the distutils document to reflect what we decide. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From lars at ibp.de Tue Jul 24 20:11:57 2007 From: lars at ibp.de (Lars Immisch) Date: Tue, 24 Jul 2007 20:11:57 +0200 Subject: [Distutils] We need to make a decision wrt distribution names (second try) In-Reply-To: References: <46A636DB.50105@ibp.de> Message-ID: <46A640ED.8070406@ibp.de> Hi, >>> Obviously, we are having a debate about what forms distribution >>> names can take. I think we need a decision. >> >> Thanks for bringing this up. >> >>> Does anyone know if there are existing rules for distribution >>> names? I can't find them if there are. Up until now, I think we've >>> been in somewhat of a prototyping mode, but I think it's time to >>> move beyond that. >>> I strongly suggest that we need an official specification that says: >>> - what's a legal distribution name and >>> - what the equivalence rules for distribution names are. >> >> Comparison rules are also important: >> >> Is artin-1.2-rc2 < artin-1.2? > > Note that these are not distribution names. Well, that depends on how > you define "distribution names". Sigh. The dsitribution names I'm > trying to talk about don't have version numbers. I don't see a > particular reason why these distribution names have to be ordered, I see. Sorry for the drive-by-shooting. Still, I'd like a stated convention how version numbers are compared. I believe this would be good for setuptools also. But the issue is separable from project naming conventions. - Lars From martin at v.loewis.de Tue Jul 24 20:21:11 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 24 Jul 2007 20:21:11 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <20070724175013.238323A40A7@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <46A509D4.3070108@v.loewis.de> <20070723204446.294223A40B2@sparrow.telecommunity.com> <46A51A0C.2090800@v.loewis.de> <20070723211919.361E23A403D@sparrow.telecommunity.com> <0CED8E91-F8C1-4951-A4C6-F7DDA81BE027@zope.com> <20070724153349.CAF4B3A40AE@sparrow.telecommunity.com> <46A639A7.7090305@v.loewis.de> <20070724175013.238323A40A7@sparrow.telecommunity.com> Message-ID: <46A64317.6090907@v.loewis.de> >> > Because if case actually made a difference, we couldn't have both >> > packages installed in the same directory, could we? >> >> Right. However, there is a difference between case-insensitive, >> and case-preserving. > > I don't understand your statement here, nor what is supposed to follow > from it. Clearly, on a case-insensitive file system, project names differing only in case cannot coexist. That doesn't mean that all references to the project should be case-normalized (e.g. lower-cased). So even if project names compare case-insensitive, there still should (could) be a "right" spelling, the one that the package author wants to see. This is the spelling that others then should use. So I still don't see why the file names on disk have any effect on the lookup setuptools do to the index. > Jim's objection was that if it's possible to get case-correction from > the index, people will declare setup.py dependencies with incorrect > case, leading to other packages having indirect dependencies with > incorrect case, leading to lots of package index lookups. I don't think that was his objection. IIUC, he complains about incorrect spellings as bad, period - regardless of whether they also have a performance effect. It's like spelling your name "Philipp" - that's a bad thing to do, independent of whether it also makes you harder to find (which it actually doesn't, thanks to Google). > This objection is relevant only to requirements which differ from the > actual project name only by their case. A non-registered package lookup > is going to fail no matter what, and thus isn't going to wind up in a > setup.py without a dependency_links specifier that will prevent it being > looked up in the package index to begin with. Right. However, if setuptools would stop making case insensitive lookups to the index, lookups to unregistered packages would become more efficient. Regards, Martin From pje at telecommunity.com Tue Jul 24 20:44:08 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 24 Jul 2007 14:44:08 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A64317.6090907@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <46A509D4.3070108@v.loewis.de> <20070723204446.294223A40B2@sparrow.telecommunity.com> <46A51A0C.2090800@v.loewis.de> <20070723211919.361E23A403D@sparrow.telecommunity.com> <0CED8E91-F8C1-4951-A4C6-F7DDA81BE027@zope.com> <20070724153349.CAF4B3A40AE@sparrow.telecommunity.com> <46A639A7.7090305@v.loewis.de> <20070724175013.238323A40A7@sparrow.telecommunity.com> <46A64317.6090907@v.loewis.de> Message-ID: <20070724184151.1EAE53A40A7@sparrow.telecommunity.com> At 08:21 PM 7/24/2007 +0200, Martin v. L?wis wrote: > >> > Because if case actually made a difference, we couldn't have both > >> > packages installed in the same directory, could we? > >> > >> Right. However, there is a difference between case-insensitive, > >> and case-preserving. > > > > I don't understand your statement here, nor what is supposed to follow > > from it. > >Clearly, on a case-insensitive file system, project names differing >only in case cannot coexist. That doesn't mean that all references >to the project should be case-normalized (e.g. lower-cased). > >So even if project names compare case-insensitive, there still >should (could) be a "right" spelling, the one that the package >author wants to see. This is the spelling that others then should >use. Well, that spelling will certainly show up everywhere. Setuptools is case-preserving, *except* with regard to installing egg files on case-insensitive filesystems (as defined by what os.path.normcase does on a given platform). When it installs an egg, it normalizes the case of the target path. In all other matters it is case-insensitive for comparison, but case-preserving of the inputs it receives. > > Jim's objection was that if it's possible to get case-correction from > > the index, people will declare setup.py dependencies with incorrect > > case, leading to other packages having indirect dependencies with > > incorrect case, leading to lots of package index lookups. > >I don't think that was his objection. IIUC, he complains about >incorrect spellings as bad, period - regardless of whether they also >have a performance effect. It's like spelling your name "Philipp" - >that's a bad thing to do, independent of whether it also makes you >harder to find (which it actually doesn't, thanks to Google). It's actually more like spelling my name "phillip", which is arguably still spelled correctly, if punctuated poorly. :) And it's also an answer to the wrong question: the *first* question is whether we should allow "phillip" and "Phillip" to co-exist in the package index. If not, then there is the question of whether there is any reason to be case-sensitive with respect to searching. If we are agreed that having projects whose names differ only by case is a bad idea, then the latter question is considerably less controversial. > > This objection is relevant only to requirements which differ from the > > actual project name only by their case. A non-registered package lookup > > is going to fail no matter what, and thus isn't going to wind up in a > > setup.py without a dependency_links specifier that will prevent it being > > looked up in the package index to begin with. > >Right. However, if setuptools would stop making case insensitive >lookups to the index, lookups to unregistered packages would become >more efficient. I'm not sure I follow you. If a non-registered package is used as a dependency, the setup() will need to specify dependency_links, in which case PyPI will not be consulted. From martin at v.loewis.de Tue Jul 24 20:54:24 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 24 Jul 2007 20:54:24 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <20070724184151.1EAE53A40A7@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <46A509D4.3070108@v.loewis.de> <20070723204446.294223A40B2@sparrow.telecommunity.com> <46A51A0C.2090800@v.loewis.de> <20070723211919.361E23A403D@sparrow.telecommunity.com> <0CED8E91-F8C1-4951-A4C6-F7DDA81BE027@zope.com> <20070724153349.CAF4B3A40AE@sparrow.telecommunity.com> <46A639A7.7090305@v.loewis.de> <20070724175013.238323A40A7@sparrow.telecommunity.com> <46A64317.6090907@v.loewis.de> <20070724184151.1EAE53A40A7@sparrow.telecommunity.com> Message-ID: <46A64AE0.7000307@v.loewis.de> >> Right. However, if setuptools would stop making case insensitive >> lookups to the index, lookups to unregistered packages would become >> more efficient. > > I'm not sure I follow you. If a non-registered package is used as a > dependency, the setup() will need to specify dependency_links, in which > case PyPI will not be consulted. Ah, ok. So is it then correct that setuptools never looks at pypi/, unless the user misspelled a package name on the command line? Regards, Martin From jim at zope.com Tue Jul 24 21:20:40 2007 From: jim at zope.com (Jim Fulton) Date: Tue, 24 Jul 2007 15:20:40 -0400 Subject: [Distutils] We need to make a decision wrt distribution names (second try) In-Reply-To: <46A640ED.8070406@ibp.de> References: <46A636DB.50105@ibp.de> <46A640ED.8070406@ibp.de> Message-ID: <1FE480D2-27FD-4F42-82FD-06C387805EE2@zope.com> On Jul 24, 2007, at 2:11 PM, Lars Immisch wrote: > Hi, > >>>> Obviously, we are having a debate about what forms distribution >>>> names can take. I think we need a decision. >>> >>> Thanks for bringing this up. >>> >>>> Does anyone know if there are existing rules for distribution >>>> names? I can't find them if there are. Up until now, I think >>>> we've been in somewhat of a prototyping mode, but I think it's >>>> time to move beyond that. >>>> I strongly suggest that we need an official specification that >>>> says: >>>> - what's a legal distribution name and >>>> - what the equivalence rules for distribution names are. >>> >>> Comparison rules are also important: >>> >>> Is artin-1.2-rc2 < artin-1.2? >> Note that these are not distribution names. Well, that depends on >> how you define "distribution names". Sigh. The dsitribution names >> I'm trying to talk about don't have version numbers. I don't see >> a particular reason why these distribution names have to be ordered, > > I see. Sorry for the drive-by-shooting. np > Still, I'd like a stated convention how version numbers are > compared. I believe this would be good for setuptools also. setuptools has this. It would be nice bless it in a PEP. > But the issue is separable from project naming conventions. Yup. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Tue Jul 24 21:46:42 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 24 Jul 2007 15:46:42 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A64AE0.7000307@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <799F00B4-AEAB-446D-B45A-B96B089C6C2C@zope.com> <20070723152015.E7AFA3A403D@sparrow.telecommunity.com> <46A4D0BE.4030706@palladion.com> <46A509D4.3070108@v.loewis.de> <20070723204446.294223A40B2@sparrow.telecommunity.com> <46A51A0C.2090800@v.loewis.de> <20070723211919.361E23A403D@sparrow.telecommunity.com> <0CED8E91-F8C1-4951-A4C6-F7DDA81BE027@zope.com> <20070724153349.CAF4B3A40AE@sparrow.telecommunity.com> <46A639A7.7090305@v.loewis.de> <20070724175013.238323A40A7@sparrow.telecommunity.com> <46A64317.6090907@v.loewis.de> <20070724184151.1EAE53A40A7@sparrow.telecommunity.com> <46A64AE0.7000307@v.loewis.de> Message-ID: <20070724194426.828EE3A40A7@sparrow.telecommunity.com> At 08:54 PM 7/24/2007 +0200, Martin v. L?wis wrote: > >> Right. However, if setuptools would stop making case insensitive > >> lookups to the index, lookups to unregistered packages would become > >> more efficient. > > > > I'm not sure I follow you. If a non-registered package is used as a > > dependency, the setup() will need to specify dependency_links, in which > > case PyPI will not be consulted. > >Ah, ok. So is it then correct that setuptools never looks at pypi/, >unless the user misspelled a package name on the command line? Pretty much, yes. From noah.gift at gmail.com Tue Jul 24 21:57:30 2007 From: noah.gift at gmail.com (Noah Gift) Date: Tue, 24 Jul 2007 15:57:30 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <20070724194426.828EE3A40A7@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <20070723211919.361E23A403D@sparrow.telecommunity.com> <0CED8E91-F8C1-4951-A4C6-F7DDA81BE027@zope.com> <20070724153349.CAF4B3A40AE@sparrow.telecommunity.com> <46A639A7.7090305@v.loewis.de> <20070724175013.238323A40A7@sparrow.telecommunity.com> <46A64317.6090907@v.loewis.de> <20070724184151.1EAE53A40A7@sparrow.telecommunity.com> <46A64AE0.7000307@v.loewis.de> <20070724194426.828EE3A40A7@sparrow.telecommunity.com> Message-ID: On 7/24/07, Phillip J. Eby wrote: > At 08:54 PM 7/24/2007 +0200, Martin v. L?wis wrote: > > >> Right. However, if setuptools would stop making case insensitive > > >> lookups to the index, lookups to unregistered packages would become > > >> more efficient. > > > > > > I'm not sure I follow you. If a non-registered package is used as a > > > dependency, the setup() will need to specify dependency_links, in which > > > case PyPI will not be consulted. > > > >Ah, ok. So is it then correct that setuptools never looks at pypi/, > >unless the user misspelled a package name on the command line? > > Pretty much, yes. Would it be a bad idea to suggest the case insensitive lookup happen against a local flat file that gets diff'd from PyPI? Then only the culprit gets punished using their own CPU :) From martin at v.loewis.de Tue Jul 24 22:02:27 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 24 Jul 2007 22:02:27 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <20070723211919.361E23A403D@sparrow.telecommunity.com> <0CED8E91-F8C1-4951-A4C6-F7DDA81BE027@zope.com> <20070724153349.CAF4B3A40AE@sparrow.telecommunity.com> <46A639A7.7090305@v.loewis.de> <20070724175013.238323A40A7@sparrow.telecommunity.com> <46A64317.6090907@v.loewis.de> <20070724184151.1EAE53A40A7@sparrow.telecommunity.com> <46A64AE0.7000307@v.loewis.de> <20070724194426.828EE3A40A7@sparrow.telecommunity.com> Message-ID: <46A65AD3.3060607@v.loewis.de> > Would it be a bad idea to suggest the case insensitive lookup happen > against a local flat file that gets diff'd from PyPI? Then only the > culprit gets punished using their own CPU :) What does it mean to "diff a flat file from PyPI"? Regards, Martin From noah.gift at gmail.com Tue Jul 24 22:07:15 2007 From: noah.gift at gmail.com (Noah Gift) Date: Tue, 24 Jul 2007 16:07:15 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A65AD3.3060607@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <20070724153349.CAF4B3A40AE@sparrow.telecommunity.com> <46A639A7.7090305@v.loewis.de> <20070724175013.238323A40A7@sparrow.telecommunity.com> <46A64317.6090907@v.loewis.de> <20070724184151.1EAE53A40A7@sparrow.telecommunity.com> <46A64AE0.7000307@v.loewis.de> <20070724194426.828EE3A40A7@sparrow.telecommunity.com> <46A65AD3.3060607@v.loewis.de> Message-ID: On 7/24/07, "Martin v. L?wis" wrote: > > Would it be a bad idea to suggest the case insensitive lookup happen > > against a local flat file that gets diff'd from PyPI? Then only the > > culprit gets punished using their own CPU :) > > What does it mean to "diff a flat file from PyPI"? I am familiar with an open source project called Radmind. It maintains machines be keeping a local transcript with all of the files and "overloads" on it. When you modify the file system you diff the changes into an overload and put them on the server. When the client asks for an update, the client checks to see if its transcript files are the same. If they are then it does nothing. If it different the file(s) get updated. Then the magic is that the search and replace for which files it needs to grab are done locally using ton's of local CPU. When the client resolves all of the files it needs, it then grabs them from the server. It is a nifty design: http://rsug.itd.umich.edu/software/radmind/ So, if someone does an "incorrect" search, easy_install checks to see first if it has the latest "file". If not, it then replaces its local index. Then the search happens locally, not being going back and forth to the server. > > Regards, > Martin > -- http://www.blog.noahgift.com From martin at v.loewis.de Tue Jul 24 22:23:58 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 24 Jul 2007 22:23:58 +0200 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <20070724153349.CAF4B3A40AE@sparrow.telecommunity.com> <46A639A7.7090305@v.loewis.de> <20070724175013.238323A40A7@sparrow.telecommunity.com> <46A64317.6090907@v.loewis.de> <20070724184151.1EAE53A40A7@sparrow.telecommunity.com> <46A64AE0.7000307@v.loewis.de> <20070724194426.828EE3A40A7@sparrow.telecommunity.com> <46A65AD3.3060607@v.loewis.de> Message-ID: <46A65FDE.7080806@v.loewis.de> > I am familiar with an open source project called Radmind. It > maintains machines be keeping a local transcript with all of the files > and "overloads" on it. When you modify the file system you diff the > changes into an overload and put them on the server. That's still a lot of terminology which I don't understand, and have no intuition for, perhaps because English is not my native language. I give up trying to understand - just to give you an idea: What's a "transcript of files"? How do you "overload" on it (why is "to overload" used with the preposition "on")? How do I "diff" a change "into" "an overload" (which now is a noun, it seems)? > So, if someone does an "incorrect" search, easy_install checks to see > first if it has the latest "file". If not, it then replaces its local > index. Then the search happens locally, not being going back and > forth to the server. I think this brings us to the real issue: you asked whether this would be a bad idea to suggest that? I now think "perhaps not bad, but unhelpful, unless you also contribute an implementation of it". It's a change to setuptools, which is still mostly a one-man-show, (IIUC), so proposing ideas in general is futile (as for most software with a single author - including PyPI); the single author cannot possibly implement all the ideas people have. Regards, Martin From noah.gift at gmail.com Tue Jul 24 22:37:49 2007 From: noah.gift at gmail.com (Noah Gift) Date: Tue, 24 Jul 2007 16:37:49 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <46A65FDE.7080806@v.loewis.de> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <20070724175013.238323A40A7@sparrow.telecommunity.com> <46A64317.6090907@v.loewis.de> <20070724184151.1EAE53A40A7@sparrow.telecommunity.com> <46A64AE0.7000307@v.loewis.de> <20070724194426.828EE3A40A7@sparrow.telecommunity.com> <46A65AD3.3060607@v.loewis.de> <46A65FDE.7080806@v.loewis.de> Message-ID: My real motive is selfishness. I like that easy_install in not case sensitive, as I and other people I am helping to learn Python. I just hope that doesn't go away. My suggestion is mored geared toward, how do I "keep" that feature :) > no intuition for, perhaps because English is not my native language. > I give up trying to understand - just to give you an idea: I apologize, I can be very lazy when I type. > I now think "perhaps not bad, but > unhelpful, unless you also contribute an implementation of it". > It's a change to setuptools, which is still mostly a one-man-show, > (IIUC), so proposing ideas in general is futile (as for most software > with a single author - including PyPI); the single author cannot > possibly implement all the ideas people have. The basic algorithm is that a local index of PyPi could be kept in one file. If an incorrect search was made, the first action to occur would be to check if the local file was the same as the file on the server. If not, it would sync the changes with svn. Then easy_install would try to do lookups against the local file to find a match. I am happy to help if you need help. I am particular interest in easy_install as I am writing a chapter on it for an O'Reilly book as well, again a partially selfish motive :) Noah > > Regards, > Martin > -- http://www.blog.noahgift.com From barry at python.org Tue Jul 24 23:21:41 2007 From: barry at python.org (Barry Warsaw) Date: Tue, 24 Jul 2007 17:21:41 -0400 Subject: [Distutils] entry points Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Just a quick question as I'm starting to play with this feature. Why do entry point values use a colon to separate the module from the callable? Couldn't you just as easily use a "normal" Python dotted path, the last component of which would of course be the callable? E.g. instead of setup( # ... entry_points = {'blogtool.parsers': '.rst = some_module:SomeClass'} ) use setup( # ... entry_points = {'blogtool.parsers': '.rst = some_module.SomeClass'} ) That would seem a bit more Pythonic. Cheers, - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRqZtZnEjvBPtnXfVAQIOHwP6AgX58dLBIzqvI7wo+z9MOIXm/rUIDikX j6Zc5S/97Fc9n3qjRlKf/fTT229oQVSIsPJgxvfXxWKYS5Qlk+YqhoACI9BuTbrm KshRSiPzFPy2eqjk287sHmrpq1MJWwrccjF4lSROn6VDcojlW/t2p6FY/+Gvj3au CPhRo/Amr+M= =8qj5 -----END PGP SIGNATURE----- From pje at telecommunity.com Wed Jul 25 00:09:30 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 24 Jul 2007 18:09:30 -0400 Subject: [Distutils] entry points In-Reply-To: References: Message-ID: <20070724220713.C34383A40A7@sparrow.telecommunity.com> At 05:21 PM 7/24/2007 -0400, Barry Warsaw wrote: >Why do entry point values use a colon to separate the module from the >callable? First off, an entry point need not be callable in the general case. Classes and arbitrary data structures can be entry points, too. Heck, Chandler parcel entry points are just a module name. > Couldn't you just as easily use a "normal" Python dotted >path, the last component of which would of course be the callable? > >E.g. instead of > >setup( > # ... > entry_points = {'blogtool.parsers': '.rst = some_module:SomeClass'} >) > >use > >setup( > # ... > entry_points = {'blogtool.parsers': '.rst = some_module.SomeClass'} >) > >That would seem a bit more Pythonic. Perhaps, but it's also a PITA to implement correctly in the presence of things like classmethods and other objects that could be legitimate targets for entry points. It requires a fair amount of error trapping, and the precise errors aren't always the same from one version of Python to the next. (Note, btw, that a module itself can be a valid entry point.) So, even though I implemented that algorithm correctly in peak.util.imports many years ago (available now as a standalone package "Importing" from the Cheeseshop), I decided to never do it again anywhere else. And since pkg_resources needs to be able to run without any other packages installed, I did the practical thing instead of the pure thing, by requiring people to be explicit instead of implicit. :) From pje at telecommunity.com Wed Jul 25 00:13:48 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 24 Jul 2007 18:13:48 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <20070724175013.238323A40A7@sparrow.telecommunity.com> <46A64317.6090907@v.loewis.de> <20070724184151.1EAE53A40A7@sparrow.telecommunity.com> <46A64AE0.7000307@v.loewis.de> <20070724194426.828EE3A40A7@sparrow.telecommunity.com> <46A65AD3.3060607@v.loewis.de> <46A65FDE.7080806@v.loewis.de> Message-ID: <20070724221128.D9B193A40A7@sparrow.telecommunity.com> At 04:37 PM 7/24/2007 -0400, Noah Gift wrote: >The basic algorithm is that a local index of PyPi could be kept in one >file. If an incorrect search was made, the first action to occur >would be to check if the local file was the same as the file on the >server. If not, it would sync the changes with svn. Then >easy_install would try to do lookups against the local file to find a >match. Note that there are a lot of ways you can implement something like this without even involving me on the client or Martin on the server. For example, setuptools.package_index uses urllib2 for all its URL access, so installing an "opener" that does caching before invoking easy_install is possible. You can also subclass the easy_install command class and the PackageIndex class, or tell the easy_install command class to use a different PackageIndex implementation. In the long run, I'd like to add some entry points to allow people to extend the search mechanism in such ways, but for now you can certainly hack subclasses easily enough and make your own alternative commands, as Jim has done for integrating zc.buildout with setuptools. From richardjones at optushome.com.au Wed Jul 25 00:14:22 2007 From: richardjones at optushome.com.au (Richard Jones) Date: Wed, 25 Jul 2007 08:14:22 +1000 Subject: [Distutils] [Catalog-sig] We need to make a decision wrt distribution names (second try) In-Reply-To: <46A640ED.8070406@ibp.de> References: <46A640ED.8070406@ibp.de> Message-ID: <200707250814.22196.richardjones@optushome.com.au> On Wed, 25 Jul 2007, Lars Immisch wrote: > Still, I'd like a stated convention how version numbers are compared. I > believe this would be good for setuptools also. Currently PyPI sorts releases using distutils.version.LooseVersion It uses distutils.version.StrictVersion when parsing "provides, "requires" and "obsoletes" setup.py package meta-data. Richard From noah.gift at gmail.com Wed Jul 25 00:30:31 2007 From: noah.gift at gmail.com (Noah Gift) Date: Tue, 24 Jul 2007 18:30:31 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <20070724221128.D9B193A40A7@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <20070724184151.1EAE53A40A7@sparrow.telecommunity.com> <46A64AE0.7000307@v.loewis.de> <20070724194426.828EE3A40A7@sparrow.telecommunity.com> <46A65AD3.3060607@v.loewis.de> <46A65FDE.7080806@v.loewis.de> <20070724221128.D9B193A40A7@sparrow.telecommunity.com> Message-ID: On 7/24/07, Phillip J. Eby wrote: > At 04:37 PM 7/24/2007 -0400, Noah Gift wrote: > >The basic algorithm is that a local index of PyPi could be kept in one > >file. If an incorrect search was made, the first action to occur > >would be to check if the local file was the same as the file on the > >server. If not, it would sync the changes with svn. Then > >easy_install would try to do lookups against the local file to find a > >match. > > Note that there are a lot of ways you can implement something like > this without even involving me on the client or Martin on the > server. For example, setuptools.package_index uses urllib2 for all > its URL access, so installing an "opener" that does caching before > invoking easy_install is possible. You can also subclass the > easy_install command class and the PackageIndex class, or tell the > easy_install command class to use a different PackageIndex implementation. > > In the long run, I'd like to add some entry points to allow people to > extend the search mechanism in such ways, but for now you can > certainly hack subclasses easily enough and make your own alternative > commands, as Jim has done for integrating zc.buildout with setuptools. > Great suggestion! I really like that idea. Does this mean it is also easy to point to another local repository that is available via NFS? I guess a local http mirror would work just as well, if you told the opener about it. This seems like a good way to instruct a sysadmin on how to setup a local customized infrastructure! > -- http://www.blog.noahgift.com From pje at telecommunity.com Wed Jul 25 01:05:32 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 24 Jul 2007 19:05:32 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <20070724184151.1EAE53A40A7@sparrow.telecommunity.com> <46A64AE0.7000307@v.loewis.de> <20070724194426.828EE3A40A7@sparrow.telecommunity.com> <46A65AD3.3060607@v.loewis.de> <46A65FDE.7080806@v.loewis.de> <20070724221128.D9B193A40A7@sparrow.telecommunity.com> Message-ID: <20070724230313.35F703A40A7@sparrow.telecommunity.com> At 06:30 PM 7/24/2007 -0400, Noah Gift wrote: >Does this mean it is also easy to point to another local repository >that is available via NFS? I guess a local http mirror would work >just as well, if you told the opener about it. You don't even need to do that. See the -i/--index-url option to easy_install, for how to specify a specific package index URL to be used in place of PyPI. >This seems like a good way to instruct a sysadmin on how to setup a >local customized infrastructure! See also: http://peak.telecommunity.com/DevCenter/EasyInstall#creating-your-own-package-index http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api From barry at python.org Wed Jul 25 01:06:21 2007 From: barry at python.org (Barry Warsaw) Date: Tue, 24 Jul 2007 19:06:21 -0400 Subject: [Distutils] entry points In-Reply-To: <20070724220713.C34383A40A7@sparrow.telecommunity.com> References: <20070724220713.C34383A40A7@sparrow.telecommunity.com> Message-ID: <770EA661-4474-40DF-8B0F-1BF9D9FC6FDA@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jul 24, 2007, at 6:09 PM, Phillip J. Eby wrote: > At 05:21 PM 7/24/2007 -0400, Barry Warsaw wrote: >> Why do entry point values use a colon to separate the module from the >> callable? > > First off, an entry point need not be callable in the general > case. Classes and arbitrary data structures can be entry points, > too. Heck, Chandler parcel entry points are just a module name. Fair enough. That wasn't entirely evident from the documentation, but maybe I wasn't looking in the right place or didn't grok what I was reading. ;) By now I've gotten my first plugin to work, and it's really quite awesome. Between setuptools and zope.interface, I'm just very psyched that I now have a principled and standard way of providing extensibility for my application. One thing I think will have to be done at some point is to consolidate all the documentation on the telecommunity site. I find myself with three open browser tabs (setuptools, PkgResources, and EasyInstall). There's /a lot/ of information there. I vote the PSF pays for a tech writer to put it all together into a single cohesive manual. Thanks for the explanation regarding the choice of ':'. I suspected you had a good reason for it. :) - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRqaF7nEjvBPtnXfVAQL85wQArnlzXZCd20QhGfk53ac9F0JB1tj0FcYP JMofaetnelHti/Fh/uMEK+S4Uxf1RnEwQwixXvk9YXvGwlVnrZINPELlx57FB4tB xpNlcwtkBRa0HCTP1LtyniQ0h+4PbHIHeNmvFskQ7+5YJCdGZYX5vffOScFlomSF tMtbU1XrZqQ= =KrB4 -----END PGP SIGNATURE----- From waterbug at pangalactic.us Wed Jul 25 03:56:17 2007 From: waterbug at pangalactic.us (Stephen Waterbury) Date: Tue, 24 Jul 2007 21:56:17 -0400 Subject: [Distutils] [Catalog-sig] We need to make a decision wrt distribution names In-Reply-To: <20070724174248.F40AA3A40A7@sparrow.telecommunity.com> References: <7AC9ED0E-FFAF-4493-9EBF-068538F2ABA9@zope.com> <46A62DA7.9000304@pangalactic.us> <46A63240.7070003@pangalactic.us> <20070724174248.F40AA3A40A7@sparrow.telecommunity.com> Message-ID: <46A6ADC1.4000008@pangalactic.us> Phillip J. Eby wrote: > At 01:09 PM 7/24/2007 -0400, Stephen Waterbury wrote: >> Actually, I wasn't confused. :) I'd suggest a convention that allows >> a distribution "title" (e.g., "Zope", "Twisted", etc.) and a >> distribution "name" that would simply be the name of the >> distribution's top-level package (e.g., "zope", "twisted", etc.), > > This proposal would rule out namespace packages ... I thought about that. The rule for namespace distributions would be to allow dotted names, e.g. "zope.interface", "zope.schema", etc., as are often currently used. In fact, in a real sense, those *are* the top-level packages of namespace packages. > in addition to being > incompatible with existing distribution names. I thought the point was to come up with a new distribution naming convention, because there currently isn't one -- but the naming convention has to be consistent with all existing distribution names? Seems a tough constraint. > Note that package != distribution ... Yes, I knew that. Of course, now the discussion seems to suggest "project" or "project release" might be a better name than "distribution", and I agree with that. > -- a distribution may contain zero or > more packages (even top-level) ... Indeed, and I've always disliked multiple top-level packages in an [installable unit]. I never liked ZODB strewing top-level packages all over site-packages. (But I do like ZODB -- thanks Jim et al.! I'd just much prefer that it have a top-level "zodb" package.) Of course, eggs make site-packages dirs look much tidier, but I'd still prefer that each [installable unit] have a top-level package, because then it's obvious where imported modules come from just by looking at their top-level namespace. > *and* a single package (top-level or > otherwise) may be spread over more than one distribution. IMO, a package that's spread over more than one distribution should probably not be top-level in both distributions. :) BTW, I am not emotionally attached to this proposal (good thing, eh? ;), but there are a couple of principles in it that I thought deserved a little bit of logical advocacy, e.g.: * if a package deserves a "top-level" namespace, it probably also deserves have its own [installable unit]. * although package != [installable unit], I still think it's not illogical to use the top-level package of an [installable unit] as part of its canonical unique identifier. But admittedly one would have to agree with some of my other points above to agree with that. Steve From pje at telecommunity.com Wed Jul 25 04:19:03 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 24 Jul 2007 22:19:03 -0400 Subject: [Distutils] [Catalog-sig] We need to make a decision wrt distribution names In-Reply-To: <46A6ADC1.4000008@pangalactic.us> References: <7AC9ED0E-FFAF-4493-9EBF-068538F2ABA9@zope.com> <46A62DA7.9000304@pangalactic.us> <46A63240.7070003@pangalactic.us> <20070724174248.F40AA3A40A7@sparrow.telecommunity.com> <46A6ADC1.4000008@pangalactic.us> Message-ID: <20070725021710.B45F43A40A7@sparrow.telecommunity.com> At 09:56 PM 7/24/2007 -0400, Stephen Waterbury wrote: >I thought the point was to come up with a new distribution naming >convention, Nope, just clarify the rules for *distinguishing* projects by name -- a much less ambitious goal, since it's pretty easy to do with little or no impact on existing projects. A new naming convention isn't in scope, since it would require a "boil the ocean" renaming effort to implement, assuming you could get everyone to agree in the first place. From waterbug at pangalactic.us Wed Jul 25 04:56:03 2007 From: waterbug at pangalactic.us (Stephen Waterbury) Date: Tue, 24 Jul 2007 22:56:03 -0400 Subject: [Distutils] [Catalog-sig] We need to make a decision wrt distribution names In-Reply-To: <20070725021710.B45F43A40A7@sparrow.telecommunity.com> References: <7AC9ED0E-FFAF-4493-9EBF-068538F2ABA9@zope.com> <46A62DA7.9000304@pangalactic.us> <46A63240.7070003@pangalactic.us> <20070724174248.F40AA3A40A7@sparrow.telecommunity.com> <46A6ADC1.4000008@pangalactic.us> <20070725021710.B45F43A40A7@sparrow.telecommunity.com> Message-ID: <46A6BBC3.7000505@pangalactic.us> Phillip J. Eby wrote: > At 09:56 PM 7/24/2007 -0400, Stephen Waterbury wrote: >> I thought the point was to come up with a new distribution naming >> convention, > > Nope, just clarify the rules for *distinguishing* projects by name -- > a much less ambitious goal, since it's pretty easy to do with little > or no impact on existing projects. > > A new naming convention isn't in scope, since it would require a > "boil the ocean" renaming effort to implement, assuming you could get > everyone to agree in the first place. Indeed. Boiling the ocean will have to wait. I still think putting multiple top-level packages in a single installable is a mistake. ;) Peace. Steve From jim at zope.com Wed Jul 25 15:25:38 2007 From: jim at zope.com (Jim Fulton) Date: Wed, 25 Jul 2007 09:25:38 -0400 Subject: [Distutils] [Catalog-sig] We need to make a decision wrt distribution names In-Reply-To: <20070725021710.B45F43A40A7@sparrow.telecommunity.com> References: <7AC9ED0E-FFAF-4493-9EBF-068538F2ABA9@zope.com> <46A62DA7.9000304@pangalactic.us> <46A63240.7070003@pangalactic.us> <20070724174248.F40AA3A40A7@sparrow.telecommunity.com> <46A6ADC1.4000008@pangalactic.us> <20070725021710.B45F43A40A7@sparrow.telecommunity.com> Message-ID: On Jul 24, 2007, at 10:19 PM, Phillip J. Eby wrote: > At 09:56 PM 7/24/2007 -0400, Stephen Waterbury wrote: >> I thought the point was to come up with a new distribution naming >> convention, > > Nope, just clarify the rules for *distinguishing* projects by name -- > a much less ambitious goal, since it's pretty easy to do with little > or no impact on existing projects. I mostly agree, except that I think we also need to define what is legal in a project name. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Wed Jul 25 15:30:45 2007 From: jim at zope.com (Jim Fulton) Date: Wed, 25 Jul 2007 09:30:45 -0400 Subject: [Distutils] [Catalog-sig] We need to make a decision wrt distribution names In-Reply-To: <46A6ADC1.4000008@pangalactic.us> References: <7AC9ED0E-FFAF-4493-9EBF-068538F2ABA9@zope.com> <46A62DA7.9000304@pangalactic.us> <46A63240.7070003@pangalactic.us> <20070724174248.F40AA3A40A7@sparrow.telecommunity.com> <46A6ADC1.4000008@pangalactic.us> Message-ID: <1882F28C-AC83-4EEE-9F86-979CF5DEB88E@zope.com> On Jul 24, 2007, at 9:56 PM, Stephen Waterbury wrote: > Phillip J. Eby wrote: >> At 01:09 PM 7/24/2007 -0400, Stephen Waterbury wrote: >>> Actually, I wasn't confused. :) I'd suggest a convention that >>> allows >>> a distribution "title" (e.g., "Zope", "Twisted", etc.) and a >>> distribution "name" that would simply be the name of the >>> distribution's top-level package (e.g., "zope", "twisted", etc.), >> >> This proposal would rule out namespace packages ... > > I thought about that. The rule for namespace distributions would > be to > allow dotted names, e.g. "zope.interface", "zope.schema", etc., as are > often currently used. In fact, in a real sense, those *are* the > top-level packages of namespace packages. Those are the top-level packages of those distributions. >> in addition to being >> incompatible with existing distribution names. > > I thought the point was to come up with a new distribution naming > convention, because there currently isn't one -- but the naming > convention has to be consistent with all existing distribution > names? Seems a tough constraint. No, my proposal was to define: - Rules for constructing *legal* (as opposed to "good") project names - Rules for variations on project names. ... >> -- a distribution may contain zero or >> more packages (even top-level) ... > > Indeed, and I've always disliked multiple top-level packages in an > [installable unit]. No offense intended, but this seems arbitrary to me. Note that not only can a distribution contain more than one package, it can contain no packages. >> *and* a single package (top-level or >> otherwise) may be spread over more than one distribution. > > IMO, a package that's spread over more than one distribution should > probably not be top-level in both distributions. :) Phillip was (I think) referring to namespace packages. Namespace packages are a very important tool for maintaining some sanity in package naming. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From james at NBN.ac.za Wed Jul 25 17:18:48 2007 From: james at NBN.ac.za (James Dominy) Date: Wed, 25 Jul 2007 17:18:48 +0200 Subject: [Distutils] Questions regarding the compilation of a non-extension shared library Message-ID: <46A769D8.7090000@nbn.ac.za> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I'm have a shared library that I would like to distribute along with my python package, written in pure C. It requires the correct installation of other software at linktime, and at runtime. The package a ctypes wrapper for SUNDIALS (a mathematical C library). I would normally simply compile my shared library on the command line as follows - --- snip: shell output 1 --- $ gcc -c -o sundials_core_aux.o sundials_core_aux.c $ gcc --shared -fPIC -Wl,-solibsundials_core_aux.so.1 -o libsundials_core_aux.so.1.0.0 sundials_core_aux.o $ /usr/lib/gcc/i686-pc-linux-gnu/3.4.6/../../../../i686-pc-linux-gnu/bin/ld: warning: creating a DT_TEXTREL in object. - --- snip --- To get rid of the DT_TEXTREL warning, I simply add -fPIC to compilation step. - --- snip: shell output 2 --- $ gcc --shared -fPIC -c -o sundials_core_aux.o $ gcc --shared -fPIC -Wl,-solibsundials_core_aux.so.1 -o libsundials_core_aux.so.1.0.0 sundials_core_aux.c $ - --- snip --- This produces no errors or warnings. Obviously this is a non-portable solution, and I'm trying to use distutils to compile the library. But when I do, using - --- snip: setup.py --- compiler = ccompiler.new_compiler() compiler.compile(['sundials_core_aux.c']) compiler.link_shared_lib(['src/sundials_core_aux.o'], compiler.library_filename('sundials_core_aux', 'shared'), 'dist') - --- snip --- The DT_TEXTREL warning appears. I've tried any number of combinations of extra_preargs and extra_postargs using '-fPIC' and '-Wl,-so...', to no avail, and equally, these are not portable arguments. Should the warning just be ignored, or is there a way I can get rid of it? Thanks, James -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) iD8DBQFGp2nYolkHbC41Z6URAguzAJ9GWnXSVsHl6T9M126jv5e3N6hxcACguPPv aCJiGxN3M8TVJ84oYYBsc7U= =I2Vn -----END PGP SIGNATURE----- From robert.kern at gmail.com Wed Jul 25 20:49:30 2007 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 25 Jul 2007 13:49:30 -0500 Subject: [Distutils] Questions regarding the compilation of a non-extension shared library In-Reply-To: <46A769D8.7090000@nbn.ac.za> References: <46A769D8.7090000@nbn.ac.za> Message-ID: James Dominy wrote: > Hi, > > I'm have a shared library that I would like to distribute along with my python package, written in pure C. It requires the > correct installation of other software at linktime, and at runtime. The package a ctypes wrapper for SUNDIALS (a mathematical C > library). You might want to take a look at OOF2. If includes a distutils command for building a shared library. I believe it tries to install the library to $prefix/lib which may not be what you want, but it might help you to find the right incantation to at least build the thing. http://www.ctcms.nist.gov/oof/oof2/index.html Will your code be open source? We have many requests for SUNDIALS wrappers on the scipy list, and it would be great to satisfy them. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From rajeev.nair at gridstoneresearch.com Wed Jul 25 21:29:05 2007 From: rajeev.nair at gridstoneresearch.com (Rajeev Nair) Date: Thu, 26 Jul 2007 00:59:05 +0530 Subject: [Distutils] DistUtils does not install Message-ID: <00de01c7cef2$11de7e70$8801a8c0@gsr.com> Hi Folks, I am a complete newbie to python and its modules. I have got the following error upon installing DistUtils modules Python setup.py install Traceback (most recent call last): File "setup.py", line 30, in ? packages = ['distutils', 'distutils.command'], File "/usr/local/Distutils-1.0.2/distutils/core.py", line 101, in setup _setup_distribution = dist = klass(attrs) File "/usr/local/Distutils-1.0.2/distutils/dist.py", line 130, in __init__ setattr(self, method_name, getattr(self.metadata, method_name)) AttributeError: DistributionMetadata instance has no attribute 'get___doc__' What am I missing here ? Tried Installing some python-devel modules too but they just wouldn't install complaining about the python version to be 2.4. I have installed the same version of python though. The OS is Suse x86_64 GNU/Linux Please help. Regards, Rajeev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070726/43a71a66/attachment.htm From pje at telecommunity.com Wed Jul 25 22:20:30 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 25 Jul 2007 16:20:30 -0400 Subject: [Distutils] DistUtils does not install In-Reply-To: <00de01c7cef2$11de7e70$8801a8c0@gsr.com> References: <00de01c7cef2$11de7e70$8801a8c0@gsr.com> Message-ID: <20070725201812.789783A40E6@sparrow.telecommunity.com> At 12:59 AM 7/26/2007 +0530, Rajeev Nair wrote: >Hi Folks, > >I am a complete newbie to python and its modules. I have got the >following error upon installing DistUtils modules > >Python setup.py install > >Traceback (most recent call last): > File "setup.py", line 30, in ? > packages = ['distutils', 'distutils.command'], > File "/usr/local/Distutils-1.0.2/distutils/core.py", line 101, in setup > _setup_distribution = dist = klass(attrs) > File "/usr/local/Distutils-1.0.2/distutils/dist.py", line 130, in __init__ > setattr(self, method_name, getattr(self.metadata, method_name)) >AttributeError: DistributionMetadata instance has no attribute 'get___doc__' > >What am I missing here ? Tried Installing some python-devel modules >too but they just wouldn't install complaining about the python >version to be 2.4. I have installed the same version of python though. > >The OS is Suse x86_64 GNU/Linux > >Please help. The distutils are included with modern Python versions; you should not attempt to install the standalone version, as it is almost certainly *way* out of date. If your operating system does not include the distutils as part of its standard Python installation, you'll need to install whatever package (usually python-devel) contains them. To check if you already have the distutils installed correctly, start your python interpreter and "import distutils". If that doesn't give you an error, they're already installed. From sklein at cpcug.org Wed Jul 25 22:42:25 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Wed, 25 Jul 2007 16:42:25 -0400 (EDT) Subject: [Distutils] Namespace packages and egg-info confusion when using bdist_rpm Message-ID: <64930.207.188.248.157.1185396145.squirrel@www.cpcug.org> I've been trying to build rpms of enthought system components. Some of them use namespace packages. Those packages have the required __init__.py files containing "__import__('pkg_resources').declare_namespace(__name__)". According to the Setuptools Guide, these __init__.py files are not to be packaged when using "system" packaging (such as bdist_rpm) and when I run bdist_rpm they are not included in the rpms. However, there are egg-info directories and nspkg.pth files produced, included in the rpms, and installed in site-packages. When I tried to run the enthought example applications, I had various kinds of import failures. I found that fixing them required putting the unpackaged __init__.py files into the relevant directories in the namespace packages. The various rpms were being placed in the proper locations where they should have been found for importing. There were some strange errors, such as in one case when I tried to test the imports interactively an "import a.b.c" worked, but an "import a.b.c as c" failed. Before putting in the __init__.py files, I tried removing the egg-info and nspkg.pth files from site_packages. That did not improve things. I also noticed that some other packages had pth files in site-packages, so that may be what is needed if the egg-info and nspkg.pth are removed, although I don't know why or what the pth files are supposed to contain. I think setuptools is somehow not properly processing bdist_rpm if there are namespace packages involved. Also, the setuptools don't seem to have an option to not create the egg info and nspkg.pth files and/or to not package them if bdist_rpm is being used. Stan Klein From arve.knudsen at gmail.com Wed Jul 25 22:51:40 2007 From: arve.knudsen at gmail.com (Arve Knudsen) Date: Wed, 25 Jul 2007 22:51:40 +0200 Subject: [Distutils] Regeneration of source manifest In-Reply-To: References: Message-ID: I take it there is no interest in fixing this problem? Or is it simply too much hassle?? Arve On 7/10/07, Arve Knudsen wrote: > > Hello > > As far as I can tell setuptools does not always regenerate the source > manifest when MANIFEST.in changes, contrary to what the documentation > says. That is, if I add a directive to exclude files with the .txt extension > to MANIFEST.in (after having previously including them), and run the sdist > command, such files are still included. I have to remove the .egg-info for > things to work as expected. This must clearly be a bug? > > Regards, > Arve Knudsen > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070725/6c75a5f3/attachment.html From pje at telecommunity.com Thu Jul 26 00:10:54 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 25 Jul 2007 18:10:54 -0400 Subject: [Distutils] Namespace packages and egg-info confusion when using bdist_rpm In-Reply-To: <64930.207.188.248.157.1185396145.squirrel@www.cpcug.org> References: <64930.207.188.248.157.1185396145.squirrel@www.cpcug.org> Message-ID: <20070725220835.061D23A40BF@sparrow.telecommunity.com> At 04:42 PM 7/25/2007 -0400, Stanley A. Klein wrote: >I've been trying to build rpms of enthought system components. Some of >them use namespace packages. Those packages have the required __init__.py >files containing >"__import__('pkg_resources').declare_namespace(__name__)". > >According to the Setuptools Guide, these __init__.py files are not to be >packaged when using "system" packaging (such as bdist_rpm) and when I run >bdist_rpm they are not included in the rpms. However, there are egg-info >directories and nspkg.pth files produced, included in the rpms, and >installed in site-packages. So far, so good - that's exactly how it's supposed to work. > When I tried to run the enthought example >applications, I had various kinds of import failures. I found that fixing >them required putting the unpackaged __init__.py files into the relevant >directories in the namespace packages. > >The various rpms were being placed in the proper locations where they >should have been found for importing. There were some strange errors, >such as in one case when I tried to test the imports interactively an >"import a.b.c" worked, but an "import a.b.c as c" failed. > >Before putting in the __init__.py files, I tried removing the egg-info and >nspkg.pth files from site_packages. That did not improve things. It sure as heck wouldn't. They need to be there, and the nspkg.pth files need to be processed by Python during its startup. >I think setuptools is somehow not properly processing bdist_rpm if there >are namespace packages involved. Also, the setuptools don't seem to have >an option to not create the egg info and nspkg.pth files and/or to not >package them if bdist_rpm is being used. This is by design. It is almost certainly the case that either: 1. the nspkg.pth files are not being processed when Python starts 2. something is deleting the namespace packages from sys.modules between the time the .pth files are processed, and the time you attempt to import them. To verify this, I would say, start Python with the bdist_rpms installed as directed (i.e., don't mess with what's installed), and see if there are sys.modules entries for the namespace package(s). If there are, then the problem is happening later - i.e., something being done by your code. If the namespace packages aren't in sys.modules, then there is a problem with the .pth files at startup; try using "python -v" to see if there are any errors shown. From pje at telecommunity.com Thu Jul 26 00:19:53 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 25 Jul 2007 18:19:53 -0400 Subject: [Distutils] Regeneration of source manifest In-Reply-To: References: Message-ID: <20070725221733.7D1593A403D@sparrow.telecommunity.com> At 10:51 PM 7/25/2007 +0200, Arve Knudsen wrote: >I take it there is no interest in fixing this problem? Or is it >simply too much hassle?? > >Arve > >On 7/10/07, Arve Knudsen < >arve.knudsen at gmail.com> wrote: >Hello > >As far as I can tell setuptools does not always regenerate the >source manifest when MANIFEST.in changes, contrary to what the >documentation says. That is, if I add a directive to exclude files >with the .txt extension to MANIFEST.in (after having previously >including them), and run the sdist command, such files are still >included. I have to remove the .egg-info for things to work as >expected. This must clearly be a bug? > >Regards, >Arve Knudsen Your email got lost in the flurry of PyPI-related emails. I am looking at the code and don't see how this condition could be produced, at least not the way you seem to be describing it, because the template is processed after the old filelist is read in. So if there are exclusion commands in the template, these should *always* be applied to the resulting SOURCES.txt. Could you give me the exact old MANIFEST.in contents, the new MANIFEST.in contents, and the egg-info/SOURCES.txt? Better yet, can you give me a small setup.py and steps that I can use to reproduce the problem? From rajeev.nair at gridstoneresearch.com Thu Jul 26 09:06:51 2007 From: rajeev.nair at gridstoneresearch.com (Rajeev Nair) Date: Thu, 26 Jul 2007 12:36:51 +0530 Subject: [Distutils] DistUtils does not install In-Reply-To: <20070725201812.789783A40E6@sparrow.telecommunity.com> Message-ID: <010101c7cf53$8c418d90$8801a8c0@gsr.com> Thanks, Philip. I installed python-devel from the SUSE installation CD and that helped. I was able to import distutils without any error from the python prompt. -----Original Message----- From: Phillip J. Eby [mailto:pje at telecommunity.com] Sent: Thursday, July 26, 2007 1:51 AM To: Rajeev Nair; distutils-sig at python.org Subject: Re: [Distutils] DistUtils does not install At 12:59 AM 7/26/2007 +0530, Rajeev Nair wrote: >Hi Folks, > >I am a complete newbie to python and its modules. I have got the >following error upon installing DistUtils modules > >Python setup.py install > >Traceback (most recent call last): > File "setup.py", line 30, in ? > packages = ['distutils', 'distutils.command'], > File "/usr/local/Distutils-1.0.2/distutils/core.py", line 101, in setup > _setup_distribution = dist = klass(attrs) > File "/usr/local/Distutils-1.0.2/distutils/dist.py", line 130, in __init__ > setattr(self, method_name, getattr(self.metadata, method_name)) >AttributeError: DistributionMetadata instance has no attribute 'get___doc__' > >What am I missing here ? Tried Installing some python-devel modules >too but they just wouldn't install complaining about the python >version to be 2.4. I have installed the same version of python though. > >The OS is Suse x86_64 GNU/Linux > >Please help. The distutils are included with modern Python versions; you should not attempt to install the standalone version, as it is almost certainly *way* out of date. If your operating system does not include the distutils as part of its standard Python installation, you'll need to install whatever package (usually python-devel) contains them. To check if you already have the distutils installed correctly, start your python interpreter and "import distutils". If that doesn't give you an error, they're already installed. From dpeterson at enthought.com Thu Jul 26 10:02:02 2007 From: dpeterson at enthought.com (Dave Peterson) Date: Thu, 26 Jul 2007 03:02:02 -0500 Subject: [Distutils] Best practices on distributing documentation and examples with eggs? Message-ID: <46A854FA.3080906@enthought.com> Over on the enthought-dev mailing list we're having a bit of a discussion on what the best way to distribute documenation and examples is for projects that we distribute binary eggs for. The general consensus is that it would be very nice indeed if there was a way to generate a tarball, or platform install, of just documentation and examples so that people wouldn't need to download a full, presumably significantly larger, source tarball. Another option would be that eggs included the documentation and examples and that, during the installation of the egg, those docs and examples were relocated to a common location (outside of the zip) to make access by users more convenient. This latter idea is similar to how Ruby Gems deal with docs. I don't claim to be a distutils or setuptools guru, so it shouldn't be too surprising that I can't seem to find anything about a setuptools or distutils command to do either of these. Am I missing something? If not, does it seem like something that might be worthy of putting into setuptools? -- Dave From noah.gift at gmail.com Thu Jul 26 17:30:07 2007 From: noah.gift at gmail.com (Noah Gift) Date: Thu, 26 Jul 2007 11:30:07 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: <20070724230313.35F703A40A7@sparrow.telecommunity.com> References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <20070724194426.828EE3A40A7@sparrow.telecommunity.com> <46A65AD3.3060607@v.loewis.de> <46A65FDE.7080806@v.loewis.de> <20070724221128.D9B193A40A7@sparrow.telecommunity.com> <20070724230313.35F703A40A7@sparrow.telecommunity.com> Message-ID: On 7/24/07, Phillip J. Eby wrote: > At 06:30 PM 7/24/2007 -0400, Noah Gift wrote: > >Does this mean it is also easy to point to another local repository > >that is available via NFS? I guess a local http mirror would work > >just as well, if you told the opener about it. > > You don't even need to do that. See the -i/--index-url option to > easy_install, for how to specify a specific package index URL to be > used in place of PyPI. > > > >This seems like a good way to instruct a sysadmin on how to setup a > >local customized infrastructure! > > See also: > > http://peak.telecommunity.com/DevCenter/EasyInstall#creating-your-own-package-index > > http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api > > Phillip, Thanks for the pointer. I am have been rereading the documentation on easy_install this week. I have one other question for you. Have you envisioned a workflow that interacts with RPM's or other package management systems? I know my company uses RPM's for example to package up our Python Packages, and many shops use RPM and/or debian packaging systems. Thanks, Noah -- http://www.blog.noahgift.com From pje at telecommunity.com Thu Jul 26 17:52:58 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 26 Jul 2007 11:52:58 -0400 Subject: [Distutils] Best practices on distributing documentation and examples with eggs? In-Reply-To: <46A854FA.3080906@enthought.com> References: <46A854FA.3080906@enthought.com> Message-ID: <20070726155222.18CB43A40D7@sparrow.telecommunity.com> At 03:02 AM 7/26/2007 -0500, Dave Peterson wrote: >Over on the enthought-dev mailing list we're having a bit of a >discussion on what the best way to distribute documenation and examples >is for projects that we distribute binary eggs for. The general >consensus is that it would be very nice indeed if there was a way to >generate a tarball, or platform install, of just documentation and >examples so that people wouldn't need to download a full, presumably >significantly larger, source tarball. Another option would be that >eggs included the documentation and examples and that, during the >installation of the egg, those docs and examples were relocated to a >common location (outside of the zip) to make access by users more >convenient. This latter idea is similar to how Ruby Gems deal with docs. > >I don't claim to be a distutils or setuptools guru, so it shouldn't be >too surprising that I can't seem to find anything about a setuptools or >distutils command to do either of these. Am I missing something? There are a few different ways you could do this. The easiest would be to put a docs subdirectory either inside one of your packages or inside your .egg-info directory, then use the pkg_resources resource or metadata APIs to list and extract them. One advantage to using something like .egg-info/docs would be that this could perhaps be recognized by some sort of "standard" tools in the future. >If not, does it seem like something that might be worthy of putting into >setuptools? It might be worth establishing convention(s) for, yes. Tools could then be developed around them, including perhaps changes to easy_install. From pje at telecommunity.com Thu Jul 26 17:54:41 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 26 Jul 2007 11:54:41 -0400 Subject: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index. In-Reply-To: References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <20070724194426.828EE3A40A7@sparrow.telecommunity.com> <46A65AD3.3060607@v.loewis.de> <46A65FDE.7080806@v.loewis.de> <20070724221128.D9B193A40A7@sparrow.telecommunity.com> <20070724230313.35F703A40A7@sparrow.telecommunity.com> Message-ID: <20070726155223.C988E3A40F0@sparrow.telecommunity.com> At 11:30 AM 7/26/2007 -0400, Noah Gift wrote: >Have you envisioned a workflow that interacts with RPM's or other >package management systems? I know my company uses RPM's for >example to package up our Python Packages, and many shops use RPM >and/or debian packaging systems. Running bdist_rpm on a setuptools-based package produces a usable RPM, but you have to manually translate any dependencies and supply them as bdist_rpm options, either on the command line or in the project's setup.cfg. From sklein at cpcug.org Thu Jul 26 18:34:29 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Thu, 26 Jul 2007 12:34:29 -0400 (EDT) Subject: [Distutils] Namespace packages and egg-info confusion when using bdist_rpm In-Reply-To: <20070725220835.061D23A40BF@sparrow.telecommunity.com> References: <64930.207.188.248.157.1185396145.squirrel@www.cpcug.org> <20070725220835.061D23A40BF@sparrow.telecommunity.com> Message-ID: <1537.207.188.248.157.1185467669.squirrel@www.cpcug.org> On Wed, 2007-07-25 at 18:10 -0400, Phillip J. Eby wrote: > At 04:42 PM 7/25/2007 -0400, Stanley A. Klein wrote: > >I've been trying to build rpms of enthought system components. Some of them use namespace packages. Those packages have the required __init__.py > >files containing > >"__import__('pkg_resources').declare_namespace(__name__)". > > > >According to the Setuptools Guide, these __init__.py files are not to be > >packaged when using "system" packaging (such as bdist_rpm) and when I run > >bdist_rpm they are not included in the rpms. However, there are egg-info > >directories and nspkg.pth files produced, included in the rpms, and installed in site-packages. > > So far, so good - that's exactly how it's supposed to work. > > > When I tried to run the enthought example > >applications, I had various kinds of import failures. I found that fixing > >them required putting the unpackaged __init__.py files into the relevant > >directories in the namespace packages. > > > >The various rpms were being placed in the proper locations where they should have been found for importing. There were some strange errors, such as in one case when I tried to test the imports interactively an "import a.b.c" worked, but an "import a.b.c as c" failed. > > > >Before putting in the __init__.py files, I tried removing the egg-info and > >nspkg.pth files from site_packages. That did not improve things. > > It sure as heck wouldn't. They need to be there, and the nspkg.pth files need to be processed by Python during its startup. > > > >I think setuptools is somehow not properly processing bdist_rpm if there > >are namespace packages involved. Also, the setuptools don't seem to have > >an option to not create the egg info and nspkg.pth files and/or to not package them if bdist_rpm is being used. > > This is by design. It is almost certainly the case that either: > > 1. the nspkg.pth files are not being processed when Python starts > > 2. something is deleting the namespace packages from sys.modules between the time the .pth files are processed, and the time you attempt to import them. > > To verify this, I would say, start Python with the bdist_rpms > installed as directed (i.e., don't mess with what's installed), and see if there are sys.modules entries for the namespace > package(s). If there are, then the problem is happening later - i.e., something being done by your code. If the namespace packages aren't in sys.modules, then there is a problem with the .pth files at startup; try using "python -v" to see if there are any errors shown. I think the "almost" allows more than the two alternative possibilities you provide, and I think that is the situation. Something else must be going on. I disabled the __init__.py's on the namespace packages by renaming them. A sys.modules.keys() shows the namespace packages in the list. Then I interactively did the first three statements of the test_traits.py program: import unittest from enthought.traits.api import * import enthought.traits.standard as standard The last statement failed with an error: Traceback (most recent call last): File "test_traits.py", line 23, in ? import enthought.traits.standard as standard AttributeError: 'module' object has no attribute 'traits' If I then interactively do "import enthought.traits.standard", it works without error. If I redo the same steps using python -v, I get the error message only after a lot of imports. The last output before the error message is # /usr/lib/python2.4/site-packages/enthought/traits/ui/editors.pyc matches /usr/lib/python2.4/site-packages/enthought/traits/ui/editors.py import enthought.traits.ui.editors # precompiled from /usr/lib/python2.4/site-packages/enthought/traits/ui/editors.pyc Traceback (most recent call last): File "", line 1, in ? AttributeError: 'module' object has no attribute 'traits' The file editors.py first does: from toolkit \ import toolkit which points to toolkit.py. That file contains some statements that were the failure point when I tried to run an example from the Traits UI Guide. The relevant statements in toolkit.py are: from enthought.traits.api \ import HasTraits, HasPrivateTraits, TraitError from ui_traits \ import SequenceTypes #-------------------------------------------------------------------------------# Constants: #------------------------------------------------------------------------------- # List of implemented UI toolkits TraitUIToolkits = [ 'wx', 'null' ] #------------------------------------------------------------------------------- # Data: #------------------------------------------------------------------------------- # The current GUI toolkit object being used _toolkit = None #------------------------------------------------------------------------------- # Low-level GUI toolkit selection function: #------------------------------------------------------------------------------- def toolkit ( *toolkits ): """ Selects and returns a low-level GUI toolkit. Use this function to get a reference to the current toolkit. """ global _toolkit if len( toolkits ) == 0: if _toolkit is not None: return _toolkit toolkits = TraitUIToolkits for toolkit_name in toolkits: try: package = 'enthought.traits.ui.' + toolkit_name module = __import__( package ) _toolkit = getattr( module.traits.ui, toolkit_name ).toolkit return _toolkit except ImportError: pass else: raise TraitError, ("Could not find any UI toolkit called: %s" % ', '.join( toolkits )) When I restart the interactive session as python -v, do the import statements in toolkit.py , and then do toolkit_name = "wx" package = 'enthought.traits.ui.' + toolkit_name module = __import__( package ) _toolkit = getattr( module.traits.ui, toolkit_name ).toolkit it finds the enthought.traits.ui.wx toolkit, does the imports required by the "module = __import__( package )" statement and then fails on the next statement with an error: >>> _toolkit = getattr( module.traits.ui, toolkit_name ).toolkit Traceback (most recent call last): File "", line 1, in ? AttributeError: 'module' object has no attribute 'traits' If I manually restore the __init__.py files in the three namespace packages, I don't see the namespace packages on python startup when I do the sys.modules.keys(), but I can run the test_traits.py and the examples from the Traits UI Guide without a problem. In particular, I don't get the attribute error that I got without the __init__.py files in place. Also, I don't know how Ryan May "hand-rolled" his Gentoo packages, but he said he had a similar problem and fixed it by putting in the __init__.py files. It looks like without the __init__.py files, the imports are being properly found and imported, but some lack of information somewhere is causing the getattr statement to fail. The purpose of the statement appears to be to return which lower level GUI toolkit was found and imported. I hope this provides everyone enough information to figure out what is happening. Stan Klein -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070726/9270e050/attachment.html From pje at telecommunity.com Thu Jul 26 18:46:05 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 26 Jul 2007 12:46:05 -0400 Subject: [Distutils] Namespace packages and egg-info confusion when using bdist_rpm In-Reply-To: <1537.207.188.248.157.1185467669.squirrel@www.cpcug.org> References: <64930.207.188.248.157.1185396145.squirrel@www.cpcug.org> <20070725220835.061D23A40BF@sparrow.telecommunity.com> <1537.207.188.248.157.1185467669.squirrel@www.cpcug.org> Message-ID: <20070726164426.C20083A40BF@sparrow.telecommunity.com> At 12:34 PM 7/26/2007 -0400, Stanley A. Klein wrote: >I disabled the __init__.py's on the namespace packages by renaming them. A >sys.modules.keys() shows the namespace packages in the list. Then I >interactively did the first three statements of the test_traits.py >program: > >import unittest >from enthought.traits.api import * >import enthought.traits.standard as standard > >The last statement failed with an error: > >Traceback (most recent call last): > File "test_traits.py", line 23, in ? > import enthought.traits.standard as standard >AttributeError: 'module' object has no attribute 'traits' > >If I then interactively do "import enthought.traits.standard", it works >without error. Aha. Try this. In each nspkg.pth file, add "import x" lines, where "x" is any namespace package referenced in that file. That is, if the file refers to enthought.traits, add a line saying "import enthought.traits" at the *end* of the file. Do this in every file, for every module mentioned in that file. Get rid of the __init__.py's and give it a whirl. I think that what's happening is that the .pth files I'm generating are not settting sys.modules['enthought'].traits = sys.modules['enthought.traits']; adding the import statements should fix that. If it works, I'll change setuptools to generate the files with the added import statements. Thanks for taking the time to help track this down. It looks like I only tested this with top--level namespace packages (e.g. peak.*, zope.*) and not sub-packages (like enthought.traits). From sklein at cpcug.org Thu Jul 26 19:37:57 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Thu, 26 Jul 2007 13:37:57 -0400 (EDT) Subject: [Distutils] Namespace packages and egg-info confusion when using bdist_rpm In-Reply-To: <20070726164426.C20083A40BF@sparrow.telecommunity.com> References: <64930.207.188.248.157.1185396145.squirrel@www.cpcug.org> <20070725220835.061D23A40BF@sparrow.telecommunity.com> <1537.207.188.248.157.1185467669.squirrel@www.cpcug.org> <20070726164426.C20083A40BF@sparrow.telecommunity.com> Message-ID: <1568.207.188.248.157.1185471477.squirrel@www.cpcug.org> On Thu, July 26, 2007 12:46 pm, Phillip J. Eby wrote: > At 12:34 PM 7/26/2007 -0400, Stanley A. Klein wrote: >>I disabled the __init__.py's on the namespace packages by renaming them. >> A >>sys.modules.keys() shows the namespace packages in the list. Then I >>interactively did the first three statements of the test_traits.py >>program: >> >>import unittest >>from enthought.traits.api import * >>import enthought.traits.standard as standard >> >>The last statement failed with an error: >> >>Traceback (most recent call last): >> File "test_traits.py", line 23, in ? >> import enthought.traits.standard as standard >>AttributeError: 'module' object has no attribute 'traits' >> >>If I then interactively do "import enthought.traits.standard", it works >>without error. > > Aha. Try this. In each nspkg.pth file, add "import x" lines, where > "x" is any namespace package referenced in that file. That is, if > the file refers to enthought.traits, add a line saying "import > enthought.traits" at the *end* of the file. Do this in every file, > for every module mentioned in that file. Get rid of the > __init__.py's and give it a whirl. > > I think that what's happening is that the .pth files I'm generating > are not settting sys.modules['enthought'].traits = > sys.modules['enthought.traits']; adding the import statements should > fix that. If it works, I'll change setuptools to generate the files > with the added import statements. > > Thanks for taking the time to help track this down. It looks like I > only tested this with top--level namespace packages (e.g. peak.*, > zope.*) and not sub-packages (like enthought.traits). > The only three namespace packages are enthought, enthought.traits, and enthought.traits.ui. Here is the full list of pth files: [stan at localhost ~]$ ls /usr/lib/python2.4/site-packages/enthought.*pth /usr/lib/python2.4/site-packages/enthought.etsconfig-2.0b1.dev_r12883-py2.4-nspkg.pth /usr/lib/python2.4/site-packages/enthought.io-2.0b1.dev_r12810-py2.4-nspkg.pth /usr/lib/python2.4/site-packages/enthought.naming-2.0b2.dev_r12810-py2.4-nspkg.pth /usr/lib/python2.4/site-packages/enthought.pyface-2.0b2.dev_r12984-py2.4-nspkg.pth /usr/lib/python2.4/site-packages/enthought.resource-2.0b1.dev_r12810-py2.4-nspkg.pth /usr/lib/python2.4/site-packages/enthought.resource_type-2.0b2.dev_r12810-py2.4-nspkg.pth /usr/lib/python2.4/site-packages/enthought.sweet_pickle-2.0b2.dev_r12810-py2.4-nspkg.pth /usr/lib/python2.4/site-packages/enthought.traits-2.0b2.dev_r12984-py2.4-nspkg.pth /usr/lib/python2.4/site-packages/enthought.traits.ui.wx-2.0b2.dev_r12984-py2.4-nspkg.pth /usr/lib/python2.4/site-packages/enthought.type_manager-2.0b1.dev_r12810-py2.4-nspkg.pth /usr/lib/python2.4/site-packages/enthought.util-2.0b2.dev_r12981-py2.4-nspkg.pth Most of the files looked like: import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('enthought',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('enthought',new.module('enthought')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) I left those alone, and did only the following two: [stan at localhost ~]$ cat /usr/lib/python2.4/site-packages/enthought.traits-2.0b2.dev_r12984-py2.4-nspkg.pth import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('enthought',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('enthought',new.module('enthought')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('enthought', 'traits')); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('enthought.traits',new.module('enthought.traits')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('enthought', 'traits', 'ui')); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('enthought.traits.ui',new.module('enthought.traits.ui')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) import enthought import enthought.traits import enthought.traits.ui and [stan at localhost ~]$ cat /usr/lib/python2.4/site-packages/enthought.traits.ui.wx-2.0b2.dev_r12984-py2.4-nspkg.pth import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('enthought',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('enthought',new.module('enthought')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('enthought', 'traits')); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('enthought.traits',new.module('enthought.traits')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('enthought', 'traits', 'ui')); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('enthought.traits.ui',new.module('enthought.traits.ui')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) import enthought import enthought.traits import enthought.traits.ui I had the same error. Should I have put in an "import enthought" in all the others? Stan Klein From pje at telecommunity.com Thu Jul 26 20:01:00 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 26 Jul 2007 14:01:00 -0400 Subject: [Distutils] Namespace packages and egg-info confusion when using bdist_rpm In-Reply-To: <1568.207.188.248.157.1185471477.squirrel@www.cpcug.org> References: <64930.207.188.248.157.1185396145.squirrel@www.cpcug.org> <20070725220835.061D23A40BF@sparrow.telecommunity.com> <1537.207.188.248.157.1185467669.squirrel@www.cpcug.org> <20070726164426.C20083A40BF@sparrow.telecommunity.com> <1568.207.188.248.157.1185471477.squirrel@www.cpcug.org> Message-ID: <20070726175840.249E33A403D@sparrow.telecommunity.com> At 01:37 PM 7/26/2007 -0400, Stanley A. Klein wrote: >I had the same error. Should I have put in an "import enthought" in all >the others? No, apparently the manual import doesn't help. Presumably the ones you changed need something more like this: import enthought; enthought.traits = sys.modules['enthought.traits'] import enthought.traits; enthought.traits.ui = sys.modules['enthought.traits.ui'] I thought the import would be sufficient, but apparently it's not. :( Again, let me know if it works so I can change setuptools accordingly. From sklein at cpcug.org Thu Jul 26 21:05:09 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Thu, 26 Jul 2007 15:05:09 -0400 (EDT) Subject: [Distutils] Best practices on distributing documentation and examples with eggs? Message-ID: <1599.207.188.248.157.1185476709.squirrel@www.cpcug.org> On Thu, July 26, 2007 12:46 pm, Phillip J. Eby wrote: At 03:02 AM 7/26/2007 -0500, Dave Peterson wrote: >>Over on the enthought-dev mailing list we're having a bit of a >>discussion on what the best way to distribute documenation and examples >>is for projects that we distribute binary eggs for. The general >>consensus is that it would be very nice indeed if there was a way to >>generate a tarball, or platform install, of just documentation and >>examples so that people wouldn't need to download a full, presumably >>significantly larger, source tarball. Another option would be that >>eggs included the documentation and examples and that, during the >>installation of the egg, those docs and examples were relocated to a >>common location (outside of the zip) to make access by users more >>convenient. This latter idea is similar to how Ruby Gems deal with docs. >> >>I don't claim to be a distutils or setuptools guru, so it shouldn't be >>too surprising that I can't seem to find anything about a setuptools or >>distutils command to do either of these. Am I missing something? >There are a few different ways you could do this. The easiest would >be to put a docs subdirectory either inside one of your packages or >inside your .egg-info directory, then use the pkg_resources resource >or metadata APIs to list and extract them. One advantage to using >something like .egg-info/docs would be that this could perhaps be >recognized by some sort of "standard" tools in the future. >If not, does it seem like something that might be worthy of putting into >setuptools? >It might be worth establishing convention(s) for, yes. Tools could >then be developed around them, including perhaps changes to easy_install. This relates to a question I asked this list earlier this month (but didn't get a response). For Linux systems the Linux Standards Base references the Unix Filesystem Hierarchy Standard (that applies to all Unix systems as well). The FHS specifies that documentation files (other than specially formatted items like man pages) go into /usr/share/doc/package_name_and_version These sometimes include examples, demos, and similar files. For example, the docs on my FC5 system for inkscape go in /usr/share/doc/inkscape-0.45.1. In that case the doc files are a typical minimal set: /usr/share/doc/inkscape-0.45.1/AUTHORS /usr/share/doc/inkscape-0.45.1/COPYING /usr/share/doc/inkscape-0.45.1/ChangeLog /usr/share/doc/inkscape-0.45.1/NEWS /usr/share/doc/inkscape-0.45.1/README and the examples, tutorials, clipart, and many miscellaneous files are in /usr/share/inkscape. The actual executables are in /usr/bin. In some cases the documentation is created as a separate package. For example, Python does this for its html-based docs and on my FC5 system, the python html docs are in /usr/share/doc/python-docs-2.4.3/. Similar considerations go to configuration files that are supposed to go into /etc. There are a number of other rules, and they are generally observed by systems that use rpm and deb packaging such as Fedora and Ubuntu. I couldn't figure out how to make this happen when using bdist_rpm, which is why I asked the earlier question. It seems to me that the only way using current Python packaging features would be to put the docs somewhere in the Python site-packages hierarchy and do a post-install scripted move to where they belong under the LSB/FHS rules. It would be preferable to get them to go where they belong without the need for post-install scripting. Stan Klein From barry at python.org Thu Jul 26 20:55:29 2007 From: barry at python.org (Barry Warsaw) Date: Thu, 26 Jul 2007 14:55:29 -0400 Subject: [Distutils] Best practices on distributing documentation and examples with eggs? In-Reply-To: <20070726155222.18CB43A40D7@sparrow.telecommunity.com> References: <46A854FA.3080906@enthought.com> <20070726155222.18CB43A40D7@sparrow.telecommunity.com> Message-ID: <660761AD-A57D-4D81-A0F5-E328043D8B34@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jul 26, 2007, at 11:52 AM, Phillip J. Eby wrote: > There are a few different ways you could do this. The easiest would > be to put a docs subdirectory either inside one of your packages or > inside your .egg-info directory, then use the pkg_resources resource > or metadata APIs to list and extract them. One advantage to using > something like .egg-info/docs would be that this could perhaps be > recognized by some sort of "standard" tools in the future. I always thought .egg-info was just an artifact of the build. Recommending that things get put there means it should get version controlled. But I've found that occasionally I have to blow away .egg-info files to get 'setup.py develop' to work properly (i.e. like a 'make clean' step). - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRqjuInEjvBPtnXfVAQJLrAP/S/q2EFM7gnokBTsdgoEjXXn1LXtEWwkP OJaPu4a0jymVy7+R0o9eGXSorOaAO0vN/ay7LGXvkllWTv0pOLa6aQ01cQlEuuhk GLdw+eLVOKmTWnhv4448YhgD/r74ywBWJ9rYD+Ygg93bNHbyTZU/zaoCc2SLpzYq CrVpeK4ArBM= =KfaK -----END PGP SIGNATURE----- From sklein at cpcug.org Thu Jul 26 21:17:52 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Thu, 26 Jul 2007 15:17:52 -0400 (EDT) Subject: [Distutils] Namespace packages and egg-info confusion when using bdist_rpm In-Reply-To: <20070726175840.249E33A403D@sparrow.telecommunity.com> References: <64930.207.188.248.157.1185396145.squirrel@www.cpcug.org> <20070725220835.061D23A40BF@sparrow.telecommunity.com> <1537.207.188.248.157.1185467669.squirrel@www.cpcug.org> <20070726164426.C20083A40BF@sparrow.telecommunity.com> <1568.207.188.248.157.1185471477.squirrel@www.cpcug.org> <20070726175840.249E33A403D@sparrow.telecommunity.com> Message-ID: <1609.207.188.248.157.1185477472.squirrel@www.cpcug.org> It worked. Problem solved. Thanks. Stan Klein On Thu, July 26, 2007 2:01 pm, Phillip J. Eby wrote: > At 01:37 PM 7/26/2007 -0400, Stanley A. Klein wrote: >>I had the same error. Should I have put in an "import enthought" in all >>the others? > > No, apparently the manual import doesn't help. Presumably the ones > you changed need something more like this: > > import enthought; enthought.traits = sys.modules['enthought.traits'] > import enthought.traits; enthought.traits.ui = > sys.modules['enthought.traits.ui'] > > I thought the import would be sufficient, but apparently it's not. :( > > Again, let me know if it works so I can change setuptools accordingly. > -- From dpeterson at enthought.com Thu Jul 26 21:13:20 2007 From: dpeterson at enthought.com (Dave Peterson) Date: Thu, 26 Jul 2007 14:13:20 -0500 Subject: [Distutils] Best practices on distributing documentation and examples with eggs? In-Reply-To: <1599.207.188.248.157.1185476709.squirrel@www.cpcug.org> References: <1599.207.188.248.157.1185476709.squirrel@www.cpcug.org> Message-ID: <46A8F250.5090509@enthought.com> Stanley A. Klein wrote: > On Thu, July 26, 2007 12:46 pm, Phillip J. Eby wrote: > At 03:02 AM 7/26/2007 -0500, Dave Peterson wrote: > >>> Over on the enthought-dev mailing list we're having a bit of a >>> discussion on what the best way to distribute documenation and examples >>> is for projects that we distribute binary eggs for. The general >>> consensus is that it would be very nice indeed if there was a way to >>> generate a tarball, or platform install, of just documentation and >>> examples so that people wouldn't need to download a full, presumably >>> significantly larger, source tarball. Another option would be that >>> eggs included the documentation and examples and that, during the >>> installation of the egg, those docs and examples were relocated to a >>> common location (outside of the zip) to make access by users more >>> convenient. This latter idea is similar to how Ruby Gems deal with docs. >>> >>> I don't claim to be a distutils or setuptools guru, so it shouldn't be >>> too surprising that I can't seem to find anything about a setuptools or >>> distutils command to do either of these. Am I missing something? >>> > > >> There are a few different ways you could do this. The easiest would >> be to put a docs subdirectory either inside one of your packages or >> inside your .egg-info directory, then use the pkg_resources resource >> or metadata APIs to list and extract them. One advantage to using >> something like .egg-info/docs would be that this could perhaps be >> recognized by some sort of "standard" tools in the future. >> > > >> If not, does it seem like something that might be worthy of putting into >> setuptools? >> > > >> It might be worth establishing convention(s) for, yes. Tools could >> then be developed around them, including perhaps changes to easy_install. >> > > > This relates to a question I asked this list earlier this month (but > didn't get a response). For Linux systems the Linux Standards Base > references the Unix Filesystem Hierarchy Standard (that applies to all > Unix systems as well). The FHS specifies that documentation files (other > than specially formatted items like man pages) go into > > /usr/share/doc/package_name_and_version > > These sometimes include examples, demos, and similar files. For example, > the docs on my FC5 system for inkscape go in > /usr/share/doc/inkscape-0.45.1. In that case the doc files are a typical > minimal set: > > /usr/share/doc/inkscape-0.45.1/AUTHORS > /usr/share/doc/inkscape-0.45.1/COPYING > /usr/share/doc/inkscape-0.45.1/ChangeLog > /usr/share/doc/inkscape-0.45.1/NEWS > /usr/share/doc/inkscape-0.45.1/README > > and the examples, tutorials, clipart, and many miscellaneous files are in > /usr/share/inkscape. The actual executables are in /usr/bin. > > In some cases the documentation is created as a separate package. For > example, Python does this for its html-based docs and on my FC5 system, > the python html docs are in /usr/share/doc/python-docs-2.4.3/. Similar > considerations go to configuration files that are supposed to go into > /etc. There are a number of other rules, and they are generally observed > by systems that use rpm and deb packaging such as Fedora and Ubuntu. > > I couldn't figure out how to make this happen when using bdist_rpm, which > is why I asked the earlier question. It seems to me that the only way > using current Python packaging features would be to put the docs somewhere > in the Python site-packages hierarchy and do a post-install scripted move > to where they belong under the LSB/FHS rules. It would be preferable to > get them to go where they belong without the need for post-install > scripting. > In further discussions at Enthought, we're now thinking that creating a 'docs' dir as a top-level under python itself makes sense, and then the installation of eggs could copy docs and/or examples there in a manner similar to how it handles scripts into the platforms scripts dir. It would make sense that they should be put in dirs named with a format like - so that the user could differentiate the docs for each project they installed. If this was done, it seems like it wouldn't be too hard for Stan, and other rpm builders, to either create symbolic links or copy these to /usr/share/doc in order to maintain compatibility with the LSB. One other thought we've been throwing around is an idea of adding something to easy-install so that users can still get a minimal binary distribution if they want, and then, if they want docs, run a command like: easy_install --get-docs enthought.traits where easy_install would then introspect the enthought.traits egg and figure out how to retrieve the docs and install them into ../python/docs. If we found time to work on implementing something, is there a preference for which path we should pursue? -- Dave From arve.knudsen at gmail.com Thu Jul 26 21:47:56 2007 From: arve.knudsen at gmail.com (Arve Knudsen) Date: Thu, 26 Jul 2007 21:47:56 +0200 Subject: [Distutils] Regeneration of source manifest In-Reply-To: <20070725221733.7D1593A403D@sparrow.telecommunity.com> References: <20070725221733.7D1593A403D@sparrow.telecommunity.com> Message-ID: On 7/26/07, Phillip J. Eby wrote: > > At 10:51 PM 7/25/2007 +0200, Arve Knudsen wrote: > >I take it there is no interest in fixing this problem? Or is it > >simply too much hassle?? > > > >Arve > > > >On 7/10/07, Arve Knudsen < > >arve.knudsen at gmail.com> wrote: > >Hello > > > >As far as I can tell setuptools does not always regenerate the > >source manifest when MANIFEST.in changes, contrary to what the > >documentation says. That is, if I add a directive to exclude files > >with the .txt extension to MANIFEST.in (after having previously > >including them), and run the sdist command, such files are still > >included. I have to remove the .egg-info for things to work as > >expected. This must clearly be a bug? > > > >Regards, > >Arve Knudsen > > Your email got lost in the flurry of PyPI-related emails. I am > looking at the code and don't see how this condition could be > produced, at least not the way you seem to be describing it, because > the template is processed after the old filelist is read in. So if > there are exclusion commands in the template, these should *always* > be applied to the resulting SOURCES.txt. > > Could you give me the exact old MANIFEST.in contents, the new > MANIFEST.in contents, and the egg-info/SOURCES.txt? > > Better yet, can you give me a small setup.py and steps that I can use > to reproduce the problem? Sorry, I can't seem to reproduce the exact problem (it doesn't help that I don't recollect the exact case). What I do notice when testing however, is that if I first add a directive in MANIFEST.in to include ez_setup.py and run sdist, remove the directive and run sdist again, ez_setup.py is still in SOURCES.txt. So it seems that sdist somehow remembers previous include directives for files that aren't included by default. Arve -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070726/5f72c8f7/attachment.html From pje at telecommunity.com Thu Jul 26 21:50:56 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 26 Jul 2007 15:50:56 -0400 Subject: [Distutils] Best practices on distributing documentation and examples with eggs? In-Reply-To: <46A8F250.5090509@enthought.com> References: <1599.207.188.248.157.1185476709.squirrel@www.cpcug.org> <46A8F250.5090509@enthought.com> Message-ID: <20070726194837.60CF73A40BF@sparrow.telecommunity.com> At 02:13 PM 7/26/2007 -0500, Dave Peterson wrote: >In further discussions at Enthought, we're now thinking that creating a >'docs' dir as a top-level under python itself makes sense, and then the >installation of eggs could copy docs and/or examples there in a manner >similar to how it handles scripts into the platforms scripts dir. It >would make sense that they should be put in dirs named with a format >like - so that the user could differentiate the >docs for each project they installed. If this was done, it seems like >it wouldn't be too hard for Stan, and other rpm builders, to either >create symbolic links or copy these to /usr/share/doc in order to >maintain compatibility with the LSB. This isn't a bad idea, although I think that there are some hurdles that need to be worked out. My general thought is that, were easy_install to support documentation, it would need to take an option to determine the documentation base directory, and have a --no-docs install option. One issue is that under most RPM systems, package names are more like 'python-foobar-1.2', and 'foobar' may not even be the exact distutils name of the package. So, installing the docs to the correct location in the general case may be more complex. Note, however, that bdist_rpm offers a --doc-files option, which of course can be configured via setup.cfg. I have no idea how it works, though, so don't ask me. :) >One other thought we've been throwing around is an idea of adding >something to easy-install so that users can still get a minimal binary >distribution if they want, and then, if they want docs, run a command like: > easy_install --get-docs enthought.traits >where easy_install would then introspect the enthought.traits egg and >figure out how to retrieve the docs and install them into ../python/docs. Actually, if the default is for easy_install to install documentation, then simply rerunning easy_install should suffice (easy_install on an already-installed package makes the package active and installs scripts, so installing docs should work the same way.) From pje at telecommunity.com Thu Jul 26 21:44:55 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 26 Jul 2007 15:44:55 -0400 Subject: [Distutils] Best practices on distributing documentation and examples with eggs? In-Reply-To: <660761AD-A57D-4D81-A0F5-E328043D8B34@python.org> References: <46A854FA.3080906@enthought.com> <20070726155222.18CB43A40D7@sparrow.telecommunity.com> <660761AD-A57D-4D81-A0F5-E328043D8B34@python.org> Message-ID: <20070726194836.89D0B3A403D@sparrow.telecommunity.com> At 02:55 PM 7/26/2007 -0400, Barry Warsaw wrote: >On Jul 26, 2007, at 11:52 AM, Phillip J. Eby wrote: > >>There are a few different ways you could do this. The easiest would >>be to put a docs subdirectory either inside one of your packages or >>inside your .egg-info directory, then use the pkg_resources resource >>or metadata APIs to list and extract them. One advantage to using >>something like .egg-info/docs would be that this could perhaps be >>recognized by some sort of "standard" tools in the future. > >I always thought .egg-info was just an artifact of the build. No - it's also a place for metadata that doesn't live under a specific package directory. >Recommending that things get put there means it should get version >controlled. Yep. > But I've found that occasionally I have to blow >away .egg-info files to get 'setup.py develop' to work properly (i.e. >like a 'make clean' step). Really? I'd consider that to be a bug. Could you perhaps be more specific? Note that if you are using a revision control system, and setuptools deletes a file from there, it might get restored by your revision control, and that could potentially be a problem unless you also manually delete the file. From barry at python.org Thu Jul 26 22:07:33 2007 From: barry at python.org (Barry Warsaw) Date: Thu, 26 Jul 2007 16:07:33 -0400 Subject: [Distutils] Best practices on distributing documentation and examples with eggs? In-Reply-To: <20070726194836.89D0B3A403D@sparrow.telecommunity.com> References: <46A854FA.3080906@enthought.com> <20070726155222.18CB43A40D7@sparrow.telecommunity.com> <660761AD-A57D-4D81-A0F5-E328043D8B34@python.org> <20070726194836.89D0B3A403D@sparrow.telecommunity.com> Message-ID: <0D3D78A3-DBC6-42D6-AFD9-3973C08EA040@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jul 26, 2007, at 3:44 PM, Phillip J. Eby wrote: > At 02:55 PM 7/26/2007 -0400, Barry Warsaw wrote: > >> On Jul 26, 2007, at 11:52 AM, Phillip J. Eby wrote: >> >>> There are a few different ways you could do this. The easiest would >>> be to put a docs subdirectory either inside one of your packages or >>> inside your .egg-info directory, then use the pkg_resources resource >>> or metadata APIs to list and extract them. One advantage to using >>> something like .egg-info/docs would be that this could perhaps be >>> recognized by some sort of "standard" tools in the future. >> >> I always thought .egg-info was just an artifact of the build. > > No - it's also a place for metadata that doesn't live under a > specific package directory. > >> Recommending that things get put there means it should get version >> controlled. > > Yep. > > >> But I've found that occasionally I have to blow >> away .egg-info files to get 'setup.py develop' to work properly (i.e. >> like a 'make clean' step). > > Really? I'd consider that to be a bug. Could you perhaps be more > specific? Now that I know that, I'll investigate further next time I see something 'weird'. I'll keep you posted. > Note that if you are using a revision control system, and > setuptools deletes a file from there, it might get restored by your > revision control, and that could potentially be a problem unless > you also manually delete the file. When does setuptools delete a file there? - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRqj/BXEjvBPtnXfVAQICXQP9F4Iv3ew0BufrhfZt1WYUCnqNF08DiXCK xSvX2KrCc3vV6Z67kFNHJsdAZnJzQR8N0x2n82TXo/q0/36Jcpyt8rp7dIdvD93I mDZtyyk2cFxuRW78xOtYw3lpUL75e1YbVMzSyhjwaXMdnKNYTfqRyfHkaR/dBz0H hKbHNuroUWg= =/4VX -----END PGP SIGNATURE----- From pje at telecommunity.com Thu Jul 26 22:18:07 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 26 Jul 2007 16:18:07 -0400 Subject: [Distutils] Best practices on distributing documentation and examples with eggs? In-Reply-To: <0D3D78A3-DBC6-42D6-AFD9-3973C08EA040@python.org> References: <46A854FA.3080906@enthought.com> <20070726155222.18CB43A40D7@sparrow.telecommunity.com> <660761AD-A57D-4D81-A0F5-E328043D8B34@python.org> <20070726194836.89D0B3A403D@sparrow.telecommunity.com> <0D3D78A3-DBC6-42D6-AFD9-3973C08EA040@python.org> Message-ID: <20070726201548.4F70A3A403D@sparrow.telecommunity.com> At 04:07 PM 7/26/2007 -0400, Barry Warsaw wrote: >On Jul 26, 2007, at 3:44 PM, Phillip J. Eby wrote: >>Note that if you are using a revision control system, and >>setuptools deletes a file from there, it might get restored by your >>revision control, and that could potentially be a problem unless >>you also manually delete the file. > >When does setuptools delete a file there? Whenever a setuptools "egg_info.writers" plugin does. :) More usefully, most files generated by setup() arguments (e.g. entry_points) will be deleted the next time you run egg_info, if you removed that argument from setup(). From mhammond at skippinet.com.au Fri Jul 27 08:18:15 2007 From: mhammond at skippinet.com.au (Mark Hammond) Date: Fri, 27 Jul 2007 16:18:15 +1000 Subject: [Distutils] distutils.util.get_platform() for Windows In-Reply-To: <7CDD8143-AE94-410C-8F93-ECF6124DDD77@zooko.com> Message-ID: <10fc01c7d015$eb24b310$090a0a0a@enfoldsystems.local> > > http://en.wikipedia.org/wiki/X86-64 notes that 'x64' is a common > > name, so > > how does 'win-x64' and 'win-ia64' sound as a compromise? > I'm happy > > to let > > any other informal "votes" make a final decision though... > > As one who is forever switching back and forth among operating > systems (Windows, sometimes with cygwin, Linux, Mac OS X, > Solaris), I > would be happy to see the standard names that I'm used to, i.e. the > output of config.guess [1], which I think is "x86_64" for the > architecture and "win32" for the operating system. I agree in principle - if config.guess has standardized, we should adopt it. > Actually config.guess doesn't run on Windows unless with cygwin or > mingw, in which case it outputs "cygwin" or "mingw" for the > operating > system. But it definitely outputs "x86_64" on other operating > systems on x86-64 architectures. My Vista x64 box has a (32bit - can't locate a 64bit) cygwin installed. This is what I get: sh-3.2$ /usr/share/automake-1.10/config.guess i686-pc-cygwin sh-3.2$ uname -a CYGWIN_NT-6.0-WOW64 vista-64 1.5.24(0.156/4/2) 2007-01-31 10:57 i686 Cygwin My XP x86 box gives the exact same result for config.guess (i686), so I'm a little confused by this. Does anyone know what I am missing? But either way, I have to concede that my preference for x86 is somewhat arbitrary, so x86_64 appears to win the vote (although polls remain open until checkin time ) I've created a patch at http://python.org/sf/1761786 with x86_64 - I'd welcome any feedback etc. Note the patch also changes bdist_msi to use get_platform() for the final .msi created. Cheers, Mark From sklein at cpcug.org Fri Jul 27 21:47:08 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Fri, 27 Jul 2007 15:47:08 -0400 (EDT) Subject: [Distutils] How can I get setuptools to include files beyond what find_packages() finds? Message-ID: <1944.207.188.248.157.1185565628.squirrel@www.cpcug.org> I'm trying to package an rpm for enthought kiva. The regular setup.py uses the numpy distutils because of some cpp functions that have to be compiled and are somehow tied to numpy. Someone recently did a "build in place" program that uses the existing setup.py (that I renamed setup.original.py) and builds the .so files in the regular source directory hierarchy. I did that and then tried to run setuptools (python setup.py bdist_rpm) using a straightforward setup.py. It included all the python files but missed the *.so files. I can run kiva examples if I manually put the *.so files into the proper place in site-packages, so I know they are needed. I tried including packages = find_packages(), package_data = {'': ['*.so']}, in the setup.py, but it still missed the *.so files. What am I doing wrong? Stan Klein From pje at telecommunity.com Fri Jul 27 21:55:06 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 27 Jul 2007 15:55:06 -0400 Subject: [Distutils] How can I get setuptools to include files beyond what find_packages() finds? In-Reply-To: <1944.207.188.248.157.1185565628.squirrel@www.cpcug.org> References: <1944.207.188.248.157.1185565628.squirrel@www.cpcug.org> Message-ID: <20070727195248.774E73A40A8@sparrow.telecommunity.com> At 03:47 PM 7/27/2007 -0400, Stanley A. Klein wrote: >I'm trying to package an rpm for enthought kiva. The regular setup.py >uses the numpy distutils because of some cpp functions that have to be >compiled and are somehow tied to numpy. > >Someone recently did a "build in place" program that uses the existing >setup.py (that I renamed setup.original.py) and builds the .so files in >the regular source directory hierarchy. I did that and then tried to run >setuptools (python setup.py bdist_rpm) using a straightforward setup.py. >It included all the python files but missed the *.so files. I can run >kiva examples if I manually put the *.so files into the proper place in >site-packages, so I know they are needed. > >I tried including > > packages = find_packages(), > package_data = {'': ['*.so']}, > >in the setup.py, but it still missed the *.so files. > >What am I doing wrong? You don't want package_data. setup() needs to actually list the extensions. More precisely, the distutils "install_lib" command needs to install the extensions when requested to do so by bdist_rpm. The preceding applies to any distutils setup, not just setuptools. From steve at holdenweb.com Sat Jul 28 14:42:13 2007 From: steve at holdenweb.com (Steve Holden) Date: Sat, 28 Jul 2007 08:42:13 -0400 Subject: [Distutils] Cygwin (1.5.24) PIL(1.1.6) Install Problem: tight looping In-Reply-To: References: <469827D7.5090400@holdenweb.com> <4699CBD4.8090007@users.sourceforge.net> Message-ID: <46AB39A5.3080007@holdenweb.com> Steve Holden wrote: > Yaakov (Cygwin Ports) wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA256 >> >> Steve Holden wrote: >>> I have tried to install it, but each time I do I see the first compile >>> command start to run and the process hangs. If I CTRL/C the setup.py >>> and restart it, it begins the next compile (implying that the previous >>> one succeeded), which again hangs. I see python.exe still running for >>> each install run I have interrupted, which I have to kill manually. >> >> Try rebaseall. >> > Unfortunately I did that before I posted. Sorry, should have mentioned it. > [Note for distutils-sig readers: I am trying to track down a build issue with version 1.1.6 of the PIL under Cygwin - nobody else reports any successful or failed experiences building this package]. Here's the output from a failed setup run with a couple of debug prints inserted which should report how sub-process termination occurred - it hangs after this output. $ python setup.py build running build running build_py running build_ext building '_imaging' extension SPAWN: ['gcc', '-fno-strict-aliasing', '-DNDEBUG', '-g', '-O3', '-Wall', '-Wstrict-prototypes', '-DHAVE_LIBZ', '-IlibImaging', '-I/usr/include', '-I/usr/include/python2.5', '-c', 'libImaging/Chops.c', '-o', 'build/temp.cygwin-1.5.24-i686-2.5/libImaging/Chops.o'] PATH? 1 V: 0 D:0 gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -DHAVE_LIBZ - IlibImaging -I/usr/include -I/usr/include/python2.5 -c libImaging/Chops.c -o bui ld/temp.cygwin-1.5.24-i686-2.5/libImaging/Chops.o Are we done yet? Waiting on pid 3280 As a further follow up, I extracted the _spawn_all function and ran it under command line control. Everything seems to work fine with other subtasks, so I am wondering whether this is a failure specific to gcc, which would seem kind of unlikely. So I ran the same compile using my test function standing alone, and see sholden at bigboy ~/Imaging-1.1.6 $ python ~/Projects/Python/spawntest.py gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -DHAVE_LIBZ -IlibImaging -I/usr/include -I/usr/include/python2.5 -c libImaging/Chops.c -o build/temp.cygwin-1.5.24-i686-2.5/libImaging/Chops.o Are we done yet? Waiting on pid 3244 Got pid, status 3244 0 Got WIFEXITED 0 So it appears unlikely to be gcc-specific, leaving me wondering what exactly is the difference between the build environment and my tests. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden --------------- Asciimercial ------------------ Get on the web: Blog, lens and tag the Internet Many services currently offer free registration ----------- Thank You for Reading ------------- From sklein at cpcug.org Mon Jul 30 21:15:55 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Mon, 30 Jul 2007 15:15:55 -0400 (EDT) Subject: [Distutils] How can I get setuptools to include files beyond what find_packages() finds? In-Reply-To: <20070727195248.774E73A40A8@sparrow.telecommunity.com> References: <1944.207.188.248.157.1185565628.squirrel@www.cpcug.org> <20070727195248.774E73A40A8@sparrow.telecommunity.com> Message-ID: <3182.207.188.248.157.1185822955.squirrel@www.cpcug.org> On Fri, 2007-07-27 at 15:55 -0400, Phillip J. Eby wrote: > At 03:47 PM 7/27/2007 -0400, Stanley A. Klein wrote: > >I'm trying to package an rpm for enthought kiva. The regular setup.py uses the numpy distutils because of some cpp functions that have to be compiled and are somehow tied to numpy. > > > >Someone recently did a "build in place" program that uses the existing setup.py (that I renamed setup.original.py) and builds the .so files in the regular source directory hierarchy. I did that and then tried to run > >setuptools (python setup.py bdist_rpm) using a straightforward setup.py. > >It included all the python files but missed the *.so files. I can run kiva examples if I manually put the *.so files into the proper place in site-packages, so I know they are needed. > > > >I tried including > > > > packages = find_packages(), > > package_data = {'': ['*.so']}, > > > >in the setup.py, but it still missed the *.so files. > > > >What am I doing wrong? > > You don't want package_data. setup() needs to actually list the extensions. More precisely, the distutils "install_lib" command needs to install the extensions when requested to do so by bdist_rpm. > > The preceding applies to any distutils setup, not just setuptools. This is very confusing. The documentation of distutils is very sparse in this area and doesn't really cover this issue. I looked at the code and couldn't figure out anything. I got a little bit out of the "--help install_lib" command line option, but couldn't figure it out from that. It appears that for extensions, distutils expects to build/compile them. There is some kind of option called skip-build that I don't know how to invoke and am not sure how will affect optimize=1 in the setup.cfg file, that calls for it to build the pyc and pyo files that are needed. I also couldn't figure out how to specify the .so files to be included. I don't need to build the .so files. They are already built. That had to be done using the build-in-place and the numpy distutils for reasons I don't fully understand but are related to the use of numpy. The reason I tried to use package_data was because I wanted setuptools to include the files the same as would be done if they were data files -- just put them into the package as they are. The find_packages() function identifies .py files to be included in the package. I want the already-built .so files to be included also. How do I make that happen? Stan Klein -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070730/15fd699d/attachment.html From pje at telecommunity.com Mon Jul 30 22:44:59 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 30 Jul 2007 16:44:59 -0400 Subject: [Distutils] How can I get setuptools to include files beyond what find_packages() finds? In-Reply-To: <3182.207.188.248.157.1185822955.squirrel@www.cpcug.org> References: <1944.207.188.248.157.1185565628.squirrel@www.cpcug.org> <20070727195248.774E73A40A8@sparrow.telecommunity.com> <3182.207.188.248.157.1185822955.squirrel@www.cpcug.org> Message-ID: <20070730204237.D00953A406B@sparrow.telecommunity.com> At 03:15 PM 7/30/2007 -0400, Stanley A. Klein wrote: >I don't need to build the .so files. They are already built. That had to >be done using the build-in-place and the numpy distutils for reasons I >don't fully understand but are related to the use of numpy. Have you tried building them with setuptools, using the numpy distutils 'build_ext' command, using: setup( cmdclass = dict(build_ext = numpy_build_ext_class_here), ext_modules = list_of_Extension_objects, ... ) Unless there is a radical difference between numpy distutils and the regular distutils, you should be able to do this. Just find numpy's "build_ext" class, and define the appropriate Extension() objects (for the things to build) in your setup script. Setuptools will then delegate the building to numpy, but handle the installing itself. Again, this is assuming that numpy's distutils extensions don't do anything unfriendly like completely redefine how extension objects work or assume that their commands will be only mixed with other numpy commands. (Setuptools doesn't make such assumptions, and tries to leave the normal distutils stuff alone as much as possible.) From robert.kern at gmail.com Tue Jul 31 00:17:40 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 30 Jul 2007 17:17:40 -0500 Subject: [Distutils] How can I get setuptools to include files beyond what find_packages() finds? In-Reply-To: <20070730204237.D00953A406B@sparrow.telecommunity.com> References: <1944.207.188.248.157.1185565628.squirrel@www.cpcug.org> <20070727195248.774E73A40A8@sparrow.telecommunity.com> <3182.207.188.248.157.1185822955.squirrel@www.cpcug.org> <20070730204237.D00953A406B@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 03:15 PM 7/30/2007 -0400, Stanley A. Klein wrote: >> I don't need to build the .so files. They are already built. That had to >> be done using the build-in-place and the numpy distutils for reasons I >> don't fully understand but are related to the use of numpy. > > Have you tried building them with setuptools, using the numpy > distutils 'build_ext' command, using: > > setup( > cmdclass = dict(build_ext = numpy_build_ext_class_here), > ext_modules = list_of_Extension_objects, > ... > ) > > Unless there is a radical difference between numpy distutils and the > regular distutils, you should be able to do this. Just find numpy's > "build_ext" class, and define the appropriate Extension() objects > (for the things to build) in your setup script. Setuptools will then > delegate the building to numpy, but handle the installing itself. > > Again, this is assuming that numpy's distutils extensions don't do > anything unfriendly like completely redefine how extension objects > work or assume that their commands will be only mixed with other > numpy commands. (Setuptools doesn't make such assumptions, and tries > to leave the normal distutils stuff alone as much as possible.) I think we're getting into confusing territory by trying to get workarounds for workarounds. Let me try to take us a step back and focus on the initial problem which is that bdist_rpm is not working with enthought.kiva. The existing setup script already does build extensions just fine; they're just not being picked up by bdist_rpm. A suggestion from a coworker of mine prompted Stanley to look at using a script that we have for building enthought.kiva inplace (there are a few more options that are needed beyond "python setup.py develop"); however, it wasn't really a suggestion to use that as basis for building an RPM. numpy.distutils extends distutils in three ways which are important for enthought.kiva: * automatically adds the location of the numpy headers to the include_dirs of Extensions. (easily replaced) * adds a build_src command that allows users to give Python functions in the sources list of an Extension. These functions will be called to actually generate the real source files. (hard to replace) * allows subpackages to have their own build information which is assembled by the top-level setup.py script. This is mostly legacy from when the enthought package was monolithic and doesn't strictly need to continue. I won't go into details since I don't think it's part of the problem. (straightforward, but time-consuming to replace) numpy.distutils tries hard to not step on setuptools' toes. We actually check if setuptools is in sys.modules and use its command classes instead of distutils' as the base classes for our commands. However, it's possible that neglect of our bdist_rpm implementation has caused the implementations to diverge and some toe-stepping has taking place. The main problem is that bdist_rpm is not working on enthought.kiva. Most likely, this is the fault of numpy.distutils. However, this is a bug that needs to be caught and fixed. Working around it by doing an --inplace build and then trying to include the extension modules as package_data is not likely to work and is not a solution. I'm not usually a Redhat guy, so I don't have much experience with bdist_rpm; however, numpy.distutils has had problems with bdist_rpm in the past. I'm trying to get an environment working on a Redhat machine, and will try to build an RPM for enthought.kiva and try to see the problem first-hand. I've looked over Stanley's emails on the subject, and don't see enough information for me to really pin down the problem. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pje at telecommunity.com Tue Jul 31 00:44:41 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 30 Jul 2007 18:44:41 -0400 Subject: [Distutils] How can I get setuptools to include files beyond what find_packages() finds? In-Reply-To: References: <1944.207.188.248.157.1185565628.squirrel@www.cpcug.org> <20070727195248.774E73A40A8@sparrow.telecommunity.com> <3182.207.188.248.157.1185822955.squirrel@www.cpcug.org> <20070730204237.D00953A406B@sparrow.telecommunity.com> Message-ID: <20070730224239.A546A3A406B@sparrow.telecommunity.com> At 05:17 PM 7/30/2007 -0500, Robert Kern wrote: >The main problem is that bdist_rpm is not working on enthought.kiva. Most >likely, this is the fault of numpy.distutils. However, this is a bug >that needs >to be caught and fixed. Working around it by doing an --inplace build and then >trying to include the extension modules as package_data is not likely to work >and is not a solution. Yep; that's why I recommend finding the actual problem -- thanks for pointing it out more specifically. >I'm not usually a Redhat guy, so I don't have much experience with bdist_rpm; >however, numpy.distutils has had problems with bdist_rpm in the past. The keys to making bdist_rpm work for a distutils extension are: 1. Make sure you install stuff to the right place 2. Implement get_outputs() correctly The most likely source of the problem is if your get_outputs() isn't listing the .so files; in that case they'd won't be listed in INSTALLED_FILES, which is used by bdist_rpm. The other possibility is that the files aren't getting installed by install_lib, as install_lib expects to just copy the build tree to the target. If the built files aren't in the right place, then they won't get installed even if they're listed in the files. You can easily test this by running an install command using --root=/some/dir --record=SOME_FILE. If this doesn't install something like "/some/dir/usr/lib/python2.x/site-packages/whatever/something.so", or the file isn't listed in SOME_FILE, you can track it down further from there -- it's not a problem specific to bdist_rpm in that case. You can also see if it's correct with or without setuptools, to see if the interaction is in fact there. >I'm trying >to get an environment working on a Redhat machine, and will try to >build an RPM >for enthought.kiva and try to see the problem first-hand. I've looked over >Stanley's emails on the subject, and don't see enough information for me to >really pin down the problem. I'd suggest trying an install --root/--record test first, since if that doesn't work right, you know RPMs aren't even involved. So that'll help narrow down where the actual issue lies. From sklein at cpcug.org Tue Jul 31 01:37:06 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Mon, 30 Jul 2007 19:37:06 -0400 (EDT) Subject: [Distutils] (no subject) Message-ID: <3600.207.188.248.157.1185838626.squirrel@www.cpcug.org> Phillip J. Eby wrote: > At 03:15 PM 7/30/2007 -0400, Stanley A. Klein wrote: >>> I don't need to build the .so files. They are already built. That had to >>> be done using the build-in-place and the numpy distutils for reasons I >>> don't fully understand but are related to the use of numpy. > >> Have you tried building them with setuptools, using the numpy >> distutils 'build_ext' command, using: >> >> setup( >> cmdclass = dict(build_ext = numpy_build_ext_class_here), >> ext_modules = list_of_Extension_objects, >> ... >> ) >> >> Unless there is a radical difference between numpy distutils and the >> regular distutils, you should be able to do this. Just find numpy's >> "build_ext" class, and define the appropriate Extension() objects >> (for the things to build) in your setup script. Setuptools will then >> delegate the building to numpy, but handle the installing itself. > >> Again, this is assuming that numpy's distutils extensions don't do >> anything unfriendly like completely redefine how extension objects >> work or assume that their commands will be only mixed with other >> numpy commands. (Setuptools doesn't make such assumptions, and tries >> to leave the normal distutils stuff alone as much as possible.) >I think we're getting into confusing territory by trying to get workarounds >for workarounds. Let me try to take us a step back and focus on the initial >problem which is that bdist_rpm is not working with enthought.kiva. The >existing setup script already does build extensions just fine; they're just >not being picked up by bdist_rpm. A suggestion from a coworker of mine >prompted Stanley to look at using a script that we have for building >enthought.kiva inplace (there are a few more options that are needed beyond >"python setup.py develop"); however, it wasn't really a suggestion to use >that as basis for building an RPM. > >numpy.distutils extends distutils in three ways which are important for >enthought.kiva: > > * automatically adds the location of the numpy headers to the >include_dirs of Extensions. (easily replaced) > > * adds a build_src command that allows users to give Python functions in >the sources list of an Extension. These functions will be called to >actually >generate the real source files. (hard to replace) > > * allows subpackages to have their own build information which is >assembled by the top-level setup.py script. This is mostly legacy from when >the enthought package was monolithic and doesn't strictly need to continue. >I won't go into details since I don't think it's part of the problem. >(straightforward, but time-consuming to replace) >numpy.distutils tries hard to not step on setuptools' toes. We actually >check if setuptools is in sys.modules and use its command classes instead >of distutils' as the base classes for our commands. However, it's possible >that neglect of our bdist_rpm implementation has caused the implementations >to diverge and some toe-stepping has taking place. >The main problem is that bdist_rpm is not working on enthought.kiva. Most >likely, this is the fault of numpy.distutils. However, this is a bug that >needs to be caught and fixed. Working around it by doing an --inplace build >and then trying to include the extension modules as package_data is not >likely to work and is not a solution. > >I'm not usually a Redhat guy, so I don't have much experience with >bdist_rpm; however, numpy.distutils has had problems with bdist_rpm in the >past. I'm trying to get an environment working on a Redhat machine, and >will try to build an RPM for enthought.kiva and try to see the problem >first-hand. I've looked over Stanley's emails on the subject, and don't see >enough information for me to really pin down the problem. Robert - Thanks for illuminating the issue. The problem I had was as follows. Fedora (also RedHat) uses SE-Linux, which needs to know all the files expected to be in sensitive directories such as the Python site-packages. This includes the pyc and pyo files ordinarily generated by Python as the .py files are executed. It turns out that to do a bdist_rpm for Fedora, it is necessary to create a setup.cfg file containing the lines: [install] optimize = 1 or to add these lines to the existing setup.cfg file. If that is not done, the result in Fedora is an unpackaged files error. This is due to the fact that if distutils/setuptools doesn't cause the pyc and pyo files to be created, Fedora will create them but they won't be properly handled in the spec file created by distutils/setuptools. In trying to do bdist_rpm with kiva, I got the unpackaged files error. This implies that numpy distutils did not properly handle the optimize=1 in the setup.cfg (when I did "python setup.py bdist_rpm"). That's when I went to the workaround that resulted in this thread. I hope this clarifies the problem. Thanks. Stan Klein From robert.kern at gmail.com Tue Jul 31 01:34:00 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 30 Jul 2007 18:34:00 -0500 Subject: [Distutils] (no subject) In-Reply-To: <3600.207.188.248.157.1185838626.squirrel@www.cpcug.org> References: <3600.207.188.248.157.1185838626.squirrel@www.cpcug.org> Message-ID: Stanley A. Klein wrote: > Robert - > > Thanks for illuminating the issue. > > The problem I had was as follows. Fedora (also RedHat) uses SE-Linux, > which needs to know all the files expected to be in sensitive directories > such as the Python site-packages. This includes the pyc and pyo files > ordinarily generated by Python as the .py files are executed. > > It turns out that to do a bdist_rpm for Fedora, it is necessary to create > a setup.cfg file containing the lines: > > [install] > optimize = 1 > > or to add these lines to the existing setup.cfg file. > > If that is not done, the result in Fedora is an unpackaged files error. > This is due to the fact that if distutils/setuptools doesn't cause the pyc > and pyo files to be created, Fedora will create them but they won't be > properly handled in the spec file created by distutils/setuptools. > > In trying to do bdist_rpm with kiva, I got the unpackaged files error. > This implies that numpy distutils did not properly handle the optimize=1 > in the setup.cfg (when I did "python setup.py bdist_rpm"). That's when I > went to the workaround that resulted in this thread. > > I hope this clarifies the problem. Not quite. I don't know what "the unpackaged files error" looks like. Can you try Phillip's suggestion using --root and --record and show us the results? Did you run "python setup.py build" before "python setup.py bdist_rpm"? We've often seen problems with the dependency-handling between distutils commands. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From sklein at cpcug.org Tue Jul 31 01:57:27 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Mon, 30 Jul 2007 19:57:27 -0400 (EDT) Subject: [Distutils] How can I get setuptools to include files beyond what find_packages() finds? Message-ID: <3617.207.188.248.157.1185839847.squirrel@www.cpcug.org> Oops, I forgot to copy the subject line. Phillip J. Eby wrote: > At 03:15 PM 7/30/2007 -0400, Stanley A. Klein wrote: >>> I don't need to build the .so files. They are already built. That had to >>> be done using the build-in-place and the numpy distutils for reasons I don't fully understand but are related to the use of numpy. > >> Have you tried building them with setuptools, using the numpy >> distutils 'build_ext' command, using: >> >> setup( >> cmdclass = dict(build_ext = numpy_build_ext_class_here), ext_modules = list_of_Extension_objects, >> ... >> ) >> >> Unless there is a radical difference between numpy distutils and the regular distutils, you should be able to do this. Just find numpy's "build_ext" class, and define the appropriate Extension() objects (for the things to build) in your setup script. Setuptools will then delegate the building to numpy, but handle the installing itself. > >> Again, this is assuming that numpy's distutils extensions don't do anything unfriendly like completely redefine how extension objects work or assume that their commands will be only mixed with other numpy commands. (Setuptools doesn't make such assumptions, and tries to leave the normal distutils stuff alone as much as possible.) >I think we're getting into confusing territory by trying to get workarounds >for workarounds. Let me try to take us a step back and focus on the initial >problem which is that bdist_rpm is not working with enthought.kiva. The >existing setup script already does build extensions just fine; they're just >not being picked up by bdist_rpm. A suggestion from a coworker of mine >prompted Stanley to look at using a script that we have for building >enthought.kiva inplace (there are a few more options that are needed beyond >"python setup.py develop"); however, it wasn't really a suggestion to use >that as basis for building an RPM. > >numpy.distutils extends distutils in three ways which are important for enthought.kiva: > > * automatically adds the location of the numpy headers to the >include_dirs of Extensions. (easily replaced) > > * adds a build_src command that allows users to give Python functions in >the sources list of an Extension. These functions will be called to >actually >generate the real source files. (hard to replace) > > * allows subpackages to have their own build information which is >assembled by the top-level setup.py script. This is mostly legacy from when >the enthought package was monolithic and doesn't strictly need to continue. >I won't go into details since I don't think it's part of the problem. >(straightforward, but time-consuming to replace) >numpy.distutils tries hard to not step on setuptools' toes. We actually check if setuptools is in sys.modules and use its command classes instead >of distutils' as the base classes for our commands. However, it's possible >that neglect of our bdist_rpm implementation has caused the implementations >to diverge and some toe-stepping has taking place. >The main problem is that bdist_rpm is not working on enthought.kiva. Most likely, this is the fault of numpy.distutils. However, this is a bug that needs to be caught and fixed. Working around it by doing an --inplace build >and then trying to include the extension modules as package_data is not >likely to work and is not a solution. > >I'm not usually a Redhat guy, so I don't have much experience with bdist_rpm; however, numpy.distutils has had problems with bdist_rpm in the >past. I'm trying to get an environment working on a Redhat machine, and >will try to build an RPM for enthought.kiva and try to see the problem >first-hand. I've looked over Stanley's emails on the subject, and don't see >enough information for me to really pin down the problem. Robert - Thanks for illuminating the issue. The problem I had was as follows. Fedora (also RedHat) uses SE-Linux, which needs to know all the files expected to be in sensitive directories such as the Python site-packages. This includes the pyc and pyo files ordinarily generated by Python as the .py files are executed. It turns out that to do a bdist_rpm for Fedora, it is necessary to create a setup.cfg file containing the lines: [install] optimize = 1 or to add these lines to the existing setup.cfg file. If that is not done, the result in Fedora is an unpackaged files error. This is due to the fact that if distutils/setuptools doesn't cause the pyc and pyo files to be created, Fedora will create them but they won't be properly handled in the spec file created by distutils/setuptools. In trying to do bdist_rpm with kiva, I got the unpackaged files error. This implies that numpy distutils did not properly handle the optimize=1 in the setup.cfg (when I did "python setup.py bdist_rpm"). That's when I went to the workaround that resulted in this thread. I hope this clarifies the problem. Thanks. Stan Klein -- Stanley A. Klein, D.Sc. Managing Principal Open Secure Energy Control Systems, LLC 8070 Georgia Avenue Silver Spring, MD 20910 301-565-4025 From philipp at weitershausen.de Tue Jul 31 08:48:35 2007 From: philipp at weitershausen.de (Philipp von Weitershausen) Date: Tue, 31 Jul 2007 08:48:35 +0200 Subject: [Distutils] Best practices on distributing documentation and examples with eggs? In-Reply-To: <20070726194836.89D0B3A403D@sparrow.telecommunity.com> References: <46A854FA.3080906@enthought.com> <20070726155222.18CB43A40D7@sparrow.telecommunity.com> <660761AD-A57D-4D81-A0F5-E328043D8B34@python.org> <20070726194836.89D0B3A403D@sparrow.telecommunity.com> Message-ID: <46AEDB43.6020104@weitershausen.de> Phillip J. Eby wrote: > At 02:55 PM 7/26/2007 -0400, Barry Warsaw wrote: > >> On Jul 26, 2007, at 11:52 AM, Phillip J. Eby wrote: >> >>> There are a few different ways you could do this. The easiest would >>> be to put a docs subdirectory either inside one of your packages or >>> inside your .egg-info directory, then use the pkg_resources resource >>> or metadata APIs to list and extract them. One advantage to using >>> something like .egg-info/docs would be that this could perhaps be >>> recognized by some sort of "standard" tools in the future. >> I always thought .egg-info was just an artifact of the build. > > No - it's also a place for metadata that doesn't live under a > specific package directory. > >> Recommending that things get put there means it should get version >> controlled. > > Yep. But isn't .egg-info completely recreatable through setup.py? Given that all it really does is present information from setup.py in a different manner, I've always been putting it on my svn:ignore list and only version-controlled the canonical source -- setup.py. That's becaue I can see and have seen the information get out of synch easily, e.g. if somebody changes setup.py but forgets to run setup.py egg_info. -- http://worldcookery.com -- Professional Zope documentation and training From pje at telecommunity.com Tue Jul 31 18:31:33 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 31 Jul 2007 12:31:33 -0400 Subject: [Distutils] Best practices on distributing documentation and examples with eggs? In-Reply-To: <46AEDB43.6020104@weitershausen.de> References: <46A854FA.3080906@enthought.com> <20070726155222.18CB43A40D7@sparrow.telecommunity.com> <660761AD-A57D-4D81-A0F5-E328043D8B34@python.org> <20070726194836.89D0B3A403D@sparrow.telecommunity.com> <46AEDB43.6020104@weitershausen.de> Message-ID: <20070731162913.1125B3A40F7@sparrow.telecommunity.com> At 08:48 AM 7/31/2007 +0200, Philipp von Weitershausen wrote: >But isn't .egg-info completely recreatable through setup.py? Not if you're putting other data there, no. Chandler, for example, has an i18n framework that stores message catalogs and other localized resources there. Of course, it's certainly possible to use egg_info.writers plugins that would copy those things there, but it's a lot easier for development if you don't, as it would require a build step every time you changed something. From sklein at cpcug.org Tue Jul 31 21:34:23 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Tue, 31 Jul 2007 15:34:23 -0400 (EDT) Subject: [Distutils] How can I get setuptools to include files beyond what find_packages() finds? Message-ID: <54560.207.188.248.157.1185910463.squirrel@www.cpcug.org> Robert Kern wrote: Stanley A. Klein wrote: >> Robert - >> >> Thanks for illuminating the issue. >> >> The problem I had was as follows. Fedora (also RedHat) uses SE-Linux, which needs to know all the files expected to be in sensitive directories >> such as the Python site-packages. This includes the pyc and pyo files ordinarily generated by Python as the .py files are executed. >> >> It turns out that to do a bdist_rpm for Fedora, it is necessary to create >> a setup.cfg file containing the lines: >> >> [install] >> optimize = 1 >> >> or to add these lines to the existing setup.cfg file. >> >> If that is not done, the result in Fedora is an unpackaged files error. >> This is due to the fact that if distutils/setuptools doesn't cause the pyc >> and pyo files to be created, Fedora will create them but they won't be properly handled in the spec file created by distutils/setuptools. >> >> In trying to do bdist_rpm with kiva, I got the unpackaged files error. This implies that numpy distutils did not properly handle the optimize=1 >> in the setup.cfg (when I did "python setup.py bdist_rpm"). That's when I >> went to the workaround that resulted in this thread. >> >> I hope this clarifies the problem. >Not quite. I don't know what "the unpackaged files error" looks like. Can you >try Phillip's suggestion using --root and --record and show us the results? Did >you run "python setup.py build" before "python setup.py bdist_rpm"? We've often >seen problems with the dependency-handling between distutils commands. I tried your second suggestion and it didn't work. Running bdist_rpm starts the whole process from scratch. Regarding Phillip's suggestion, when running bdist_rpm I don't have direct access to --root and --record. However, in trying to track it down, I think I may have found something that is contributing to the problem. When running bdist_rpm, the install command options appear to be set in a script located in the %install part of the rpm spec file. Most likely, the install is called by the rpm program and is controlled by the spec file. The install script in the kiva spec file is "python setup.py install --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES". The install scripts in the other spec files (that worked) are "python setup.py install --single-version-externally-managed --root= $RPM_BUILD_ROOT --record=INSTALLED_FILES" I looked at the numpy distutils. In many cases they check to see if setuptools is installed, and if so they import the relevant setuptools modules (otherwise they import the distutils modules). In particular, the install command does that. However, in the numpy distutils, the bdist_rpm command does not. It only uses the distutils bdist_rpm, that produces the spec file install script shown above. It seems to me that the setuptools install is being called via the numpy distutils without the --single-version-externally-managed option it would expect if its own bdist_rpm were being used to build the spec file. I don't know if that is enough to get the install to ignore including the pyc and pyo files in the INSTALLED_FILES as essentially specified in the setup.cfg option, but it is clearly a glitch that could cause unexpected behavior. The issue with the .so files arose because of my effort to work around the issue of the pyc and pyo files. I think the proper choice of bdist_rpm to import may be closer to the cause of the original problem. I tried to do something to fix the numpy distutils bdist_rpm.py (by trying to follow what was done in install.py) but it didn't work and I got an error message I didn't understand. Stan Klein -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070731/e6fe2c46/attachment.html From robert.kern at gmail.com Tue Jul 31 21:36:24 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 31 Jul 2007 14:36:24 -0500 Subject: [Distutils] How can I get setuptools to include files beyond what find_packages() finds? In-Reply-To: <54560.207.188.248.157.1185910463.squirrel@www.cpcug.org> References: <54560.207.188.248.157.1185910463.squirrel@www.cpcug.org> Message-ID: Stanley A. Klein wrote: > I tried to do something to fix the numpy distutils bdist_rpm.py (by > trying to follow what was done in install.py) but it didn't work and I > got an error message I didn't understand. I'd like to help, but if you don't copy the exact error message here, I can't. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From sklein at cpcug.org Tue Jul 31 23:16:08 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Tue, 31 Jul 2007 17:16:08 -0400 (EDT) Subject: [Distutils] How can I get setuptools to include files beyond what find_packages() finds? Message-ID: <4235.207.188.248.157.1185916568.squirrel@www.cpcug.org> Robert Kern wrote: Stanley A. Klein wrote: >> I tried to do something to fix the numpy distutils bdist_rpm.py (by >> trying to follow what was done in install.py) but it didn't work and I >> got an error message I didn't understand. >I'd like to help, but if you don't copy the exact error message here, I can't. OK, here is my revised numpy/distutils/commands/bdist_rpm.py (trying -- obviously not well -- to follow what was done in numpy/distutils/commands/install.py: import os import sys if 'setuptools' in sys.modules: import setuptools.command.bdist_rpm as old_bdist_rpm class bdist_rpm(old_bdist_rpm): pass else: from distutils.command.bdist_rpm import bdist_rpm as old_bdist_rpm class bdist_rpm(old_bdist_rpm): def _make_spec_file(self): spec_file = old_bdist_rpm._make_spec_file(self) # Replace hardcoded setup.py script name # with the real setup script name. setup_py = os.path.basename(sys.argv[0]) if setup_py == 'setup.py': return spec_file new_spec_file = [] for line in spec_file: line = line.replace('setup.py',setup_py) new_spec_file.append(line) return new_spec_file And here is the error message I got (using the kiva setup.py): [stan at localhost enthought.kiva_2.0]$ python setup.py bdist_rpm Traceback (most recent call last): File "setup.py", line 2, in ? from numpy.distutils.core import setup File "/usr/lib/python2.4/site-packages/numpy/distutils/core.py", line 32, in ? from numpy.distutils.command import bdist_rpm File "/usr/lib/python2.4/site-packages/numpy/distutils/command/bdist_rpm.py", line 6, in ? class bdist_rpm(old_bdist_rpm): TypeError: Error when calling the metaclass bases module.__init__() takes at most 2 arguments (3 given) BTW, below is what the unpackaged files error looks like (with a traceback generated by having the DISTUTILS_DEBUG environment variable set). In a much earlier thread I learned that this can usually be resolved by setting [install] optimize=1 in setup.cfg. --------------------------------------------------------------------------- Installed (but unpackaged) file(s) found: /usr/lib/python2.4/site-packages/enthought/__init__.pyc /usr/lib/python2.4/site-packages/enthought/__init__.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/benchmark.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/benchmark.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/conv.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/conv.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/dash.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/dash.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/lion.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/lion.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/lion_data.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/lion_data.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/polygon_hit_test.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/polygon_hit_test.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/rect.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/rect.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/simple.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/simple.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/simple2.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/simple2.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/simple_clip.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/simple_clip.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/star.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/star.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/star1.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/star1.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/star2.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/star2.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/star_path.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/star_path.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/sub_path.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/sub_path.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/text_ex.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/text_ex.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/tk_ex.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/tk_ex.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/wx_brain.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/wx_brain.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/wx_demo.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/wx_demo.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/wx_image_copy.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/wx_image_copy.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/wx_lines.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/wx_lines.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/wx_lion.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/wx_lion.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/affine_matrix_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/affine_matrix_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/clip_to_rect_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/clip_to_rect_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/compiled_path_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/compiled_path_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/gcmemtest.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/gcmemtest.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/graphics_context_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/graphics_context_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/image_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/image_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/join_stroke_path_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/join_stroke_path_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/points_in_polygon_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/points_in_polygon_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/rgba_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/rgba_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/stroke_path_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/stroke_path_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_arc.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_arc.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_draw_dash.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_draw_dash.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_ft.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_ft.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_graphics_context_system.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_graphics_context_system.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_image3.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_image3.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_save.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_save.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_utils.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_utils.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/unicode_font_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/unicode_font_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/tests/affine_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/tests/affine_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/tests/basecore2d_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/tests/basecore2d_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/tests/mac_simpletest.pyc /usr/lib/python2.4/site-packages/enthought/kiva/tests/mac_simpletest.pyo /usr/lib/python2.4/site-packages/enthought/kiva/tests/macport_test.pyc /usr/lib/python2.4/site-packages/enthought/kiva/tests/macport_test.pyo Traceback (most recent call last): File "setup.py", line 108, in ? version = '2.0b1', File "/usr/lib/python2.4/site-packages/numpy/distutils/core.py", line 174, in setup return old_setup(**new_attr) File "/usr/lib/python2.4/distutils/core.py", line 149, in setup dist.run_commands() File "/usr/lib/python2.4/distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/usr/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/usr/lib/python2.4/distutils/command/bdist_rpm.py", line 365, in run self.spawn(rpm_cmd) File "/usr/lib/python2.4/distutils/cmd.py", line 398, in spawn spawn(cmd, search_path, dry_run= self.dry_run) File "/usr/lib/python2.4/distutils/spawn.py", line 37, in spawn _spawn_posix(cmd, search_path, dry_run=dry_run) File "/usr/lib/python2.4/distutils/spawn.py", line 165, in _spawn_posix raise DistutilsExecError, \ distutils.errors.DistutilsExecError: command 'rpmbuild' failed with exit status 1 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070731/ce4bc621/attachment.htm From robert.kern at gmail.com Tue Jul 31 23:55:14 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 31 Jul 2007 16:55:14 -0500 Subject: [Distutils] How can I get setuptools to include files beyond what find_packages() finds? In-Reply-To: <4235.207.188.248.157.1185916568.squirrel@www.cpcug.org> References: <4235.207.188.248.157.1185916568.squirrel@www.cpcug.org> Message-ID: Stanley A. Klein wrote: > Robert Kern wrote: > Stanley A. Klein wrote: > >>> I tried to do something to fix the numpy distutils bdist_rpm.py (by >>> trying to follow what was done in install.py) but it didn't work and I >>> got an error message I didn't understand. > >> I'd like to help, but if you don't copy the exact error message here, I > can't. > > OK, here is my revised numpy/distutils/commands/bdist_rpm.py (trying -- > obviously not well -- to follow what was done in > numpy/distutils/commands/install.py: > > import os > import sys > > if 'setuptools' in sys.modules: > import setuptools.command.bdist_rpm as old_bdist_rpm > class bdist_rpm(old_bdist_rpm): > pass > > else: > from distutils.command.bdist_rpm import bdist_rpm as old_bdist_rpm Note the difference between these two lines. You imported a module, not the class inside it. Try this instead: if 'setuptools' in sys.modules: from setuptools.command.bdist_rpm import bdist_rpm as old_bdist_rpm else: from distutils.command.bdist_rpm import bdist_rpm as old_bdist_rpm -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco